Tracking and Analyzing Student Success Data Part 3

Tracking and Analyzing Student Success Data

Part 3

State/Territory Area (km2)

Afghanistan 645,807 606,467

Albania 28,748 25,643

Algeria 2,381,741 2,556,276

American Samoa 197 205

Andorra 464 577

Anguilla 96 88

Antigua & Barbuda 442 498

Argentina 2,777,409 2,537,555

Armenia 29,743 25,859

Aruba 193 210

Australia 7,682,557 7,365,389

Austria 83,858 81,434

Azerbaijan 86,530 91,752

Bahamas 13,962 9,125

Bahrain 694 725

Bangladesh 142,615 98,722

Barbados 431 517

Belgium 30,518 25,417

Belize 22,965 20,648

Benin 112,620 108,768

… … …

Which column lists the correct areas?

A Bit of History

Late 1800s: Astronomer Simon Newcomb noticed that the early pages of log table books were more grubby than the later pages Users were looking-up numbers that started with digit 1 more often than numbers starting with, say, digit 5.

If the leading (first) digit is d, then the frequency of occurrence (probability) of the leading digit is

Log10 (1 + 1/d)

If the leading (first) digit is d, then the frequency of occurrence (probability) of the leading digit is

Log10 (1 + 1/d)

Leading digit (d)

1 2 3 4 5 6 7 8 9

Probability of

occurrence

30% 18% 12% 10% 8% 7% 6% 5% < 5%

Number Leading (first) digit

350 3

42057 4

0.64 6

A Bit of History (cont’d)

Late 1930s: Physicist Frank Benford rediscovered Newcomb’s formula: Log10 (1+1/d) Benford’s Law

Source: The Law of Anomalous Numbers, F. Benford, Proceedings of the American Philosophical Society, Vol. 78, 1938, pp: 551-572.

Source: The Law of Anomalous Numbers, F. Benford, Proceedings of the American Philosophical Society, Vol. 78, 1938, pp: 551-572.



Albania 28,748 25,643

Algeria 2,381,741 2,556,276


Andorra 464 577

Anguilla 96 88


Argentina 2,777,409 2,537,555

Armenia 29,743 25,859

Aruba 193 210

Australia 7,682,557 7,365,389

Austria 83,858 81,434


Bahamas 13,962 9,125

Bahrain 694 725


Barbados 431 517

Belgium 30,518 25,417

Belize 22,965 20,648

Benin 112,620 108,768

… … …



Albania 28,748 25,643

Algeria 2,381,741 2,556,276


Andorra 464 577

Anguilla 96 88


Argentina 2,777,409 2,537,555

Armenia 29,743 25,859

Aruba 193 210

Australia 7,682,557 7,365,389

Austria 83,858 81,434


Bahamas 13,962 9,125

Bahrain 694 725


Barbados 431 517

Belgium 30,518 25,417

Belize 22,965 20,648

Benin 112,620 108,768

… … …

Correct List

Sou

rce:

Impr

ovis

ed fr

om R

. M. F

ewst

er, T

he A

mer

ican

Sta

tistic

ian,

Feb

. 200

9, V

ol. 6

3, N

o. 1

.

Benford’s Law

in

MasteringTM

Rasil Warnakulasooriya

Pearson, Boston

A Quick Demo of MasteringTM

masteringphysics.com

Thanks to Claire Masson for the Demo slides

A Tutorial Problem in MasteringPhysics®

Student enters 0 to Part A

Students can open Hints

Declarative Hint

Procedural Hint

Feedback within Hints

Student obtains the correct answers

Time to Completion Data

Student Item Time interval (sec.)

[finished time-start

time]

First (leading) digit

S1 Tutorial X 350 3

S1 EOC Y 1048 1

S3 EOC Y 67 6

S4 Tutorial Z 42179 4

… … … …

MasteringPhysics® time data follow Benford’s Law

closely

1 2 3 4 5 6 7 8 90.00

0.05

0.10

0.15

0.20

0.25

0.30

Pro

babili

ty

First Digit

Observed Benford's Law

= 0.006

MasteringChemistry® time data follow Benford’s Law closely (better than in

physics)

1 2 3 4 5 6 7 8 90.00

0.05

0.10

0.15

0.20

0.25

0.30 = 0.002 P

robabili

ty

First Digit


But what can we do with it?

What can the deviations from the Benford’s Law tell

us?

Deviation from Benford’s Law

1 2 3 4 5 6 7 8 90.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35Student 1

Pro

babili

ty

First Digit

Benford's Law Observed

less deviation

1 2 3 4 5 6 7 8 90.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Pro

bab

ility

First Digit

Benford's Law ObservedStudent 2

more deviation

Final Exam Score vs. Deviation from Benford’s Law

(Chemistry)

Thanks to professors Randall Hall & Leslie Butler for providing the exam score data

Final Exam Score vs. Deviation from Benford’s Law

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40

-3

-2

-1

0

1

2

3

Correlation = -0.2095% CI [-0.34, -0.10]

Fin

al E

xam

Sco

re(in s

tandard

devi

atio

n u

nits

)

Deviation from Benford's Law

For every 1/5th of deviation from Benford’s Law, the exam score is decreased by 1/2 standard deviation

To what degree is the average time

correlated with the final exam scores?

-0.03

(about 7 times less than the correlation found by using only the first digits; almost no correlation)

The correlation (-0.2) is small but surprisingly good (?)

* Typical correlations encountered in educational research ~ 0.3 * High school GPA & SAT®-math correlated at 0.23

* SAT®-math and first-year college GPA correlated at 0.26

Source: College Board Research Report No 2008-5: Validity of the SAT® for Predicting First-Year College Grade Point Average, Jennifer L. Kobrin, et al.

Can we identify at-risk students?

Using average time: 0% of the students can be flagged

Students who sat for the final exam

Students who did not sit for the final exam

Using Benford’s Law: ~19% of the students can be flagged

Students who sat for the final exam

Students who did not sit for the final exam

* In time to completion data the first digit (or deviations from the Benford’s Law) carries more actionable information than all of the digits combined!

Summary

* Possibility of weighting the deviations from Benford's Law to identify at-risk students so that instructors can intervene effectively

* Possibility of weighting the deviations from Benford’s Law in deciding on specific study plans/personalized learning resources within Mastering

State/Territory Area (miles2)


Albania 11,100 9,901

Algeria 919,595 986,983


Andorra 179 223

Anguilla 37 34


Argentina 1,072,363 979,755

Armenia 11,484 9,984

Aruba 75 81

Australia 2,966,250 2,843,791

Austria 32,378 31,442


Bahamas 5,391 3,523

Bahrain 268 280


Barbados 166 200

Belgium 11,783 9,814

Belize 8,867 7,972

Benin 43,483 41,996

… … …

1 2 3 4 5 6 7 8 90

5

10

15

20

25

30

35

% o

ccurr

ence

first digit (time to completion in s)


Chemistry

1 2 3 4 5 6 7 8 90

5

10

15

20

25

30

35

% o

ccurr

ence

first digit (time to completion in min)


Chemistry

Documents

Tracking and Analyzing Student Success Data Part 3