9 Nov 2011COMP80131-SEEDSM21 Scientific Methods 1 Barry & Goran ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 2:

9 Nov 2011 COMP80131-SEEDSM2 1

Scientific Methods 1

Barry & Goran

‘Scientific evaluation, experimental design

& statistical methods’

COMP80131

Lecture 2: Statistical Methods-Basics

www.cs.man.ac.uk/~barry/mydocs/myCOMP80131


Scientific Methods 1

• Scientific evaluation: derivation of useful & reliable statements about some new or existing scientific idea based on an accumulation of evidence which is often in the form of tables of numerical values.

• Experimental design: how to generate the quantifiable outputs, the systematic observation & measurement of these outputs and the recording of the resulting data. The experiments are normally designed to test some theoretical prediction of what the researcher expects to happen – a ‘research hypothesis’

• Statistical methods: the means of deriving the required useful and reliable statements from numerical evidence.


Scientific Enquiry• It may be argued that:

– ‘Scientific researchers propose hypotheses as explanations of phenomena & design experimental studies to test these hypotheses’.

• It may also be argued otherwise.• Wider domains of inquiry may combine many independently

derived hypotheses.• Or not have hypotheses at all, other than contrived ones such as:

– ‘This idea can (not) be implemented’


Philosophy of Science• Concerns: the underpinning logic of the scientific method,

what separates science from non-science,

the ethics implicit in science.

• Assumes: reality is objective and consistent,

humans have the capacity to perceive reality accurately,

rational explanations exist for elements of the real world.

• Logical Positivism & other theories claim to have defined the

logic of science, but have all been been challenged.

• Ludwig Wittgenstein (1889-1951) got his PhD in Manchester


Objectivity, repeatability & full disclosure

• Scientific inquiry is intended to be as objective as possible, to reduce biased interpretations of results.

• Procedures must be reproducible (i.e. repeatable)• Researchers should:

– document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them.

• This practice is called ‘full disclosure’.• Allows the methodology & the statistical reliability of the data

to be verified.


References on Statistics

1. DJ Hand ‘Statistics – a very short introduction’ Oxford UP 2008

2. Schaum’s Outlines ‘Prob & Stats’ 2009

3. WG Hopkins ‘A new View of Statistics’ (Google it)

4. ‘Why is my evil lecturer forcing me to learn statistics?’ (Google it – forget it!!)


Tables of ResultsEngli Maths Phys Chem Hist Fren Music Art Avge 81 67 60 104 89 97 72 30 75.0 91 32 42 34 24 65 81 61 53.8 13 123 45 22 92 61 114 11 60.1 91 65 80 23 95 47 101 33 66.9 63 58 44 6 38 58 36 21 40.5 10 28 69 24 84 91 20 102 53.5 28 20 60 18 46 38 -3 79 35.8 55 0 44 85 35 23 11 112 45.6 96 38 49 17 11 42 45 48 43.3 96 21 48 83 80 27 8 101 58.0 16 68 55 35 69 44 40 55 47.8 97 41 64 13 91 63 -13 33 48.6 96 100 34 19 34 53 81 -10 50.9 49 92 70 17 13 39 63 -19 40.5 80 55 58 3 58 87 68 28 54.6 14 42 45 95 63 30 64 46 49.9 42 82 49 19 88 40 42 16 47.3 92 18 53 80 0 52 -17 108 48.3 79 69 53 29 0 6 59 31 40.8 96 31 62 40 77 23 50 65 55.5

A fictitious set of exam results.

A sample of 20 students out of a population of 1000.

Complete file is:

ExamData.xls or ExamData.dat

www.cs.man.ac.uk/~barry


A bit of MATLAB

[Marks,Headings]=xlsread('ExamData.xls');

[nRows,nCols] = size(Marks);

Headings(1,1:nCols))

Marks

Reads in marks from Excel spreadsheet into an array ‘Marks’.

Headings read in separately.

Miss out ‘;’ to display. ‘%’ is comment.


A bit more MATLAB% Row with mean of each column:

Me = mean(Marks)

% Row with standd deviations of cols:

St_devs = std(Marks)

% Row with variances of cols:

Variances = var(Marks)

Statistics printed out: Engli Maths Phys Chem Hist Fren Music Art Avge

Means: 52.2 49.2 49.7 49.6 55.7 51.0 48.4 50.7 50.8Std_devs: 28.2 27.2 10.5 31.5 33.3 28.6 33.4 34.1 8.7Variances: 795 741 110 990 1109 819 1115 1165 75.5


Definitions: mean46850699-423023163860-345 030

Here is a col of marks, say for French.

The mean is the average. It is about 27.

This is a ‘statistic’ which summarizes the column of data.

Alternatives exist: e.g. median & mode

It allows comparisons to be made.

If the average is 31 next year, we can hypothesise that the students are better, better taught or the exam was easier, (or maybe the exam room was warmer).

(Is the increase of 4 statistically significant?)


Definitions: variance46

8

50

6

99

-42

30

23

16

38

60

-3

45

0

30

On the right is another column. Mean is also 27.

But it is much less ‘spread out’ – its variance is less.

All students are getting close to the same mark.

Maybe the exam is not well designed to test ability.

If there are N marks, subtract the mean from each of them, square them add up the squared values then divide by N-1.

282629253024272628272826252927

N

nn meanx

NVariance

1

2)()1(

1

Another ‘statistic’: 1068 (left) & 2.86 (right)

Measure of ‘spread’


Definitions: std_deviation46

8

50

6

99

-42

30

23

16

38

60

-3

54

0

30

This is the square root of the variance.

Also a measure of ‘spread’

Yet another ‘statistic’: 32.7 (left)

1.69 (right)

Many alternatives exist

282629253024272628272826252927


Population-mean & sample-mean• Simplest statistic is probably the mean or average. • Given a table of 20 marks, average is easily found & understood. • Questions arise if we consider this batch of students to be a ‘sample’

of a much larger ‘population’ of say 1000 students taking exams. • How representative is this batch’s average, called a ‘sample-mean’,

likely to be of the mean for the whole population, i.e.the ‘population mean’?

• A question that arises all the time in statistical methods. • A 2nd example: if there is a population of 50 million people in the

UK, we take a ‘sample’ of 1000 people, measure their heights & compute the average, how close will be this ‘sample mean’ to the true mean for the whole population?

• How reliable will sample-mean be as estimate of population-mean? • Same question can be asked about other statistics, e.g.. variance.


Back to MATLAB• Divide the 1000 marks into batches & compute the sample mean for

each batch.

True Means: 52.2 49.2 49.7 49.6 55.7 51.0 48.4 50.7 50.8------------------------------------------------------------------------------Means: 50.0 58.7 51.0 46.7 43.7 62.3 61.1 36.9 51.3 52.7 Means: 48.5 51.8 57.8 47.2 45.6 47.7 53.7 50.6 48.0 44.5 Means: 49.5 48.6 30.9 53.9 43.7 53.6 46.6 50.4 56.9 48.4 Means: 44.5 68.2 48.1 55.9 48.0 52.5 54.0 42.2 50.3 56.8 Means: 52.2 39.9 38.1 69.9 50.4 61.9 57.2 50.6 49.5 59.8 Means: 59.0 61.5 39.5 54.9 42.6 44.0 50.6 41.0 62.1 48.9 Means: 44.6 56.1 48.7 49.9 44.3 48.4 39.1 52.4 56.6 43.5 Means: 62.8 49.6 55.7 42.9 48.8 42.1 60.7 66.5 41.8 55.2 Means: 51.7 52.3 53.2 48.2 48.1 69.1 49.8 57.0 50.1 53.4 Means: 49.9 47.4 54.1 50.4 67.2 51.6 42.9 56.1 52.5 44.9 Means: 55.8 46.1 48.5 55.8 54.7 54.5 39.3 49.9 43.8 53.1 Means: 50.4 44.1 55.5 46.6 47.8 41.7 47.9 57.5 53.7 51.5 Means: 52.8 67.2 47.8 46.7 53.3 53.8 46.9 51.3 48.5 58.6 Means: 47.0 48.6 56.4 50.3 50.9 56.4 50.0 52.1 42.5 50.5 Means: 54.2 50.0 52.3 51.0 52.3 50.9 50.8 63.5 48.6 58.6 Means: 56.3 51.1 54.0 53.9 64.0 48.8 50.8 44.3 62.2 61.8 Means: 40.9 53.3 52.8 56.9 51.2 61.1 57.6 56.8 50.1 37.6 Means: 53.0 55.9 38.8 47.2 49.0 62.2 49.1 39.4 54.6 49.5 Means: 47.8 51.4 48.2 45.9 48.2 53.6 54.0 43.6 49.1 48.3 Means: 38.9 51.9 52.0 60.7 44.1 44.2 70.8 51.3 49.9 46.8 Means: 52.6 54.9 54.9 50.8 43.8 53.5 50.9 58.3 40.1 48.9 Means: 52.5 68.1 53.3 46.1 60.1 53.4 52.0 48.3 51.5 55.5 Means: 60.0 45.7 45.5 45.7 50.5 51.8 44.8 50.1 54.2 65.9

Sample means for

50 batches of 20

Look at col 1 (Engl)


50 batches of 20 (column 1)

5 10 15 20 25 30 35 40 45 500

10

20

30

40

50

60

70

80

90

100

Batch

Sam

ple

mea

n

Look at spread over all batches for column 1

Remember pop-mean 52.2

Mean (of sample-means) =52.2

Variance = 32


20 batches of 50 (column 1)

2 4 6 8 10 12 14 16 18 200

10

20

30

40

50

60

70

80

90

100

Batch

Sam

ple

mea

n

Variance has reduced.Mean of sample-means = 52.2

Variance = 18.2


10 batches of 100

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

Batch

Sam

ple

mea

n

Mean of sample-means = 52.2

Variance = 7.28


Distributions

• Histogram divides domain (x-axis) into say 10 or 20 regions & plots the number of marks that fall in each region.

• In MATLAB:• figure(1); hist(Marks(:,1),20);• figure(2); hist(Marks(:,2),20);• figure(3); hist(Marks(:,3),20); etc.


Histogram for col 1 (English)

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

Evenly distributed across the domain.

Looks like a ‘uniform’ distribution


Histogram for col 2 (Maths)

-40 -20 0 20 40 60 80 100 120 1400

50

100

150

Looks a bit ‘Gaussian’ or ‘normal’Mean 50


Histogram for col 3 (Phys)

10 20 30 40 50 60 70 80 900

20

40

60

80

100

120

140

Also looks ‘Gaussian’

Mean 50 with smaller variance


Histogram for col 4 (Chem)

-20 0 20 40 60 80 100 1200

20

40

60

80

100

120

140

Bi-modal distribution


Column 5(Hist)

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

A bit strange


Col 6 (French)

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

Uniform again?


Column 7 (Music)

-50 0 50 100 150 2000

20

40

60

80

100

120

Gaussian again?


Col 8 (Art)

-100 -50 0 50 100 150 2000

20

40

60

80

100

120

140

Gaussian again?


Col 9 (Average)

20 30 40 50 60 70 800

20

40

60

80

100

120

Gaussian?


Some questions for you

•Analyse the ficticious exam results & comment on features.•Compute means, stds & vars for each subject & histograms for the distributions.•Make observations about performance in each subject & overall•Do marks support the hypothesis that people good at Music are also good at Maths?•Do they support the hypothesis that people good at English are also good at French?•Do they support the hypothesis that people good at Art are also good at Maths?•If you have access to only 50 rows of this data, investigate the same hypotheses•What conclusions could you draw, and with what degree of certainty?


Correlation

• Measure of how two columns are related.• Let cols be x and y:• Correlation coefficient:

yx

N

nynxn meanymeanx

varvar

))((1


Scatter plot col 1 against col 1

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

col 1

col 1

Corr coeff = 1

Positive correlation


Scatter plot col 1 against -col 1

Corr-coeff = -1

Negative correlation

0 10 20 30 40 50 60 70 80 90 100-100

-90

-80

-70

-60

-50

-40

-30

-20

-10

0

col 1

col 1


Scatter plot col 1(Eng) – col 2(Maths)

0 10 20 30 40 50 60 70 80 90 100-40

-20

0

20

40

60

80

100

120

140

Corr coeff = 0.04

(close to zero)

Very weak or no correlation


Scatter plot col 2(Maths) – col 7(Mus)

Corr coeff = 0.8

(strong +ve corr)

-50 0 50 100 150 200-40

-20

0

20

40

60

80

100

120

140

col 7

col 2


Scatter plot col 2(Maths) – col 8(Art)

-40 -20 0 20 40 60 80 100 120 140-100

-50

0

50

100

150

200

col 2

col 8

Corr coeff = -0.8

Strong –ve correlation


Correlation

In MATLAB: corr(Marks)

1.00 -0.037 -0.029 -0.068 -0.04 0.012 -0.015 0.013 0.34

-0.037 1.00 -0.0014 0.051 -0.033 0.003 0.79 -0.82 0.365

-0.029 -0.0014 1.00 -0.042 0.03 0.009 0.017 0.011 0.15

-0.068 0.051 -0.042 1.00 -0.013 -0.055 0.048 -0.031 0.42

-0.04 -0.033 0.03 -0.013 1.00 -0.053 0.002 -0.006 0.43

0.012 0.003 0.009 -0.055 -0.053 1.00 -0.004 -0.009 0.363

-0.015 0.79 0.017 0.0476 0.0021 -0.004 1.00 -0.66 0.48

0.013 -0.82 0.011 -0.031 -0.0061 -0.009 -0.66 1.00 -0.16

0.34 0.37 0.15 0.42 0.43 0.363 0.48 -0.16 1.00

Documents

9 Nov 2011COMP80131-SEEDSM21 Scientific Methods 1 Barry & Goran ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 2: