Upload
lillian-arnold
View
232
Download
0
Embed Size (px)
Citation preview
Inferential StatisticsStandard Error of the MeanSignificanceInferential tests you can use
1
Pengenalan kepada Statistik Inferensi
Perhatikan simbol/formula
2
t =
XA
—XB
—
XA 2 XA 2( )
n1
- XB 2 XB 2( )
n2
-( ) ( )+[ ]1
n1
1
n2
+( )x
-
(n1-1) + (n2-1)
3
Don’t Panic !
Don’t Panic !
t =XA
—XB
—
XA 2 XA 2( )
n1
- XB 2 XB 2( )
n2
-( ) ( )+[ ]1
n1
1
n2
+( )x
-
Compare with SD formula
(n1-1) + (n2-1)
Difference betweenmeans
Basic types of statistical treatment
4
o Descriptive statistics which summarize the characteristics of a sample of data
o Inferential statistics which attempt to say something about a population on the basis of a sample of data - infer to all on the basis of some
Statistical tests are inferentialStatistical tests are inferential
Two kinds of descriptive statistic:
5
o Measures of central tendency– mean
– median – mode
o Measures of dispersion (variation)– range – inter-quartile range– variance/standard deviation
Or where about on the measurement scale most of the data fall
Or where about on the measurement scale most of the data fall
Or how spread out they are
Or how spread out they are
The different measures have different sensitivity and should be used at the appropriate times…
The different measures have different sensitivity and should be used at the appropriate times…
Mean
6
Sum of all observations divided by the number of observations
In notation:
Mean uses every item of data but is sensitive to extreme ‘outliers’
i = 1
n
xi
n
Refer to handout on notationRefer to handout on notation
See example on next slideSee example on next slide
Variance and standard deviation
7
A deviation is a measure of how far from the mean is a score in our dataSample: 6,4,7,5 mean =5.5Each score can be expressed in terms of distance
from 5.56,4,7,5, => 0.5, -1.5, 1.5, -0.5 (these are distances
from mean)Since these are measures of distance, some are
positive (greater than mean) and some are negative (less than the mean)
TIP: Sum of these distances ALWAYS = 0
To overcome problems with range etc. we need a better measure of spread
To overcome problems with range etc. we need a better measure of spread
Symbol check
x
(x x)
Called ‘x bar’; refers to the ‘mean’
Called ‘x minus x-bar’; implies subtracting the mean from a data point x. also known as a deviation from the mean
8
Two ways to get SD
9
sd (x x)2
n
•Sum the sq. deviations from the mean•Divide by No. of observations•Take the square root of the result
•Sum the squared raw scores•Divide by N•Subtract the squared mean•Take the square root of the result
sd x2
n x
2
x x
2 4 2 4 2 4 2 4 2 4 3 9 3 9 4 16 4 16 5 25
x = 29 x = 952
2
s = -2
n x 2x
= -95
102.9
2
= -9.5 8.41
= 1.09
= 1.044 If we recalculate the variance with the 60 instead of the 5 in the data…
If we recalculate the variance with the 60 instead of the 5 in the data…
x x
2 4 2 4 2 4 2 4 2 4 3 9 3 9 4 16 4 16 60 3600
x = 84 x = 36702
2
s = -2
n x 2x
= -3760
108.4
2
= -367 70.56
= 296.44
= 17.22
If we include a large outlier:
Note increase in SD
Like the mean, the standard deviation uses every piece of data and is therefore sensitive to extreme values
Like the mean, the standard deviation uses every piece of data and is therefore sensitive to extreme values
Mean
Two sets of data can have the same mean but different standard deviations.
The bigger the SD, the more s-p-r-e-a-d out are the data.
On the use of N or N-1
sd (x x)2
n
When your observations are the complete set of people that could be measured (parameter)
When you are observing only a sample of potential users (statistic), the use of N-1 increases size of sd slightly
13
sd (x x)2n 1
Summary
Mode •
Median •
Mean •
Range •
Interquartile Range •
Variance / Standard Deviation •
Most frequent observation. Use with nominal data‘Middle’ of data. Use with ordinal data or when data contain outliers‘Average’. Use with interval and ratio data if no outliers
Dependent on two extreme valuesMore useful than range. Often used with medianSame conditions as mean. With mean, provides excellent summary of data
Measures of Central Tendency
Measures of Dispersion
Deviation units: Z scores
15
z x x
sd
Any data point can be expressed in terms of itsDistance from the mean in SD units:
A positive z score implies a value above the meanA negative z score implies a value below the mean
Interpreting Z scores
16
Mean = 70,SD = 6Then a score of 82
is 2 sd [ (82-70)/6] above the mean, or 82 = Z score of 2
Similarly, a score of 64 = a Z score of -1
By using Z scores, we can standardize a set of scores to a scale that is more intuitive
Many IQ tests and aptitude tests do this, setting a mean of 100 and an SD of 10 etc.
Comparing data with Z scores
17
You score 49 in class A but 58 in class B How can you compare your performance in both?
Class A: Class B:Mean =45 Mean =55SD=4 SD = 6
49 is a Z=1.0 58 is a Z=0.5
With normal distributions
18
Mean, SD and Z tables
In combination provide powerful means of estimating what your data indicates
Graphing data - the histogram
0
10
20
30
40
50
60
70
80
90
100
19
NumberOf errors
The categories of data we are studying, e.g., task or interface, or user group etc.
The frequency of occurrence for measure of interest,e.g., errors, time, scores on a test etc.
1 2 3 4 5 6 7 8 9 10Graph gives instant summary of data - check spread, similarity, outliers, etc.
Graph gives instant summary of data - check spread, similarity, outliers, etc.
Very large data sets tend to have distinct shape:
0
10
20
30
40
50
60
70
80
20
Normal distribution
21
Bell shaped, symmetrical, measures of central tendency convergemean, median, mode are equal in normal
distributionMean lies at the peak of the curve
Many events in nature follow this curveIQ test scores, height, tosses of a fair coin,
user performance in tests,
The Normal Curve
22
MeanMedianMode
50% of scoresfall below meanf
NB: position of measures of central tendency
NB: position of measures of central tendency
Positively skewed distribution
23
Mode Median Mean
f
Note how the various measures of central tendency separate now - note the direction of the change…mode moves left of other two, mean stays highest, indicating frequency of scores less than the mean
Note how the various measures of central tendency separate now - note the direction of the change…mode moves left of other two, mean stays highest, indicating frequency of scores less than the mean
24
Negatively skewed distribution
Mean Median Mode
f
Here the tendency to have higher values more common serves to increase the value of the mode
Here the tendency to have higher values more common serves to increase the value of the mode
Other distributions
25
BimodalData shows 2 peaks separated by trough
MultimodalMore than 2 peaks
The shape of the underlying distribution determines your choice of inferential test
Bimodal
26
f
MeanMedian
Mode Mode
Will occur in situations where there might be distinct groups being tested e.g., novices and experts
Note how each mode is itself part of a normal distribution (more later)
Will occur in situations where there might be distinct groups being tested e.g., novices and experts
Note how each mode is itself part of a normal distribution (more later)
Standard deviations and the normal curve
27
Mean
1 sd
f
1 sd
68% of observationsfall within ± 1 s.d.
95% of observations fallwithin ± 2 s.d. (approx)
1 sd1 sd
Z scores and tables
28
Knowing a Z score allows you to determine where under the normal distribution it occurs
Z score between:
0 and 1 = 34% of observations1 and -1 = 68% of observations etc.
Or 16% of scores are >1 Z score above mean
Check out Z tables in any basic stats book
Remember:
29
A Z score reflects position in a normal distribution
The Normal Distribution has been plotted out such that we know what proportion of the distribution occurs above or below any point
Importance of distribution
30
Given the mean, the standard deviation, and some reasonable expectation of normal distribution, we can establish the confidence level of our findings
With a distribution, we can go beyond descriptive statistics to inferential statistics (tests of significance)
So - for your research:
31
Always summarize the data by graphing it - look for general pattern of distribution
Then, determine the mean, median, mode and standard deviation
From these we know a LOT about what we have observed
Inference is built on Probability
32
Inferential statistics rely on the laws of probability to determine the ‘significance’ of the data we observe.
Statistical significance is NOT the same as practical significance
In statistics, we generally consider ‘significant’ those differences that occur less than 1:20 by chance alone
Calculating probability
33
Probability refers to the likelihood of any given event occurring out of all possible events e.g.:Tossing a coin - outcome is either head or tail
Therefore probability of head is 1/2Probability of two heads on two tosses is 1/4 since the
other possible outcomes are two tails, and two possible sequences of head and tail.
The probability of any event is expressed as a value between 0 (no chance) and 1 (certain)
At this point I ask people to take out a coin and toss it 10 times, noting the exact sequence of outcomes e.g.,
h,h,t,h,t,t,h,t,t,h.
Then I have people compare outcomes….
At this point I ask people to take out a coin and toss it 10 times, noting the exact sequence of outcomes e.g.,
h,h,t,h,t,t,h,t,t,h.
Then I have people compare outcomes….
Sampling distribution for 3 coin tosses
34
0
1
2
3
4
0 heads 1
1 head 3
2 heads 3
3 heads 1
Probability and normal curves
35
Q? When is the probability of getting 10 heads in 10 coin tosses the same as getting 6 heads and 4 tails?HHHHHHHHHHHHTHTHHTHT
Answer: when you specify the precise order of the 6 H/4T sequence:(1/2)10 =1/1024 (specific order)But to get 6 heads, in any order it is:
210/1024 (or about 1:5)
What use is probability to us?
36
It tells us how likely is any event to occur by chance
This enables us to determine if the behavior of our users in a test is just chance or is being affected by our interfaces
Determining probability
37
Your statistical test result is plotted against the distribution of all scores on such a test.
It can be looked up in stats tables or is calculated for you in EXCEL or SPSS etc
This tells you its probability of occurrenceThe distributions have been determined by
statisticians.
Introduce simple stats tables here :
Introduce simple stats tables here :
What is a significance level?
38
In research, we estimate the probability level of finding what we found by chance alone.
Convention dictates that this level is 1:20 or a probability of .05, usually expressed as : p<.05.
However, this level is negotiableBut the higher it is (e.g., p<.30 etc) the more
likely you are to claim a difference that is really just occurring by chance (known as a Type 1 error)
What levels might we chose?
39
In research there are two types of errors we can make when considering probability:Claiming a significant difference when there
is none (type 1 error)Failing to claim a difference where there is
one (type 2 error)The p<.05 convention is the ‘balanced’
case but tends to minimize type 1 errors
Using other levels
40
Type 1 and 2 errors are interwoven, if we lessen the probability of one occurring, we increase the chance of the other.
If we think that we really want to find any differences that exist, we might accept a probability level of .10 or higher
Thinking about p levels
41
The p<.x level means we believe our results could occur by chance alone (not because of our manipulation) at least x/100 timesP<.10 => our results should occur by chance 1 in
10 timesP<.20=> our results should occur by chance 2 in
10 times
Depending on your context, you can take your chances :)
In research, the consensus is 1:20 is high enough…..
Putting probability to work
42
Understanding the probability of gaining the data you have can guide your decisions
Determine how precise you need to be IN ADVANCE, not after you see the result
It is like making a bet….you cannot play the odds after the event!
Sampling error and the mean
43
Usually, our data forms only a small part of all the possible data we could collectAll possible users do not participate in a usability
testEvery possible respondent did not answer our
questions
The mean we observe therefore is unlikely to be the exact mean for the whole populationThe scores of our users in a test are not going to
be an exact index of how all users would perform
I find that this is the hardest part of stats for novices to grasp, since it is the bridge between descriptive and inferential stats…..needs to be explained slowly!!
I find that this is the hardest part of stats for novices to grasp, since it is the bridge between descriptive and inferential stats…..needs to be explained slowly!!
How can we relate our sample to everyone else?
44
Central limit theoremIf we repeatedly sample and calculate means
from a population, our list of means will itself be normally distributed
Holds true even for samples taken from a skewed population distribution
This implies that our observed mean follows the same rules as all data under the normal curve
45
2 4 6 8 10 12 14 16 18
The distribution of the means forms a smaller normal distribution about the true mean:
0 5 10 15 20
05
00
10
00
15
00
z
n = 5
mean of sample means = 10
SD of sample means = 2.41
0 5 10 15 20
02
00
40
06
00
80
0
z
n = 2
mean of sample means = 10
SD of sample means = 4.16
0 5 10 15 20
05
00
10
00
15
00
20
00
z
n = 15
mean of sample means = 10
SD of sample means = 0.87
47
True for skewed distributions too
Mean
f
Plot of means from samples
Here the tendency to have higher values more common serves to increase the value of the mode
Here the tendency to have higher values more common serves to increase the value of the mode
How means behave..
48
A mean of any sample belongs to a normal distribution of possible means of samples
Any normal distribution behaves lawfullyIf we calculate the SD of all these means,
we can determine what proportion (%) of means fall within specific distances of the ‘true’ or population mean
But...
49
We only have a sample, not the population…
We use an estimate of this SD of means known as the Standard Error of the Mean
SE SD
N
Implications
50
Given a sample of data, we can estimate how confident we are in it being a true reflection of the ‘world’ or…
If we test 10 users on an interface or service, we can estimate how much variability about our mean score we will find within the intended full population of users
Example
51
We test 20 users on a new iPad:Mean error score: 10, sd: 4What can we infer about the broader user
population?According to the central limit theorem, our
observed mean (10 errors) is itself 95% likely to be within 2 s.d. of the ‘true’ (but unknown to us) mean of the population
The Standard Error of the Means
52
SE s.d .(sample)
N
4
20
4
4.470.89
If standard error of mean = 0.89
53
Then observed (sample) mean is within a normal distribution about the ‘true’ or population mean:So we can be
68% confident that the true mean=10 0.89 95% confident our population mean = 10 1.78 99% confident it is within 10 2.67
This offers a strong method of interpreting of our data
Issues to note
54
If s.d. is large and/or sample size is small, the estimated deviation of the population means will appear large.e.g., in last example, if n=9, SE mean=1.33 So confidence interval becomes 10 2.66
(i.e., we are now 95% confident that the true mean is somewhere between 7.34 and 12.66.
Hence confidence improves as sample increases and variability lessensOr in other words: the more users you study, the
more sure you can be….!
Exercise:
55
If the mean = 10 and the s.d.=4, what is the 68% confidence interval when we have: 16 users? 9 users?
If the s.d. = 12, and mean is still 10, what is the 95% confidence interval for those N?
Answers:
9-11
8.66-11.33
4-16
2-18
Answers:
9-11
8.66-11.33
4-16
2-18
Exercise answers:
56
If the mean = 10 and the s.d.=4, what is the 68% confidence interval when we have:
16 users?= 9-11 (hint: sd/n = 4/4=1) 9 users? = 8.66-11.33
If the s.d. = 12, and mean is still 10, what is the 95% confidence interval for those N? 16 users: 4-16 (hint: 95% CI implies 2 SE either side of
mean)9 users: 2-18
Recap
57
Summarizing data effectively informs us of central tendencies
We can estimate how our data deviates from the population we are trying to estimate
We can establish confidence intervals to enable us to make reliable ‘bets’ on the effects of our designs on users
Comparing 2 means
58
The differences between means of samples drawn from the same population are also normally distributed
Thus, if we compare means from two samples, we can estimate if they belong to the same parent population
This is the beginning of significance testing
This is the beginning of significance testing
SE of difference between means
59
[x 1 x
2] 2
x 12
x 2
SEdiff .means SE(sample1)2 SE(sample2)2
This lets us set up confidence limits for the differences between the two means
Regardless of population mean:
60
The difference between 2 true measures of the mean of a population is 0
The differences between pairs of sample means from this population is normally distributed about 0
Consider two interfaces:
61
We capture 10 users’ times per task on each.
The results are:
Interface A = mean 8, sd =3Interface B = mean 10, sd=3.5
Q? - is Interface A really different?
How do we tackle this question?
Calculate the SE difference between the means
62
SEa = 3/10 = 0.95
SEb= 3.5/ 10=1.11
SE a-b = (0.952+1.112) = (0.90+1.23)=1.46
Observed Difference between means= 2.0
95% Confidence interval of difference between means is 2 x(1.46) or 2.92 (i.e. we expect to find difference between 0-2.92 by chance alone).
suggests there is no significant difference at the p<.05 level.
But what else?
63
We can calculate the exact probability of finding this difference by chance:Divide observed difference between the means by the SE(diff between means): 2.0/1.46 = 1.37Gives us the number of standard deviation units between two means (Z scores)Check Z table: 82% of observations are within 1.37 sd, 18% are greater; thus the precise sig level of our findings is p<.18.
Thus - Interface A is different, with rough odds of 5:1
Hold it!
64
Didn’t we first conclude there was no significant difference?Yes, no significant difference at p<.05But the probability of getting the differences we
observed by chance was approximately 0.18 Not good enough for science (must avoid type 1 error),
but very useful for making a judgment on designBut you MUST specify levels you will accept BEFORE not
after….
Note - for small samples (n<20) t- distribution is better than z distribution, when looking up probability
Why t?
65
Similar to the normal distributiont distribution is flatter than Z for small
degrees of freedom (n-1), but virtually identical to Z when N>30
Exact shape of t-distribution depends on sample size
Simple t-test:
66
You want all users of a new interface to score at least 70% on an effectiveness test. You test 6 users on a new interface and gain the following scores:
629275688395
Mean = 79.17Sd=13.17
T-test:
67
t 79.17 70
13.17
6
9.17
5.381.71
From t-tables, we can see that this value of t exceeds t value (with 5 d.f.) for p.10 level
So we are confident at 90% level that our new interface leads to improvement
T-test:
68
t 79.17 70
13.17
6
9.17
5.381.71
SE mean
Sample mean
Thus - we can still talk in confidence intervals, e.g., We are 68% confident the mean of population =79.17 5.38
Predicting the direction of the difference
69
Since you stated that you wanted to see if new Interface was BETTER (>70), not just DIFFERENT (< or > 70%), this is asking for a one-sided test….
For a two-sided test, I just want to see if there is ANY difference (better or worse) between A and B.
One tail (directional) test
70
Tester narrows the odds by half by testing for a specific difference
One sided predictions specify which part of the normal curve the difference observed must reside in (left or right)
Testing for ANY difference is known as ‘two-tail’ testing,
Testing for a directional difference (A>B) is known as ‘one-tail’ testing
So to recap
71
If you are interested only in certain differences, you are being ‘directional’ or ‘one-sided’
Under the normal curve, random or chance differences occur equally on both sides
You MUST state directional expectations (hypothesis) in advance
Why would you predict the direction?
72
Theoretical groundsExperience or previous findings suggested the
differencePractical grounds
You redesigned the interface to make it better, so you EXPECT users will perform better….
Alternative and Null Hypotheses
Inferential statistics test the likelihood that the alternative (research) hypothesis (H1) is true and the null hypothesis (H0) is notin testing differences, the H1 would predict
that differences would be found, while the H0 would predict no differences
by setting the significance level (generally at .05), the researcher has a criterion for making this decision
Alternative and Null Hypotheses
If the .05 level is achieved (p is equal to or less than .05), then a researcher rejects the H0 and accepts the H1
If the the .05 significance level is not achieved, then the H0 is retained
Alternative and Null Hypotheses
If the .05 level is achieved (p is equal to or less than .05), then a researcher rejects the H0 and accepts the H1
If the the .05 significance level is not achieved, then the H0 is retained
Degrees of FreedomDegrees of freedom (df) are the way
in which the scientific tradition accounts for variation due to errorit specifies how many values vary
within a statistical testscientists recognize that collecting data can
never be error-freeeach piece of data collected can vary, or
carry error that we cannot account forby including df in statistical computations,
scientists help account for this errorthere are clear rules for how to calculate df
for each statistical test
Inferential Statistics: 5 StepsTo determine if SAMPLE means come from
same population, use 5 steps with inferential statistics1. State Hypothesis
Ho: no difference between 2 means; any difference found is due to sampling errorany significant difference found is not a
TRUE difference, but CHANCE due to sampling error
results stated in terms of probability that Ho is falsefindings are stronger if can reject Ho therefore, need to specify Ho and H1
Steps in Inferential Statistics 2. Level of Significance
Probability that sample means are different enough to reject Ho (.05 or .01)level of probability or level of
confidence
Steps in Inferential Statistics 3. Computing Calculated Value
Use statistical test to derive some calculated value (e.g., t value or F value)
4. Obtain Critical Valuea criterion used based on df and alpha
level (.05 or .01) is compared to the calculated value to determine if findings are significant and therefore reject Ho
Steps in Inferential Statistics
5. Reject or Fail to Reject HoCALCULATED value is compared to the
CRITICAL value to determine if the difference is significant enough to reject Ho at the predetermined level of significanceIf CRITICAL value > CALCULATED
value --> fail to reject HoIf CRITICAL value < CALCULATED
value --> reject Ho
If reject Ho, only supports H1; it does not prove H1
Testing Hypothesis
If reject Ho and conclude groups are really different, it doesn’t mean they’re different for the reason you hypothesized may be other reason
Since Ho testing is based on sample means, not population means, there is a possibility of making an error or wrong decision in rejecting or failing to reject Ho
Type I errorType II error
Testing Hypothesis
Type I error -- rejecting Ho when it was true (it should have been accepted)equal to alphaif = .05, then there’s a 5% chance of
Type I errorType II error -- accepting Ho when it
should have been rejectedIf increase , you will decrease the
chance of Type II error
Identifying the Appropriate Statistical Test of Difference
One variable One-way chi-square
Two variables(1 IV with 2 levels; 1 DV) t-test
Two variables(1 IV with 2+ levels; 1 DV) ANOVA
Three or more variables ANOVA
84
TERIMA KASIH