Statistics Boot Camp - Test Science...As the number of our observations increases, the difference between the sample mean and population mean goes to zero Our point estimate is a good

Statistics Boot Camp

Dr. Stephanie Lane

Institute for Defense Analyses

DATAWorks 2018

March 21, 2018

• Summarizing and simplifying data

• Point and interval estimation

• Foundations of statistical inference

• The process of hypothesis testing

• Common statistical tests

• A few closing tips

1

Outline of boot camp

2

Thinking about variables

Variables are characteristics that are observed or manipulated in a study

We frequently think about variables in terms of being continuous or discrete

• Continuous variables have fractional amounts• e.g., height, distance, time

• Discrete variables are distinct, separate, countable values• e.g., number of missiles, crew rank

The type of variable has implications for how we describe, analyze, visualize, and characterize our data.

3

Scales of measurement

Scale of measurement Definition ExampleNominal Named categories Wheel Type

(Tracked, Wheels)Ordinal Ordered categories Place in a race

(First, Second, Third)Interval Ordered values,

equidistantTemperature (Fahrenheit, Celsius)

Ratio Ordered values, equidistant, real zero

Distance (meters)

When possible, we prefer interval and ratio measures, as they contain more information

4

Descriptive statistics – simplify and summarize

Where do scores tend to fall (central tendency) and how spread out are they (variability)?

Central tendency aims to describe the centrality, or the typical score of a distribution

Mean, median, and mode are measures of central tendency

Variability aims to quantify the spread of a distribution, or the typical spread away from the center

Standard deviation, variance, range, and interquartile range are measures of variability

5

Central tendency

The mean is the average score of the distribution

𝑋 = 𝑖=1

𝑛 𝑋𝑖

𝑁

It is useful for describing symmetric distributions, and is frequently thought of as the “balance point” of a distribution.

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 6 7 X X X 6

Central tendency

The median is the 50th percentile, or the middle score in the distribution• It is useful for describing skewed distributions• It is robust to extreme scores, meaning that it is minimally

affected by outlying observations

The mode is the most commonly occurring score• It is useful for describing nominal data and multi-modal

distributions

7

Central tendency – reporting

Suppose we introduce a new system and administer a usability survey to operators. We ask operators to rate the usability of the system on a 1-7 scale. We plot our data and see the following results:

8

Variability – characterizing spread

9

Standard deviation is the typical spread of scores away from the mean, and is a common metric for quantifying variability

𝑠𝑋 = 𝑖=1

𝑁 𝑋𝑖 − 𝑋 2

𝑁 − 1

Like the mean, it takes all scores into account. Therefore, it is influenced by extreme scores or outliers.

The variance is the squared standard deviation, 𝑠𝑋2. We prefer reporting the standard deviation, as it is in the original units of the variable (e.g., yards, miles, points, etc.)

If we convert our scores to standard scores, we can easily quantify their distance away from the mean

10

𝑧 =𝑋 − 𝜇

𝜎

If we convert our scores to standard scores, we can easily quantify their distance away from the mean

11

Beyond descriptive statistics

12

Descriptive statistics are useful, and are a great starting point for describing and understanding your data.

But as an analysis tool, they can only get us so far.

What if we are interested in not only describing the sample, but we are also interested in drawing inference to the broader population?

For this question, we need inferential statistics.

Inferential statistics

13







14


Point estimation

15

In point estimation, we are interested in estimating some unknown parameter using a statistic (e.g., a mean) that we calculate from our sample data.

The parameter is the value that summarizes the entire population.

We use our sample statistic to draw inferences about the value in the population –

the population parameter.

Understanding vocabulary – parameter versus statistic

16

Random sample

𝜇 𝑋

Population Sample

Parameter Statistic

Examples of point estimation

We have a new communications system. We measure the time it takes to transmit a message. From the sample of data we collect during testing, we compute a mean of 𝑋 =

0.48 seconds.

Our best estimate of the population mean, 𝝁, is .48 seconds.

We have a new user interface for a software program. We observe whether operators could complete their mission using the new interface. We find that 27/30, or 90% of operators were able to successfully complete their mission.

Our best estimate of the population proportion, 𝒑𝟎, is .90.

17

How do we know our sample statistic (e.g., mean, proportion) is the best estimate of the population parameter?

18

Law of large numbers

19

As the number of our observations increases, the difference between the sample mean and population mean goes to zero

Our point estimate is a good starting point for quantifying a population parameter.

But might you also want to know…

…how much uncertainty is there?

20

Interval estimation

21

Interval estimation

22

Interval estimation provides us a range of values that have a specific likelihood of containing our population value

A confidence interval is constructed using our sample statistic ± our margin of error

The margin of error quantifies the degree of uncertainty in our estimate

𝑝 ± 𝑧𝛼2

𝑝(1 − 𝑝)

𝑛 𝑋 ± 𝑧𝛼

2

𝑠𝑋

𝑁 𝑋 ± 𝑡𝛼

2

𝑠𝑋

𝑁

What does a confidence interval tell us?

23

“If we were to repeat the study many times, 95% of the confidence intervals we constructed would contain the value of the true population parameter”

23

“This is why I hate statisticians.”

Audience poll: how does this interpretation make you feel?

What does a confidence interval tell us?

Let’s unpack this with a quick demonstration.

Suppose the true population mean is 𝜇 = 5.

I take a sample from the population, compute the sample statistic and margin of error, and construct a confidence interval.

24

Factors affecting confidence interval width

25

More variability wider confidence intervals

Smaller sample size wider confidence intervals

Larger confidence level wider confidence intervals

A wider confidence interval reflects more uncertainty in our estimate of the population parameter.







26


We can perform inference evenif we don’t know the shape of the population distribution!

The magical number 30 Central limit theorem

27

For a population with mean 𝜇 and standard deviation 𝜎, the distribution of sample means for sample size 𝑛 will have a mean of 𝜇 and a standard deviation of 𝜎/√𝑛 and will approach a normal distribution as 𝑛 approaches infinity.

What does this buy us?

Demonstrating the CLT

28

Suppose our population distribution is normally distributed, 𝜇 = 10; 𝜎 = 2

The sampling distribution of means

is normally distributed,

𝜇 = 10; 𝜎 =2

30

We take samples of 𝑁 = 30, with

replacement, from the population

Demonstrating the CLT via simulation

29

The sample means are normally distributed

But remember!

We said that the central limit theorem allowed us to know the shape of the sampling distribution of means, even if the

population distribution was not normal

30

What if the population distribution is not normal?

31

The sample means are still normally distributed

What if our population distribution is not normally distributed and our sample size is < 30?

32

The sampling distribution will resemble the population distribution

33

We shouldn’t use methods designed for normal theory here

What if the population is normally distributed and our sample size is small?

34

The sampling distribution of means will still be symmetric and approximately normal

35

We can more or less use methods based on normal theory here

The CLT is fundamental to the use of sampling distributions

36

Bottom line: the CLT allows us to perform inference on means even when we don’t know the population distribution

37







38


Hypothesis testing is a process of evaluating a claim that we make about the population

39

Hypothesis Testing

Step 1: State your hypotheses

Step 2: Set the acceptable level of risk (alpha)

Step 3: Collect data + compute test statistic

Step 4: Determine probability

Step 5: State your research conclusion

40

Step 1: State your hypotheses

41

We state two mutually exclusive hypotheses: the null hypothesis and the alternative hypotheses

• The null hypothesis is the hypothesis of “no change,” “no difference,” or “no relationship”

• The alternative hypothesis states that there is a change, a difference, or a relationship

Hypotheses can be directional or nondirectional, corresponding to one- and two-tailed tests, respectively

Step 2: Set the acceptable level of risk

42

There is no single correct answer to this!

In academic contexts, 𝛼 = .05 is frequently used as an acceptable level of risk

In defense research, sometimes 𝛼 = .20 is used as an acceptable level of risk

In pharmaceutical research, 𝛼 = .01 or even 𝛼 = .001 might be used!

The acceptable level of risk depends on the context. It should be set at the beginning, prior to analysis.

Step 3: Collecting data and computing statistic

43

There are many test statistics that can be computed during this stage, depending on your hypotheses.

We will focus on two today: 𝑧 and 𝑡

• 𝑧 is used when we know both the population mean and standard deviation, and we are testing our sample mean against the population mean

• 𝑡 is used when we do not know the population standard deviation, and we have to estimate it from our sample data

Step 4: Determine probability of your statistic

44

To determine probability, we compare our statistic to the sampling distribution of our statistic

In words: we compare our result to the results we would have observed by chance if the null hypothesis was true

Step 5: Draw a research conclusion

45

Make a decision about the null hypothesis• Reject if 𝑝 < 𝛼

• Retain if 𝑝 > 𝛼

Draw a conclusion in plain English!

We rejected the null hypothesis saying that there was no effect of training on performance, p = .032.

Training significantly affected performance. Individuals who underwent training had better performance.

Thinking about errors in hypothesis testing

DecisionRetain Null Reject Null

Trut

h

Null is true

Correct Decision

1 − 𝛼

Type I error𝛼

Null is false

Type II error𝛽

Correct Decision

1 − 𝛽

46

The null hypothesis on trial

47

Jury decisionNot guilty Guilty

Trut

h

Not guilty

Correct Decision

1 − 𝛼

Type I error𝛼

Guilty Type II error𝛽

Correct Decision

1 − 𝛽

Visualizing power, Type I error, Type II error

48

These five steps describe the process of hypothesis testing.

Let’s start with one of the most basic hypothesis tests.

49







50


Tests of one mean

51

Tests of one mean

52

Scenario: From the manufacturer, we know that the mean weight is specified to be 14000. This is our population mean, 𝜇= 14000. Suppose we have a set of N = 35 vehicles.

Research question: Do we have evidence that our sample of vehicles is significantly different from the population?

We begin by visualizing our distribution and computing basic descriptive statistics

53

In our sample, the mean weight was 𝑋 = 14076.6 and the standard deviation was 𝑠𝑋 = 102.40.

We proceed to a formal hypothesis test

54

We construct the 𝑡 statistic using

𝑡 = 𝑋 − 𝜇𝑠𝑋𝑛

We obtain t(34) = 4.43, p < .01. If we set 𝛼 = .05, then we reject the null hypothesis.

Conclusion: Our sample of vehicles is significantly heavier than the population.

What is the differenceI observed?

What is the range of differences I expected?

𝑡 = 4.43

Tests of two means

55

Tests of two means – motivation

56

Suppose we are not interested in comparing our sample to the population, but we are instead interested in comparing our samples to each other

For instance, what if we are interested in…

…comparing a new system to an old system?…comparing variant A to variant B?…comparing demographic groups?

Two-sample t-test: helmet testing

57

Scenario: we are interested in comparing the comfort of a new helmet to an old helmet. We assign 20 operators to wear the old helmet and 20 operators to wear the new helmet. We observe the number of hours until operators remove the helmet due to discomfort for both the old and new helmet.

Research question: Is the new helmet an improvement upon the old helmet?

Two-sample t-test: describe and visualize

58

Group 1 - Old Group 2 - NewMean 5.60 8.37Standard Deviation 2.03 2.49Sample Size 20 20

Two-sample t-test: inference

59

We construct the t-statistic using:

We obtain t(38) = 3.86, p < .01. At 𝛼 = .05, we reject the null hypothesis.

Conclusion: The new helmet significantly reduces discomfort.

𝑠𝑝2 =

𝑛1 − 1 𝑠12 + 𝑛2 − 1 𝑠2

2

𝑛1 − 1 + 𝑛2 − 1

𝑡 =( 𝑋1− 𝑋2) − (𝜇1 − 𝜇2)

𝑠𝑝2 1

𝑛1+

1𝑛2

We pool our sample variances together into one

estimate

𝑡 = 3.86

Test of one proportion

60

Test of one proportion – motivation

61

In testing, we are frequently interested in estimating the underlying probability of an event

… what is the probability of a successful missile launch?… what is the probability of a successful message transmission?… what is the probability of a successful torpedo hit?

Though we generally prefer continuous metrics as response variables, sometimes we can’t avoid using a 0/1 outcome variable.

Binomial distribution

62

For the binomial distribution, the following criteria must be met:

Each event yields one of two outcomes – success or failure

Each event is independent (memory-less, like a coin flip)

The underlying probability of success, 𝑝0, is constant across events

Performing inference on proportions

63

We can perform inference by comparing our observed sample proportion to the sampling distribution

The mean of the sampling distribution is 𝑝 =𝑋

𝑛

And the standard deviation is 𝑝0 1−𝑝0

𝑛

We may construct our test statistic using z = 𝑝−𝑝0

𝑝0(1−𝑝0)

𝑛

Number of successes

Number of trials

Test of one proportion

64

Scenario: Suppose we have a system for transmitting messages. We want to know if this new system is worse than P(Transmission) = .80, our population estimate for the legacy system. During the OT of this new system, we observe that 38 of 50 messages transmitted successfully.

Research question: Is the new system worse than the old system?

Relevant quantities: X = 38; 𝑛 = 50; 𝑝 = .76; 𝑝0 = .80

Compute statistic and determine probability

65

Conclusion: We cannot conclude that the new system is worse than the old system.

𝑧 = 𝑝 − 𝑝0

𝑝0 1 − 𝑝0𝑛

𝑧 = −.71

General linear model

66

67

General

Linear: the function is a linear function of the parameters

Model: it provides a description of the relationship between one response and one or more predictor

variables

General: widely applicable to estimating and testing hypotheses about parameters

Linear Model

The general linear model

68

In its simplest form, the GLM can be expressed as:

𝑦𝑖 = 𝛽0 + 𝛽1𝑥𝑖 + 𝜖𝑖

The above equation expresses a model in which we can express an individual’s outcome, 𝑦𝑖, as a function of an intercept, 𝛽0, a coefficient, 𝛽1, a predictor variable, 𝑥𝑖 , and random error, 𝜖𝑖

Simple linear regression

69

For one response variable and one predictor variable, the phrase “simple linear regression” is often used

If we have two continuous variables, we can plot a line of best fit to characterize the relationship between our predictor and our outcome variable

Ordinary least squares – line of best fit

70

residual, 𝝐𝒊

𝑦𝑖 = 𝛽0 + 𝑥𝑖𝛽1 + 𝜖𝑖

The general linear model – beyond simple linear regression

71

𝒚 = 𝑿𝜷 + 𝝐

𝒚 is a vector of 𝑁 observations on our response variable

𝑿 is an 𝑁 × 𝑝 matrix of observations on 𝑝 predictor variables

𝜷 is a 𝑝 × 1 matrix of unknown parameters

𝝐 is a vector of 𝑁 subject-specific deviations from the expected value

N x 1 N x p N x 1p x 1

GLM: Assumptions

72

Homogeneity – constant variance about any value of the regression function (e.g., deviation for each unit has the same variance), 𝜖𝑖~ 𝑁(0, 𝜎2)

Independence – errors are statistically independent• If violated, check sampling and/or design

Linearity – the expected values (means) are linear functions of the parameters

• Linear: 𝑦𝑖 = 𝛽0 + 𝛽1𝑥𝑖 + 𝜖𝑖

• Linear: 𝑦𝑖 = 𝛽0 + 𝛽1𝑥𝑖 + 𝛽2𝑥𝑖2 + 𝜖𝑖

• Not linear: 𝑦𝑖 = 𝛽0 + 𝑥𝑖𝛽1 + 𝜖𝑖

Existence – finite mean and variance

Important note about GLM assumptions

73

For least squares estimation to be valid, we only need these four assumptions.

However, to be able to perform inference (e.g., hypothesis tests), we must make one final assumption – Gaussian errors

These five assumptions (H-I-L-E-Gauss) allow for estimation and inference in regression

GLM – example

74

Scenario: We have a notional system with a new user interface designed to be simpler and clearer to operators. We want to know if the new system improves reaction time compared to the old system. We also ask operators to rate their experience on a on a 1-7 scale.

Regression equation:

𝑅𝑒𝑎𝑐𝑡𝑖𝑜𝑛 𝑡𝑖𝑚𝑒 = 𝛽0 + 𝛽1𝑆𝑦𝑠𝑡𝑒𝑚 + 𝛽2𝐸𝑥𝑝𝑒𝑟𝑖𝑒𝑛𝑐𝑒 + 𝛽3(𝑆𝑦𝑠𝑡𝑒𝑚 ∗ 𝐸𝑥𝑝𝑒𝑟𝑖𝑒𝑛𝑐𝑒) + 𝜖𝑖

main effect main effect interaction effect

In multiple regression, we have a hypothesis test for each effect

Understanding effects in multiple regression

75

Conclusion: The new system improves reaction time, particularly for individuals with low experience

Generalized linear model

76

The generalized linear model allows for the linear model to be related to our response variable via a link function

Therefore, we can extend the linear modeling framework to response variables that are not normally distributed

Common examples include:• Poisson regression (outcome is a count)• Logistic regression (outcome is binary)

The generalized linear modeling framework opens up a large number of possibilities

77Table 15.1 from Applied Regression Analysis and Generalized Linear Models by John Fox

Generalized linear model – the link function

78

We can express a generalized linear model using a linear predictor,𝜂𝑖 = 𝛽0 + 𝛽1𝑥1𝑖

with a link function 𝑔(∙) that describes the relationship between the mean 𝐸 𝑦𝑖 = 𝜇𝑖 and the linear predictor,

𝑔 𝜇𝑖 = 𝜂𝑖 = 𝛽0 +𝛽1𝑥1𝑖

We can express the normal GLM in this way using𝜂𝑖 = 𝛽0 + 𝛽1𝑥1𝑖

with an identity link function, 𝑔 𝜇𝑖 = 𝜇𝑖

Logistic regression

79

If our outcome is binary, 𝑦𝑖 ~ 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑁𝑖 , 𝑝𝑖)

then our linear predictor can be expressed as: 𝜂𝑖 = 𝛽0 + 𝛽1𝑥1𝑖

with a logit link function,

𝑔 𝜇𝑖 = 𝑙𝑜𝑔𝑖𝑡 𝜇𝑖 = log𝜇𝑖

1 − 𝜇𝑖,

yielding the regression equation,

log 𝑝

1 − 𝑝= 𝛽0 + 𝛽1𝑥1 + 𝛽2𝑥2 + …𝛽𝑘𝑥𝑘

Visualizing the logit link function

80

Don’t try to fit a straight line through 0/1 data!

Instead use a logit link function to map our linear predictor onto the binary response

The general linear model represents a powerful framework for evaluating our research hypotheses, and encompasses a variety

of statistical tests, including t-tests, ANOVA, ANCOVA, and multiple regression.

The generalized linear model is an extension that allows the linear model to be related to an outcome variable via a link

function, and includes logistic regression, Poisson regression, and multinomial regression (among others).

81







82


Make sure the test you select reflects your research question

Carefully consider outliers. Don’t just delete!

Follow good practices of data visualization

83

Matching the test to the research question

84

When selecting the correct test, there are several important questions to consider

85

Are you looking for…

… a difference?a difference in means?a difference in medians?a difference in variances?

… a relationship?a linear relationship?a nonlinear relationship?

86

Selecting the correct test

87


88


89

These flowcharts can be useful (I’ll admit).

However, adhering strictly to their use might derail you from thinking critically about the question you were actually trying to

answer.

90

Flowchart reasoning gone wrong

91

“I have a continuous outcome variable and I want to see if there is a difference between source A and source B.”

t = 1.13, p = .26 D = 0.33, p = .07

If you had also cared about a difference in spread, you would have

missed it with the t-test!

“I’ll do a two-sample t-test!”

one journey through a flowchart later…

Outliers

92

Carefully consider outliers

93

With a small sample size, even a few data points can heavily influence our results!

Data visualization

94

Meaningful data visualization

95

Two continuous variables – relationship

One continuous variable

One discrete variable

Two continuous variables – difference

Thank you!

Documents

Statistics Boot Camp - Test Science...As the number of our observations increases, the difference between the sample mean and population mean goes to zero Our point estimate is a good