Chapter 4Chapter 4

Calculating and Calculating and Evaluating ValidityEvaluating Validity

When We Have Validity . . . When We Have Validity . . .

• We have an acceptably accurate We have an acceptably accurate measurement.measurement.

• We are measuring what we intend to We are measuring what we intend to measure.measure.

• We are interpreting and applying the We are interpreting and applying the measurement appropriately.measurement appropriately.

Methods for Evaluating Validity Methods for Evaluating Validity and Their Calculationsand Their Calculations

Population and SamplePopulation and Sample

• Population: Population: – The designated group being measured The designated group being measured

• Sample: Sample: – A representative subgroup of the populationA representative subgroup of the population

• Inferential statistics:Inferential statistics:– Generalization about a population based on Generalization about a population based on

what is learned about the samplewhat is learned about the sample

Using t-tests to Compare Using t-tests to Compare Sample MeansSample Means

• Purpose of t-tests: Purpose of t-tests: – Compare two sample meansCompare two sample means– Determine if a sample represents a Determine if a sample represents a

populationpopulation

• Types:Types:– Paired vs. independentPaired vs. independent– One-tailed vs. two-tailedOne-tailed vs. two-tailed

Results of a t-testResults of a t-testfor the VOfor the VO22max-12-minute run examplemax-12-minute run example

Interpreting t-testsInterpreting t-tests

• p value: p value: – The probability that the difference between The probability that the difference between

two means is a coincidence of random two means is a coincidence of random samplingsampling

• Sometimes a t-test comparison of means Sometimes a t-test comparison of means doesn’t give us everything we need to doesn’t give us everything we need to know; we need to compare individual know; we need to compare individual differences in scores.differences in scores.

CorrelationsCorrelations

• Defined: Measurement of the strength of Defined: Measurement of the strength of the relationship between two variables.the relationship between two variables.

• Pearson Product-Moment Correlation:Pearson Product-Moment Correlation:– Compares individual differences between two Compares individual differences between two

methods of measurementmethods of measurement

Results of a PearsonResults of a PearsonProduct-Moment CorrelationProduct-Moment Correlation

for the VOfor the VO22max-12-minute run examplemax-12-minute run example

Methods for Evaluating a Methods for Evaluating a CorrelationCorrelation

• r value (or correlation r value (or correlation coefficient)coefficient)

• TrendlineTrendline

• RR2 2 (or coefficient of (or coefficient of determination)determination)

Slope of a TrendlineSlope of a Trendline

• Defined:Defined:– The change in The change in yy value per unit change in value per unit change in xx

value.value.

yy value = the slope value = the slope ** the the xx value + the value + the yy intercept (point where intercept (point where yy = 0) = 0)

OrOr

y = mx + by = mx + b

Scatter Plot Showing TrendlineScatter Plot Showing Trendline

Bland-Altman AnalysisBland-Altman Analysis

A simple approach for evaluating A simple approach for evaluating criterion validitycriterion validity——through error through error analysis: analysis:

• Data taken from an unusually diverse Data taken from an unusually diverse group of subjects will show an artificially group of subjects will show an artificially high correlation.high correlation.

• Analysis focuses attention on error Analysis focuses attention on error scores.scores.

Spreadsheet for a Spreadsheet for a Bland-Altman AnalysisBland-Altman Analysis

Bland-Altman AnalysisBland-Altman Analysis

Evaluating Validity of Ranked DataEvaluating Validity of Ranked Data

• Ordinal numbers:Ordinal numbers:– Ranked numbers that give a place in Ranked numbers that give a place in

line but no information about distances line but no information about distances between numbers; place holders.between numbers; place holders.

• Interval numbers:Interval numbers:– Numbers that are separated by equal Numbers that are separated by equal

intervals; scalar numbers.intervals; scalar numbers.

Spearman’s rhoSpearman’s rho

• A special correlation used in cases A special correlation used in cases where one variable is an ordinal where one variable is an ordinal number number

• Calculating Spearman’s rho:Calculating Spearman’s rho:– Excel doesn’t offer the calculationExcel doesn’t offer the calculation– Some websites can do it; for example, Some websites can do it; for example,

www.wessa.net/rankcorr.waspwww.wessa.net/rankcorr.wasp

Evaluating the Validity Evaluating the Validity of Criterion-Referenced Testsof Criterion-Referenced Tests

• Validity ratio: The ratio of number of Validity ratio: The ratio of number of scores classified correctly to the total scores classified correctly to the total number of scores.number of scores.

Validity ratio = (CR+NCR)/(CR+CW+NCR)Validity ratio = (CR+NCR)/(CR+CW+NCR)where CR = competent and classified correctlywhere CR = competent and classified correctly

NCW = not competent and wrongly classifiedNCW = not competent and wrongly classified CW = competent but wrongly classifiedCW = competent but wrongly classified

and NCR = not competent and classified and NCR = not competent and classified correctlycorrectly

Example Diagram for Evaluating the Example Diagram for Evaluating the Validity of a Criterion-Referenced Validity of a Criterion-Referenced

MeasurementMeasurement

Your ViewpointYour Viewpoint

• After reading the chapter, go back and read After reading the chapter, go back and read the chapter-opening quote from Henry Clay: the chapter-opening quote from Henry Clay:

““Statistics are no substitute for human Statistics are no substitute for human judgment.”judgment.”

• Think also about the famous quote, “There Think also about the famous quote, “There are three kinds of lies: lies, damned lies, and are three kinds of lies: lies, damned lies, and statistics.”statistics.”

• Do you agree with the sentiments expressed Do you agree with the sentiments expressed by these authors? Why or why not?by these authors? Why or why not?