PEOPLE ANALYT
ICS:
HOW TO BE A
SMART DATA
CONSUMER
NYC 4/22/2016
HOW TO BE A SMART DATA CONSUMER4/22/16
2
HOW TO BE A SMART DATA CONSUMER4/22/16
IT’S NOT THAT EASY…
3
HOW TO BE A SMART DATA CONSUMER4/22/16
MORE REALITY THAN FICTION…
4
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO VALIDITY OF RESULTSResource: http://horan.asu.edu/cook&
campbell.htm
From the “Bible”:
5
HOW TO BE A SMART DATA CONSUMER4/22/16
6
INTERNAL VALIDITY?
HOW TO BE A SMART DATA CONSUMER4/22/16
INTERNAL VALIDITYGiven that there is a relationship, is it plausible there are other explanations for the model?
7
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO INTERNAL… History (effects may be due to
unforeseen events) Maturation Testing (becoming test savvy) Instrumentation Statistical Regression Selection (self or convenient selection) Mortality
8
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO INTERNAL… Interactions With Selection Ambiguity About the Direction of Causal
Inference Diffusion or Imitation of Treatments Compensatory Equalization of
Treatments (coffee talk) Compensatory Rivalry by Respondents'
Receiving Less Desirable Treatments Resentful Demoralization of
Respondents Receiving Less Desirable Treatments9
HOW TO BE A SMART DATA CONSUMER4/22/16
10
EXTERNAL VALIDITY?
HOW TO BE A SMART DATA CONSUMER4/22/16
EXTERNAL VALIDITYGiven that there’s a causal relationship, how likely is it that the conclusion is generalizable across people, groups, companies, locations, and time?
11
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO EXTERNAL… Interaction of Selection and Treatment
(participants) Interaction of Setting and Treatment
(places) Interaction of History and Treatment
12
HOW TO BE A SMART DATA CONSUMER4/22/16
CONSTRUCT VALIDITY?
13
HOW TO BE A SMART DATA CONSUMER4/22/16
14
CONSTRUCT VALIDITY?
HOW TO BE A SMART DATA CONSUMER4/22/16
CONSTRUCT VALIDITYDo the relationships in the model actually reflect the meaning of variables?
15
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO CONSTRUCT… Inadequate Preoperational Explication
of Constructs Mono-Operation Bias (when the boss
asks the questions) Mono-Method Bias Hypothesis Guessing within
Experimental Conditions Evaluation Apprehension
16
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO CONSTRUCT… Experimenter Expectancies (coaching) Confounding Constructs and Levels of
Constructs Interaction of Different Treatments Interaction of Testing and Treatment
(attention time!) Restricted Generalizability Across
Constructs
17
HOW TO BE A SMART DATA CONSUMER4/22/16
18
STATISTICAL CONCLUSION VALIDITY?
HOW TO BE A SMART DATA CONSUMER4/22/16
19
STATISTICAL CONCLUSION VALIDITY?
HOW TO BE A SMART DATA CONSUMER4/22/16
STATISTICAL CONCLUSION VALIDITYAre we correctly analyzing the data?
20
HOW TO BE A SMART DATA CONSUMER4/22/16
THREATS TO STATISTICAL… Low Statistical Power Violated Assumptions of Statistical
Tests Fishing and the Error Rate Problem (in
Kansas City…) The Reliability of Measures The Reliability of Treatment
Implementation Random Irrelevancies in the
Experimental Setting Random Heterogeneity of Respondents21
HOW TO BE A SMART DATA CONSUMER4/22/16
22
HOW TO BE A SMART DATA CONSUMER4/22/16
STEVE LEVYwww.linkedin.com/in/stevenmlevywww.twitter.com/[email protected]
23