Data Consumerism HRU

PEOPLE ANALYT

ICS:

HOW TO BE A

SMART DATA

CONSUMER

NYC 4/22/2016

HOW TO BE A SMART DATA CONSUMER4/22/16

2

HOW TO BE A SMART DATA CONSUMER4/22/16

IT’S NOT THAT EASY…

3

HOW TO BE A SMART DATA CONSUMER4/22/16

MORE REALITY THAN FICTION…

4

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO VALIDITY OF RESULTSResource: http://horan.asu.edu/cook&

campbell.htm

From the “Bible”:

5

HOW TO BE A SMART DATA CONSUMER4/22/16

6

INTERNAL VALIDITY?

HOW TO BE A SMART DATA CONSUMER4/22/16

INTERNAL VALIDITYGiven that there is a relationship, is it plausible there are other explanations for the model?

7

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO INTERNAL… History (effects may be due to

unforeseen events) Maturation Testing (becoming test savvy) Instrumentation Statistical Regression Selection (self or convenient selection) Mortality

8

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO INTERNAL… Interactions With Selection Ambiguity About the Direction of Causal

Inference Diffusion or Imitation of Treatments Compensatory Equalization of

Treatments (coffee talk) Compensatory Rivalry by Respondents'

Receiving Less Desirable Treatments Resentful Demoralization of

Respondents Receiving Less Desirable Treatments9

HOW TO BE A SMART DATA CONSUMER4/22/16

10

EXTERNAL VALIDITY?

HOW TO BE A SMART DATA CONSUMER4/22/16

EXTERNAL VALIDITYGiven that there’s a causal relationship, how likely is it that the conclusion is generalizable across people, groups, companies, locations, and time?

11

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO EXTERNAL… Interaction of Selection and Treatment

(participants) Interaction of Setting and Treatment

(places) Interaction of History and Treatment

12

HOW TO BE A SMART DATA CONSUMER4/22/16

CONSTRUCT VALIDITY?

13

HOW TO BE A SMART DATA CONSUMER4/22/16

14

CONSTRUCT VALIDITY?

HOW TO BE A SMART DATA CONSUMER4/22/16

CONSTRUCT VALIDITYDo the relationships in the model actually reflect the meaning of variables?

15

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO CONSTRUCT… Inadequate Preoperational Explication

of Constructs Mono-Operation Bias (when the boss

asks the questions) Mono-Method Bias Hypothesis Guessing within

Experimental Conditions Evaluation Apprehension

16

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO CONSTRUCT… Experimenter Expectancies (coaching) Confounding Constructs and Levels of

Constructs Interaction of Different Treatments Interaction of Testing and Treatment

(attention time!) Restricted Generalizability Across

Constructs

17

HOW TO BE A SMART DATA CONSUMER4/22/16

18

STATISTICAL CONCLUSION VALIDITY?

HOW TO BE A SMART DATA CONSUMER4/22/16

19

STATISTICAL CONCLUSION VALIDITY?

HOW TO BE A SMART DATA CONSUMER4/22/16

STATISTICAL CONCLUSION VALIDITYAre we correctly analyzing the data?

20

HOW TO BE A SMART DATA CONSUMER4/22/16

THREATS TO STATISTICAL… Low Statistical Power Violated Assumptions of Statistical

Tests Fishing and the Error Rate Problem (in

Kansas City…) The Reliability of Measures The Reliability of Treatment

Implementation Random Irrelevancies in the

Experimental Setting Random Heterogeneity of Respondents21

HOW TO BE A SMART DATA CONSUMER4/22/16

22

HOW TO BE A SMART DATA CONSUMER4/22/16

STEVE LEVYwww.linkedin.com/in/stevenmlevywww.twitter.com/levyrecruitswww.recruitinginferno.comlevy.steve@gmail.com

23