Presentation Teaching Evidence-Based Management NYU Wagner 2014

Evidence-Based Practice

A New Approach of Teaching The Practice of Management

Eric Barends, CEBMa – Denise Rousseau, CMUMay 6th, 2014

Mission Today• Teaching Evidence-based practice (in 1 hour)

• Disappointment

• Inspiration

Who are we and what is our mission (today)?

Eric Barends• manager

• teacher

• director

• CEBMa

Denise Rousseau• researcher

• teacher

• professor

• CMU / Heinz / Tepper

Postgraduate Course

Postgraduate Course

Postgraduate Course

Postgraduate Course

Current developments

Education

Access to research databases

Rapid Evidence Assessments

Building a community

Postgraduate Course

CEBMa Database of Evidence Summaries

Online learning modules

Accreditational bodies

Future developments: practice

1. Some background

2. Teaching: learning principles

3. Teaching: curriculum

4. Teaching: examples

5. Your questions

Today

Evidence based management:

What is it?

1. Some background

Evidence-based practice

Central Premise:

Decisions should be based on a

combination of critical thinking

and the ‘best available evidence‘.

Evidence?

findings from scientific research,

organizational facts & figures,

benchmarking, best practices,

professional experience

16

All managers base their

decisions on ‘evidence’

17

But…many managers pay

little or no attention to the

quality of the evidence they

base their decisions on

18

Trust me, 20 years of management experience

19

SO ...

20

Teach managers how to

critically evaluate the

trustworthiness of evidence

from multiple sources

and help them find ‘the best available’ evidence

An example

An example

An example

Maslow, A.H. "A Theory of Human Motivation”

An example

An example

Professional experience and

judgment

Organizational data, facts and figures

Stakeholders’ values and concerns

Scientific research findings

AskAcquire

AppraiseAggregate

ApplyAssess

An example

Professional experience and judgment

Maslow, A.H. (1943). "A Theory of Human Motivation," Psychological Review 50(4)

Wahba, M. A., & Bridwell, L. G. (1976). Maslow reconsidered: A review of research on the need hierarchy theory. Organizational Behavior and Human Performance, 15(2)

Scientific research findings

How evidence-based are we (managers)?

“I’ve never thought I need more

evidence before making a

decision;

I know what needs to be done,

we get on with it and we get

results.”

1. Incompetent people benefit more from feedback than

highly competent people.

2. Task conflict improves work group performance while

relational conflict harms it.

3. Encouraging employees to participate in decision

making is more effective for improving organizational

performance than setting performance goals.

True (likely) or false (not likely)?

How evidence-based are we?

959 (US) + 626 (Dutch) HR professionals

35 statements, based on an extensive body of

evidence

true / false / uncertain

HR Professionals' beliefs about effective human resource practices: correspondence between research and practice, (Rynes et al, 2002, Sanders et al 2008)

Outcome: not better than random chance

EBP:

Teach managers how to

critically evaluate the

trustworthiness of evidence

from multiple sources

and help them find ‘the best available’ evidence

2. Teaching EBP:

Learning principles

Discuss with your neighbours:

When it comes to teaching,

what are important learning principles?

60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

Learning principles

1. The brain needs time to get used to new things. Longer periods in between practice sessions lead to a longer overall retention.

2. Training program should include opportunities for practice, linked to real world situations .

3. The human brain seeks comfort in what it knows and is familiar with; addressing / confronting prior knowledge increases understanding (start from where the students are)

Learning principles

4. Higher order thinking only happens when people work on questions / problems / issues themselves.

5. Learning from failure is important for learning. Speculating and predicting before finding the correct answers helps people become adaptive learners / experts.

Teaching EBP =

Small groups

Problem based

Real life cases

EBP starts with a practical question, not with an academic answer

3. Teaching EBP:

Curriculum

Limitations of human judgment & common forms of cognitive bias

Retrieving & critically assessing experiential evidence

Retrieving & critically assessing organizational evidence (qualitative & quantitative)

Searching in research databases

Efficiently reading research articles

Critically appraising evidence from research

Weighing and aggregating evidence from multiple sources

Incorporating evidence into the decision making process

Different types of decisions and decision-making processes

Assessing the outcome of decisions made

EBP: Curriculum

Professional experience and

judgment

Organizational data, facts and figures

Stakeholders’ values and concerns

Scientific research findings

AskAcquire

AppraiseAggregate

ApplyAssess

A critical and reflective attitude

Skills to distinguish trustworthy from less trustworthy evidence.

Thinking in terms of probabilities

EBP: outcome

CAT: Critically Appraised Topic

CAT: Critically Appraised Topic

A critically appraised topic (CAT) is a

structured, short (2 – 5 pages max) summary

of evidence on a topic of interest, focused

around a practical problem or question..

CAT: structure

1) Background / context

2) Question (PICOC)

3) Search strategy

4) Results / evidence summary

5) Findings

6) Limitations

7) Recommendation

CAT: 3 hits

1. Small group (2-3), scientific evidence, list of topics

2. Individual, scientific evidence & theory, own topic

3. Individual, evidence from multiple sources, real life question / issue

CAT: examples

Is there a valid and reliable way to measure the productivity of knowledge workers?

To what extent will leadership training improve the effectiveness of leaders (e.g. managers, executives), what are the characteristics of effective leadership training programs?

What research evaluating the effects of 360-degree has been published in peer-reviewed scholarly journals in the past 10 years? Which of the variables that are expected to have an impact on the effectiveness of 360-degree feedback are most widely studied and what is known of their effect?

CAT-walk

4. Teaching EBP:

Examples

Critical attitude

ASK

Critically appraisal: research findings

Organizational evidence

4. Teaching EBP: examples

Critical attitude

ASK

Critically appraisal: research findings

Organizational evidence

4. Teaching EBP: examples

Discuss with your neighbours:

What are the most common cognitive biases in management?

(give an example)

60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

We are predisposed to selectively

search for or interpret information in

ways that confirms our existing beliefs,

expectations and assumptions, and

ignore information to the contrary.

In other words, we “see what we want to see”

Confirmation bias

The amount of information increases faster than our ability to process it.

Increase of information

McKinsey (1997 / 2001)

Case study / best practice

War on Talent

1. Pattern recognition

2. Confirmation-bias

3. Small numbers fallacy

4. Outcome bias

5. Halo effect

6. Authority-bias

7. Groupthink

8. Availability bias

Biases

Assignment

One of your best friends is trader on the stock exchange. He enthusiastically tells you he has analyzed a large number of financial and economic data and that he has discovered an interesting phenomenon: "The position of the Dow Jones index multiplied by the price of oil is two days ahead of the gold price!" In other words, if both the Dow Jones and the oil price go up, the price of gold will rise within a day.

Bias? Critical questions?

We are predisposed to see order, pattern and causal relations in the world.

Patternicity: The tendency to find meaningful patterns in both meaningful and meaningless noise.

Bias: pattern recognition

Assignment

Most people will prefer doctor B

Why? Bias? Critical questions?

doctor A doctor B

Last operation: patient died on the

operating table

Last operation: patient recovered

faster than expected

Outcome bias

We are inclined to evaluate the quality of a decision (intervention, method) on the basis of its outcome.

Bias: Outcome bias

The Asch Experiment

Nasa, Challenger

Groupthink:

Groupthink is a psychological phenomenon that occurs within a group of people, in which the desire for harmony or conformity in the group results in an incorrect or irrational decision

Bias: Groupthink

Bounded rationality

System 1

Fast Intuitive, associative heuristics & biases

System 2

Slow (lazy) Deliberate, ‘reasoning’ Rational

Bounded rationality

limbic system and brainstem(system 1)

neo cortex(system 2)

Critical attitude

ASK

Critically appraisal: research findings

Organizational evidence

Teaching EBP: examples

Postgraduate Course

Postgraduate Course

1. Problem identification

2. Surfacing assumptions

3. Logic model

4. Formulating a searchable question

5. Formulate sub-questions

5 steps

Postgraduate Course

For which problem is ….. the solution?

For who(m)

Why

How big ?

How do we know (what is the evidence?)

Step 1: What is the problem?

Postgraduate Course

“Door to needle time”, 48 UK hospitals in Westmidland

Our hospital

The problem

Postgraduate Course

Discuss with your neighbours possible causes for these

differences that could not be solved by the implementation of

Lean / Six Sigma

60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

Postgraduate Course

1. Problem identification

2. Surfacing assumptions

3. Logic model

4. Formulating a searchable question

5. Formulate sub-questions

5 steps

Postgraduate Course

Logic model

Postgraduate Course

A logic model spells out the process by which a

problem or intervention is expected to produce

certain outcomes.

In making expectations (its logic) explicit, a logic

model helps identify the kind of evidence

needed.

Logic model

Critical attitude

ASK

Critically appraisal: research findings

Organizational evidence

4. Teaching EBP: examples

Critical appraisal

Always start with a practical question:- does it work?- how many employees …?- how do employees feel about …?

How could we find out?

87

Critical appraisal

Amanda Burls:

“I never tell them anything about randomization or blinding, I just ask: “How are you going to know? How would you test this if I would give you a half million dollars to test it?” And when they come up with a suggestion I say, “Alright, can you think of any reason you got the results showing it works, while in fact it doesn’t.” And they say, “Well, it could be this, it could be that” And then I say, “Ok, then redesign your study so it can’t be this,” and what they come up with are precisely those things: randomization and blinding.”

How trustworthy is this study?

Two studies, different designs

Studies with methodological flaws

Best available evidence (so what now?)

Online course!

Critical appraisal

Critical attitude

ASK

Critically appraisal: research findings

Organizational evidence

4. Teaching EBP: examples

91

Organizational facts and figures

Organizational facts and figures

Organizational facts and figures

95

Organizational facts and figures

Organizational facts and figures

Examples

1. Added value

2. Types of organizational evidence

3. Starting point: logic model

4. Statistical aspects to consider

5. Barriers to overcome

Organizational evidence

5. Barriers to overcome

1. Small numbers problem

2. Measurement errors

3. Context

4. Tainted data

5. Politics