Evidence-Base Practice

Evidence-Base Practice

Evidence-based practice?

Decisions should be based on a combination of critical thinking

and the ‘best available evidence‘.

Evidence =Various types of information

outcome of scientific research, organizational facts & data,

benchmarking, best practices, collective experience, personal

experience, intuition

All managers base their decisions on ‘evidence’

However ...

Many managers pay little or no attention to the

quality of the evidence they base their decisions on

Trust me, 20 years of management experience

Teach managers how to critically evaluate the validity,

and generalizability of the evidence and help them find ‘the best available’ evidence

Best available evidence

Experiential evidence: the professional insight, understanding, and

expertise of practitioners

Organizational evidence; data, facts and figures, business

intelligence, and benchmarks

Scientific evidence: outcome from scientific research published in

peer reviewed journals

Organizational values and stakeholders’ concerns

Evidence based decision

Best available experiential evidence

Best available organizational

evidence

Organizational values and stakeholders’

concerns

Best available scientific evidence

Decision making process




evidence


concerns


Decision making processdiagnosis intervention


It’s about probabilities(not golden bullets)

Evidence-Based Practice

1991Medicine

1998Education

1999Social care, public policy

2000Nursing

2000Criminal justice

????Management?






What’s the evidence for evidence-based practice?

Got evidence?

Forecasts or risk assessments based on the aggregated experience of multiple persons are more accurate than forecasts based on the experience of one person (provided that the forecasts are made independently before being averaged together)

- Choudhry, N.K., et al. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005; 142 (4)

- Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690

- Bauer A., et al. Forecast Evaluation with Cross Sectional Data: The Blue Chip Surveys. Economic Review, Federal Reserva bank of Atlanta, 2003.

- Servan-Schreiber, E., et al. Prediction Markets: Does Money Matter? Electronic Markets, 2004: 14 (31).- Scott Armstrong, J. Combining Forecasts, in Principles of Forecasting: A handbook for Researchers and

Practitioners, Kluwer Academic Publishers, New York, 2001

Professional judgments based on hard data or statistical models are more accurate than judgments solely based on experiential evidence

- Yaniv, I., & Choshen-Hillel, S. (2011). Exploiting the Wisdom of Others to Make Better Decisions: Suspending Judgment Reduces Egocentrism and Increases Accuracy, Journal of Behavioral Decision Making, 2012; 25 (5) p 427–434

- Lewis, M. Moneyball: The Art of Winning an Unfair Game. Barnes and Noble, 2003- Grove, W.M. Clinical Versus Statistical Prediction. Journal of Clinical Psychology, 2005; 61(10), p 1233–

1243

Got evidence?

A decision based on the combination of different types of evidence (experiential, organizational, scientific) leads to better outcomes than a decision based on just one source of evidence

- Antman, E.M. et al, A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts, JAMA, 1992: 268 (2) p 240 – 248

- McNees, S.K. The Role of Judgment in Macroeconomic Forecasting Accuracy, International Journal of Forecasting, 1990; 6 (3), p 28-299

- Silver, N. The Signal and the Noise: Why So Many Predictions Fail - but Some Don't. Penguin: London, 2012; p 286 and p 690

- Tetlock, P. E. Expert Political Judgement, Princeton, NJ: Princeton University Press, 2006

Got evidence?

Evidence-Base Practice

Post mortem analysis

Evidence-based perspective

NOT: Did they make the right decision?

BUT: Assessment of the decision making process

What kind of evidence was there? Was this the best available evidence? Is there evidence from scientific research to

support (or call into question) the approach taken?


Hospital, The Netherlands

550 beds

3300 employees

210 medical specialists

225,000 admissions

Top Clinical & Teaching hospital

Structure: Business Units

Organization

Decision: Leadership training

All managers:

Board of directors

Division managers

Unit managers

Head nurses

From: 20 business units - 20 senior managers - 40

supervisors

To:20 business units – 8 senior managers - 40

supervisors

Cause I: reorganization

Ambition

Experience

Education (MBA)

Assessment

Selection


Gap

larger span of control, more responsibilities =

extra skills They need extra skills

We need extra skills


Cause II: leadership climate

Employee / Job satisfaction: leadership

Board of

directors


Cause III: management development

Project team

10 members

Expertise (HR)

Representation (nurses, doctors,

managers)

Support base

Process: months!

Sessions with all stakeholders (medical staff, nurses, managers, staff council)

What is leadership?

What kind of leadership does the OLVG

need?

OLVG leadership vision

OLVG Leadership vision

Passion and business

Inspire and connect

Leadership concerns us all

Selection procedure

Longlist: 30 References (other hospitals, network), reputation

Shortlist: 5 Academic, Educational, Training, HR Consulting,

Wild card

First round: 3 Paper pitch: proposal based on documents

Final round: 1 Carousel: board, medical staff, head nurses,

managers



1. What was the problem / issue

2. What kind of evidence was there?

3. Was this the best available evidence?

1. Problem identification

2. Surfacing assumptions

3. Logic model

3 steps

For which problem is ….. the solution?

- For who(m)

- Why?

- How big?

- How do we know (what is the evidence?)

Step 1: What is the problem?


1. What was the problem / issue



Assumptions are often hidden

Step 2: What are the assumptions?

1. Problem identification

2. Surfacing assumptions

3. Logic model

3 steps

Logic model


1. What was the problem / issue?



Best available evidence?



evidence


concerns




extra skills They need extra skills

We need extra skills



extra skills We need extra skills


Finance? Risk management? Problem solving? Planning? Performance management? Change management? Business plan?


Employee / Job satisfaction: leadership

always – often – sometimes - never

I can always count on my manager when having job

related problems.

I can always ask my manager for support

The relationship with my manager is good.

I have conflicts with my manager

There is a good atmosphere between my manager

and me

Survey: Leadership

always – often – sometimes - never

My manager inspires me to do the best in my job

My manager sets a good example

My manager makes me feel that my work is

meaningful

My manager takes interest in my career

development.

Survey: Leadership

Survey: outcome ‘leadership’

Favourable versus unfavourable

Comparisson with 2011

Benchmark with other divisions / units

No baseline

No confidence intervals

No focus groups

No correlations / regressions

How did new managers score?

Board of directors: strategy



1. Dus wat is het probleem?

2. Hoe zeker kunnen we er van zijn dat leiderschapstraining de oplossing voor ons probleem is?

3. Zijn er andere opties mogelijk?

Selection procedure

Longlist: 30 References (other hospitals, network), reputation

Shortlist: 5 Academic, Educational, Training, HR Consulting,

Wild card

First round: 3 Paper pitch: proposal based on documents

Final round: 1 Carousel: board, medical staff, nurses, managers

leadership training: dm process

No problem definition & not objectified

No experiential evidence consulted

No organizational evidence consulted

Focus on shareholders’ concerns

Selection of training companies based on

experience, recommendation or reputation

No explicit selection criteria / procedure

Postgraduate School




evidence


concerns



✗

✗

✓

?

Critically Appraised Topic

Question: To what extent will leadership training

improve the effectiveness of the leaders (e.g.

managers, head nurses, physician

executives) of the Amsterdam Medical

Centre? In addition, what are the

characteristics of effective leadership training

programs?

leadership training: scientific evidence

15 meta analyses, 5 relevant

37 (‘systematic’) reviews, 2 relevant

Lots of relevant primary studies (by-catch)


Overall (50 years): small (.2) to moderate (.4)

effect sizes

Little information on ROI (sometimes negative)

Tailor made programs designed on the basis of an

analysis of tasks / skills are more effective and

have a positive impact on motivation


Leadership trainings that focus on general

management skills and / or interpersonal / social

skills show higher effect sizes than those based on

a specific leadership ‘model’

The trainings should be of reasonable length (>3

days) and repeated periodically to be effective


The training should include opportunities to

practice (in vivo)

A working climate that supports the ‘transfer’ to

the workplace (organization, supervisor, peers)

motivates employees to apply what they have

learned.

Reactions

Who knew?

Denial

Anger

Bargaining

Acceptance




evidence

Organizational values and stakeholders’ concerns



Think about it:

In the next weeks, before you make a decision, ask yourself

Did you ask the right questions?

What is the evidence available?

What is the quality of the evidence?

Is it the best available evidence?

Documents

Evidence-Base Practice