Evidence-Based Management: A New Approach to Teaching the Practice of Management

Evidence-Based Management: A New Approach to Teaching the Practice of ManagementRoye WernerPDW, Annual AOM 2011, San AntonioEric BarendsRob BrinerBlake JelleyDenise RousseauLori Peterson

Rob Briner EBMgt: Whats stopping us?Roye WernerWithout a map Eric Barends5-step pull approachBlake JelleyStrategy & AssignmentsLori PetersonLessons Learned- SubgroupsExperiences, Feedback, Support

PDW, Annual AOM 2011, San Antonio


Rob BrinerPart 1: EBMgt: Whats Stopping Us?

THE UNDERLYING ARGUMENTFour propositions:Research produced in business schools could be useful to organizationsDrawing on the best available evidence (including business school research) is likely to improve management decisionsManagers and organizations do not appear to be strongly aware of nor use research findingsWe need to increase awareness of and access to research findings

WHATS STOPING US?The quick fix problemThe management fad (fashion) problemWhy dont some academics and researchers like EBMgt?Why dont some managers and practitioners like EBMgt?

QUICK FIXESWhat is the quick fix? A solution whichFocuses on style and presentation not contentIs always slower than we hopedUsually doesnt workIs followed by another quick fixSo why do we do quick fixes?Can be career-enhancing for managersSpeed is often valued over accuracyDo we crave quick and easy solutions?

So who needs or wants academic research?

MANAGEMENT FADS (1)The nearly-forgotten fads

Scientific Management/TaylorismBusiness Process ReengineeringManagement by resultsExcellenceTotal Quality ManagementLearning OrganizationsKnowledge Management

The fads that havent been forgotten (yet)Talent managementManagement developmentExecutive coachingEmotional intelligenceEmployee engagementMyers Briggs Type IndicatorBelbin Team RolesGeneral concern about the destructive impact of fads from both practitioners and researchersMANAGEMENT FADS (2)

FADS* SEEM TO BE ATTRACTIVE, COMPELLING AND IRRESISTIBLEPromise to deliver a lot and fastAppear simpleNew and shinyWill make everything alright and help contain anxieties around intractable problemsHelp user feel effective and cutting edgeBits of some fads may work in some contexts

So who needs or wants academic research?

*Evidence-based management not a fad!

WHY DONT SOME ACADEMICS LIKE EBMgt?Ambivalence about the value and applicability of management researchFew incentives to do systematic reviews or CATsPrimary research (collecting new data) valued more highly than secondary research (reviewing existing data)EBMgt not academics responsibility this is about practice not researchSome concern that systematic reviews will expose the limited nature of management researchSome academics are like gurus and feel that EBMgt might show their claims to be untrueFew incentives to get involved

WHY DONT SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (1)Undermines formal authorityThey feel it constrains freedom to make managerial decisionsSpeed valued and rewarded more than accuracyFeel they cannot use their own experience and judgment (not true)Managers not necessarily rewarded for doing what works (organizations rarely evaluate)Few incentives to get involved

Huge pressure to adopt fadsHave too high expectations of evidence (e.g., that it should give the answer otherwise its not worth knowing, media representations of science focus on breakthrough single studies causes distrust of scientific findings)WHY DONT SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)

Huge incentives and pressure to adopt fads

And there we see the power of any big managerial idea [or fad or fashion]. It may be smart, like quality, or stupid, like conglomeration. Either way, if everybody's doing it, the pressure to do it too is immense. If it turns out to be smart, great. If it turns out to be stupid, well, you were in good company and most likely ended up no worse off than your competitors. Your company's board consists mostly of CEOs who were probably doing it at their companies. How mad can they get?WHY DONT SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)

Huge incentives and pressure to adopt fads

The true value of conventional management wisdom is not that it's wise or dumb, but that it's conventional. It makes one of the hardest jobs in the world, managing an organization, a little easier. By following it, managers everywhere see a way to drag their sorry behinds through another quarter without getting fired. And isn't that, really, what it's all about?

(Colvin, 2004, Fortune)

WHY DONT SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)

SO WHATS STOPPING US?Few incentives for researchers, business school teachers, or practitionersNot a case of removing barriersNeed to start doing it through our teaching and education to demonstrate the benefits of evidence-based approaches for individuals, professions, and organizationsWe are stopping us!

Part 2: Without a MapLife in the (EBMgt) information universe Roye Werner

Finding the evidence: what Ill cover: Where does this fit in the teaching of EBMgt?How is information on how to manage structured?How is research on how to manage structured?Tips for teachersTips for searchersTips for managersFuture visions

Where this fits in the 5 StepsFormulate an answerable questionSearch for the best available evidenceCritically appraise the evidenceIntegrate the evidence with your managerial expertise and organizational concerns and applyEvaluate the process

Peer-reviewed (vague estimate)Map: Management Advice What they seeFound on Google?

Articles in magazines, newspapers, trade publications, journals. Example: 1,471,332 under subject heading Management in ABI-Inform database - of those, 135,778 peer-reviewed. Print is almost gone.Books from academic, trade, and popular presses. Amazon lists 10,000 books on Management Science and 6,500 on Organizational Behavior. Also large sections in bookstores. Print still rules, but how long before ebooks take over?Websites, blogs, apps, social networks, open courseware. i.e. MIT Sloan courses, HRM Today network, My MBA Human Resources app, HRSpace twitter, business professor blogs, British Library Management PortalOrganizations. i.e. SHRM, Conference Board, CEBMa(!)Consultants. US has 361,206 management consultancies listed in Dun & Bradstreet.

Google ScholarProQuest (ABI Inform,etc.)EBSCO (Business Source Premier, etc.)JSTORLibrary discovery service, (Summon, Worldcat Local, Primo Central, etc.) or federated searching programDatabases (3 popular ones) Academic books (soon to be mixed with databases)Map: Management Advice What you want them to see

Tips for Teachers: 1Find out what databases your university library offers

Work with your librarians. They can teach classes, confer with students, create guides. They want to do this!

Tips for Teachers: 2

Example of a librarian-created guideTips for Teachers: 2

Offer hands-on training and practiceTips for Teachers: 3

Put links to resources on your course management system

Tips for Teachers: 4

Google Scholar excellent results, but text not part of the dealGood:Powerful searchRecognizedWider rangeEasy, intuitiveSome things are freeMore languagesPreferred route for manyNot so good:Less control over searchSome non-scholarly itemsDuplicatesNo citation helpConnection to univ. unclearNeeds subscriptionsTips for Teachers: 5

Some basic principlesDatabases are different from search engines more structured, can take advantage of human-assigned categoriesBoolean AND, OR, NOT, parentheses combining setsQuotation marks define a phrase usuallyTruncation symbols (often *)Field searching limiting to title, abstract, subject fieldsGive Advanced Search a tryLook for a tutorial 5 minutes there can save you much more in your search

Tips for Searchers

Public libraries have more than you thinkIs an abstract good enough? Google Scholar provides. (If not, public libraries do interlibrary loan also.)Small Business Development CentersWork with local universities and professorsBecome a member of CEBMaTips for Managers

Open Access please sign onSystematic reviews: abundant, accessible and comprehensibleAll in one place - example: ProQuests EBMedicine search model

Mobile access: get readyFuture Visions

Eric BarendsPDW, Annual AOM 2011, San AntonioPart 3: The 5-step pull approach

Half of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half.The most important thing to learn is how to learn on your own.Pioneer

An EBM approach to educationJAMA, 1992

Push vs PullPush: teaching management principles based upon a convergent body of research and telling students what to do.

Pull: teaching students how to find, appraise and apply the outcome of research (evidence) by themselves

The 5 steps of pull EBM

(0). Create awareness Formulate an answerable question Search for the best available evidence Critically appraise the evidenceIntegrate the evidence and apply Monitor the outcome

The 5 steps of pull EBM0. Aware1. Ask2. Acquire3. Appraise4. Apply5. Analyze & Adjust

Objectives of the course1. What What is evidence-based management?What does it look like in practice? 2. HowFormulate Answerable QuestionsSearch for evidenceAppraise a studyIntegrate and apply

1. Asking the right questions

Answerable questionI am a consultant, my client a large health-care organization. The board of directors has plans for a merger with a smaller healthcare organization. However, its been said that the organizational culture differs widely between the two organizations. The board wants to know if this can impede a successful outcome.

P =Population or problem I = Intervention or successfactorC = Comparison O = OutcomeC = ContextAnswerable question: PICO(C)

Answerable question: PICOCP: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?

P =back office employees, organizations with a different organizational cultureI = merger, integration back office C = organizations with a similar organizational culture O = economy of scale C = healthcare organizations, unequal Answerable question: PICOC

Step 2: Finding the best available evidence

Where do we search?

Search strategieTwo types of search strategies

Search results

Hands on!

Making sense of evidenceStep 3: Critically appraise the evidence

Critical appraisalHow to read a research article?

Critical appraisalStudy designsLevels of evidenceBias / confoundingEffect sizesExternal validity

Research designsSystematic reviews & meta analysisRandomised controlled study (experiment)Non-randomised controlled study (quasi-experiment)Correlational studyBefore-after studyQualitative research

Which design for which question?Research designsThe best evidence depends on the question type !

Levels of evidence

Levels of Evidence: internal validity

Methodological pitfalls

Bias Confounding

Step 4: Turning evidence into practice

Is your organization / division / population so different from those in the study that its results cannot apply? How relevant is the study to what you are seeking to understand or decide? What are your organizations potential benefits and harms from the intervention? Is the intervention feasible in your setting?Organization concernsAlways ask yourself to what extent the evidence is applicable in your situation:

P =back office employees vs middle managersI = integration back office vs integration board C = O = economy of scale vs shareholder valueC = healthcare organization vs financial service organizationAnswerable question: PICOC

CAT: Critically Appraised Topic

CAT: Critically Appraised TopicA critically appraised topic (CAT) is a structured, short (3 pages max) summary of evidence on a topic of interest, usually focused around a practical problem or question. A CAT is like a quick and dirty version of a systematic review, summarizing the best available research evidence on a topic.

CAT: structure Background / context Question (PICOC) Search strategy Results / evidence summary Limitations Conclusion Recommendation

CAT-walk

An example

Part 3 Initial Strategies and ReflectionsR. Blake Jelley, Ph.D.PDW, Annual AOM 2011, San Antonio

BackgroundTwo faculty members leading EBMgt @ UPEI.PhD management (Wendy Carroll)Joined UPEI 2008; seasoned executive; inaugural EMBA co-director; award-winning instructor; active researcher.PhD industrial-organizational psychology (me)Joined UPEI 2007; scientist-practitioner; 5 years in applied research (Ontario Police College) technical & peer-reviewed pubs.UPEI EMBA program launched in 2008.My course assignment: Business Research Methods.*

Focus for EBMgt teaching (to date) is the EMBA.Some preliminary work with undergraduates.Current Components.EMBA Orientation.Managing People & Organizations.Business Research Methods.Some other courses, too.EMBA Signature Project.Alternative approaches to EBMgt Teaching.(Reasonably) integrated approach (ours).An EBMgt-titled course (e.g., Rousseau).Background

Based on Jelley, Carroll, & Rousseau (in press).Various exercises and assignments across different courses.Developing Basic Awareness of EBMgt.Define EBMgt.Rynes et al. (2002) HR Knowledge Quiz.Self-Guided Field Trip.EBMgt as content (push) and process (pull).Explicit consideration of EBMgt facets.

Teaching Strategies I

Learning to Ask the Right Questions.Decision awareness; practitioner focus.Absenteeism Diagnosis exercise.Burning Question assignment.Preliminary Research Proposal assignment.Question Specification exercise.Surfacing Assumptions exercises.Teaching Strategies II

Getting the Best Available Evidence.Rousseaus rule #1.Never trust truth claims made without references! It leads directly to Rule #1.1, Do your due diligence: Check (at least a sample of) those references.Introduction to the business research landscape.Literature Search Workshops.Literature review; systematic review.Snake Oil Symposium.Primary research exercises/workshops.Teaching Strategies III

Critically Appraising the Evidence.

Brief intro. to philosophical issues/perspectives.Limited or inclusive use of evidence hierarchies.Critical appraisal of all four EBMgt elements.Research Trinity: Design, measurement, & analysis.Indicators of good research; reviewers checklists.Variable Type and Hypothesis Example exercise.Favorite Article Review exercises.Myth of Market Share reflective paper.Get the Evidence, Myth Busting, etc. group assignments.

Teaching Strategies IV

Integrating EBMgt Elements in support of Organizational Decisions and Actions.Classroom-based education provides a foundation.(For example, Rousseaus & Carrolls assignments)Thats Incredible! assignment.Persuasive Paper.The Great Debate.Learning Diary.Letter to Self (e.g., www.futureme.org).Teaching Strategies V

Evaluating the Process.After Action Review.

Evaluating our work.Pre- and Post-course Assessments.Need for measure development and formal evaluation research.Teaching Strategies VI

Realization that evidence is not often used to support decision-making.Claims of using learning at work; some specific demonstrations.Reading Method sections of peer-reviewed articles allegedly.Meta-analyses may now be favorite reading materials.Realization that information exists beyond personal experience and first page of Google hits.A lot of material; not easy; make it more down to earth; practical assignments.- BUT realization that easy may detract from learning.Variance among students in uptake and understanding.Informal Feedback

Teaching strategies and resources.Continue to develop them.Adapt existing ideas.(For me, more attention to the final two steps)Measuring the EBMgt mindset and skills.Formal research.EBMgt education evaluation.EBMgt education outcomes as predictors (mediators) of organizational criteria?Future Directions

Part 4Lessons LearnedLori Peterson, PhDCleveland State University


CAT Assignment ComponentsProfessor ProvidesStudent DeliverablesEBMgt Lecture background (covered in class 1-3)Research tipsLibrary tool overviewReal life exampleGuidance at each stepFeedback

Question developmentPICO(C) templateDraft written CAT SummaryPeer Review of draft CAT SummaryFinal written CAT SummaryPeer Review of Final CAT SummaryCAT Walk (presentation)Reflection Memo

Question Development Too broadToo narrowToo easily answeredDifficult to answer or find information Topic too clinical for Management or Administration studentsPicking a topic from the universe of health care related subjects

Research Tips and PitfallsProcrastination waiting, starting late, other classesUnable to access articles through pay servicesSolution: Interlibrary loan and access through the off-campus library log-inSorting fact from opinion piecesToo much information or question too broadDated (prior to 2000) information for evolving topicsLearning tools or tactics earlier:Examples: search engine tactics and reference managers

Writing ProcessAssignment limited page length (5 pages) and provided headings which helped with structureReading the articles was more time consuming than many expected, until they got better a selecting articles based on the abstractMost questions were slightly revised in the process (as was expected and encouraged)Writing has not been a problem, thanks, in part, to the research that is required (10 sources minimum)Getting started or staying focused while writing may be another story

CAT Summary Headings Question Background / context Results / evidence summary Comments (limitations) Recommendations Conclusion References Appendix PICO(C) TableSummary of Research TableOther relevant information (tables, charts, etc)

Peer ReviewIn class, one week before written paper dueInformalHelp with logic or gaps in content, not typosMost papers were incomplete at this pointOut of class, random assignment, 2 per studentFormal, used a templateTemplate is a work in progressDesigned to be developmental for future writingMany students accesses more papers than they were assignedHelped with questions during the CAT Walk

CAT WalkPresentation expected 4-6 minutes, with a maximum time of 10 minutes for questionsLimited slides to 6, slides were optionalTypical issues: Nervousness in presentationLack of preparationReading slides or notesBack to audience (because of reading)

Reflection MemoNumber of article abstracts reviewed, articles downloaded, and articles used in summary Why were most articles rejected?What was your search strategy? (list the tools used. Also, consider how you would explain what you did if someone were to replicate your search.) What would have made your search more effective?What will you take away from this assignment?What would you have liked to learn sooner to make this process easier for you?

Experiences, Barriers & FeedbackGroup 1: Denise & LoriGroup 2: Blake & Rob

Support, Teaching MaterialGroup 3: Roye & EricPDW, Annual AOM 2011, San AntonioGroup Discussion

The Center for Evidence-Based ManagementWWW.CEBMa.orgPresentationsHandbook of EBMGt: Jelley, R. B., Carroll, W. C., & Rousseau, D. M.:Reflections on teaching evidence-based management

Further Information

CAT: experiences so farPositiveClear (completely) new approachLibrary search useful (much more evidence than expected)Independency (managers can search for themselves)Strong arguments, empowering (evidence speaks for itself)

NegativeAccess to relevant databases (solved!)Lack of aggregated evidenceTranslating conclusions to own contextQuick and dirty: jumping to conclusion

****************PRESENTATION ONE*Introduction to Evidence-Based Practice********* *[Be sure to acknowledge Wendy and Denise RE: the chapter upon which much of this presentation is based.]**

[Mention an adapted version of Eric Barends five step pull EBMgt approach is key to our chapter on teaching EBMgt in which different courses from different masters programs are described.]**Burning Question: FINERPRP: What dont we know that matters?Question Specification: CIMO*******The question writing process was a back and forth between the students and myself. I utilized a discussion thread on blackboard for the students to post their questions as I did not want two students working on the exact same topic. Experience in previous semesters has shown that the questions evolve as the research is conducted. I did not want to limit the students and with the blackboard discussion thread, all students have access to the topics the other students have posted.

While there were some physicians in the class, some of the non-clinical students attempted the clinical topics. Clinical topics are difficult as the students would have more trouble getting access to the research, so I encouraged these students to pick a less clinical topic. One example was Migraines, I encouraged the student away from the why of migraines to the impact of work time lost due to migraines which the student was able to find information on.*Individual student assignment, no groupsAll students selected a topic they were interested in and related to the course. No two students could use the same topic

Scenario given: Student was newly hired into a high profile role. The executive leadership team has a problem they are trying to solve and they want you to research it and come back in three months with some background information and to help them make a decision. The deliverables are a 3-5 page written paper and a 4-6 minute presentation. As a new hire, this is a tremendous opportunity to make a good impression with the executives*The question writing process was a back and forth between the students and myself. I utilized a discussion thread on blackboard for the students to post their questions as I did not want two students working on the exact same topic. Experience in previous semesters has shown that the questions evolve as the research is conducted. I did not want to limit the students and with the blackboard discussion thread, all students have access to the topics the other students have posted.

While there were some physicians in the class, some of the non-clinical students attempted the clinical topics. Clinical topics are difficult as the students would have more trouble getting access to the research, so I encouraged these students to pick a less clinical topic. One example was Migraines, I encouraged the student away from the why of migraines to the impact of work time lost due to migraines which the student was able to find information on.*This is a simple template I have prepared for the students to think about their research planning. Several students commented they wished they had something like this earlier in their university days, either as undergrads for research papers, or earlier in graduate school It is a work in progress, but it does seem to help focus the students.*Student have access to university services, PubMed, OhioLink, some had research services through their employer, etc. Many utilized Google Scholar with more success in 2nd semester, surprisingly. There are some tips I think the students could learn about library research to improve their process. The pay services included Elsevier, Springer, etc and journals published by these organizations, using Interlibrary loan many of their article retrieval issues would have been solved.

Assignment required 10 sources, some reviewed 30+ articles and over time, got better at reading the abstracts to see if the paper was even helpful.

Evolving topics is a problem. With the passage of health care reform, many of the students wanted to research this topic, however, there are fewer academic articles that will help them. Encouraging them toward topics they would be able to find information on. Of course, they can research anything they want, but some topics are better left for other classes*First class received less information on the structure and struggled more with the elements. Second class received more information on what the sections mean and we spent time on what the summary research table should look like.

Only one student in two semesters submitted a research report, the others compiled research that was more like an executive summary

*The writing was not a problem as most students had much more research than they could utilize. Assignment called for 3-5 pages, all students utilized 5 pages both semesters. I am considering adding more pages to the next assignment *Reflection suggested perhaps the in class review be earlier, I will try to fit this into the schedule for next semesterIn class is not graded, but not having a draft meant no points could be awarded for the activity. I reviewed these papers, scanned and emailed comments or summarized comments to studentsStudents had experience with peer review earlier in semester through another assignmentOften students write papers and the only person who sees them is the professor, this way, others see their work and perhaps they work harder when they know their peers see their work? Note, I do not use the peer grades when developing my evaluation as there is significant grade inflation among peer grades. No one wants to mark down their classmate*Students delivered 2 Pecha Kucha during term, which helped with structure for the short presentation formatWith the 2 Pecha Kucha presentations, the 2nd class presented much better than the first class. Pecha Kucha has gotten great reviews by both MBAs and undergrads. While a short presentation can be daunting, the preparation during the semester helped with nerves and content first pecha kucha was much worse! These issues can be combated by more presentations, more weight on presentation, different audience (community leaders/alumni), any number of options, but none were so serious as to fail the student.*The Reflection Memo assignment questions are included in the assignment handout. Memos are the typical way the students communicate with me. The insights the students provide help me think about how I would change the assignment for the next time, if I offer the assignment at all. This is not an evaluation of the course, simply an insight into the assignment, which is an important distinction for some students. *** *

Documents

Evidence-Based Management: A New Approach to Teaching the Practice of Management