Using Qualitative Research Methods to Improve Physician … Grand Rounds... · ... (2014). On Methods: What’s the difference between qualitative and quantitative approaches? The

The Clinical and Translational Science Awards (CTSA) is a registered trademark of DHHS.

Using Qualitative Research Methods to Improve Physician Research Training at

the University of Cincinnati Jackie Knapke, PhD

Academic Program Director Clinical and Translational Research Training Program

University of Cincinnati

http://www.uchealth.com/


Learning Objectives

• Understand differences between qualitative and quantitative research.

• Describe how a theoretical framework can inform and influence qualitative research.

• Demonstrate how a qualitative research approach can be used to evaluate an educational program.



The MS in Clinical and Translational Research

The MS in Clinical & Translational Research degree was formally approved in 2009.

• 40 trainees currently enrolled • Over 140 graduates

The MS program provides training that helps students: • understand the principles of statistics and study design, • understand the medical literature, • communicate more effectively with a biostatistician, • prepare K and R grant applications, and • prepare for careers as independent investigators.



Previous Program Evaluation Research

• Over 200 graduates from our MS and Certificate programs.

• Alumni have obtained nearly $140 million in grant funding.

• Our current students and alumni have published over 1,500 articles in peer-reviewed journals.

• MS alumni also publish more articles than their non-alumni peers.1

• MS alumni are 3 times more likely than non-alumni fellows to have received at least 1 grant.2

¹Knapke JM, Tsevat J, Succop PA, Djawe K, Kuhnell P, Haynes EN. CTS. 2013;6(6):458-462. ²Knapke JM, Haynes EN, Kuhnell P, Tsevat J. CTS. 2015;8(1):52-56.



Quantitative studies show our MS program is highly effective . . .

So why undertake a qualitative study?



Differences between Quan & Qual Research Approaches1

Quantitative Qualitative

Focus Summarize characteristics across groups or relationships. Describe trends or relationships between variables.

Explore a topic about which little is known. Understand a phenomenon in a deep and comprehensive manner. Emergent design.

Purpose Be specific and narrow. Seek measurable, observable data on variables.

Be general and broad. Seek to understand participant experiences.

Researcher Role Detached and impartial. Objective portrayal. More deductive.

Personal involvement and partiality. Empathetic understanding. Reflexive. More inductive.

1(Ben-Eliyahu, 2014)



Quantitative Qualitative

Sampling and Sample Size

Random, Large

Purposeful, Small

Data Collection Collect mostly numerical data using instruments with pre-set protocol

Collect mostly textual or visual data using semi-structured or open-ended protocol, natural setting

Research Methods Measurements, biological samples, psychometric tests, questionnaires

Document review, interviews, focus groups, observations, creative artifacts

Data Analysis and Interpretation

Statistical analysis related to variables, interpret using prior predictions and past research

Textual analysis to describe themes and state larger meaning of findings

Differences between Quan & Qual Research Approaches1

1(Ben-Eliyahu, 2014)



Why Qualitative Research? (Maxwell, 2013; Rubin & Rubin, 2012)

• Depth

• Formative Evaluation vs.

Summative Evaluation

• Research with participants,

not on them



Qualitative Program Evaluation Research Type: Interpretive Phenomenology (Lopez & Willis, 2004) Purpose: To allow students to articulate their expectations, needs, and experiences in

the MSCTR program To develop novel training methods and/or curriculum modifications to

improve physician-scholar training



Categories: After initial line-by-line coding, create categories

from second-level coding.

Themes: Categories are refined and developed into relational

themes.

Patterns: Themes are connected to

develop constitutive patterns.

Methods: Document Review Group Level Assessment (GLA) Interviews Student Journals Survey Questionnaire Primary Method of Analysis: Modified Seven Stage Hermeneutic Analysis (Diekelmann, et al., 1989)



Recruitment and Sampling • Stratified Sampling Approach • n = 12

Participant Demographics • Sex

• equally split (6/6)

• Institutional Affiliation • 8 from CCHMC, 4 from UC

• Appointment • 10 fellows, 2 junior faculty

• Student Status/Progress • 2 early in coursework, 8 advanced, 2 alumni within last year



1. Why do trainees choose to pursue CTR training? 2. What knowledge or skills do students wish to obtain from their MS training? 3. What have students’ experiences been like thus far? 4. What areas of their educational program do trainees feel are lacking? 5. What changes would students make to the program’s curriculum and/or methods?

Research Questions

Member Checking and

Dissemination

Phase I Data

Collection Methods

• Document Review • Group Level Assessment

• Modified Seven Stage Hermeneutic Analysis • Group Level Assessment Analysis

Data Analysis

Phase I Findings

1. Why do trainees choose to pursue CTR training? 2. What knowledge or skills do students wish to obtain from their MS training? 3. What have students’ experiences been like thus far? 4. What areas of their educational program do trainees feel are lacking? 5. What changes would students make to the program’s curriculum and/or methods?

Research Questions

Member Checking and

Dissemination

Phase I Data

Collection Methods



Data Analysis

Phase I Findings

Phase II • Interviews • Student Journals • Survey

Data Collection Methods

• Modified Seven Stage Hermeneutic Analysis • Descriptive Statistics

Data Analysis

Final Findings



• Constructivism Theory for Curriculum Development (Bruner, 1960, 1985)

• Disciplinary Socialization Theory (Bess, 1978; Lave & Wenger, 1991; Merton, 1949)

• Andragogy Model (Knowles, 1973, 1980, 1984)

Theoretical Framework



Constructivism Theory for Curriculum Development (Bruner, 1960, 1985)

Four Key Characteristics of a Spiral Curriculum

• Topics are revisited. • As topics are revisited, the level of difficulty increases. • New knowledge and experiences are related to existing knowledge and experiences. • Student competency increases with each visit until overall learning objectives are achieved.

2 + 2 = 4

½ + ¾ = 1¼

4x + 5 = 47



Disciplinary Socialization Theory (Bess, 1978; Lave & Wenger, 1991; Merton, 1949)

• Students begin to

anticipate the types of attitudes and behaviors expected of them within their chosen discipline.

Stage I: Anticipatory

• Students are novices, but they carefully observe their professors and older peers to learn normative role expectations.

Stage II: Formal

•The types of behaviors that are acceptable and valued within a field are further indoctrinated into the student.

Stage III: Informal

• Final stage where students fully internalize their professional identities within their field.

Stage IV: Personal

The Four-Stage Model of Disciplinary Socialization

(based on: Stein & Weidman, 1989)



Andragogy Model (Knowles, 1973, 1980, 1984)

Four main assumptions of andragogical theory:

• Changes in Self-Concept

• Role of Experience

• Readiness to Learn

• Orientation to Learning

Pedagogy Andragogy

Greek root paid + agogus aner + agogus

Meaning children + leader of man (vs. boy) + leader of



Phase I Data

Collection Methods



Data Analysis

Phase I Findings Document Review: 1. Personal experiences (A) 2. Improve clinical care (A) 3. Mentorship/teaching (DS) 4. Collaboration (A-DS) 5. Professional goals (A) GLA: 1. Physician-specific

coursework (A-CC) 2. Directed curriculum (CC) 3. Curriculum unity (CC)

Research Process: Phase I

CC: Constructivist Curriculum DS: Disciplinary Socialization A: Andragogy



Phase I Phase II Phase II

Final Findings

Data Collection Methods

• Interviews • Student Journals • Survey


Data Analysis

Phase I Findings Document Review: 1. Personal experiences (A) 2. Improve clinical care (A) 3. Mentorship/teaching (DS) 4. Collaboration (A-DS) 5. Professional goals (A) GLA: 1. Physician-specific

coursework (A-CC) 2. Directed curriculum (CC) 3. Curriculum unity (CC)




Final Findings

Phase II Data

Collection Methods

• Interviews • Student Journals • Survey


Data Analysis

Final Findings Five Patterns: 1. Curriculum (CC) 2. Mentorship/Relationships (DS) 3. Program Structure &

Organization (A) 4. Instructional Methods (A) 5. Ancillary Student Perspectives

on MSCTR Experiences

Member

Checking and Dissemination




Final Findings: 5 Patterns with Themes 1. Curriculum

• Flexible, but Directed (A-CC) • Statistical Instruction (A-CC) • Integration (CC) • Course Expansion/Development (CC)

2. Mentorship/Relationships 3. Program Structure & Organization 4. Instructional Methods 5. Ancillary Student Perspectives on MSCTR Experiences




1. Curriculum

2. Mentorship/Relationships • Formal Mentorship (DS) • Early (DS) • Other Relationships (DS)

3. Program Structure & Organization 4. Instructional Methods 5. Ancillary Student Perspectives on MSCTR Experiences

Final Findings: 5 Patterns with Themes




1. Curriculum 2. Mentorship/Relationships

3. Program Structure & Organization

• Scheduling (A) • Administrators are critical (A) • Individualized Advising (A)

4. Instructional Methods 5. Ancillary Student Perspectives on MSCTR Experiences





1. Curriculum 2. Mentorship/Relationships 3. Program Structure & Organization

4. Instructional Methods

• Physician-centered (A-CC) • Applied (A-CC) • Hybrid/Blended Learning (A-CC)

5. Ancillary Student Perspectives on MSCTR Experiences





1. Curriculum 2. Mentorship/Relationships 3. Program Structure & Organization 4. Instructional Methods

5. Ancillary Student Perspectives on MSCTR Experiences

• Academic Integrity • Research Experience





Concluding Recommendations

1. Update the overall curriculum. a. Curriculum Overhaul b. Minor Curriculum Updates

2. Improve statistical training.

3. Invest in online courses, both existing and new.

4. Create a more structured mentorship program within the

MSCTR.



Quantitative studies show our MS program is highly effective . . .

So why undertake a qualitative study?



Acknowledgments

This research was supported by an Institutional Clinical and Translational Science Award, NIH/NCRR 5UL1RR026314.



References

Ben-Eliyahu, A. (2014). On Methods: What’s the difference between qualitative and quantitative approaches? The Chronicle of Evidence-Based Mentoring. Retrieved September 7, 2015, from http://chronicle.umbmentoring.org/on-methods-whats-the-difference-between-qualitative-and-quantitative-approaches/.

Bess, J. L. (1978). Anticipatory socialization of graduate students. Research in Higher Education, 8, 289-317. Bruner, J. (1960). The Process of Education. Cambridge, MA: The President and Fellows of Harvard College. Bruner, J. (1985). Models of the Learner. Educational Researcher, 14(6), 5-8. Creswell, J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches. Thousand Oaks, CA: Sage. Diekelmann, N., Allen, D., & Tanner, C. (1989). The NLN criteria for appraisal of baccalaureate programs: A critical hermeneutic analysis. New

York, NY: National League for Nursing. Gilbert, J. (2009). Constructivism: Learning by Doing. A Spiral Approach to Curriculum. Retrieved May 15, 2014, from

http://www.nyu.edu/classes/gilbert/musedtechwkshp/learn.html Knapke, J. M., Haynes, E. N., Kuhnell, P., & Tsevat, J. (2015). NIH granta awards as a metric of clinical and translational research training

effectiveness. Clinical and Translational Science, 8(1), 52-56. Knapke, J. M., Tsevat, J., Succop, P. A., Djawe, K., Kuhnell, P., & Haynes, E. N. (2013). Publication track records as a metric of clinical research

training effectiveness. Clinical and Translational Science, 6(6), 458-462. Knowles, M. S. (1970). The Modern Practice of Adult Education. New York, NY: Association Press. Knowles, M. S. (1973). The Adult Learner: A Neglected Species. Houston, TX: Gulf Publishing Company. Knowles, M. S. (1980). The Modern Practice of Adult Education: From Pedagogy to Andragogy. Cambridge: Adult Education Company. Lave, J., & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press. Lopez, K. A., & Willis, D. G. (2004). Descriptive versus interpretive phenomenology: Their contributions to nursing knowledge. Qualitative

Health Research, 14(5), 726-735. Maxwell, J. A. (2013). Qualitative research design—An interactive approach, 3rd edition. Thousand Oaks, CA: Sage. Merton, R. K. (1949). Social Theory and Social Structure. Glencoe, IL: The Free Press. Rubin, H. J., & Rubin, I. S. (2012). Qualitative interviewing: The art of hearing data (3rd ed.). Los Angeles: Sage. Stein, E. L., & Weidman, J. C. (1989). Socialization in graduate school: A conceptual framework. Paper presented at the Association for the

Study of Higher Education, Atlanta, GA.


http://chronicle.umbmentoring.org/on-methods-whats-the-difference-between-qualitative-and-quantitative-approaches/

http://chronicle.umbmentoring.org/on-methods-whats-the-difference-between-qualitative-and-quantitative-approaches/

http://www.nyu.edu/classes/gilbert/musedtechwkshp/learn.html

Documents

Using Qualitative Research Methods to Improve Physician … Grand Rounds... · ... (2014). On Methods: What’s the difference between qualitative and quantitative approaches? The