11

Student Reflections as DataStudent Reflections as Data

Dr. Allyson Fiona HadwinDr. Allyson Fiona Hadwin

University of Victoria University of Victoria

(EPLS & LTC)(EPLS & LTC)

hadwin@uvic.cahadwin@uvic.caUBC Institute for the Scholarship of Teaching and Learning

22

Examining and Interpreting Examining and Interpreting Data Collected over TimeData Collected over Time

Student reflection dataStudent reflection data

Student interview dataStudent interview data

33

Student Reflection DataStudent Reflection Data

JournalsJournals

PortfoliosPortfolios

Reflection assignmentsReflection assignments

44

Why do you want to look at Why do you want to look at this data in the first place?this data in the first place?

What is your research question?What is your research question?

55

There are different kinds of There are different kinds of research questionsresearch questions

Introducing the language so you know Introducing the language so you know where to look beyond this coursewhere to look beyond this course

Covered in many introductory research Covered in many introductory research methods texts in educationmethods texts in education

Creswell, J. W. (2005). Educational Research: Planning, Creswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and qualitative conducting and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.Hall.

66

Develop an understanding of what students gained through the experience (THEORY GENERATION)

What are some possible models to explain student learning in this program?

INDUCTIVE

?

77

Examine students' reflections according to theoretically driven constructs

(THEORY DRIVEN)

What kinds of sensitivities and awareness developed with respect to each of these

issues: X, Y, Z?

Theory verification

?

88

Understand the essence of student experiences of X.

What is the meaning of this learning experience for students?

Phenomenology

?

99

Compare students to one another

How were these students’ experiences similar or different?

Cross case study

?

1010

Compare students to themselves at different points in time

How did students change over the course of this instructional experience?

Within case study

?

1111

What kind of question do you What kind of question do you have?have?

Bottom Up

No idea what to expect

Looking for something

Top Down

1212

What is the question?What is the question?

A characteristic of reflection data:A characteristic of reflection data:

Usually interested in higher order, complex Usually interested in higher order, complex thinkingthinking

Critical thinkingCritical thinking

Community awarenessCommunity awareness

Self-regulated learningSelf-regulated learning

Difficult to operationalize and measureDifficult to operationalize and measure

1313

CautionCaution

Data analysis and interpretation are Data analysis and interpretation are limited (and afforded) by:limited (and afforded) by:

– the kind of data you collectedthe kind of data you collected– how you collected the datahow you collected the data– your own biases and roles in the processyour own biases and roles in the process

1414

Who is doing the analysis and Who is doing the analysis and interpretation?interpretation?

ResearcherResearcher

The student The student

Collaboration between researcher and Collaboration between researcher and studentstudent

1515

Beginning steps in the Beginning steps in the analysis processanalysis process

1.1. What is the unit of analysisWhat is the unit of analysis

2.2. Organize data according to unit of Organize data according to unit of analysisanalysis

3.3. Choose an analysis approach that is Choose an analysis approach that is consistent with your research consistent with your research questionquestion

1616

Unit of AnalysisUnit of Analysis

Individual

Responses to questions or themes Timing of events

Key Events Key Processes

1717

Data Sets Data Sets

1.1. Reflections on specific questions Reflections on specific questions collected by Dr. Dana Damian (University collected by Dr. Dana Damian (University of Victoria)of Victoria)

2.2. Reflections about studying and learning Reflections about studying and learning over 3 study sessions using computer over 3 study sessions using computer based technologies for learning based technologies for learning (collected by Allyson Hadwin)(collected by Allyson Hadwin)

1818

Data setsData sets

What unit of analysis might you What unit of analysis might you choose for each data set?choose for each data set?

Explain why?Explain why?

1919

Organize dataOrganize data

Make 2 copies of everythingMake 2 copies of everything

Make sure identifier information is on Make sure identifier information is on each document (and unit)each document (and unit)

Group data into the desired unit of Group data into the desired unit of analysisanalysis

Choose an appropriate analytic Choose an appropriate analytic approachapproach

2020

Semi-structured or Semi-structured or Loosely structured?Loosely structured?

Semi-structuredSemi-structured– Responses to specific questionsResponses to specific questions– Reflections on particular themesReflections on particular themes

Loosely structuredLoosely structured– Open discussionOpen discussion– Reflections about learning without specific Reflections about learning without specific

criteriacriteria

2121

Which data set is:Which data set is:

Semi structured?Semi structured?

Loosely structured?Loosely structured?

2222

Analytic approachesAnalytic approaches

Content analysisContent analysis

Case studyCase study

Inductive analysisInductive analysis

Narrative analysisNarrative analysis

Phenomenological analysisPhenomenological analysis

Grounded theory analysisGrounded theory analysis

2323

Combining analytic Combining analytic approachesapproaches

Often these analytic approaches are Often these analytic approaches are combined.combined.

I describe each separately to give you a I describe each separately to give you a sense of the main focussense of the main focus

2424

Content AnalysisContent AnalysisCode (from theory or data)Code (from theory or data)Pull apartPull apartRe-construct according to categories and Re-construct according to categories and themesthemesSummarized numerically & with quotesSummarized numerically & with quotesBuild theory and explanationBuild theory and explanationNot very sensitive to context of statements Not very sensitive to context of statements and ideasand ideas

2525

Narrative AnalysisNarrative Analysis

Preserves the wholePreserves the wholerepresents the perspective of the “teller”represents the perspective of the “teller”Themes in contextThemes in contextResearcher voice appears in explanation Researcher voice appears in explanation and representationand representationMany of the participants words includedMany of the participants words included

2626

Case Study AnalysisCase Study Analysis

Assemble all sources data for one case Assemble all sources data for one case (reflection, interviews, etc)(reflection, interviews, etc)Construct a case record –classifying and Construct a case record –classifying and editing raw data into an accessible packageediting raw data into an accessible packageWrite a case narrative (organized Write a case narrative (organized chronologically or thematically, or both)chronologically or thematically, or both)Pattern matching (comparing theoretically Pattern matching (comparing theoretically predicted patterns to data based patterns)predicted patterns to data based patterns)Explanation building (theory building)Explanation building (theory building)Time series analysis –trends over timeTime series analysis –trends over time

2727

Inductive analysisInductive analysis

Themes emerging from the dataThemes emerging from the data

Also top down sensitizing from the Also top down sensitizing from the researcherresearcher

Movement toward typologiesMovement toward typologies

Synthesis and explanation of typologiesSynthesis and explanation of typologies

2828

Grounded Theory analysisGrounded Theory analysis

Fine-grained analysis of small units (sentences or Fine-grained analysis of small units (sentences or phrases)phrases)

description

conceptual ordering

theorizing is interplay between data and theory

asking questions and making comparisons throughout

layers of coding from micro-analysis to modelling

iterative inductive cycle with specific stages

Constant comparative approach

Saturation

2929

Phenomenological analysisPhenomenological analysis

• -examining or uncovering the essence of a experience or behaviour

• -involves exploration of personal biases• -looking for what is "real” & teasing apart judgment• -bracketing or identifying data in its pure form

1. Epochè -look inside at personal bias2. Phenomenological reduction -bracketing from the

world and all presuppositions3. Horizontalizing -data spread out for interpretation4. Structural synthesis -deeper meaning or essence of

phenomenon is revealed

3030

Important elements of analysisImportant elements of analysis

Systematic approach (open about and Systematic approach (open about and describe the approach)describe the approach)

Coding dataCoding dataDescriptionDescriptionConceptual OrderingConceptual OrderingHigher order analysis or theorizingHigher order analysis or theorizing

Strauss and Corbin (Grounded theory)Strauss and Corbin (Grounded theory)

3131

Steps for your analysisSteps for your analysis

1.1. Focus your analysisFocus your analysis

2.2. Organize your dataOrganize your data

3.3. Decide upon and justify the Decide upon and justify the appropriate way to attack or analyze appropriate way to attack or analyze the data?the data?

4.4. (coding)(coding)

5.5. Conceptually organize data (try 2 Conceptually organize data (try 2 different ways of presenting findings different ways of presenting findings to bring meaning to themes)to bring meaning to themes)

3232

CodingCoding

Marking segments of data with symbols, Marking segments of data with symbols, descriptive words, labels or namesdescriptive words, labels or namesKeep a master list of codes, their meaning, Keep a master list of codes, their meaning, examples, & non-examplesexamples, & non-examples

Theory drivenTheory drivenData drivenData driven

3333

DescriptionDescription

Emerges from coding the dataEmerges from coding the data

What does the data sayWhat does the data say

What is importantWhat is important

What are the patterns?What are the patterns?

What are basic storiesWhat are basic stories

3434

DescriptionDescription

3535

Numerical descriptionNumerical description

3636

Conceptual Ordering -logical Conceptual Ordering -logical analysisanalysis

• Creating categories, finding means for identifying and Creating categories, finding means for identifying and displaying themesdisplaying themes

• Keeping an eye toward interpretationKeeping an eye toward interpretation

• Must be careful - once you start organizing data Must be careful - once you start organizing data according to a frame it becomes difficult to let go and according to a frame it becomes difficult to let go and try something different when it is not workingtry something different when it is not working

• MatrixesMatrixes• Concept mapsConcept maps• Summary tablesSummary tables• DiagramsDiagrams

3737

Conceptual orderingConceptual ordering

3838

Data may be organized:Data may be organized:

• Time orderedTime ordered• Role orderedRole ordered• Role by time orderedRole by time ordered• Role by groupRole by group• Conceptually clusteredConceptually clustered• Site dynamics Site dynamics • Predictor - outcomePredictor - outcome• Process -outcomeProcess -outcome• see Miles & Huberman (1984) for detailssee Miles & Huberman (1984) for details

3939

Numerical analysisNumerical analysis

4040

Interpretation -TheoryInterpretation -Theory

• Focus is illumination, extrapolation, and Focus is illumination, extrapolation, and understandingunderstanding

• Does not focus on cause and Does not focus on cause and consequenceconsequence

• Going beyond the data to connect with Going beyond the data to connect with theory or to generate theory/explanationtheory or to generate theory/explanation

• Dealing with rival explanations, Dealing with rival explanations, disconfirming cases, and data irregularities disconfirming cases, and data irregularities

4141

Higher order theorizingHigher order theorizing

4242

Group activityGroup activity

What have students learned What have students learned about their learning through the about their learning through the computer based learning activitycomputer based learning activity

4343

Analyzing data activityAnalyzing data activity

Focus your analysisFocus your analysis

Organize your dataOrganize your data

Decide upon and justify the appropriate Decide upon and justify the appropriate way to attack or analyze the data?way to attack or analyze the data?

(coding)(coding)

Conceptually organize data (try 2 Conceptually organize data (try 2 different ways of presenting findings to different ways of presenting findings to bring meaning to themes)bring meaning to themes)

4444

When a theoretical frame When a theoretical frame informs your analysisinforms your analysis

Coding scheme comes from the theoryCoding scheme comes from the theory

For example, you can use a rubric to For example, you can use a rubric to analyze (and assess) deeper level thinkinganalyze (and assess) deeper level thinking

Top Down

4545

Working with theory driven Working with theory driven constructsconstructs

Defining your constructDefining your construct– From the data (inductive)From the data (inductive)– From the theory (lens for the data)From the theory (lens for the data)

RubricsRubrics

Content analysisContent analysis

4646

An Example from Critical An Example from Critical ThinkingThinking

VanGyn, G., Ford, C., & Associates VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. (2005). Teaching Critical Thinking. Unpublished manuscript.Unpublished manuscript.

4747

Critical Thinking RubricCritical Thinking Rubric

Interviewed facultyInterviewed faculty

Examined Examined critical thinking critical thinking assignmentsassignments

looking for ways to define and assess looking for ways to define and assess critical thinkingcritical thinking

4848

Key Dimensions of critical Key Dimensions of critical thinkingthinking

Intellectual habitsIntellectual habits

Intellectual deliberationsIntellectual deliberations

Reflexive DispositionReflexive Disposition

4949

Intellectual habitsIntellectual habits

Intellectual curiosityIntellectual curiosity

Respect for truth and reasonRespect for truth and reason

Fair and open mindednessFair and open mindedness

Tolerance for ambiguityTolerance for ambiguity

Intellectual courage to take a positionIntellectual courage to take a position

Intellectual work ethicIntellectual work ethic

Willingness to engage in collaborative Willingness to engage in collaborative inquiryinquiry

5050

Intellectual deliberationsIntellectual deliberations

Identify the challenge, situation or taskIdentify the challenge, situation or taskGather, interpret and understand Gather, interpret and understand background information & evidencebackground information & evidenceApply thinking strategies relevant to the Apply thinking strategies relevant to the type of inquiry relevant to the challengetype of inquiry relevant to the challengeGenerate or select alternativesGenerate or select alternativesMake evaluative judgments among Make evaluative judgments among alternatives based on criteriaalternatives based on criteriaProvide justification for the conclusionProvide justification for the conclusion

5151

Reflexive dispositionReflexive disposition

Plan ahead for critical thinkingPlan ahead for critical thinking

Monitor its quality throughoutMonitor its quality throughout

Reflect on the strengths and limitations of Reflect on the strengths and limitations of the use of intellectual habits and the use of intellectual habits and intellectual deliberations in making a intellectual deliberations in making a judgmentjudgment

5252

Quality, rigor, credibilityQuality, rigor, credibility

Qualitative analysis is judged by different criteria Qualitative analysis is judged by different criteria largely governed by:largely governed by:

– Design chosenDesign chosen– Philosophical position (often represented Philosophical position (often represented

in designin design

Identify/reference the framework for credibility you Identify/reference the framework for credibility you are using *** Think of your writing as an opportunity are using *** Think of your writing as an opportunity to educate the reader about rigor in qualitative to educate the reader about rigor in qualitative researchresearch

5353

Internal validityInternal validity

Triangulation (data, perspective, theory)Triangulation (data, perspective, theory)

Member checksMember checks

Long-term observationLong-term observation

Peer examinationPeer examination

Participatory modelsParticipatory models

Exposing researcher biasesExposing researcher biases

5454

Fossey, et al. (2002)Fossey, et al. (2002)CongruenceCongruence

Responsiveness to social contextResponsiveness to social context

– Emergent research designEmergent research design– Sampling, data collection, analysisSampling, data collection, analysis

AppropriatenessAppropriateness

– SamplingSampling– Data collectionData collection

AdequacyAdequacy

– SamplingSampling– Data gathering and analysisData gathering and analysis– WritingWriting

5555

TransparencyTransparencyData collection and analysisData collection and analysis

AuthenticityAuthenticityPresentation of findings and interpretationsPresentation of findings and interpretations

CoherenceCoherencePresentation of findings and interpretationsPresentation of findings and interpretations

ReciprocityReciprocityAnalysis, findings and interpretationsAnalysis, findings and interpretations

TypicalityTypicalityWritten reportWritten report

PermeabilityPermeabilityFindings and interpretationsFindings and interpretationsWritten reportWritten report

5656

ReferencesReferencesCreswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and Creswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and

qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.

Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating qualitative reesarch. Australian and New Zealand Journal of Psychiatry, 36, 717-732.qualitative reesarch. Australian and New Zealand Journal of Psychiatry, 36, 717-732.

Hadwin, A. F., Hadwin, A. F., Boutara, L., Knoetze, T., & Thompson, SBoutara, L., Knoetze, T., & Thompson, S. (2004). Cross case study of self-regulation . (2004). Cross case study of self-regulation as a series of events. Educational Research and Evaluation, 10, 365-418.as a series of events. Educational Research and Evaluation, 10, 365-418.

Hadwin, A. F., Hadwin, A. F., Wozney, L., & Pontin, OWozney, L., & Pontin, O. (in press). Scaffolding the appropriation of self-regulatory . (in press). Scaffolding the appropriation of self-regulatory activity: A social constructivist analysis of changes in student-teacher discourse about a graduate activity: A social constructivist analysis of changes in student-teacher discourse about a graduate student portfolio. student portfolio. Special Issue ofSpecial Issue of Instructional Science.Instructional Science.

Hadwin, A. F., Hadwin, A. F., Wozney, L.., & Venkatesh, VWozney, L.., & Venkatesh, V. (2003, April). . (2003, April). A narrative analysis of the dynamic A narrative analysis of the dynamic interplay between students’ emerging task understanding and instructional scaffoldsinterplay between students’ emerging task understanding and instructional scaffolds . Paper . Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, USA. USA.

Krathwohl, D. (1998). Methods of educational and social Krathwohl, D. (1998). Methods of educational and social researchresearch: An integrated approach. New : An integrated approach. New York: LongmanYork: Longman

Miles, M. B., & Huberman, A. M (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage, 1994 Miles, M. B., & Huberman, A. M (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage, 1994

Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, Calif.: Sage.Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, Calif.: Sage.

Strauss, A. L., & Corbin, J. M. (1990). Basics of qualitative research: grounded theory procedures and Strauss, A. L., & Corbin, J. M. (1990). Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publicationstechniques. Newbury Park, CA: Sage Publications

VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. Unpublished manuscript.VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. Unpublished manuscript.