Srl research lecture

MEASURING SELF-REGULATED LEARNING

Piia Näykki

Learning and Educational Technology Research Unit

1

OUTLINE OF THE LECTURE

1. Key points of SRL and measurement

2. Research protocols that are currently used to measure SRL

3. Constructive critique for the measures

4. Primary vs. secondary research

5. Forecast what measurements of SRL might be in the future

27.11.2015

2

KEY POINTS IN SRL

The general agreement in the research field:

• The term SRL is associated with forms of learning that are

metacognitively guided, intrinsically motivated, and strategic

• There are a number of SRL models

• SRL models emphasizes different phases

• SRL is a cyclical process

3

This is what we already know!

hmmmmm.. But how to study SRL?

4

BRAINSTORM in pairs (5 min):

1. Think, what do you allready knowabout Self-Regulated Learning and

2. What aspects of SRL you would beinterested to study?

3. Formulate a list of interestingquestions related to SRL.

4. Prepare yourself to share your ideas to the class.

5

SHARE your ideas to the class,

As we continue the lecture, think followingquestion:

- How could this topic of my interest bestudied?

KEY POINTS OF MEASUREMENTS OF SRL

1. Measurement of SRL should reflect a model of SRL.

2. From the description of SRL it is obvious that many facets of

SRL are not readily observable.

3. One challenge of studying SRL is to find ways to document its

components.

4. How to study SRL as an aptitude and/ or as an event?

27.11.2015

6

SRL AS AN APTITUDE… AS AN EVENT

1. When SRL is measured as an aptitude, a single measurement

aggregates over or abstracts some quality of SRL based on

multiple SRL events.

Example:

- student studying for a test. A researcher may be interested in mc

monitoring concerning a rehearsal tactic.

- The student’s description of SRL might be recorded in several

forms: a rating on a questionnaire item, an interview’s

classification or the proportion of the particular kinds of notes

student has written to the textbook.

- To predict whether a student will/ will not, can/ can not act on an

SRL-related cognition.

27.11.2015

7

SRL AS AN APTITUDE… AS AN EVENT

1. When SRL is measured as an event has 3 levels: occurency,

contingency and patterned contingency.

Example:

- Student is solving a geometry problem and thinking aloud.

- The student says: ”wow, this is hard”.

- The researcher may presume that s/he have monitored the

current state of the task (task difficulty).

- What are the standards against this monitoring action has been

done? (we don’t know).

- Student’s report is interpreted as indirect evidence.

27.11.2015

8

WHY IT IS SO DIFFICULT TO MEASURE SRL

• SRL is an internal process• Can we ask individuals to tell their internal processes?

• What problems are related to that?

• SRL has an external signs,• What kind of signs?

• Are we sure that our interpretations of the signs are valid?

• What problems are related to that?

9

TENSIONS IN MEASURING SRL

• General Reliability & validity

• Big sample size vs. sample with a meaning

• Self-report critique: Self-report is too static and ”Individuals

tend to tell what they expect researchers want to hear”.

• Qualitative approaches critique: video observation methods are

subjective and based on researchers’ interpretation, not

objective.

• In sum: Some of the features of the classical

QUANTITATIVE VS. QUALITATIVE divide

10


1. Key points of measurements in general

2. SRL: components researchers and practioners seek to

measure





27.11.2015

11

PROTOCOLS FOR MEASURING SRL

Winne & Perry; Boekaerts & Corno (2005) + Handbook2011

AS AN APTITUDE

• Self-report questionnaire

• Structured interviews

• Teacher judgements

AS AN EVENT

• Observations of overt behavior (video/ real time)

• Thinking aloud protocols

• Traces of mental events and processes (computer)

• Learning diaries

• Interviews (possible with a stimulus)

12

13


1. Take your list of interesting questionsrelated to SRL.

2. Now think and discuss in pairs: howcould your topic of interest be studied?

3. Prepare yourself to share your ideas to the class.

OBSERVATIONS OF OVERT BEHAVIOR

• The famous Marshmallow experiment

http://www.youtube.com/watch?v=QX_oy9614HQ

• 1960’s Standford University

• 4 year old kids, 15 minutes to waite

• 2/3 could not waite..

• 1/3 – delayed gratification

• 14-15 years later follow-up studies with the same kids.• Around 100% of those who waited were successful in school

• Around 80% of those who could not waite were in troubles.

27.11.2015

14

http://www.youtube.com/watch?v=QX_oy9614HQ

OBSERVATIONS OF OVERT BEHAVIOR

• Capture what the learner is doing• Based on verbal (talk), actions (gestures) and interactions

• Use a coding categories (pre-defined or based on the data)

• Recording for inter-judges reliability

Problems:

• Are we really accessing the students SRL?

• Do we make inferences as researchers?

• How does a SRL ”look like”? How it can be recognized? How to

operationalize the phenomenon?

27.11.2015

15

THINKING-ALOUD PROTOCOLS

Example of thinking-aloud (02:45-03:40)

http://www.youtube.com/watch?v=px0A4jV02RA&feature=relate

d

• Students say out loud what they are doing, thinking, etc.

• Ericsson, K. A., & Simon, H. A. (1980)

• Coding and inter-judges

• Researcher presence should not affect but activate

• Direct measure on-task

• Problems: requieres training, reliability with younger children,

task difficulty, high cognitive load, frequency affected by

personal factors.

27.11.2015

16

http://www.youtube.com/watch?v=px0A4jV02RA&feature=related

DIARIES

• Students report in a scheduled basis their strategy use and

problems

• Bernhard Schmitz & Julia Klug (Handbook, 2011)

• Qualitative but also quantitative (structured questions)

• Time series analysis

• Type of Self-report

• Problems: Reactivity, high level of students implication

27.11.2015

17

INTERVIEW

• Asking questions about individuals’ self-regulated learning

• Interview can be unstructured, semi-structured or structured

• Stimulated recall interview• Participants explain what was happening in a certain situations

(i.e. in the video).

• Coding needed: Inter-judges

• Problems: off-task, students’ accuracy, laborous: transcriptions

needed.

27.11.2015

18

TRACES OF MENTAL EVENTS & PROCESSES

Example: Eye-tracking experiment, online dating

http://www.youtube.com/watch?v=WqD2pXqT0Z0

• Follow the students actions called traces.

• Students behavior (Computer technology based)• Examples: gStudy & nStudy (Winne et al.)

• Objective collection

27.11.2015

19

http://www.youtube.com/watch?v=WqD2pXqT0Z0

LOGFILE TRACES

• Time stamped records of learners activity in computer based

learning environment.• Provide data ”on the fly” of learners actions.

• Possible to capture in details each action lerner produces (Perry

& Winne, 2006).• Questionnaires and self-reports do not reliably answer what the

students actually do (Hadwin et al., 2002).

• Depending on the age group, often children cannot destinquish

they actions from actual behavior (Paris & Paris 1990).

WHEN THERE IS A NEED TO USE ON-LINE

METHODS

• Usability problems

• Learner – hypermedia interaction, such as navigation

• Modeling learners actions

• Investigation of cognitive processes during the learning

• To better capture intra-individual differences on learners

actions

(Rouet & Passerault, 1999; Winne 2001; Hadwin et al., 2007)

DURATION, FREQUENCY AND SEQUENCY OF EVENTS

• Duration = how long time something is happening• i.e. duration of reading (eg. Time spent in reading)

• duration of model events (eg. Time spent in processing or

manipulating the information).

• Frequency = how often a certain event occurs?• i.e. amount of checking the instructions.

• Sequency = how the events are sequenced.• i.e. sequency of the specific events occuring.

• Data parsing to identify the meaningful events from the

extagenous events that occur in the data (Nesbit et al., 2008).

XWSAJNSHBHCLSOD

XWSAJNSHBHCLSOD -> ABC

• gStudy is designed in order to supports students to became and practise self-regulated learning skills (Winne et al., 2005).

• gStudy provides a number of cognitive tools, for example highlighting with different labels, creating notes that prompts to elaborate or summarize or making searches from the contents of the current kit (Kumar, Groeneboer, Chu,

Jamieson-Noel. & Xin, 2006; Winne & al., 2006).

• Students cognitive tool use in gStudy is considered to reflect students use of study tactics (Hadwin

et al., 2008)

GSTUDY

LEARNING PATTERNS IN CHALLENGING LEARNING

SITUATIONS

•In the challenging learning situations, 88 learning patterns emerged between the low achieving students.•These patterns varied from lenght of three to eight. •Each pattern was used eight to twelve times between challenging gStudy sessions.

2

1 1

2

1

2 2 2 2 2

1

1 2 3 4 5 6 7 8

Label: Interesting information

Label: Important information

1 11

3 3

1 2 3 4

Label: Interestimg detail

Make note in C-map

nStudyWinne & Hadwin, 2009

TRACES PROBLEMS

• Problems: students can use actions for different purposes

(researcher inferences), Massive data: analysis problematic

26

27.11.2015

SELF-REPORT

• A self-report inventory is a type of psychological test in which

a person fills out a survey or questionnaire with or without the

help of an investigator

• Likert-scales (5 or 7)

• Quantitative data • Open questions could be qualitative

• Large sample size

• Easy interpretation

• High reliability if well-constructed

• MSLQ, LASSI, OSLQ, ILS…

27.11.2015

27

SELF-REPORT PROBLEMS

• Validity

• Honesty & accuracy (Introspective ability)

• Decontextualized• Too general if not tailored

• In sum: Like other methods have problems

27.11.2015

28

MIXED METHODS TO STUDY SRL

• Triangulation• Validity

• Different information: bigger picture

• Example of combinations:• Self-report with other

• TAP + stimulated recall

• Traces + stimulated recall

27.11.2015

29

RESEARCH EXAMPLE

Mixed methods approach: video data analysis of group interaction and

video-stimulated recall interview.

Näykki, P. Järvelä, S. Kirschner, P., & Järvenoja, H. (2014). Socio-emotional conflict in collaborative

learning – A process-oriented case study in a higher education context. International Journal of

Educational Research, 68, 1-14.

How did the students interpret the conflict?

Emma and Anna reacted the most strongly to the conflict

EMMA: ‘It was a huge shock to me; I was so surprised that

strange people can talk to each other like that. So I shut

down, and I thought, “Oh my god, can I say what I think

at all?”

ANNA: ‘I think we were overruled, and I didn’t enjoy the

group work after the conflict; it was just to get the

course done. At first, I tried to negotiate the task with

Erik, but in the end, I took a yes-man role and tried not

to care and to agree on every solution’.

Erik and Maria explained the conflict more neutrally

ERIK: ‘The personal chemistry didn’t work between two of the

girls in our group, Tiina and Maria. Actually, there were three

central characters, me, Maria, and Tiina, and then the two

that were a bit quieter, Anna and Emma. Very clearly, Maria

was bothered of Tiina’s occupation; I think she was jealous of

her and was making fun of her’. Maria and Tiina had this fight, and

after that, everything was kind of hushed up and—I don’t know—I

sometimes felt that I was some kind of a leader; others wanted me to

comment on everything’. My goal was to learn as much as possible because

this whole area is so interesting and very valuable to me’. But after the

conflict, it was more like, let’s just try to do the task, and we just tried to get

group work done as soon as possible’.

KEY POINTS OF MEASUREMENTS IN GENERAL

• Reliability:• Overall consistency of a measure: Does it measure what we are

aiming for?

• Inter-judges reliability.

• Validity:• Are these results meaningful: How to interpret the findings?

• Internal validity: extent to which a causal conclusion based on a

study is warranted. Is my study valid? Did I interpret the results

correctly?

• External validity: extent to which the results of a study can be

generalized to other situations and to other people. Does this

happen in the real world?

33


1. Key points of measurements in general

2. SRL: components researchers and practioners seek to

measure





34

PRIMARY VS. SECONDARY RESEARCH

• Primary research consists of the collection of original primary

data by the researcher. It is often undertaken after the

researcher has gained some insight into the issue by reviewing

secondary research or by analyzing previously collected primary

data.

• Secondary research (also known as desk research) involves the

summary, collation and/or synthesis of existing research rather

than primary research, where data is collected from, for

example, research subjects or experiments.

35

TYPES OF SECONDARY RESEARCH

• Different types• Systematic Review (Meta-Analysis)

• Best-Evidence Synthesis

• Narrative Review

• Different Objectives• Integrative Research Review

• Theoretical Review

• Methodological Review

• Thematic Review

• State-Of-The-Art Review

• Historical Review

• Comparison of two perspectives review

36

META-ANALYSIS

• A meta-analysis refers to:• Methods focused on contrasting and combining results from

different studies…

• …in the hope of identifying patterns among study results…

• …giving statistics to evaluate the effect of the relationships.

• It uses the different studies’ effect sizes comparing among

them taking into account other variables (e.g. sample size) to

weight the importance of a particular study for the general

research aim.

37

CRITIQUES TO META-ANALYSIS

• A meta-analysis of several small studies does not predict the

results of a single large study (medicine).

• Golden standard Randomized Controlled Trial vs. meta-

analysis –RCT is just one study.

• Publication bias: the file drawer problem.

• Different study goals, methods, etc.: Mixing apples and

oranges

• Inclusion criteria is CRUCIAL: Garbage in, garbage out

• In sum: there are bad meta-analysis as much as there are

excellent narrative reviews. MA per se does not guarantee

quality.

38

META-ANALYSIS AND SRL

SRL relatively a new field

• Four MAs• Hattie, Biggs, & Purdie, (1996).

• Dignath, & Büttner, (2008).

• Sitzmann, & Ely, (2011).

• Dignath, Büttner, & Langfeldt, (2008).

39

DIGNAGTH’S MA FEATURES

• Goal: reviewed 48 studies derived from 30 articles on the

effectiveness of self-regulatory training with primary school

students.

• Detailed eligibility criteria.

• SRL training programmes have a positive effect on learning,

strategy use & motivation, even for primary school children

• Highest benefits can be gained in mathematics, motivational

outcomes, and cognitive & metacognitive strategies (in line

with Hattie et al.,1996).

• Primary students benefit even more than older (in line with

Hattie et al.,1996).

40


• Surprisingly, effect sizes were significantly higher for

interventions that did not train students by means of group

work but…

• Several meta-analyses investigated this method and revealed

positive effects:• In the studies included in this meta-analysis, we found only very

little information about the implementation of group work in the

learning setting.

• Hence, a possible reason for the negative effect of group work on

training effects at primary school level might be that students

were not used to work in groups and did not receive enough

instruction about collaboration.

• Too many self-report data (2006)

41


• Summarizing the most effective characteristics of interventions

yields that a training programme should be based on social-

cognitive theories, should train cognitive (especially

elaboration and problem solving strategies), metacognitive

(especially planning strategies), and motivational strategies

(especially feedback), and provide knowledge about strategy

use and about its benefit.

• Future research: learning environment & how teachers can be

trained.

• PREP EXAMPLE (Preparing Teacher Education Students for 21st Practice Learning)

42

43


1. Take your list of how to study (a certaintopic of) SRL

2. Now think and discuss in pairs: whatdo you think about measuring SRL now?

3. Prepare yourself to present 3 keypoints to the class.

FORECASTING THE FUTURE

11/27/2015

44

FORECASTING THE FUTURE

11/27/2015

45

• Too little has been achieved yet in measuring SRL as an event.• Challenges in this arena are significant!

• Protocols are needed for collecting longitudinal measurements

that span multiple brief episodes as well as expended periods

(i.e. grade levels).

• Methods are needed that characterize temporally unfolding

patterns of engagement with tasks• in terms of the tactics and strategies that constitute SRL.

• In terms of comparing patterns over time

• New ideas from psychophysiological measurements• Triangulation needed!

CONCLUSIONS

Different ways to measure• Choose the one that fits your research goals.

• But don’t avoid because lack of knowledge.

• Try to triangulate data• Don’tbe afraid to be innovative but remember:

• Reliability and validity

27.11.2015

46

CONCLUSIONS

1. Measuring intervenes in a student’s environment.• We design measurement with an intention to cause the student to

recall or to generate a particular kind of response.

WHEN WE MEASURE, WE CHANGE THE ENVIRONMENT – ALLWAYS.

27.11.2015

47

REFERENCES

• Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention.

Applied Psychology-an International Review-psychologie Appliquee-revue Internationale, 54(2), 199-231.

• Butler, D. L. (2011). Investigating self-regulated learning using in-depth case studies. In B. J. Zimmerman & D. H.

Schunk (Eds.), Handbook of self-regulation of learning and performance(pp. 346-360). New York: Routledge.

• Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL

model. Educational Psychologist, 46(1), 6 -25. doi: 10.1080/00461520.2011.538645

• Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215-251. doi:

10.1037/0033-295X.87.3.215

• Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students' strategic

processing. British Journal of Educational Psychology, 77(2), 351-378. doi: 10.1348/000709906x106147

• Schmitz, B., Klug, J., & Schmidt, M. (2011). Assessing self-regulated learning using diary measures with university

students. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance(pp.

251-266). New York: Routledge.

• Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated engagement in learning. In D. Hacker, J. Dunlosky &

A. Graesser (Eds.), Metacognition in educational theory and practice(pp. 277-304). Hillsdale, NJ: Erlbaum.

• Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect. In D. J.

Hacker, J. Dunlosky & A. C. Graesser (Eds.), Handbook of Metacognition in Education(pp. 299-315). New York:

Routledge.

• Zimmerman, B. J., & Schunk, D. H. (2011). Handbook of self-regulation of learning and performance. New York:

Routledge.

27.11.2015

48

REFERENCES

• Cooper, H. (2010). Research synthesis and meta-analysis. Thousand Oaks, California: SAGE.

• Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis

on intervention studies at primary and secondary school level. Metacognition and Learning, 3, 231-264. doi:

10.1007/s11409-008-9029-x

• Dignath, C., Büttner, G., & Langfeldt, H. (2008). How can primary school students learn self-regulated learning

strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review,

3(2), 101-129. doi: 10.1016/j.edurev.2008.02.003

• Dochy, F. (2006). A guide for writing scholarly articles or reviews for the Educational Research Review. Educational

Research Review. http://www.journals.elsevier.com/educational-research-review/

• Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis.

Review of Educational Research, 66(2), 99-136. doi: 10.3102/00346543066002099

• Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational

attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421-442. doi: 10.1037/a0022777

27.11.2015

49

THANK YOU!

QUESTIONS AND COMMENTS?

[email protected]

50