Observational Methods January 20, 2010. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri,

Observational Methods

January 20, 2010

Today’s Class

• Survey Results• Probing Question from Wed, Jan. 20• Observational Methods• Probing Question for Fri, Jan. 22• Survey

Survey Results• Only 3 responses

– I will be switching to a better technology for the next survey

• Mostly positive

• I’ve read through your comments, and will try to take them into account

• Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better

• The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

Today’s Class


ENTITATIVE

HOLISTIC

ESSENTIALISTEXISTENTIALIST

Today’s Class


Observational methods: top level

• You don’t manipulate the research setting in any way

• You just look at what is happening, or what has happened, and you note it

• Later, you analyze the data


• This class we will focus on quantitative observational methods– Where observations are noted in terms of a

defined set of categories

– Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data


• This week we will focus on quantitative observational methods

• Next week we will focus on qualitative methods– Where observations are noted in a more free-form

way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

What is the research setting?

• Where the learning (or other phenomenon you want to study) is taking place

What is the research setting?

• Usually a real learning setting– Classroom– University computer lab– Dorm room– Workplace– Any others?

• Occasionally a lab setting– Needs to be justification for why the lab setting is valid

Types of observation

• Enhanced video coding• Video coding• Field observation• Screen replay coding• Text replay coding

• (and more)

High-level notes

• High-level methodological similarity

• Can be used for many of the same research goals

• Advantages and disadvantages to each

Types of observation(that we’ll focus on today)

• Video coding• Field observation• Text replay coding

Types of observationthat we’ll focus on today


Let’s watch a video

• http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related– From 0:00 to 1:15

http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related

http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related

What student behaviors did you observe?

• Please explain both what the behavior is, and how you could recognize it from the video

• Does anyone disagree with how a behavior is defined? Why?

List of behaviors

• Let’s briefly write down a list of behaviors seen

OK, let’s watch it again

• Please write every behavior you see that did not fit into the original coding scheme

What additional student behaviors did you observe?

• Please explain both what the behavior is, and how you could recognize it from the video

• Does anyone disagree with how a behavior is defined? Why?

What is the total list of behaviors?

Now let’s make a coding scheme

• A list of “interesting” behaviors, how they are defined, and a code letter

• Example:– Hint abuse (a type of gaming the system)– Clicking through hints at high speed until reaching

the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step


• Should not be infinitely long – eliminate– “Uninteresting” behaviors – be guided by • your intuition• prior theory and empirical results

– “Extremely rare” behaviors – under approximately 1 in 200

– Behaviors occuring between 1-5% of the time are often very interesting behaviors!

– Effects on learning are often disproportionate• Gaming the system occurs 3-5% of the time, and

correlates to learning at 0.3-0.4

Which behaviors in your list

• Are “uninteresting”?• Are “extremely rare”?


• And combine related behaviors– For example, different types of gaming the system

(e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together• common

– antecedents (poorly known skill)– characteristics (very fast responses)– results (poor learning gains)

• differences are no longer considered theoretically interesting (some debate on this still)• tend to co-occur

Which behaviors

• Should be combined?

Our Coding Scheme

OK, let’s watch it for the 3rd time

• Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them)

• Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds– If 2 behaviors observed, pick the dominant one– If you are not sure, put down a “?”– Do NOT talk to your partner

OK, now work with your partner

• Find out what % of time you agreed

• Find out where and how you disagreed

• There are better ways to assess inter-rater reliability, we’ll discuss these next class

Each group’s % error

What categories did you disagree on?

OK, now let’s watch

• Same video, but times 1:06 to 2:10

• Observe the same student

• Follow the same procedure as before

Were the behaviors the same in this segment?

• It is important to:– Develop your coding scheme based on sufficient

initial qualitative observation (and/or theory and prior work)

– Note behaviors that do not fit in your coding scheme

OK, now work with your partner

• Find out what % of time you agreed

• Find out where and how you disagreed

Across sessions

• Did your agreement go up or go down?

• By how much?

A good strategy

• Repeatedly code the same data separately and compare

• Until your inter-rater reliability is “high enough” (more next time)

• And then code different data separately

Incidentally

• There is nothing “magic” about the video coding method we used

• It was largely selected for tractability in class

Other things we could have done

• Watch the same time segment multiple times

• Advantages

• Disadvantages


• Watch the same time segment multiple times

• Advantages: Greater certainty in coding

• Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting


• Code when the behavior changes, not on a pre-selected time interval

• Advantages

• Disadvantages


• Code when the behavior changes, not on a pre-selected time interval

• Advantages: Get better assessment of the duration of each behavior

• Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)


• Code the first action in a time window, not the predominant one

• Advantages

• Disadvantages


• Code the first action in a time window, not the predominant one

• Advantages: Less ambiguity and decision-making

• Disadvantages: May be less representative; very brief behaviors may be missed


• What else?

A limitation of one camera

What is she doing?

Other ways to use video

• Multiple cameras

• Addresses issue of the student off-screen• Synchronization becomes a challenge• Time to code increases with each camera you

add


• Individual student video

• Often from web-cams on top of machines

Example

Example

• https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H

https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H

https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H

Note that

• The specific D’Mello study we read was a lab study, but the same method has been used in classrooms (without the emote-aloud)(cf. Arroyo et al, 2009)

• I chose the D’Mello paper in part because of the very clear description and comparisons of alternatives


• Other thoughts?



Methods

• Similar, but live and in the field rather than via video

• Different challenges• Some advantages

Sampling

• One challenge that emerges in field observations is sampling

Sampling

• Usually not desirable to follow only one student throughout the entire observation period

• Usually not tractable to have one observer per student

• Usually not possible to fully code multiple students at the same time

Field Sampling Strategies

• Switch between students in pre-determined order according to pre-determined schedule

• Watch or public timekeeper(image from Rodrigo et al, 2008)

Field Sampling Strategies

• Focus on a specific behavior and code it whenever it occurs, including which student engaged in the behavior

Observer effects

• Another challenge in live observation is observer effects

• A person hovering over your shoulder will impact your behavior

Observer effects:What you can do

• Obfuscation strategies• Desensitization strategies

Obfuscation strategies

• Don’t appear to be observing the student you are actually observing– Observation from a distance– Side glances– Peripheral vision observation



• Easiest to do• Depends on classroom layout• Not possible to assess some things



• Not always effective



• Takes training, but is the most powerful

Desensitization strategies

• Spend a class period walking around the classroom and noting random symbols in your clipboard

• Whenever a student asks you what you’re doing, show them the clipboard and explain that you’re just there to study the software for your professor and won’t be giving any information to the teacher

• After a certain point, they will decide you’re not important and begin to behave in normal (frighteningly anti-social) ways

Advantages of field observations

Faster to conduct

• About 3x faster(Baker, Corbett, & Wagner, 2006)

Faster to conduct

• Go out to class Versus• Go out to class, then analyze in lab

• Also– The ability to stop and re-watch is frequently used

in video coding

No camera & microphone limitations

• You can stand close enough to really see & hear what’s going on

• If something is obscured you can change your position

What is she doing?

What is his facial expression?

With video…

• You have to get the data collection right, even before you’ve had time to do analysis

With field observations…

• You can quickly see whether you’re missing key information, and adjust your method (Note: adjusting your method in the *middle* of a study is a bad idea. Better to adjust your method, and restart, and just lose the first day’s data)

Major limitation

• Impossible to change coding categories to follow up on a question that emerges when you analyze the data

• Impossible for data to be re-used by alternate research group with different coding scheme

Other thoughts on field observation?



Text replays

• Pretty-prints of student interaction behavior from the logs

Examples

Sampling

• You can set up any sampling schema you want, if you have enough log data

• 5 action sequences• 20 second sequences• Every behavior on a specific skill, but other

skills omitted

Sampling

• Equal number of observations per lesson• Equal number of observations per student

• Observations that machine learning software needs help to categorize (“biased sampling”)

Major Advantages

• Both video and field observations hold some risk of observer effects

• Text replays are based on logs that were collected completely unobtrusively

Major Advantages

• Blazing fast to conduct– 8 to 40 seconds per observation

Notes

• Decent inter-rater reliability is possible(Baker, Corbett, & Wagner, 2006)(Baker, Mitrovic, & Mathews, under review)

• Agree with other measures of constructs(Baker, Corbett, & Wagner, 2006)

• Can be used to train machine-learned detectors(Baker & de Carvalho, 2008) (Baker, Mitrovic, & Mathews, under review)

Major Limitations

• Limited range of constructs you can code

• Gaming the System – yes• Collaboration in online chat – yes

(Prata et al, 2008)• Frustration, Boredom – sometimes• Off-Task Behavior outside of software – no• Collaborative Behavior outside of software – no

Major Limitations

• Lower precision (because lower bandwidth of observation)

Other thoughts on text replays?

Other Quantitative Observation Methods

When?

• When might you want to use some of the other methods listed here?

Any thoughts?

• About some of these related methods?

Not as common

• I only know of a 2 papers using screen replay in education– That group (Helen Pain and her students) now

uses transcript coding, field coding, and video coding, but don’t use screen replays anymore

• I don’t know of any group coding using Super-Fidelity data, although D’Mello and Arroyo both use that kind of data in machine learning

High-level thoughts on quantitative observation methods

Generalizability

• Important to think about where you data came from, and what conclusions you can draw

What can you conclude?

• Data on one class day• Data on one lesson• Data on full year

• Data from 1 student• Data from 10 students• Data from 1 class• Data from 1 school• Data from 3 schools with different profiles• Data from 100 schools

Feasibility of collection for each method

• Data on one class day• Data on one lesson• Data on full year

• Data from 1 student• Data from 10 students• Data from 1 class• Data from 1 school• Data from 3 schools with different profiles• Data from 100 schools

Things you can code

• Teacher behaviors• Student individual behaviors• Student collaborative behaviors• Student affect• What else?

What method(s) work for each of these?

• Teacher behaviors• Student individual behaviors• Student collaborative behaviors• Student affect• What else?

Coding teacher behavior

• http://www.youtube.com/watch?v=gHzTUYAOkPM

http://www.youtube.com/watch?v=gHzTUYAOkPM

http://www.youtube.com/watch?v=gHzTUYAOkPM

Standard coding schemes

• It is worth noting that many research areas have standard coding schemes that have been developed and refined across many research projects, and are used by many researchers– Lauren Resnick and Sarah Michaels’s accountable

talk (strategies used by students and teachers)– Baker et al: engagement-related behaviors in

classrooms using educational software– D’Mello et al: affect in educational settings

Advantages to using these schemes

• You don’t need to defend your choice of scheme in your write-up

• Easier to compare your results to previous results and identify what’s new/different

Disadvantages to using these schemes

• You insulate yourself to some degree from obtaining surprising/new/exciting results

Which types of research programs would use these methods?

ENTITATIVE

HOLISTIC


ENTITATIVE

HOLISTIC


Today’s Class


Probing Question

• For Friday, you will read• D'Mello, S., Taylor, R.S., Graesser, A. (2007) Monitoring Affective

Trajectories during Complex Learning. Proceedings of the 29th Annual Meeting of the Cognitive Science Society, 203-208

• Which used data from a lab study

• If you wanted to study affective transitions in real classrooms, which of the methods we discussed today would be best? Why?

Today’s Class


Please complete this survey, and put it in this envelope

The End

Documents

Observational Methods January 20, 2010. Today’s Class Survey Results Probing Question from Wed, Jan. 20 Observational Methods Probing Question for Fri,