Coordinated Reasoning and the “Written Response Effect”

Michael W. Creane

University of California, Berkeley

November 5th, 2019

Today’s Agenda

• Study Origins

• Overview of Coordination

• Exploring Coordination as a measurable construct

• Exploring Coordinated Reasoning Complexity across verbal and written interviews.

• The “Written Response Effect”

• Conclusions

A Troubling Contradiction

• Apparent gulf between quality of conversations with students and their written responses on quizzes/exams

• Potential real world consequences:• Systematic underestimation of student ability through written assessment?

• Written open responses may present a latent bias towards those who produce more exposition

• Even with no bias towards increased exposition, does quality of responses change consistently across individuals?

What is Coordination?

• A process of “weighing and balancing different considerations and goals” (Turiel & Gingo, 2017, pg. 210)• When reasoning and judgments within or between domains are

contradictory, it does not prevent an individual from developing a singular judgment of alright or not alright. • Consider the Heinz Dilemma• Where does that first response (alright/not alright) come from?

• This ‘singular’ (alright/not alright) judgment, on it’s face, can be seen as a distillation or a reduction of the thought processes (coordination) that led to it.

Coordination as a process

Participant makes judgment of Alright or Not Alright (C)

Coordinated Reasoningby participant (B)Scenario

given and Question asked by researcher (A)

Participant evaluates judgment(E)Researcher asks Why? (D)

So while coordinated reasoning (B) is a process that involves weighing and balancing various components of a situation, it leads to a ‘relatively simple’ judgment of yes/no, alright/not alright(C) that should reflect this complexity when evaluation is explained further (E)

In short, the evaluation given should be a strong approximation of the participant’s coordinated reasoning.

Construct Development

• Guttman Mapping Sentence : • Take a theorist’s statement and break it down into measurable components.

“The process of coordination involves weighing and balancing the different considerations when drawing conclusions within the parameters of social situations” Turiel 2008.

• Category (Domain)

• Quantity (Categories represented x points of view)

• Presence of Contradictions (Probe for firmness of conviction)

• Mode of Resolution (Resolved or situation-dependent)

Coordinated Reasoning Construct Map

Respondent's Level + Response Criteria for Level

Complex Coordinated (3) Individuals who can coordinate a complex set of considerations leading to an unambiguous

resolution↑

a) Addressed, incorporated and evaluates multiple points of information, even if information is apparently conflicting b) within multiple domains AND/OR a

single domain from multiple unique viewpoints, c) while addressing the domain(s) used in the question, d) and ultimately coming to a clear and

defensible resolution regarding their justification for their judgment.2.5 contains elements of level 2 and 3 responses ↑

Complex Uncoordinated (2) Individuals who can consider, but not coordinate a complex set of considerations, leading to an

ambiguous resolution↑

a) Addressed, incorporated and evaluates multiple points of information, even if information is apparently conflicting b) within multiple domains AND/OR a

single domain from multiple unique viewpoints, c) response may or may not be domain concordant, d) and ultimately does not come to a clear and defensible

resolution regarding their justification for their judgment.1.5 contains elements of level 1 and 2 responses ↑

Simple Subordinated (1) Individuals who are limited to evaluating a single point of consideration in coming to an

unambiguous resolution↑

a) May address, incorporated and evaluates one or multiple points of information, b) within a single domain from a single viewpoint or operates as

though all individuals hold a single view, c) response may or may not be domain concordant, d) and ultimately comes to a clear resolution regarding their

justification for their judgment.

Unjustifiable (0)↑

Unable to justify response (Because, I dunno, etc)

Un-codeable (unused) - Unintelligible response, unable to answer or comprehend question

Items Design

• Stories:

SharingChris and Jeff are sitting together in the cafeteria at school for

lunch. Today Jeff forgot his lunch at home. Jeff asks Chris to

share his food. Chris believes that everyone deserves to have

some food to eat for lunch, so he offers to share half of his food

with Jeff.

Q: Is it alright or not alright for Chris to share his lunch with Jeff?

Why or why not?

The teacher comes over and tells Chris not to share his food with

Jeff because there is a school rule against sharing food, so Chris

stops sharing.

Q: Is it alright or not alright for the teacher to tell Chris not to

share his lunch?

Why or why not?

After the teacher leaves, Chris still shares his lunch with Jeff and

the teacher does not see.

Q: Is it alright or not alright that Chris shared his lunch when the

teacher told him not to?

Why or why not?

After lunch, the teacher asked Chris if he had shared his lunch

after they talked. What should Chris say?

Why?

Story Student Motive Teacher Motive

Sharing Food Moral Conventional

Believes everyone

deserves to eat some

food for lunch

School rule against

sharing food

Include Moral Conventional

Believes everyone should

be included in a group

School rule re:

max group size

Harm Moral Conventional

Believes it is wrong to

hurt on purpose

Team rule that

players must hurt

on purpose

Cheating Moral Conventional

Believes everyone should

be included in a group

School rule against

students choosing

own groups

Item 2: The teacher comes over and tells Chris not to share his food with Jeff because there is a school rule against sharing food, so Chris stops sharing.Q: Is it alright or not alright for the teacher to tell Chris not to share his lunch?Why or why not?

• Complex Coordinated: I think it’s not alright to do that. (Why?) Because it’s not the school’s lunch, it’s my lunch. Maybe the school can give him one. But the school gives you the lunch and you get to decide what to do with the lunch. It was not correct for the teacher to do that. (Does it matter why the teacher is telling Chris not to share his lunch with Jeff?) Only if there were cases before if someone put something in the food and gave it to someone (Would it be alright for the teacher to enforce the rule in that case?) No, because if a big problem at the school, then, but they don’t have to enforce the rule, they should just remind them of the situation or problem.

• Complex Uncoordinated: It depends on the teacher. If the teacher feels that that’s OK, or if they’d rather go by the rules of the school, that’s OK too. (Does it matter that Chris wants to share his food for the reason he does?). No, the teacher gets to decide.

• Simple Subordinated: I think it may be alright (Why?) Because there might be germs, and someone might get them. (And what might be the outcome if someone gets germs?) They could get fever, or sick, or jungle rot.

Hypotheses for current study

• Student responses to written questions would be more attenuated and more reliant on local contextual cues from the story than their verbal counterparts. Thus, there may be unique step difficulties associated with written responses compared to verbal.

• A Partial Credit Logistic Model would better fit the data in comparison to a Rating Scale Logistic Model

• There is not expected to be any evidence of gender based DIF

• There is expected to be a moderate to strong correlation between participant age and ability level.

Sample Size and Demographics

Interview (16

Items) 30

Written

Questionnaire

(23 Items) 22

Written

Questionnaire

(7 Items) 142

All Items 12

Grade Frequency

2 2

3 2

4 1

5 2

6 32

7 56

8 55

9 3

10 10

11 4

12 2

13 1

Total 170

Average Grade 7.388235

Local Tutoring Center, Friend’s and Family’s Children

Local Tutoring Center

Oakland DBME Data Set

Local Tutoring Center (Linking Cases)

Measures of Reliability & Validity

• Interrater reliability • Whole score Cohen’s Kappa= .84

• Within half score Cohen’s Kappa=.92

• Split Halves Reliability= .88

• Person Separation Reliability (5pl model): .94

• Internal Validity: Cronbach’s Alpha: .97• Strong internal consistency

• Exit interviews (12)• 80% of linked individuals perceived the written form of the items to be more difficult

than the interview form.• 83% reported having further considerations on the questions that they did not write

down

SEM and Test InformationTest Information CurveStandard Error of Measurement Graph

Response Score Distributions Paper vs. Interview

0 1 2 3 4 5

# Paper 21 185 46 30 30 27

# Interview 0 139 101 48 77 130

3.9E-122

Score

χ2=115.94, df=5, p<.001

Looking at 834 total coded responses, the distributions are clearly different between the two modes of assessment

Rasch vs. Rating Scale vs. Partial Credit Model

• A Rasch Model is a logistic (probabilistic) model of dichotomous outcomes (only one step between levels)

• A Rating Scale Model involves 3 or more levels and is akin to stacking multiple Rasch models • Same increase in ability needed to go from Simple Subordinated to Complex

Uncoordinated and from Complex Uncoordinated to Complex Coordinated• Think of a ladder with equal spaces between rungs

• A Partial Credit Model is a more flexible ladder• Allows for different size steps in ability between levels

• Think of a ladder with different size gaps between rungs

Partial Credit Model was a better fit than Rating Scale: X^2=246.7, df=126, p<.0001

Implications of using PCM vs. Rating Scale• In 15/16 items, there is a difficulty associated with

going from Unjustifiable (alright or not alright, but I dunno why) to simple subordinated• This was apparent in the PAPER version only

So there is a unique difficulty associated with written responses that is not present in verbal interviews

Comparing Item Difficulties

Item Interview Difficulty Interview Error Paper Difficulty Difference

Study1 0.207 0.179 1.0665 0.8595

Study2 0.429 0.177 1 0.571

Study3 0.351 0.18 1.07475 0.72375

Study4 0.527 0.197 0.74725 0.22025

Share1 0.581 0.188 1.02325 0.44225

Share2 -0.011 0.193 0.87375 0.88475

Share3 0.156 0.177 0.937 0.781

Share4 0.184 0.182 0.48075 0.29675

Harm1 -0.302 0.175 0.729 1.031

Harm2 -0.16 0.169 1.056 1.216

Harm3 0.021 0.175 1.06125 1.04025

Harm4 -0.432 0.179 0.98475 1.41675

Cheat1 0.43 0.202 0.7835 0.3535

Cheat2 0.033 0.176 0.8195 0.7865

Cheat3 0.059 0.185 0.79425 0.73525

Cheat4 0.399 0.18 0.7695 0.3705

ALL 16 ITEMS were considerablymore difficult for participants on paper than their interview counterparts. All paper difficulties fell OUTSIDE 99% CI of interview difficulties

Item Interview Difficulty Interview Error Paper Difficulty Difference

Study1 0.207 0.179 1.0665 0.8595

Study2 0.429 0.177 1 0.571

Study3 0.351 0.18 1.07475 0.72375

Study4 0.527 0.197 0.74725 0.22025

Share1 0.581 0.188 1.02325 0.44225

Share2 -0.011 0.193 0.87375 0.88475

Share3 0.156 0.177 0.937 0.781

Share4 0.184 0.182 0.48075 0.29675

Harm1 -0.302 0.175 0.729 1.031

Harm2 -0.16 0.169 1.056 1.216

Harm3 0.021 0.175 1.06125 1.04025

Harm4 -0.432 0.179 0.98475 1.41675

Cheat1 0.43 0.202 0.7835 0.3535

Cheat2 0.033 0.176 0.8195 0.7865

Cheat3 0.059 0.185 0.79425 0.73525

Cheat4 0.399 0.18 0.7695 0.3705

Scores were re-estimated based on the assumption that “I dunno” is a possible answer in an interview (despite not occurring due to probing questions)

A Sample of the Linked CasesLevel

difference

Per

Scenario

1 3 1 3 3 2.5 2 2 2 -0.5 1 -1 1.5

2.5 2 1 1.5 1 1 1 1 -1.5 -1 0 -0.5 -3

1.5 1.5 2.5 1.5 1.5 1.5 1.5 1 0 0 -1 -0.5 -1.5

1 1.5 2 2.5 1 2 2 2 0 0.5 0 -0.5 0

1 1.5 2.5 3 1 2 2 1.5 0 0.5 -0.5 -1.5 -1.5

1.5 1 1 1 1.5 1 1 1 0 0 0 0 0

1 1.5 1.5 2.5 3 3 2.5 1.5 2 1.5 1 -1 3.5

1 1 1 1.5 1 1 1 1 0 0 0 -0.5 -0.5

2.5 1.5 1.5 3 1.5 1.5 1 1.5 -1 0 -0.5 -1.5 -3

1 1.5 2 1.5 2.5 2 2.5 2 1.5 0.5 0.5 0.5 3

1 1 1 1 1 1 1 1 0 0 0 0 0

1.5 1 1 1 1 1 1 1 -0.5 0 0 0 -0.5

1.5 1 1.5 1.5 3 2 3 3 1.5 1 1.5 1.5 5.5

1 1 1 2 1 1 1 1.5 0 0 0 -0.5 -0.5

2 1.5 2.5 3 3 2.5 3 3 1 1 0.5 0 2.5

2 2 2 3 3 3 3 3 1 1 1 0 3

1 1 1 1 2.5 2 1 1.5 1.5 1 0 0.5 3

1 1.5 1 1 1 1.5 1.5 1 0 0 0.5 0 0.5

1 1 1 2.5 1.5 2.5 2.5 3 0.5 1.5 1.5 0.5 4

1.5 1 1 1 1 1 1.5 1 -0.5 0 0.5 0 0

2 1.5 2.5 1.5 2 3 3 2.5 0 1.5 0.5 1 3

1 1.5 1 3 2.5 2.5 3 2.5 1.5 1 2 -0.5 4

1 1 2 1 1 1.5 1.5 1 0 0.5 -0.5 0 0

1 1 1 1 1.5 1 1 1.5 0.5 0 0 0.5 1

24

Paper Response Levels Interview Response Levels Interview Response - Paper Response• Scores from 6 individuals

• Tracked scores on like items from interview to paper:• 39/96 cells-

Interview score Higher

• 37/96 cells-Same score

• 20/96 cells-Paper score Higher

• Multinomial Dist: (P<.0001)

Item DifficultiesNot only did every item become more difficult when responses were written down, but it was not a linear transformation. In some cases previously difficult questions became easy (Study 4) and in other previously easy questions became difficult (Harm 1,2)

Spearman’s Rho= -.01 (p>.95)

Study 2 Interview Rank Order

Study 2 Paper Rank Order

Examining DIF: Do Males and Females reason differently?

male female St. Dev t-score

Step 1 0.04189 0.05216 0.77 -0.1739

Step 2 0.03379 -0.00135 0.77 0.595007

Step 3 0.03119 -0.00213 0.77 0.56419

Step 4 0.02776 -0.03304 0.77 1.029494

Step 5 0.05351 0.03055 0.77 0.388769

Study 2 DIF Results (all results p>.05)Study 1 DIF Results (F=1, M=2)

Conclusions

• Student response complexity is not solely a function of their knowledge, but also of environmental support for their expression• Written responses may have a tendency to underestimate an

individual’s true reasoning complexity ability• Individuals ‘sacrificed’ approximately 1.4 steps per item to ‘the written

response effect’. • Written responses seem to ‘increase” item difficulty by ~.733 logits on average

• Interviews with probing questions may have a tendency to more accurately estimate an individual’s true reasoning complexity ability• If I’m testing coordinated reasoning complexity, perhaps it’s more accurate

to look at ‘written coordinated reasoning complexity’ vs. ‘conversational coordinated reasoning complexity’.

Additional Takeaways:

1) There appears to be no difference in coordinated reasoning complexity abilities between males and females

2) There is a moderate (.4) positive correlation between age and ability level1) Yes it improves with age, BUT there is also considerable variation within ages

3) Every instance of measurement entails at least two abilities:1) The ability that the researcher intends to measure

2) The ability of the participant to engage with the method of assessment