Cognitive Approaches to the Winograd Schema Challenge:

Challenges and Complications

Jesse MuSupervised by Joshua Hartshorne and

Tobias Gerstenberg

A Mystery of Pronoun Resolution

Q: Who does “he” refer to?Pronoun resolution techniques fail!(Hobbs Algorithm, Centering Theory)

• A lost to B at tug-of-war although he was weak.• A didn’t beat B at tug-of-war because he was strong.etc…

Resolution encoded in meaning, not syntax

• A beat B at tug-of-war because he was strong.• A beat B at tug-of-war because he was weak.

Why Winograd Schema?

Empirical Evaluations

1. Troy (Novelty)2. College (Learning)

Empirical Evaluations

1. Troy (Novelty)2. College (Learning)

resolution = 0.913

strengthL = 0.347

strengthR = 0.802

resolution = 0.043

strengthL = 0.872

strengthR = 0.252

1. Christopher beat Nicholas at tug-of-war, because he is strong.2. Brian didn't beat Gabriel at tug-of-war, because he is strong.3. Andrew beat Joseph at tug-of-war, because he is weak.4. Adam didn't beat Jack at tug-of-war, because he is weak.5. Christian lost to Austin at tug-of-war, because he is strong.6. Daniel didn't lose to Seth at tug-of-war, because he is strong.7. Dylan lost to Ethan at tug-of-war, because he is weak.8. Richard didn't lose to Carlos at tug-of-war, because he is weak.

9. Daniel beat William at tug-of-war, although he is strong.10. Isaiah didn't beat Luis at tug-of-war, although he is strong.11. Brandon beat Ryan at tug-of-war, although he is weak.12. Marcus didn't beat Charles at tug-of-war, although he is weak.13. Benjamin lost to Noah at tug-of-war, although he is strong.14. Patrick didn't lose to Bryan at tug-of-war, although he is strong.15. Samuel lost to Robert at tug-of-war, although he is weak.16. Cole didn't lose to Jackson at tug-of-war, although he is weak.

1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.

9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

Nontroy Troy

5. Christian lost to Austin at tug-of-war,because he is strong.

1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.

9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

Nontroy Troy

4. Adam didn’t beat Jack at tug-of-war,because he is weak.

Pronoun Resolution (Because)Non-troy Troy

Strength Judgments (Because)Non-troy Troy

??Pronoun Resolution (Because)

Non-troy Troy

Strength Judgments (Because)Non-troy Troy

??

A Tale of Two Interpretations11. Brandon beat Ryan at tug-of-war,

although he is weak.

He = Brandon?He = Ryan?

Although [he=Brandon] is weak, Ryan is even weaker.

A Tale of Two Interpretations11. Brandon beat Ryan at tug-of-war,

although he is weak.

Although [he=Ryan] isweak, Brandon is even weaker.

Although [he=Brandon] is weak, the God of Weaklings helped him win.

Nontroy Troy

11. Brandon beat Ryan at tug-of-war,although he is weak.

r = 0.958

r = -0.985

1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.

9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

r = 0.958

r = -0.985

Empirical Evaluations

1. Troy (Novelty)2. College (Learning)

Willowbrook College

Pinehurst University

1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

Willowbrook College

Pinehurst University

17. A beat B at tug-of-war,because he is from Willowbrook College

??

1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is Willowbrook College.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is from Willowbrook College.19. A beat B at tug-of-war, because he is from Pinehurst University.20. A didn't beat B at tug-of-war, because he is from Pinehurst University.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.

“College” Questions17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is from Willowbrook College.19. A beat B at tug-of-war, because he is from Pinehurst University.20. A didn't beat B at tug-of-war, because he is from Pinehurst University.21. A lost to B at tug-of-war, because he is from Willowbrook College.22. A didn't lose to B at tug-of-war, because he is from Willowbrook College.23. A lost to B at tug-of-war, because he is from Pinehurst University.24. A didn't lose to B at tug-of-war, because he is from Pinehurst University.25. A beat B at tug-of-war, although he is from Willowbrook College.26. A didn't beat B at tug-of-war, although he is from Willowbrook College.27. A beat B at tug-of-war, although he is from Pinehurst University.28. A didn't beat B at tug-of-war, although he is from Pinehurst University.29. A lost to B at tug-of-war, although he is from Willowbrook College.30. A didn't lose to B at tug-of-war, although he is from Willowbrook College.31. A lost to B at tug-of-war, although he is from Pinehurst University.32. A didn't lose to B at tug-of-war, although he is from Pinehurst University.

1-16: Control17-32: CollegeProcedure

½ Control + ¼ College

Diagnostic Message

½ Control + ¾ College

Order Effects for Control Questions

1. Christopher beat Nicholas at tug-of-war, because he is strong.

Collapse into Control/Pre-diagnostic/Post-diagnostic1. A beat B at tug-of-war, because he is strong/from Willowbrook College.2. A didn't beat B at tug-of-war, because he is strong/from Willowbrook College.3. A beat B at tug-of-war, because he is weak/from Pinehurst University.4. A didn't beat B at tug-of-war, because he is weak/from Pinehurst University.5. A lost to B at tug-of-war, because he is strong/from Willowbrook College.6. A didn't lose to B at tug-of-war, because he is strong/from Willowbrook College.7. A lost to B at tug-of-war, because he is weak/from Pinehurst University.8. A didn't lose to B at tug-of-war, because he is weak/from Pinehurst University.9. A beat B at tug-of-war, although he is strong/from Willowbrook College.10. A didn't beat B at tug-of-war, although he is strong/from Willowbrook College.11. A beat B at tug-of-war, although he is weak/from Pinehurst University.12. A didn't beat B at tug-of-war, although he is weak/from Pinehurst University.13. A lost to B at tug-of-war, although he is strong/from Willowbrook College.14. A didn't lose to B at tug-of-war, although he is strong/from Willowbrook College.15. A lost to B at tug-of-war, although he is weak/from Pinehurst University.16. A didn't lose to B at tug-of-war, although he is weak/from Pinehurst University.

1. A beat B at tug-of-war,because he is strong/from

Willowbrook College.

(Left) (Right)

(hypothetical)

Pronoun resolution

1. A beat B at tug-of-war,because he is strong/from

Willowbrook College.

(Left) (Right)

(hypothetical subject bias)

Pronoun resolution

1. A beat B at tug-of-war,because he is strong/from

Willowbrook College.

(Left) (Right)

(results)

Pronoun resolution

Collapse into Control/Pre-diagnostic/Post-diagnostic1. A beat B at tug-of-war, because he is strong/from Willowbrook College.2. A didn't beat B at tug-of-war, because he is strong/from Willowbrook College.3. A beat B at tug-of-war, because he is weak/from Pinehurst University.4. A didn't beat B at tug-of-war, because he is weak/from Pinehurst University.5. A lost to B at tug-of-war, because he is strong/from Willowbrook College.6. A didn't lose to B at tug-of-war, because he is strong/from Willowbrook College.7. A lost to B at tug-of-war, because he is weak/from Pinehurst University.8. A didn't lose to B at tug-of-war, because he is weak/from Pinehurst University.9. A beat B at tug-of-war, although he is strong/from Willowbrook College.10. A didn't beat B at tug-of-war, although he is strong/from Willowbrook College.11. A beat B at tug-of-war, although he is weak/from Pinehurst University.12. A didn't beat B at tug-of-war, although he is weak/from Pinehurst University.13. A lost to B at tug-of-war, although he is strong/from Willowbrook College.14. A didn't lose to B at tug-of-war, although he is strong/from Willowbrook College.15. A lost to B at tug-of-war, although he is weak/from Pinehurst University.16. A didn't lose to B at tug-of-war, although he is weak/from Pinehurst University.

5. Christian lost to Austin at tug-of-war, because he is strong/from

Willowbrook College.

(Left) (Right)

(hypothetical)

Pronoun resolution

5. Christian lost to Austin at tug-of-war, because he is strong/from

Willowbrook College.

(Left) (Right)

(hypothetical subject bias)

Pronoun resolution

5. Christian lost to Austin at tug-of-war, because he is strong/from

Willowbrook College.

(Left) (Right)

(results)

Pronoun resolution

1. Pre-diagnostic subject bias

2. Post-diagnostic moves in direction of control resolution

1. A beat B at tug-of-war,because he is strong/from

Willowbrook College.

(Weak) (Strong)

(hypothetical)

StrengthL StrengthR

(Weak) (Strong)

1. A beat B at tug-of-war,

1. A beat B at tug-of-war,because he is strong/from

Willowbrook College.

(Weak) (Strong)

(hypothetical)

StrengthL StrengthR

(Weak) (Strong)

5. Christian lost to Austin at tug-of-war, because he is strong/from

Willowbrook College.

(Weak) (Strong)

(hypothetical)

StrengthL StrengthR

(Weak) (Strong)

5. Christian lost to Austin at tug-of-war, because he is strong/from

Willowbrook College.

(Weak) (Strong)

(hypothetical)

StrengthL StrengthR

(Weak) (Strong)

r = 0.323

Pre/Control Correlation

Post/Control Correlation

r = 0.990

Comparison

Empirical Evaluations

1. Troy (Novelty)2. College (Learning)

Empirical Evaluations

1. Troy (Novelty)2. College (Learning)

Cognitive Approaches to the Winograd Schema Challenge:

Challenges and Complications

Jesse MuSupervised by Joshua Hartshorne and

Tobias Gerstenberg