Imaginary Worlds and Imaginary Worlds and Grounded (Embodied) Grounded (Embodied)

CognitionCognition

John B. BlackJohn B. Black

Teachers CollegeTeachers College

Columbia UniversityColumbia University

Research with Gordon BowerResearch with Gordon Bower

Studied Students Reading and Studied Students Reading and Remembering StoriesRemembering StoriesFound That Switches in Point of View Found That Switches in Point of View Lead to Longer Reading Times and Lead to Longer Reading Times and Memory Errors – Spatial LayoutMemory Errors – Spatial LayoutFound That Actions That Changed The Found That Actions That Changed The Story World Were Remembered BetterStory World Were Remembered BetterSeemed to be Story World Separate From Seemed to be Story World Separate From Symbolic Content of Story (Mental Model)Symbolic Content of Story (Mental Model)

Symbolic Content vs Model WorldSymbolic Content vs Model World

Johnson-Laird Proposed Mental Models Johnson-Laird Proposed Mental Models That Involved Imagining Models of Logical That Involved Imagining Models of Logical ExpressionsExpressionsKintsch Proposed Situation Models of Kintsch Proposed Situation Models of Textual Content Separate from Textbase Textual Content Separate from Textbase of Propositional Networksof Propositional NetworksPhilosophers like Frege and Carnap Philosophers like Frege and Carnap Distinguish Between Sense and Distinguish Between Sense and Reference Reference

My Research at TCMy Research at TC

Shifted from Text to MultimediaShifted from Text to Multimedia

What is the Nature of These Mental What is the Nature of These Mental Referent or Imaginary Worlds?Referent or Imaginary Worlds?

How Does One Teach For Imaginary How Does One Teach For Imaginary World Level of Understanding?World Level of Understanding?

World of a CathedralWorld of a Cathedral

Multimedia System to Learn About St. John’s Multimedia System to Learn About St. John’s Cathedral (Van Esselstyn and Black)Cathedral (Van Esselstyn and Black)

Learned a lot from just Floor Plan (survey map Learned a lot from just Floor Plan (survey map knowledge) and Textknowledge) and Text

Providing Virtual Tour (dynamic route map Providing Virtual Tour (dynamic route map knowledge) only helped when given rationale for knowledge) only helped when given rationale for spatial layoutsspatial layouts

Constructing this Cathedral World needed Constructing this Cathedral World needed reasons for spatial layoutreasons for spatial layout

Multimedia CathedralMultimedia Cathedral

Research with Maggie ChanResearch with Maggie Chan

Crucial Element of Imaginary World Crucial Element of Imaginary World Reasoning is Functional RelationsReasoning is Functional RelationsFunctional Relations Describe How One Functional Relations Describe How One Entity Changes as a Function of Change Entity Changes as a Function of Change in Anotherin AnotherDeeper Lever Understanding Than Deeper Lever Understanding Than Structural Causal RelationsStructural Causal RelationsShowed Direct Manipulation Animation Showed Direct Manipulation Animation Good Way to Learn ThisGood Way to Learn This

Roller Coaster Simulation GameRoller Coaster Simulation Game

Roller Coaster Game StudiesRoller Coaster Game Studies

Snapshots (like Comics) Better than Snapshots (like Comics) Better than System Controlled AnimationSystem Controlled Animation

Consistent with Many Studies of Animation Consistent with Many Studies of Animation in Tversky Survey in Tversky Survey

““Direct Manipulation” of one Factor with Direct Manipulation” of one Factor with Animation of Other Factors Gave Better Animation of Other Factors Gave Better Learning (Chan and Black)Learning (Chan and Black)

Experimental DesignExperimental DesignIV1: Presentation Formats:

text-only, text-plus-static diagrams, text-plus-direct-manipulation animation

IV2: System Complexity:

simple (Energy conversion in a playground swing),

moderate (Energy conversion in a roller coaster ride),

complicated (Energy conversion of pole vaulting)

Dependent Measures:

DV1: Verbal Recall (summary)

DV2: Visual Recall (drawing)

DV3: What-if Scenarios

DV4: What’s wrong here? Case study

DV5: Transfer

Direct Manipulation Direct Manipulation

AnimationAnimationSimple system --- Simple system --- a playground swing a playground swing

Direct Manipulation Direct Manipulation

AnimationAnimation

Complex system --- pole-vaultingComplex system --- pole-vaulting

77thth Grade Grade

What-if scenariosWhat-if scenarios

66thth Grade Grade

Result Varies within Type of Result Varies within Type of Student and System ComplexityStudent and System Complexity

General form seems to be for simple General form seems to be for simple systems and capable students text is systems and capable students text is sufficientsufficient

Also for capable students and intermediateAlso for capable students and intermediate

systems text plus diagrams sufficientsystems text plus diagrams sufficient

But for complex systems for any students But for complex systems for any students and less capable students for any system and less capable students for any system need direct manipulation animationneed direct manipulation animation

Video Games for LearningVideo Games for Learning

There is a Lot of Hype About Video There is a Lot of Hype About Video Games as Learning EnvironmentsGames as Learning Environments

However, Almost No Studies That Show However, Almost No Studies That Show This – in fact, Some Show ReverseThis – in fact, Some Show Reverse

But We Have Found Video Games Good But We Have Found Video Games Good as Part of Learning Environment But Not as Part of Learning Environment But Not By ThemselvesBy Themselves

Video Games as Preparation for Video Games as Preparation for Future LearningFuture Learning

CivilizationCivilization

Preparation for Future Learning Preparation for Future Learning Video Game StudyVideo Game Study

Compared Experts in Civilization Compared Experts in Civilization

Formally Learning New MaterialFormally Learning New Material

Civilization Experts Were Better at Civilization Experts Were Better at Formally Learning New Material –Formally Learning New Material –

Particularly Systems RelatedParticularly Systems Related

Thus Video Games May Provide Thus Video Games May Provide

Experiences asExperiences as

Preparation for Future LearningPreparation for Future Learning

Reflective Questions with GamesReflective Questions with Games

Studied Business Student Learning From Studied Business Student Learning From Business Simulation Game (from HBS)Business Simulation Game (from HBS)Students Learned Much More If Game Students Learned Much More If Game Supplemented with Reflective Questions Supplemented with Reflective Questions About Strategies Used and Relation to About Strategies Used and Relation to Background ReadingBackground ReadingArmy Has Found Learning With America’s Army Has Found Learning With America’s Army Game Occurs in After Action Army Game Occurs in After Action Reports Which Provide ReflectionReports Which Provide Reflection

Grounded (Embodied) CognitionGrounded (Embodied) Cognition

Currently Think Imaginary Worlds Involves Currently Think Imaginary Worlds Involves Grounded or Embodied Cognition Grounded or Embodied Cognition

This is a Perceptual Simulation to This is a Perceptual Simulation to Supplement (Provide Mental Referents)Supplement (Provide Mental Referents)

Recent Neural Imaging Studies Show Recent Neural Imaging Studies Show Perceptual Areas of Brain Activated in Perceptual Areas of Brain Activated in Symbolic Tasks in Addition to Symbolic Symbolic Tasks in Addition to Symbolic AreasAreas

How To Investigate This?How To Investigate This?

Glenberg Text ComprehensionGlenberg Text Comprehension

ARIAARIA

REALREAL

??????

ARIA Project –ARIA Project –Associating Real and Imaginary Associating Real and Imaginary

AgentsAgents

ARIA Project Provides Embodiment by Having ARIA Project Provides Embodiment by Having Students Work With Robot Simulations on Students Work With Robot Simulations on Computer Screen Then Construct Robots to Do Computer Screen Then Construct Robots to Do It In Real WorldIt In Real World

Also Provides Embodiment By Having Students Also Provides Embodiment By Having Students Act Out Using There Own Bodies What they Act Out Using There Own Bodies What they Want Robots to DoWant Robots to Do

Being Done as After School Program at The Being Done as After School Program at The School at Columbia U.School at Columbia U.

REAL—Reflective Agent Learning REAL—Reflective Agent Learning environmentenvironment

Provides Embodiment by Allowing Learner Provides Embodiment by Allowing Learner to Input Knowledge Representations in to Input Knowledge Representations in Head of AgentsHead of Agents

Then See How Agent with KnowledgeThen See How Agent with Knowledge

Behaves in Virtual WorldBehaves in Virtual World

Various Depictions of Results ReflectsVarious Depictions of Results Reflects

What is Good and Bad About KnowledgeWhat is Good and Bad About Knowledge

Then Try AgainThen Try Again

REAL BusinessREAL Business

REAL Business (Design Mode)REAL Business (Design Mode)

REAL Business (Game/Reflection Mode)REAL Business (Game/Reflection Mode)

Constructing Dynamic Imaginary Constructing Dynamic Imaginary WorldsWorlds

Virtual Worlds with Animated Agents can be Virtual Worlds with Animated Agents can be Effective for Learning ifEffective for Learning if

Spatially MeaningfulSpatially MeaningfulCausally TransparentCausally TransparentDirectly Manipulate Factors and SeeDirectly Manipulate Factors and See

Animated ResultsAnimated ResultsGets Internalized as Imaginary WorldGets Internalized as Imaginary World

Augmented Reality Promises to GiveAugmented Reality Promises to GiveSame Benefits Better Integrated withSame Benefits Better Integrated withReal WorldReal World