Upload
beverly-park-woolf
View
9
Download
1
Embed Size (px)
Citation preview
Beverly Park Woolf College of Information and Computer Sciences,
University of Massachusetts-Amherst [email protected]
Detection of and Response to Online Users’ Emotion
Department of Quantitative Health SciencesUniversity of Massachusetts Medical School
March 24, 2017
Bounce between two systems . . .
Mathematics Tutor: Exists and contains most of the described features.
Patient Care Tutor: Proposed and can be built with these features.
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Assess Patient Learning
Agenda
Caring: A Definition
Caring implies actions based on another person’s wants and desires (Noddings, 1986).
Entities that care consider the other person’s point of view, her needs and what she expects of us (Cooper, 2003).
Caring includes the ability to empathize with others and take responsibility for their needs.
Caring Impacts Learning
Findings from neuroscience suggest that all learning is affective in nature (Damasio, 1994).Students learn most effectively in a climate in which people care about them (Cooper, 2003).A major weakness of traditional psychology is to separate the intellect and affect (Vygotsky, 1986). Every person’s idea contains an affective attitude.
When Caring is Present . .
Positive emotions and interactions create the ambience for learning and enable a student’s brain to remain curious and open (Cooper, 2002; 2003).
A continuing, positive sense of self produces a constant positive feeling throughout the body, which leads to greater openness and willingness to engage in interactions (Winkley, 1996). Babies brains grow when they feel cared for.
When Caring is NOT Present . . .
Students flounder in internal confusion (Cooper, 2003).
Negative affect tends to produce a shutting down of one’s self, a withdrawal, stimulating protection and defense.
Research Questions about Emotion
How are users’ emotions evidenced and measured?
How do emotions predict learning?
How accurate are emotion models (e.g., Markov or Bayesian Models) at predicting future emotions from student behaviors?
How effective are interventions at responding to emotion??
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Model the Patient
Model the Disease
Personalize Tutoring
Assess Learning
Intelligent TutoringSystems
Adele explains the importance of palpating the patient’s abdomen.
Background Research: Shaw, Johnson , & Ganeshan (1999). In Proceedings of the third annual conference on Autonomous Agents (pp. 283-290). ACM.
Adele advising in a critical care scenario on the World Wide Web. Pedagogical Agents on the Web
Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
Adele instructs a student to answer a quiz when the student selects a urine dipstick test. Pedagogical Agents on the Web
Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan
Background Research: Taking the Time to Care: Empowering Low Health Literacy Hospital Patients
with Virtual Nurse AgentsTimothy W. Bickmore, Laura M. Pfeifer
College of Computer & Information Science Northeastern University {bickmore,laurap}@ccs.neu.edu
• Ninety million Americans have inadequate health literacy, resulting in a reduced ability to read and follow directions in the healthcare environment.
• Animated, empathic virtual nurse interface for educating and counseling hospital patients with inadequate health literacy in their hospital beds at the time of discharge.
• Results indicate that hospital patients with low health literacy found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or nurse.
Patient Interacting with Virtual Nurse
Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer
Chi 2009
Sample dialogue - relational aspects highlighted Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer
Chi 2009
Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer
Chi 2009
Self-report Ratings of the Virtual Nurse (mean (SD))
Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer
Chi 2009
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
MathSpring
An intelligent tutor for mathematics;
Contains hundreds of math problem, grades 5-10;
Used by tens of thousands of students;
Aligned with Common Core standards;
Detects and responds to student emotion;
Improves student passing of state standardized tests;
Uses multimedia characters to support student emotion;
Built by Woolf & Arroyo at UMass-Amherst.
23
The MathSpring System
Pedagogical AgentsAdult Users Require Adult Companions
Help to develop a positive emotional environment; Agents offers advice and encouragement.
Empathize with patientAgents express full sentences of cognitive, meta-cognitive and emotional feedback.
Build patients’ self-esteem and self-worthAgents are gendered and multicultural: White, Hispanic, African-American
Mass Statewide Standardized Tests
Passing grades for experimental (dark grey) and control group (light grey); same grade, same school and same teacher.
Students represented by the yellow/green polygon used the Math Tutor; those represented by the blue polygon did not. The distribution of students using the tutor is shifted towards the right, towards more proficiency and above proficiency. Students were matched in terms of teacher; all were 7th grade students.
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Sensors to Detect Student EmotionDavid Cooper, Ph.D. Dissertation
Sensors to measureStudent Emotion
(Clockwise): mental state camera, skin conductance bracelet, pressure sensitive mouse, pressure sensitive chair.
31
Use Sensors
32
The Students
Rural-Area High School in MA (35 students)Geometry and Algebra classes
UMASS 114 (29 students)Math for Elementary School Teachers
33
Students Self-Report EmotionsFour bipolar emotional axes
How are you feeling? Please rate your level of interest in this
Using Sensors to Measure Emotion:A Linear Regression Model
R represents the fit of the modelN is the number of cases available, for each emotion self-report and each sensor
Significantresults
Student self-report
Cooper et al., UMAP 2009;Arroyo et al., AIED 2009
David Cooper, Ph.D. Dissertation
36
Models of Emotions with SensorsFrom Tutor-Context Variables and Sensors
Linear Models to Predict Emotions Variables Entered in Stepwise Regression
Confidence InterestFrustration Excitement
SitForwardStdev
“Concentrating”Max. Probability
Camera Facial Detection Software
SitForwardMean
Seat Sensor
# Hints Seen
Solved? 1st Attempt
# Incorrectattempts
Gender Ped. Agent
Seconds to1st Attempt
Time in Tutor
Seconds To Solve
Tutor Context Variables (for the last problem)Tutor Only All Sensors+Tutor
R=0.53 R=0.43 R=0.37R=0.49
R=0.72 R=0.70 R=NAR=0.82
Sensor Variables (Mean, Min, Max, Stdev for the last problem)
Results of Emotion Prediction Studies
Sensors predict self-report, accuracy 75-80%. Sensors improve student emotion self-reports. Self-reports predicts post-tutor math performance, attitudes and perceptions of tutoring software.
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Computer Vision to Monitor & Measure Grit
• Several written Instruments are used to measure non-cognitive skills, primarily self-report surveys.– Self-regulation– Motivation strategies (MSLQ)
• Students with grit & persistence have good self-regulation skills:– Goal setting, self-monitoring, and self-instruction; – They manage their emotions and waning motivation
Head Position
The right mouth corner is tracked during student-computer interactions with significant changes of head position and orientation
Low Level Features
Extraction of low-level features from face, (a) furrows, (b) iris, and (c) mouth, for FACS-based facial expression analysis, yielding the interpretation “student exhibits anger.”
Trace Student Attention
The Mathspring GUI overlaid with a “heat map” of student attention, as measured
by computer-vision. In red, screen regions the student particularly focused
on
Engaged while reading and solving the problem
Head tilts
Starting to get annoyed with the learning companion
• Student was actually fine with the learning companion at first but seems to get annoyed later on
• Seem to be offended when the learning companion states “See, I told you that hints are really helpful”. He never took any hints and did everything correctly on every first try
• Finally closes the learning companion as soon as it pops up after doing six problems
Looking to the right side of the screen means he is looking at learning companion
Closing learning companion
Companion removed by
student.
Summary for GritCamera6
• Student is very engaged• Student takes time to answer every question
– Made no mistake– Took no hint– Got it right in every first try
• Takes time to answer survey– Only two survey shows up
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Cardiac Tutor, Built at UMass-Amherst
The Cardiac Tutor teaches advanced cardiac support techniques to medical personnel. It presents cardiac problems and, using a variety of steps, students select various interventions. It provides clues, verbal advice, and feedback to personalize and optimize the learning..
Simulated patient in the Cardiac Tutor. The intravenous line has been installed
( “IV in ”); chest compressions are in progress, ventilation has not yet begun,
and the electronic shock system is discharged. Eliot, Williams & Woolf, 1996, An intelligent learning
environment for advanced cardiac life support.
Rashi Case DescriptionBuilt at UMass-Amherst
Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
Rashi Interview Tool
Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
Rashi Exam Tool
Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
Rashi Lab Test Tool
Rashi Inquiry Notebook
Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
Rashi Argument Editor
Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment
Background Research: Medical Knowledge
Bioworld patient scenario and evidence palette.Susanne Lajoie 2001
Constructing knowledge in the context of BioWorld
Model Medical Knowledge
(Left) Conducting Diagnostic Tests; (Right) constructing the closing argument Lajoie 2001; Constructing knowledge in the context of BioWorld
Lajoie & Greer, 2003 Establishing an Argumentation Environment to Foster Scientific Reasoning With Bio-World
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Research Questions
• What to do in the moment when users are frustrated, bored, etc.?– Increase Challenge? Decrease Challenge?– Provide extra scaffolds?– Provide “affective” scaffolds? What are those?– Encourage users to stop and think about what is going
on?• How to measure changes in student affect, or to
capture micro-changes in student affective states?
Interventions have a strong impact on Learning
• Average effect size of metacognitive instruction across 20 studies was .72, a very large effect.
– Teach a skill; track the student, measure the skill before and after use of the software.
– Measure a student’s aptitude for learners trained in a skill against students who are not trained in that skill.
Many people: • relate to computers in the same way they relate to
humans (Nass, 2010);• continue to engage in frustrating tasks significantly
longer after an empathic digital response (Picard); • have lowered stress levels after receiving an empathetic message from a
digital character (Arroyo et al., 2009); • recalled more information when interacting with an
artist agent compared to scientist agent; • report reduced frustration and more general interest
when working with gender-matched characters.
People and Agents
Affirmation Theory Messages
Affirmation Theory: Propose that users’ motivation is rooted in their belief about why they succeed or fail. Students can be taught to understand that failure is the result of a lack of effort instead of a lack of ability.
Example Messages “People have myths about math, like, that only some people are good in
math. The truth is that we can all be successful in math if we give it a try”
“We will learn new skills only if we are persistent. If we are very stuck,
let's call the teacher, or ask for a hint!”
“When we realize we don't know why the answer was wrong, it helps us
understand better what we need to practice.”
Effort Interaction Messages
Effort Interactions: Acknowledge effort and incorrect answers. The goal is to make students realize that praise is not always appropriate and that effort is the primary goal.
Example Messages“That was too easy for you. Let's hope
the next one is more challenging so that we can learn something.”!”
“Good job! See how taking your time to work through these questions can
make you get the right answer?”
Incorrect ResponseStudent effort shown/correct response
Student effort shown /incorrect response
Agent Emotion
Frustrated students are supported by helpful companions.
Arroyo et al., AIED2009
• Our current Gold standard/Truth: self-reports• During Experiment:
– Ask students about their “interest” level and their “excitement” level every 7 minutes, on average.
Gathering Student Affect “During”
Low
HighNeutral / Middle
Large Data Sets
EventLog Table of a Math Tutoring System. 571,776 rows, just in a year time.
The Student Progress Page
73
Dovan Rai, PhD, WPI
Three Experimental Conditions
74
• No access to the Student Progress Page
• “My Progress” button was present (student choice)
• Prompt invitation to see “my progress” upon bored (disinterested) or unexcited
• Force student to see my progress when student bored (disinterested) or unexcited
How to analyze changes in student affect, from moment to moment?
MARKOV CHAIN MODELS
Student InterestTen thousand Data Points N~230.
SPPAbsent
SPPPresent
SPP Prompted
SPP Forced
How to compare Markov Chain Models, quantitatively?
• Probability of student following a specific path• What is the probability that a student will end
up excited, after 3 transitions?• Given that they started in a specific state?
Resulting Data
Introduction and Care for the Patient
Model the Patient
Model Emotion (Sensors & Computer Vision)
Model the Medical Domain
Provide Interventions
Propose Health Care Tutor
Agenda
Health Care Research Questions
Which interventions improve patients’ affect and learning?
Which interventions are best for which patients?
What is the impact of deeper companion characteristics?
Do affective characters make users “feel” better?
How does the gender (ethnicity) of companions impact patients’ attitudes/emotions/learning?
Detect patient emotion, using Computer Vision or sensors (camera, wrist band, pressure mouse, seat sensors).
Dynamically collect data streams of students’ physiological activity and self-reports of emotions.
Apply interventions once emotion is detected, e.g., animated embodied agents
Integrate computer vision techniques to improve detection of emotion.
Train researchers and educators in techniques to recognize and respond to patients’ emotion and make predictions over large data sets.
Proposed Health Care System
Study Design
Patients
Control
ExperimentalIntervention
Control
ExperimentalIntervention
Visit # 1 Visit # 2 Visit # 3 Visit # 4
Randomized
• Survey• Audio tape with physician
• Survey• Audio tape with physician
• Survey• Audio tape with physician
• Survey
• Survey • Survey• Survey
Baseline data
Greenfield et al., 1986; Patients’ participation in Medical Care General Internal Medicine
Emotion is vital to learning and using technology to recognize users’ emotion has led to powerful performance results.
Detecting and responding to users’ emotion while they work with online environments improves learning.
Summary
Thank You !Any Questions?
Detection of and Response to Online Users’ Emotion
Department of Quantitative Health SciencesUniversity of Massachusetts Medical School
March 24, 2017