View
219
Download
0
Category
Preview:
Citation preview
Facial Affect Displays During Tutoring Sessions
Mattijs Ghijsen – Dirk Heylen*
Rieks op den AkkerAnton Nijholt
2
INES
3
Facial Expressions
• What do they tell about how the student is experiencing the situation?
• How might we be able to use this to improve the tutoring system?
4
Previous Work
• Based on– Student Activity– Difficulty– Errors
• Adjust– Adjust teaching strategy– Adjust dialogue move
5
Pilot
• Collect data• See what happens
6
Interpreting
• If some expression occurred• What triggered it?• What does it mean?
= How can a system make sense of what happened and react appropriately?
7
Elements to consider
• Events:– system– student
• Appraisals• Facial Expressions
8
Facial Expression Dictionary?
• Emotions• Mental state• Conversational function• Adaptor
9
More complex
• Using Stimulus Evaluation Checks:– Situation: how could it be evaluated– Face: what could have triggered it
• Why do it?• Some way of flexibility:
– if match between f.e.-sec and sit.-sec: ok– if no match than: reason
10
Component Process Model
• Scherer• Eclectic• Appraisal process
– Stimulus Evaluation Checks– 5 criteria– Performed sequentially
11
Stimulus Evaluation Checks
Novelty change in the pattern of external or internal stimulation
Intrinsic pleasantness
whether a stimulus event is pleasant, inducing approach tendencies, or unpleasant, inducing avoidance tendencies
Goal/Need significance
whether a stimulus event is relevant to important goals or needs of the organism; outcome consistent with expectations; conducive or obstructive to reaching goals
Coping potential
evaluating the causation of a stimulus and the coping potential available, particularly the degree of control over the event and relative power
Norm/Self compatibility
evaluating whether the event, particularly an action conforms to social norms, cultural conventions…
12
Component Process Model
Function Subsystem Component
Evaluation of stimulation
Information processing
Cognitive
System regulation Support Neurophysiological
Preparation and direction of action
Executive Motivational
Communication and direction of action
Action Expressive
Monitoring, attention, focus, reflection
Monitor Subjective feeling
13
SEC AU
Novelty High Low
Sudden 1 + 2 + 5 + 26/27(1/2=brow raiser; 5= eyelid; 26/27=mouth)
Familiarity 4b + 7
Predictability 4b + 7
14
INES
15
The Exercise
• Ask patient to put arm on table• Disinfect part of the arm• Insert the needle• Inject medicine
16
Movie
17
Movie
18
Movie
19
Movie
20
Movie
21
Movie
22
Movie
23
Situation – Expression
Patient asks for clarification raised brows (2), frown (1), head pulled back (1)
Patient repeatedly does not understand what student says
smile (2)
Patient pulls up her sleeve smile (4), raised brows (2), nod (1), pull head back (1)
Student is disinfecting the arm smile (7), raised brow (1)
24
Facial Expression Appraisal
Expression Novel Pleasant Goal significant
Coping Potential
Smile (22) 3 18 19 14
Raised eyebrows (11)
7 4 2 0
Mouth corners pulled down (2)
2 1 1 1
26
For our application
• Facial expressions tell us something
• that might be of use.– Students are finding it pleasant– Students are concentrating– Students are not frustrated– Students were not expecting
something
27
Discussion
• Not only emotion• Pilot: not real data• Complicated procedure• Many things indeterminate
28
Questions:
• How to classify situations in terms of SECs?– Situation: what to take into account?– Appraisal not situation– Coder consistency
• Facial expression:– how to analyse in terms of SECs– ambiguity – incompleteness
• How to make a system work with such information structures?
29
Problem
• Matching 1) potential appraisals of situation
types with2) potential significance of facial
expression
• 1) = hard to do for types• 2) = lots of ambiguity
30
So
• Why do it?• Some way of flexibility:
– if match between f.e.-sec and sit.-sec: ok
– if no match than: reason
Recommended