Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Conversational Informatics (E), October 3, 2018
1. Introduction—Conversational Informatics—
Toyoaki NishidaKyoto University
Copyright © 2018, Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, At ,Inc. All Rights Reserved.
Why do we study conversation?
AI People
Data
Service
Common ground
Conversation as continuous update of the common ground
Conversations in different settings
Laboratories (controlled) Wild (uncontrolled)
Augmented (unrestricted)
Main Focus: Conversation
Shot by Ms. Sutasinee Thovuttikul with permission at a market in Chiang Mai, Thailand
Eye gaze
Hand gesture PostureFacial expression
AskingNegotiating
Proposing
Conviviality
Social networksTrust
Conversation is a complex business
[A marketplace in Chiang Mai; courtesy of Sutasinee Thovuttikul]
Challenge: A robot that can participate in conversation
Long-term goal
Not just conversational but also empathic
History of conversational systems development
t1990 2000 201019801970
Natural language dialogue systems
Speech dialogue systems
Multi-modal dialogue systems
Embodied Conversational Agents / Intelligent Virtual Human
Story Understanding systems
Conversational Systems
Transactional systems
Interactional systems
Affective Computing
Cognitive systems
Natural language question answering systems
The Knowledge Navigator
Our research group
SPOC (2005)
SituatedKeyword-based
EgoChat (2000)
CoMeMo Community (1998) POC TV(2001)
Physical robots
Vicky (2008)
Sustainable Knowledge Globe (2006)
Has addressed numerous aspects
[Nishida 1998][Nishida 1998]
CoMeMo Community
[Kubota 2000]
Using alterego agents
[Kubota 2000]
EgoChat
Knowledge cardKnowledge card
Knowledge cardKnowledge card
Knowledge card server
Knowledge card
presentation by ECA
Knowledge card editor
Knowledge channel control policy
landscapes
[Nishida DNIS 2003]
Knowledge Card Circulation
EgoChat -- demo
[Kubota+ 2003]
[Nishida 2002]
POC: Public Opinion Channel [Nishida 1999]
POC TV (KDDI FTTH Trial)
Capturing conversation quanta using auxiliary devices
P
A
B
button press
utterance 1 utterance 2reaction to the present
reaction to the past
proaction to the future
the timing of button press
and conversation quantum
Analysis
[Kubota and Saito 2005]
In-vehicle conversation circulation system
[Okamura+ 2007]
conversation assistance and capture
by recognizing nonverbal behaviors (e.g., pointing)
driving simulatorautomobile with video recorder capture and reuse conversation in automobile
- personal experience and social interaction -
detect pointing, display names and previous conversations
Capture
Driving experiences data base
Interactive presentation
Driving experience recording system
Scenes+GPS info
[Nishida 2008]
User behavior
Intelligent Driving Simulator
The purpose: to help people share and evolve the driving experiences
data base by allowing them to conversationally annotate driving scenes.
Vickey: In-vehicle conversation circulation system
The Sustainable Knowledge Globe (SKG)
[Kubota+ 2004]
[Kubota+ 2004]
The Sustainable Knowledge Globe (SKG)
[Nakano 2005]
SPOC
[Terada+ 2002]
Autonomous mobile chair
Conversational artifacts shows how synthetic
characters or intelligent robots use eye gaze, gestures
and other non-verbal communicators to interact.
Conversational contents looks at developing
techniques for acquiring, editing, distributing and utilising
the contents that are produced and consumed in
conversation.
Conversation environment design explains
techniques for creating intelligent virtual environments
and for representing individuals within a virtual
environment
Conversation measurement, analysis and modelling
demonstrate how conversational
behaviour can be measured and
analyzed.
Buy your copy here
or online at
www.Wiley.com
An Engineering Approach
http://as.wiley.com/WileyCDA/WileyTitle/productCd-0470026995,descCd-tableOfContents.html
Our research group addressed
SPOC (2005)
SituatedKeyword-based
EgoChat (2000)
CoMeMo Community (1998) POC TV(2001)
Gesture-based
Vicky (2008)
Sustainable Knowledge Globe (2006)
Has addressed numerous aspects
Not realistic!
But limited
Approach 1: Making environment playful
Smart conversation space that encompasses
participants and referents of conversation.
-> Engaged conversational interactions
-> More insights about the common ground
Augmentation
by MR (VR—AR)
Daily living space
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
[Nitschke 2013]https://www.youtube.com/watch?v=V-9SKpcMrzk
Conversations in an artificially created environment
ICIE: Immersive Collaborative Interaction Environment
Appearance Architecture Projection
Inside view
https://www.youtube.com/watch?v=wxkZ9armrI8
Interactive Dome
DEAL: Platform for ICIE
[Ohmoto 2011]
Learn from Playful interactions
Shared interaction environment cohabited by people and agents
[Suyama, Ohmoto 2015]
Player #1 in ICIE #1
Red hat
Player #2 in ICIE #2
Green hat
Player #3 (agent)
Blue hat
Analyze the behaviors of
participants by integrating audio-
visual and
physiological .measurement
Approach 2: Understanding by measurement and analysis
Approach 2: Understanding by measurement and analysis
Measurement in an open spaceMeasuring conversation in a fully equipped space
Conversation in a specific spatial arrangement Measurement in an augmented space
Corneal Imaging CameraScene cameraEye camera
• Lightweight and versatile system
• Appl.: Google Glass like HMD, unconstrained setups
Corneal Image Feature Matching
Problem:
Local feature correspondence + RANSAC does
not work due to large noise in eye images
Approach:
1. Formulate problem as registration of 3D spherical light maps of eye and scene image
2. Single point algorithm for robust alignment
Non-intrusive Eye Gaze Tracking (EGT) by corneal imaging
Intentions are dynamic and tacit
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Facilitative agent that can track dynamic and tacit intentions
Degrees of concentration
Level of proficiency
Tacit social signals and mental process beneath surface
Approach 3: Learning by imitation
Endow robots with an ability
of autonomously imitating
human behaviors.
Approach 3: Learning by imitation—Generic framework
Measurement Corpus Generalization Dialogue
patterns
[Nishida et al 2014]
Endow robots with an ability
of autonomously imitating
human behaviors.
Interactions from observation—General framework
Causes
Causes
Causes
1a
2a
3a
t
t
t
[Nishida et al 2014]
Problem formulation:Find approximately repeated subsequences in a longer time series.
(1) Motif Discovery—Finding Patterns of Interaction
[Nishida et al 2014]
Future
Change angle
GH
Past Futuret
( ) ( ) ( );...; 1H t seq t n seq t= − −
( ) ( ) ( )1 ;...;G t seq t seq t n= + +
( ) ( ) ( )
( ) 1
1
1
ˆ
f
f
T
i i i
l
i i
i
l
i
i
s t t t
cs
x
c
t
=
=
= −
=
TtVtStUtH )()()()( =
Find optimal lPggT uutGtG =)()(
Find optimal lF
11and,)( +−= jjjF
g
ii liut
fT
ll
T
lli li
tUU
tUUt = ,
)(
)()(
)()()()()(ˆ)(~ tttttxtx PFPF −−=
Learning by imitation
Robust Singular Spectrum Transform
[Mohammad 2009]
[Mohammad 2009]
Learning by demonstration
[Mohammad 2016]
Application
Platform Evaluation
Content production
Model building
Analysis
Theory
Measurement
Conversational interactions
Conversational Informatics
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Building conversational
systems
Understanding conversation
Limitations
Our conversational agents do not produce
a strong sense of presence.
We lack the common ground!
Common ground
Physical configuration
Imaginary scene shared by participants
Communal background including the cultural and biological aspects
Common ground
Bargaining scenario (successful)
C: Oh, OK. How much is this one?
S: This one? It worth nothing, you can have it [for free].
C: Haha, well, please...
S: The price is 500 tomans.
C: Oh! 500 tomans? It is very expensive? Do you have anything cheaper?
S: This is from a very good brand, we have the no-name ones, you can see
up there, and they are cheaper.
C: But the colors of those do not match my table. I would love to buy this if
you give me a very good discount. You know, I am about to marry,
marriage itself costs a lot, so it would be nice if you would do that.
S: It is your marriage, you should treat us! … Because it is a happy event I
can give it for 450.
C: 450 is still a little bit expensive. Why not we make a deal for 400?
S: 400 is too low, but since it is my first sell of the day, and you are getting
married, and everything looks very great I can give it for 430, but you
know, I haven’t sold it less than 450 so far.
C: You have been so nice so far, why not getting rid of that 30 and make it
round for 400. I am sure you are making lots of purchases today with
other customers and you can compensate for that. And I’m going to be a
frequent customer from now on, I will come to your shop again.
S: We would love to have you again, but the price is somehow at its
margin. But, I can guarantee it for one year for you, and I can give it to
you for 420, final! This is very good brand, you would never regret it, you
would love it, you can have my word that it works for you perfectly and
for one year if anything happens to that you can bring it to us and we can
fix it for you for free. Ok, I’ll write 420 for you.
C: Ok, thank you, you did me a good favor. Thanks a lot.
S: Oh please. We enjoy you being here and please come back soon.
C: Yeah I will. I’m actually going to do more purchases from now on from
your shop. I had a good experience.
S: Sure! Have a good time.
C: Hi. How are you?
S: Hi. Welcome to this shop.
C: Thank you. How much are these handicrafts here?
S: Oh, these are called Khaatam .The big ones are 120, [middle
sized ones for] 80 tomans and [small ones are] 50 tomans.
C: Um, this is Termeh, right?
S: Yes!
C: What about the Termeh?
S: Actually we have different types of Termeh. We have the red
ones there starts from 50 and the brown ones start from 150
and these special ones starts from 300.
C: You know what? I am about to marry, and I am looking for
something very nice to decorate my table. Its color is brown.
So, what do you suggest?
S: Oh! Congratulations for your marriage!
C: Thank you.
S: We usually go for the red ones for brown tables, but because
you are a great customer and you have good taste, probably
you would go for this one, this is very beautiful and it is from a
very famous designer.
Ostensible offer
[Mirzaei+ 2017]
How much
is this one?
I expect him to
understand that
I want to know
the price.
Immediate quote
is not good.
Successful Scenario: Extract Analysis
I am asking him to
give me the
information on the
price so that I can
decide whether to
buy it or not.
She is asking
how much I
expect her to
pay for this item.
She does not expect
me to quote the price
immediately. It is
against convention.
[Mirzaei+ 2017]
Have it
for free
Look at his body
language.
He shows his
sincerity.
Successful Scenario: Extract Analysis
Two arms extended forward with palms up
He is being kind
and polite but his
offer is not real.
He expects me to
decline the offer to
balance the equity.
She will be happy
by my ostensible
offer and she will
ostensibly decline.
I show her my
sincerity. I’m making
a non-real offer.
[Mirzaei+ 2017]
Haha
Successful Scenario: Extract Analysis
She enjoyed it
and she knows
the offer is not
real.
I expect him to
be relieved by
receiving my
declination.
I understand you are
being very nice and I
acknowledge your
politeness.
She acknowledged
my offer. She is nice
and the atmosphere
of the conversation
is good.
She expects me to
continue with the
convention (i.e.,
going with 2nd offer)
[Mirzaei+ 2017]
Synthetic evidential study
Synthetic evidential study (SES) combines dramatic role play and group discussion to help people spin stories by bringing together partial thoughts and evidence.
Componentize
Reuse
SES session Interpretation archive
Structured collection of {story, background, critique}Agent Play
Dramatic role play
Group discussions
[Nishida et al 2015]
At the beginning of the 18th century, a feudal lord named Asano Takumi-no-kami
Naganori was in charge of a reception for envoys from the Imperial Court in Kyoto.
Another feudal lord, Kira Kozuke-no-suke Yoshinaka, was appointed to instruct
Asano in the ceremonies. On the day of the reception, while Kira was talking with
Yoriteru Kajikawa, a lesser official, at “Matsu no Roka” (“Hallway of Pine Trees”) in
Edo Castle, Asano came up to them screaming “This is for revenge!!” and slashed
Kira twice with a short sword. Soon after the incident, Kajikawa restrained Asano,
who was then imprisoned. The reason for the attack was not known, though it was
widely believed that Kira had somehow humiliated Asano. Ultimately Asano was
sentenced to commit seppuku, a ritual suicide, but Kira went without punishment.
Hallway of Pine Trees (from Chushingura)
Kira Kozuke-no-suke Yoshinaka
Asano Takumi-no-kami Naganori
Yoriteru Kajikawa
Why was it possible?
How did it happen?
What did each think?
Dramatic Role Play
Group play capture
Agent play
Discussion phase
T. Ookaki, M. Abe, M. Yoshino, Y. Ohmoto and T. Nishida. Synthetic Evidential Study for Deepening
Inside Their Heart. IEA/AIE 2015.
Asano
Kira
Kajikawa
Third person view First person view
Discussions
1. Common ground for society and AI
2. Conversation as a contiguous update of common ground.
3. Conversation envisioning as visualization of conversation and common ground
4. Conversation envisioner as a platform for conversation envisioning
Conversation as
contiguous update
of common ground
Analysis
Training
Assistance
Conversation envisioner
Vision — Conversation Envisioner
TSEiA: The Story Envisioning Agent
[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]
TSEiA: The Story Envisioning Agent
[Qiang Zhang. TSEiA: The Story Envisioning Agent, HCII 2018, Student Design Competition (to be presented)]
FACSvatar: An Asynchronous Open Source Modular Framework From Face to FACS Based Avatar Animation
Flow of data in framework
Facial expression data can be used with any character[van der Struijk+ 2018] Stef van der Struijk, Hung-Hsuan Huang, Maryam Sadat Mirzaei, Toyoaki Nishida. FACSvatar: An Open Source Modular Framework for Real-Time FACS based Facial Animation, IVA 2018 (accepted)
Role-play Envision
L2 Learners New L2 Learners Teacher
AI Actor AI Assistance
Conversation Envisioning for CALL Purposes
CALL: Computer-Assisted Language Learning
[Mirzaei+ EutoCALL 2018]
Agenda
CreditsWill be awarded based on a report on subjects given at the class. Due date (January 31st, 2019)
Agenda (planned)
1. October 3: Introduction by Nishida2. October 10: History of Conversational Systems by Nishida 3. October 17: Methodologies for Conversational System Development (1) by Nishida4. October 24: Methodologies for Conversational System Development (2) by Nishida5. October 31: Methodologies for Conversational System Development (3) by Nishida
(November 7: no class)6. November 14: Cognitive Interaction Design by Ohmoto7. November 21: Smart Conversation Space by Ohmoto
(November 28: no class)8. December 5: Measurement, Analysis and Modeling by Ohmoto9. December 12: Affective Computing, Theory of Mind by Nishida10. December 19: Learning by Imitation by Nishida 11. December 26: Aspects of Conversation – 1 by Nishida12. January 9: Aspects of Conversation – 2 by Nishida13. January 16: Function of Nonverbal Behaviors in Conversation by Nishida14. January 23: Synergy by Nishida
Course materials available from: http://www.ii.ist.i.kyoto-u.ac.jp/?page_id=5881