correction of oblig. 1 presentation of oblig. 2€¦ · •Vowels have distinct formant structures that can help us distinguish them tirsdag 2. oktober 2012 @ 2012, Pierre Lison -

Exercise session: correction of oblig. 1 & presentation of oblig. 2

Pierre Lison, Language Technology Group (LTG)

Department of Informatics

Fall 2012, October 1 2012

tirsdag 2. oktober 2012

@ 2012, Pierre Lison - INF5820 course

Outline

• Correction of assignment 1

• Presentation of assignment 2

2tirsdag 2. oktober 2012


Outline

•Correction of assignment 1

• Presentation of assignment 2



Question 1: analysis of dialogue

• Turns mostly structured by

• Dialogue structure (questions followed by answers, etc.)

• Complete grammatical units

• + probably intonation, non-verbal cues etc. (although we don’t have direct access to them)




• Speech acts:

• Assertives: «he sent it this afternoon I think», «we’ll be in Trondheim that day»

• Directives: «could you help me carry these groceries in the kitchen?»

• Commissives: «sure will do»

• Expressives: «poor you», «too bad»




• Grounding and clarification:

• Lots of backchannels («mm» etc.)

• Clarification requests («which email?»)

• Explicit feedback («oh i see», «poor you»), on several levels (perception, understanding, attitudes)

• Implicit feedback («he sent his email this afternoon» - «didn’t check my email this afternoon»)




• Conversational implicatures:

• «how was your day?» «well my boss put one more big pile ....»

• «I didn’t check my email this afternoon»

• «We’ll be in Trondheim that day»

• The indirect request «why don’t you tell him that ...» could also be seen as a conversational implicature




• Deictic markers:

• Pronouns: I, you, we, my

• Adverbs: there

• Demonstratives: these, this

• Temporal phrases: this afternoon, the following weekend

• Tense markers: «he sent it»

• etc.



Question 2: shared intentionality

• Shared intentionality

9

our cognition is directed towards things in the world (desires, goals, attention)




• Shared intentionality

10

our cognition is directed towards things in the world (desires, goals, attention)

we can share this intentionality with others (joint attention, collaborative activities)




• Shared intentionality presupposes advanced skills for reading the intentions (and beliefs) of others

• Humans can also have joint commitments towards goals and coordinate action plans to realise them

• Spoken dialogue is a prototypical example of collaborative activity:

• Need to read and interpret each other’s intentions

• Gradual refinement & expansion of common ground

• Joint attention towards particular topics in focus

• Role of imitation and alignment



Question 3: phonetics

• Word boundaries difficult to detect on the waveform

• Stops: silence followed by a «burst»

• Fricatives: uneven, aperiodic waveform

12

Time (s)3.731 5.141

-0.2334

0.1682

0

untitled

Time (s)0.7956 2.387

-0.07474

0.07803

0

untitled




• Pitch variation indicates the utterance’s intonation (e.g. declarative vs interrogative sentence)

13

• Vowels have distinct formant structures that can help us distinguish them




• Intensity (in dB) can be calculated with the following formula:

14

for a sample of N time points t1,... tN

P0 is the human auditory threshold, = 2 x 10-5 Pa

Intensity = 10 log10Power

P0= 10 log10

1

NP0

N�

i=1

y(ti)2

Power =1

N

N�

i=1

y(ti)2




15

[Marta’s solution]



Question 4: probabilistic models

16

[Henning’s solution]




17





18





19

P (B|JC,¬MC)

= αP (JC,¬MC|B)P (B)

= α

�

a={T,F}

P (JC,¬MC,A = a|B)

P (B)

= α

�

a={T,F}

P (JC|A = a)P (¬MC|A = a)P (A = a|B)

P (B)

= α

�

a={T,F}

P (JC|A = a)P (¬MC|A = a)

�

e={T,F}

P (A = a|B,E = e)P (E = e)

P (B)

Marginalisation on A

Bayes’ rule

Marginalisation on E

NB: there are several alternative ways to get to this result




• Then we simply calculate the results for Burglary=true and Burglary=false and renormalise at the end:

20

P (¬B|JC,¬MC) = α× (0.9× 0.3× 0.29× 0.002

+0.9× 0.3× 0.001× 0.998 + 0.05× 0.99× 0.71× 0.002

+0.05× 0.99× 0.999× 0.998)× 0.999 = α× 0.049798

P (B|JC,¬MC) = α× (0.9× 0.3× 0.95× 0.002

+0.9× 0.3× 0.95× 0.998 + 0.05× 0.99× 0.05× 0.002

+0.05× 0.99× 0.05× 0.998)× 0.001 = α× 0.0002589

Renormalising with α = 19.977, we then have P(B|JC, ¬MC) = 0.00517



Outline

• Correction of assignment 1

•Presentation of assignment 2



Assignment 2

• In this assignment, you will be asked to develop a small dialogue system

22

• The domain is a simple, simulated human-robot interaction scenario:

• A robot is standing on a table where several objects are present

• The robot can move around on the table, perceive the objects in front of him, and pick them up



Assignment 2

• Basic skeleton for the dialogue system is provided

• The speech recognition and synthesis will be done «in the cloud», using the API provided by AT&T (but you will have to provide a speech recognition grammar for the ASR)

• Libraries for recording/playing sounds and connecting to AT&T servers for the ASR and TTS

• As well as a GUI for the simulation and display of the dialogue history

• Your task is to implement a dialogue policy processing the input (N-best list) and returning an appropriate action:

23

public Action processInput(NBest u_u, DialogueState dstate, WorldState wstate) { ... }



Assignment 2

• The dialogue policy must be able to handle the following:

• Greetings («hi») and closings («goodbye»)

• Commands to move the robot («go forward», «turn left», «turn right»)

• Talking about objects in the scene («do you see an object?», «what do you see?»)

• Picking up and releasing objects («pick up the object», «put down the object»)

• The system must also be able to produce clarification requests («sorry could you repeat?») when things are unclear



Assignment 2

• You can implement your policy in Java or in Python

• For Python, you’ll have to use Jython (a python implementation running on the JVM, and which can directly use Java classes)

• The package is now ready to use

• You’re of course free to help me improve or debug it (the code is made available on a SVN server)

• Submission deadline: October 21st


Documents

correction of oblig. 1 presentation of oblig. 2€¦ · •Vowels have distinct formant structures that can help us distinguish them tirsdag 2. oktober 2012 @ 2012, Pierre Lison -