1

To Example or Not To Example, That is the Natural Language Question

Frederick Parkinson, PhD, Project Manager, User Interface DesignerDeborah Rapsinski, Senior User Interface Designer

2

Introduction

• This paper presents best practices around the prompting of natural language recognition states– Structure of prompt– Use of examples

3

What we will talk about

• Why we need best practices around Natural Language (NL)?

• What makes designing for NL challenging?

• What makes for a good NL response?

• The Nuance best practice for NL

• Evidence to support this strategy

4

Why we need best practices for NL

• NL deployments are becoming more common

• NL deployments incur additional cost, so the demand for quick ROI is more acute

• Best practices allow adopters to achieve ROI goals more quickly by allowing designers to leverage “lessons learned” from previous deployments

5

Designing for NL is challenging

• Directed Dialog versus Natural Language

• Why NL is appealing

• The Natural Language Paradox– Natural Language states are unnatural– While callers are encouraged to respond naturally, the fact that

they are interacting with a computer makes the interaction unnatural

6

How Natural Language systems work

• Two tasks:– recognition of words in the utterance– interpreting the meaning of those words

• How Natural Language systems are developed– trained against corpora of tens of thousands of utterances– training data is tagged by hand by humans

• To work well, the recognizer has to pick out a few salient words that map to specific caller intentions

7

What makes a good Natural Language response

• A few meaningful words– four to ten words long– one or two meaningful words

• Good examples of responses– “I need to locate a pharmacy,” “I want to order a refill,”

“my power is out,” “close account”

• Bad examples of responses– “I need help,” “inquiry,”

“I need to check about amounts of several drugs that I already have when it's time to order new ones I don't want to order more than I can use”

8

The Nuance best practice

• Provide the caller with examples– pro: addresses the “deer in the headlights” some callers face– con: humans don’t offer examples

• Examples provide just enough structure to help callers formulate their responses

• “In a few words, please tell me why you’re calling today. You can say things like: ‘I need my account balance,’ or ‘what’s the status of my order?’ Okay, go ahead.”

9

What makes a good example?

Shows callers how to respond 1 statement, 1 question

Succinct and Unambiguous Fewest number of turns todestination

Makes use of caller profile If past due:“I’d like to make

payment arrangements.”

10

• Rotate examples to teach callers of what can be handled – Offer 2 examples at the initial prompt

• Reduce to 1 for re-prompting or looping through NL state

• Balance caller goals with business goals– Use examples that will, e.g. encourage callers to pay their bill or

inform them of new product offerings

Best Practice, Continued

11

Evidence

• Usability Testing– Inferred success through completed tasks– Post-test interviews

• Experiments– No example, 2 statements, 2 questions, 1 statement and 1

question…

• Deployment Data– Fluency, delays– Number of words– Value of responses

12

Evidence: Number of Words Used

< 4 Words Between 4 & 10 > 20 Words

No ExampleExample Used

13

Evidence: Fluency

Disfluency Slight Delay Long Delay

No ExampleExample Used

14

Conclusion

• We’ve presented the Nuance best practice for the structure of the Natural Language question– Include examples– Examples include one statement, one question– Examples balance callers’ goals and business goals

• We’ve presented evidence in support of this best practice that shows that callers prefer hearing examples and that examples improve the quality of callers’ responses to the Natural Language question