16th IEEE International Conference on Robot & Human Interactive CommunicationAugust 26 - 29, 2007 / Jeju, Korea

A Spoken Dialogue System for Electronic Program GuideInformation Access

Seokhwan Kim, Cheongjae Lee, Sangkeun Jung, and Gary Geunbae Lee

Department of Computer Science and EngineeringPohang University of Science and Technology, South Korea

e-mail:(megaup, lcj80, hugman, gblee)(postech.ac.kr

Abstract-In this paper, we present POSTECH Spoken Di-alogue System for Electronic Program Guide InformationAccess (POSSDS-EPG). POSSDS-EPG consists of automaticspeech recognizer, spoken language understanding, dialoguemanager, system utterance generator, text-to-speech synthe-sizer, and EPG database manager. Each module is designed andimplemented to make an effective and practical spoken dialoguesystem. In particular, in order to reflect the up-to-date EPGinformation which is updated frequently and periodically, weapplied a web-mining technology to the EPG database manager,which builds the content database based on automatically ex-tracted information from popular EPG websites. The automat-ically generated content database is used by other modules inthe system for building their own resources. Evaluations showthat our system performs EPG access task in high performanceand can be managed with low cost.

I. INTRODUCTION

During the past few years, digital broadcasting serviceshave been widely used. One ofthe main advantages of digitalbroadcasting services is that the users can get more advancedinformation on broadcasting contents. The Electronic Pro-gram Guide (EPG) is the most widely used application ofdigital broadcasting services, and it provides detailed infor-mation on TV programs, such as program titles, channelnames, genres, actors, program start times and so on. How-ever the current EPG systems are still difficult to use forobtaining some information, even though they aim to provideuser-friendly services. It is mainly due to the limitation offormer interfaces, such as the remote controller which is usedas the main input device in most ofEPG systems.

In order to make an EPG system user friendly, we de-veloped POSSDS-EPG, a spoken dialogue system for EPGinformation access. In the past few years, spoken dialoguesystems have been used in various applications for naturaland convenient interface with users [1], and recently, theinterest in spoken dialogue systems has been sharply in-creasing. There have also been some studies on spoken lan-guage dialogue interfaces for operating a TV and retrievinginformation on TV programs [2].

POSSDS-EPG consists of several sub-modules which aregeneral in most of spoken dialogue systems for other domains,such as automatic speech recognizing module (ASR), spokenlanguage understanding module (SLU), dialogue managingmodule (DM), system utterance generating module (SUG)and text-to-speech synthesizing module (TTS). Besides thesedomain independent modules, we incorporated the EPG da-

tabase managing module by using the web mining metho-dology. This is a necessary part of the system due to thecharacteristic of EPG data of being updated frequently andperiodically. In order to constantly maintain up-to-date data,we developed a web mining module which automaticallyextracts data for building EPG database from websites deal-ing with information on TV programs. By using the automaticweb mining module, we can reduce the cost and effort forconsistently maintaining EPG database. In this paper, wedescribe the major methodologies behind POSSDS-EPG.

II. POSSDS-EPG: POSTECH SPOKEN DIALOGUESYSTEM FOR EPG DOMAIN

POSSDS-EPG consists of a set of appropriate modulesthat are designed to be connected to each other according tothe order. An overview of the system is shown in Fig 1. Theoverall system aims to output the synthesized spoken re-sponse corresponding to an input utterance spoken by theuser.

In order to handle the spoken input utterance, the ASRmodule is operated first, which recognizes the user utterance.The recognized result is used for the input of the spokenlanguage understanding module. The SLU module extractssemantic concepts from the user utterance and constructs thepre-defined semantic frame by using extracted results. Thedialogue manager generates system responses with the se-mantic frame and the discourse history. The discourse historyis a set of semantic frames in one dialogue session. The resultof the dialogue manager is represented by correspondingsystem action tags selected from the pre-defined system ac-

Fig. 1 Overview ofPOSSDS-EPG System Architecture

978-1 -4244-1635-6/07/$25.00 ©2007 IEEE.

MP-13

178

tion tag set. Each system action tag is defined by consideringthe expected response of the system at each situation of thedialogue. The manner of the system response is determinedby considering the system action tag in the system utterancegenerating module. The system utterance generating moduleproduces literal system utterances by assembling retrievedrecords from the EPG database with system utterance tem-plates which are determined by given system action tags. Thefinal result of the overall system is produced by the TTSmodule, which synthesizes the spoken utterance from theliteral utterance.

Each module in POSSDS-EPG is based on do-main-independent methodologies, with the consequence thatit can be easily applied to other domains. To build a spokendialogue system for some domain, it is essential to constructthe contents database for the domain, since all the modulesneed to take respective models and rules for the given domain,and they can be built from the contents database on the do-main. In case of the EPG domain, we not only have to con-sider constructing the contents database initially, but alsomaintaining it continuously, due to the characteristic of theEPG contents of being updated frequently and periodically.Maintaining the contents database manually is time con-suming, labor intensive, and expensive. To reduce the cost ofthe task, we developed the EPG database managing modulebased on the web mining technologies. It automatically ex-tracts the up-to-date information from websites which pro-vide information on TV programs. Extracted data are storedand managed in the form of the schema of the content data-base which can be directly used in other modules.

In following subsections, we describe detailed propertiesand methodologies of each module.

A. Automatic Speech Recognizer

The speech recognizer in POSSDS-EPG was developedbased on HTK (Hidden Markov Model Toolkit). The re-cognizer uses a pre-trained dialogue acoustic model andadopts the EPG domain specific language model. To build thelanguage model, the candidate utterances that have highprobability of being spoken by users are required. We gen-erate the candidate utterances automatically by using the di-alogue examples in the existing example database and theretrieved result from the up-to-date EPG database. Table 1shows an example of the automatically generated candidate

Table. 1 Examples of the automatically generated candidate utterances

utterances. Each dialogue example record in the exampledatabase not only has the utterance itself, but also the taggedinformation on the named entities contained in the utterance.The candidate utterances are generated by replacing eachnamed entity in the existing utterance with the correspondingentity in the retrieved results. The candidate utterances areused for building the up-to-date language model for thespeech recognizer.

B. Spoken Language Understanding

The SLU module of POSSDS-EPG was constructed by aconcept spotting approach which aims to extract only theessential information for predefined meaning representationslots [3]. The semantic frame is made up of these slots in-cluding dialogue act, main action, and component slots forthe EPG domain. An example of the semantic frame for theEPG domain is shown in Table 2.

Each slot value is selected from the corresponding prede-fined tag set shown in Table 3. While the dialog act tagrepresents the grammatical function ofthe utterance, the mainaction tag mainly comprises the semantic function of it. Dueto the characteristics ofthe dialog act and the main action tag,each dialog act tag deals with domain-independent concepts,while each of the main action tag is defined for the do-main-specific task ofthe dialog manager for the EPG domain.The component slots are used for representing named entitiesin the utterance.We regarded the SLU problem as a classification problem,

Table.2 An example of the semantic frame

7 2SBS 'a-i -[TIo 1p1?User Utterance (Geu-leom SBS deu-la-ma-neun eon-je ha-ji?)

Then, when do the SBS dramas start?

Dialog Act When-question

Main Action Search_start_time

Component Slots [channel = SBS, genre = 4fp[]

Table.3 The list of the predefined slots in the semantic frame

for the EPG domain

Dialog act

Wh_Question Yes_No-Question Accept

Reject Statement Request

Express Conventional_Open Conventional-CloseMain Action

SearchProgram Search-channel Search-daySearch_starttime Search endtime Search_currenttime

Search_currentdate Move channel Alarm

Record TV-on TV-offComponent slot

Genre Channel Date

Start-time End-time Cast

Day Program

An Existing Utterance

I want to watch drama Hae-Sin around 9 pm.

[genre = drama], [program-name = Hae-Sin], [time = 9 pm]

Retrieved Results

[genre = movie], [program_name= Monster], [time = 11 pm][genre = sports], [program-name= Basketball], [time = 7 pm]

Candidate Utterances

I want to watch movie Monster around 11 pm.I want to watch sports Basketball around 7 pm.

179

which can be solved by statistical machine learning frame-works. To build a statistical model for the SLU problem, weshould prepare the training corpus containing utterances thathave high probability ofbeing spoken by users. We can easilycreate a training corpus by reusing the candidate utterancesthat are used for building the language model in the speechrecognizer, which is referenced in the previous subsection.

C. Dialogue Manager

To develop an effective and practical spoken dialoguesystem, we proposed the situation-based dialogue manage-ment method using dialogue examples in our previous work[4]. Fig 2 illustrates an overall strategy of the example-baseddialogue modeling. For the system utterance generation, weautomatically construct and index a dialogue example data-base from the dialogue corpus. The dialogue manager re-trieves the best dialogue example for the current dialoguesituation, which includes a current user utterance, semanticframe and discourse history. From the retrieved result, thedialogue manager determines the system action tag from thepre-defined tag set shown in the Table 4.

D. System Utterance Generator

The system utterance generator generates the literal sys-tem utterances based on the system action tag and the utter-ance generating template. Each system action tag has at leastone utterance generating template which is constructed ma-nually. The system utterance generating task is advanced byfilling slots in the template with proper values, such as re-trieving results from the EPG database, slot values in thesemantic frame, and constituents in the discourse history.Table 5 shows an example of this task.

DialogueUser's Utterance Crpus

UserSemantc Discourse AutomaticTie-breakingf f tIndexing

abe4h lst otnh preetie systematonagSystem Actil-on<Rsalttonse Sayer Seleteertina pD

Informchannl Inorpoga Ifrm castl|

Inform 4tim e lsInormhepositfived Informacinegtive

Specify Confirm Finish

Table.5 An example of the system utterance generation

System Action Tag Inform Channel

[program name]° [channel]OIlk1 >FL[kUtterance Template ( [program name] eun [channel] e-seo hap-ni-da)

[program name] is broadcasted on [channel].

Slot Values [program name =o t!, channel = KBS]

to d-° KBS OI k1 L-1 rc [System Utterance (Hae-Sin eun KBS e-seo hap-ni-da)

Hae-Sin is broadcasted on KBS.

E. EPG Database Manager

The main purpose ofthe EPG database manager is to builda content database for the other modules in POSSDS-EPGwith minimal human effort. Fortunately, there are severalwebsites which provide the information on TV programs, andwe can obtain well-formed data for EPG from these websites.We chose an EPG website (http://www.epg.co.kr) dealingwith the information on Korean TV programs. The EPG da-tabase manager builds a contents database from the infor-mation on the website following the procedure shown in Fig 3.Firstly, unnecessary parts of the web page are filtered out onthe content filtering step. Since we are mainly concerned onthe information in the form of the table, several tables aredistilled through the first step. Then, the EPG databasemanager extracts the needed information from the tables re-ferring to the structure and the header information of thegiven table. The extracted data are stored in content databasefor the POSSDS-EPG on the last step. All the steps in theEPG database manager are currently implemented by usingthe website specific wrapper rules.

III. EVALUATIONS AND IMPLEMENTATION

A. Automatic Speech Recognizer

In order to evaluate the effectiveness of the automaticallymanaged spoken dialogue system, we measured the user sa-tisfaction of the system and compared it with our previoussystem which is manually tuned for the fixed EPG database.The user satisfaction was defined with the linear interpolationof three different measures: user perception of Task Com-pletion Rate (TCR), Mean Recognition Accuracy (MRA),and Success Turn Rate (STR) instead of the elapsed time.

Jr

wl

wl

ultulllg h.

Fig.3 A procedur ofweb-mning-baseEPGcontntdatabaEPG DB

Fig.3 A procedure of web-mining-based EPG content database building

180

Each was weighted by a factor of 1/3, so that the maximumvalue of the user satisfaction is one. We asked each of 5 testvolunteers to perform 5 different EPG tasks. The volunteersevaluated every system's response in each dialogue turn.

As we can see in Table 6, the user satisfaction of the newsystem which is automatically managed was 0.73 for spokeninput, while the previous system achieved a user satisfactionof 0.75 for spoken input. It means that we can automaticallymanage the POSSDS-EPG, which has a close performance tothe manually built system with much less human effort.

B. Implementation

We implemented POSSDS-EPG to be separated into threeunits ofprograms including the engine part, the graphical userinterface (GUI) part, and the information feeding part. Theengine part contains the above-mentioned modules except tothe EPG database manager. The engine part was developedby using standard C++ library and can run under both Linuxand Windows platform, but the GUI part of the system canonly be operated under the Windows platform, because it isimplemented by using Microsoft Visual Studio's MFC library.Fig 4 shows a screen shot of the POSSDM-EPG, which isdisplayed through the GUI part ofthe system on the Windowsplatform. While the engine part and the GUI part are operatedtightly coupled, the information feeding part is independentfrom the other parts of the system. The information feedingpart which incorporates the EPG database manager moduleruns at offline to automatically build a content database.

IV. CONCLUSION

This paper describes the POSSDS-EPG, a spoken dialo-gue system for EPG information access. To make an effectiveand practical spoken dialogue system, we incorporatedsuitable methodologies for each module, such as ASR, SLU,DM, and SUG. In addition to these modules, we implementedthe EPG database manager which automatically builds thecontent database for the other modules in the system by usingthe web-mining methodologies. The automatically generatedcontent database is used by each module to build its ownresources. The evaluation result demonstrates that the EPG

Table.6 The comparison of the dialogue performances between the ma-

nually built EPG database and the automatically built one

JUCK U.I7 U.0l

STR 0.65 0.62

MRA 0.85 0.85

User Satisfaction 0.75 0.73

TCR: User Perception of Task Completion Rate

STR: Success Turn Rate

MRA: Mean Recognition AccuracyUser Satisfaction = aTCR + bSTR + rMRA

Fig.4 A screen shot of the POSSDM-EPG operated

on the Windows platform

database manager can dramatically reduce the cost of con-structing and maintaining our system. Currently the EPGdatabase manager was implemented using site-specificwrapper rules, but we will reform the system to be operatedwithout regarding the type of the target website in order tocollect wider information from more diverse websites.

V. ACKNOWLEDGEMENT

This research was supported by the Intelligent RoboticsDevelopment Program, one of the 21st Century FrontierR&D Programs funded by the Ministry of Commerce, In-dustry and Energy of Korea, and also supported by the Invi-tation for Proposals on Internet Services Theme funded byMicrosoft Research Asia.

VI. REFERENCES

[1] J. Allen, D. Byron, M. Dzikovska, G. Ferguson, L. Galescu, and A.Stent, "Toward conversational human-computer interaction," AlMagazine, vol. 22, no. 4, 2001, pp 27-37,.

[2] J. Goto, M. Miyazaki, and T. Kobayakawa, "A TV Agent System thatIntegrates Knowledge and Answers Users' Questions" Proceedings ofthe 2006 International Conference on Intelligent User Interfaces, pp300-302, 2006.

[3] J. Eun, C. Lee, and G. Lee, "An Information extraction approach forspoken language understanding" in Proceedings of the 8th Interna-tional Conference on Spoken Language Processing, 2004, pp.2145-2148.

[4] C. Lee, S. Jung, J. Eun, M. Jeong and G. Lee, "A Situation-basedDialogue Management Using Dialogue Examples" in Proceedings ofthe international conference on acoustics, speech and signalprocessing 2006, vol. 1, 2006, pp. 69-72.

181