halef Documentation

ETS

Apr 02, 2018

Contents

1 OpenVXML Without Tears 1

2 Halef Setup Process 19

i

ii

CHAPTER 1

OpenVXML Without Tears1

Authors Vikram Ramanarayanan and Eugene Tsuprun (with inputs from the OpenVXML Setup and UserGuides)

1.1 Overview

In this manual, you will learn how to:

• Install Eclipse and the OpenVXML plugin for Eclipse

• Create your OpenVXML first project—a “Hello world” chatbot application

• Deploy the application

• Use halefBot (a text interface to Halef) to test out the application

• Use WebGGS to add logging and semantic categorization capabilities

1.1.1 Prerequisites

This tutorial assumes some familiarity with regular expressions, which we use to search for patterns in the users’responses (see RegexOne tutorial.)

1.2 Installing Eclipse and OpenVXML

Eclipse is an IDE (interactive development environment) that will allow you, among other things, to design callworkflows compatible with Halef.

1 The absence of tears is not guaranteed.

1

halef Documentation

1.2.1 Install Java Development Kit 8

JDK 8 is a prerequisite for Eclipse and OpenVXML. Download and install it.

1.2.2 Install Eclipse

Download and install Eclipse. Do not use the latest version—only this one.

1.2.3 Install the OpenVXML plug-in for Eclipse

Download OpenVXML 5.1. Here is the compiled OpenVXML binary

Unzip the file.

Start Eclipse.

From the menu, select Help → Install New Software. . . .

In the Install dialog that appears, in the “Work with:” field at the top, hit Add. . . .

Hit Local. . . and select the unzipped directory: ‘‘OpenVXML_5.1.0201506111143 ‘‘.

Make sure all the check boxes in the bottom part of the screen are unchecked (Show only the latest versions of availablesoftware, Group items by category, etc.)

Select All packages.

2 Chapter 1. OpenVXML Without Tears1

halef Documentation

On the Review Licenses screen, there aren’t actually any licenses to review. Just click I accept the terms. . . and thenFinish.

During the installation you’ll need to click OK once, to allow unsigned content to be installed.

When the installation is complete, restart Eclipse.

Now that you have installed OpenVXML, change your perspective (the windows and tabs you see in Eclipse). Fromthe menu, select Window → Open Perspective → Other. . . → OpenVXML.

1.3 Creating a Hello World Project

In this part of the tutorial, you will create and deploy a text-based chatbot that says “Hello World”.

1.3.1 Import the starter project

The Starter war file helps you get started designing your chatbot quickly.

1. Download the Starter war file: https://github.com/etsuprun/halefdocs/raw/master/Starter.war

1.3. Creating a Hello World Project 3

halef Documentation

2. Open Eclipse. Choose a new workspace.(Note: it is best practice to create a new workspace for each applicationyou are working on.)

3. Open the OpenVXML perspective: Window → Open Perspective → Other. . . and choose “OpenVXML”.

4. File → Import. . . Choose “General” -> “Existing Projects into Workspace” and click on “Next”. Then, selectthe button next to “Select archive file:”, hit Browse. . . and navigate to the location of the .war file you justdownloaded. In the drop-down menu with file extensions (which is located in the lower right corner of thedialog box on Windows), choose *.*. Then choose the .war file that you would like to import and select “Open”.

5. Hit “Finish” in the window that appears.

6. You may or may not get an error message. If you do, just restart Eclipse.

1.3.2 Set up a text-based Hello World canvas

Every OpenVXML project consists of two components:

• The Voice, which stores all your audio files (for speech applications)

• The Workflows, a visual representation of the chatbot application you want to design.

In Project Explorer, expand Deploy_Workflow and then Workflow Design. Open up Main Canvas.canvas. As the namesuggests, this is the canvas on which you will be drawing a flowchart-like representation of your chatbot. You shouldnow see something like this:

The Project Explorer (on the left) is where you should see the Voice and Workflow projects in your Eclipse workspace.

The Design Area (in the middle) shows the call flow of the currently selected canvas.

The Voice Pallet (on the right) displays the available blocks for the application.

The Starter warfile only has a Begin block (denoting the start of the callflow) and the Submit block (marking wherethe call ends).

Drag the block called PlayPrompt onto the canvas. PlayPrompt outputs something to your user–either a sound file orsome text. Let’s have it say “Hello world!”:

4 Chapter 1. OpenVXML Without Tears1

halef Documentation

1. Double-click the PlayPrompt on your canvas

2. Under Media, click on Not Configured.

3. Press Add Entry.

4. In Content Type, choose Text.

5. In the text box, type “Hello world!”

6. Hit OK three times.

Now that you’ve created the PlayPrompt, let’s make sure your application knows to play it when you start interactingwith the chatbot. To do this, click on the arrow in the lower right corner of the Begin block and drag and drop it ontothe PlayPrompt block. In the ensuing dialog box, select Continue and hit OK.

Now, let’s connect the PlayPrompt to the Submit block to mark the end of the application. Connect your PlayPromptto Return the same way: click the little arrow on the PlayPrompt, drag it to Submit, and hit OK.

Your canvas should now look like this:

1.3. Creating a Hello World Project 5

halef Documentation

1.4 Creating a Branching Application

Let’s now create a more complex callflow. In this section of the tutorial, you will build a text-based chatbot that will:

1. Ask the user if they like pizza

2. Save the user’s response into the database on our server

3. Categorize the user’s response into the semantic categories of “yes” or “no”

4. Follow up with an appropriate response (“Me too! I love pizza!” or “I’m sorry to hear you don’t like pizza.”)

5. Ask for clarification, if the original response was not understood

Our callflow will look like this:

6 Chapter 1. OpenVXML Without Tears1

halef Documentation

1.4.1 Configure the question block

In contrast to a PlayPrompt, which plays back a message for the user and expects no response, a Question block allowsus to prompt the user for a response. In this case, we’ll be asking if the user likes pizza.

Drag a Question block onto your canvas and connect the Begin block to it. Double click it to edit:

1. Set User Input Style to “S” and leave the drop-down at “Voice Only”. (Do this first.)

2. Set the Name for the question block. This name is arbitrary and will just help you identify the block on yourcanvas.

3. Set a Variable Name for the variable that will store the response for our question. Our convention is to start thevariable name with A_, for instance, A_do_you_like_pizza.

4. Double-click on Not Configured next to Prompt (in the Media tab). Press “Add Entry”. In the dialog box thatappears, set Content Type to Text. Type “Do you like pizza?” in the text area. That’s the text that will be shownto the test taker. Hit OK twice.

5. Double-click on Not Configured next to Voice Grammar. Choose “Grammar File” (from the dropdown menu)and then type ignore.wfst. This is the name of the language model Halef will be using when converting theuser’s speech input into text. Because we are building a text-based chatbot for now, we don’t need to customizea language model. We do, however, need to specify a value here, because Halef expects one.

Your question block should look like this:

1.4. Creating a Branching Application 7

halef Documentation

1.4.2 Create a Script block to handle semantic classification

Script blocks allow you to use the JavaScript language to manipulate variables, communicate with external services,and control the flow of the application.

In this application, we will use the script block to classify the response to the question “Do you like pizza?” into oneof two categories: yes or no. We will also send the user’s response to a back-end service, which will then store it intoa database.

Fortunately, you don’t need to know JavaScript to achieve the above goals. We’ve created a tool called WebGGS thatwill write the script for you.

Drag a Script block and connect the Question block to it.

Now, open WebGGS: https://videotelephony.halef-research.ets.org/webggs/

In WebGGs, define the variable name: A_do_you_like_pizza

8 Chapter 1. OpenVXML Without Tears1

halef Documentation

Add some regular expressions representing potential responses to the question “Do you like pizza?” and correspondingsemantic categories. Your WebGGS box should look like this:

Now, hit the right arrow to generate the script. You should the following output:

This script will do the following:

• Log A_do_you_like_pizza (the variable containing the response to the question block) to the server

• If the response contains the strings “yes” or “yeah”, set the variable SC_do_you_like_pizza to equal to“yes”.

• If the response contains the string “no”, set the variable SC_do_you_like_pizza to equal to “no”.

1.4.3 Create a Branch block

Drag a Branch block from the Voice pallet and connect the Script block to it. The Branch block is what allows to routethe application in accordance with the semantic category of the response to “Do you like pizza?”

We will build exit paths to deal with three types of responses:

• responses that fall into the category of “yes” (users who like pizza)

• responses that fall into the category of “no” (users who don’t like pizza)

• responses which did not fall into either category (i.e., what our script failed to categorize)

Open the Branch block and hit “Add Branch”.

Exit Path Name is an arbitrary name, but we recommend keeping it consistent with the name of the semantic category.Let’s first make a branch for the “yes” category. Set Exit Path Name to yes.

The Expression is a JavaScript statement that should return true or false. If the statement is true, the call willbe routed through this exit path. In this case, we will want to enter: Variables.SC_do_you_like_pizza =="yes"

1.4. Creating a Branching Application 9

halef Documentation

Now, make another exit path for the “no” category. The Exit Path Name should say no, and the Expression shouldread: Variables.SC_do_you_like_pizza == "no"

Your Branch properties should now look like this:

We do not need to define an Expression for the third exit path (neither “yes” nor “no”). This so-called Default pathwill be triggered if the JavaScript expressions for all the other semantic categories were computed to false.

1.4.4 Add the PlayPrompts

Depending on the category of the response, we will respond with a relevant PlayPrompt—”Me too! I love pizza!”,“I’m sorry to hear you don’t like pizza.”, or “Sorry, I don’t understand.”. Create three new PlayPrompts that say this.

1.4.5 Add the connectors

Draw an arrow from your Branch to “Me too! I love pizza!” In the ensuing dialog box, choose the “yes” exit path.

Now, draw an arrow from the Branch to “I’m sorry to hear you don’t like pizza.” Choose the “no” exit path.

Draw an arrow from the Branch to “I don’t understand” and choose the Default exit path. If the system doesn’tunderstand the response, let’s ask the user to repeat it. To do so, connect the “I don’t understand” PlayPrompt back tothe “Do you like pizza?” question to create a loop.

10 Chapter 1. OpenVXML Without Tears1

halef Documentation

Finally, connect the remaining two play prompts to a Submit block to mark the end of the application.

1.4.6 Save and export your project

We need to export the newly created project into a Web ARchive (WAR) application that can be served by the webserver as VoiceXML and then read by Halef’s Voice Browser. A voice browser browses voice/speech web pages (inthe VoiceXML format) much like Firefox or Chrome browse HTML pages.

1. Save the project: File → Save All

2. Go to File → Export . . .

3. Under Voice Tools, choose “Web Application”

4. Select “Archive file”.

5. Choose where you’d like to save the file. We recommend saving it in a git repository for better version control.

1.4.7 Test your application on halefBot

halefBot is the text-based interface to Halef.

Open up halefBot URL (ask your system administrator for the URL).

The Start URL is [your war file name without ‘.war’]/Deploy_Workflow/Begin.

For instance, if you called your war file “helloworld.war”, the Start URL is helloworld/Deploy_Workflow/Begin.

Open your project in halefBot and try a few different responses:

1.4. Creating a Branching Application 11

halef Documentation

You’ll note that the response in Russian (“ ”) was not in our grammar, and so it correctly went to the Default exit path.The other two responses (“yeah of course I love pizza!” and “not so much”) were successfully captured by our regularexpressions.

1.5 Adding Counters

In our pizza callflow, there is no set limit as to how many times the user will get to the “I don’t understand” dialogstate. As long as they keep saying something we don’t have in our semantic categories, the conversation can go onindefinitely.

We can use simple counter logic to set a limit. Let’s do this:

1. Initialize a counter and set it to 0.

2. Each time we hit the Default exit path (the response does not fit into the “yes” or “no” semantic categories),increment the counter by 1.

3. If the counter is greater than 2, exit the application. Else, keep asking the user if they like pizza.

12 Chapter 1. OpenVXML Without Tears1

halef Documentation

We’ll revise our callflow to look like this:

First, let’s initialize the counter. Open the Begin block and add a new variable calledcounter_do_you_like_pizza. Set Type to Decimal, and Value to 0.

Note: we strongly recommend keeping your counter variable names exactly consistent with the prompt variable names(A_do_you_like_pizza in this case).

1.5. Adding Counters 13

halef Documentation

In the first version of this callflow, our Default exit path was connected directly to the “I don’t understand” PlayPrompt.Now, we’ll want to change this logic so that before we end up at “I don’t understand”, we’ll first make sure that theuser hasn’t already been there twice.

Add another Script block to your canvas. If you’d like, give it a name like “increment counter”. Set the Default exitpath of the branch to go to this new script block. In your script block, add this JavaScript to increment the counter:

Variables.counter_do_you_like_pizza = Variables.counter_do_you_like_pizza + 1;

Now, after we run this code, we’ll want to decide whether the counter is greater than 2. To help with this, let’s use theDecision block from the Voice Pallet.

The Decision block helps us formulate a single JavaScript expression that evaluates to true or false. Selectcounter_do_you_like_pizza on the left, Greater Than (>) as the comparison operator, then Expression of 2on the right.

Just like a Branch block, a Decision block supports multiple exit paths. Connect the true exit path to anotherPlayPrompt that says something like: “Sorry you’re having trouble. Please try again later.” Connect the false exitpath to our “I don’t understand” PlayPrompt, which then is connected back to the original prompt.

Let’s save, deploy, and try this in halefBot:

14 Chapter 1. OpenVXML Without Tears1

halef Documentation

After the third time we hit the Default exit path, the counter was greater than 2, and so we got kicked out from theapplication.

1.6 Appendix A: Portals: Extending Workflows to Span Multiple Can-vases

Using multiple canvases is a great way to separate an application into more manageable pieces.

To add a new design canvas to a Workflow (and configure a portal between the new and existing canvas):

1. Right-click on the Workflow Design folder for Deploy_Workflow in the Project Explorer and select:

New → Other → Voice Tools Wizard -> Design Document. Hit “Next”. This will open the new designdocument wizard. Give the new canvas a name and hit “Finish”.

2. Enter a name for the new design canvas in the input box (for example, “SecondCanvas”. Note: This name mustbe unique amongst the existing design canvases in the application.

3. When you open up the new canvas, you’ll see a Portal Exit block on the new canvas.

1.6. Appendix A: Portals: Extending Workflows to Span Multiple Canvases 15

halef Documentation

4. Let’s set up a Portal Entry and connect it to the second canvas. Drag and drop Portal Entry from the Voice Palletonto the original canvas. Click on the new block and choose the Portal Exit to connect it to.

1.7 Appendix B: Creating an OpenVXML project from scratch

The easiest way to get started with an OpenVXML project is to follow the instructions above and import Starter.war,a starter project. But if you really want to create your project from scratch using the Eclipse plugin, here are theinstructions.

First, let’s create a new voice project: File → New → Project → Voice Tool Wizards → Voice

Call the voice project HelloWorld_Voice.

You should now see the voice project in your Project Explorer pane:

Now, create the interactive workflow: File → New → Project → Voice Tool Wizards → Interactive Workflow

The application name for this and all Halef workflows should be Deploy_Workflow. Hit Next.

In the Branding dialog box, leave Brands at Default. Hit Next.

On the Interaction Type Support dialog box, leave Voice Interaction checked. Hit Next.

On the Language Support dialog box, associate your workflow with your voice by clicking Not Configured and choos-ing HelloWorld_Voice:

16 Chapter 1. OpenVXML Without Tears1

halef Documentation

You have now created your voice and your workflow, and you have associated the two. You should see both Hel-loWorld_Voice and Deploy_Workflow in the Project Explorer pane.

1.8 Appendix C: Troubleshooting

Symptom: A Question dialog state does not respond to speech input Try this:

1. Look in the Tomcat log to see if there was any error

2. If the log indicates instead that the last action was fetching the grammar file: a. Ensure that the grammar file inthe war has Linux line endings: Export it, remove it from the war, check the line endings in a text editor, and re-import to the war. b. Ensure that the grammar element in the vxml shown in the Tomcat log has mode=”voice”and there is no grammar element with mode=”dtmf”.

• If the vxml has a problem:

(a) Import the war into OpenVxml and open the Question node

(b) Change Settings > User Input Style = Hybrid

(c) If Media > Dtmf Grammar shows anything other than “Not Configured”, then double-click thedisplayed value and set Grammar File to an empty string.

(d) If Media > Voice Grammar shows “Not Configured”, then double-click “Not Configured”, setthe dropdown to Grammar File, and type the filename of the grammar file (including the fileextension) into the textbox.

(e) Change Settings > User Input Style = “Voice Only” with the S shown in blue and the D shown ingray.

1.8. Appendix C: Troubleshooting 17

halef Documentation

(f) Save and then Export.

18 Chapter 1. OpenVXML Without Tears1

CHAPTER 2

Halef Setup Process

How to setup Kaldi Recognizer and Test Recognition Model

How to setup Cairo Speech Server

How to setup JvoiceXML Voice Browser Application

2.1 How to setup Kaldi Recognizer and Test Recognition Model

1. Make sure that you have latest Ubuntu updates

sudo apt-get update

sudo apt-get dist-upgrade

sudo apt-get upgrade

2. Install libraries required for Kaldi recognizer

sudo apt-get install zlib1g-dev

sudo apt-get install libatlas3-base

sudo apt-get install libwebsockets-dev

sudo apt-get install libssl-dev

3. Create Kaldi directory, clone custom Kaldi repository and run build script

mkdir -p /export/Apps/KALDI/

cd /export/Apps/KALDI/

git clone git://git.code.sf.net/p/halef/cassandra halef-cassandra

cd /export/Apps/KALDI/halef-cassandra

bash ./BUILD_KALDI.sh

4. Copy recognition models and binaries to destination directories

19

halef Documentation

mkdir -p /export/Apps/KALDI/2015-12-21/deployment/bin

cd /export/Apps/KALDI/2015-12-21/deployment

cp /export/Apps/KALDI/2015-12-21/halef-cassandra/kaldi-trunk/src/online2bin/STRM-ASR-server/export/Apps/KALDI/2015-12-21/deployment/bin

chown -R <your_login>:<your_group> /export/Apps/KALDI/2015-12-21

5. Copy recognition models and startup scripts

cd /export/Apps/KALDI/2015-12-21/deployment/

tar xzvf models.tar.gz

tar xzvf 9404.tar.gz

6. Start Kaldi instance

cd /export/Apps/KALDI/2015-12-21/deployment/srv/9404

./server.sh &

2.2 How to setup Cairo speech server

To setup Cairo instances a convenient setup script has been created.

1. Download setup script and Kaldi settings files by opening the following links and clicking “Download this file”link on top of each page.

https://sourceforge.net/p/halef/halef-cairo/ci/master/tree/cairo-SETUP/SETUP.sh

https://sourceforge.net/p/halef/halef-cairo/ci/master/tree/cairo-SETUP/SETUP.kaldi.properties

2. Copy downloaded SETUP.* files into /export/Apps directory on the server and edit them. You will see di-recory settings and database connection settings in the top part of SETUP.sh file which you need to correct.SETUP.kaldi.properties should contain at least 1 entry defining item name, server ip and port on which Kaldi islistening. For example: default=localhost:9404

3. Make sure that you have ~2GB of free space in /export/Apps and server can connect to the Internet for Cairodownload.

4. To install Cairo with Java, please run setup in unattended mode:

./SETUP.sh –quiet

Cairo installation can take considerable time depending on your Internet connection speed.

5. You can install and run Cairo as service by running CairoService script (please review CairoService file contentfirst to check if all directories are correct there):

cd /export/Apps/Cairo/cairo-VM/scripts

sudo CairoService install

sudo service cairo start

2.3 How to setup JvoiceXML Voice Browser Application

1. For JvoiceXML we recommend using latest version directly from JvoiceXML developer repository (https://github.com/JVoiceXML/JVoiceXML) :

20 Chapter 2. Halef Setup Process

halef Documentation

mkdirs -p /export/Apps/JVXML

cd /export/Apps/JVXML

git clone https://github.com/JVoiceXML/JVoiceXML.git

2. Configure JvoiceXML by copying DB logging file from Cairo:

cp /export/Apps/Cairo/cairo-VM/config/halef-logging-db.xml /export/Apps/JVXML/JvoiceXML/main/

and edit mrcpv2 settings to point to Cairo server by replacing IP for cairoAddress and cairoHost in the file:

/export/Apps/JVXML/JvoiceXML/org.jvoicexml.implementation.mrcpv2/build.gradle

3. To compile and run JvoiceXML you need Gradle version 3 or higher. Because system apt-get will install onlyversion 2 it is recommended to download Gradle directly from:

https://services.gradle.org/distributions/gradle-3.4.1-bin.zip

and symply unpack it into /export/Apps

4. You can build and run JvoiceXML using the following commands (or put them into shell script):

cd /export/Apps/JVXML/JvoiceXML/main/

export JAVA_HOME=/export/Apps/jdk

export PATH=/export/Apps/jdk/bin/java:/export/Apps/gradle-3.4.1/bin:$PATH

nohup gradle run

2.3. How to setup JvoiceXML Voice Browser Application 21