1 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Interactive Knowledge Capture for Problem-Solving Systems Jim Blythe Yolanda Gil Jihie

1USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture

Interactive Knowledge Capturefor Problem-Solving Systems

Jim Blythe

Yolanda Gil

Jihie Kim

www.isi.edu/ikcap


Outline

Main elements of our approach

Experiences and lessons from earlier work: monolithic systems

New work on the Calo project:

open systems


Acquiring procedure knowledge from users

Each user can have unique requirements of a system Their requirements will change over time, perhaps

frequently In travel planning, may be different for each trip

Users need to be able to modify procedure representations in intelligent systems to address their needs

Complements example-based learning


Example: adding procedure knowledge to a travel planning tool

An assessment tool makes judgments about travel itineraries: e.g., the airline should be United or American e.g., the hotel should be within walking distance, unless I am

renting a car

Need to add procedure knowledge to tell the system to make a new kind of judgment:

“the hotel can cost up to 20% more than the government per diem rate for the city.”

or supporting procedures:

“to estimate driving time, divide the distance by 55”


Expect’s Support for procedure KA:Key Technologies

Where does the user start? An acquisition wizard guides the user to start the KA process through

a dialog, based on problem-solving methods. KA takes many steps; users will be lost…

The acquisition wizard manages the process from end to end. Users don’t know the computer language.

An English-based procedure editor– Users modify the English paraphrase of the formal representation.

A search-based expression composer– Suggests valid reformulations of user sentences.

How do users ensure all the needed information is added? An interdependency analyzer understands which pieces of knowledge

are used to solve a problem.



Meta-reasoning: About how knowledge fragments are combined to

solve a goal About missing information, effects of changes to KB About the dialog and context

Putting knowledge in the user’s terms: Generate text descriptions to hide the syntax Browse-and-replace interface for procedures Search-based reformulations of free text Test knowledge on real examples


Outline


Experiences and lessons from earlier work: monolithic systems using Expect


open systems


Acquisition wizard

Dialog with user to start the process.

Some questions use menus or text input.

Others use the English editor to refine procedural knowledge


English-based Procedure editor

NL description of method

Alternatives forselected text fragment

(multiply

(obj (look-up

(obj fsa-per-diem-hotel-rate)

(for (r-city ?hotel))))

(by 1.2))


Test new knowledge immediately

Each element is defined and checked

using constraints

Softools, developed under the DARPA AcT program


U: new port: HavanaS: I need to know if it is an airport or a seaportU: seaportS: I need to know the location and the berths

...

...

r-location

r-berths

r-pols

r-piers

r-storage-area

(evaluate (obj coa) (wrt logistics))

... (r-location port)

... (r-berths seaport)

port

airport seaport

inlandwaterwayseaport

maritimeseaport

Domain Ontology Problem-Solving Methods

Global Procedure Analysis:Test interdependencies, both rule-rule and rule-data

INTERDEPENDENCIES

Interdependencies guide Knowledge Acquisition


When combined, the tools use information from each other

Application

Acquisitionwizard

Acquisitionanalyzer

Interdependencyanalyzer

Procedure editor

Relation/concept editor

Instance editor

Expression composer


The acquisition wizard

Guides the user through the initial steps of adding new knowledge.

Structures the knowledge to be added using default procedural knowledge.

Questions are generated from a problem-solving theory.


Problem-solving theory for plan evaluation

A hierarchy of generic types of plan judgments with default procedural knowledge.

DEFINED: check that the value is less than the maximum value

ASK USER: compute a maximum value for each object

judgment

global judgment local judgment bounds check extensional check

completeness judgment

hotel cost judgment

upper bound lower bound positive negative

“Warn if the value is too large?”


Benefits of integration: The acquisition wizard and the method editor

Each component receives information from the other that helps the user:

The wizard provides to the editor: An initial version of the method, with the correct capability An expectation of the result type of the method

The editor provides to the wizard: A more detailed method result type Used to help classify the new task in the ontology


bounds check

upper bound lower bound

“Warn if the value is too large?”


The expression composer

Assists users in formalizing informal statements by suggesting composite expressions using terms the system understands.

Uses an ontology of terms and relations, and synonyms derived from WordNet with extensions.

Makes breadth-first forward search, matching keywords


Expression composer example

User types: “max staging post landing”

Tool suggests: “find the maximum of the landing distance available of the runways of the forward staging post”.

find

object of

landing-distance-available

runways

?forward-staging-post

maximum

Function call

reformulation

Typed variablecombining queries

Information element


Integrated with the method editor

Constructs compound terms in KB that include user terms and have desired type

Anytime breadth-first search through space of terms


Outline


Experiences and lessons from earlier work: monolithic systems


open systems


Tailor: interactive task acquisition forthe Calo office assistant

A personal assistant that manages everyday tasks. Learns from experience, interacts with the user

naturally through several channels. Large DARPA-funded project run by SRI.

Goal2ACT8

sleeping

Fact1ACT2normal

Goal3ACT3

sleeping

Intention Graph

Cue: (TEST (source vision new))

ACT2

Process Library

Process Execution

(source commanders-vision new)

(ACHIEVE (satisfy Incoming-INs))

New Facts & Goals

ExternalWorld

1

2

3

4

5

6

7

8

Cue: (ACHIEVE (satisfy source))

ACT1Facts&

Goals

(ACHIEVE (derive IN-list source))

ACT1current

Spark is Calo’s task manager.

(A PRS-based system developed by Myers et al.)


Approach and challenges in acquiring knowledge within CALO

Carry key elements forward: Explicit reasoning about knowledge dependencies

and acquisition context Expressing the procedures in the user’s terms

Earlier work used Expect, a monolithic system using custom problem-solving system, language, methods, ontology,..

Now use Spark’s performance element and procedures, ontologies provided by several groups.


Example: purchasing a laptop

Spark manages the workflow as the user purchases a laptop. Choose a model, find bids, get authorization, track

purchase…

During the process, Spark finds that the order cannot be completed, because a manager who must authorize the purchase is not available.

User should be able to tell the system “You don’t need authorization when the cost is less than $2000”


Initial solution

Make direct analysis of the Spark procedures Encapsulates a model of Spark’s behaviour Requires additions to language, e.g. types, purpose of

subtasks

Combine the procedure analysis with the expression composer to help interpret user sentences.

Dialog is currently implicit Currently provide follow-up questions based on analysis.


Expression composer suggests valid

condition

User can explore modifications based

on different assumptions

Response to“You don’t need authorization when the cost is less than $2000”

Automatic generation of text from Spark

procedure definitions


Tailor: summary and future work

Currently allow users to modify existing procedures, using global analysis and expression composer.

Next:

Support defining new tasks, integrate with advice for Spark.

Model dialog: build templates for supporting theory, working with Allen/Ferguson on dialog model.


Summary

Our work allows users to add procedure knowledge to both custom and pre-existing intelligent systems.

Combination of global analysis and term reformulation can allow users and the system to reach shared understanding.

Interactive knowledge capture complements example-based learning to allow intelligent systems to adapt.

Evaluation strategies: Ablation user studies with fixed and free tasks. Good results with Expect & Constable, still to do with Tailor.

Documents

1 USC INFORMATION SCIENCES INSTITUTE Interactive Knowledge Capture Interactive Knowledge Capture for Problem-Solving Systems Jim Blythe Yolanda Gil Jihie