Intelligent User Interfaces Frank Shipman Department of Computer Science Texas A&M University...

Intelligent User Interfaces

Frank Shipman

Department of Computer Science

Texas A&M University

E-mail: shipman@cs.tamu.edu

What this is about

Designing, building, and evaluating intelligent user interfaces

Particular technologies used to create intelligent user interfaces

Issues concerning applicability of intelligent user interfaces

Intelligent user interfaces (IUI)?

Systems that provide interactive support based on embedded AI mechanisms

Interfaces to AI functionality and knowledge representations

Adaptive systems Cooperative problem-solving systems

Technologies I

User models,Situation models, and Programming by Demonstration

Adaptive Interfaces

Requirements:– interface that can be adapted– user or situation model– adaptation strategy

Frequently used for providing assistance or training to user

User Model

“any information which a program has which is specific to a particular user.

The information itself could range from a simple count of errors, to some complicated data structure which purports to represent relevant parts of the user’s knowledge of the problem domain.”

Stereotyping vs. Individual

Stereotyping (canonical user modeling)– provide interfaces for classes of users– classes might be based on skill (novices,

midrange users, experts) or role Individual approach

– dynamically adapt to suit each user– can be based on observed use of system

or self assessment

Representations for User Models

Descriptive method– modeling the user’s observed behavior– describes what system has seen user do

Skill-based or cognitive method– attempt to model the internal cognitive

models and processes of the user– represents background knowledge, goals,

plans, preferences, misconceptions

Acquisition of User Model

Model based on a combination of:– Observations of system use

• statistical history• chronological history

– Self-assessment– Testing

How can model evolve over time?– Any of the above

Berkeley UNIX Consultant

Goal: Provide help to new UNIX users

Generates user model based on “successful” use of UNIX commands– Explanations of difficult commands can

make use of student’s knowledge.

Intelligent Tutoring Systems

Task is generally well-known– assignments given to student by system– systems track partial completion

Systems keep record of student’s success and failure.– used to determine future assignments– used to determine how to help when

student has difficulties

Situation Models

Components of situation:users, system, environment

Users– multiple user models

System– hardware constraints and load– device / resource availability

Representing the Environment

Identifying environmental influences– anticipating use situations– classes of use vs. detailed model of

environment

Monitoring environment– direct input devices– user description

Mars Medical Assistant

Goal: Provide medical support for astronauts on three year trip to Mars

Consider educational, consultation, and emergency situations

Models of user and patient– limited highly-trained user community– no new users joining during mission

Other Adaptive Systems

Typing completion– suggests completions for partial terms based

on prior use Emacs suggestions

– notifies user when more efficient method available to complete task

Computer Chess Game– determines quality of own play based on

perceived level of opponent

Programming by Demonstration

Generalizing from demonstrated action and situation sequences to programs

Difficulties:– knowing what must stay the same– knowing what are variables and their types– connecting to programmed application

code

Programming by Demonstration Systems Peridot -- demonstration of simple

interface Marquise -- demonstration of graphical

editors including palettes and modes DEMO -- demonstrating dynamically

created objects DEMO2 -- refinement by system based

on multiple demonstrations

Pavlov

Focus on programming animation Includes:

– graphical objects– models of motion and time

Stimulus-response demonstration– modes for creating objects and behaviors– mode for demonstrating interaction

Technologies II

Presentation generation,Design Environments, and Interface agents

Presentation Generation

Generating dynamic links to information – enabling user-controlled flow

Generating presentations based on current situation and/or user– Use of user or situation model

Generating rhetorical structure/transition– Scripting events– Media-based decisions

Presentations and Explanations

Examples:– Explainer (Redmiles)– Explainable Expert System (Moore)– Story Presentation System (Sgouros, …)

Explainer

Domain: Graphical program explanation for software reuse

Creates links between perspectives on software including source code, documentation, execution information, application domain view

Provides user multiple points of access to better inform about software

Explainable Expert System (EES) Explains different outcomes in an expert

system / planner Generates natural language to answer

user’s questions Keeps dialog history to provide

differential descriptions

EES Architecture for Explanation

Query analyzer Text planner Sentence generator

User goals Dialog history Focus Information

Knowledge Base Plan OperatorsUserQuestion Response

EES Example

User: “Describe Inderal”

System: “Inderal a drug that …”

User: “Describe Elavil”

System: “Like Inderal, Elavil is used …”

User: “Describe Cafergot”

System: “Cafergot is very different from the drugs we have been talking about. …”

Story Presentation System

A dynamic dramatization method for narrative presentations

Architecture:

PlotAnalysis

DramatizationPresentation

Manager

Dramatic EffectsLibrary

Original StoryMaterial

Symbolic PlotDescription

StoryPresentation

Story Presentation Analysis

Plot analysis models: – physical and emotional state changes– positive and negative interference among

characters Dramatization uses plot analysis to

determine dramatic events in story– Lifeline, Rising complication, Reversal of

fortune, Dramatic irony, Happy end

Story Presentation Results

Presentation manager adds dramatic effects to original story material to emphasize dramatic events in story

Effects include– audio: selection of noises or music– images and video: flashbacks,

flashforwards, images of other characters

Design Environments

Provide a software environment supporting the activities part of design.– specification, construction– argumentation, documentation,

communication Examples:

– Framer (Lemke, Fischer)– JANUS (Morch, McCall, Fischer , ...)

Framer

Knowledge-based support for interface design

Approach:direct manipulation interface builder

Framer 1 -- construction kit approach

Framer 2 -- design environment

Design Environment Components (1) Checklist

– system provides decomposition of task,– user identifies current focus

Palette & Workspace– system provides primitive components– user identifies components used and

organization of components in design

Design Environment Components (2) Specification sheets

– system brings design issues to user’s attention, presents potential answers, and explains significance and consequences of design choices

– user symbolically specifies answers to design issues

Design Environment Components (3) Critics

– system points out sub-optimal design decisions, explains why this is believed, and provides heuristics for making decisions

– users may accept or reject the system’s critique

Design Environment Components (4) Catalog

– system provides examples– user selects designs to reuse and modify

Code generator– system generates an executable

representation of designed interface

Other Design Environments

JANUS -- kitchen design– designed for non-technical users

XNetwork -- computer network design– identified need for simulation component

VDDE -- voice dialog design– another type of interface design with

interesting constraints

Software Agents

One view:

Software processes that have non-trivial tasks delegated to them which require independent action and a report on the results.

Issues for Software Agents (1)

Personification– Should agents be represented as a living

or animated character?– Does it improve adoption of software?– Does it create inflated expectations?– Is it just too annoying?

Issues for Software Agents (2)

Trust and Competence– How does user develop an informed level of

trust?– Can agent give self-assessment on likely

outcome of task? Delegation

– How can user delegate tasks?– How can user check on status of delegated

tasks?

Issues for Software Agents (3)

Control– How does user set limits on the agent’s

activity?– When does the agent get to interrupt the user

(mixed-initiative dialog)? Dealing with multiple agents

– How can the user manage many agents? – How can interactions between agents be

predicted?

Information Retrieval Agents

Watch user patterns to infer interests or goals which are used to classify, rank, or suggest new information

Examples:– INFOSCOPE: patterns in Netnews use– BASAR: patterns in Web access

Issue: the “cold start” problem– must watch a while before suggesting

Social Filtering

Finding elements liked by others (with similar preferences)– requires some notion of preferences– improves with more users

Examples– Tapestry -- rating of documents– GroupLens -- collaboration & user profiles– Amazon.com and CD-NOW

Technologies III

Knowledge manipulation and Using recognized structure

Interacting with Knowledge

User tasks– Adding knowledge– Editing rule bases and object hierarchies

Examples– HITS Knowledge Editor (Terveen)– Modifier (Girgensohn)– Hyper-Object Substrate (Shipman)

Knowledge Representations

Informal– text, graphics, audio, video

Semi-formal– hypertext, argumentation

Formal– frames, semantic nets, scripts, rules,

inheritance hierarchies,

HITS Knowledge Editor

Knowledge editor for CYC project

Difficulties of knowledge representationformalization - articulation in precise detail

comprehension - complex vocabulary, size

modification - location and consistency

Terveen’s Design Principle #1

Provide a workspace in which users and systems can jointly construct and manipulate a context for problem-solving, and in which the state of the problem-solving is represented visibly.

Terveen’s Design Principle #2

Deliver intelligent assistance through critics.

Terveen’s Design Principle #3

Exploit the interactive potential of computational media to manage the user-system interaction according to conventions that are appropriate to the role of each party in the interaction.

Support Provided by HKE

Inferences -- information inferred from workspace and existing KB

Troubles -- inconsistencies between workspace and KB

Suggestions -- relevant representational issues for users to consider

Modifier

Support for End-User Modifiability– Users are not knowledge engineers

Example:– Adding new object class to existing system

Support: suggestions based on similarity of features and efficiency of representation

Hyper-Object Substrate

Formalization as difficulty– cognitive load– tacit knowledge– premature structure

Supporting incremental formalization– Flexible knowledge representation– Suggestions based on informal content– Automating knowledge acquisition?

Using Recognized Structure

When can it be used without user acknowledging correctness?

Examples: TileBars (Hearst)

• graphically presents results of text analysis inferring topic changes within text corpus

Data Mining• search for patterns within database

Non-Verbal Representations

Non-verbal communication allows expression of emergent thoughts

Systems need to recognize implicit structure

Examples:– VIKI (Marshall, Shipman, …)– PerSketch (Saund, Moran)– Tivoli (Moran, van Melle, Chui, …)

VIKI: An Analyst’s Workspace

Task: collect, organize, and interpret information

Characteristics:– evolving understanding– ephemeral structures– emerging visual languages

Goal: Recognize visual structures and languages

PerSketch: Adaptive Drawing Tool

Recognize multiple potential connections between strokes in drawing

Enables cleaning up drawings, editing drawn objects

Example:

TIVOLI: Meeting Room Support

Manipulating structures on an electronic whiteboard

Recognize gestures, characters, etc. Recognize lists, groupings Map changes in visual structure to

underlying database

Design Considerations

Issues of applicability,

Evaluating interfaces, and

Discussion

IUI: Appearing Too Smart?

A central danger of creating IUI is, like ELIZA, appearing too smart– Creating unrealistic expectations– Communication difficulties

• false alarm• garden path

Design guideline: do not hide system’s knowledge or lack thereof

Evaluation Methods

Interviews Cognitive Walkthroughs Questionnaires Videotapes of use Usage data from software monitoring Comparative evaluations Pre- and post- use testing

Evaluating Interfaces

Deciding between observational and empirical methods– Is there a specific question being asked?– How mature is the technology?– Is having real tasks in a natural setting

important?

Expectation Agents

Support for greater user participation in software design– Software designers encode expectations of

use which are matched against actual use.– When expectations are not met, a dialog

can be started between designers and users.