Cognitive assistance at work

© 2015 IBM Corporation

Hamid R. Motahari-Nezhad

IBM Almaden Research Center

San Jose, CA

Cognitive Assistance at WorkCognitive Assistant for Employees and Citizens

AAAI 2015 Fall Symposium


2

Workers and Work Management

https://www.centrodeinnovacionbbva.com/en/innovation-edge/social-business/social-trend

Onboarding, Orientation and Growth

Com

munic

atio

n a

nd

in

tera

ctio

ns

Work

an

d P

roje

ct

Manag

em

ent

Knowledge Worker


The Work Practices of Human Administrative Assistants

Human assistant activities

–Calendaring• Scheduling, information formatting and preparation

–Task Management

–Email Management• Filtering emails,

• Email classification

Interruption management

–Mediating interruption

–Prioritizing interruptions

Taking care of routine tasks

–Tracking

–Following up

–Travel arrangement, and preparation

–Reminding, and organizing

–Managing work of human• Pre-processing

• Filtering

• Prioritizing

• Compiling information and reports

3

An assistant “will remove much of the burden of administrative chores from its human user and

provide guidance, advice, and assistance in problem solving and decision making.” Gutierrze and Hilfdalgo,

1988

© 2013 IBM Corporation4

Credit: Rob Koplowitz, IBM Insight 2015


COGNITIVE ASSISTANCE

5


Cognitive Assistant

A software agent (cog) that

– “augments human intelligence” (Engelbart’s definition1 in 1962)

– Performs tasks and offer services (assists human in decision making and taking actions)

– Complements human by offering capabilities that is beyond the ordinary power and reach of human (intelligence

amplification)

A more technical definition

– Cognitive Assistant offers computational capabilities typically based on Natural Language Processing (NLP),

Machine Learning (ML), and reasoning chains, on large amount of data, which provides cognition powers that

augment and scale human intelligence

Getting us closer to the vision painted for human-machine partnership in 1960:

– “The hope is that, in not too many years, human brains and computing machines will be coupled together very

tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way

not approached by the information handling machines we know today”

“Man-Computer Symbiosis , J. C. R. Licklider IRE Transactions on Human Factors in Electronics, volume HFE-1,

pages 4-11, March 1960

6 1 Augmenting Human Intellect: A Conceptual Framework, by Douglas C. Engelbart, October 1962


History of Cognitive Assistants from the lens of AI

7

1945

Memex (Bush)

1962

NLS/Augment

(Engelbart)

1955/6

Logic Theorist

(Newwell, Simon, 1955)

Checker Player

(Samuel, 1956)

Touring Test,

1950

Thinking machines

1966

Eliza

(Weizenbaum)

1965-1987 DENDRAL

1974-1984 MYCIN

1987 Cognitive Tutors

(Anderson)

Apple’s Knowledge

Navigator System

Expert Systems

1965-1987 1992-1998

Virtual Telephone

Assistant

Portico, Wildfire,

Webley;

Speech Recognition

Voice Controlled

2002-08

DARPA PAL

Program

CALO

IRIS


Modern Cognitive Assistants: State of the art (2008-present)

Commercial

Personal Assistants and Bots– Siri, Google Now, Microsoft

Cortana, Amazon Echo, FB M

– Braina, Samsung's S Voice, LG's

Voice Mate, SILVIA, HTC's Hidi,

Nuance’ Vlingo

– AIVC, Skyvi, IRIS, Everfriend,

Evi (Q&A), Alme (patient

assistant)

– Viv (Global Brain as a Service)

– x.ai, Telegram bots

Cognitive and intelligent

platforms– IBM Watson

– Wolfram Alpha

– Saffron 10

– Vicarious (Captcha)

Open Source/Research

OAQA

DeepDive

OpenCog

YodaQA

OpenSherlock

OpenIRIS

iCub EU projects

Cougaar

Inquire* (intelligent textbook)

8

* Curated knowledge base


A Society of Interacting Cognitive Agents (Bots) and Humans

9

Cognitive Agent to

Agent

Human to Human

Cognitive Agent to Human

Human-Cog interaction

Cog-Cog interaction

Cog-mediated Human Interaction

Natural Language

Natural Language, or ACL?

ACL: Agent Communication Language, KQML, etc.

Natural Language-ACL-Natural Language

Weather

Cog

Health Agent

Personality

Insight Cog.

Provider

Cogs

Travel Cog 1

Travel Cog 2

Planning a Vacation

Trip

Considering preferences,

experience, conditions, cost,

Availability, etc.

Mediated and facilitated by Cogs


A major challenge towards offering cognitive assistance: Building knowledge models, and knowledge acquisition

“For an artifact, a computational intelligence, to be able to behave with high levels of performance on complex intellectual

tasks, perhaps surpassing human level, it must have extensive knowledge of the domain”

The challenge of AI in making progress toward building human-like artifacts:

– Knowledge representation, and (especially) knowledge acquisition

Approaches

– Build a large knowledge base by reading text

– Distilling from the WWW a huge knowledge base

Semantic Web and Linked Data methods over the last decade extensively has explored building models, ontologies and

rule-set that contributes to WWW knowledge representation

– Manual and semi-automated, focused on curated ontologies

– Community participation in building ontologies have resulted in creation of large knowledge bases: DBPedia, Yago,

Wikidata, Freebase, MediaWiki, etc.

– Ontologies are expensive to build and scale, and are generic in nature

10

EDWARD A. FEIGENBAUM, Some Challenges and Grand Challenges for

Computational Intelligence, Journal of the ACM, Vol. 50, No. 1, January 2003, pp. 32–40


Lesson Learned from Watson in Jeopardy

“The Watson program is already a breakthrough technology in AI. For many years it had been largely assumed that for a

computer to go beyond search and really be able to perform complex human language tasks it needed to do one of two

things: either it would “understand” the texts using some kind of deep “knowledge representation,” or it would have a complex

statistical model based on millions of texts.”– James Hendler, Watson goes to college: How the world’s smartest PC will revolutionize AI, GigaOm, 3/2/2013

Breakthrough:

– Developing a systematic approach for scalable knowledge model building from large, less reliable data sources, and

deploying a large array of individually imperfect techniques to find right answers

• Building and curating a robust, and comprehensive knowledge base and ruleset is laborious, time consuming and

slow

• Watson approach for building on massive, mixed curated and not-curated and less reliable information sources with

uncertainty has proved effective

11

Source:

Inquire Intelligent

Book


Towards Mass Computing as the Shared Characteristic of Recent Computing Advances

12

Scalable Computing over

Massive Commodity Hardware

Building Stronger

Super Computers

Cloud Computing

Crowd Computing

Advanced individual

algorithms

Mass computing applied to AIComplex array of algorithms applied to make

sense of data, and offer cognitive assistance

Big

Data

Complex

Analytics


COGNITIVE ASSISTANT FOR WORKERS

13


eAssistant: a cognitive assistant for the enterprise

A mobile intelligent assistant for the enterprise that assist a user (worker) to be

more productive by supporting following a methodology of monitor, process,

recommend and do actions with the following capabilities

– Understands human language

– Monitors input channels including email, calendar chat and enterprise

information sources

– Builds a model of the user and the world, and is situational aware (context)

– Offer assistance by

• Pre-processing information, and presenting information in desired format

• Categorizing and filtering information

• Gathering and organizing information

• Scheduling meetings

• Identifying requests, and organizing to-dos of its human subject

• Assists in performing tasks such as organizing events, travel assistant, and

learns new tasks

• And, suggest taking certain actions to its human subject that supports

increasing productivity, and growth

14


eAssistant: Cognitive Assistant Types in Work Environment

Personal (employee) eAssistant

– Personal eAssistants have access to the data space (and applications) that the principal has access to with the same

level of visibility

– While eAssistant is proactive in making suggestions, it takes action under the control and direction of the principal

Assistant’s eAssistant

– An assistant to Human Assistants helping them to become more productive, and focus on work that require human

judgment

Expert/Process eAssistants

– Assistants that are experts in a specific domain such as travel policy, human resources, etc.

15

Cognitive Assistant Platform

Individual cognitive

agents

Assistant’s Cognitive

AgentsExpert Cognitive

Agents

Systems of cognitive agents that

collaborate effectively with one

another to support human activities.

Interactions types need to be supported:

• cog-to-cog interactions,

• human-cog interactions, and

• cog-backed human-to-human interactions


Actionable Statement Identification Over Unstructured Conversations

16

Email, Chat, and Calendaring apps are

the most used channels for doing work

in the enterprise

Addressing the work organization and

management for Knowledge workers:

monitoring communication channels (email,

chat), and:

- capturing, prioritizing and organizing work

of a worker

- Identifying actionable statements

(requests, commitments, questions) and

track them over the course of

conversations

© 2013 IBM Corporation17

Inbox - Verse Highlighting actionable statements Recommending fulfilment actions

IBM Insight 2015 – The session on “Given your collaboration tools a brain”

© 2013 IBM Corporation18IBM Insight 2015 – The session on “Given your collaboration tools a brain”

Send File Action Archetype Send File Action Archetype Send File Action Archetype


Invite/Calendar Action Archetype Automated Invite Parameters Extraction Calendar Entry Creation


Integration with Watson Health Integration with Watson Health Integration with Watson Health


eAssistant App and APIs

21

Watson (& BigInsight NLP) Apps and Services on BlueMix

Co

llab

ora

tio

n T

oo

ls

Enterprise Repositories, Applications and Data Sources

Feeds

RepositoriesDocument collections

…

eAssistant Apps

Personal Knowledge

Graph Builder

Conversation Analytics, Auto-Response,

Prioritization

Calendar and Scheduling Assistant

Context-aware Information

Finder

To-do, Task and Process

Assistant

Cognitive Work Assistant APIs

Semantic Role Labeling

POS taggingDependency

AnalysisCo-reference

resolutionNamed Entity Recognition

Knowledge GraphBuilder


COGNITIVE BPMCognitive Assistance for Case Management

22


Cognitive assistance in case management for knowledge workers

Knowledge workers are involved in handling cases in the work context (in domains such as social care, legal,

government services, citizen services, etc.).

Cognitive case management is about providing cognitive support to knowledge workers in handling customer cases.

Cognitive assistance for employees: Handling and managing cases involves understanding policies, laws, rules,

regulations, processes, plans, as well as customers, surrounding world, news, social networks, etc.

Cognitive assistance for customers/agents: Assists citizens by empowering them by knowing their rights and

responsibilities, and helping them to expedite the progress of the case

23

Citizens

Assistant

Business

Employees/

agents

Plansworkflows

Rules

Policies

Regulations

Templates

Instructions/

Procedures

ApplicationsSchedules

Communications such as

email, chat, social media,

etc.

Organization

Cog. Agent

Unstructured Linked Information


Spectrum of work, and Cognitive Assistance

24

Ref: Motahar-Nezhad, Swanson, 2013, and Sandy Kemsley, Column2

Spectrum of Work

Cognitive BPM


Cognitive BPM Systems

A Cognitive BPM system offers the computational capability of a cognitive system to provide

support the whole lifecycle of processes over structured and unstructured information sources,

and continuously discovers, learns and acts to achieve a desired process outcome

– Meets two pressing needs: supporting complex process decisions, and processing large

amount of data

– Intelligent and integrated process (model) definition, reasoning and adaptation

• Process is not assumed apriori defined; discovered, learned and customized based on

accumulated knowledge and experience

– Intelligence supporting the execution of process

• When, What action (how) and whom to contact

– The need for revisiting some basic abstractions of BPM

• Real-world, and real-time course of actions

• New information availability changes course of actions in a plan

• Fluid actions/tasks, notion of task completion, and process/plan adaptation

25


Process Definition,

Discovery, Learning

Process Enactment

Process/ Environment

Sensing

Process Analytics

Proactive/ Reactive Process

Response/Adaptation

Cognitive BPM Lifecycle

26

Environment Sensing

Data

sources

Data Processing/

Analytics

Process

Composition /

Enactment Update

Process

Monitoring/Analytics

IoT


Use Case: Cognitive Case Assistance

Assume an executive admin is managing an event organization process for their department

– Step 1: sending invite to an event to employees in their department, through email and requests for RSVP

• Cognitive BPM (1): Q&A ability for the admin: How many have confirmed, how many pending, how

many not answered

• Cognitive BPM (2): Predictive analytics: how many will eventually RSVP?

• Cognitive BPM (3): Diagnostic analytics: why some not accepted (customers in case of marketing

case)?

– Step 2: Ordering place, food, transportation, etc

• Cognitive BPM (1): tracking of the process steps, which vendor have replied, which ones pending,

have questions, etc.

• Cognitive BPM (2): keeping track of synchronization and consistency (dates, amounts, numbers, etc.)

among different steps

– Step 3: Pre-event steps

• Reminding people who have RSVPed

• Compiling and sending logistic information (from different steps)

27


Research in Support of Cognitive BPM in Work Assistant Space

Task, commitment and task lifecycle extraction from workers interactions over email and chat

28

Anup K. Kalia, Hamid R. Motahari Nezhad, Claudio Bartolini, Munindar P. Singh: Monitoring Commitments in People-Driven Service Engagements. IEEE SCC

2013: 160-167


Research Directions

Abstractions and models for Cognitive Work Assistants and Cognitive Processes

Knowledge representation models, and scalable knowledge acquisition methods from

unstructured information (text, image, etc.) and building actionable knowledge graphs

Cognitive Work Assistants

–Cognitive augmentation of workers in work environments, and in process management

Cognitive Process Management System

–Analytics on unstructured information to support process understanding

–Analytics to support process adaptation, customization and configuration

–Proactive process adaptation

Learning and teaching tasks and processes to cognitive agents

–Interactive learning where cognitive agents ask process questions

–Gradual learning through experience, and process improvement

29


QUESTIONS? Thank You!

30

Data & Analytics

Cognitive assistance at work