Writing and Speech Recognition

1

Speech, Ink, and Slides: The Interaction of Content Channels

Richard AndersonCrystal HoyerCraig PrinceJonathan SuFred VideonSteve Wolfman

Repeat Intro of Self

Mention:-Richard-Jonathan

In Audience

2

Background

Content channels simply refers to the various sources of information in some context (e.g. audio, slides, digital ink, video, etc.)

Our focus is on the use of digital ink in the classroom setting

We want to capture/playback/analyze these channels intelligently

3

Why do we want to analyze content channels?

We want to make it easier to interact with electronic materials Better search and navigation of

presentations Accessibility for the

hearing/learning/visually impaired Generating text transcripts Recognizing high level behaviors

Conversion to: Braille/Screen Reader

4

Distance Learning Classes

5

Classroom Presenter

General tool for giving presentations on the Tablet PC

Many similar systems – our findings applicable to all such systems

Enables writing directly on the slides Tablet PC enables high-quality digital ink Used in over 100 courses so far Allows us to collect real usage data

6

Questions We Wanted to Explore

High Level Question: What is the potential for automatic analysis of archived content?

Other Questions: How well can digital ink be recognized by itself? How closely are different content channels tied

together? Speech and Ink? Ink and Slide Content?

Can we identify high level behaviors by analyzing the content channels?

7

Research Methodology

1. We wanted to understand what real presentation data is like

2. We collected several 100’s of hrs. of recorded lectures from distance learning classes

3. Analyzed the data in various ways to help answer our guiding questions.

• Note: All examples given here are from real presentations!

8

Outline

Motivation Handwriting Recognition Joint Writing and Speech Recognition Attentional Mark Identification Activity Inference: Recognizing

Corrections

9

Handwriting Recognition

Classroom lectures on Tablet PC offer interesting challenges for handwriting recognition Somewhat Awkward

• Small Surface to Write On• Bad Angle to the Tablet PC

Hastily Written• Concentrating on Speaking• Excited / Nervous

10

Recognition Examples

The Good:

The Bad:

The Ugly:

Mark: Success/Failure

11

Recognition Procedure

Studied isolated words/phrases written on slides

Removed all non-textual ink Fed through the Microsoft Handwriting

Recognizer No training done!

12

Handwriting Recog. Results

260 (21%)18 (1%)123 (10%)850 (68%)Total

58 (11%)2 <(1%)46 (9%)408 (79%)Prof. E

111 (26%)9 (2%)45 (11%)262 (61%)Prof. D

19 (44%)1 (3%)5 (11%)18 (42%)Prof. C

71 (29%)6 (2%)26 (10%)146 (59%)Prof. B

1 (6%)0 (0%)1 (6%)16 (88%)Prof. A

NoneCloseAlternateExact

Mention That These Results Are Surprisingly Good!

Each Row Represents a Different Lecturer

13

Outline


Corrections

Look at Potential

14

Joint Writing and Speech Recognition

Co-expression of ink and speech Is digital ink spoken as it is written?

Yes, but how often? How “closely” to the written text?

Can speech be used to disambiguate handwriting?

Can handwriting be used to disambiguate speech? (incl. deictic references)

In Time/Accuracy, Wanted Empirical Evidence

15

Examples

Difficult for Speech and Ink Recognition

Difficult Written Abbreviations

Speech/Ink Used to Disambiguate Ink/Speech

DigiMon

Java 2 Enterprise Edition

Eswaran, Gray, Loric, Traiger

corn flakes

16

Experiment

Examined instances of isolated word writing Selected word writing episodes at random

but uniformly from the various instructors Generated transcripts manually from the

audio Checked whether the instructor spoke the

exact word written Measured the time between the written and

spoken word

17

Speech/Text Co-occurrence Results

Exact Approx None Simul 0-2s > 2s

A 1 (100%) 0 (0%) 0 (0%) 1 (100%) 0 (0%) 0 (0%)

B 9 (75%) 3 (25%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)

C 9 (82%) 2 (18%) 0 (0%) 10 (91%) 1 (9%) 0 (0%)

D 12 (86%) 2 (14%) 0 (0%) 10 (71%) 4 (29%) 0 (0%)

E 9 (56%) 7 (44%) 0 (0%) 7 (44%) 4 (25%) 5 (31%)

Total 40 (74%) 14 (26%) 0 (0%) 40 (74%) 9 (17%) 5 (9%)

Each Row Represents a Different Lecturer

18

Outline


Corrections

19

Attentional Mark Identification

Attentional Marks are… First step is to Identify a stroke as a

mark Tying Attentional Marks to slide

content is important Attentional Ink provides a concrete link

between speech and slide content!

20

Example

21

Method

Segmentation Few strokes Close spatial and temporal proximity

Mark Recognition Created hand tuned classifiers for:

Circles, Lines, Bullets/Ticks

Matched with slide content

22

Experiment

1. Identified and Classified Attention Marks by Hand

Two different people per slide Identified type of mark as well as slide

content mark referred to

2. Identified Attention Marks Automatically

3. Compared Resulting Identification

23

Content Matching Issues

Hard to determine exactly what content a mark refers to

Not just a recognition Issue, but also related to HOW people draw

24

Content Matching Cont.

Granularity of content parsing can be an issue

25

Attentional Ink Recognition Accuracy

532118 (22%)50 (9%)35 (7%)329 (62%)

8735 (40%)0 (0%)0 (0%)52 (60%)Bullets

33966 (20%)44 (13%)22 (6%)207 (61%)Underlines

10617 (16%)6 (6%)13 (12%)70 (66%)Circles

Non-MatchCloseExact to Punctuation

Exact

26

Outline


Corrections

27

Recongizing Corrections

Why? Want to answer the broad question:

- “Can we recognize patterns of activity by analyzing the ink and speech channels?”

Useful for Presenters- Occurs frequently (about 1-3 per lecture)

But Non-trivial

Our vision allows falsepositives

28

Recognizing Corrections

Identified Six Types of Corrections

Looked through large # of lectures, wide range of marks

29

Example Results

No Table Because: 1. Not a robust experiment2. Proof of Concept

30

Wrap-up

We wanted to understand the nature of real data to direct our focus when building tools for automatic analysis

Our studies provided the necessary understanding to accomplish this

31

Wrap-up (Cont.)

Specific Results: Basic handwriting recognition is

surprisingly good Very strong co-occurrence of written and

spoken words We were able to identify attentional

marks and the content associated with them

Activity Recognition: There are certain high-level activities that we can identify

ALL OPEN for Refinement

32

Questions?

[email protected]

[email protected]

Classroom Presenter Websitehttp://www.cs.washington.edu/education/dl/presenter/

Business

Writing and Speech Recognition