Smart Email System

Aarthi NatarajanSuhaib A. ObeidatGanesh Sridharan

OutlineOutline Motivation High-level Design Functional Design E-mail message and address book

sorting/prioritizing Device Control Using E-mail Voice Interface Implementation Performance Metrics Future work Conclusions

MotivationMotivation

Reduce Human Distraction Free hands, eyes Less time to achieve tasks (user behavior,

voice) Least amount of attention

Provide more services and flexibility Access over a phone Remote control of devices Ability to prioritize.

HLDHLD

User Interface

(GUI + Voice-Enhanced)

User Behavior Monitor

(capturer)SMTP IMAP

E-mail System

Interface

Message Sorting and Prioritizing

Device Control Module

Arrival Notification

Module

Display ManagementDisplay Management

1. Sorting of emails – based on the user access patterns

2. Sorting the address book – based on the frequency of correspondence.

3. Notifying the user of new message arrival:i. Access to the user’s calendar ii. frequency of correspondenceiii. user-assigned trustiv. subject keywords

Store E-mails Addresses and PrioritiesStore E-mails Addresses and Priorities

Emails addresses, user access pattern, frequency of correspondence and trust level are stored as an XML element in an XML file.

Whenever the e-mail client is active, it is loaded into the memory from the file system as stored in a hash table.

Periodically, the XML file is updated from the hash table.

Device Control Using E-mailDevice Control Using E-mail

Reduces human distraction by eliminating the need for physical movement.

Ability to control different appliances/devices remotely: Devices connected to serial or parallel

port Why not USB ? future work.

Auto-response to provide feedback.

Device Control Using E-mail-ContDevice Control Using E-mail-Cont

An e-mail address is assigned for the purpose of hardware control (e.g., my_home@domain.com).

Sender address is checkedA digital signature can be used for security purposes.

Computer that has control of the h/w is on-line all time, with the e-mail client running.Upon message arrival, subject line is parsed:

E.g. “A.C: Switch On”

Create an e-mail message destined to the user, with the result of the operation (e.g., success, fail, device is not connected).

Voice InterfaceVoice Interface Speech Synthesis:

Text-to-speech (TTS) Notification, message text, etc.

Speech Recognition Speech-to-text (STT) Rule-based grammar (higher accuracy of

recognition). As opposed to dictation free recognition.

Temporal-awareness.

Speech SynthesisSpeech Synthesis

Synthesizers provide the computer with the ability to speak.

Users and applications provide text to a speech synthesizer, which is then converted to audio.

Bad news: does not sound natural.

ApplicationSpeech

Synthesizer

“Computers

can speak”

Computers can speak

Speech RecognitionSpeech Recognition

1. Structure Analysis: start and end of paragraphs, sentences, and other structures

2. Text pre-processing: abbreviations, dates, numbers, currency amounts (etc.)

3. Text-to-phoneme conversion: times t ay m s4. Prosody analysis: determine appropriate prosody

for the sentence5. Waveform production:

• Concatenation of chunks of recorded human voice• Formant synthesis: signal processing techniques based

on knowledge of how phonemes sound and how prosody affects those phonemes

Why VoiceWhy VoiceAllows access over a phone

User’s hands are occupied

User has physical disability (e.g., limited use of hands).

User’s eyes are looking at something other than the screen (e.g., driving, maintenance and repair, etc).

User has physical disability (e.g., visual impairment)

Challenges InvolvedChallenges InvolvedTransience: “what did you say” ?

Invisibility: what actions to perform ?

Asymmetry: people can speak faster than they can type, but listen much more slowly than they can read.

Synthesis quality: recorded or synthesized ?

Recognition: flexibility vs. accuracy.

Design Issues in Speech ApplicationsDesign Issues in Speech Applications Feedback and Latency:

People read meaning in pauses Speech applications cause pauses in places where they

do not naturally belong. Prompts:

Assessing the tradeoff between flexibility and performance.

Explicit prompts: when user must be tightly constrained Implicit prompts: when application is able to accept more

flexible input

Handling errors: no repetition, more constrained grammar.

E-mail Protocols-IMAPE-mail Protocols-IMAP IInternet MMessage AAccess PProtocol Incoming Mail Protocol Provide support for different access

modes: Online:NSF-like (connection

maintained throughout) Offline: download and delete from

server (periodic connections. Disconnected (hybrid) :download,

manipulate, upload. Offline paradigm allows minimum

connect time Constructs to permit online

performance optimization, especially over low-speed links.

E-mail Protocols-SMTPE-mail Protocols-SMTP

SSimple MMail TTransfer PProtocolSending e-mail messages between servers or from a client to a serverProved useful in the wireless domain (e.g., used in SMS).Drawback: not fast enough (e-mail was not intended for wireless).

E-mail Protocols-MIMEE-mail Protocols-MIME

MMulti-purpose IInternet MMail EExtensions.

How messages must be formatted so that they can be exchanged between different email systems.

Compatible with WAP.

ImplementationImplementation Java-based e-mail client.

IMAP, SMTP and MIME

XML-based display manager JSAPI for speech synthesis and recognition

FreeTTS speech synthesizer(written entirely in Java).

ViaVoice speech recognizer (from IBM). JavaComm for hardware control.

Mapping device name to corresponding port. Interfacing from the mail system to the particular device.

Performance MetricsPerformance Metrics

Subjective Tests:User Distraction.Usability study (I.e., Flexibility of Use,

UI convenience).Accuracy (e.g., of the voice interface,

the remote control of the hardware).

Future WorkFuture Work

Extending the hardware control capabilities of the system

E.g., wireless access to the different devicesHarmonic control of the overall system.

Providing speech recognition allowing for user-machine dialogues.Web-based ImplementationAccess over a phone

ConclusionsConclusions

Capabilities of a context-aware e-mail system can go way beyond the traditional functionality.

Implementing voice-enhanced systems introduces many issues.

Ubiquitous Computing is a revolutionary rather than evolutionary field.