Designing the UI for the

Internet of Things

Tim Lynch Design lead, mobile and consumer products

Nuance Communications

@clampants | @NuanceInc

Voice interfaces for connected

experiences

#IoTUI

#SXSW

A little bit about me…

(Yes, we beat the record.)

I live in Boston, where I

am the lead UX

Designer for web/mobile

devices at Nuance. I

love Austin, have family

here, and almost moved

here, yet somehow I

ended up living in a city

with 800 ft. of snow in a month and below 0° F

winters. So thank you for

having me and letting

me thaw.

A little bit about Nuance… We innovate technology to reduce the distance between

want and get.

“Tweet this and say not a

bad way to start the day”

“Bring up the record for

patient Nicole Redman”

A little bit about our design team…

We are a diverse team of

interaction and visual

designers, user

researchers, dialog

designers across the

country, redefining speech

experiences across

handsets and tablets,

automotive, television,

desktop, IoT, wearables,

augmented reality,

gaming… The list goes on.

Did I mention it’s fun?

Narrative design is core to what we

do as designers. We have the ability

to pull from general design experience

to create meaningful voice

experiences for users.

The landscape of the

internet of things

We – designers and consumers – face

challenges each day when designing

for the Internet of Things:

• Smaller (or varying)

screen sizes

• Designing for

transmodal

experiences

• Disparate device

branding and cues

• User expectations of

immediacy

• And so on.

When we talk about devices that make

up the Internet of Things, what comes

to mind?

Light bulbs, thermostats, smart hubs, speakers, Crock Pots,

refrigerators…

These devices are the poster children of

connected devices – the IoT we know

and love today. They are generally

“smart devices” that…

a) Are loaded with sensors

b) Can “talk” (convey data and information) to us and

each other

But when we consider all the things that

are taking in and communicating data,

we often overlook the things we already

know –things that were smart and

sensor-laden well before the phrase

“Internet of Things.”

Sometimes, these things act as the

interface for other connected devices.

Other times, these things unto

themselves have become connected

devices.

Then there are

connected things

that feel like

science fiction…

By 2020, there could be

200-billion connected

things, from smart dust

to entire cities.*

How will we interact

with it all?

* http://www.intel.com/content/www/us/en/internet-of-things/infographics/guide-to-iot.html

This shift in ecosystem presents a

fundamental challenge. As people living in this world of connected experiences, we need to

interact with these things in personal, meaningful ways… and our

devices need to interact with us similarly.

Interacting with

all these things

Small screens

Pebble Time

Consider different form factors

(screens).

Screens

Honeywell WiFi Smart

Thermostat

No screens

Kohler Moxie

Showerhead &

Speaker

As these devices pervade our

everyday lives – and become

smaller, pushed into the background,

and more personal – our interactions

with them become more enmeshed

in the day-to-day.

Context: How are things used?

Life becomes the context.

Different inputs and outputs.

From Nicolas Nova’s “Curious Rituals” - https://curiousrituals.wordpress.com/

We’re in the wild-west with these

devices. Each device seems to put a

stake in the ground with some novel

way to interact with it.

Touches, gestures, and swipes on one device trigger one

response, while on another they trigger something

completely different.

Method’s Henri (via FastCoDesign) - http://method.com/work/ixda15

“With the rise of smart objects and the connected home,

we’ve found that products increasingly need to

communicate even without a screen, through things like

light and sound patterns.” –Daniel Nacamuli

Speech as a

unifying modality

Speech is the

simplest and

most human

communication

method.

Using speech, the user

interface becomes almost

invisible and the

experience is as natural as

part of the day-to-day.

The goal is to

communicate with

devices as we

would each other

– as humans.

How Voice Activates and Advances the Human-Computer Relationship - by Clifford Nass - http://goo.gl/YYozKs

Dialog evokes meaning,

identity, emotion, and trust.

Let’s pretend you have a connected

mattress.

The Chattress

Hey friend

:)

Foundations of the ‘voice interface’

The Chattress

You seem down

today…what’s

wrong?

ASR

Recognize the

words

NLU

Meaning

behind the

words

Dialog

Appropriate

response

Nothing,

mattress :(

Intelligence is the secret sauce that

permeates the entire experience,

influencing the conversational

partnership we have and creating a

much more personalized experience.

Intelligence & personalization

The intelligent Chattress

Intelligence

Context Knowledge

Memory

‘Smarts’ can manifest itself

through context, memory,

and knowledge.

Design considerations (Tweet this section)

A speech experience itself, though

– without a holistic and thoughtful

design – won’t make for a good

user experience.

Speech experiences will fail (or fail to be adopted)

when they:

• Fail to meet people’s expectations

• Don’t take into account context or other modalities

• Are hidden or unclear in purpose

• Ignore conversational norms

• Treat errors as dead-ends

Understand

expectations

Leverage the

strengths of

speech

Partner with

other modalities

Introduce

yourself

Frame the

scope

Support what

is natural

Provide

conversational

feedback

Identify errors

as opportunities

Deliver a

consistent point

of view

To remedy this, consider these speech

design factors.

1. Understand expectations.

People don’t want to

speak to their

devices simply to

have a

conversation…

they want to get

things done.

Design with those

things in mind.

Police Dog, Tess (via the State Library of New South Wales, NZ - https://flic.kr/p/5TJoyH)

2. Leverage the strengths of speech.

Speech can empower goal-oriented tasks, streamline existing

flows, and improve the experience in certain contexts.

3. Partner with other modalities.

Rarely should speech be

thought about as the only

modality.

Use it to support and

amplify other modalities.

4. Introduce yourself.

Make speech obvious and well-integrated into the full

experience. Once discovered, people will experiment.

5. Frame the scope with guidance.

The promise of natural language is you can say anything.

A challenge is you think you can say anything…

6. Support what is natural.

The “natural” in “natural language” can mean lengthy

phrases… but it can also mean simple fragments. Natural

language should encompass structured commands, but be

able to extend out to full, grammatically correct sentence

structure.

7. Provide conversational feedback.

Speech systems should follow our own conversational

norms, conveying they are listening and understanding.

8. Identify “errors” as opportunities.

“Sorry, I’m not

hearing anything.

Try checking your

mic settings.”

Audio

“Sorry, what time

was that?”

Recognition

“Did you mean

two people or two

o’clock?”

Interpretation

“Ah…I can’t help

you with that yet,

but try this…”

Dialog

It’s OK if something goes wrong – as long as people

understand what happened (and what to do to fix it).

(Tweet this)

9. Deliver a consistent

point of view.

Dialog, TTS, visual, audio, interaction, scope, content, form

factor… should all work in concert across devices.

So, if you take away one thing,

it should be…

Thoughtfully-designed speech

systems allow us to meaningfully

interact with our connected

devices.

Thank you!

Tim Lynch Design lead, mobile and consumer products

Nuance Communications

@clampants | @NuanceInc

#IoTUI

#SXSW