From Interaction to Understanding

From Interaction to Understanding

Mark Billinghurst

[email protected]

The HIT Lab NZ, University of Canterbury

May 21st 2015

My Childhood Home

The Vision - Auckland

The Road Trip

Happy Trails

The Reality

Essential Supplies

Mount Messenger

Are We There Yet?

Vision vs. Reality

Lesson’s Learned !  Enjoy the journey !  Travel with others !  Have a well equipped vehicle !  The end always seems nearer than it really is !  Don’t be disappointed in the outcome !  Most Important: Have a Clear Vision

HUMAN COMPUTER INTERACTION

Processing Power

User Interaction

Hard Wire

Gesture/Speech

???

Punch Card/Tape

Mouse/GUI

HCI VISIONS

Courtesy Matt Rettig, CMU

Vannevar Bush (1945)

!  Memex – As we may think

A Memex is a device in which an individual stores all his books … and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

Vannevar Bush

Ivan Sutherland

!  Sketchpad - 1963

The Ultimate Display The ultimate display would .. be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs .. would be confining, and a bullet .. would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.

Ivan E. Sutherland

Sutherland, I.E. The ultimate display. Information Processing 1965, Proc. IFIP Congress 65, 506-508.

Realizing the Vision !  3D computer graphics !  Physical input devices !  Intuitive interaction

Sutherland, I. E. (1968, December). A head-mounted three dimensional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I (pp. 757-764). ACM.

Richard Bolt !  Founder MIT Media Lab !  The Human Interface: Where People

and Computers Meet !  Van Nostrand Reinhold, 1984.

Interacting with machines via speech, gesture, and gaze will not only be possible, but in fact will become the way most people--most of the time, and for most purposes--will deal with them.

Richard Bolt

Put That There (1979)

!  Speech/gesture interaction

Ubiquitous Computing

“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”

Mark Weiser July 23, 1952 - April 27, 1999

!  dadfas


!  Mark Weiser, Xerox PARC !  TAB, Slate, Wall display


Smart Home Sensor Networks

Invisible Interfaces

Jun Rekimoto, Sony CSL

Hiroshi Ishii

Tangible User Interfaces (Ishii 97) !  Create digital shadows

for physical objects !  Foreground

!  graspable UI

!  Background !  ambient interfaces

i/O Brush (Ryokai, Marti, Ishii)

Virtual Reality

!  sddg

Virtual Reality

!  Immersive VR ! Head mounted display, gloves !  Separation from the real world

1977 – Star Wars

Augmented Reality

Augmented Reality ! Defining Characteristics

! Combines Real and Virtual Images -  Both can be seen at the same time

!  Interactive in real-time - The virtual content can be interacted with

! Registered in 3D - Virtual objects appear fixed in space

2008 - CNN

Example: Puteko

!  http://colarapp.com/

Microsoft HoloLens

!  Stereo see-through display !  Self contained !  Room scale tracking !  Gesture/speech interaction

!  df

From Hiroshi Ishii

!  asfas

MY VISION

!  daf

!  sdfs

!  sdfgs

!  axcvxca

Vision

“Using technology to create shared emotional experiences between users

and so create a deeper sense of empathy and understanding”

Empathic Computing 1. Computing systems that can understand your feelings and emotions 2. Computing systems that help you better understand the feelings of others

Affective Computing

!  Ros Picard – MIT Media Lab !  http://affect.media.mit.edu

Appliances That Make You Happy

!  Jun Rekimoto – Univ. Tokyo !  Smile detection + smart devices

Can we develop interfaces that allow us to be more empathetic to others?

Empathy Computing Requirements

! Basic Requirements ! Making the technology transparent

! Empathy Definition ! Seeing with the eyes of another ! Hearing with the ears of another ! Feeling with the heart of another

Movies are like a machine that generates Empathy

Roger Ebert

Using VR for Empathy

!  USC Project Syria (2014) !  VR Experience of Terrorist attack in Aleppo

Using AR for Empathy

! Augmented Reality can: ! Remove technology barriers ! Enhance communication ! Change perspective ! Share experiences ! Enhance interaction in real world

Wearable Computer

▪  Wearable computer – Ego-Vision view ▪  camera + processing + display + connectivity

Current Collaboration on Wearables

!  First person remote conferencing/hangouts !  Limitations

!  Single POV, no spatial cues, no annotations, etc

Sharing Space: Social Panoramas

!  Capture and share social spaces in real time !  Enable remote people to feel like they’re with you

Technology

!  Google Glass !  Image capture, viewpoint sharing

!  Remote device (desktop, tablet) !  Immersive viewing, live annotation

Capture Emotions Data Capture

Feature Detection

Emotion Recognition

Emotion Representation

Empathic User Interface

Hardware

User Interface

Sensor Systems/Software

!  Different systems developed/evaluated !  SSI framework (emoVoice, etc) !  e-Health, bitalino, arduino bio-sensors ! Wearscript wearable eye-tracker

!

!

Wearable Interface

!  Google Glass + e-Health + Spydroid + SSI ! Measure GSR, pulse oxygen, ECG, pitch !  Share video and audio remotely !  Represent emotions back to Glass user (4 states)

! !

Desktop Interface

!  Live video, real time emotion data !  See what sender sees, emotion representation

!

Emotion Representation

!  How can we show what you’re feeling? !  Tested - Raw data, Visual tinting, Emotion labeling

!

! !Excited Happy

Early Results !  Colour overlay + video stream preferred

!  Easier to understand

!  Disagreement over best colour coding !  Different emotional response with diff. stimuli

!  Scary movie best

!  Remote users felt connected to local user ! Understanding of emotions

Analysis

Table 2. Correlation between different streaming scenarios obtained from the feedback of the local and remote users

Experiment 3 (Creating a shared experience while watching an audio Visual Watching an audio visual)

Happy Sad Neutral Excited Case1

(Only video stream) 0.77 0.84 0.85 0.85 Case 2 (Case 1 + Color Overlay) 0.92 0.91 0.85 0.98

Case 3 (Case 2 + Graphical Analysis) 0.86 0.75 0.85 0.94

Looking to the Future

What’s Next?

Capturing Space: Real World Capture

!  Hands free AR !  Portable scene capture (color + depth)

!  Projector/Kinect combo, Remote controlled pan/tilt

!  Remote expert annotation interface

Remote Expert View

Brain to Brain Control

!  Rajesh Rao, University of Washington !  First Brain to Brain control

System Architecture

Scaling Up

!  Seeing actions of millions of users in the world !  Augmentation on city/country level

AR + Smart Sensors + Social Networks

!  Track population at city scale (mobile networks) !  Match population data to external sensor data

! medical, environmental, etc

!  Mine data to improve social services

Research Challenges !  How to convey emotion? !  How to measure empathy? !  Interface/interaction models? !  How to communicate emotion? !  Scaling up to city/country scale?

CONCLUSIONS

Take Home Messages !  Find people to travel with !  Equip the vehicle !  Enjoy the journey !  Don’t be disappointed in the outcome Most Important: Find your vision

More Information •  Mark Billinghurst

– [email protected]

– @marknb00

•  Website – www.hitlabnz.org