Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/267861785

ADualSceneCameraEyeTrackerforInteractionwithPublicandHand-heldDisplays

ARTICLE

READS

19

3AUTHORS,INCLUDING:

AndreasBulling

MaxPlanckInstituteforInformatics

55PUBLICATIONS535CITATIONS

SEEPROFILE

HansGellersen

LancasterUniversity

243PUBLICATIONS4,938CITATIONS

SEEPROFILE

Availablefrom:AndreasBulling

Retrievedon:06November2015

A Dual Scene Camera Eye Tracker forInteraction with Public and Hand-held Displays

Jayson Turner1, Andreas Bulling1,2, and Hans Gellersen1

1 Lancaster University, United Kingdomj.turner@lancaster.ac.uk,hwg@comp.lancs.ac.uk

2 University of Cambridge, United Kingdomandreas.bulling@acm.org

Abstract. Advances in eye tracking technology have allowed gaze tobecome a viable input modality for pervasive displays. Hand-held devicesare typically located below the visual field of a standard mobile eyetracker. To enable eye-based interaction with a public display and ahand-held device, we have developed a dual scene camera system withan extended field of view. Our system enables new interaction techniquesthat take advantage of gaze on remote and close proximity displays toselect and move information for retrieval and manipulation.

Keywords: Dual Scene Camera, Extend Visual Field, Gaze Interaction,Public Display, Hand-held Device

1 Introduction

Recent advances in eye tracking equipment have enabled mobile daily-life record-ings and pervasive gaze-based interaction [1]. This gives rise to its potential asan input modality for pervasive displays. We are particularly interested in theuse of mobile eye tracking for gaze-based interaction across remote displays andhand-held devices (see Figure 1(a)). In previous work we have described severalinteraction techniques specifically designed to utilise gaze on both such displays.Stellmach et al. introduced several techniques that use gaze and touch input forremote target selection [2].

To map the gaze of a user to a screen in the environment, the screen mustbe detected via the scene view of the eye tracker [3]. This is problematic with ahand-held device due to the close proximity of the device and the viewing angleof the scene camera. To bring the device in to view the user must lower theirhead beyond a comfortable limit. To address this issue, Schneider et al. used agaze-actuated camera in combination with a static wide angled scene camera sothat a wide angle of view may be documented [4].

Figure 1(b) shows the natural head position of a user looking at a hand-helddevice that is not fully visible, as well as the head position needed to bring thehand-held device in to view of the scene camera. This position is uncomfortablefor the user and does not allow them to properly operate the device.

Fig. 1. (a) A user interacting with a situated display and a hand-held device. (b)(left)User looking naturally at hand-held device below, the device is not fully visible in thescene video feed. (b)(right) User lowers head beyond a comfortable limit to bring thedevice in to view of the scene camera.

2 Dual Scene Camera Eye Tracker

We propose to use an additional scene camera to extend the visual field of amobile eye tracker. As a basis we used an off-the-shelf tracker from SensoMo-toric Instruments (SMI) that integrates a UI-1221LE-C camera produced by IDSImaging, its imaging sensor is 4.512 x 2.880 mm in size. The default lens pro-vided by SMI has a 3.6 mm focal length. The angle of view (α) can be calculatedby the sensor dimension (d) and lens focal length (f) as follows: α = 2 arctan d

2f .This yields an angle of view of ˜44◦vertically and ˜64◦horizontally.

We augment this system with an additional camera mounted via a 3D printedacrylonitrile butadiene styrene (ABS) pivotable mount that clips firmly on tothe existing scene camera (see Figure 2(a)). Each camera uses a 3.0 mm lensand yields ˜51◦vertical and ˜74◦horizontal angle of view. When combined, thevertical angle of view is increased up to ˜102◦depending on the tilt of the topcamera. Figure 2(b)(right) shows how the additional camera provides an extrafeed that is used to detect a display below the original scene cameras angle ofview.

The additional camera also provides a scene view with minimal lens distor-tion and high spatial resolution, using a wider angled lens such as a fish eyewould negatively impact these attributes. In previous work we have shown thatthis approach improves usability with no significant impact on interaction per-formance when compared to the standard optical configuration [5].

Infrared illuminated images of the eye are reflected via a hot mirror to acamera attached to the eye tracker, this is as not to obstruct the users view withhardware, see Figure 2(a). When looking at a hand-held device, it is common forusers eyelids to occlude their pupils. Our system uses an algorithm by Swirskiet al. that enables us to detect the pupil from below its optical axis [6]. Toestimate gaze, the pupil centre must be mapped to scene camera coordinates, a9 point calibration technique is used to find a homographic relationship betweenthese. Due to parallax errors caused by the scene camera and the users eye notbeing on the same optical axis, the system requires two calibrations, one for

Fig. 2. (a) (1) Two USB cameras mounted vertically on an adjustable mount. (2) Eyecamera and hot mirror. (b) (left) Thresholded image used to find screens in the sceneview. (right) Extended scene view now able to detect both displays.

the distant public display and one for the closer hand-held device. The systemswitches calibrations depending on which screen is in view or which is closest tothe users gaze point.

The intrinsic parameters of each camera are determined using an offlinecamera calibration routine that is part of the Open Computer Vision library(OpenCV). Radial and tangential distortions caused by the lens are corrected ineach incoming frame before they are used.

Our system is able to detect and differentiate between two displays, one 50”Plasma TV and one tablet PC. Screens are detected by converting incomingvideo frames to Hue, Saturation and Value (HSV) colour space and thresholdedto find the pixels with the highest Value (see Figure 2(b)(left)). Contour findingis then used to look for rectangles in the image. As the tablet PC is operatedby touch, it is possible for the display to become occluded by the users hand,disabling rectangle detection. To resolve this we also search for blobs, sort themby area and find their bounding box and convex hull. Each bounding box corneris matched to its closest convex hull point, if a corner of the display is occluded itis estimated using the corresponding bounding box corner point. Each screen hasa different ratio of sides, this is used to determine which one has been detected.

For every incoming scene frame, we find the four corner points of each de-tected display and the four points representing the bounds of each displays coor-dinate system. These are used to calculate a perspective transform. This trans-form is then applied the users gaze in scene view coordinates to obtain their gazein screen coordinates.

3 Interaction Techniques & Application

As our system is capable of mapping gaze to both public and hand-held displays,this opens up avenues for new interaction paradigms. We will demonstrate howthis system can be used to interact with a public bulletin display and a tabletdevice. The bulletin display contains URLs, contact details, images and dates forbusinesses, events, and advertisements. The system uses techniques designed to

fulfil the following tasks: Retreive/Return: Movement of an object to and froma remote display and a hand-held device. Relocate: Movement of an object fromone location to another within one display.

The system will utilise three interaction techniques proposed in previous workthat use a combination of gaze and touch [7]:

– Eye Cut & Paste Look at object on display, tap on hand-held to pick up,object is ’cut’ from view, look at drop location, tap to drop. The same orderof events can be used for both tasks.

– Eye Drag & Drop Similar to Eye Cut & Paste but selection and drop areconfirmed with touch hold and release respectively.

– Eye Summon & Cast Bring object to hand-held (Summon): look at objecton remote display, swipe down on hand-held, the object is moved to thelocation of the swipe. Return object (Cast): look at drop location on remotedisplay, swipe upwards over the object on the hand-held device, the objectis moved to the gaze location. These steps can be used to relocate an objectby changing the drop location in the Cast stage.

As an example, a user may wish to obtain a telephone number from the publicdisplay to make a call. Using Eye Drag & Drop the user can look at a contactnumber, perform a touch hold gesture, direct their gaze to the phone applicationon their hand-held device and perform a touch release to drop the number onto the application icon. This would then place a call to the retrieved number.The user may rearrange objects on the public display and post new media fromthe hand-held device such as text, images and calendar events. They may alsomanipulate existing media by bringing it closer for editing to then return it toits original location.

References

1. Bulling, A., Gellersen, H.: Toward mobile eye-based human-computer interaction.Pervasive Computing, IEEE 9(4) (october-december 2010) 8 –12

2. Stellmach, S., Dachselt, R.: Look & touch: Gaze-supported target acquisition. In:Proceedings of the 2012 annual conference on Human factors in computing systems.CHI ’12, ACM (2012)

3. Mardanbegi, D., Hansen, D.W.: Mobile gaze-based screen interaction in 3d envi-ronments. In: Proc. NGCA, ACM Press (2011) 2:1–2:4

4. Schneider, E., Villgrattner, T., Vockeroth, J., Bartl, K., Kohlbecher, S., Bardins, S.,Ulbrich, H., Brandt, T.: EyeSeeCam: An Eye Movement-Driven Head Camera forthe Examination of Natural Visual Exploration. Annals of the New York Academyof Sciences 1164(1) (2009) 461–467

5. Turner, J., Bulling, A., Gellersen, H.: Extending the visual field of a head-mountedeye tracker for pervasive eye-based interaction. In: Proceedings of the 2012 Sympo-sium on Eye-Tracking Research and Applications. ETRA ’12, ACM Press (2012)

6. Swirski, L., Bulling, A., Dodgson, N.: Robust, real-time pupil tracking in highlyoff-axis images. In: Proceedings of the 2012 Symposium on Eye-Tracking Researchand Applications. ETRA ’12, ACM (2012)

7. Turner, J., Bulling, A., Gellersen, H.: Combining gaze with manual interaction toextend physical reach. In: Proc. PETMEI, ACM Press (2011) 33–36