““It's always incumbent on a researcher to make the effort to It's always incumbent on a researcher to make the effort to reach out to the community to engage with the population reach out to the community to engage with the population

they're serving.”they're serving.”

Experiments with the See ColOr Interface

Guido BolognaCVML, University of Geneva, Switzerland

3

Outline

• Introduction• Local Perception Module• Experiments in the past• New experiments• Conclusion

4

Introduction

• Blind people40 millions in 2020 (90% after the age of 1 year)4 millions in OCDE countries

• Long term goal : sensing the environment with soundsAction is key to perception

• A number of pixels are directional sound sources

Local Perception ModuleLocal Perception Module

6

See ColOr Seeing Colours with an Orchestra

head-centric

stereovision

distance

video

saxophone

piano

colour encoding2D sound spatialization

headphones

25 points

7

8

Colour encoding

• Colour system: Hue-Saturation-Luminance

• Hue: musical instrument• Saturation: 4 instrument

notes• Luminance: 2 instruments

with different notes

9

HueOboe

Viola

Violin pizzicato

Flute

Trumpet

Piano

Saxophone

10

Saturation and luminance

• 4 levels of saturation

• 8 levels of luminance

Human voiceDouble bass

Low High

11

Prototypes

– Stereoscopic or Kinect camera– Sonification of a row containing 25 points– CIPIC Kemar HRTFs (2D spatialization)– HSL colour coding– Two depth coding schemes– Lower peripheral resolution

[15 12 9 7 5 3 3 2 2 1 1 1 1 1 1 1 2 2 3 3 5 7 9 12 15]

12

Depth coding (first mode)

For all sonified pixels (25) :• 90 ms for undetermined depth (D);• 160 ms for 0 <D<1• 207 ms for 1<D<2• 254 ms for 2<D<3• 300 ms for D > 3

13

Depth coding (second mode)

• The strategy here is to sonify the dominant colour of the row with only 1 sound source (providing average depth)

• The centroid of the first red area is laterally spatialized

[R R R R R B B J B B B R R G G G J J B B B G G G G] [R R R R R ]

• The volume V after 3 meters starts to decrease:F(V) = V * exp(-k*D)

Experiments in the PastExperiments in the Past

15

16

Left picture : the 6 experiment participants interpreted the major colours as the sky the sea and the sun; clouds were more difficult to infer (2 subjects); instead of ducks, all the subjects found an island (with yellow sand) or a ship.

Right picture : 4 participants correctly recognised the tree with the sky and the grass; one subject qualified the tree as a strange dark object and finally the last individual imagined a nuclear explosion !

17

• All individuals found one of the red doors in a time range between 4 and 9 minutes.

18

Pairing colored socks

19

Participant Time (mn) Success rate (pairs)P1 12 5P2 24 5P3 7 5P4 6 3P5 16 5Average 13.0 (7.3) 4.6 (0.9)

Participant Time (mn) Success rate (pairs)P6 5 5P7 10 5P8 4 5P9 4 5P10 18 5Average 8.2 (6.0) 5.0 (0.0)

20

Following a red path

• In this experiment the goal was to follow a coloured path in an outdoor environment.

21

Participant Path Length (m) Speed Average (m/h)P1 M+C = 88 723P2 M = 84 710P3 M+B = 110 485P4 M+A = 93 656P5 M = 84 484P6 M+A+C = 97 600P7 M+A+C = 97 451P8 M = 84 869P9 M = 84 1326P10 M = 84 1096Average 90.5(8.7) 740.0(284.6)

Find a red tee shirt

New ExperimentsNew Experiments

19/04/23 25

Current Prototype with Kinect

Alerting System

Object Recognition Text reading

Local Perception Global Perception

See ColOr – CVML (2013)

The Spinning ChairEXPERIMENT ONE:

1

2

3

4

5

1

Time (

minu

tes)

Finding JuanEXPERIMENT THREE:

2

4

6

8

10

1

Time (

minu

tes)

Walking Towards a WallEXPERIMENT TWO:

Grasping ObjectsEXPERIMENT FOUR:

ConclusionsConclusions

35

• Sonification of two very important vision features (colour and depth) made it possible to achieve a number of tasks related to mobility and recognition.

• We will pursue to define new challenging experiments requiring the activation of several See ColOr modules.

• We hope that new small 3D RGB cameras will be in the market soon (like the new Kinect Capri) !