Rewarding the Original: Explorations in Joint User-SensorMotion Spaces

John Williamsonjhw@dcs.gla.ac.uk

Roderick Murray-Smithrod@dcs.gla.ac.uk

School of Computing Science, University of Glasgow, G12 8QQ, Scotland, UK

ABSTRACTThis paper presents a systematic and general technique forestablishing a set of motions suitable for use with sensorsystems, by drawing performable and measurable motionsdirectly from users. It uses reinforcement which rewardsoriginality to induce users to explore the space of motionsthey can perform. A decomposition of movements into mo-tion primitives is constructed, among which a meaningfuloriginality metric can be defined. Because the originalitymeasure is defined in terms of the sensed input, the result-ing space contains only movements which can both be per-formed and sensed. We show how this can be used to eval-uate the relative performance of different joint user-sensorsystems, providing objective analyses of gesture lexiconswith regard to the technical limitations of sensors and hu-mans. In particular, we show how the space of motionsvaries across the arm for a body-mounted inertial sensor.

ACM Classification KeywordsH.5.2 [Information Interfaces And Presentation]: User Inter-faces - Input devices and strategies;

Author Keywordsmotion; gesture; reinforcement; originality, inertial; novelty.

INTRODUCTION“At Sea Life Park we shaped creativity in two dolphins[...] by reinforcing anything the animals did that wasnovel and had not been reinforced before. Soon thesubjects caught on and began “inventing” often quiteamusing behaviors.”

This quote, from Karen Pryor’s book [13] on reinforcementtraining for animals, illustrates an unusual application of pos-itive feedback: promoting creativity by rewarding novel be-haviour. Reinforcement is often used to shape behaviourinto specific forms, narrowing behaviours down to a tem-plate. But it can be used for the opposite effect; to promotethe discovery of new behaviours by rewarding originality. Inthis paper we explore how this strategy can be used to mapthe capabilities of a joint user-sensor system by extracting aspace of useful motions directly from users themselves.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHI’12, May 5–10, 2012, Austin, Texas, USA.Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.

The purpose of these explorations is to derive a systematicmethod for studying the capabilities of input devices in con-text. There is now an enormous and ever-expanding diver-sity of devices which capture movement for the purposes ofcontrolling a computer. From familiar pointing devices liketouch screens and mice to more esoteric vision based sys-tems, pressure-sensing and inertial-sensing hardware, thesedevices are incorporated into feedback loops where commu-nication between human and computer can take place. Withthe exception of brain-computer interfaces and a few otherbio-sensors (such as galvanic skin response), all of theseinput devices, indirectly or otherwise, measure the effectsmuscle activity for the purpose of determining intention.

It is challenging to work out how to most effectively designnew input systems, or how to best exploit the capabilitiesof existing devices. There are a number of well-developedspecific interaction paradigms for different classes of inputdevice. Pointing movements, for example, have been ex-tensively studied (e.g. [17], [11]) and there exists a range oftechniques for analysing performance and designing interac-tions where pointing is practical. Static pattern-based “ges-ture recognizers”, which match defined sensor sequences toa set of symbols, have been widely explored for non-pointingdevices, especially inertial sensor and vision-based systems.Scoditti et. al. [15], for example, lays out a taxonomy ofmovements for accelerometer control from a symbolic lan-guage perspective. Still other interaction paradigms use gen-eral dynamical systems (of which pointing systems are an el-ementary instance), such as the motion selection techniquesof Williamson et. al. [18] and Fekete et. al. [2]. Many inputtechniques are under-exploited, however; capacitive sensorarrays are often used to detect touches as a set of contactpoints, but there is much more usable information that couldbe extracted (e.g. from pose variations).

While these specific paradigms are very powerful and are es-sential for the construction of working, usable systems, it isinteresting to consider how input mechanisms can be quanti-fied in a broader sense, and how factors which influence theuse of these mechanisms can be analysed within a coherentframework. This paper looks at the essential characteristicsof one aspect of input mechanisms – determining the spaceof motions which are distinguishable and controllable, anddevelops a very general technique for exploring, comparingand quantifying these spaces.

High reliability, low variability.Low information capacity: few symbols but very precise.

Low reliability, high variabilityLow information capacity: many symbols but imprecise

High reliability, high variabilityHigh information capacity: many symbols and very precise

Figure 1: A sketch of variability and reliability. Imaginary distributions of sensor sequences projected onto a 2D plane are shown, where eachdistribution represents one “symbol” and the shaded area shows the range of measured value for repeated performances. To have high informationcapacity, there must be both variety and precision, so that there is a partition of the space which separates the symbols well.

Information, semantics and input devicesThe quality of a user input system, in terms of its purely util-itarian benefits, can be quantified by its information through-put. The more information that is communicated, the betterthe interface. This is the basis of many input device stud-ies in HCI, such as those using Fitts’ Law [3] to estimatethe information capacity of pointing devices. This narrowfocus on information transfer rates excludes many impor-tant factors in the quality of an interaction, including er-gonomics, aesthetics, social acceptability, cost and physicaldimensions, but it captures the essential quality which dis-tinguishes effective input systems from ineffective ones.

In our stance, there is a clear separation between the processof communicating intention and the associated semantics.Analysing interaction this way ignores the relation betweenmovements themselves and the meaning a system ascribesto them; there is no consideration of whether flicking a fin-ger is a more suitable operation than shaking a fist for someparticular purpose. In building concrete systems, exploitingpreconceived notions and metaphors inherent in human ac-tivity is essential method to address aesthetic and subjectiveaspects of interaction. But here we focus on a high-levelanalysis of the limits of an input mechanism without regardto metaphor or idiom. In this way we can separate, controland measure factors such as encumbrance or social contextwhich impact upon the usability of an interactive system.

To be more precise, the sense of “input system” or “inputmechanism” as used here, is a system which encompassesthe combination of the involved human parts and the physi-cal sensors. A system which uses accelerometers for sensingmotion of the hand is quite different from one that is attachedto a knee, and different still from a system which uses in-frared proximity to sense motion of the hand, and these allform separate input mechanisms.

For a one-dimensional signal, Shannon’s classic formula re-lates the signal to noise ratio and bandwidth to the through-put in an analog channel subject to additive white Gaussiannoise:

C = B log2

(1 +

S

N

),

giving a transfer rate (capacity) of C bits per second from abandwidth (variability of the signal) B and a signal-to-noiseratio (reliability or repeatability of the signal) S/N . Al-

though this simple relation is only true for one-dimensionalchannels of a very specific type, the principles of variabil-ity and reliability generalise to more complex interfaces. Wecan thus split the analysis into two subproblems (Figure 1):

• Variability The “size” of the space of possible movementsthat a user can achieve, or more specifically the subset ofthe sensor measurements a user can reach. The larger therepertoire of movements available, the more efficient thecommunication will be.

• Reliability The precision with which actions can be re-produced at will. The more precisely state sequences canbe reproduced – from the perspective of the sensors – themore quickly information can be conveyed. If movementshave a variation which is not under the control of a user,the information capacity of the channel will be diminishedas some movements will be indistinguishable.

Two further concepts flow from these two basic factors:

• Transmissibility How quickly and easily the skills re-quired to interact with the system can be communicatedor discovered. This is affected by feedback from the sys-tem and by the ease of learning movements from others,and depends on factors such as similarity to previouslyencountered interactions, and the affordances of the inputdevice.

• Recognisability How effectively the software can mapsensor sequences into classes that represent user inten-tion. Even if the user can reliably generate a huge varietyof sensor sequences quickly and accurately, there must besystematic way of mapping these onto useful actions.

Each of these aspects must be addressed when designing aninput mechanism or when evaluating its performance. Al-though many of these issues can be addressed for specificinteraction styles (e.g. via Fitts’ law measurements for point-ing devices), it would be useful to have more general tech-niques which presuppose less about the interaction style tobe used. This is particularly the case for sensors for which“traditional” mappings such as targeting are not appropriate.

This paper focuses on solving the first problem: establish-ing the repertoire of motions which can be used for input.This problem is tackled by constructing a decomposition ofmovements into “motion primitives” which efficiently en-code the current state and its local temporal variation. A

similarity metric between these primitives is defined, andan experimental protocol which rewards users for perform-ing actions which maximise this metric is constructed, andthus rewards users for exploring more of the potential mo-tion space. The result is a map of possible motion primitivesfor a specific user and input mechanism.

Gesture designThe problems of designing “gesture-based” interactions –which often means any interaction involving non-traditionalinput devices – have been studied widely. Many papers havelaid out strategies for creating gesture systems, sometimesinvolving gesture creation by designers [20] and sometimesinvolving user-generated gestures [10], [19]. Although thesestudies are valuable in constructing viable gesture lexicons,they often proceed without reference to the technical con-straints of recognition.

What distinguishes the work presented here is the principledanalysis of spaces formed by the combined constraints ofusers and the sensing hardware, but without regard to spe-cific domains. The motion exploration techniques specifi-cally identify the technical limitations of input mechanisms.Any design process will have to incorporate domain-specificconstraints, but the analytic process we present provides asolid foundation from which to work.

EXPLORING THE SPACE OF GESTURESOne of the major issues with any new input technique is es-tablishing what we term a joint user-sensor space. In gen-eral, the space that is measured by a set of sensors (for exam-ple, a vector of pixel brightnesses, in a camera-based motioncapture system) is very different from the one in which usersare expressing themselves (which might be the position oftheir fingertip relative to their torso). Motions, as measuredby the sensor are sequences in sensor space – the space de-fined by the vector of values that a given sensor produces.Users perform actions in user space, which is a more dif-ficult concept to pin down; it clearly involves the muscularsystem but motions are not usually imagined or controlledas variations in joint angles or other physiological measure-ments. It might best be defined as the space in which usersthink they are controlling (i.e. an intentional space), and canvary from task to task. The role of proprioception, opticalflow, and other sources of feedback are more relevant in thecontrol of action than the resulting physical state changes.

Many parts of the sensor space correspond to impossiblereal-world configurations (in a camera system, imagine animage of a user with multiple heads), and many motions thatcan be performed cannot be sensed (for a camera, the posi-tion of the finger when the relevant hand is occluded by theother arm). Only those motions which are both possible andcan be sensed are communicative. These set of these mo-tions form the joint user-sensor space and ideally a systemwould use only motions within that set and would use mo-tions from the whole of this set. These criteria will optimisethe information transfer, by minimising useless signals, andby maximising the set of distinct signals that can be used.

Motion primitives: elements of gestureUsing a very liberal definition, a gesture can be considereda deliberate movement for the purposes of communicating

[8]. The problem to be addressed can then be stated as iden-tifying what set of “gestures” can be performed and sensedreliably; those gestures which lie in the joint user-sensorspace. By quantifying this space, we can obtain the super-set of all movements which could be used in a gesture-basedsystem. Gestures, in this general sense, are unbounded inlength. A gesture could last a few hundred millisecondsor several minutes, and treating “complete” movements istherefore difficult. The space of length-unbounded gesturesis obviously very large, and there is not an obviously correctway to make like-for-like comparisons between motions ofdifferent lengths.

If we imagine that gestures could be broken down into a setof elementary units, so that any movement could be analysedas the composition of those elementary “motion primitives”,the problem of establishing a consistent and meaningful ges-ture space would be much simpler. The simplest dissectionis to divide motions into equal length sections and analysethese separately. For example, all motions could be split intoequal 0.1 second blocks and analysis carried out on thesesections, and assuming that gestures can be resynthesizedby concatenating these sections, the gesture space would be-come a catalogue of these sections. In essence, we can forma codebook of motion vectors. This concatenative approachis merely the simplest of many approaches that could be en-visaged, and for some types of motion (e.g. rhythmic mo-tion, as discussed by Ijspeert et. al. [5]) there are more ele-gant decompositions. The subdivision of generable motionsinto components on which metrics can be defined is the keyto creating processes that can reward originality.

JOINT USER-SENSOR SPACESThere is not one constant joint user-sensor space; there aremany spaces and they are very variable in nature. Even ifthe sensing hardware and interpreting software remain con-stant, the human element of the interaction varies signifi-cantly. Different people have quite different motions thatthey are capable of and willing to make. Those who arehighly skilled in manual tasks may well have a richer rangeof motions they can draw upon and greater precision in theircontrol of movements. An expert in martial arts might, forexample, have much greater control over the velocity pro-files of their movements than someone who lacks such train-ing. Those who have motor impairments will have access toa smaller subset of the possible motion space. Individualswill also vary over time, both on short scales (for exampleas consequence of exhaustion) and on longer time scales (asan effect of growth, weight change or age-related reducedmobility). Capturing a systematic picture of the accessibleregions of the gesture space across population groups givesdesigners a powerful tool with which to customize interfacesto satisfy specific needs. We would like both to identify theset of communicative gestures, and to map the qualities ofthose within that set. These qualities might include the stren-uousness of movement, or the subtlety of the gesture.

Physical constraintsTemporary physical constraints will also affect the space ofmotions accessible. As an example, consider trying to op-erate a gesture recognizer while carrying a heavy bag inthe gesturing arm. The effect of physical constraints, such

as being seated versus standing, wearing constrictive cloth-ing, or being constrained within a vehicle can also be mea-sured. Some constraints will affect the reliability of move-ment, rather than variability, by adding noise to the process,and this will not be captured in a motion map. But manyconstraints limit the range of movement and can be mappedout in the joint user-sensor space.

Social contextsThere are also social constraints on gestures that can be per-formed in different situations in front of different audiences(see for example Rico and Brewster [14]). Movements whichare quite acceptable in private at home may not be so accept-able on the pavement of a busy street. The effect of these var-ious constraints on the input device at hand can be preciselyexamined by comparing spaces where audiences are variedand identifying which motions are excluded. For example, amobile phone with an accelerometer-based interface can beevaluated to identify which motions are usable in a privatesetting and which are then unreasonable in a public setting.

False positives and common movementsThe consideration of the joint-user space also leads to exam-ination of regions of the space rendered unusable becausethey are too frequently performed inadvertently. Motionswhich occur as part of everyday behaviour might be subtleand unobtrusive, but it can be very hard for a system to dis-tinguish between these movements executed as a deliberateattempt to communicate and those which were not intendedto be recognised. Foot tapping is a good example; this isa simple and unobtrusive movement, but a poor choice forcommunication if used alone. People tap their feet idly andsuppressing this behaviour to restrict it only for input pur-poses is burdensome. We will not treat this in the studiespresented, but it is well within the scope of the analyticaltechniques presented here to map out such reserved spacesby capturing motions during everyday activity and excludingthem from the available space.

IntuitivenessThe word “intuitive” is often used to describe interfaces with-out a clear definition of what it implies. If we take the mean-ing of intuitive to describe something which is familiar be-fore it has been used, we can analyze input systems in termsof which movements are spontaneously generated. By cap-turing the exploration process, not just the set of commu-nicative motion primitives, a picture of the motions whichthe interaction inspires can be built up. If a new input mecha-nism is proposed, and we wish to choose some set of motionsto use as controls, those which occur “naturally” – thosewhich are afforded by the physical construction of the inputdevices or the feedback provided – will minimise the bur-den of learning to control the device. This “seed set” canbe identified from the motions which are generated earlyin the motion exploration process. Those motions which,across a population, are only generated after other move-ments have been exhausted are presumably less obvious thanthose which are generated at the beginning. Primitives canbe ranked according to their “time of invention” and thisranked repertoire can then be used either to design gestures– by sequencing elements with low rank – or to test the in-

tuitiveness of a proposed gesture by computing the averagerank of its component primitives.

REINFORCEMENT FOR EXPLORATIONThe basis of the exploration process is reinforcement learn-ing, where the aim is to reward the novelty of a behaviour,rather than its similarity to a desired behaviour. Rather thanshaping behaviour, we seek to draw candidate motions fromusers’ minds, encouraging them to vary their movement asmuch as possible. This is reminiscent of the “superstitiousperception” process discovered in Skinner’s early work withpigeons. This demonstrated that animals could be condi-tioned into generating very peculiar behaviours, if a rein-forcer such as food, was released in a manner completelyunrelated to the behaviour [16]. Recent work by Gosselin et.al. [4] built on this to obtain images of archetypal objectsby averaging the result of a selection process where subjectsculled a set of purely random images to include only thoseresembling the imagined object. The Cybernetic Serendipitysystem [12] is a very early example of a system – in this casean installation – which responds specifically to novelty in in-put, triggering the movement of mobiles when musical inputwas sufficiently different from previous patterns the systemhad “heard”.

Our system rewards users for novelty. The hypothesis un-derlying this approach is that if users are reinforced whenthey do something novel, they will eventually exhaust therange of distinguishable behaviours. Obviously, the abilityto explore the space is dependent on the imagination of theparticipants, and since the only feedback is related to nov-elty, any variation must by driven by the user.

MODELLINGAny analysis of the joint user-sensor space must rely uponsome model of that space. Such a model is an assumptionabout how we expect the user-sensor system to behave, andthis choice will constrain the results that can be obtained.Although any analysis cannot be assumption free, we canmake the exploration as general as possible by making themodelling assumptions as unrestrictive as possible.

The representation of the movement section should com-pactly capture the nature of the movement; a trivial record-ing of sensor values is very simple to implement but for mostsensors will require a great deal of data to represent what isa much simpler underlying process. Identifying a reasonableset of latent variables – compressing the representation – willreduce the potential for “over-counting” the set of availablemotions. The more compactly a representation can capturethe space of movements without distortion, the more mean-ingful distance metrics in that space become.

Sequencing of primitivesThe motion sequencing approach could, in its most trivialform, be reduced to analysis of a sequence of static poses; inthe case of a camera system, this would be analysing videoas a “bag of frames”. This, however, ignores the fundamen-tally dynamic nature of human movement. Human move-ment is a continuous physical process driven by muscularmotion and a methodology which takes a static approachgreatly over-counts the space of possible motions, because

feasible movements lie along smooth, achievable trajecto-ries. A representation which captures temporal aspects isessential to modelling the space of motions well.

Simply modelling the transitions between static poses is anunsatisfactory way to map out the gesture space. We couldfor example, identify a range of limb poses and ask usersto transition between random pairs. Even if the poses werewell-selected, this analysis would be dominated by transientphenomena as static poses are left and entered, and wouldfail to model the co-articulation between dynamic move-ments. Dynamical models of human movement often ap-proximate limb motion as a spring-like system ([6], [1]) whichhave clearly continuous motion; a hand in oscillation cannotbe well approximated by transitions between static poses inthat cycle.

Polynomial approximationOne elementary and compact dynamical model is to repre-sent the current state state along with (smoothed) time deriva-tives. A limb measurement, for example, which included po-sition, velocity, acceleration and jerk (the 3rd order deriva-tive) can represent a wide variety of motions, and this rep-resentation captures the physical essence of motion, if weassume limb motion can be well-modelled by a linear differ-ential equation over short periods of time.

We generally do not, however, want to assume a mappingfrom sensor space to the physical movement space, as thismust necessarily be quite specific, and limits the generalityof the exploration technique. Instead, the motion vectors canbe represented directly as sensor values, along with their var-ious derivatives. By dividing each one-dimensional signalinto segments, and fitting a polynomial to those segments,motions can be represented in a simple form which encodesthe time-evolution in a physically-relevant way.

For sensors with very high inherent dimension (like cam-eras), this is not a practical approach, and some pre-processingmust be applied to extract relevant features (e.g. by trackingobjects). Lower-dimensional sensors, such as accelerome-ters, pressure sensors or styli can be directly mapped with-out feature extraction. An extension of this technique wouldbe to automatically learn a low-dimensional manifold froma series of sensor measurements, and then use this learnedlow-dimensional mapping as the input to the motion explo-ration system. Assuming that the mapping extracted rel-evant latent variables, this would extend the technique tovery high-dimensional systems without having to constructa manual model.

Similarity metricsThe exploration of the space hinges on rewarding users fordoing something novel. This requires a careful definitionof novelty, as once chosen, the experimental results will de-pend upon this originality metric. One simple model is theEuclidean distance between motion vectors. This, however,performs poorly when the scales of different elements of thesensor vector are widely distributed.

To calculate the novelty of the movement, a suitable similar-ity metric is required. The choice of metric is an assump-tion; we have chosen here to use the mean distance of the

nearest k neighbours using the Mahalanobis metric (or gen-eralized interpoint distance). The Mahalanobis distance iseffective because it accounts for the differences in scale ofthe various variables and linear interdependencies betweenthem (see Figure 2). Using the mean distance to the k near-est neighbours mitigates the impact of outliers that mightaffect the results if distance to the nearest point was simplyused.

Original space Whitened space

Figure 2: The Mahalanobis metric gives get a scale-independent mea-sure. This is equivalent to decorrelating and rescaling data to a spherewith unit covariance before comparing. The nearest k neighbours un-der this metric are found, and the mean distance to them becomes theoriginality value of that point. If this value is large enough then M∗

(green star) joins the repertoire.

Other scale-invariant metrics could also be used; the co-sine metric

(s = cos(θ) =

Mi·Mj

‖Mi‖‖Mj‖

)might be effective

for high-dimensional spaces, but the Mahalanobis distanceis well-understood, robust and compensates for linear corre-lations.

Testing the quality of modelsMaking assumptions about the nature of movements is in-escapable in the construction of models. However, it is pos-sible to evaluate the quality of these assumptions system-atically. Given a concatenative model with fixed windows,a data set can be processed with a range of windows andmovement models, and the model error can be comparedagainst the compression achieved (e.g. by considering therate-distortion curve). A model which concisely representsthe motions with minimal error provides an optimal space inwhich to judge originality.

CONCRETE IMPLEMENTATIONTo apply these techniques, we built a system for perform-ing motion exploration using an inertial sensor. This sensorcan be attached to various body parts to sense motions ofthose parts, and the system can map out motions availableacross users and sensor locations. This system uses onlyaudio feedback as a reinforcer, so that participants do nothave to divert their visual attention. In our system, we havedefined a motion primitive as the measurement of vector ofsensor values at a point in time, along with a number of theestimated derivatives of this action – the polynomial approx-imation described above. This combined vector representsa snapshot of the evolution of the sensor values at a partic-ular time. A 4th order model was used, which captures thecurrent state of a sensor stream and its first three derivatives.A least-squares fit of a polynomial of appropriate order isperformed on a sliding window. Derivatives of order greater

than three are likely to be subject to poor fitting, and alsohave less obvious significance as high order derivatives arenot something that humans are accustomed to controlling.

In the measurements presented, a 0.65 second running win-dow was used, as a compromise between capturing the finestdetails of movements and capturing movements of sufficientmagnitude to be useful for communication. This choice isobviously an assumption, but this assumption is numericallyjustified in the results section.

ImplementationThe novelty detector captures data from a SHAKE SK71

inertial sensor, which provides 3-axis accelerometer, gyro-scope and magnetometer outputs. The accelerometer, gy-roscope and magnetometer measure in the range −6g – 6g,−900 – 900 degrees/second and −0.2mT – 0.2mT respec-tively, all with 12 bit resolution.

Figure 4: The SK7 inertial sensor, as mounted on the hand. The elasticmounting minimises slippage.

Each sample therefore consists of a 9 element vector S =[ax, ay, ax, gθ, gφ, gψ,mx,my,mz]. We do not attempt tomap these values into an alternative space (e.g. to map to de-vice orientation), partly to reduce the assumptions required,but also to capture all motions that the hardware can sense(so including motions involving linear acceleration such asflicks or lunges).

The inertial sensors are sampled synchronously at 32Hz; thesensor streams are filtered on the hardware to bandlimit thembefore decimating from an original sampling rate of 1024Hz.A 32Hz rate is sufficient to capture the details of most limbmovement. At a 0.65s window length one motion primi-tive is therefore extracted from 21 samples of data (the win-dow time is adjusted slightly to be exactly 21 samples inlength). A bank of Savitsky-Golay filters are applied to thesignal, each designed to estimate one smoothed derivativeefficiently. These coefficients are concatenated into a singlevector M∗ =

[ax,

ˆdaxdt ,

ˆdaxdt2 , . . .

]with q = 9 × 4 = 36

dimensions, which represents the motion in sensor space.

When computing the novelty of a motion, we represent therepertoire (or codebook) of distinct motions as a set of vec-tors M1 . . .MN . These are the set of motions which the“novelty detector” has determined are distinct enough to main-tain. It is initially empty, and is populated as the motion cap-1http://code.google.com/p/shake-drivers/

ture progresses. As described, the mean Mahalanobis dis-tance to the k nearest neighbours is used; thus to computethe distance between a new vector M∗ and an existing sam-ple in the repertoire Mj , we compute

d(M∗,Mj) =√

(M∗ −Mj)TΣ−1(M∗ −Mj),∀Mj ∈M,

where Σ is the covariance matrix of the repertoireM1 . . .MN .We then find the k smallest distances d(Mj ,M

∗) and thelog mean of this, dµ(k)(M∗) = log 1

k

∑kj=1 d(Mj ,M

∗), isthe similarity to the existing samples in the repertoire. Ifdµ(k)(M

∗) > ∆, where ∆ is a threshold setting the “nov-elty cutoff”, then the sample M∗ is added to the repertoire.By adjusting ∆ and k the sparsity of the space, and thus thetime taken to exhaust it, can be controlled. ∆ is proportionalto a confidence interval on the interpoint distances.

In practice, calculating the matrix Σ, which represents thecovariance of the repertoire, is quite expensive, and is doneonly when a number of new samples have been added tothe repertoire. The current implementation computes thiscovariance matrix for every fifty new vectors added to therepertoire. The matrix is initialised to a covariance matrixestimated from a pilot trial; it is quickly replaced as the newrepertoire accumulates. As the covariance matrix is neces-sarily updated during the motion capture, the similarity met-ric changes slightly throughout the process. These changesare gradual and result in a smooth variation in the original-ity feedback. The adaptive nature of the metric maintainsreinforcement at a roughly constant rate over individuals.

Audio feedback: originality rewardThe “reward” for novel movements is an increase in the ap-parent liveliness of the audio feedback presented. Whena new element is inserted into the repertoire, the distancedµ(k)(M

∗) is accumulated into a leaky integrator, which de-cays exponentially at a constant rate. This leaky integra-tor is fed to the audio output to produce the positive feed-back for novel movements. We use a flexible granular syn-thesiser to produce the feedback. The integrator output ad-justs the density of grains and the spectral brightness of thesource waveforms such that more novel movements makebrighter, more intense sounds, which gradually decay awayto duller, sparser sounds as originality decreases. Eventuallythe sound fades away to a low hum when no new vectors arebeing added to the repertoire. The synthesiser is configuredto sample snippets of ≈ 40ms from a short excerpt of jazz.This results in an aesthetically pleasant texture which can bemodulated continuously with a degree of subtlety.

EXPERIMENTAs an example of motion space analysis, an experiment wasconducted with the inertial sensor based exploration system.The exploration was carried out in two conditions: condi-tion A, with the sensor attached to the dominant hand; andcondition B, with the sensor attached to the arm just abovethe elbow. The hypothesis is that the range of reasonablemotions should be smaller for the elbow than for the hand,because the available degrees of freedom are substantiallylower. The experiment is designed to test whether the pro-cesses presented here could quantify this difference reliably,

(a) (b)

OriginalRunning fitc

x

x2

x3

(c)

Figure 3: The process of extracting motion primitives. A short window of the signal is extracted (a), lowpass filtered (b) (original dashed, filteredin solid). An estimate of the value and derivatives at the central point is made (c) (for the lower blue curve in (b)). These coefficients become therepresentation of that movement. Only one axis is shown in (c) for clarity; the motion vector is the concatenation of these coefficients for all axes.

0.65 seconds0.65 seconds

Figure 5: Self organizing map, showing the log density of points on the projected motion space for all participants across all conditions. The breakoutsshow the video sequence for the nearest vector to that point, and the motion curves for that vector. The fitted curve (from the motion vector coefficientsMi) is shown as a solid line, the true raw data as a dotted line, and the area between is shaded.

0.08 0.16Compression ratio

0.0

0.2

0.4

Media

n R

MSE (

std.

devs.

)

ConstantLinearQuadraticCubicQuartic

Figure 6: Median root mean squared error of Savitsky-Golay fit ver-sus “compression” (1/window length), computed for the entire dataset.Choices near the origin represent a good trade-off between error andcompactness. The cubic, 0.65 second model used is shown as a blackstar, near the optimum.

and to provide a pool of captured data for exploratory anal-ysis.

The experiment was conducted in a laboratory. There wereN = 20 participants, 16 male, 4 female, recruited froma pool of computer science students, with an age range of18–40. All participants gave informed consent. The SK7sensor pack was affixed with an elastic strap (see Figure 4).This prevented the sensor from being moved relative to thebody and so eliminates symmetric motions where the samemovement is performed with the sensor axes in a differentalignment and also prevents users from performing “mo-tions” such as dropping the device or throwing it into theair. Participants were requested to keep their legs and torsorelatively steady and concentrate on moving their dominantarm. We did not aim to completely suppress natural mo-tion of the rest of the body, but to focus intentional move-ment on the relevant area. The participants motions werealso recorded with a video camera. This is not used as in-put, but the captured video stream is synchronized with theSK7 sensor stream. This provides a visual reference for themovements performed.

All participants completed both conditions in counterbal-anced order. Each condition was conducted as a set of threesubtrials, lasting three minutes, with a break between each.Each subtrial continued from the point the previous one left

0 200 400 600Time (s)

0

2500

5000

Num

ber

of

vect

ors

Number of vectors

Figure 7: The total number of primitives versus time, for all partici-pants. Measurements for condition A (hand) are shown as solid lines,measurements for condition B (elbow) are shown as dashed. The graphshows a polynomial fit to the true data; the original raw data is shownas a dotted line for comparison.

0 200 400

Time (s)

80

0

log(|det(Σ)|)

Volume of the space explored over time

Figure 8: The “size” of space explored. This graph shows the log de-terminant of the covariance matrix of vectors seen so far for each par-ticipant log(| det(Σ)|). | det(Σ)| is proportional to the hypervolume ofthe covariance ellipse; larger values indicate more of the space beingexplored. A value of 0 (|det(Σ)|=1) means that the volume of the co-variance ellipse was equal to that of the covariance of the entire dataset. The Box plot to the right shows the final distribution of values.

off, for a total of nine minutes of exploration in each con-dition. The conditions were broken into subtrials to combatfatigue from long performances. The implementation usesk = 5 nearest neighbours and a value of ∆ = 1.50 for thenovelty cutoff.

ANALYSISIn total, across both conditions, there were 203671 36 di-mensional motion vectors captured from a total of exactly6 hours of exploration. Figure 6 shows error against modelsize for a range of polynomial orders and window lengths;a 0.65 second cubic model is close to the optimum for low-order polynomial models.

The number of unique vectors generated by each participantin each condition is shown in Figure 7. The vectors gener-ally asymptotically approach a value which varies by par-ticipant, with a mean value ≈ 5000. Because the process

AJ JM NU

ELBOW

HAND

Figure 9: Individual (log) densities for a selection of participants. Up-per row shows the maps for the elbow condition, lower shows the mapfor the hand condition.

Figure 10: (left, red) Average total acceleration across the space of mo-tions. The three smaller images below show the individual maps for thex, y, z axes respectively. (right, blue) Average total angular velocity ofmotions. The lower images show the maps for each of the three axes.

is adaptive (the covariance matrix is continuously recom-puted), there is not a huge variation between the conditions.This is to be expected, as the feedback is designed to fall offat a roughly constant rate regardless of the absolute motionspace explored, in order to maintain user motivation. How-ever, Figure 8 shows a plot of the estimated “volume” ofspace explored in each condition; it is clear from this thatthere was a much smaller portion of the space explored inthe elbow condition. This clearly supports the hypothesisthat the range of available motions is reduced in the case ofthe elbow as compared to the hand.

Motion mapsThe motion vectors forming the repertoire are difficult to vi-sualise directly, primarily because of their high dimension.The space can be more easily visualised by performing di-mensional reduction to a 2D layout. In our analysis, wehave used a standard self-organizing map ([7], [9]) on thewhitened feature vectors to reduce the 36-dimensional vec-tors to a 2D map. The maps used here are discrete 48×48 ar-rays with a rectangular neighbourhood function. Each point

Hand Elbow

Union

Intersection

Rank

All

Energy

(a) (b)

(c) (d)

(e)

(f)

Figure 11: (a,b,c,d) Union and intersection of all motion maps for thetwo conditions. Each map is the sum/product of the normalised den-sities for each participant. (e) Map coloured by average time of in-vention; brighter areas were discovered later in the process. (f) Mapcoloured by average energy in |G/s|. Brighter colours involve moreviolent motion.

Figure 12: Movement energy (root-mean-square derivative of acceler-ation), in G per second, versus Mahalanobis distance (linear regressionas a red line). There is a strong inverse correlation between how violenta movement is and how likely users are to perform it.

on this map represents a vector in the original high dimen-sional space, and the points are organized so that nearby vec-tors on the 2D map are nearby in the high-dimensional space.For consistency between plots, the dimensional reductionwas computed from all data recorded (across all participants,and all conditions), and each separate condition/participantmap was obtained as a subset of this master projection. Allof the visualisations are thus directly comparable.

We constructed an interactive tool which uses the synchro-nized video recordings to show snapshots of motions. Thistool finds the point on the map nearest the cursor, looks upthe index in the video and plays the relevant section, or gen-erates a composite image for that motion. The compositeimages show the moving parts of the frame in colour, fadingfrom purple to yellow, against a monochrome background.

Figure 5(a) shows a map of the space explored by all users.Maps for individual participant/condition pairs can be pro-duced; some examples are shown in Figure 9. The maps canbe coloured by objective measurements; Figure 10 showsmaps coloured by total acceleration and by total angular ve-locity. Motions which involve twisting around the forward

axis can, for example, easily be picked out from these maps.Figure 11(e) colours movements by time-of-invention (move-ments that were generally found later are coloured brighter).Comparing with Figure 10 it can be observed, for exam-ple, that movements involving twisting tend to be discoveredlater. Figure 11(f) shows a map the energy of movementsestimated from the root mean square derivative of accelera-tion, where the derivatives are estimated from the raw datacorresponding to the vectors at that position. Figure 12 plotsthis energy measure against Mahalanobis distance (i.e. howclose the motion was to the overall distribution of the reper-toire). It can be seen that energetic movements are distinctlyrarer than calmer ones. Densities can also be formed fromthe (continuous) intersection, union or difference of a pair ofmaps; for example Figure 11(a) shows all motions that usersperformed in the hand condition, while Figure 11(c) and (d)shows movements performed by all users in the hand andelbow condition respectively.

One notable qualitative result is that the motions on videobear little relation to the sensor measurements. Arm move-ments with very different scales can result in very similarinertial sensor sequences and vice versa; the mapping be-tween inertial space and human-centered space is complexand non-obvious. This partially explains the difficulty inconstructing inertial sensor based interfaces beyond simplemetaphors such as shaking and tilting.

These analyses demonstrate how movement spaces can beefficiently categorised and compared. Motions can be anal-ysed with cross-referenced objective metrics to identify fea-tures of interest or relevant trends. This study focused onvariations across body locations, but other variables of inter-est, such as sensor types, demographics, social context, orphysical encumbrances are equally open to exploration.

Estimating information transfer ratesOur process estimates the repertoire of possible motions. Itidentifies the variability of the channel, under the restrictionsof the motion primitive model. Measuring the informationcapacity of an input system would require measures of thevariability and the reliability. Measuring reliability (the pre-cision with which motions can be controlled or repeated) isbeyond the scope of this paper, but an analysis which couldidentify reliability of motions across a motion space couldbe combined with the techniques presented here to providebounds on information transfer rates.

ApplicationIn applying these techniques, there are two essential ele-ments: defining novelty and providing reward. We haveshown that simple modulation of responsiveness is a suffi-cient reward to drive users to explore different behaviours.The major challenge is deriving suitable novelty metrics, andthe key issue there is constructing a decomposition of senseddata into comparable units which also capture the essence ofthe communication. With many simple input mechanismsthe fixed time polynomial representation given here wouldbe suitable (e.g. with a pressure sensor) but other modali-ties will require more careful definition of primitives whichcompress sensed data into meaningfully comparable units.

CONCLUSIONSThe exploration of potential input mechanisms is often nec-essarily driven by ingenuity in imagining metaphors. Whena new avenue for input presents itself, interactive systems de-signers have to conceive of ways of building an interactionaround the mechanism; conversely, input devices are oftendesigned to fit an established metaphor. While this is stillessential, the techniques given here extract maps of motionprimitives with only very limited assumptions and create aplatform for comparing and analysing input mechanisms.Reinforcement of novelty is a systematic and straightfor-ward way of identifying joint user-sensor spaces.

This approach does not provide a complete strategy for de-signing for new input devices. Meaning must still be as-cribed to gestures, which requires the imagination of design-ers in creating idioms; recognition techniques must still becreated to categorize movements accurately; and the role offeedback in the interaction process is unaddressed. How-ever, the techniques laid out here present designers with apalette of suitable elements which can be woven into aninteraction; provide measures for comparing the capabili-ties of input mechanisms under different constraints; andgive implementers a suite of tools with which to rate anddissect proposed gesture sets. A full information theoreticanalysis will require the development of techniques for sys-tematically analysing the precision of movements across awhole space. Probabilistic methods are increasingly beingused to analyse interaction, but such methods often requirea space over which measurements can be normalised. Thishas been previously generally inaccessible, but the processgiven here provides an efficient way of extracting a suitableglobal space of movements.

We have shown here how the exploration process can be ap-plied to an inertial sensor but these techniques are easy toapply to other sensors. The techniques provide designerswith a rich set of exploratory and analytical tools for build-ing movement-based systems. The decomposition of con-tinuous gestures into simple primitives makes it possible tosystematically compare complete gesture spaces. The out-puts of these analyses are well-founded objective measureswhich necessarily take into account both the restrictions ofinput devices and the limitations of the humans using them.These can be used to compare contexts, populations, devicesand metaphors within a coherent framework.

ACKNOWLEDGEMENTSThis work was supported by a fellowship from the Scot-tish Informatics and Computing Science Alliance (SICSA)and by the PASCAL2 Network of Excellence. All (non-video) data from this project is freely available online atwww.dcs.gla.ac.uk/˜jhw.

REFERENCES

1. R. M. Alexander. Principles of animal locomotion.Princeton University Press, 2003.

2. J.-D. Fekete, N. Elmqvist, and Y. Guiard.Motion-pointing: target selection using ellipticalmotions. In CHI ’09, pages 289–298. ACM, 2009.

3. P. M. Fitts. The information capacity of the humanmotor system in controlling the amplitude ofmovement. J. Exp. Psych., 47(6):381–391, 1954.

4. F. Gosselin and P. G. Schyns. Superstitious PerceptionsReveal Properties of Internal Representations.Psychological Science, 14(5):505–509, 2003.

5. A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learningattractor landscapes for learning motor primitives. InNIPS’03, pages 1523–1530. MIT Press, 2003.

6. R. J. Jagacinski and J. M. Flach. Control theory forhumans : quantitative approaches to modelingperformance. L. Erlbaum Associates, 2003.

7. J. Kangas, T. Kohonen, and J. Laaksonen. Variants ofself-organizing maps. IEEE Tran. Neural Networks,1(1):93 –99, 1990.

8. A. Kendon. Gesture: Visible Action as Utterance.Cambridge University Press, 2004.

9. T. Kohonen. Self-organized formation of topologicallycorrect feature maps. Biological Cybernetics,43:59–69, 1982.

10. C. Kray, D. Nesbitt, J. Dawson, and M. Rohs.User-defined gestures for connecting mobile phones,public displays, and tabletops. In MobileHCI ’10, pages239–248, 2010.

11. I. S. MacKenzie. Fitts’ law as a research and designtool in human-computer interaction. Hum.-Comput.Interact., 7:91–139, 1992.

12. G. Pask. A comment, a case history and a plan. InJ. Reichardt, editor, Cybernetics, Art and Ideas, pages76–99. New York Graphic Society, 1971.

13. K. Pryor. Don’t Shoot the Dog: The New Art ofTeaching and Training. Ringpress Books, 2002.

14. J. Rico and S. Brewster. Usable gestures for mobileinterfaces: evaluating social acceptability. In CHI ’10,pages 887–896, 2010.

15. A. Scoditti, R. Blanch, and J. Coutaz. A noveltaxonomy for gestural interaction techniques based onaccelerometers. In IUI ’11, pages 63–72. ACM, 2011.

16. B. F. Skinner. ‘superstition’ in the pigeon. Journal ofExperimental Psychology, 38(2):168–172, 1948.

17. R. W. Soukoreff and I. S. MacKenzie. Towards astandard for pointing device evaluation, perspectives on27 years of Fitts’ law research in HCI. Int. J.Hum.-Comput. Stud., 61:751–789, 2004.

18. J. Williamson and R. Murray-Smith. Pointing without apointer. In CHI ’04, pages 1407–1410. ACM, 2004.

19. J. O. Wobbrock, M. R. Morris, and A. D. Wilson.User-defined gestures for surface computing. In CHI’09, pages 1083–1092, 2009.

20. K. Yatani, K. Tamura, K. Hiroki, M. Sugimoto, andH. Hiromichi. Toss-it: Intuitive information transfertechniques for mobile devices using toss and swingactions. IEICE - Trans. Inf. Syst., E89-D:150–157,2006.