Sketch-Based Robot Programming

Christine M. Barber2, Robin J. Shucksmith2, Bruce MacDonald2, Burkhard C. Wunsche1

1Department of Computer Science, The University of Auckland, New Zealand.2Department of Electrical and Computer Engineering, The University of Auckland, New Zealand.

Email: {cbar141,rshu008}@aucklanduni.ac.nzb.macdonald@auckland.ac.nz,burkhard@cs.auckland.ac.nz

Abstract

Robots are rapidly becoming a part of everyday life and have now moved from industrial environmentsto household, medical and entertainment applications. In order to make full use of robots new interfacesneed to be developed, which allow inexperienced human users to instruct (program) robots, without havingto understand programming and the underlying electronics and mechanics. In this paper we present anovel sketch-based interface for robot programming. We have identified applications which are difficultto represent algorithmically, but can be easily represented with sketch input. We then define a rangeof sketch impressions allowing the user to define a wide range of behaviours within these applicationdomains. Our system uses a Pioneer robot with an arm and a fixed overhead camera. The user sketchesinto the camera view and the sketch input is interpreted, mapped into the 3D domain, and translatedinto robot interactions. Current applications include specification of robot paths and obstacles, coveringregions (e.g., patrolling in security applications or seed sowing in agricultural applications), and directingthe robot arm, e.g., to pick up objects. A user evaluation of the system demonstrates that the interface isintuitive and, with the exceptions of controlling the arm, all interactions are perceived as easy to perform.

Keywords: robot programming, sketch-based interfaces, human-robot interaction, augmented reality

1 Introduction

Robots are becoming increasingly prevalent intoday’s society, but use of robots is limited by thedifficulty in directing (programming) them. Mostproduction robots used in consumer and industryapplications today are fully autonomous, with aclearly defined hardcoded set of functionalities.

Many different approaches to human-robot in-teraction have been explored. The traditionalapproach is to program robots using one of manyprogramming languages available. This requiresexpert knowledge of the robot hardware and alsosoftware programming skills. This process is wellabove the knowledge of the ordinary consumer andit is a complex and time consuming process even foran expert. Alternatively, robots can be controlledusing joysticks or a keyboard, but this limits thenumber of possible tasks, and requires training andmental skills (user must take on the robot’s view).A novel intuitive interface is needed that simplifiesinteraction with and control of robots.

Gesture recognition and speech recognition haveboth been suggested as possible interfaces [1]. Al-though these are natural communication methods

978-1-4244-9631-0/10/$26.00 c©2010 IEEE

for humans, speech and voice commands sufferfrom innate ambiguity so more general behaviourand tasks are hard to convey, e.g., to accuratelyspecify an area for the robot to control. Analternative approach is sketch-based interfaces,which are rapidly gaining in popularity with theincreased uptake of consumer-level touch sensitivedevices. The premise is that a user draws afree form picture on the screen with a mouse orstylus. These sketches are then interpreted bysoftware, providing more complex functionality.Sketch-input is a promising approach because ofits intuitive pen-and-paper metaphor. Sketchingavoids the need for 3D input devices, does usuallynot require a 3D mental model, is supported bymany social networking tools (hence potentiallyfacilitating remote and collaborative work), andis available on a wide variety of platforms suchas PCs with Windows 7, iPhones, and interactivewhite boards. Furthermore it has been shown thatsketching encourages creativity [2] and enablesusers to concentrate on the overall problems ratherthan details [3]. By combining sketch and a realtime camera representation of the environmentan even richer experience can be provided. Forinstance, objects for the robot to manipulate canbe identified easily by sketching a circle around

their image in the view. Note that this idea canbe considered as a type of Augmented Realityinterface, which are becoming more and morepopular in robotics.

This project aims to design and create a sketch-based interface that can be used with minimalinstruction to control a robot to perform tasksrelating to real world applications. Section 2covers previous relevant work on human-robotinteraction and sketch-based interfaces. Section 3identifies requirements for our applications, whichare used to motivate the system design presentedin section 4. We explain two components of thesystem, the sketch recogniser and the robot controlin more detail in section 5 and 6, respectively.Section 7 gives implementation details. The resultsof evaluating our system are found in section 8 andare followed by the conclusion and an outlook onfuture work in section 9.

2 Related Work

2.1 Human Robot Interaction

A variety of interfaces has been developed tofacilitate human-robot interaction including speechrecognition, gesture recognition, facial recognition,and interfaces that use peripherals such as atouch screen, mouse, joystick or keyboard. Speechrecognition has been successfully employed in ap-plications with a limited number of commands,e.g., to find our where people want to go and guidethem [4]. The ability to ask questions and ges-ture recognition, such as detecting where the userpoints to, can be used to resolve ambiguities [5].Other interfaces to control robots include joysticksand peripherals that take inputs from buttons [6].These interfaces are intuitive and hence easy touse for most people. However, they only allowinteraction via a set of predefined functions sincethe input has a clearly defined semantic. Touchscreens are being used by the U.S. Army tocontrol Experimental Unmanned Vehicles (XUVs),since they reduce supervisory workload, give newoperational flexibility and increase the span ofcontrol [7]. Other applications include search andrescue robots [8] and social robots [9]. However,touch screen interfaces can suffer from too muchinformation being presented on the screen [8].

2.2 Sketch-Based Interfaces for Ani-mation and Robot Control

The most common application of sketching forcontrolling motion is character animation. Ani-mated 3D characters and robots are both usuallyrepresented as a skeleton of rigid bones connectedby joints. Steger represents 2D motions with

directed motion paths [10]. Disparate motionsare synchronized using events which are indicatedby time stamps along the motion paths. MotionDoodles allow the user to sketch a motion pathfor a sketched character which can consist of up toseven components with predefined functionalities(head, body, arms, etc.) [11]. The second approachfor sketch-based animation is to draw key posesand extract motion from them. This can beachieved by sketching skeletons for key framesand interpolating them [12, 13]. An alternativeapproach is to draw perspective 2D sketches ofdifferent poses from a single view point and totranslate differences in bone length into 3D posi-tions and hence motions [14].

Interfaces that use sketches to control robots havebeen devised for robot navigation, controllingmultiple robots and controlling household robots.Skubic et al. achieve robot navigation by drawinglandmarks on a PDA [15] and control formations ofmultiple robots by sketching different symbols [16].Sakamoto et al. control vacuuming robots by strokegestures and sketching motion paths and operationareas [17].

3 Requirement Analysis

The goal of our research is to provide an interfacefor applications where an inexperienced user needsto control a robot using an unconstrained set ofmotions. Five key application areas were identi-fied: household robots for tasks such as mowing,vacuuming and fetching objects, surveillance wherethe robot patrols and protects a certain area,industry where a robot may need to move arounda factory and manipulate products with a roboticarm, agriculture where a robot might need tosow seed, monitor crops and apply pesticides, andeducation to provide users with insight into robotcontrol.

These applications can be characterised by thefollowing tasks: (1) Motion paths and target:many household and industrial settings (e.g., au-tomated forklifts) require the robot to move to agiven object, either directly or via a given path.There must be a simple way of specifying realworld points for the robot to manoeuvre to andpaths to follow. Moving of objects requires thesequences of tasks to be specified, for instancemove to the object, pick it up and then moveelsewhere in the room. Furthermore, in factoryand outdoor environments robots often must stayoff certain areas, e.g., in order to avoid dangerousobstacles or collisions with pedestrian. There mustbe a method of indicating areas that the robotmust not travel through. (2) Area coverage:In surveillance, agriculture and household tasks,

such as cleaning, the robot must cover a prescribedarea, which can have an arbitrary shape. (3) Pickup and drop off: In household and industrialapplication it is common for robots to pick upan object and drop it off at another location.Examples are nursing robots fetching medicine andautomated forklifts shifting pallets.

In all applications the robot should be able tobe stopped immediately. This is particularlyimportant in large scale industrial robots wheresafety is critical. The ability to pause the robotis also desirable, e.g., for maintenance tasks inindustrial applications, or for cleaning applicationswhere users may wish to stop the robot whilst theyare in the room.

4 System Design

4.1 Physical Setup

In order to control the robot in the environment aview of the environment is required. One solutionis to provide a map, e.g., created by sketch input,or automatically using a SLAM (Simultaneouslocalization and mapping) algorithm. We useinstead a ceiling mounted calibrated camera since areal view of the environment is most intuitive, weused a similar set-up in previous research whichaccelerates development, and because cameras areinexpensive and often already exist in indoor envi-ronments for security purposes. We use a Pioneer3 robot with attached arm. A fiducial markeris attached to its top so it can be tracked bythe overhead camera, and its position computedusing using ARToolKitPlus, which is a part ofARDev [18]. ARDev is also used to display theimage of the current view from the camera in real-time on the environment window in our interface.Two input windows exist as illustrated in figure 1:the real-world view from the overhead camera anda 3D OpenGL window displaying a robot model.

Figure 1: The environment window (left) and 3D robot

model window (right) for sketch input.

4.2 Logical Setup

Figure 2 shows an overview of the logical designof our system. ARDev renders the sketches drawnby the user within the environment window and

renders the 3D robot model. When input fromthe mouse or touch screen is received the 2Dsketched input is mapped to ARDev’s 3D worldco-ordinates. An event is then fired in ARDevto get the mouse co-ordinates which are sent tothe Sketch Manager. The Sketch Manager isresponsible for the recognition and classificationof sketches and how ARDev will display thosesketches. If the sketch corresponds to any of thenine sketches able to be recognised by the systemthe Sketch Manager fires an event to the RobotControl. The Robot Control then interprets thesketches so they can be executed as behavioursby the robot using the robot’s current positionobtained from ARDev.

Figure 2: System Design.

To control the robot, the user sketches on the twoviews shown in figure 1. The first view is of theenvironment. In this window the sketch recogniseris notified of any mouse motion or clicks andrecords each sketch. Once each sketch is recordedit is classified by the sketch recogniser to determinewhat action has been drawn. Upon successfulclassification, events are sent to the robot controlsystem where they are recorded. Each event isprocessed in sequence by interpreting its associatedsketch as real world behaviour. This involvesprojecting the sketch into real world points as wellas processing of the sketch to break it down intoa series of actionable tasks. ARDev provides therobot control subsystem with information aboutthe robot’s current location by detecting a fiducialmarker in the image. Using this localisationinformation, the Pioneer robot can then be con-trolled via a robot interface to perform the requiredbehaviour.

The second view displays a 3D robot model; mousemotion in this view is handled by the robot modelsubsystem. This allows the user to manipulate a3D model of a robotic arm using sketches. Thepositions of this arm are passed to the robot controlwhich handles updating the position of the physicalarm to match the robot model.

4.3 Sketch Symbols and Semantic

The left part of figure 1 illustrates the typesof sketch input recognised by our system. Thesymbols cover all of the functionalities describedin section 3. In order to simplify usage and recog-nition we chose the simplest and most intuitiverepresentations we could find.

For directing a robot to a point a cross is used sincethis avoids ambiguities with other functionalitieswhich could occur when using, e.g., a small circle.Paths are defined with an arrow since this avoidsambiguities compared to lines used in Sakamotoet al.’s solution [17]. Arrows situated on therobot rotate it to the arrows heading. A regionof interest (closed contour) is used to indicate anarea for the robot to patrol/cover. Double headedarrows are used to specify routes to patrol. Forreasons of simplicity a triangle is used to initiatea camera capture while a rectangle specifies anobstacle or area to be avoided. A more generalrepresentation for the future might be a closedcontour with a cross or scribble inside. For stopand pause a square and parallel lines were used,respectively. The reasoning being a large numberof people already understand the functionalityintended, based on their experience with musicplayers. Following a similar logic scribbles wereused for deletion since this functionality is similarto using an eraser on paper, blackboards and indrawing programs.

The window with the 3D robot model allows usersto rotate components by sketching a line startingon the part to be moved. In our case the onlymovable part is the arm. The user can rotate thewhole arm, the three hinge joints and can open andclose the gripper. Sketch input is translated to a3D rotation by using the trackball concept [19]: avirtual sphere is placed over the window centeredat the joint to be rotated. The start and end pointof the sketch are then projected onto the spheresurface. Connecting the centre of the rotation(the selected joint) to the points results in twovectors. The rotation axis is obtained from thecross product of these vectors and the rotationangle from the angle between them. If the joint isnot a ball joint only rotations around the movabledirections are applied.

5 Sketch Recogniser

The Sketch Recogniser has to recognise single anddouble sided arrows, scribbles, closed contours,and various symbols. Recorded sketches are firstprocessed to compute basic properties and are thentested with a series of classifiers. The sketch is thenstored as either a classified or unclassified sketch.Notification through call-back to the robot control

subsystem occurs whenever a sketch is created ordeleted.

5.1 Sketch Processing

Once the sketch stroke has been captured it isprocessed to reduce noise. This occurs in twostages, first the raw data is smoothed by an elevenpoint weighted moving average, which acts as alow pass filter removing small perturbations in theinput which can arise due to an unsteady hand.Next the sketch is broken down into a series ofstraight line segments using a split and mergetechnique. This involves continuously splitting theset of raw sketch points if the straight line error istoo large. This is continued until lines are fitted toall points with an acceptable error. This reducesnoise and computation time, as every pixel doesnot have to be considered when performing sketchrecognition or when displaying the sketch to screen.The points that define the start and the end of eachline are stored and become the “control points” ofthat sketch object.

The following classifiers make use of the length ofa sketch, its number of direction changes, anglesbetween straight lines and the size and shape ofits oriented bounding box, which is the smallestpossible rectangle that can be placed around it.The oriented bounding box for a sketch a iscalculated immediately after it has been sketched,and its height and width are denoted by aoobH andaoobW , respectively.

5.2 Sketch Classification

A variety of classifiers have been implementedbased on the above and some additional properties.

A scribble is characterised by a dense packingwithin its bounding box and a high number ofdirection changes or a circular repeating motion.Experiments were conducted to derive the con-stants in the following equations capturing theseproperties:

length(a)2 ∗ (aoobH + aoobW )

> 1.9

#DirectionChanges

#ControlPoints− 2> 0.18

5.5 ∗ length(a)aoobH + aoobW

+#DirectionChanges

#ControlPoints− 2> 2.4

The cross and pause sign are characterised by twoapproximately straight lines a and b (width >>height for the oriented bounding box) with similarlength. For the cross the bounding box centres aresimilar and the bounding boxes roughly orthogonalto each other, whereas for the pause sign they areroughly parallel.

Arrows and double arrows are recognised by arelatively long straight or curved sketch whose endpoint has an arrow head, i.e., either it meets twoshort sketches which have a roughly 45 degreeangle with its tangent at the end point, or itsendpoint is close to the sharp angle of a “<” shapedsketch.

Squares, triangles and rectangles are assumed to besketched in a single stroke. To initially filter outshapes that aren’t enclosures, a condition is placedon the separation of the start and end points of asketch a:

|astart − aend| < 0.125 ∗ (aoobH + aoobW ) (1)

In order to differentiate between triangles, squaresand rectangles the distance of each point ai to thesketch’s centre of mass (COM) is considered:

COM =1

numPixels∗

numPixels−1∑i=0

ai (2)

This method is scaling and rotation invariant andresults in a more flexible and simpler implementa-tion.

Figure 3: Plot of the distances of a sketch’s pixels to

its centre of mass for a square, triangle and rectangle

symbol.

A plot of the distance of each pixel to the centre ofmass is shown for a triangle, square and rectanglein figure 3. It can be seen that the representationprovides a unique signature of each type of shapeallowing them to be easily differentiated. A peakdetector is used to determine the number andlocation of each local maxima and minima in thedistance profile. The peak detector uses a non zero-derivative method to avoid false detections whichcan occur due to accidental zero crossings [20].

The number of maxima and minima are thenused to classify the shape. Squares have typicallyfour maxima and minima, triangles three andrectangles either two or four depending on therectangles width. This potential difference occursfor extremely narrow rectangles because the smalltroughs in the peaks relating to the short sides ofthe rectangle are ignored due to the peak detection

threshold. When the shape has four maximaand minima the height and width are used todifferentiate between rectangles and squares. Forthe shape to be considered a square the width andheight of a bounding box must differ by at most25%. The maxima identified should relate to thecorners of the triangle, rectangle or square. Falsedetections can occur when shapes other than thisare drawn. To allow for this the smoothed pixeldata is segmented at the maximum points andlines are fitted between them using a modified leastsquares technique. More details are found in [21].

6 Robot Control

Once sketches have been successfully classified therobot control subsystem handles the higher levelinterpretation of each recognised sketch. It alsohandles all communication with the robot, readingfrom its sensors and issuing commands for therobot to perform.

6.1 Task Queue and Control Loop

The robot control system allows multiple tasks tobe queued using a two-sided queue, providing formore complex behaviour. When a new sketch isclassified it is converted to a new task and placedat the end of the queue. Certain sketches, e.g., asquare representing a “stop” sign, are placed onthe front of the task list, to interrupt the system.Other sketches can act as interrupts depending onthe current state of the system. For instance thefirst pause sketch drawn will interrupt the systembut subsequent pauses will be added to the backof the list. When a sketch is deleted the robotcontrol code is notified and then searches throughthe task queue deleting any tasks associated withthat sketch.

A simple control loop is used to command therobot. During each iteration three main functionsare performed: updating the robot’s position uti-lizing the calibrated overhead camera; checkingfor any interrupts, such as if any of the robot’sbumpers have been pressed; and if there are nointerrupts it processes the task at the front of thequeue. The tasks fall into five classes: roboticarm manipulation, robot motion, forbidden regiondefinition, image capture, and system control. Thetwo system control tasks are stop and pause. Whenprocessing the stop task, it simply clears the sketchcontext and task queue. A pause task specifies nobehaviour so the system is held in a busy wait modeuntil the pause sketch is erased.

6.2 Robotic Arm Manipulation

When the system is initialised the robot controlsystem places the arm in a known initial position,to ensure the robot arm position initially matchesthat of the 3D model. When the user rotatesthe arm in the 3D model a change in angle fromthis initial position is generated for each joint andcommunicated to the control system. This mustthen be mapped to the possible positions that eachjoint can move to. This mapping is achieved bylinearly scaling the rotation to match the rotationon the robotic arm. To account for any differencebetween the initial position of the arm and the3D model the rotations are adjusted by a jointdependent constant value

6.3 Mapping Sketch Input into the 3DWorld

In order for 2D sketches to specify 3D motions theymust be mapped into the 3D environment of therobot. Using a calibrated camera this is achievedby casting a ray from the view point through apoint of the sketch (parallel to the view plane).Next the intersection with the ground plane, situ-ated at the origin, is found. This intersection pointprovides the point’s 2D coordinates in the robot’splane of motion.

6.4 Robot Motion

The path to a target point, indicated by a cross, isa straight line. Sketched “no go” areas are avoidedby circumventing them until a line to the targetpoint can be traced which does not intersect thatobstacle. The robot itself has laser sensors andavoids obstacles in a similar way. While morecomplicated path planning algorithms could beutilised, this algorithm was sufficient for testingthe given set-up.

Arrows can encode both a path for the robotto follow or a rotation of the robot if drawn onit. To determine if the arrow is drawn on therobot, the robot is modelled as a circle (radius200mm) for ease of calculation. Since the robotis roughly circular anyway, this does not result ina substantial error. The start and end points ofthe arrow are projected into the robot plane andtheir distance to the robot’s centre calculated. Ifthis falls inside the circle then the robot is simplyturned to the heading formed by the last twocontrol points of the arrow. If the arrow sketch fallsoutside of the circle, path following is intended.The waypoints for arrow sketches are formed fromtheir control points. Waypoints too close togetherare removed.

In order to patrol a region of interest the regionis covered with parallel lines aligned with the

width of its oriented bounding box. We performthis “scanline” computation in pixel space sincethe current set-up does not have any significantperspective distortion. For large scale applicationsrequiring high precision the scan lines should becomputed in world-space.

The above algorithms compute a series of way-points, which the robot processes sequentially. Tomanoeuvre itself to a waypoint the control algo-rithm maps the point into the robot plane and thencommands the robot to turn at a rate proportionalto the difference between its current heading anda straight line to the waypoint. Once the robot isat the correct heading it begins moving forwardat a speed dependent on the distance to thewaypoint. If the task does not involve patrolling,once the robot reaches the final waypoint the robotis stopped and the task removed from the FIFO. Ifthe robot is in a patrolling mode then the controlsystem simply controls the robot to each waypointin reverse.

7 Implementation

All sketch recognition and robot control softwarewas implemented in C++ to provide good perfor-mance. ARToolKitPlus was used to provide identi-fication of the fiducial marker in the camera image.ARDev interprets the camera calibration data andhandles projections between the image plane androbot plane as well as controlling the rendering ofall objects in OpenGL. ImageMagick++ was usedfor conversion between image file formats. Player,a robot control interface, is used to communicatewith the robot and control its sensors and systems.

8 User Evaluation and Results

To adequately assess whether the implementeddesign satisfies the specified requirements, eightsubjects were asked to perform a series of tasksrepresentative of real world applications. Fiveof the subjects were male, three female, all wereaged between 21 and 22. Half had a robotics orprogramming background. The subjects were onlyprovided with a list of shapes and their associatedbehaviour, no details on how to draw the shapeswere provided.

The tasks included guiding the robot around itsenvironment, including through an imaginary ob-stacle, selecting an area to patrol, taking a picture,stopping and pausing the robot during a task andmanipulating the robotic arm to pick up a foamsquare. Examples are shown in figure 4. Thesubjects were then asked to assess the system usinga 5-level Likert scale as well as to provide general

Figure 4: Examples of sketch-based robot program-

ming: (a)-(c) patrolling an area, (d)-(f) path following,

(g) camera shot, (h) indicating obstacles/danger areas,

(i) target point, (j)-(m) picking up an object.

feedback. The results of the quantitative questionsasked are presented in figure 5.

Figure 5: Results of our usability study. Answers

are on a 5-level Likert scale ranging from 1 “strongly

disagree” to 5 (“strongly agree”).

The results of the user study indicate that mostfunctionalities were intuitive and easy to achieve(4.375 out of 5). The functionalities patrolling andstop, pause and resume also achieved very highscores of 4.75 and 4.4, respectively. The mainproblem were the tasks related to manipulatingthe robot arm - especially “picking up an object”which received a score of only 2.431 out of 5. Therobotic arm was incorporated into the design asit is advantageous in industrial and educationalapplications. A common user complaint aboutthe robotic arm was that it was difficult to gainfeedback on where the arm was relative to theobject. This occurred because the movement hadto be visually monitored and sketch behaviourdid not match expectations due to the trackball

movement not feeling natural. These problemscould be overcome in future by providing real timecamera feedback of the area in front of the robot. Aprincipal problem with the robot arm is that it hasmany axes resulting in multiple possible motionsfor an action. Furthermore its manipulation isin 3D rather than on a 2D plane, i.e., the armhas 6 degrees of freedom (3 for position and 3 fororientation, of the end of the hand). This couldbe resolved with a more abstract design whereusers sketch the “semantic” of an action, ratherthe motion of its components. An example wouldbe to circle the object to be picked up and to sketcha line from the gripper to the object. An inbuiltalgorithm would then compute the optimal armmotion.

Users also commented that the sketch recognitionwas not as robust as they would have hoped. Forinstance several subjects drew squares or triangleswith multiple strokes, or a scribble as a loosecrossed motion. The sketch recognition shouldbe extended to support recognition of shapes withmultiple strokes. Furthermore, it would be usefulto perform a large scale survey investigating howusers draw our proposed sketched robot controlsymbols and use this to improve our sketch recog-niser.

9 Conclusion and Future Work

Sketching is a promising approach to allow in-experienced people to design simple robot pro-grams. We identified five typical applications andderived three common tasks: moving a robot,patrolling an area and picking/dropping objects.A range of sketch symbols were presented andimplemented for performing these tasks. Userevaluation concluded that this type of system wasintuitive and easy to use, thus this type of interfacecould provide a solution to the complexity seenin current human robot interfaces. The roboticarm component presented some challenges to users,highlighting the need for further development inthis area. In future work we want to definemore high-level behaviour, which requires moreintelligent robot control software and sensors, andwe want to explore sketch-based interfaces withoutceiling mounted camera, e.g., by using SLAMalgorithms.

References

[1] M. Urban and P. Bajcsy, “Fusion of voice, ges-ture, and human-computer interface controlsfor remotely operated robot,” in Proceedingsof International Conference on InformationFusion, Jul. 2005, p. 8.

[2] M. D. Gross and E. Y.-L. Do, “Ambiguousintentions: a paper-like interface for creativedesign,” in UIST ’96: Proceedings of the 9thannual ACM symposium on User interfacesoftware and technology. ACM, 1996, pp.183–192.

[3] Y. Y. Wong, “Rough and ready prototypes:lessons from graphic design,” in CHI ’92:Posters and short talks of the 1992 SIGCHIconference on Human factors in computingsystems. ACM, 1992, pp. 83–84.

[4] M. Shiomi, D. Sakamoto, T. Kanda, C. T.Ishi, H. Ishiguro, and N. Hagita, “A semi-autonomous communication robot: a fieldtrial at a train station,” in Proceedings of the3rd ACM/IEEE international conference onHuman robot interaction (HRI ’08). NewYork, NY, USA: ACM, 2008, pp. 303–310.

[5] T. Takahashi, S. Nakanishi, Y. Kuno, andY. Shirai, “Human-robot interface by verbaland nonverbal behaviors,” in Proceedings ofIntelligent Robots and Systems, Oct. 1998,pp. 924–929.

[6] T. H. Song, J. H. Park, S. M. Jung, and J. W.Jeon, “The development of interface devicefor human robot interaction,” in Proceedingsof the International Conference on Control,Automation and Systems (ICCAS ’07), Oct.2007, pp. 640–643.

[7] S. G. Hill and B. Bodt, “A field experimentof autonomous mobility: operator workloadfor one and two robots,” in Proceedings ofthe ACM/IEEE international conference onHuman-robot interaction (HRI ’07). NewYork, NY, USA: ACM, 2007, pp. 169–176.

[8] M. W. Kadous, R. K.-M. Sheh, and C. Sam-mut, “Effective user interface design forrescue robotics,” in Proceedings of the1st ACM SIGCHI/SIGART conference onHuman-robot interaction (HRI ’06). NewYork, NY, USA: ACM, 2006, pp. 250–257.

[9] D. F. Glas, T. Kanda, H. Ishiguro, andN. Hagita, “Simultaneous teleoperation ofmultiple social robots,” in Proceedings of the3rd ACM/IEEE international conference onHuman robot interaction (HRI ’08). NewYork, NY, USA: ACM, 2008, pp. 311–318.

[10] E. Steger, “Sketch-based animationlanguage,” http://www.cs.toronto.edu/∼esteger/sketchlang/index.html, 2004.

[11] M. Thorne, D. Burke, and M. van de Panne,“Motion doodles: an interface for sketch-ing character motion,” ACM Transactions on

Graphics (TOG), vol. 23, no. 3, pp. 424–431,2004.

[12] Q. L. Li, W. D. Geng, T. Yu, X. J. Shen,N. Lau, and G. Yu, “Motionmaster: author-ing and choreographing kung-fu motions bysketch drawings,” in Proceedings of the 2006ACM SIGGRAPH/Eurographics symposiumon Computer animation (SCA ’06). Aire-la-Ville, Switzerland, Switzerland: EurographicsAssociation, 2006, pp. 233–241.

[13] S. Takahashi, Y. Kato, and E. Shibayama,“A new static depiction and input techniquefor 2d animation,” in 2005 IEEE Symposiumon Visual Languages and Human-CentricComputing, 2005, pp. 296–298.

[14] J. Davis, M. Agrawala, E. Chuang,Z. Popovic, and D. Salesin, “A sketchinginterface for articulated figure animation,”in Proceedings of the 2003 ACMSIGGRAPH/Eurographics symposiumon Computer animation (SCA ’03), 2003, pp.320–328.

[15] M. Skubic, C. Bailey, and G. Chronis,“A sketch interface for mobile robots,” inProceedings of the International Conferenceon Systems, Man and Cybernetics, Oct. 2003,pp. 919–924.

[16] M. Skubic, D. Anderson, S. Blisard,D. Perzanowski, W. Adams, J. G. Trafton,and A. C. Schultz, “Using a sketch padinterface for interacting with a robot team,”in AAAI’05: Proceedings of the 20th nationalconference on Artificial intelligence. AAAIPress, 2005, pp. 1739–1740.

[17] D. Sakamoto, K. Honda, M. Inami, andT. Igarashi, “Sketch and run: a stroke-based interface for home robots,” in CHI’09: Proceedings of the 27th internationalconference on Human factors in computingsystems. ACM, 2009, pp. 197–200.

[18] T. Collett, “ARDev homepage,” 2010, http://sourceforge.net/projects/ardev/.

[19] J. Hultquist, “A virtual trackball,” inGraphics gems. San Diego, CA, USA: Aca-demic Press, 1990, pp. 462–463.

[20] E. Billauer, “peakdet: Peak detection usingMATLAB,” 2008, http://www.billauer.co.il/peakdet.html.

[21] R. Shucksmith, “Sketch-based robot controland training,” Department of Electrical &Computer Engineering, University of Auck-land, New Zealand, Part IV Research ProjectReport, Sep. 2010.