PhonePaint: Using Smartphones asDynamic Brushes with InteractiveDisplays

Jian ZhaoDepartment of ComputerScienceUniversity of Torontojianzhao@dgp.toronto.edu

Fanny ChevalierDepartment of ComputerScienceUniversity of Torontofanny@dgp.toronto.edu

Karan SinghDepartment of ComputerScienceUniversity of Torontokaran@dgp.toronto.edu

Copyright is held by the author/owner(s).

AbstractWe present PhonePaint, a novel technique that allowsusers to sketch on interactive displays using asmartphone as a configurable paintbrush. PhonePaintprovides a casual and ubiquitous painting experiencethat is suitable for a broad audience, and offers a richsketching interaction where the stroke parameters aredynamically controlled by the built-in sensors of thephone. We describe a proof-of-concept implementationof PhonePaint with minimal hardware added to aregular smartphone. We conducted a light-weightqualitative evaluation of the prototype. The resultsshow that participants enjoyed the drawing processwith PhonePaint and were able to create interestingvisuals with minimal training.

Author KeywordsSketch Interface, Painting, Touch DevicesACM Classification KeywordsH.5.2 [Information Interfaces and Presentation]: UserInterfaces

IntroductionPainting, drawing and sketching is a ubiquitous form ofcreative expression, be it to communicate ideasthrough visuals, share artistic illustrations, or simplydoodle thoughts. The considerable differences

between traditional painting with brushes and digitalpainting with mice have long been a barrier to creativeand freeform sketching on computers. However, thepractice of digital painting has drastically increased withthe release of interactive displays and styluses, whichoffer a more engaging and direct user experience.

With these digital supports, one category of drawingapplications is typically designed for simple, portableand quick sketching needs. As such, they are oftenlimited to the most primitive drawing tools, confiningusers to a rough and rudimentary style. Conversely,another category of sketching interfaces aims to offer aricher experience. These, which are mostly attached tostationary devices, are typically employed byprofessional software, where brush types and many oftheir parameters can be fine tuned through a complexgraphical user interface. Such systems, althoughoffering the best flexibility, can be difficult andintimidating for new users to learn, and thusdiscouraging for novice or casual users.

In this paper, we are interested in exploring if and howa light-weight and everyday device such as asmartphone can provide an easy and portable as wellas customizable and rich enough drawing experiencewith interactive displays. We propose PhonePaint, anovel technique designed to support an engagingtangible-input painting process through the flexiblecontrol of brush parameters with the interactiveaffordances of the device, namely its accelerometer,gyroscope and touch display. PhonePaint allowssimultaneous drawing at two levels: with the device asa brush on an interactive display, and directlymanipulating the phone to dynamically control brushparameters and appearance while drawing strokes.

Related WorkThere exists a large body of research on digitalpainting systems that aim to reproduce and extend thetraditional drawing experience with real brushes. Oneapproach is to algorithmically simulate the ordinarypaintbrushes with software, such as Adobe Photoshopand Illustrator. The software simulations can be usedwith input devices other than a mouse, e.g., a stylus, tofurther enhance the user experience of painting viatangible inputs. For example, FluidPaint [7] allowsartists to paint with real wet brushes in which thefootprints of the brush are captured by a transparentsurface illuminated by infrared LEDs.

Another approach is to augment the tangible brushinput device with extra and special hardware.IntuPaint [8] uses a self-illuminated LED brush whosebristles are made of semi-transparent nylon fibers.Conte [9], also equipped with a LED, enables easymode switching (e.g., color selection) by using itsdifferent corners and faces in contact with a tabletopsurface. I/O brush [4] uses a tiny camera that allows topick colors from the real-world.

Researchers have explored the use of portable devicesas styluses on interactive displays. Hardy et al. [1]proposed a technique allowing users to pair a mobiledevice with a display and perform selections through aNFC/RFID tag grid. PhoneTouch [5] further refines theselection by correlating the phone accelerometer andtabletop input events.

Beside selection tasks, there exist studies on using asmartphone with the surface in a stylus-like fashion.Jang et al. [2] extended smartphone with an opticalsensor allowing users to perform flick-based browsing,selections and text input. Schmidt et al. [6] proposed

an interaction style for mobiles and surfaces based ondirect manipulations with the phone as a tangiblestylus, facilitating seamless interaction across deviceboundaries. McAdam et al. [3] compared a dialmanipulation task with 3 interaction types and foundthat people thought the tangible use of the surface waseasier than direct touch.

In summary, there has been a consequent effort onaugmenting tangible input devices to improve the digitalpainting experience on the one hand. On the otherhand, smart phones has proven to be good candidatesto serve as a stlyus on interactive display. However, asfar as we know, no work to date has explored bringingthose two pieces of the puzzle together. WithPhonePaint, we propose to use of the smart phonedevice as a tangible paintbrush, leveraging the sensordata to dynamically control stroke parameters.

The PhonePaint Design

TOUCH SCREEN ACCELEROMETERa b

Figure 1: Example of dynamicbrush manipulations using thephone built-in sensors: (a) fingertouch location can be used tocontrol the stroke width(up-down) and opacity(left-right), and (b) tilt angles canbe used to change the strokeperpendicular opacity gradient.

The design of PhonePaint is motivated by the manyfeatures of a smartphone that we believe are essentialfor an effective tangible device to serve as a brush fordigital painting on interactive displays.

Prevalence and Portability. The ubiquitous usage ofsmartphones nowadays makes the potential brushesprevalent [3]. Smartphones are portable and easy tobe associated with other digital devices, thereforepeople can perform painting on interactive displays atdifferent locations, including private and public places.

Personalization and Collaboration. The phone touchdisplay can be used as a dedicated interface forparameter configurations (e.g., a settings panel madeof sliders and buttons), allowing personalized sketchingstyles adjusted on the brush device itself [6]. Unlike

traditional styluses, this affordance naturally supportedby smartphones enables a potential convenientmulti-user collaborative painting experience.

Dynamic Brush Manipulation. Figure 1 illustrates someexamples of leveraging the built-in phone sensors todynamically manipulate brush parameters whilepainting strokes. First, finger manipulation on thephone touch display offers two extra degrees offreedom (x and y coordinates) for directly varying brushproperties, e.g., stroke width, color hue ortransparency. Second, additional built-in sensors of thephone can be utilized for dynamic brush interactions.For example, the device orientation interpreted can beused for intuitive mode switching between drawing anderasing. Mapping tilt angles to transparency gradientsof stroke colors provides a more realistic paintingexperience (mimicking that only part of the brushesbristles is in contact with the canvas).

Cross-Device Data Transfer. The brush can benefitfrom additional features usually supported bysmartphones, including the inter-device communicationand camera capabilities [6]. For instance, brushsettings and sketches can be fluidly transferredbetween different phones or between the phone andthe display, to support easy sharing, recording andcollaboration. The digital camera can collect additionalinput from the surrounding environment, such as usingreal-world imagery as a color or texture palette [4].

Proof-of-Concept ImplementationWe created a research prototype using a HTC TouchPro 2 as the PhonePaint brush and a Microsoft Surfaceas the painting display. At this stage, we chose toimplement features that are critical for the painting

process. Features such as data transfer and camerausage, secondary to our goal, are left for future work.

The Phone Brush

(a)

(b)

Figure 2: Proof-of-conceptimplementation of PhonePaint:(a) adding minimal hardware tothe phone, i.e., two LEDs and abattery, and (b) the brushparameters configuration panel.

Ideally we aim not to add any extra hardware with thephone to maintain its ubiquitousness. However, limitedby current settings of smartphones, we attachedminimal hardware—two infrared LEDs (5mm diameter,1.2V)—in order to make the footprints of the phonedetectable by the surface (Figure 2-a). The LED on thetop of the phone is used to track the device in thedrawing mode. When the phone is flipped, the one onthe bottom indicates the location of the eraser. An AAbattery was used to supply power for the two LEDs.The physical dimensions of the phone brush are116×59.2×17.3mm and its total weight (including theextra hardware) was 195g

We developed a customized system for configuring thebrush parameters and sending data to the surface viathe Bluetooth network. Figure 2-b shows theconfiguration user interface of PhonePaint, in which theuser can manipulate: stroke colors and color gradients,brush initial width, and mapping mechanisms betweenfinger touch locations on the phone display and thechosen brush parameters. The phone accelerometerand gyroscope data was used to: 1) compute thephone tilting angles for dynamically changing theopacity gradient perpendicular to the stroke drawingdirection, and 2) distinguish whether the brush is in thedrawing mode or the erasing mode as described in theprevious section. While our prototype only implementsassociations of sensor data with a few drawingparameters, other parameter mappings can be easilyadded in a similar fashion.

The Painting SurfaceThe Microsoft Surface (30 inch diagonal, 1024×768pixels) was raised to a height of 30cm to allow users tocomfortably reach the full surface of the display, whenstanding beside it. Since an important affordance ofPhonePaint is to incorporate the brush settings directlyon the brush device, the interactive display itself waslimited to a simple drawing canvas. The surface, thus,only received data sent by the phone, and interpretedthe brush parameters to render strokes, using thetracking information of the phone footprint coordinates.To facilitate the evaluation of PhonePaint, wedeveloped a simple graphic user interface as shown inFigure 3-a, which we discuss in detail below.

User StudyWe conducted a light-weight evaluation with thepurpose of gathering initial user feedback and studyingthe usability of PhonePaint. Six volunteers (3 females),aged 22-31, participated in the study. Participants weremembers of the university community (i.e., graduatestudents and researchers). All are right-handed, andnone of them had prior professional artistic training.

First, participants performed 24 trials of structureddrawing tasks for reproducing example strokes on ablank canvas, as shown on the top-left corner of thesurface (Figure 3-a). Textual instructions were alsoprovided for describing the conditions required forcompleting the trial. Creative manipulations of variousbrush properties (e.g., stroke colors, gradients orwidths) was encouraged as long as the requiredinstructions were obeyed. Subjects could restart a trialas many times as they desired until satisfaction (notime limit was imposed).

For the example strokes, we controlled 4 independentvariables: the stroke type (straight or curvy line), thecolor mode (unicolor or three-point gradient), opacitygradient mode (with or without phone tilt angles tochange stroke opacity gradient) and touch displaycontrol mode (not using the touch display, using touchpositions to change the brush transparency, or changethe brush width), for a total of 24 conditions.

(a)

(b)

Figure 3: Participants usingPhonePaint (a) to reproduce agiven stroke shown on thetop-left corner and, (b) forfreestyle drawing.

After completing all the trials, participants were given10min for free-form painting (Figure 3-b), after whichthey filled out a post-study questionnaire. We alsoconducted a short informal interview to collect theirgeneral feedback. The whole study took about 30min.

Results and DiscussionsQualitative AssessmentFigure 4 shows the results of the post-questionnaire.The overall user reactions were very encouraging.Participants generally thought that PhonePaint wasflexible, novel and compelling to control multiple strokeparameters in real-time with phone sensors (Q1:µ = 3.8). They were eager to have such application ontheir personal phones (Q5: µ = 4.4). But we observedlarger variances on Q4 (σ = 1.0) and Q6 (σ = 1.1)related to aspects of collaborative and public drawing,which may due to participants’ various personalpreferences about privacy issues.

Task Observations and Initial User FeedbackDuring structured drawing time, we observed thatparticipants were able to familiarize themselves withthe PhonePaint brush after a very short time practice.All participants completed the task fluidly asking foronly little assistance on setting the brush parameterswith the configuration panel (Figure 2-b). All the trials

were achieved fairly quickly by participants (averagecompletion time 18s including the configurations ofbrush parameters), except for the ones trials requiringthe use of the phone touch display and the phone tiltangle that were more time consuming (30s onaverage). Overall, participants were able to easily andaccurately reproduce the given sample strokes,indicating that PhonePaint is a suitable candidate as aconfigurable brush for digital painting.

1 2 3 4 5

Q1

Q2

Q3

Q4

Q5

Q6

ScoreQ1. I find that the control of the stroke colors, width and opacity is flexibleQ2. I enjoy the process of painting with the phoneQ3. I feel creative when I use this systemQ4. It would be more fun if I could draw with others with this systemQ5. I would like to have an application like this one on my phoneQ6. I would like to have access to sketching displays everywhere

Figure 4: Average usability assessment measured on a5-point Likert scale (1: strongly disagree, 5: strongly agree)

In free-form drawing part, participants showedenthusiasm and reported that the whole drawingprocess to be enjoyable, interesting and engaging.Some more “talented artists” were able to produce verycompelling and interesting sketches like those ofFigure 5. Three participants were very excited andeven requested more time to finish their artwork,looking at the sketches and saying: “I should show it tomy friends”. All the participants particularly liked usingthe tilt angle to modify dynamic brush parameters: “It’slovely. I can draw fancy strokes by just rotating mywrist”. Two participants tried almost all the features ofPhonePaint by themselves, including different ways ofcontrolling the dynamic brush parameters. In contrast,some participants sticked to a set of few features, such

as changing the stroke opacity gradient with tilt angles.For these participants, the experimenter sporadicallysuggested exploration by giving some hints. This maybe because of participants’ different personalpreferences and prior experiences.

Figure 5: Examples of free-formsketches performed byparticipants

A notable issue was that several participants found itdifficult to use the touch display for dynamic brushparameter control and it became harder when using incombination of tilting angles. However, someparticipants used the second hand on the touch screenfor more accuracy, which could overcome this issue tosome extent, similar to the interaction in [6].Participants also reported that PhonePaint can becomecumbersome and tiring for fine-stroke sketching.Similar issues have been pointed out in [2, 3].However, such fine-drawing tasks tend to be rare innon-professional and casual sketches which aretypically used for quick illustrations. PhonePaint isdesigned to provide a portable and easy sketchingexperience with a rich dynamic manipulation of strokeparameters. In this light, we believe the prevalence andubiquitousness of the approach is of a higherimportance. During our study, participants felt that thephone and the tabletop surface worked well as an unit,and the design of PhonePaint provided them with aninterface easy to grasp and use.

Conclusion and Future WorkWe have introduced PhonePaint, a technique designedfor informal ubiquitous sketching, where a smartphoneis used as a paintbrush in a novel way to draw on aninteractive displays. PhonePaint aims at improving theusers’ experience of digital painting through directmanipulations on the brush itself to control brushparameters, by leveraging the phone built-in sensors.

Our initial study suggests that PhonePaint is aneffective and convenient tool for perform basiceveryday drawing tasks.

In the future, we wish to explore other interactiontechniques to improve the overall user experience ofdigital painting using a similar system. One possibilitycould be enabling haptic feedback to convey additionalinformation of stroke parameter settings. We also planto extend PhonePaint to support collaborative drawing.Finally, we want to conduct further evaluations on bothmulti-user and individual painting scenarios to betterunderstand the capabilities of this idea.

References[1] R. Hardy and E. Rukzio. Touch & interact: touch-based

interaction of mobile phones with displays. In Proc. ofMobileHCI, pp. 245-254, 2008.

[2] B. Jang, M. Lee, and G. J. Kim. CornerPen: smart phone is thepen. In Proc. of int. conf. on Human-computer interaction. Part3:76-83, 2011.

[3] C. McAdam and S. Brewster. Using mobile phones to interactwith tabletop computers. In Proc. of ITS, pp. 232-241, 2011.

[4] K. Ryokai, S. Marti, and H. Ishii. I/O brush: drawing witheveryday objects as ink. In Proc. of CHI, pp. 303-310, 2004.

[5] D. Schmidt, F. Chehimi, E. Rukzio, and H. Gellersen.PhoneTouch: a technique for direct phone interaction onsurfaces. In Proc. of ACM UIST, pp. 13-16, 2010.

[6] D. Schmidt, J. Seifert, E Rukzio, and H. Gellersen. Across-device interaction style for mobiles and surfaces. In Proc.of DIS, pp. 318-327, 2012.

[7] P. Vandoren, L. Claesen, T. Van Laerhoven, J. Taelman, C.Raymaekers, E. Flerackers, and F. Van Reeth. FluidPaint: aninteractive digital painting system using real wet brushes. InProc. of ITS, pp. 53-56, 2009.

[8] P. Vandoren, T. Van Laerhoven, l. Claesen, J. Taelman, C.Raymaekers, and F. Van Reeth. IntuPaint: Bridging the gapbetween physical and digital painting. In IEEE workshop ontabletops and interactive surfaces, pp. 65-72, 2008.

[9] D. Vogel and G. Casiez. Conte: multimodal input inspired by anartist’s crayon. In Proc. of UIST, pp. 357-366, 2011.