1

CrowdVision: A Computing Platform for VideoCrowdprocessing Using Deep Learning

Zongqing Lu Member, IEEE , Kevin Chan Member, IEEE , and Thomas La Porta Fellow, IEEE

Abstract—Mobile devices such as smartphones are enabling users to generate and share videos with increasing rates. In somecases, these videos may contain valuable information, which can be exploited for a variety of purposes. However, instead of centrallycollecting and processing videos for information retrieval, we consider crowdprocessing videos, where each mobile device locallyprocesses stored videos. While the computational capability of mobile devices continues to improve, processing videos using deeplearning, i.e., convolutional neural networks, is still a demanding task for mobile devices. To this end, we design and build CrowdVision,a computing platform that enables mobile devices to crowdprocess videos using deep learning in a distributed and energy-efficientmanner leveraging cloud offload. CrowdVision can quickly and efficiently process videos with offload under various settings anddifferent network connections and greatly outperform the existing computation offload framework (e.g., with a 2× speed-up). In doingso CrowdVision tackles several challenges: (i) how to exploit the characteristics of the computing of deep learning for video processing;(ii) how to parallelize processing and offloading for acceleration; and (iii) how to optimize both time and energy at runtime by justdetermining the right moments to offload.

Index Terms—Crowdprocessing, video, offload, convolutional neural networks.

F

1 INTRODUCTION

MOBILE devices such as smartphones are enabling usersto generate and share videos with increasing rates. In

some cases, these videos may contain valuable information,which can be exploited for a variety of purposes.

In this paper, we consider a video classification problemwhere a task issuer asks the crowd to identify relevantvideos about a specific object or target. This requires objectdetection to be performed on videos to detect these objectsof interest within the frames of the videos. The limitationsof mobile devices for these types of applications are wellknown. The most obvious challenge is the computationalrequirement. Although it continues to improve, video pro-cessing using deep learning, i.e., Convolutional Neural Net-works (CNNs), is still a demanding task for mobile devices[1].

We anticipate that video processing can be performedwithin a network of mobile devices and a powerful cloudenvironment. Individuals have the option to process thevideos locally or offload them to a highly capable computingunit in the cloud. Coupled with the computation limitationis the cost to transmit, whether it be via WiFi or cellular (4GLTE). Transmitting videos to the cloud over wireless linksconsumes considerable energy. Additionally, when usingcellular networks, there may be data budgets that limit datatransfer without incurring extra expenses. As a result, weconsider the coupled problem of constrained resources ofcomputing, data transmission, and battery.

Due to these limitations, users may be reluctant to par-ticipate in such requested video processing tasks. However,

• Z. Lu is with the Department of Computer Science, Peking Univer-sity. E-mail: zongqing.lu@pku.edu.cn. K. Chan is with Army ResearchLaboratory. E-mail: kevin.s.chan.civ@mail.mil. T. La Porta is with theDepartment of Computer Science and Engineering, Pennsylvania StateUniversity. E-mail: tlp@cse.psu.edu.

users could be motivated by various incentive mechanisms,such as [2], which are not the focus of this paper. Forthis paper, we consider a variation on a crowdsensing-type scenario that we call crowdprocessing. Instead of userscollecting and sharing data to solve a larger problem, weexplore the computing capability of mobile devices andallow individuals to process some (or all) of the data ratherthan having some centralized entity process everything.

We design and build CrowdVision, a computing plat-form that enables mobile devices to crowdprocess videosusing deep learning in a distributed and energy-efficientmanner leveraging cloud offload. In doing so we tackleseveral challenges. First, to design a computing platformspecifically for video processing using deep learning, weshould take into account the characteristics of the com-puting of deep learning. By deploying and measuring thecomputing of CNNs on mobile devices, we identify batchprocessing, which makes deep learning different from com-mon computational tasks and can be exploited for comput-ing acceleration. Second, as frames extracted from a videoarrive at a certain rate, to take advantage of the batchprocessing towards optimizing the processing time, we needto determine when to perform each batch with how manyframes. However, the problem turns out to be a NP-hardproblem. By carefully balancing the waiting time for framesto be available and the processing time incurred by eachadditional processing batch, we are able to determine thebatch processing to optimize the processing parallelized byoffloading, even with an energy constraint that is imposedby the user. Third, When the data rate for offloading varieswidely, it is hard for mobile devices to acknowledge whenoffload benefits. By correlating signal strength, data rate,and power based on offloading attempts, we are able tosense and seize the right moments when offloading benefitsboth processing time and energy at runtime.

2

We envision CrowdVision to be useful for a variety ofapplications that require content information of videos fromthe crowd. One common use case is emergency responsesituations, where municipal agencies ask the public to assistin identifying terrorists or criminals through scanning oftheir videos, as the FBI did after the Boston Marathonbombing. CrowdVision could handle this request in a smartand automatic way; i.e., it first filters videos based on lo-cation and timestamp, and then collectively performs deeplearning to find the object of interest.

Contributions: (i) We measure and characterize the pro-cessing time and resource usage for each component ofvideo processing using deep learning (§3). (ii) We design asplit-shift algorithm that parallelizes frame offload and localdetection to optimize the processing time by taking advan-tage of batch processing and two algorithms that optimizethe processing time with an energy constraint (§4). (iii) Wedesign an adaptive algorithm featured with a backoff mech-anism, which determines the offloading at runtime towardsoptimizing both completion time and energy (§5). (iv) Weimplement CrowdVision on Android with a GPU-enabledserver for offload (§6). (v) We show experimentally thatCrowdVision greatly outperforms the existing computationoffload framework (e.g., with a 2× speed-up) and improvesspeed and energy usage of video crowdprocessing, and thesplit-shift algorithm closely approximates the optimum (§7).

2 RELATED WORK

There are several examples of crowdsensing frameworkssimilar to CrowdVision. Medusa [3] is a platform that per-forms crowdsensing tasks through the transfer and process-ing of media (text, images, videos). Pickle [4] is a crowd-sensing application to conduct collaborative learning. Gi-gaSight [5] is a cloud architecture for continuous collectionof crowd-sourced video from mobile devices. In contrast tothese frameworks, CrowdVision is a computing platform forvideo crowdprocessing using deep learning rather than thesystem or platform that motivates users and collects senseddata. For the crowdprocessing task, CrowdVision attemptsto optimize the performance of the task execution whileconsidering the energy usage and data usage of the par-ticipating mobile devices through computation offload.

There is a large body of work that studies computationoffload for mobile devices [6]. These can be summarilyclassified into two categories: building general models forcomputation offload such as [7], [8], [9], [10] and computa-tion offload based applications such as [11], [12], [13]. MAUI[7] investigates code offload to reserve energy for computa-tion. The feasibility of computation offload is studied in [8]in term communications networks. COSMOS is presentedin [9], which bridges the gap between quick response re-quests by mobile devices and the long setup time of cloudframeworks. Offload scheduling is addressed in [10] whichconsiders the tail energy of cellular network. Computationoffload has been considered and implemented in variousapplications, e.g., for interactive perception applications[12], social sensing [11] and wearables [13]. In additionto offloading, Tango [14] considers application replicationon mobile devices and servers, where the active instanceof the application switches between the two systems to

accelerate network-intensive applications. However, due tothe characteristics of the computing of deep learning, noneof these works can be directly and effectively applied to ourapplication and offloading problem.

Optimizing the computing of deep learning applicationson mobile devices has been recently investigated by com-pressing parameters of CNNs [15], by distributing com-putation to heterogeneous processors on-board [16], [17],by trading off accuracy and resource usage [18], and bymobile GPUs [19]. However, currently, these approachesrequire custom software or hardware. Nevertheless, thesepotential processing approaches can be easily integratedwith CrowdVision whenever they are available for off-the-shelf mobile devices.

3 OVERVIEW

CrowdVision is a distributed computing platform that en-ables mobile devices to participate in the crowdprocessingof videos. In this environment, a task issuer initiates aquery by which mobile devices are requested to identifysome object of interest within its locally stored videos. Weconsider a crowdprocessing approach to perform object de-tection/classification of videos. This task includes filteringof videos based on metadata, and then processing the videosto perform object detection. This takes several steps asdescribed below. Caffe [20], [21], a deep learning framework,is currently employed to perform object detection on framesextracted from videos. Although CrowdVision is currentlybuilt on top of Caffe, it is compatible with other deeplearning frameworks, e.g., Torch [22] and TensorFlow [23].

The details of the query can be of varying specificity(e.g., “find people wearing red hats walking a dog in thepark on Tuesday evening”). These details may allow forthe individuals to filter out irrelevant videos (based onlocation or timestamp). At the expense of some energyusage, the user can offload the videos to the cloud, wherehigh performance computing can quickly process the videoswithout energy usage concerns. Alternatively, the user canprocess some of the frames locally and offload others.

Once these videos are processed using deep learningeither locally on the mobile device or remotely on the cloud,the task issuer will receive information about the videosrelated to the query. For frames that are processed on themobile devices, the user will forward either the tags, theframes of interest, or the entire video to the cloud.

Two important aspects of this process are not in the scopeof this paper. First, incentive mechanisms for crowdprocess-ing are important but not the focus of this paper. Second,participation in the task may reveal personal informationand intrude on users’ privacy, but users are allowed to filterout their personal videos. More sophisticated and rigoroustreatment of security and privacy control is left as futurework.

The overview of CrowdVision is depicted in Fig. 1. Amobile device has several options to perform object detec-tion on each related video. Performing object detection ina video entails two main steps. First, video frames must beextracted from the video and turned into images. Second,object detection is performed on the images. These two

3

Fig. 1. Overview of CrowdVision.

functions may be performed locally on a mobile device, or inthe cloud, or distributed between the two systems.

As illustrated in Fig. 1, by considering several inputparameters and constraints, e.g., network connections, bat-tery life, and data budget, a mobile device can perform(i) video offload by sending the whole video to the cloud.Alternatively, the user may perform frame extraction locallyand then offload specific frames to the cloud. In this case,it needs to determine whether each frame is processed by(ii) frame offload in which the frame is sent to the cloud fordetection or (iii) local detection where the frame is detectedon the mobile device.

In CrowdVision, we consider both the task issuer’s queryrequirements and the resource usages of mobile devices.From the perspective of the task issuer, the quality ofCrowdVision’s response to the query can be evaluated bytwo metrics. First is the timeliness of the response, mea-suring the time required to process all related videos oneach mobile device. Second, the accuracy of the response ismeasured by the frames identified as containing the eventthat actually contains the event, which is determined by theCNN model. In this work, we consider timeliness. Fromthe users’ perspective, its primary criteria is the resourceefficiency. Moreover, although the computational capabilityof mobile devices continues to improve, video processingusing deep learning on mobile devices is still limited andresource intensive. Therefore, the cloud is provided by taskissuers to assist in video processing.

When mobile devices perform video processing withoffloading, they may be connected to the cloud via WiFi orcellular networks. In these two circumstances, users mayhave different priorities in terms of budgeting resources.When using WiFi, users may not care about the resourceusage as long as video processing does not affect the normaluse of their mobile devices; or they may only care aboutbattery life when battery recharging is not accessible. Whenusing cellular networks, battery life and data usage may be-come the primary concerns since it is usually inconvenientto recharge mobile device when connected cellular networksand cellular data plans may be limited. Therefore, the designof CrowdVision takes both of these into consideration.

Through the characterization of processing time, energy,and cellular data usage for video processing, we designCrowdVision. By considering inputs from the processingtask, constraints of users, and various network scenarios, weoptimize cloud-assisted video crowdprocessing as a func-tion of these design parameters. CrowdVision determines

all 15 10 6 5 3 10

10

20

extracted frames per second

processin

gtim

e(s)

1

(a) total processing time

all 15 10 6 5 3 10

10

20

extracted frames per second

processin

gtim

e(m

s)

1

(b) processing time per frame

Fig. 2. Processing time of different extraction rates on a 30-second1080P video using DSP on smartphone.

how to quickly and efficiently process each video and takesadvantage of some actions that can be done in parallelor pipelined. In the following, we describe each of theseprocessing components.

3.1 Frame ExtractionFrame extraction is used to take individual video frames andtransform them into images upon which object detectionmay be performed. To target objects with different dynamicswithin the video, the task issuer may request a differentframe extraction rate. For example, for an object moving ata high speed like a car, the rate should be high enough to notmiss the object. For a slowly moving object like a pedestrian,a lower frame extraction rate can be used, which eases thecomputing burden. The setting of the frame extraction ratefor object detection is important but it is not the focus ofthis paper. CrowdVision takes frame extraction rate as aparameter defined by the task, denoted as er frame persecond (fps).

Fig. 2a and 2b, respectively, illustrate the total processingtime and the processing time per frame for frame extractionusing a hardware codec (DSP) with different extraction rateson a 30-second 1080p video (30fps) on a Galaxy S5 (unlessstated otherwise all measurements are performed on theGalaxy S5). In Fig. 2a, the total processing time for extractingall the frames takes about 13 seconds, and it decreases asthe extraction rate reduces. The processing time per frame,as illustrated in Fig. 2b, slightly and linearly increases as theextraction rate decreases, and the average is about 16 ms.

Moreover, as shown in Table 1, the power of frameextraction is about 1W, and the CPU usage is only about4%.

3.2 DetectionDetection is the process of determining if the object of in-terest is in a frame. For detection, we currently use AlexNet[24], a 8-layer CNN, on Caffe to perform classification, butwe do not have any restriction on CNN models1. Although

1. The processing time of detection varies over different CNN mod-els. However, the processing time of detection is just a system parame-ter in CrowdVisionand CrowdVision supports any CNN model.

TABLE 1Power and CPU usage of frame extraction and object detection on

smartphone

Power CPU usage

Frame Extraction 1178 mW 4%Object detection 2191 mW 25%

4

0 5 10 150

5

10

α = 240msβ = 400ms

# of processed framesprocessin

gtim

e(s) individually

in batch

1

Fig. 3. Processing time of detection performed individually or in batchon smartphone.

Caffe can be accelerated by CUDA-enabled GPUs [25], itis currently not supported by GPUs on off-the-shelf mobiledevices. Caffe in both the cloud and mobile devices is thesame, but the cloud is equipped with powerful CUDA-enabled GPUs and can perform object detection hundredsof times faster than mobile devices.

We find that the processing time of detection is affectedby the batch size, e.g., processing two frames in a batch takesless time than that of two frames that are detected individu-ally. Fig. 3 shows the measured processing time of detectionperformed individually and in a batch. The processing timegrows linearly with the increase of the number of framesfor both cases. However, batch processing performs muchbetter, where the intercept (α) is about 240ms and the slope(β) is 400ms. That means if we process ten frames one byone, it takes about 6.4s, while if ten frames are processed ina batch, it takes about 4.24s. The difference grows with theincrease of the number of frames. For detection, it is betterto put more frames in a batch to reduce processing time.However, the system must wait longer to get the extractedframes from the video. Therefore, it is difficult to determinethe best batch size.

This characteristic of batch processing commonly existsin the computing of CNNs on both CPUs and GPUs, andit also drives the design of CrowdVision’s modules of lo-cal detection and frame offload. Although CrowdVision isdesigned based on the detection using mobile CPUs, it canbe easily adapted to mobile GPUs when they are availablefor the acceleration of deep learning on off-the-shelf mobiledevices, because this just requires a change of system pa-rameters. Moreover, despite the fact that mobile GPUs canperform several times faster than CPUs, offloading may stillbe needed since CNN models go deeper and computingbecomes much more difficult (e.g., the state-of-the-art CNN,ResNet [26], has more than a thousand layers).

Table 1 gives the measured power and CPU usage ofperforming detection on the smartphone. The power is2191mW, while the CPU usage is 25% (occupied one offour cores). Although object detection requires considerablecomputation, together with frame extraction, it will notaffect the normal operations of mobile devices, and thuswe also consider performing video processing on mobiledevices locally.

3.3 Offloading

Mobile devices can choose between video offload and localprocessing as depicted in Fig. 1. When choosing local pro-cessing, they then decide between frame offload and local

detection. These decisions are all affected by network con-ditions that lead to divergent throughput and power. More-over, mobile devices can be connected to the cloud via eitherWiFi or cellular. Each may have different characteristics, andusers may also have different concerns in each scenario.Therefore, we separate these two scenarios throughout thedesign of CrowdVision for easy of presentation. However, itdoes not mean that the solution introduced for one scenarioworks exclusively in that scenario. It depends on situationalcharacteristics.

Note that due to the characteristic of batch processing,none of existing offloading schemes can directly and effec-tively apply to deep learning for video processing becausethey cannot take advantage of batch processing to reducethe completion time and energy expenditure.

4 PROCESSING UNDER WIFI

When mobile devices are connected to the cloud with WiFi,energy usage may or may not be the concern. Therefore,in this section, we study the optimization of the process-ing time on a mobile device, and then we investigate theoptimization while also considering energy usage as a con-straint, which can be imposed by the user to govern theamount of energy the device uses for each task.

4.1 Optimizing Completion Time

When a mobile device receives a task, it first parses thetask to find the related videos based on metadata, such aslocation information, or timestamp. The completion time isthe time spent processing these related videos. We say avideo is processed when the video is offloaded to the cloud,or frames are extracted and processed by some combinationof the cloud and local device. Multiple videos are processedin serial; i.e., we do not consider parallel video offloadingand local processing, since it is not resource-efficient. Asvideos are processed in serial, optimizing the completiontime of all related videos on a mobile device is equal tominimizing the completion time of each video. Therefore,for each video, CrowdVision first estimates and comparesthe completion time of video offload and local processing.

Mobile devices are usually connected to WiFi when usersare at home or at work. In such environments, the channelconditions commonly vary slightly. Therefore, similar toMAUI [7], for the processing under WiFi, the data ratebetween a mobile device and cloud can be considered to bestable during a short period time. Note that the environmentwith a highly dynamic WiFi data rate can be handled bythe adaptive algorithm discussed in §5. Like MAUI, mobiledevices probe the data rate by sending 10KB data to thecloud periodically. Let r denote the data rate.

For each video, the processing time of video offload canbe easily estimated. However, for local processing, we haveto consider the time spent on extracting a frame, offloadinga frame, and detecting a frame, and then choose betweenframe offload and local detection wisely for each frame toobtain minimal completion time of local processing.Processing time on frame extraction. As shown in Fig. 2b,the processing time of extracting a frame linearly increasesas extraction rate decreases. Given an extraction rate defined

5

by the task, we can easily obtain the processing time toextract a frame from a specific video. Note that videos withdifferent resolutions (e.g., 720p, 1080p, 4K) have varied pro-cessing times due to different decoding workload. However,the processing time on the frames from videos with the samespecifications varies only slightly based on our experiments.Processing time on local detection. As illustrated in Fig. 3,the processing time of frame detection can be calculated asα + βx, where x is the number of frames included in aprocessing batch. However, since extracted frames do notbecome available at the same time, and batch processingrequires all the frames to be processed to be availablebefore processing, the optimized processing time of multipleextracted frames cannot be simply computed as above. Thisproblem turns out to be non-trivial.

Assume there will be totally n frames extracted froma video. Each frame will be available at a time interval γ(i.e., the time to extract a frame). To minimize the processingtime, we need to optimally determine how to process theseframes, i.e., how many processing batches are needed andhow many frames each batch should process.

To mathematically formulate the problem, let us assumethe number of batches is also n (n can be seen as themaximum number of batches needed to process the frames)and let yi denote the number of frames for batch i ∈ [1, n],where yi ≥ 0 (yi = 0 means batch i does not process anyframe). Let xi denote the time interval between the starttimes of batch i and i+ 1, x0 denote the waiting time beforeprocessing the first batch, xn denote the processing time ofthe last batch. Note that if yi and yi+1 are both zero, xi isalso zero. Then, our problem can be formulated as an IntegerLinear Programming (ILP) problem as following:

minn∑i=0

xi (1)

s.t.n∑i=1

yi = n, (2)

k∑i=0

xi ≥k+1∑i=1

γyi, k = 0, . . . , n− 1 (3)

xi + biα ≥ α+ βyi, i = 1, . . . , n (4)n(1− bi) ≥ yi ≥ 0, i = 1, . . . , n (5)bi = {0, 1}, i = 1, . . . , n. (6)

Constraint (2) regulates that all frames are processed. Con-straint (3) makes sure that the number of frames to be pro-cessed by a batch are available before the start of the batch.Moreover, the previous batch must have been completedalready before processing a batch, which can be representedas xi ≥ α + βyi if yi > 0 and else xi ≥ 0. Constraints(4)(5)(6) are the workaround for this. If yi > 0, bi is zeroaccording to constraint (5). If yi = 0, constraint (4) is equalto xi+biα ≥ α. As our problem is a minimization problem, bwill be equal to one. Since the problem (1) is ILP (NP-hard),the optimal solution of minimizing the processing time ondetection costs too much, even for a small n. For example,for a instance of n = 100 with parameters obtained fromFig. 2 and 3, GLPK takes 217 seconds to solve the problemoptimally on a 16-core workstation.

Processing time on frame offload. Besides detecting frameslocally, mobile devices can also choose to offload frames tothe cloud to accelerate the processing. Let δ denote the timea mobile device spends offloading a frame, where δ = df/rand df is the data size of an extracted frame. Note thatthe extracted frame may be resized to feed different CNNmodels and thus df is not a fixed size. When offloading aframe takes less time than extracting a frame (i.e., δ ≤ γ),the completion time of local processing is about γn, whichis the processing time of frame extraction. However, whenδ > γ (the most common case), there will be frame backlogand local detection is needed. However, as discussed above,minimizing the processing time of local detection is alreadyan NP-hard problem. The problem that considers both frameoffloading and local detection to minimize the processingtime of local processing is even more difficult to solve.Therefore, we propose a split-shift algorithm to solve theproblem.

4.2 Split-Shift Algorithm

For each extracted frame, we have two options: frame of-fload and local detection, which are two processes workingin parallel to reduce the completion time of a video. Intu-itively, the offloading process should keep sending extractedframes to the cloud. For the detection process, it is better toreduce the number of processing batches (i.e., increase batchsizes), since each additional batch incurs more processingtime. However, as the batch processing requires the inputframes be available before the processing starts, it is betterto not wait too long for the extracted frames. Based on thisintuition, we design the split-shift algorithm.

If only frame offload is deployed, the completion timeis δn (accurately it is δn + γ, but γ is small and cannot bereduced and thus we consider δn for simplicity). We treatthe detection process as a helper to reduce δn. The main ideais to shift the workload from the offloading process to thedetection process to balance the completion times of thesetwo processes by determining the number of processingbatches and the number of frames to be processed in eachbatch.

First, let us assume that all frames are available at thebeginning. Then, let n∗p denote the number of frames to bedetected locally that minimizes the completion time, and wehave

δ(n− n∗p) = α+ βn∗p,

which can be employed to approximate the case whereδ � γ and α � γ. Moreover, n∗p can be seen as themaximum number of frames to detect for all cases. Asframes are extracted at a certain rate, the number of framesfor location detection should be less than n∗p.

Since the offloading process keeps sending frames to thecloud, for the detection process, the extracted frame arrivesevery δγ

δ−γ , denoted by γ′. Then, frame detection can bedetermined by the following steps. Let b denote the numberof processing batches and initially b = 1.1. First, we compare n∗pγ

′ and α. If n∗pγ′ ≤ α, then we can

calculate np by solving

δ(n− np) = npγ′ + α+ βnp,

6

where np = b δn−αγ′+β+δ c. For this case, there is only one

processing batch and np frames.2. If n∗pγ

′ > α, which means the waiting time before n∗pframes are available is more than the processing timefor an additional processing batch (i.e., α), it is better toschedule more than one batch. Therefore, we increase bby one. Then, n1

p, i.e., the number of frames of the firstprocessing batch, will be calculated by

n1pγ′ + α+ βn1

p ≥ n∗pγ′

to guarantee that all other frames (i.e., n∗p − n1p) are

available for detection after n1p is processed, and n1

p =

dn∗pγ′−α

γ′+β e.3. However, there may still be n1

pγ′ > α. If so, it is better

to split the first processing batch into two, similar to theprevious step. The split process continues until n1

pγ′ ≤ α

and then we have the number of scheduled batches andalso the number of frames for each batch.

4. n∗p is derived based on the assumption stated above.However, the frames to be locally detected must be lessthan n∗p. Therefore, we need to rebalance the completiontime between these two processes by shifting framesfrom local detection to frame offloading. To do so, firstwe calculate np by

δ(n− np) = n1pγ′ + αb+ βnp.

If∑bi=1 n

ip − np ≥ nbp, decrease b by one and recal-

culate this equation. The shift process is repeated until∑bi=1 n

ip − np < nbp and nbp is set to nbp + np −

∑bi=1 n

ip.

Finally, local detection will process total np frames by bbatches, and each batch i will process nip frames.The computational complexity of the algorithm is O(n),

which is desirable for mobile devices. Moreover, it is alsoeasy to be implemented on mobile devices; i.e., the offload-ing process simply keeps sending frames one by one untilframe queue is empty, while the detection process initiatesbatch processing when the number of frames required byeach batch are available in the frame queue.

Overall, for each video, a mobile device first estimatesthe completion time of video offload and local processing,respectively, and then chooses the approach that has a bettercompletion time.

4.3 Optimization with Energy Constraint

In this section, we first look into the energy consumption ofeach processing module and then investigate the problem ofoptimizing the completion time with an energy constraint.

Fig. 4 shows the measured power of WiFi transmissionin terms of different data rates on the smartphone. Thepower linearly increases with the data rate, similar to thatshown in [27]. From the experiment, the sending state ofWiFi consumes about 1W, denoted by ei, and the powerincreases about 25mW for every 1Mbits/s increase in datarate, denoted by ∆e. For a video v with data rate dv andlength lv , the energy consumption of offloading video vusing WiFi can be represented as

Ev =dvlvr

(ei + r∆e).

0 2 4 60

1

2

3

data rate (MB/s)

power

(W)

1

Fig. 4. Power of WiFi in terms of different uplink rates on smartphone.

Let ee denote the power of frame extraction and ep denotethe power of frame detection, which are measured on thesmartphone and shown in Table 1. Then, the energy con-sumption of frame extraction is

Eev = lvreγee.

The minimal energy consumption of locally detecting allframes extracted from video v is

Epv = (α+ βlvre)ep.

Local detection can yield Epv only when all frames are pro-cessed in one batch. Moreover, the energy cost of offloadingall frames extracted from video v is

Eov =lvredfr

(ei + r∆e).

When considering energy consumption only, it is better tooffload frames rather than to perform local detection whenEpv > Eov . Also, it is straightforward to decide betweenvideo offloading and local processing based on Ev , Eev , Epvand Eov . However, the optimization of energy consumptionmay not necessarily optimize the completion time; i.e., wecannot always optimize the energy consumption and com-pletion time simultaneously.

Since the minimum and the maximum energy consump-tion for video processing can be easily calculated, we willlet users to choose between the minimum and maximumamount of energy to be consumed. Then, the selectedamount of energy E′ can be used as a constraint to optimizethe completion time.

Let Ep denote the sum of Eev and Epv , Eo denote thesum of Eev and Eov , Emin = min{Ev, Ep, Eo}, and Tmin =min{T ∗, Tv}, where T ∗ denotes the minimum completiontime of local processing and Tv denotes the completion timeof video offloading. Since the processing option that con-sumes the most energy may not be the fastest one and videoprocessing can only be accelerated locally by parallelizingframe offloading and local detection, we choose the energyconsumption of the fastest processing as the maximumenergy consumption, denoted as Emax = E(Tmin). By doingso, we can guarantee that, given an energy constraint, thecompletion time can be optimized.

In general, the solution to optimize the completion timewith an energy constraint works as follows. First, we caneasily calculate the energy consumption of each processingoption and leverage the split-shift algorithm to computeT ∗, and accordingly E(T ∗). Then, by comparison, we haveTmin, Emin, and Emax. When Emin = Ev and Tmin = Tv ,video offloading is the most efficient and fastest way; i.e.,

7

it optimizes the energy consumption and completion timesimultaneously. When Emin = Ev and Tmin = T ∗, thenwe need to obtain the minimum completion time for localprocessing with the energy constraint E′. Let Tmin(E′) de-note the minimum completion time. If Tmin(E′) > Tv , videooffloading will be selected, otherwise, the local processingthat produces Tmin(E′) is chosen. When Emin = Eo orEmin = Ep, we need to compare the local processing thatyields Tmin(E′) with video offloading. If Tmin(E′) ≥ Tvand E′ ≥ Ev , video offloading is preferred, otherwise, localprocessing is selected.

Therefore, we can see that to process each video, we haveto solve the optimization problem that minimizes the com-pletion time of local processing with an energy constraint.This problem is non-trivial when Emin < E′ < Emax anddfr > γ. To solve the problem, we need to consider two

cases: Emin = Eo and Emin = Ep.Frame Offloading with Extra Energy. When Emin = Eo,then Eo < Ep, which generally means offloading a frameconsumes less energy than detecting a frame. Therefore, theproblem is how to maximally reduce the completion timeby parallelizing local detection with frame offloading byexploiting the extra energy ∆E = E′ − Emin. The localdetection can be determined as follows to obtain Tmin(E′).1. The spending of the extra energy can be represented as

∆E = (bα+ βnmax)ep − nmaxdfr

(ei + r∆e). (7)

The first part of the r.h.s of (7) is the energy cost to per-form detection on nmax frames and the second part is theenergy to offload nmax frames. From (7), the maximumnumber of locally detected frames can be calculated as

nmax = b ∆E − bαepβep − dfei

r − df∆ec.

Note that nmax frames are processed by b batches, whereb is set to one initially.

2. Let T pmin(nmax, b) denote the optimized completion timeof local detection, which can be obtained by solving (1)and approximated by the split process of the split-shiftalgorithm. If T pmin(nmax, b) ≤ (n − nmax)

dfr , Tmin(E′) =

(n− nmax)dfr .

3. If T pmin(nmax, b) > (n − nmax)dfr and b = 1, when

nmaxγ′ ≤ α, the local detection completion time can-

not be reduced by increasing the number of processingbatches. Therefore, we need to reduce nmax to ensurenmaxγ

′ + α + βnmax ≤ (n − nmax)dfr and then we have

nmax = b ndf−αrdf+rβ+rγ′ c and Tmin(E′) = (n− nmax)

dfr .

4. If T pmin(nmax, b) > (n− nmax)dfr , b = 1 and nmaxγ

′ > α,or if T pmin(nmax, b) > (n − nmax)

dfr and b > 1, we

need to examine whether the increase of the number ofprocessing batches can reduce the local detection com-pletion time. If the completion time cannot be reduced,Tmin(E′) = T pmin(nmax, b); otherwise, b is increased byone. Since increasing the processing batches increases theenergy consumption in order to process nmax frames (bythe additional α), we need to recalculate nmax accordingto (7) and thus the process has to be repeated. Theprocess stops until T pmin(nmax, b) cannot be decreased,i.e., T pmin(nmax, b) ≤ T pmin(nmax, b+ 1).

Local Detection with Extra Energy. When Emin = Ep, theextra energy ∆E = E′ − Ep can be utilized to increase thenumber of processing batches, and it can also be used forframe offloading. Therefore, we need to compare these twooptions in terms of energy cost and completion time, whichcan be determined as follows.1. Initially, since Epv is the minimal energy consumption

of local detection, we have one processing batch and nframes to be processed; i.e., b = 1 and nmax = n.

2. For each iteration, first we calculate γ′ (i.e., δγδ−γ )

and T pmin(nmax,b). Then, we compare Tpmin(nmax,b)

αep

−Tpmin(nmax,b+1)

αepand rβ

df (ei+r∆e)−rβep , which representthe efficiency of increasing the processing batch andframe offloading, respectively, to reduce the completiontime in terms of energy cost.

3. We will choose the more efficient option for each itera-tion. If the processing batch is selected, we reduce ∆E byαep and increase b by one, and then repeat the process. Ifframe offloading is more efficient, we first decrease nmax

by one and then determine whether n−nmax frames canbe offloaded before nmax frames are detected locally. Ifso, we need to update δ to Tp

min(nmax,b)

n−nmax, which will be

used to recalculate γ′. Due to the change of γ′, we needto reset b = 1 and ∆E = E′ − ep(α + βnmax) − df

r (n −nmax)(ei + δer), and repeat the process.

4. The process stops when ∆E ≤ 0 or when n − nmax

frames cannot be offloaded within T pmin(nmax, b). Finally,local detection will process nmax frames using b batches,n − nmax frames will be offloaded, and Tmin(E′) =T pmin(nmax, b).The solutions of both cases have low computational

complexity O(n2) and can be easily implemented on mobiledevices to obtain the completion time of a video with energyconstraints.

5 PROCESSING UNDER CELLULAR

When mobile devices have only cellular connections, it isbetter to not offload videos, due to the limitation of cellularuplink speed and limitations on data usage. Since datatransmission rates may vary widely over time, e.g., due tomovement, the solution for processing under WiFi may notbe valid for the cellular scenario.

Obviously, if all frames are detected locally, there is nocellular data usage. However, this may consume signifi-cant amounts of energy and severely increase completiontime. Although users may be more sensitive about datausage rather than energy, energy usage is a concern. Datausage can be easily controlled, while the energy cost offrame offload varies with data transmission rate and signalstrength, and thus it is hard to tell how much energy will beconsumed to offload a frame beforehand.

Therefore, for processing under cellular, we consider theproblem of optimizing both completion time and energyconsumption with a data usage constraint so that decisionsare made while considering tradeoffs between these twoobjectives. However, due to the variation of cellular datarates, we cannot solve the problem traditionally by Paretooptimal solutions. Thus, we design an adaptive algorithmthat makes a decision on each extracted frame (i.e., between

8

0 200 400 600 8000

1

2

3

data rate (KB/s)po

wer

(W)

1

Fig. 5. Power of LTE in terms of different uplink rates on smartphone.

frame offload and local detection) towards optimizing bothobjectives.

To perform video processing under cellular, users mustspecify a data usage constraint D′ between zero and a max-imum, which can be easily calculated, i.e., video duration× frame extraction rate × frame data size. If D′ is equalto zero, the problem is trivial and all extracted frames aredetected locally. If the user does not care data usage, D′ issimply set to the maximum. In the following, we considerthe case that D′ is greater than 0.

5.1 Estimation of Uplink Rate and PowerBefore deciding between detection and offloading for eachframe, we need to know the cost in terms of processingtime and energy. For frame detection, both processing timeand power are stable and can be accurately estimated,although the processing time varies with the number ofprocessing batches. The difficulty lies in estimating these forframe offload. The offloading time and power are related tomany factors, such as signal strength, channel conditionsand network traffic. However, from measurements on thesmartphone, we can see the power of LTE uplink exhibitsan approximately linear relationship with data rates asdepicted in Fig. 5. Similar results are also found in [28].Therefore, during a short period time, we can explore thehistory of previous frame offloads to correlate data ratesand power. Moreover, among the factors that affect cellularuplink rates, signal strength, which is mainly impacted bythe users’ location and movement, can be treated as anindicator of data rate during a short period of time, wherethe coefficients of the linear relation between data rates andpower are steady.

To estimate the uplink data rate and power, first werecord the data rate and energy consumption for eachoffloaded frame. Then, these records can be exploited toderive the linear relation between data rate and power usingregression, to maintain an up-to-date correlation estimate.Moreover, we also record the cellular signal strength levelduring each frame offload. Since generally data rate has alinear relationship with signal strength during short periodsof time, we can exploit current signal strength level toroughly estimate current data rate based on the records ofpreviously offloaded frames. Then, the estimated data rate isexploited to gauge energy consumption to offload a frame.

5.2 Adaptive AlgorithmAs aforementioned, we try to optimize completion timeand energy consumption together. However, due to the

dynamics of cellular uplink data rate and power, we proposea tailored solution for this specific problem rather than ascalar treatment of these two objectives.

For the processing under cellular, we have two options:frame offload and local detection. Frame offload may im-prove completion time or energy consumption or both,depending on the cellular data rate and power consumedby cellular during offloading. For a number of frames, localdetection can be exploited to reduce the completion time byincreasing the number of processing batches, although thisincurs an additional energy cost. Moreover, local detectionusually takes multiple frames as input, and once it startsprocessing, the frames should not to be offloaded to avoidduplicate processing. Therefore, it is difficult to determinethe number of frames included in batch processing, sinceoffloading frames may improve performance during batchprocessing.

Frame offload may be exploited to reduce both com-pletion time and energy consumption simultaneously; localdetection cannot optimize these two objectives together, andhence it may be preferred only when both completion timeand energy cannot benefit from frame offload. In addition,we do not parallelize frame offload and local detection, sincethis always sacrifices one objective for another. Based theseconsiderations, we design an adaptive algorithm for the pro-cessing under cellular, towards optimizing both completiontime and energy cost with the cellular data usage constraint,which works as follows.

Frames are continuously extracted from a video into aframe queue. When a frame is available, a mobile devicewill offload a frame to the cloud. If both the offloading timeand energy consumption are less than those estimated forlocal detection, this offloading is called a successful attempt;otherwise, it is referred to as an unsuccessful attempt.

After the first successful attempt, the mobile device willoffload another frame. If this is also a successful attempt,then we can derive the linear relationship between datarate and energy consumption, and the relation betweensignal strength and data rate. For subsequent offloads, themobile device can estimate the completion time and energyconsumption based on current signal strength and the linearfunctions. If both are less than those estimated for localdetection, it will offload another frame.

Whenever an attempt is unsuccessful, or the estimateindicates an unsuccessful attempt will occur, the mobiledevice will switch to local detection and set a backoff timer toω for frame offload (i.e., frame offload will not be performedduring ω). Let T pmin(nr) denote the minimal completion timeif all the remaining frames nr are detected locally, no denotethe number of previously offloaded frames, and nd denotethe maximum number of frames that can be offloaded to thecloud under the data usage constraint D′, where nd = bD

′

dfc.

Then, ω =Tpmin(nr)

nd−no2u, where u is the number of consecutive

unsuccessful attempts.For local detection, each time the mobile device will

process as many frames from the frame queue (it may waitfor frames to be extracted from a video), which can becompleted within ω. After a timeout of the backoff timer, themobile device will switch back to frame offload. The processiterates until frame offload reaches the limit nd or all frames

9

t

successful attemptfailled attempt

ω =2T p

min(n−2)nd−2

failled attempt

ω =22T p

min(n−6)nd−3

successful attempts

ω =2T p

min(n−14)nd−6

estimated failledattempt

Local detection

Frame o�load

1

Fig. 6. Illustration of video processing under cellular.

are processed. If limit is reached, then local detection will bethe only option to process the remaining frames.

Frame offload and local detection are performed al-ternately to find the moments when frame offload canimprove both energy and completion time, or when localdetection should be performed. The backoff timer exponen-tially increases with the number of consecutive unsuccessfulattempts. This is designed to capture different networkconditions. When the network condition is constantly poor,offloading is less frequently attempted and the frequency isexponentially reduced. When the network condition varies,offloading is attempted more frequently to seize the mo-ments when frame offload benefits both.

We use Fig. 6 as an example to illustrate how theadaptive algorithm works. Since the first offloading attemptis successful, it offloads the second frame. However, theoffloading time is more than detecting a frame locally. Thus,the frame offloading backoff timer is set at ω =

2Tpmin(n−2)

nd−2and local detection is performed instead during ω. After thetimeout, another frame is offloaded, but it fails. Therefore,the mobile device switches to local detection again, andthe backoff timer is set to 22Tp

min(n−6)

nd−3 since there are twoconsecutive attempt failures. After switching back to frameoffload, several successful attempts are made, and hence themobile device can estimate the data rate and the energyto be consumed based on the signal strength. However, asubsequent estimate indicates that the next attempt will beunsuccessful and hence it performs local detection instead.

Instead of parallelizing frame offload and local detection,we choose to perform them alternatively. Frame offload isemployed to optimize both completion time and energyconsumption. However, due to the variation of network con-ditions, frame offload is selected only if it outperforms localdetection in terms of both completion time and energy andmeets the data usage constraint. The adaptive algorithm issimple and efficient, and it can also be easily implemented.

6 IMPLEMENTATION

We have implemented CrowdVision on Android and aworkstation with a GTX TITAN X GPU as the cloud to assistmobile devices for video crowdprocessing. Tasks are issuedfrom the workstation to mobile devices through GoogleCloud Messaging. Messaging between the cloud and mobiledevices are implemented using Protocol Buffers. The archi-tecture and App of CrowdVision are illustrated in Fig. 7. Theimplementation of CrowdVision on mobile devices consistsof three components: core services, monitors, and a GUI.Core Services. Core services contain an executor service,a frame extraction callable, a Caffe callable, an offload-ing callable, a multithreaded frame queue, and a video

Fig. 7. Architecture and App of CrowdVision.

database. The video database stores the metadata of localvideos such that CrowdVision can quickly screen videos forthe processing task. The Multithreaded frame queue storesextracted frames from videos and supports multithreadingaccess. The executor servce is able to call any of the callablesto perform frame extraction, Caffe for frame detection, oroffloading of a frame or video. The executor service takesinputs from tasks, GUI and monitors, and employs theselected strategy to process each video.Monitors. There are two monitors to measure network stateand battery level, which provide inputs to the core services.The network monitor tells the cores services current net-work connection with distinct metrics for WiFi or cellularnetworks. For WiFi, it measures the uplink data rate froma mobile device to cloud using probing data. For cellular,it monitors the signal strength level. The battery monitormeasures the energy consumption for each offloaded framefor the cellular case.GUI. The GUI allows mobile users to configure the energyusage, cellular data usage, and access to videos. Energycontrol ranges from the minimum to the maximum energycost of local video processing. It is enabled using WiFi. Forcellular, the data usage limits can be specified by users.Additionally, users can select videos to not be processedby CrowdVision, for privacy concerns or any other manualfiltering requirements.

7 EVALUATION

In this section, we evaluate the performance of Crowd-Vision. We first compare CrowdVision against alternativesunder various system settings based on empirically gath-ered measurements to understand when and why Crowd-Vision outperforms alternatives and then confirm the gainsof CrowdVision through experiments on our testbed. Wealso investigate how the split-shift algorithm approximatesthe optimum for determining processing batches.

7.1 Performance under Various SettingsWe first use empirically gathered measurements of process-ing time and energy taken from a Galaxy S5 to investigatethe impact of system parameters, such as WiFi/cellulardata rate, frame data size and frame extraction rate. UnderWiFi, CrowdVision is compared against MAUI [7] (the mostpopular generic computation offload system), where MAUIis adapted to optimize the completion time. Under cellular,

10

0 1000 2000 30000

20

40

60

80

WiFi data rate (KB/s)

completiontim

e(s)

MAUICrowdVision

1(a) df = 500KB/s, re = 6fps

0 200 400 600 8000

10

20

30

40

frame data size (KB)

completiontim

e(s)

MAUICrowdVision

1(b) re = 6fps, r = 2000KB/s

0 5 10 15 200

10

20

30

40

frame extraction rate (fps)

completiontim

e(s)

MAUICrowdVision

1(c) df = 500KB/s, r = 2000KB/s

Fig. 8. Performance of CrowdVision and MAUI under WiFi in terms of WiFi data rate, frame data size, and frame extraction rate.

as MAUI does not adapt to varying data transmission rates,CrowdVision is compared against basic processing options(i.e., solely frame offload and solely local detection). Thesimulation is carried out on a 1080p 30-second video. Notethat the video specification and duration do not affect theirrelative performance.WiFi. Fig. 8a illustrates the completion time of MAUI andCrowdVision in terms of WiFi data rate, where the framedata size df is 500kB and frame extraction rate re is 6fps.When the WiFi data rate is high enough, e.g., 2620KB/s inFig. 8a, video offload costs the least and both MAUI andCrowdVision choose video offload and hence perform thesame. Similarly, when the WiFi data rate is low enough,local detection will be selected by both MAUI and CrowdVi-sion and thus they perform equivalently again, e.g., 20KB/sin Fig. 8a. When between these two rates, CrowdVision out-performs MAUI. This is because the split-shift algorithmemployed by CrowdVision can improve completion time bytaking advantage of batch processing and by parallelizingframe offload and local detection. The completion time ofCrowdVision is as low as 60% of MAUI as depicted inFig. 8a.

Since CNN models may require different resolutions ofimages as input, we also investigate the effect of frame datasize on the completion time. As illustrated in Fig. 8b, similarto the effect of WiFi data rate, when frame data size is largeenough, it is always better to offload videos. When framedata size is small enough, frame offload is the best option.

0 20 40 60 80 10080

100

120

140

(E′ − Emin)/(Emax − Emin) (%)

energy

consum

p.(J)

Energy Consump.

30

60

90

completiontim

e(s)Completion Time

1(a) Emin = Eo

0 20 40 60 80 100150

200

250

(E′ − Emin)/(Emax − Emin) (%)

energy

consum

p.(J)

Energy Consump.

60

70

80

completiontim

e(s)Completion Time

1(b) Emin = Ep

Fig. 9. Optimized completion time with energy constraint, where (a) r =500KB/s, df = 200KB, re = 6fps, (b) r = 40KB/s, df = 200KB, re =6fps.

In both regions, MAUI and CrowdVision perform equally.However, between them, CrowdVision always outperformsMAUI as depicted in Fig. 8b due to the same reason dis-cussed above.

To detect different objects, different frame extractionrates may be defined by the task issuer. Similar to the effectof WiFi data rate and frame data size, CrowdVision outper-forms MAUI as shown in Fig. 8c.

Fig. 9 illustrates the completion time with differentenergy constraints between Emin and Emax. In Fig. 9a,Emin = Eo (i.e., frame offload consumes the least energy),Emax is the energy consumption of the fastest processingoption. When the energy constraint increases from Emin toEmax (i.e., from 80J to 120J), the completion time graduallydecreases from 75 seconds to 35 seconds. Fig. 9b shows thecase that Emin = Ep (i.e., local detection consumes the leastenergy). As the energy constraint is relaxed, the completiontime approaches the minimum. From Fig. 9 and Fig. 9b, wecan see that our solution acts as a linear function to correlatecompletion time and energy constraint.Cellular. Based on the measurements of the LTE module ona Galaxy S5 under different uplink rates, as illustrated inFig. 5, we perform the evaluation over cellular networks.

To model the dynamics of cellular data rate, we adopt aMarkov chain [29]. Let R denote a vector of transmissionrates R = [r0, r1, . . . , rl], where ri < ri+1. The Markovchain advances at each time unit. If the chain is currentlyin rate ri, then it can change to adjacent rate ri−1 or ri+1,or remain in ri, but staying in current rate has a largerprobability than changing. Therefore, for a given vector, e.g.,

Frame O�oad Local Detection CrowdVision0

100

200 147.3

136.2 133.5

ener

gyco

nsum

p.(J

) Energy Consump.

0

70

14084.2

60.6 60.8

com

plet

ion

time

(s)Compeletion Time

1

(a) [20, 100, 150, 250, 400 KB/s]

Frame O�oad Local Detection CrowdVision0

100

200127.9

136.2 125.8

ener

gyco

nsum

p.(J

) Energy Consump.

0

70

140

61.5

60.6 53.6

com

plet

ion

time

(s)Compeletion Time

1

(b) [20, 100, 150, 500, 600 KB/s]

Fig. 10. Performance under different cellular data rates, where df = 100KB, re = 6 fps.

11

0.7 1.66 4.930

10

20

30

18.6

106.88.6 6.7 5.8

WiFi uplink data rate (MB/s)

completiontim

e(s) MAUI CrowdVision

1(a) df = 519KB, re = 1fps

72 252 519020406080100

13.5

46.6

84.6

7.523

43.9

frame data size (KB)

completiontim

e(s) MAUI

CrowdVision

1(b) r = 0.7MB/s, re = 5fps

1 5 100

10

20

1.7

6.9

13.1

1.5

6.3

11

frame extraction rate (fps)

completiontim

e(s) MAUI CrowdVision

1

(c) r = 1.66MB/s, df = 252KB

MAUI CrowdVision0

10

20

9.25.9

106.8

completiontim

e(s) Estimate Measure

1(d) r = 1.66MB/s, df =519KB, re = 1fps

0 50 10020

30

40

23.8

26.628.2

(E′ − Emin)/(Emax − Emin) (%)

energy

consum

p.(J)

Energy Consump.

4

5

65.4

5

4.3

completiontim

e(s)Completion Time

1

(e) r = 1.66MB/s, df =519KB, re = 1fps

Fig. 11. System performance of different processing options with various settings and different WiFi data rates.

of five rates, the transition matrix is defined as

P =

r0 r1 r2 r3 r4

r0

r1

r2

r3

r4

2/3

2/9

0

0

0

1/3

5/9

2/9

0

0

0

2/9

5/9

2/9

0

0

0

2/9

5/9

1/3

0

0

0

2/9

2/3

.

In the experiments, we consider two vectors of cellular datarates, which are R1 = [20, 100, 150, 250, 400] (KB/s) andR2 = [20, 100, 150, 500, 600] (KB/s), and the time unit ofthe Markov chain is two seconds. Moreover, we set the datausage constraint of CrowdVision to half of total extractedframes.

Fig. 10a and 10b illustrate their performance under R1

and R2, respectively. As depicted in Fig. 10a, CrowdVi-sion outperforms frame offload in terms of both completiontime and energy; its performance is similar to local detec-tion. When cellular data rates increase to R2 in Fig. 10b,frame offload performs better than before as expected. Sincecellular data rates do not affect local detection, its perfor-mance remains the same. CrowdVision performs the best. Ithas less completion time than others and slightly less energyconsumption than frame offload, with half of cellular datausage.

CrowdVision outperforms frame offload and local detec-tion under different cellular data rates, because the adaptivealgorithm of CrowdVision is designed to adopt differentprocessing options according to real-time cellular data rateand energy to be consumed. Generally, when the cellulardata rate is high, it tends to offload frames, otherwise, itis apt to perform local detection. Moreover, the backoffmechanism of the adaptive algorithm avoids unnecessaryframe offload when the cellular data rate is low.

7.2 Performance on TestbedTestbed. We deployed CrowdVision on a Galaxy S5, whichcan connect to the Internet using either WiFi or cellular. Theworkstation can be reached by a public IP address. For theexperiment using WiFi, we configured a WiFi router withdifferent 802.11 protocols to get different uplink data rates.The experiments under cellular (4g LTE) were carried out atthree different locations (i.e., a lab, a lobby, and a restaurantin downtown) to acquire different signal strengths, uplinkdata rates, and traffic conditions. The performance is eval-uated based on the processing of a 1080p 30-second videounder different settings in terms of completion time andenergy, where energy is measured by the Monsoon powermonitor.

WiFi. Fig. 11 illustrates the completion time of MAUI andCrowdVision. The experiments are conducted using thefollowing parameters: uplink data rates (0.7, 1.66 and 4.93MB/s), frame data sizes (72, 252, and 519 KB, which arethe sizes of JPG images with resolutions 640×360, 1280×720and 1920×1080, respectively), and frame extraction rates (1,5, 10 fps). Fig. 11a, 11b and 11c depict the completion timein terms of WiFi uplink data rate, frame data size and frameextraction, respectively. They exhibit the similar pattern asdepicted in Fig. 8a, 8b and 8c. CrowdVision outperformsMAUI in all the settings. For example, when WiFi uplinkdata rate is 0.7MB/s, the completion time of CrowdVi-sion is less than a half of MAUI (i.e., 2x speed-up). Theseexperiments further confirm the gain of CrowdVision overMAUI. Moreover, as shown in Fig. 11d, the estimatedcompletion time is very close to the measured completiontime. Therefore, CrowdVision can reliably make the bestdecision based on the estimate. When an energy constraintis imposed, CrowdVision can improve the completion timeas the energy constraint is relaxed, and vice versa, as illus-trated in Fig. 11e, which confirms the effectiveness of theoptimization with energy constraints.Cellular. We measured the average uplink data rates at thethree locations and correlated the performance with theserates, as illustrated in Fig. 12. When the data rate is low,CrowdVision outperforms frame offload in terms of bothcompletion time and energy. Since CrowdVision has to sendsome frames to acknowledge the current cellular data rate,it incurs slightly longer completion time and uses moreenergy than local detection. When the data rate increases,CrowdVision and frame offload perform better than be-fore. CrowdVision has similar completion time with localdetection but uses less energy. When the data rate furtherincreases, CrowdVision is faster and uses less energy thanthe others. When the data rate is sufficiently high such thatoffloading a frame commonly outperforms locally detecting

200 400 6000

20

40

60

80

cellular rates (KB/s)

com

plet

ion

time

(s)

Local DetectionFrame O�oadCrowdVision

1(a) completion time

200 400 6000

20

40

60

80

cellular rates (KB/s)

ener

gyco

nsum

p.(J

) Local DetectionFrame O�oadCrowdVision

1(b) energy consumption

Fig. 12. System performance of different processing options under dif-ferent cellular uplink rates, where df = 252KB, re = 1fps.

12

10 50 1000

20

40

60

# of frames

completiontim

e(s)

OptimumSplit-Shift

1(a) α = 240, β = 400, γ = 16

10 50 1000

1

2

3

4

# of frames

completiontim

e(s)

OptimumSplit-Shift

1(b) α = 40, β = 20, γ = 16

Fig. 13. Comparison between Optimum and Split-Shift.

a frame in both completion time and energy cost, Crowd-Vision tends to offload more frames but avoids the timewhen the data rate is low. Therefore, CrowdVision adapts todifferent cellular data rates to reduce completion time andsave energy.

7.3 Performance of Split-Shift

Although the split-shift algorithm is designed to handlethe local processing under WiFi, it can also be exploited tosolve the ILP problem (1) by assuming δ → +∞ (recall δis the time spent to offload a frame). The ILP problem ismeaningful. It can be employed to minimize the processingtime of deep learning on frames extracted from videos,no matter the processing is performed on CPUs or GPUs.Therefore, we investigate the performance of the split-shiftalgorithm on (1), comparing to the optimum obtained byGLPK using LP relaxation and integer optimization on asmall scale.

Fig. 13a illustrates their comparison with the parametersof batch processing on the CPU of Galaxy S5. We cansee split-shift achieves the optimum under this setting.In Fig. 13b, we use the parameters of Tegra K1 GPU onperforming AlexNet, where α and β is close to γ (thetime spent to extract a frame from a video). Under thissetting, although split-shift continuously deviates from theoptimum with the increase of the frame number, split-shiftis still close to the optimum. The different performanceof split-shift under these two settings can be explainedintuitively as: (i) when detecting a frame takes much longertime than extracting a frame from a video, it is relativelyeasy to determine the optimal batch processing; (ii) whenthey are close, it is much more difficult to do so. This isalso evidenced by GLPK, which spent much more time onfinding the optimal solution under the setting of Fig. 13bthan Fig. 13a for the same number of frames (minutes vs.hours). In summary, split-shift, a suboptimal algorithm withthe complexity O(n), is practical and affordable to solvethe ILP problem, which commonly exists in optimizing theperformance of deep learning applications.

8 DISCUSSION

Crowdprocessing. Unlike crowdsourcing or crowdsensing,we define crowdprocessing as a computing-oriented ap-proach. Instead of simply sensing and sharing data, wefocus on exploring the computing capability of mobiledevices; i.e., mobile devices process their own data using

deep learning and results are centrally collected to solveproblems. For crowdprocessing, the key problem, which isalso the focus of this paper, is to enable resource-constrainedmobile devices to efficiently perform such complex comput-ing.Compatibility. CrowdVision is currently built on top ofCaffe. However, as shown in Fig. 7, Caffe works only asa callable to perform detection on mobile device and cloud.Therefore, it can be easily replaced by other deep learningframeworks. Moreover, CrowdVision can be easily adaptedto mobile GPUs when they are available for the accelerationof deep learning on off-the-shelf mobile devices, because it isjust the change of system parameters. It is worth noting thatthe computing capability of mobile devices even equippedwith mobile GPUs is still far behind GPU-accelerated cloud.Generality. The characteristic of batch processing com-monly exists in the computing of deep learning, not justfor convolutional neural networks. Therefore, CrowdVi-sion that is particularly designed to take advantage of batchprocessing can be used for other applications with minormodifications. Moreover, the split-shift algorithm can beexploited to determine batch processing so as to optimizethe performance and resource usage in these applicationswith/without computation offload.

9 CONCLUSIONS

In this paper, we present CrowdVision, a computing plat-form for crowdprocessing videos using deep learning.CrowdVision is designed to optimize the performance onmobile devices with computational offload and by takinginto consideration the characteristics of the computing ofdeep learning for video processing. CrowdVision is im-plemented and evaluated on the off-the-shelf smartphone.Experimental results demonstrate that CrowdVision greatlyoutperforms the existing computational offload system orbasic processing options under various settings and net-work conditions. We envision CrowdVision to be a greatcomputing framework for crowdprocessing applications us-ing deep learning.

ACKNOWLEDGMENTS

This work was supported in part by the Army Research Lab-oratory under Cooperative Agreement Number W911NF-09-2-0053. A preliminary version of this work will appear inthe Proceedings of INFOCOM 2018 [30].

REFERENCES

[1] Z. Lu, S. Rallapalli, K. Chan, and T. La Porta, “Modeling theresource requirements of convolutional neural networks on mobiledevices,” in MM’17.

[2] D. Yang, G. Xue, X. Fang, and J. Tang, “Crowdsourcing to smart-phones: incentive mechanism design for mobile phone sensing,”in MobiCom’12.

[3] M.-R. Ra, B. Liu, T. L. Porta, and R. Govindan, “Medusa: Aprogramming framework for crowd-sensing applications,” in Mo-biSys’12.

[4] B. Liu, Y. Jiang, F. Sha, and R. Govindan, “Cloud-enabled privacy-preserving collaborative learning for mobile sensing,” in Sen-Sys’12.

[5] P. Simoens, Y. Xiao, P. Pillai, Z. Chen, K. Ha, and M. Satya-narayanan, “Scalable crowd-sourcing of video from mobile de-vices,” in MobiSys’13.

13

[6] K. Kumar, J. Liu, Y.-H. Lu, and B. Bhargava, “A survey ofcomputation offloading for mobile systems,” Mobile Networks andApplications, vol. 18, no. 1, pp. 129–140, 2013.

[7] E. Cuervo, A. Balasubramanian, D.-k. Cho, A. Wolman, S. Saroiu,R. Chandra, and P. Bahl, “Maui: making smartphones last longerwith code offload,” in MobiSys’10.

[8] M. Barbera, S. Kosta, A. Mei, and J. Stefa, “To offload or notto offload? the bandwidth and energy costs of mobile cloudcomputing,” in INFOCOM’13.

[9] C. Shi, K. Habak, P. Pandurangan, M. Ammar, M. Naik, andE. Zegura, “Cosmos: computation offloading as a service formobile devices,” in MobiHoc’14.

[10] Y. Geng, W. Hu, Y. Yang, W. Gao, and G. Cao, “Energy-efficientcomputation offloading in cellular networks,” in ICNP’15.

[11] K. K. Rachuri, C. Mascolo, M. Musolesi, and P. J. Rentfrow,“Sociablesense: exploring the trade-offs of adaptive sampling andcomputation offloading for social sensing,” in MobiCom’11.

[12] M.-R. Ra, A. Sheth, L. Mummert, P. Pillai, D. Wetherall, andR. Govindan, “Odessa: enabling interactive perception applica-tions on mobile devices,” in MobiSys’12.

[13] J. Huang, A. Badam, R. Chandra, and E. B. Nightingale,“Weardrive: fast and energy-efficient storage for wearables,” inATC’15.

[14] M. S. Gordon, D. K. Hong, P. M. Chen, J. Flinn, S. Mahlke, andZ. M. Mao, “Accelerating mobile applications through flip-flopreplication,” in MobiSys’15.

[15] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantizedconvolutional neural networks for mobile devices,” in CVPR’16.

[16] N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, L. Jiao,L. Qendro, and F. Kawsar, “Deepx: A software accelerator for low-power deep learning inference on mobile devices,” in IPSN’16.

[17] P. Georgiev, N. D. Lane, K. K. Rachuri, and C. Mascolo, “Leo:Scheduling sensor inference algorithms across heterogeneous mo-bile processors and network resources,” in MobiCom’16.

[18] S. Han, H. Shen, M. Philipose, S. Agarwal, A. Wolman, and A. Kr-ishnamurthy, “Mcdnn: An approximation-based execution frame-work for deep stream processing under resource constraints,” inMobiSys’16.

[19] S. Rallapalli, H. Qiu, A. J. Bency, S. Karthikeyan, R. Govindan, B. S.Manjunath, and R. Urgaonkar, “Are very deep neural networksfeasible on mobile devices?” in HotMobile’16.

[20] “Caffe,” http://caffe.berkeleyvision.org/.[21] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,

S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecturefor fast feature embedding,” in MM’14.

[22] “Torch,” http://torch.ch/.[23] “Tensorflow,” http://www.tensorflow.org/.[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifi-

cation with deep convolutional neural networks,” in NIPS’12.[25] “CUDA,” http://www.nvidia.com/.[26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for

image recognition,” in CVPR’16.[27] R. Friedman, A. Kogan, and Y. Krivolapov, “On power and

throughput tradeoffs of wifi and bluetooth in smartphones,” IEEETransactions on Mobile Computing, vol. 12, no. 7, pp. 1363–1376,2013.

[28] J. Huang, F. Qian, A. Gerber, Z. M. Mao, S. Sen, and O. Spatscheck,“A close examination of performance and power characteristics of4g lte networks,” in MobiSys’12.

[29] A. Fu, P. Sadeghi, and M. Medard, “Dynamic rate adaptationfor improved throughput and delay in wireless network codedbroadcast,” IEEE/ACM Transactions on Networking, vol. 22, no. 6,pp. 1715–1728, 2014.

[30] Z. Lu, K. Chan, and T. La Porta, “A computing platform for videocrowdprocessing using deep learning,” in INFOCOM’18.