DeepIoT: Compressing Deep Neural Network Structures forSensing Systems with a Compressor-Critic Framework
Shuochao YaoUniversity of Illinois Urbana
Yiran ZhaoUniversity of Illinois Urbana
Aston ZhangUniversity of Illinois Urbana
Lu SuState University of New York at
Tarek AbdelzaherUniversity of Illinois Urbana
ABSTRACTRecent advances in deep learning motivate the use of deep neutralnetworks in sensing applications, but their excessive resource needson constrained embedded devices remain an important impediment.A recently explored solution space lies in compressing (approximat-ing or simplifying) deep neural networks in some manner beforeuse on the device. We propose a new compression solution, calledDeepIoT, that makes two key contributions in that space. First,unlike current solutions geared for compressing specic types ofneural networks, DeepIoT presents a unied approach that com-presses all commonly used deep learning structures for sensingapplications, including fully-connected, convolutional, and recur-rent neural networks, as well as their combinations. Second, unlikesolutions that either sparsify weight matrices or assume linear struc-ture within weight matrices, DeepIoT compresses neural networkstructures into smaller dense matrices by nding the minimumnumber of non-redundant hidden elements, such as lters and di-mensions required by each layer, while keeping the performanceof sensing applications the same. Importantly, it does so using anapproach that obtains a global view of parameter redundancies,which is shown to produce superior compression. e compressedmodel generated by DeepIoT can directly use existing deep learn-ing libraries that run on embedded and mobile systems withoutfurther modications. We conduct experiments with ve dierentsensing-related tasks on Intel Edison devices. DeepIoT outperformsall compared baseline algorithms with respect to execution timeand energy consumption by a signicant margin. It reduces thesize of deep neural networks by 90% to 98.9%. It is thus able toshorten execution time by 71.4% to 94.5%, and decrease energyconsumption by 72.2% to 95.7%. ese improvements are achievedwithout loss of accuracy. e results underscore the potential ofDeepIoT for advancing the exploitation of deep neural networks onresource-constrained embedded devices.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prot or commercial advantage and that copies bear this notice and the full citationon the rst page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permied. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specic permission and/or afee. Request permissions from firstname.lastname@example.org.SenSys 17, Del, Netherlands 2017 ACM. 978-1-4503-5459-2/17/11. . .$15.00DOI: 10.1145/3131672.3131675
ACM Reference format:Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, and Tarek Abdelzaher.2017. DeepIoT: Compressing Deep Neural Network Structures for SensingSystems with a Compressor-Critic Framework. In Proceedings of SenSys 17,Del, Netherlands, November 68, 2017, 14 pages.DOI: 10.1145/3131672.3131675
1 INTRODUCTIONis paper is motivated by the prospect of enabling a smarter
and more user-friendly category of every-day physical objects ca-pable of performing complex sensing and recognition tasks, such asthose needed for understanding human context and enabling morenatural interactions with users in emerging Internet of ings (IoT)applications.
Present-day sensing applications cover a broad range of areasincluding human interactions [27, 59], context sensing [6, 34, 39,46, 54], crowd sensing [55, 58], object detection and tracking [7,29, 42, 53]. e recent commercial interest in IoT technologiespromises a proliferation of smart objects in human spaces at a muchbroader scale. Such objects will ideally have independent means ofinteracting with their surroundings to perform complex detectionand recognition tasks, such as recognizing users, interpreing voicecommands, and understanding human context. e paper exploresthe feasibility of implementing such functions using deep neuralnetworks on computationally-constrained devices, such as Intelssuggested IoT platform: the Edison board1.
e use of deep neural networks in sensing applications has re-cently gained popularity. Specic neural network models have beendesigned to fuse multiple sensory modalities and extract temporalrelationships for sensing applications. ese models have shownsignicant improvements on audio sensing , tracking and local-ization [8, 38, 52, 56], human activity recognition [37, 56], and useridentication [31, 56].
Training the neural network can occur on a computationallycapable node and, as such, is not of concern in this paper. e keyimpediment to deploying deep-learning-based sensing applicationslies in the high memory consumption, execution time, and energydemand associated with storing and using the trained network onthe target device. is leads to increased interest in compressingneural networks to enable exploitation of deep learning on low-endembedded devices.
We propose DeepIoT that compresses commonly used deep neu-ral network structures for sensing applications through deciding
SenSys 17, November 68, 2017, Del, Netherlands Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, and Tarek Abdelzaher
the minimum number of elements in each layer. Previous illumi-nating studies on neural network compression sparsify large denseparameter matrices into large sparse matrices [4, 22, 24]. In contrast,DeepIoT minimizes the number of elements in each layer, whichresults in converting parameters into a set of small dense matrices.A small dense matrix does not require additional storage for elementindices and is eciently optimized for processing . DeepIoTgreatly reduces the eort of designing ecient neural structures forsensing applications by deciding the number of elements in eachlayer in a manner informed by the topology of the neural network.
DeepIoT borrows the idea of dropping hidden elementsfrom a widely-used deep learning regularization method calleddropout . e dropout operation gives each hidden elementa dropout probability. During the dropout process, hidden ele-ments can be pruned according to their dropout probabilities. ena thinned network structure can be generated. However, thesedropout probabilities are usually set to a pre-dened value, suchas 0.5. Such pre-dened values are not the optimal probabilities,thereby resulting in a less ecient exploration of the solution space.If we can obtain the optimal dropout probability for each hiddenelement, it becomes possible for us to generate the optimal slim net-work structure that preserves the accuracy of sensing applicationswhile maximally reducing the resource consumption of sensing sys-tems. An important purpose of DeepIoT is thus to nd the optimaldropout probability for each hidden element in the neural network.
Notice that, dropout can be easily applied to all commonly usedneural network structures. In fully-connected neural networks,neurons are dropped in each layer ; in convolutional neuralnetworks, lters are dropped in each layer ; and in recurrentneural networks, dimensions are reduced in each layer . ismeans that DeepIoT can be applied to all commonly-used neuralnetwork structures and their combinations.
To obtain the optimal dropout probabilities for the neural net-work, DeepIoT exploits the network parameters themselves. Fromthe perspective of model compression, a hidden element that isconnected to redundant model parameters should have a higherprobability to be dropped. A contribution of DeepIoT lies in ex-ploiting a novel compressor neural network to solve this problem.It takes model parameters of each layer as input, learns parameterredundancies, and generates the dropout probabilities accordingly.Since there are interconnections of parameters among dierentlayers, we design the compressor neural network to be a recurrentneural network that can globally share the redundancy informationand generate dropout probabilities layer by layer.
e compressor neural network is optimized jointly with the orig-inal neural network to be compressed through a compressor-criticframework that tries to minimize the loss function of the originalsensing applicaiton. e compressor-critic framework emulates theidea of the well-known actor-critic algorithm from reinforcementlearning , optimizing two networks in an iterative manner.
We evaluate the DeepIoT framework on the Intel Edison comput-ing platform , which Intel markets as an enabler platform for thecomputing elements of embedded things in IoT systems. We con-duct two sets of experiments. e rst set consists of three tasks thatenable embedded systems to interact with humans with basic modal-ities, including handwrien text, vision, and speech, demonstratingsuperior accuracy of our produced neural networks, compared to
others of similar size. e second set provides two examples ofapplying compressed neural networks to solving human-centriccontext sensing tasks; namely, human activity recognition and useridentication, in a resource-constrained scenario.
We compare DeepIoT with other state-of-the-art magnitude-based  and sparse-coding-based  neural network compressionmethods. e resource consumption of resulting models on the IntelEdison module and the nal performance of sensing applications areestimated for all compressed models. In all experiments, DeepIoTis shown to outperform the other algorithms by a large margin interms of compression ratio, memory consumption, execution time,and energy consumption. In these experiments, when comparedwith the non-compressed neural networks, DeepIoT is able to re-duce the model size by 90% to 98.9%, shorten the running time by71.4% to 94.5%, and decrease the energy consumption by 72.2% to95.7%. When compared with the state-out-the-art baseline algo-rithm, DeepIoT is able to reduce the model size by 11.6% to 83.2%,shorten the running time by 60.9% to 87.9%, and decrease the energyconsumption by 64.1% to 88.7%. Importantly, these improvementsare achieved without loss of accuracy. Experiments demonstratethe promise of DeepIoT in enabling resource-constrained embeddeddevices to benet from advances in deep learning.
e rest of this paper is organized as follows. Section 2 introducesrelated work on optimizating sensing applications for resource-constrained devices. We describe the technical details of DeepIoTin Section 3. We describe system implementation in Section 4. eevaluation is presented in Section 5. Finally, we discuss the resultsin Section 6 and conclude in Section 7.
A key direction in embedded sensing literaure is to enable run-ning progressively more interesting applications under the morepronounced resource constraints of embedded and mobile devices.Brouwers et al. reduced the energy consumption of Wi-Fi basedlocalization with an incremental scanning strategy . Hester et al.proposed an ultra-low-power hardware architecture and a compan-ion soware framework for energy-ecient sensing system .Ferrari et al. and Schuss et al. focused on low-power wirelesscommunication protocols [13, 41]. Wang et al. enabled energy e-cient reliable broadcast by considering poorly correlated links .Saifullah et al. designed a scalable and energy-ecient wirelesssensor network (WSN) over white spaces . Alsheikh et al. dis-cussed about data compression in WSN with machine learningtechniques .
Recent studies focused on compressing deep neural networks forembedded and mobile devices. Han et al. proposed a magnitude-based compression method with ne-tuning, which illustratedpromising compression results . is method removes weightconnections with low magnitude iteratively; however, it requiresadditional implementation of sparse matrix with more resource con-sumption. In addition, the aggressive pruning method increases thepotential risk of irretrievable network damage. Guo et al. proposeda compression algorithm with connection splicing, which providedthe chance of rehabilitation with a certain threshold . How-ever, the algorithm still focuses on weight level instead of structurelevel. Other than the magnitude-based method, another series of
Compressing Deep Neural Network Structures for Sensing Systems SenSys 17, November 68, 2017, Del, Netherlands
Compressor Neural Network
Original Neural Network
Figure 1: Overall DeepIoT system framework. Orange boxesrepresent dropout operations. Green boxes represent param-eters of the original neural network.works focused on the factorization-based method that reduced theneural network complexity by exploiting low-rank structures inparameters. Denton et al. exploited various matrix factorizationmethods with ne-tunning to approximate the convolutional oper-ations in order to reduce the neural network execution time .Lane et al. applied sparse coding based and matrix factorizationbased method to reduce complexity of fully-connected layer andconvolutional layer respectively . However, factorization-basedmethods usually obtain lower compression ratio compared withmagnitude-based methods, and the low-rank assumption may hurtthe nal network performance. Wang et al. applied the informa-tion of frequency domain for model compression . However,additional implementation is required to speed-up the frequency-domain representations, and the method is not suitable for modernCNNs with small convolution lter sizes. Hinton et al. proposed ateacher-student framework that distilled the knowledge in an en-semble of models into a single model . However, the frameworkfocused more on compressing model ensemble into a single modelinstead of structure compression.
Our paper is partly inspired by deep reinforcement learning.With the aid of deep neural networks, reinforcement leaning hasachieved great success on Atari games , Go chess , andmultichannel access .
To the best of our knowledge, DeepIoT is the rst framework forneural network structure compressing based on dropout operationsand reducing parameter redundancies, where dropout operationsprovide DeepIoT the chance of rehabilitation with a certain probabil-ity. DeepIoT generates a more concise network structure for trans-planting large-scale neural networks onto resource-constrainedembedded devices.
3 SYSTEM FRAMEWORKWe introduce DeepIoT, a neural network structure compression
framework for sensing applications. Without loss of generality,before introducing the technical details, we rs...