Machine Learning empowered Occupancy Sensing for Smart

Machine Learning empowered Occupancy Sensing for Smart Buildings

Han Zou 1 Hari Prasanna Das 1 Jianfei Yang 2 Yuxun Zhou 1 Costas Spanos 1

AbstractOver half of the global electricity consumption isattributed to buildings, which are often operatedpoorly from an energy perspective. Significant im-provements in energy efficiency can be achievedvia intelligent building control techniques. To real-ize such advanced control schemes, accurate androbust occupancy information is highly valuable.In this work, we present a cutting-edge WiFi sens-ing platform and state-of-the-art machine learningmethods to address longstanding occupancy sens-ing challenges in smart buildings. Our system-atic solution provides comprehensive fine-grainedoccupancy information in a non-intrusive andprivacy-preserving manner, which facilitates eco-friendly and sustainable buildings.

1. IntroductionEnergy consumption of buildings, both residential and com-mercial, account for more than 50% of global electricityand 40% CO2 emissions worldwide (Allouhi et al., 2015;Jia et al., 2018). In efforts to improve energy efficiency inbuildings, researchers and industry leaders have attemptedto implement various control and automation approachesalongside techniques like incentive design and price adjust-ment to more effectively regulate the energy usage. Lighting,heating, ventilation and air-conditioning (L-HVAC) systemsare the most energy consuming components in a building,which contribute over 70% of the total building energy con-sumption (Jin et al., 2014; Zou et al., 2017a; Weekly et al.,2018). Prior studies have shown that huge amounts of en-ergy is wasted from L-HVAC systems in unoccupied spacessince most of Building Management Systems (BMSs) oper-ate based on static schedules (Yang et al., 2016; Pan et al.,2015). Occupancy based control algorithms for L-HVAChave proven effective in saving considerable amount ofenergy (Zou et al., 2018a). Thus, accurate and robust occu-pancy and occupant activity sensing in buildings serves as

1University of California, Berkeley, USA 2Nanyang Techno-logical University, Singapore. Correspondence to: Han Zou <[email protected]>.

the precursor for intelligent control and energy efficiency.

With the pervasive availability of WiFi infrastructure in bothresidential and commercial buildings, and nearly every mo-bile device (MD) is embedded with a WiFi module, WiFihas been acknowledged as the most promising modality forindoor context-aware services and location-based services(Lymberopoulos et al., 2015; Lu et al., 2016; Jia et al., 2016;Zou et al., 2017c; Yang et al., 2018a). In this work, weintroduce WiFi enabled occupancy sensing platform, andpropose Machine Learning (ML) algorithms, including ad-versarial learning, deep learning and transfer learning, toaddress longstanding challenging problems in occupancysensing, such as extensive human intervention, environmen-tal and temporal dynamics.

2. WiFi-based Occupant Positioning SystemWiFi-based Occupant Positioning System (OPSs) usuallyadopt fingerprinting method as the localization engine asit can capture signal variations in complex indoor environ-ments (Zou et al., 2016). It comprises of 2 steps: 1) Offlinesite survey phase: Received Signal Strength (RSS) measure-ments from multiple WiF access points (APs) at predefinedcalibration points (CPs) and their physical coordinates arecollected and formed a fingerprint database (a.k.a radiomap); 2) Online testing phase: the location of a MD isestimated by comparing the real-time RSS readings to thefingerprints stored in the database. Two major bottleneckshinder WiFi-based OPS for pervasive implementation: 1)the offline site survey process is extremely labor-intensiveand time-consuming; 2) the manual calibrated radio mapis vulnerable to temporal and spatial dynamics. Thus, anautomatic scheme capable of constructing and updating theradio map for adaptive and robust localization is desired.

2.1. Adversarial Learning for Radio Map Adaptation

To overcome these bottlenecks, we propose WiGAN, an au-tomatic fine-grained radio map construction and adaptationscheme that is empowered by Gaussian Process Regression(GPR) and Generative Adversarial Network (GAN) (Good-fellow et al., 2014). Indoor space can be classified into2 categories based on the accessibility: 1) free space (e.g.corridors, open space) where occupants can move freely;2) constrained space where areas are blocked by furniture


Real RSS data at CPscollected by mobilerobot in free space

GaussianProcess

RegressionGeneratorRSSfgp

DiscriminatorReal

Fake

AdversarialLoss

RSSfreal

RSSfgan Coordinates

of CPs in freespace

GaussianProcess

RegressionGeneratorRSSsgp RSSsgan Coordinates of VPs in

constrained space

Synthesize radio map in constrained space

noise

noise

Adversarial learning with real radio map obtained in free space

Figure 1. An overview of WiGAN.

Figure 2. (a) Constructed spatial map via LiDAR SLAM with APlocations; (b) WiGAN generated radio map of AP8 .

(e.g. table in cubicles and conference rooms) or personaloffice that requires special authentication. In free space,we develop a mobile robotic platform to construct and up-date the spatial map (via LiDAR SLAM) and radio mapsimultaneously and automatically.

In constrained space, we propose WiGAN to synthesizerealistic RSS via GPR conditioned GAN. The overview ofWiGAN is presented in Fig. 1. Its functioning includes 4steps: Step 1: a GPR model GP is constructed using thereal RSS values collected by the robot in the free space, aswell as their 2D coordinates (Sf , lf ) to capture the RSSvariations on a rough level; Step 2: coarse RSS estima-tions S

f

GP from GP are adopted as inpurt for the genera-tor G of GAN instead of random noise. As a result, theprobability space on the generator gets further refined, tobetter describe the real RSS distribution in the latent fea-ture space. The input for the discriminator D are randomlysampled batch of real RSS data Sf ∼ Pr(S

f ) and synthe-

sized RSS data SfGP ∼ Pn(S

f

GP) in the free space. Sim-ilar to GAN, the parameters of G and D are optimized inmin-max fashion as minG maxD ESf∼Pr(Sf )[logD(Sf )]+

ESfGP∼Pn(S

fGP)

[log(1−D(G(Sf

GP)))]. Following steps de-tail the procedure to generate RSS values in constrainedspace. Step 3: we leverage the GPR model to generatecoarse estimations at some virtual points (VPs) S

f

GP ; Step4: S

f

GP are used as input for the generator. The fine-grainedRSS estimations at VPs in the constrained space are calcu-lated via the generator G trained in Step 2 as: S

s= G(S

f

GP).In this manner, a fine-grained radio map that covers both

free and constrained space is automatically generated andcontinuously updated, substantially facilitating the large-scale implementation of WiFi OPS.

The experimental results are presented in Fig. 2. WiGANcaptures the irregular RSS distributions in complex indoorspaces and outperforms pure GPR by 44.9%. By leveragingthe radio map synthesized via WiGAN, 1.96 m localiza-tion accuracy is achieved while tremendously reducing theoverhead of time and labor for manual site survey process.By leveraging the occupancy location information providedby our WiFi OPS, our occupancy-driven lighting controlachieves 93% energy savings compared to conventionallighting control scheme. Moreover, WiGAN can be ex-tended for other RF signal’s radio map construction andadaptation, e.g. GPS and LTE. This substantially broad-ens the application of GAN framework to an entirely newdomain of Internet of Things (IoT).

3. Device-free Human Activity RecognitionWith the booming development of IoT, billions of WiFienabled IoT devices, such as thermostats, smart speaker,switch and TV, are en-route to being widely deployed inindoor environments. The WiFi connection among them pro-vides a rich web of reflected rays that spread every corner.Although these WiFi signals are designed for communi-cation and data transmission, they have great potential toprovide a unique opportunity to sense nearby human activi-ties, requiring no privacy intrusive cameras or inconvenientwearables. We explore Channel State Information (CSI)from WiFi physical layer, which reveals detailed informa-tion about how human movement interferes the WiFi signalpropagation from one node (transmitter) to another node(receiver) (Zou et al., 2018b). We develop an OpenWrtfirmware that can run on commercial WiFi routers for CSIdata acquisition, creating potential for large-scale real-lifeapplications (Zou et al., 2017b).

3.1. Deep Learning for Human Activity Recognition

Conventional feature extraction algorithms require exten-sive human intervention and expert knowledge. To addressthis issue, we treat CSI time-series data from multiple sub-carriers as the source of ‘video monitoring ’for occupancysensing (Yang et al., 2018b). As shown in Fig. 4, differ-ent human activities generate distinct perturbations on CSIreadings. We divide these CSI time-series data into smallchunks and the data in each window forms a CSI frame.These CSI frames are served as the input dataset for ourdeep learning engine. Fig. 4 illustrates the network architec-ture of our proposed deep learning method. We first sanitizethe inherent noise in each CSI frame and learn a sparse rep-resentation of it using an autoencoder (AE) module. Afterthat, we use a convolutional neural network (CNN) module


Figure 3. Proposed Adversarial Domain Adaptation.

Figure 4. WiFi CSI frames depicting different activities and theDNN architecture for device-free human activity recognition.

to extract the most discriminative local features from theoutput of AE. Since the sequence of CSI frames can beviewed as consecutive video, the temporal dependenciesamong them are vital properties for accurate human activ-ity recognition. Therefore, we use LSTM to capture them.Since all the parameters in the DNN are automatically fine-tuned in an end-to-end fashion, excessive expert knowledgeis not needed for feature engineering, making it much moreefficient and extendable. Real-world experiments were con-ducted which demonstrate that the proposed method canidentify a number of human activities (e.g. sit, stand, walk,run) with 97.6% recognition accuracy by leveraging onlytwo commodity WiFi routers (Zou et al., 2018c).

3.2. Domain Adaptation for Robust Sensing

Another challenging task is to improve the adaptabilityof the sensing system (i.e. to make it operate in an en-tirely new environment with minimal re-configuration andre-calibration). To address this, we propose a novel adver-sarial domain adaptation (ADA) scheme (Zou et al., 2019).Fig. 3 illustrates its training procedure, which consists of4 steps. Step 1: Train a source encoder Ms and a sourceclassifier Cs by optimizing the loss (LCs

) to construct agood baseline of the feature space and the classifier. Step2: Train a target encoder Mt while giving freedom to thesource encoder Ms to fine-tune via adversarial learningby optimizing the discriminator loss (LD), the source en-

Figure 5. Confusion matrices for gesture recognition via (a) Sourceonly network (b) Proposed ADA method.

coder loss (LMs) and the target encoder loss (LMt

). Inthis manner, both unlabeled target data and labeled sourcedata are embedded to a domain-invariant feature space de-fined by both domains, in a way that a domain discriminatorcannot distinguish the domain labels of them. Step 3: Ashared classifier Csh is constructed with the labeled sourcedomain data by optimizing loss (LCsh

). Step 4: Employthe trained target encoder Mt to map test samples fromthe target domain into the domain-invariant feature spaceand use the shared classifier Csh to identify the categoryof each testing sample in the target domain. The trainingof the proposed ADA scheme is thusly equivalent to solv-ing minCsh

minMt,MsmaxD minMs,Cs

LCs+LD+LMs

+LMt

+ LCsh. Experiments were conducted in 2 conference

rooms with different sizes to validate the propose method forspatial adaptation of WiFi enabled device-free gesture recog-nition. Volunteers performed 6 common gestures, movinga hand right and left, up and down, push and pull betweenthe two IoT devices. Fig. 5 compares the confusion matri-ces of the gesture classification accuracies with source onlynetwork and with our proposed ADA method.

4. ConclusionIn this work, we introduced our WiFi sensing platform andour proposed ML algorithms to address longstanding occu-pancy sensing challenges in smart building. Our systematicsolution performs occupancy sensing and behavior infer-ence in a non-intrusive and privacy-preserving manner. Thisin turn enables occupancy adaptive L-HVAC building con-trol achieving energy efficiency and reduce CO2 emissionswhile maintaining the occupant comfort and productivity.


ReferencesAllouhi, A., El Fouih, Y., Kousksou, T., Jamil, A., Zeraouli,

Y., and Mourad, Y. Energy consumption and efficiencyin buildings: current status and future trends. Journal ofCleaner Production, 109:118–130, 2015.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,Warde-Farley, D., Ozair, S., Courville, A., and Bengio,Y. Generative adversarial nets. In Advances in neuralinformation processing systems, pp. 2672–2680, 2014.

Jia, R., Jin, M., Zou, H., Yesilata, Y., Xie, L., and Spanos, C.Mapsentinel: Can the knowledge of space use improveindoor tracking further? Sensors, 16(4):472, 2016.

Jia, R., Jin, B., Jin, M., Zhou, Y., Konstantakopoulos, I. C.,Zou, H., Kim, J., Li, D., Gu, W., Arghandeh, R., et al. De-sign automation for smart building systems. Proceedingsof the IEEE, 106(9):1680–1699, 2018.

Jin, M., Zou, H., Weekly, K., Jia, R., Bayen, A. M., andSpanos, C. J. Environmental sensing by wearable devicefor indoor activity and location estimation. In IndustrialElectronics Society, IECON 2014-40th Annual Confer-ence of the IEEE, pp. 5369–5375. IEEE, 2014.

Lu, X., Wen, H., Zou, H., Jiang, H., Xie, L., and Trigoni,N. Robust occupancy inference with commodity wifi.In 2016 IEEE 12th International Conference on Wirelessand Mobile Computing, Networking and Communications(WiMob), pp. 1–8. IEEE, 2016.

Lymberopoulos, D., Liu, J., Yang, X., Choudhury, R. R.,Handziski, V., and Sen, S. A realistic evaluation and com-parison of indoor location technologies: Experiences andlessons learned. In Proceedings of the 14th internationalconference on information processing in sensor networks,pp. 178–189. ACM, 2015.

Pan, J., Jain, R., Paul, S., Vu, T., Saifullah, A., and Sha,M. An internet of things framework for smart energy inbuildings: designs, prototype, and experiments. IEEEInternet of Things Journal, 2(6):527–537, 2015.

Weekly, K., Jin, M., Zou, H., Hsu, C., Soyza, C., Bayen,A., and Spanos, C. Building-in-briefcase: A rapidly-deployable environmental sensor suite for the smart build-ing. Sensors (Basel, Switzerland), 18(5), 2018.

Yang, J., Santamouris, M., and Lee, S. E. Review of occu-pancy sensing systems and occupancy modeling method-ologies for the application in institutional buildings. En-ergy and Buildings, 121:344–349, 2016.

Yang, J., Zou, H., Jiang, H., and Xie, L. Carefi: Sedentarybehavior monitoring system via commodity wifi infras-tructures. IEEE Transactions on Vehicular Technology,2018a.

Yang, J., Zou, H., Jiang, H., and Xie, L. Device-free occu-pant activity sensing using wifi-enabled iot devices forsmart homes. IEEE Internet of Things Journal, 5(5):3991–4002, 2018b.

Zou, H., Jin, M., Jiang, H., Xie, L., and Spanos, C.Winips: Wifi-based non-intrusive ips for online radio mapconstruction. In 2016 IEEE Conference on ComputerCommunications Workshops (INFOCOM WKSHPS), pp.1081–1082. IEEE, 2016.

Zou, H., Jiang, H., Yang, J., Xie, L., and Spanos, C. Non-intrusive occupancy sensing in commercial buildings. En-ergy and Buildings, 154:633–643, 2017a.

Zou, H., Zhou, Y., Yang, J., Gu, W., Xie, L., and Spanos, C.Freecount: Device-free crowd counting with commoditywifi. In Global Communications Conference (GLOBE-COM), 2017 IEEE. IEEE, 2017b.

Zou, H., Zhou, Y., Yang, J., Gu, W., Xie, L., and Spanos,C. Freedetector: Device-free occupancy detection withcommodity wifi. In Sensing, Communication and Net-working (SECON Workshops), 2017 IEEE InternationalConference on, pp. 1–5. IEEE, 2017c.

Zou, H., Zhou, Y., Jiang, H., Chien, S.-C., Xie, L., andSpanos, C. J. Winlight: A wifi-based occupancy-drivenlighting control system for smart building. Energy andBuildings, 158:924–938, 2018a.

Zou, H., Zhou, Y., Yang, J., Gu, W., Xie, L., and Spanos,C. Wifi-based human identification via convex tensorshapelet learning. In AAAI Conference on Artificial Intel-ligence (AAAI), pp. 1711–1718, 2018b.

Zou, H., Zhou, Y., Yang, J., and Spanos, C. J. Towardsoccupant activity driven smart buildings via wifi-enablediot devices and deep learning. Energy and Buildings, 177:12–22, 2018c.

Zou, H., Zhou, Y., Yang, J., Liu, H., Das, H., and Spanos,C. Consensus adversarial domain adaptation. In AAAIConference on Artificial Intelligence, 2019.

Documents

Machine Learning empowered Occupancy Sensing for Smart