Web-based Platform for Evaluation of RF-basedIndoor Localization Algorithms

Filip Lemic∗, Vlado Handziski∗, Niklas Wirstrom†, Tom Van Haute‡, Eli De Poorter‡,Thiemo Voigt†, Adam Wolisz∗

∗Telecommunication Networks Group (TKN), Technical University of Berlin (TUB)†SICS Swedish ICT

‡Department of Information Technology (INTEC), Ghent University - iMinds{lemic,handziski,wolisz}@tkn.tu-berlin.de

Abstract—The experimental efforts for optimizing the perfor-mance of RF-based indoor localization algorithms for specificenvironments and scenarios is time consuming and costly. Inthis work, we address this problem by providing a publiclyaccessible platform for streamlined experimental evaluation ofRF-based indoor localization algorithms, without the need of aphysical testbed infrastructure. We also offer an extensive set ofraw measurements that can be used as input data for indoorlocalization algorithms. The datasets are collected in multipletestbed environments, with various densities of measurementpoints, using different measuring devices and in various scenarioswith controlled RF interference. The platform encompasses twocore services: one focused on storage and management of rawdata, and one focused on automated calculation of metrics forperformance characterization of localization algorithms. Toolsfor visualization of the raw data, as well as software libraries forconvenient access to the platform from MATLAB and Python, arealso offered. By contrasting its fidelity and usability with respectto remote experiments on dedicated physical testbed infrastruc-ture, we show that the virtual platform produces comparativeperformance results while offering significant reduction in thecomplexity, time and labor overheads.

I. INTRODUCTION

The high demand for reliable location information in indoorspaces like buildings, tunnels, etc., where global satellite-based navigation systems are typically not usable, has ledto an intense focus on alternative approaches. Many of theproposed solutions leverage intrinsic characteristics of theindoor environment in order to provide the users with estimatesof their location. Using Radio Frequency (RF) signal featuresis one of the most popular approaches, as illustrated by thenumerous works done in this area, with [1]–[4] being justa few of many examples. Although rapid progress has beenachieved in the baseline performance of these algorithms, theirpractical application still requires extensive experimentationefforts for tuning the performance at the specific conditionsof the deployment scenarios and environments, which areassociated with high time and cost overheads.

In this work we aim at reducing these hindrances byproviding: (i) detailed public datasets of raw RF data that canbe used as input to indoor localization algorithms, collectedin a wide range of conditions (different types of devices,different types of interference, different environments) and (ii)a web-based platform that streamlines the utilization of these

datasets for the purpose of performance evaluation of indoorlocalization algorithms.

The provided platform1 supports evaluation of RF-basedindoor localization algorithms without the need for experi-mentation on dedicated physical experimental infrastructure.As shown in Figure 1, the platform encompasses two coreservices. The first one is a service for storage and managementof raw measurement data that can be used as an input tothe localization algorithms that the experimenters want toevaluate. The second service offers automated calculation of astandardized set of indoor localization performance metricslike localization accuracy, response time and power con-sumption. The platform includes a tool for visualization ofthe raw data that supports experimenters in the process ofreviewing the offered datasets. We also provide a set ofSoftware Development Kits (SDKs) for Python and MATLAB,providing functions for streamlined access to raw data and forcalculation of performance metrics, which further simplifiesthe evaluation process.

Raw$data$storage$service$

Metrics$calcula2on$service$

API$ API$

Algorithm$

Wrapper$

Wrapper$

get_measurements()$

raw$data$input$

calculate_metrics()$

metrics$input$Raw$data$

visualiza2on$tool$

Experimenter$

“browsing”$of$the$raw$data$ fineAtuning$and$

parameteriza2on$of$the$algorithm$

using$the$evalua2on$results$

Figure 1: Main components of the evaluation platform

From the experimenter’s perspective, with the help of theraw data visualization tool, the experimenter is able to visu-alize and “browse” the available datasets. After selecting thedesired raw dataset, the experimenter can use it as input to analgorithm to be evaluated. Based on the results of the perfor-mance evaluation, the experimenter is able to parametrize orfine-tune the algorithm for improving its performance.

1 A prototype of the platform is available at http://ebp.evarilos.eu.

The currently offered datasets are collected on threetestbeds, giving experimenters the possibility to evaluate theiralgorithms in diverse environments. The raw data that canbe used as input to the evaluated algorithms is collectedunder different scenarios, with different densities of evaluationpoints, using different hardware and with different sourcesof controlled RF interference, providing a wide range ofexperimentation possibilities. This data was collected with thehelp of an automated testbed infrastructure described in [5].We envision further additions to the current datasets, both fromour side and from the researchers interested in contributing.

The rest of this paper is structured as follows. In Section IIwe provide an overview of the related work. Section IIIdescribes the design and implementation of the platform, andthe format of the stored raw data. In Section IV we presentthe currently available datasets and outline how some openresearch questions in RF-based indoor localization can beaddressed using the available raw data. Section V comparesthe results and overheads of evaluating an example indoorlocalization algorithm with the help of the virtual platform andits datasets with one using remote experiments on dedicatedphysical testbed infrastructure. Finally, Section VI concludesthe work and gives directions for future improvements.

II. RELATED WORK

Public raw datasets represent an important bedrock of scien-tific research in disciplines ranging from astronomy to studiesof climate change and the humane genome. Dedicated sites,like the Amazon Public Dataset [6], act as central repositoriesof popular and large datasets, streamlining their use in cloud-based frameworks. These datasets can be leveraged as commonbenchmarks for assessing the relative performance of differentsystems. Popular examples are the use of NIST LanguageRecognition datasets in the domain of language recognition [7]or ImageNet datasets in computer vision [8].

In the domain of RF-based indoor localization, in additionto our work, there have been few previous efforts to virtu-alize aspects of the experimental evaluation of localizationalgorithms. One example is the VirTIL testbed [9]. While inthe VirTIL testbed authors focus on the evaluation of range-based indoor localization algorithms, by providing the rangemeasurements at various locations in their testbed, in ourwork we promote and offer unprocessed low-level data suchas Received Signal Strength Indicator (RSSI) and Time ofArrival (ToA) measurements that are used in a wide range ofindoor localization approaches. As a result, the number of RFalgorithms that can be evaluated is extended, and can includescene analysis (fingerprinting), hybrid and proximity basedalgorithms. In addition, we offer data from several testbedswith artificial interference patterns, offering users a possibilityof evaluating their algorithms in different environments andinterference scenarios.

III. PLATFORM OVERVIEW

This section presents the components of our platform forvirtual evaluation of RF-based indoor localization algorithms.

The overview of the platform indicating the relation amongindividual components is given in Figure 1. The platformconsists of a raw data storage service used for storing the rawdata that can be inputed into an indoor localization algorithmto be evaluated. The platform further provides a raw datavisualization tool, that enables experimenters to easily visual-ize collected information stored in the provided measurementcollections. Furthermore, the experimenter is able to fetch anduse a desired raw dataset though a set of software SDKs.Finally, a set of standardized metrics for characterizing theperformance of an algorithm can be obtained through onefunction call to a metrics calculation service.

The raw data storage service stores collected measurementsthat can be used for indoor localization evaluation purposes.A detailed description of the service and its functions isgiven in [9]. The measurements are stored together with thelocations where they are taken, annotated with the locations oftransmitting devices, the metadata describing the environmentand the hardware used for collection of the raw data. Theservice provides a publicly available Application ProgrammingInterface (API) for managing the stored data, where theexperimenter can “search” the stored datasets and select adesired one.

The platform further consists of a set of software SDKsor wrappers developed for Python and MATLAB program-ming languages, due to their popularity in rapid prototypingof algorithms, including those for indoor localization. Theexperimenter can use the wrappers to fetch measurementsthrough a single function call. The experimenter is then ableto input the fetched data to an algorithm to be evaluated. TheSDKs provide interaction with the cloud service for the datastorage as shown in Listing 1. Using the get measurementscommand, the experimenter is able to fetch the data froman experiment or a specific measurement. With the filteringcommand, it is possible to filter the fetched data based ondesired parameters, such as number of measurements, wirelesschannel or transmitting device.

Listing 1: Important Python SDK functionsdef g e t m e a s u r e m e n t s ( d a t a b a s e , expe r ime n t , measurement ) ;def f i l t e r i n g ( measurement , num meas , channe l , s e n d e r ) ;def c a l c u l a t e m e t r i c s ( d a t a ) ;

The output of an algorithm, i.e. estimated locations, and thematching ground-truth coordinates where the measurementswere taken, can be passed to a metrics calculation service usingthe calculate metrics command provided by the wrappers. Bydoing that the experimenter is able to obtain a standardizedset of performance metrics, allowing him to easily evaluatethe performance of an algorithm, using the experimentallycollected data as the input and receiving a set of standardizedmetrics as the output of the procedure [10].

A. Design GoalsMain requirements for the service are simplicity of usage

for the experimenters, extensibility, reliability, fast data flow,remote usage of the service, and language and platformindependence.

a) Usability: We provide a set of services for easyscoping and filtering of the raw data, so that experimenterscan selectively request a record at a specific location co-ordinates and for a given technology, and consequently getthe desired dataset in an efficient way. The advantage of ourapproach, in comparison to plain “downloading” alternative,lies in the fact that experimental raw datasets for evaluationof indoor localization algorithms can be very large. Especiallyfor “universal” datasets that can be used for evaluation ofdifferent localization algorithms, the aim is to collect dataat high spatial sampling densities and using diverse hardwareequipment. But any particular algorithm would likely use onlya small subset of this data in a given evaluation campaign. Theapproach of disseminating the whole raw data sets as simple“downloadable” archives is thus very inefficient and leaves tothe experimenter the burden of local filtering. The alternativethat we offer, an online service for accessing and managing ofthis data, is much more convenient for the experimenters andreduces the requirements for local computational resources.

b) Extensibility: One of the main concerns for the de-velopment of the service for management of the raw datais extensibility. We aimed at developing a service whereexperimenters are able to add or remove parameters to thedata records according to their specific needs. Furthermore,the overhead of storing these customized data records into thedatabase should be minimal and should not require complexschema adaptations.

c) Fast and reliable remote access: Another core designgoal was communication efficiency and scalability. We aimedat a cloud-based platform supporting public remote access fora large number of users, without the need of downloading andrunning the whole platform on a local machine.

d) Language and platform independence: Finally, thedesign goal is to make the platform available for various pro-gramming languages and operating systems. In other words,in order to make the platform usable for a large number ofexperimenters, with expertise in different software platformsand languages, we aimed in developing it so that it is languageand platform independent.

B. Platform Implementation

As described previously, the main components of the plat-form are the raw data storage service, service for calculationof the performance metrics, set of SDKs for simple eval-uation with Python and MATLAB programming languagesand a visualization tool for convenient data browsing. Whilethe other components provide straightforward functions andsimplifications for the experimenters, due to its central role,the implementation of the raw data storage service is worthexplaining in more detail. It is implemented as a web servicedeveloped in Python 2.7 using the Flask module. The Flaskmodule provides a simple way of creating RESTful webservices. We leveraged this to develop multiple functions forsupporting raw data storage and management. Raw data isstored in a MongoDB database, an open-source documentdatabase and the leading Not only SQL (NoSQL) database

written in C++. It can store JavaScript Object Notation (JSON)based messages using Binary JSON (BSON) format of data.Our data messages are defined as Protocol Buffer structures, away of encoding structured data using an efficient and exten-sible binary format. The service we developed, together withthe MongoDB database, is running on a EC2 instance in AWS.Amazon Elastic Compute Cloud (Amazon EC2) is a webservice that provides scalability and platform independence ofour services. The experimenter is able to communicate withthe services through properly defined HTTP requests.

The design goal of extensibility was achieved using ProtocolBuffer (version 2) for defining message types and MongoDBfor storing those messages. In Protocol Buffers, each datastructure that needs to be encoded is encapsulated in the formof a message. The specification of the internal structure of themessages is done in special protocol files that have the .protoextension. The specification is performed using a simple, butpowerful, domain specific data specification language thatallows easy description of the different message fields, theirtypes, optionality, etc. Using the Protocol Buffer compilerprotoc the .proto specification files can be compiled to generatedata access classes in number of languages like C++, Javaand Python. These classes provide simple accessors for eachfield (like query() and set query()) as well as methods toserialize/parse the whole structure to/from raw bytes. NoSQLdatabases employ less constrained consistency models thantraditional relational databases. By using a NoSQL or schema-less type of database, i.e. MongoDB, the service enables thestorage of any type of defined message, without the need ofchanging the code and/or the database itself.

RESTful web services enable remote access to the data us-ing simple HTTP requests. Protocol Buffers serialize messagesinto binary streams which support fast communication be-tween the experimenters and the platform. To sum up, RESTfulweb service as a part of service for management of the rawdata provides remote access to the service. High availability issupported by running the service in the Amazon cloud. Finally,the fast data flow from experimenter to database is achieved byusing Protocol Buffer serialization and MongoDB schema-lessdatabase for storing the binary data.

Using the Protocol Buffer compiler message specificationfiles (.proto) can be compiled to generate data access classesin a number of programming languages like C++, Java andPython. Furthermore, due to the fact that communication withthe cloud is done using HTTP requests, it is possible to managedata from different users’ platforms, and also using differentprogramming languages (most of the modern programminglanguages provide libraries enabling HTTP requests). To con-clude, by using Protocol Buffers and a RESTful web serviceit is possible to manage raw data from different platforms andusing different programming languages.

C. Data Structure

NoSQL databases usually provide the following hierarchy.On the top level is a set of databases. Each database consistsof a number of collections (the equivalent to tables in rela-

tional databases). Finally, each collection consists of a set ofmessages which contain data. We follow the same principlein our raw data storage service. The data storage hierarchyof the raw data storage service is given in Figure 2. Thedatabase for storing the raw data from the indoor localizationexperiments consists of a set of collections (called “Experi-ments” in our case), where a collection stores all raw datafrom one round of measurements, i.e. from one measurementsurvey. Each collection consists of messages that are related tophysical locations where the measurements are taken, whereeach location is defined with coordinates (x,y,z) in relationto a predefined zero-point. Besides the database for storingraw data, there is also a database for storing the metadatadescribing the raw data. The explicit split between the dataand metadata is done to reduce the amount of data that has tobe fetched from the platform, i.e. in case where a lot of datashares the same metadata, the metadata has to be fetched onlyonce and applied to all the data. Also, the explicit split enablesreusage of one metadata message for multiple repetitions of thesame measurement survey, which decreases the redundancy inthe stored information. This database contains descriptions ofdifferent experiments or measurement surveys performed forcollecting the raw data.

Raw$data$ Metadata$

Experiment*1* Experiment*M* Experiment*1*…*

Measurement*at*loca4on*1*

Measurement*at*loca4on*N*

Metadata*related*to*experiment*K**

…*

DATABASES*

COLLECTIONS*

MESSAGES*

Experiment*K*

Figure 2: Hierarchy of the service for managing the raw data

A snapshot of the raw data in JSON representation is givenin Listing 2. This sample data consists of the RSSI mea-surements, information about the measurements (timestamp,transmitter ID, run number, etc.) and of the location of thetransmitting and receiving devices. Alternatively, instead ofstoring RSSI values, it is also possible to store ToA, Angle ofArrival (AoA) or Link Quality Indicator (LQI) measurements,or any combination of those measurements.

Listing 2: Raw data format1 { receiver_id: "MacBook Pro",2 run_nr: 13,3 timestamp_utc: 1373126790,4 sender_id: "tplink08",5 sender_bssid: "64:70:02:3e:aa:11",6 rssi: -42,7 channel: "11",8 receiver_location: {9 room_label: "FT226",

10 coordinate_z: 9.53,11 coordinate_y: 1.67,12 coordinate_x: 23.9},13 sender_location: {14 room_label: "FT226",15 coordinate_z: 10.9,16 coordinate_y: 0.7,17 coordinate_x: 31},}

IV. OVERVIEW OF THE AVAILABLE DATASETS

The current datasets consist of IEEE 802.11b/g andIEEE 802.15.4 RSSI measurements, IEEE 802.15.4 ToA andIEEE 802.15.4 LQI measurements, collected using differentdevices at different locations in various environments andscenarios. A device for collecting the measurements has beencarried to each measurement point using an autonomousmobility platform, which is a robotic mobility platform thatcan carry a system for collecting the raw data. It is capable ofself-navigating in indoor office spaces, achieving positioningerrors smaller that 15 cm in average [5], which is comparableto the positioning errors achieved by a person carrying a local-ization device. As has been shown in a large scale evaluationof multiple indoor localization solutions in [11], [12], theaverage localization errors of current state-of-the-art solutionsare mostly higher than 1 m, meaning that the accuracy of theused autonomous mobility platform is sufficient to serve assource of ground-truth information.

In addition to the simplification of the experiment processachieved by the automation, the mobility platform has anadditional benefit in that it removes the influence that aperson’s body can have on a measuring device, i.e. a deviceis entirely static during the measurement collection, there isno shadowing due to the person’s body, and the height of adevice is always the same in respect to the floor. All thesefeatures produce raw measurements that are less influencedby the experimenter, which increases the objectiveness incomparability of multiple algorithms and gives more accurateintrinsic performance results of an algorithm. The disadvan-tage is, however, the fact that removing the influence of theexperimenter gives less realistic real-life conditions, becausein many real-life localization scenarios a localization deviceis carried by a person.

In the following we overview the publicly available datasetsand discuss how they help to solve open questions in RF-basedindoor localization.

a) Diversity of environments: One open research ques-tion in RF-based indoor localization is related to havinga solution that provides reasonably accurate and real-timeperformance in diverse environments [13], [14]. The pro-vided measurements collected in various environments giveexperimenters a possibility to evaluate the stability of theirsolution in diverse conditions, with an aim on solving theaforementioned research challenge. Current datasets have beencollected in three different testbeds (TWIST [15] and w-iLab.t I/II [16]). The footprints of the testbed environments aregiven in Figure 3, with the locations of measurement points forthe densest measurement surveys indicated with red dots. Thefirst two testbeds represent typical office environments. Themeasurements were performed in static environments, i.e. withno people moving, without introducing additional unaccountedsources of RF interference, etc. The third testbed representsa big open space environment, such as a warehouse or anunderground garage. This environment is shielded with nopeople moving and with effectively no RF interference.

(a) TWIST testbed

(b) w-iLab.t I testbed

(c) w-iLab.t II testbed

Figure 3: Footprints of the testbed environments

b) Diversity of densities of measurements points: Themeasurements have also been collected with different densitiesof measurement points, which has various advantages. Firstly,some of the indoor localization algorithms, e.g. fingerprinting-based ones, need a training dataset with typically high density.The most dense dataset can in that case be used as the trainingdataset for the algorithm. Furthermore, it is still unclear howto select the measurement points for the objective evaluationof RF-based indoor localization algorithms [17]. By providing,in terms of realistic possibilities and time constrains, a highlydense set of measurement points, we provide the opportunityof evaluating the influence of the number and locations ofmeasurement points on a large set of algorithms, which couldaddress the mentioned research question. Finally, an open re-search question than can be addressed with the offered datasetsis the trend in spatial distribution of localization errors [12],[18]. For the TWIST testbed environment as an example, thefootprints with different densities of measurements points andwith measurement points indicated with red dots, are given inFigure 4 (measurement points destinies of 22.5 m2, 11 m2 and5 m2 per point, respectively).

c) Diversity of interference scenarios: It is shown thatRF interference can highly influence the performance of RF-

(a) Density 1

(b) Density 2

(c) Density 3

Figure 4: Various densities in the TWIST testbed

based indoor localization algorithms [12], [19]. There is,however, no work on improving the interference robustnessof indoor localization algorithms, and by providing the mea-surements collected in the environment with high interference,we hope to provide users the possibility of improving theinterference robustness of their algorithms. The generatedinterference scenarios range from a scenario in which noartificial interference is generated and the uncontrolled inter-ference is minimized, to scenarios with various types of RFinterference. In the scenarios with interference the interferencewas artificially generated and the interference types vary fromjamming using IEEE 802.11 or IEEE 802.15.4 devices toIEEE 802.11 traffic as the interference source.

d) Diversity of localization devices: While it is knownthat the device diversity is one of the biggest obstacles in hav-ing widely used indoor localization service [20], this problemhas not been fully addressed yet. By providing measurementsfrom various devices we hope to provide possibility of improv-ing the performance of RF-based indoor localization in thisarea. We have collected measurements from two extensivelyused technologies for RF-based indoor localization, namelyIEEE 802.11 and IEEE 802.15.4. The full datasets for differenttechnologies are available for the TWIST testbed environment.

In Figure 5 we show the locations of controllable IEEE802.11 (blue squares) and IEEE 802.15.4 (red points) nodesthat have been used as anchors in TWIST testbed [15]. Inour data collections the measurements from these devicesare annotated with with the location of the transmitters. Incase of the IEEE 802.11 anchor nodes, we have used sixTP LINK 4300 wireless routers, set on WiFi channel 11 andwith a transmission power of 20 dBm (100 mW). Apart fromthese devices, we also stored the data from all other visibleAccess Points (APs) in the environment, but without theirlocations. For the IEEE 802.15.4 technology we have beenusing TelosB nodes deployed in our TWIST testbed as anchornodes. We used various devices as receivers for obtaining theprovided datasets. Namely, for the IEEE 802.11 technologywe used the MacBook Pro and Lenovo ThinkPad laptops,Google Nexus 7 tablet and Nexus S smartphone. For theIEEE 802.15.4 technology we have been using the TelosBand stm32w nodes as measuring devices.

IEEE#802.11#anchors#

IEEE#802.15.4#anchors#

Figure 5: TWIST testbed - transmitting devices

V. EVALUATION OF THE VIRTUAL PLATFORM

In order to evaluate the usability of the collected rawdatasets and the time efforts of using the virtual platform forevaluation of RF-based indoor localization algorithms we haveexecuted a set of experiments described in this section. Weused a simple WiFi beacon packets RSSI-based fingerprintingalgorithm that uses quantiles of beacon packets RSSI valuesand Pompeiu-Hausdorff (PH) distance to calculate similar-ities between training and user-generated fingerprints [14].Further, we evaluated its performance firstly by using theinfrastructure for evaluation of RF-based indoor localizationalgorithms described in [5] and secondly by leveraging the rawdataset collected under similar conditions by using the virtualplatform. The aim of the evaluation is to demonstrate thatthe achieved performance metrics of the evaluated algorithm,i.e. the accuracy and delay (response time), do not practicallydiffer in a case when the proposed platform is used, in com-parison to a case when real testbed infrastructure is used forevaluation. Secondly, while it is clear that, in comparison to us-ing the infrastructure for experimentation, using the proposedplatform provides lower-complexity experimentation for theexperimenter, we aim on showing that there is also a significantreduction in time-efforts needed for experimentation.

Since the impact of the “staleness” of recorded data de-pends on the specific conditions in the environment, in thecomparison of performance of the evaluated fingerprinting

algorithm we have collected a raw dataset and performed anexperimental run using the infrastructure on the same day andin a static environment. The presented results are not intendedto prove that recorded data can replace many runs with realhardware, due to changes in the conditions that can occurwith time. In other words, the same raw datasets collectedin different time snapshots will differ, due to an intrinsicrandomness in wireless environment, and for capturing thisdifference different runs with a real hardware are necessary.On the contrary, the presented results capture realisticallythe specific conditions in a time snapshot and can serve asbasis for fair comparison between the results obtained byleveraging the raw dataset as an input to the algorithm and theresults obtained by performing an experimental run leveragingthe evaluation infrastructure, i.e. requesting the algorithm toprovide an estimate at each evaluation point.

In the experiments using the infrastructure we evaluated theperformance of a given algorithm, which was running on aMacBook Pro laptop, in four interference scenarios in TWISTtestbed (in details described in [12]) and in 20 evaluationpoints for each interference scenario (Figure 4b). In the firstinterference scenario, i.e. “Reference scenario”, no artificialinterference was generated and the influence of uncontrolledinterference was minimized by performing the experiment instatic environment to ensure consistency and minimize theeffects of shadowing and RF interference. In the “Interferencescenario 1” the artificial interference was WiFi traffic on oneWiFi channel. Finally, in the “Interference scenario 2” and “In-terference scenario 3” artificially generated interference wasdistributed jamming on IEEE 802.15.4 channel and jammingon IEEE 802.11 channel, respectively.

Using the platform for virtual evaluation we selected anappropriate dataset in which a MacBook Pro was used as thedevice for collecting the raw data, the raw data was collectedin the same 20 evaluation points, in the same testbed and in thesame four interference scenarios. The functions of the Pythonwrapper were used to fetch the raw data and input it to thealgorithm, and to calculate the performance metrics. The delaymetric in this case is the sum of the time needed to collect themeasurements and the time needed for the processing of theraw data and providing the location estimates.

The summarized results for 20 evaluation points in eachinterference scenario are given in Table I. As indicated in thetable, the results of the evaluation using the virtual platform arecomparable to the results using the infrastructure specificallydesigned for evaluation purposes. Small differences in thelocalization errors and room accuracies are caused by anunavoidable randomness of the wireless environment.

From the time perspective, the evaluation using the virtualplatform takes only a few (tens of) seconds, once the evalua-tion chain (fetch the raw data - input the data to an algorithm -request metrics - get metrics) is set. In comparison, one run ofan experiment with 20 evaluation points using the automatedinfrastructure for experimentation takes around 1 min perpoint, or around 20 min all together. From our experience,we can argue that manual experimentation takes even more

TABLE I: Evaluation results obtained using the infrastructure vs. using the platform for evaluation

Scenario Reference scenario Interference scenario 1 Interference scenario 2 Interference scenario 3Metric Infrastructure Platform Infrastructure Platform Infrastructure Platform Infrastructure PlatformMean error [m] 2.82 2.91 3.84 3.72 5.20 4.91 4.02 4.26Median error [m] 2.61 2.45 3.11 2.87 4.81 4.41 3.56 3.6175 perc. error [m] 3.87 3.74 5.14 4.87 6.46 6.22 5.36 5.4590 perc. error [m] 4.19 4.56 5.87 5.44 9.49 8.81 7.37 7.44Min. error [m] 0.47 0.22 1.26 1.17 0.88 0.56 0.79 1.02Max. error [m] 5.76 6.41 6.70 6.53 11.46 8.31 10.93 11.65Room acc. [%] 70.0 65.0 50.0 55.0 45.0 50.0 45.0 45.0Mean delay [s] 20.08 21.02 19.58 21.05 20.58 20.76 20.02 20.55Median delay [s] 20.05 21.20 19.83 21.09 20.45 20.55 20.05 20.6375 perc. delay [s] 20.11 21.32 19.95 21.78 20.71 20.91 20.11 20.7690 perc. delay [s] 20.25 21.51 20.03 21.95 21.64 22.08 20.16 20.87Min. delay [s] 19.66 20.12 19.71 19.56 19.92 20.03 19.67 20.12Max. delay [s] 21.02 22.23 20.11 22.04 21.94 22.65 20.26 21.04

time, in comparison to using the automated infrastructure,which again favors using the virtual platform for evaluation.If we take into account also the time necessary for settingup a comparable experimental setup (on the order of days orweeks), the benefit of using the proposed virtual platform iseven higher.

VI. CONCLUSION AND FUTURE WORK

This work describes a platform for virtual experimentalevaluation of RF-based indoor localization algorithms in asimple and effective way, without the need of a testbedinfrastructure. We firstly provide an overview of the designideas that guided the development of the platform and itshigh-level architecture, together with the selected implemen-tation frameworks. We further describe the currently availabledatasets that can be used for evaluation of RF-based indoorlocalization algorithms, together with the experimentation pos-sibilities that they enable. Finally, we demonstrate the usageof the virtual platform and assess the overhead and obtainedperformance evaluation metrics by performing the same setof experiments using the platform and a dedicated physicaltestbed infrastructure. The advantage of having the virtualplatform is visible in its simplicity for the experimenters andthe reduction of the time needed for experimentation. Whenusing the platform the experimenter is provided with easy touse, low-complexity access to the infrastructure, which doesnot have the burden for handling the complex componentsof the infrastructure for evaluation. In comparison to manualevaluation, when using the platform the experimenter doesnot have to carry a measuring device and does not haveto care about finding the exact coordinates for the ground-truth locations, which are both labor and time consuming.By using the platform it is possible to leverage exactly thesame dataset for comparing different algorithms or differentparameterizations of an algorithm, which removes the pre-viously described randomness in the wireless environmentand increases the objectivity of comparison between differentalgorithms. Our future work will be oriented on increasing theaccuracy of ground-truth locations and on collecting additionalmeasurements, with the focus on device diversity, more densemeasurements and AoA measurements.

ACKNOWLEDGMENTS

This work has been partially funded by the EuropeanCommission (FP7-ICT-FIRE) within the project EVARILOS(grant No. 317989). The author Filip Lemic was partiallysupported by DAAD (German Academic Exchange Service).

REFERENCES

[1] P. Bahl and V. N. Padmanabhan, “Radar: an in-building rf-based userlocation and tracking system,” in INFOCOM’00, IEEE, 2000.

[2] N. B. Priyantha et al., “The cricket location-support system,” inMOBICOM’00, ACM, 2000, pp. 32–43.

[3] T. He et al., “Range-free localization schemes for large scale sensornetworks,” in MOBICOM’03, ACM, 2003, pp. 81–95.

[4] D. Niculescu and B. Nath, “Ad hoc positioning system (aps) usingaoa,” in INFOCOM 2003, IEEE, vol. 3, 2003, pp. 1734–1743.

[5] F. Lemic et al., “Infrastructure for Benchmarking RF-based IndoorLocalization under Controlled Interference,” in UPINLBS’14, 2014.

[6] Amazon, Amazon Public Datasets. [Online]. Available: https://aws.amazon.com/datasets.

[7] A. F. Martin and M. A. Przybocki, “NIST 2003 Language RecognitionEvaluation,” in INTERSPEECH, 2003.

[8] O. Russakovsky et al., ImageNet Large Scale Visual RecognitionChallenge, 2014.

[9] F. Lemic and V. Handziski, “Data Management Services for Evalua-tion of RF-based Indoor Localization,” Telecommunication NetworksGroup, Tech. Rep. TKN-14-002, 2014.

[10] F. Lemic, “Service for Calculation of Performance Metrics of IndoorLocalization Benchmarking Experiments,” Telecommunication Net-works Group, Tech. Rep. TKN-14-003, 2014.

[11] D. Lymberopoulos et al., “A Realistic Evaluation and Comparisonof Indoor Location Technologies: Experiences and Lessons Learned,”2015.

[12] F. Lemic et al., “EVARILOS Open Challenge: Track 3,” Telecommu-nication Networks Group, Tech. Rep. TKN-14-005, 2014.

[13] E. Elnahrawy et al., “The limits of localization using signal strength:a comparative study,” in Sensor and Ad Hoc Communications andNetworks (IEEE SECON 2004), IEEE, 2004, pp. 406–414.

[14] F. Lemic et al., “Experimental Decomposition of the Performance ofFingerprinting-based Localization Algorithms,” in IPIN’14, 2014.

[15] V. Handziski et al., “TWIST: a scalable and reconfigurable testbed forwireless indoor experiments with sensor network,” in RealMAN’06,2006.

[16] S. Bouckaert et al., “The w-ilab. t testbed,” in Testbeds and ResearchInfrastructures. Development of Networks and Communities, 2011.

[17] T. V. Haute et al., “The EVARILOS Benchmarking Handbook: Eval-uation of RF-based Indoor Localization Solutions,” in MERMAT’13,2013.

[18] S. Adler et al., “Experimental Evaluation of the Spatial Error Distri-bution of Indoor Localization Algorithms,” in UPINLBS’14, 2014.

[19] A. Behboodi et al., “Interference Effect on Localization Solutions:Signal Feature Perspective,” in VTC2015-Spring, 2015.

[20] J.-g. Park et al., “Implications of Device Diversity for OrganicLocalization,” in INFOCOM, 2011,, 2011, pp. 3182–3190.