Deep Learning based Wireless Localization for Indoor NavigationRoshan Ayyalasomayajula, Aditya Arun, Chenfeng Wu, Sanatan
Sharma, Abhishek Sethi, Deepak Vasisht and Dinesh Bharadia
1
Mobicom 2020
https://wcsng.ucsd.edu/dloc/
Outdoor Localization
2
Outdoor Localization
2
Navigation
https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-
navigation/
Autonomous Driving
https://play.google.com/
Delivery Bots
https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-
aebebe557ad4
Autonomous Drones
Outdoor Localization
2
Navigation
https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-
navigation/
Autonomous Driving
https://play.google.com/
Delivery Bots
https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-
aebebe557ad4
Autonomous Drones
Outdoor Localization
2
https://www.nasa.gov/sites/default/files/gps_constellation_0.jp
g
Navigation
https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-
navigation/
Autonomous Driving
https://play.google.com/
Delivery Bots
https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-
aebebe557ad4
Autonomous Drones
Indoor Localization
3
Indoor Localization
3
https://www.clipartkey.com/upic/4785/ https://medium.com/@brian.moran_91776/top-10-reasons-how-indoor-navigation-helps-shopping-malls-
be6376bf3d2
Indoor Localization
3
https://www.clipartkey.com/upic/4785/ https://medium.com/@brian.moran_91776/top-10-reasons-how-indoor-navigation-helps-shopping-malls-
be6376bf3d2
https://jonnegroni.com/2015/04/15/the-humans-of-wall-e-were-probably-better-off-without-him/
Lack of Context
4
Lack of Context
4
19, 13
X-axis (m)
Y-a
xis
(m
)
Lack of Context
4
19, 13
X-axis (m)
Y-a
xis
(m
)
Lack of Context
4
19, 13
X-axis (m)
Y-a
xis
(m
)
Lack of Context
4
19, 13
X-axis (m)
Y-a
xis
(m
)
WiFi based localization
5
WiFi based localization
5
MonoLocoMobiSys'18
Chronos NSDI'16
ToneTrackMobicom'15
SpotFiSigcomm'15
ArrayTrackNSDI'13
WiFi based localization
5
MonoLocoMobiSys'18
Chronos NSDI'16
ToneTrackMobicom'15
SpotFiSigcomm'15
ArrayTrackNSDI'13
Median: few decimeters
WiFi based localization
5
MonoLocoMobiSys'18
Chronos NSDI'16
ToneTrackMobicom'15
SpotFiSigcomm'15
ArrayTrackNSDI'13
Median: few decimeters >10% of cases: few meters
Deep Learning based Wireless Localization for Indoor
NavigationDLoc and MapFind
6
Deep Learning based Wireless Localization
7
Deep Learning based Wireless Localization
Localization: Novel learning based approach to solve for the environment dependent localization.
7
Deep Learning based Wireless Localization
Localization: Novel learning based approach to solve for the environment dependent localization.
Context: Bot that collects both Visual and WiFi data.
7
Table
TablesDesk
Deep Learning based Wireless Localization
Localization: Novel learning based approach to solve for the environment dependent localization.
Context: Bot that collects both Visual and WiFi data.
Dataset: Deployed it in 8 different in a Simple and Complex Environment
7
Table
TablesDesk
Deep Learning based Wireless Localization
Localization: Novel learning based approach to solve for the environment dependent localization.
Context: Bot that collects both Visual and WiFi data.
Dataset: Deployed it in 8 different in a Simple and Complex Environment
Results: Shown a 85% improvement compared to state of the art at 90th
percentile.
7
Table
TablesDesk
Challenge: Multipath, Non-Line of Sight
8
Challenge: Multipath, Non-Line of Sight
8
r
A
P
Smartphon
e
θ
Challenge: Multipath, Non-Line of Sight
8
r
A
P
Smartphon
e
Reflecto
r
θ
Challenge: Multipath, Non-Line of Sight
8
A
P
Smartphon
e
Reflecto
r
Obstacl
e
Challenge: Multipath, Non-Line of Sight
8
A
P
Smartphon
e
Reflecto
r
Obstacl
e
Need Knowledge of Environment
Requirements to design the neural network
9
Requirements to design the neural network
9
Input Representation
Requirements to design the neural network
9
Input Representation
Output/Target Representation
Requirements to design the neural network
9
Input Representation
NetworkOutput/Target Representation
Requirements to design the neural network
9
Input Representation
NetworkOutput/Target Representation
Objective/LossFunction
Requirements to design the neural network
9
Input Representation
NetworkOutput/Target Representation
Objective/LossFunctionGradient
Input Representation: Raw CSI data
10
Input Representation: Raw CSI data
10
Maximillian Arnold et. al., SCC 2019
Michal Nowicki et. al., ICA, 2017
Xuyu Wang, et al., IEEE Access 5, 2017
Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017
Input Representation: Raw CSI data
10
Maximillian Arnold et. al., SCC 2019
Michal Nowicki et. al., ICA, 2017
Xuyu Wang, et al., IEEE Access 5, 2017
Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017
Complex Channel Values and AWG noise
Input Representation: Raw CSI data
10
Can we represent them as images?
Maximillian Arnold et. al., SCC 2019
Michal Nowicki et. al., ICA, 2017
Xuyu Wang, et al., IEEE Access 5, 2017
Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017
Complex Channel Values and AWG noise
Input Representation: AoA-ToF images
11
Input Representation: AoA-ToF images
11
Input Representation: AoA-ToF images
11
Does not have context of Space and AP locations
Input Representation: XY images
AoA-ToF (Polar) to XY (cartesian)
12
Angle
of A
rriv
al,
𝝷°
Time of
Flight(m)
Input Representation: XY images
AoA-ToF (Polar) to XY (cartesian)
12
Angle
of A
rriv
al,
𝝷°
X axis
(m)
Time of
Flight(m)
Y a
xis
(m)
Polar to Cartesian
Input Representation: XY images
AoA-ToF (Polar) to XY (cartesian)
12
Angle
of A
rriv
al,
𝝷°
X axis
(m)
Time of
Flight(m)
Y a
xis
(m)
Polar to Cartesian
Context embedded Input Representation
Location Decoder Output targets
13
Location Decoder Output targets
13
Location Decoder Output targets
13
Location Decoder Output targets
13
Image-to-Image translation problem
Network Architecture
14
Network Architecture
14
1. Conv2d
[7,7,1,3]
2. Instance norm
3. ReLU
1. Conv2d
[3,3,2,1]
2. Instance norm
3. ReLU
Resnet Block* 1. ConvTranspose2d
[3,3,2,1]
2. Instance norm
3. ReLU
1. Conv2d
[7,7,1,3]
2. Instance norm
3. Sigmoid
1. Conv2d
[7,7,1,3]
2. Instance norm
3. Tanh
6 Resnet Blocks3 Resnet
Blocks
Location
DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
Network Architecture
14
1. Conv2d
[7,7,1,3]
2. Instance norm
3. ReLU
1. Conv2d
[3,3,2,1]
2. Instance norm
3. ReLU
Resnet Block* 1. ConvTranspose2d
[3,3,2,1]
2. Instance norm
3. ReLU
1. Conv2d
[7,7,1,3]
2. Instance norm
3. Sigmoid
1. Conv2d
[7,7,1,3]
2. Instance norm
3. Tanh
Llocation
6 Resnet Blocks3 Resnet
Blocks
Location
DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
Location Loss
Closeness in MSE sense
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]
15
Location Loss
Closeness in MSE sense
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]
Penalize multiple peaks
15
Location Loss
Closeness in MSE sense
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]
Penalize multiple peaks
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2 𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇 + λ 𝐿1[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 ]
15
High 90th percentile errors: Asynchronous Clocks
16
High 90th percentile errors: Asynchronous Clocks
16
r
AP
Smartphon
e
High 90th percentile errors: Asynchronous Clocks
16
r
Δr
AP
Smartphone
ToF offset
17
ToF offset compensation
18
DLoc: Network Architecture
19
6 Resnet Blocks3 Resnet
Blocks
Location DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
Offset Corrected Images
DLoc: Network Architecture
19
6 Resnet Blocks3 Resnet
Blocks
Location DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
6 Resnet Blocks
Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)
256
12
8 64 4
Offset Corrected Images
Insight: Single source
20
DLoc: Network Architecture
21
6 Resnet Blocks3 Resnet
Blocks
Location DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛+
6 Resnet Blocks
Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)
256
12
8 64 4
Offset Corrected Images
DLoc: Network Architecture
21
6 Resnet Blocks3 Resnet
Blocks
Location DecoderEncoder
4 64
128256 256
256
12
8 64 1
Input Images
Output Location
𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛
𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦
+
6 Resnet Blocks
Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)
256
12
8 64 4
Offset Corrected Images
Offset Compensation Loss
Defines consistency across images
𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 =1
𝑁𝐴𝑃
𝑖=1
𝑁𝐴𝑃
𝐿2[𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐸(𝐻)) − 𝑇𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦]𝑖
22
Offset Compensation Loss
Defines consistency across images
𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 =1
𝑁𝐴𝑃
𝑖=1
𝑁𝐴𝑃
𝐿2[𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐸(𝐻)) − 𝑇𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦]𝑖
22
Context: MapFind
23
Context: MapFind
23
WiFi Client
LiDAR
RGB-D
Server
Kobuki
Context: MapFind
23
Table
TablesDesk
Context: MapFind
23
How much data is needed?
Table
TablesDesk
Path Planning
24
Path Planning
Maximize coverage
Minimize traversal length
24
Path Planning
Maximize coverage
Minimize traversal length
24
Context Enabled Accurate Indoor Localization
Results
25
Setup
26
Setup
26
Complex High-multipath and
NLOS environment (1500 sq.
ft.)
AP
1
AP
3
AP
2
Plasma
Screens
Plasma
Screens
AP4 -
NLOS
Setup
26
Complex High-multipath and
NLOS environment (1500 sq.
ft.)
AP
1
AP
3
AP
2
Blocka
ge
Plasma
Screens
Plasma
Screens
AP4 -
NLOS
Setup
26
Complex High-multipath and
NLOS environment (1500 sq.
ft.)
Simple LOS based
environment (500 sq. ft.)
AP
3 AP
2
AP
1
Reflect
or
AP
1
AP
3
AP
2
Blocka
ge
Plasma
Screens
Plasma
Screens
AP4 -
NLOS
DLoc Result – Complex Environment
27
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)
0.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc Result – Complex Environment
27
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)
0.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Complex Environment
27
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)
0.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Complex Environment
27
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)
0.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Complex Environment
27
0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)
0.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Simple Environment
28
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)
00.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc Result – Simple Environment
28
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)
00.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Simple Environment
28
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)
00.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
DLoc Result – Simple Environment
28
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)
00.10.20.30.40.50.60.70.80.9
1
CD
F
DLoc
SpotFi
Baseline DL Model
Accurate Indoor Localization
Generalization across multiple setups
29
Generalization across multiple setups
29
Setup-1 Setup-2
Setup-3 Setup-4
Generalization across multiple setups
29
Setup-1 Setup-2
Setup-3 Setup-4
Trained on
Setup
Tested on
Setup
Median Error (cm)
90th Percentile Error (cm)
DLoc SpotFi DLoc SpotFi
1,3,4 2
1,2,4 3
1,2,3 4
Generalization across multiple setups
29
Setup-1 Setup-2
Setup-3 Setup-4
Trained on
Setup
Tested on
Setup
Median Error (cm)
90th Percentile Error (cm)
DLoc SpotFi DLoc SpotFi
1,3,4 2 198 420
1,2,4 3 154 380
1,2,3 4 161 455
Generalization across multiple setups
29
Setup-1 Setup-2
Setup-3 Setup-4
Trained on
Setup
Tested on
Setup
Median Error (cm)
90th Percentile Error (cm)
DLoc SpotFi DLoc SpotFi
1,3,4 2 71 198 171 420
1,2,4 3 82 154 252 380
1,2,3 4 105 161 277 455
Open-Sourced Dataset
30
Open-Sourced Dataset
30
• Enabling Baseline comparison for all algorithms
Open-Sourced Dataset
30
• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization
Open-Sourced Dataset
30
• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program
Open-Sourced Dataset
Labelled WiFi CSI data (WILD-v1)
8 different setups
4 different days
108K datapoints
2 different environments
30
https://wcsng.ucsd.edu/wild/
• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program
Open-Sourced Dataset
Labelled WiFi CSI data (WILD-v1)
8 different setups
4 different days
108K datapoints
2 different environments
30
https://wcsng.ucsd.edu/wild/
WILD-v2 Coming Soon
• 20 different setups
• 10 different days
• 1 million datapoints
• 8 different environments
• 20 different AP locations
• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program
Conclusion and Future Work
• Novel Deep Learning based algorithm with 85% incremental performance
compared to state-of-the-art.
• MapFind we have collected over 108k datapoints (and expanding) that is open-
sourced.
• Enabling large scale and autonomous indoor navigation
31
https://wcsng.ucsd.edu/dloc/