HUMAN ACTIVITY RECOGNITION USING BODY
AREA SENSOR NETWORKS
By
BO XU
Bachelor of Science in Electronic Engineering
University of Science and Technology Beijing
Beijing, China
2005
Submitted to the Faculty of the Graduate College of the Oklahoma State University in partial fulfillment of the requirements for the Degree of MASTER OF SCIENCE
May, 2009
ii
HUMAN ACTIVITY RECOGNITION USING BODY
AREA SENSOR NETWORKS
Thesis Approved: Xiaolin Li
Thesis Adviser Weihua Sheng Venkatesh Sarangan
A. Gordon Emslie
Dean of the Graduate College
iii
ACKNOWLEDGMENTS
I would like to express my gratitude to my supervisor, Dr. Xiaolin Li, whose
expertise, understanding, and patience, added considerably to my graduate experience. I
appreciate his vast knowledge and skill in many areas, and his assistance in writing this
thesis. I would also like to thank other members of my committee, Dr. Weihua Sheng,
and Dr. Venkatesh Sarangan for the assistance they provided at all levels of the research
project. Finally, I would like to thank my parents to support me go to United States,
where I learned a lot both in my study and in my life.
iv
TABLE OF CONTENTS Chapter Page 1. INTRODUCTION .....................................................................................................1
1.1 Wireless Sensor Networks .................................................................................1 1.2 Wireless Body Area Networks...........................................................................2 1.3 Hidden Markov Model .......................................................................................3 2. REVIEW OF LITERATURE AND SYSTEM ARCHITECTURE ..........................5 2.1 The Application of Hidden Markov Model .......................................................5 2.2 System Architecture for HMM Based Human Activities Identification ...........6 3. METHODOLOGY OF HUMAN ACTIVITIES RECOGNITION ...........................7 3.1 Data Collection ..................................................................................................7 3.1.1 Hardware Selection ...................................................................................7 3.2 TinyOS ...............................................................................................................9 3.3 Data Collection ..................................................................................................9
3.3.1 Sensor Nodes Data Collection ..................................................................9 3.3.2 Wireless Pulse Oximeter Data Collection ...............................................12 3.4 Data Parsing .....................................................................................................12 3.4.1 Observation Sequence Classification ......................................................13 3.4.2 Hidden States Definition .........................................................................16 3.5 Hidden Markov Model Training using Baum-Welch Algorithm ....................17
3.5.1 Baum-Welch Algorithm ..........................................................................18 3.6 Hidden States Decoding ...................................................................................20 4. EXTENDED KALMAN FILTER IN INDOOR LOCALIZATION .......................22
4.1 Kalman Filter ...................................................................................................22 4.2 Extended Kalman Filter ...................................................................................25 4.3 Application of Extended Kalman Filter in Indoor Localization ......................26
4.3.1 RSSI Based Distance measurement ........................................................26 4.3.2 System Model .........................................................................................28
v
Chapter Page
5. CONCLUSION AND FUTURE WORK ................................................................32 5.1 Human Activities Recognition Result… .........................................................32 5.2 Indoor Localization Result………………………………………………….. 40
5.2.1 Experiment Set-up ..................................................................................40 5.2.2 Experiment Result ...................................................................................42
5.3 Conclusion and Future Work ...........................................................................51 REFERENCES ............................................................................................................53
vi
LIST OF TABLES Table Page Table 2.1 PC/Laptop Side Architecture ........................................................................ 6 Table 2.2 Motes Side Architecture ................................................................................6
vii
LIST OF FIGURES Figure Page Figure 3.1 Sensor nodes distributed in one subject ....................................................10 Figure 3.2 Crossbow Micaz sensor node and MTS400 sensor board ........................11 Figure 3.3 Data Collection Mechanism .....................................................................12 Figure 3.4 Four Parts of Wireless Pulse Oximeter ....................................................12 Figure 3.5 Component Structure in Wireless Pulse Oximeter ...................................13 Figure 3.6 Observation Sequence Classification .......................................................15 Figure 3.7 States Transition Form for Hidden States ................................................17 Figure 4.1 Relation between distance and RSSI ........................................................28 Figure 5.1 Reluctant Acceleration Data for Proposition 1 .........................................32 Figure 5.2 Hidden States Decoding for Proposition 1 ...............................................33 Figure 5.3 Reluctant Acceleration Data for Proposition 2 .........................................34 Figure 5.4 Hidden States Decoding for Proposition 2 ...............................................34 Figure 5.5 Reluctant Acceleration Data for Proposition 3 .........................................36 Figure 5.6 Hidden States Decoding for Proposition 3 ...............................................36 Figure 5.7 Accuracy of human activities recognition with HMM and without
HMM........................................................................................................37 Figure 5.8 Subject using his/her own model and Subject using other trainee’s
model........................................................................................................38 Figure 5.9 Comparison of accuracy for human activities with one sensor, two
sensors and three sensors mounted on each subject ................................39 Figure 5.10 Comparison of accuracy among sensors mounted on different parts of
human body ............................................................................................40 Figure 5.11 Two target nodes deployed vertically to record three axis
acceleration data .....................................................................................42 Figure 5.12 Real route coordinate, EKF implemented coordinate, in a
1.8m*2.1m*1.5m area (walk diagonally) ..............................................42 Figure 5.13 X-axis data for EKF coordinate, real route coordinate ...........................43 Figure 5.14 Y-axis data for EKF coordinate, real route coordinate ...........................43 Figure 5.15 Z-axis data for EKF coordinate, real route coordinate ...........................44 Figure 5.16 Average error for EKF in three axis in the 1.8m*2.1m*1.5m area ........44 Figure 5.17 Real route coordinate, EKF implemented coordinate, calculated
coordinate in a 1.8m*2.1m*1.5m area (walk straightly) .......................45 Figure 5.18 X-axis coordinate of EKF and real route ................................................46 Figure 5.19 Y-axis coordinate of EKF and real route ................................................46 Figure 5.20 Z-axis coordinate of EKF and real route ................................................46 Figure 5.21 Average error for EKF in three axis in the 1.8m*2.1m*1.5m
viii
Figure Page
area .........................................................................................................47 Figure 5.22 Real route coordinate, EKF implemented coordinate, in a
1.8m*6m*1.5m area (walk diagonally) .................................................48 Figure 5.23 X-axis coordinate of EKF and real route ................................................49 Figure 5.24 Y-axis coordinate of EKF and real route ................................................49 Figure 5.25 Z-axis coordinate of EKF and real route ................................................49 Figure 5.26 Average error for EKF in three axis in the 1.8m*6m*1.5m
area (walk diagonally) .............................................................................50
1
CHAPTER 1
INTRODUCTION
1.1 Wireless Sensor Networks
As an emerging technology that bridges the physical world and the digital
information world, wireless sensor networks provide a tiny, low-power and low cost but
powerful platforms for data collection and transmission [1]. A wireless sensor network
consists of distributed smart devices equipped with processor, radio, memory, and
sensors, monitoring physical or environmental conditions cooperatively in multiple
modalities, for example, temperature, humidity, sound, motion, vibration and so on.
Wireless sensor networks are now used in many areas, such as traffic control, precision
agriculture, environment and habitat monitoring, healthcare applications, home
automation, and so on.
Wireless sensor networks consist of many sensor nodes which can be considered as
miniature computers. Each sensor node is equipped with a microcontroller, a radio
transceiver or other communications device, and an energy supply, usually a battery.
This thesis will focus on human activity recognition using wireless sensor networks.
Each human being will be equipped with several sensor nodes in his/her body to build up
a body area sensor network. Sensor data will be saved and later processed for training and
analysis purpose. The other topic investigated in this thesis is indoor localization. Our
method uses RSSI based distance measurement and Extended Kalman Filter. The
2
distance measured is used as input data in the Extended Kalman Filter to get the location
of a target node. In the next section, we will briefly introduce body area sensor networks.
1.2 Wireless Body Area Sensor Networks
Wireless body area sensor networks (WBASNs) are an extension of wireless sensor
network,which focus on the application of wireless sensor networks in human body.
The health care system will benefit from introducing the continuous and low-cost
monitoring tools with real time updates of medical sensor reading. A wearable wireless
body area network can integrate several smart sensors being used for computer assisted
rehabilitation and early detection of medical conditions.
T.G Zimmerman firstly presented the concept of wireless body sensor networks in the
article [2]. There is a lot of space for research in the WBASNs. WBASNs consist of
sensor nodes which can be either wearable or inside human body, monitoring vital data of
the subject and then send the data to the gateway. This thesis will only cover the
application of wearable sensors mounted on human body.
In WBASNs, two types of device are necessary: The first type of device is the wearable
sensor device which can transmit data to host. The second type of device is the host, such
as a base node connected with the laptop. A number of wearable sensor nodes mounted
on different parts of human body such as arms, legs, toes, and waist will transmit the data
to host where raw data will be processed.
With the growing technology in wireless sensor network and wearable computing,
human activities monitoring and physiological monitoring become practical. We briefly
review the previous work on human activities monitoring in wireless sensor networks.
3
Live Net is a long-term health monitoring project conducted in the media lab of MIT,
which provides human activities monitoring and danger warning mechanism based on the
real time sensor data.[9] In [7], measured acceleration and angular velocity data gathered
through wearable sensors is introduced to determine a user’s location between pre-
selected locations and classify walking ,sitting, and standing behaviors. In [8], a gait
assessment system is developed for elderly monitoring. And many others [10,11,12,13,14]
also discuss physical activity recognition using acceleration sensors.
In this thesis, we will use Crossbow Micaz sensor node attached with Crossbow
MTS400 accelerometer sensor board to get acceleration from different parts of human
body. Three sensor nodes are placed at different parts of the subject, which are left toe,
right outer leg joint, and right side of waist. We will use another sensor node attached
with wireless pulse oximeter to get the heart rate data. Heart rate data plays an assistant
role in this thesis, which will be used for fall detection.
Series of human activities can be regarded as Markov process. Individual human
activity can be regarded as a state in Markov model. The activity series can not be
observed directly from acceleration data. Therefore, Hidden Markov Model (HMM) is
introduced to find the most probable hidden states of human activities. This thesis uses
observation sequence (acceleration data) which generated by sensor nodes to estimate the
hidden states (human activity series) under the HMM.
1.3 Hidden Markov Model
Hidden Markov Model is a statistical model. In a Hidden Markov Model, the system
being processed is considered as a Markov process, which has unknown parameters. The
4
hidden parameters are determined from the observable sequence. The state transition
probabilities are the only parameters in a Markov Model as the state is observable
according to the observation sequence. In Hidden Markov Model, the state is not directly
observable according to the observation sequence. The application of Hidden Markov
Model is to discover the sequence of states using observable sequence. There are three
common algorithms to solve problems regarding to Hidden Markov Model:
Forward-Backward algorithm: If the parameters of HMM have been given, calculates
the probability of an observed sequence and figure out the hidden state associated with
the observed sequence.
Viterbi Algorithm: If the parameters of HMM and observation sequences have been
given, find out the hidden states which lead to the highest probabilities that generate the
observed sequence.
Baum-Welch Algorithm: If the parameters of HMM are not given, only the
observation sequence is given, find out the value of state transition probabilities and
observation transition probabilities. The parameters of HMM will be fully discovered
after running the Baum-Welch Algorithm.
In this thesis, firstly, we define the parameters for the Hidden Markov Model
according to the system being modeled. Secondly, we have the system model trained by
using Baum-Welch algorithm. Finally, the Viterbi algorithm is introduced to identify the
activity series based on the new observation sequences. Meanwhile, forward-backward
algorithm is conducted in order to compute the probability of a particular output sequence.
The indoor localization will be discussed in the later chapter.
5
CHAPTER 2
REVIEW OF LITERATURE AND SYSTEM ARCHITECHTURE
2.1 The application of Hidden Markov Model
Hidden Markov Model (HMM) was introduced and developed in the late 1960s [15].
L.E Baum, T. Petrie, N. Weiss, and G. Soules published the famous Baum-Welch
Algorithm that solved the model training problem [16] in 1968. The Baum-Welch
Algorithm can be regarded as an Expectation-Maximization (EM) method for maximum
likelihood calculation. In [20], properties of the EM algorithm are proved by C.F.J. Wu.
In [21], S.E. Levinson, L.R. Rabiner, and M.M. Sondhi published another EM method for
HMM training which laid a solid foundation for the application of HMM. Thereafter,
HMM was widely used in speech recognition [25]. L.R. Rabiner gave a classic tutorial on
how to apply the Hidden Markov Model to speech recognition in [24]. Nowadays, HMM
is also used for handwriting recognition [23] and other areas.
Introducing Hidden Markov Model for human activities recognition in wireless sensor
networks is a new research area. A project named SATIRE carried out by UIUC provides
a smart jacket which monitors the daily activities of human being and performs the
outdoor localization with the help of GPS component mounted on the sensor node. The
smart jacket serves as a wearable platform for personal health monitoring. [28]. A study
integrated with optimization approach in 3-D model based framework is introduced in
[29] for posture estimation and a motion-tracking scheme with GA based particle filter is
6
described as a solution for motion capture problem.
2.2 System Architecture for HMM Based Human Activities Identification
The following tables show the system architecture for human activities recognition in
this thesis.
Application Layer (Simple user interface in Cygwin) Interpretation Layer (Hidden states decoding) Parsing Layer (Raw sensor data parsing) Physical Layer (USB connection between PC/Laptop and Gateway)
Table 2.1 PC/Laptop side architecture
Application Layer (motes programmed for realizing certain function) Sensor Layer (Synchronization protocol, Memory logging protocol) Physical Layer (UART and RF radio communication)
Table 2.2 Motes side architecture
In chapter three, we will show the choice of hardware, mechanism of data collection and
data parsing. After that, we will train the system model with Baum-Welch algorithm
using observed sequences that come from the acceleration data generated by sensor nodes.
In the hidden states decoding step, Viterbi algorithm is applied to find the Viterbi path,
which is the most probable hidden states that lead to the observation sequences. In
chapter five, we will show all the result graphs and discuss several issues regarding to
Hidden Markov Model and indoor localization with Extended Kalman Filter.
7
CHAPTER 3
METHODOLOGY OF HUMAN ACTIVITIES RECOGNITION
3.1 Data Collection
3.1.1 Hardware Selection
In this thesis, we use Crossbow Micaz sensor node with Crossbow MTS400 sensor
board to get sensor data. Crossbow Micaz sensor is equipped with a 7.38 MHz Atmel
processor, 128 KB program memory, 4 KB RAM, and 512 KB non-volatile storage. The
radio component is a Chipcon SmartRF CC2420, with 2.4GHz frequency, Manchester
encoding, and linear RSSI (received signal strength indicator). Output power is digitally
programmable by setting the PA POW register.
MTS400CA sensor board utilizes the latest generation of IC-based surface mount
sensors. It combines several independent sensors into one sensor board. Specifications
are Dual-axis Accelerometer: Analog Devices ADXL202JE with Acceleration range ±2g
Barometric Pressure Sensor: Intersema MS5534AM Ambient Light Sensor TAOS
TSL2550D, Relative Humidity and Temperature Sensor: Sensirion SHT11.
Acceleration data can be used to reconstruct activities based on motion patterns. This
thesis demonstrates such reconstruction using Hidden Markov Models which are fed with
raw acceleration measurements to recognize activity types. Accelerometers are the most
important hardware in our experiments as activity measurement is based on the data
collected by accelerometers. In this thesis, we use Analog Devices ADXL202JE, which
8
has a range of ±2 g on both axes and a noise density and sensitivity of 200 μgHz and
167mV/g respectively.
The hardware for activity recognition has the following responsibilities: Sensing,
Storage, Energy supply, and Communication. Acceleration data is stored in the flash
memory of Crossbow Micaz sensor node during the experiments.
Wireless pulse oximeter will be introduced in the experiment as well. Pulse oximeter
monitors the heart rate and oxygenation of a subject and plays an important role in patient
health monitoring. In our experiment, the wireless pulse oximeter is placed on the
fingertip of a subject, and a light containing both red and infrared wavelengths passes
from one side of finger to the other side. Oxygenation is made by measuring the ratio of
changing absorbance of the red and infrared light. The changing absorbance can be
achieved by measuring the changing absorbance of the red wavelengths and infrared
wavelengths.
The wireless pulse oximeter used in this thesis is developed by Harvard Sensor
Network Lab and Smith Medical Group. It can collect heart rate (HR) data and oxygen
saturation (SpO2) data in a frequency of 80Hz. The program embedded in the wireless
pulse oximeter is modified according to the CodeBlue project carried out by Harvard
University. The average adult heart beats at about 70 bpm (males) and 75 bpm (females)
in a stationary motion; however, the heart rate varies among people and can be
significantly lower in endurance athletes. By monitoring the heart rate of one person will
assist us to recognize the human activity accurately.
In this thesis, heart rate data is imported as an assistant indicator for human activities
recognition, usually, there is significant difference between stationary motion and non-
9
stationary motion. But on the other hand, heart rate changes rapidly and varies from
person to person, even for the same human being, different physical health status will
effect heart rate greatly, therefore, the heart rate data is not considered as a vital
parameter in the observation sequence and is only taken into account when acceleration
data can not well support the human activities recognition.
3.2 TinyOS
All the programs running in the motes are supported by TinyOS. TinyOS is a free and
open source operating system designed for the implementation of wireless sensor
networks (WSNs). TinyOS is a component-based embedded operating system written in
nesC programming language, which is optimized for the memory limitation of sensor
networks. Components in TinyOS are connected to each other with interfaces. The
components and interfaces packages include communication, routing, sensing, actuation
and storage. Event driven execution model is optimized throughout the system and a task
queue is out there to receive the tasks posted by incoming event. All tasks in the queue
are executed in a FIFO manner. The program size is effectively minimized because of the
event driven architecture since only the necessary and essential components are contained.
All the programs used in sensor nodes are programmed in NesC language.
3.3 Data Collection
3.3.1 Sensor Nodes Data Collection
10
In our experiment, each subject is equipped with four sensors. Three of them are
Crossbow Micaz sensor node with Crossbow MTS400 sensor board attached. The fourth
one is the Crossbow Micaz sensor node with wireless pulse oximeter attached.
For each subject, sensor one is placed at the left toe, sensor two is placed at the outer
joint of right leg, and sensor three is placed at the right side of waist as shown in Figure
3.1.
Figure 3.1 Sensor nodes distributed in one subject.
Figure 3.2 shows the model of Crossbow Micaz sensor node and Crossbow MTS400
sensor board which is the same as the sensor package in the red circle in Figure 3.1.
11
Figure 3.2 Crossbow Micaz sensor node and MTS400 sensor board.
For the time synchronization purpose, a specific mechanism is introduced to ensure
all the sensor nodes start to work at the same time. The mechanism is simple, after
turning on the sensor nodes, the base station which connects to the laptop will send a
‘start’ message to all other sensor nodes, which will start working once they receive the
message. As the sensor nodes are very close to the laptop, the time delay can be neglected.
They will receive the message and start to work at the same time. All the data will be
saved in the memory of the Micaz sensor nodes once they start working.
Once the whole experiment is finish, wee need to get the data from the memory of the
sensor, the mechanism is as follow. The base will send a TOSMsg to certain sensor, when
the sensor receives the TOSMsg, it will send all of the saved data back to base node. Base
node is attached on Crossbow MIB520-USB-Gateway which connects with a laptop.
When base node receives these data, data will be saved in a file for further processing.
Figure 3.3 shows the data collection mechanism:
12
Base StationSensor node
Send TosMsg to start or stop
sensor working
Start or stop logging acceleration
data after receiving TosMsg
Base Station Sensor node
Send TodMsg asking for sending back
acceleration data.
Send acceleration data back to base
Figure 3.3 Data collection mechanism
3.3.2 Wireless Pulse Oximeter Data Collection
Wireless pulse oximeter includes four separate components. The first part is Crossbow
Micaz sensor node. The second part is an intermediate board which connects pulse
oximeter with Micaz sensor node. The third part is pulse oximeter which can get heart
rate and SpO2 data from the finger sensor. The forth part is the finger sensor.
Figure 3.4 shows the components of wireless pulse oximeter:
Figure 3.4 Components of wireless pulse oximeter
13
The data collection mechanism is the same as discussed in section 3.3.1.
The program structure used in wirelesss pulse oximeter is shown in Figure 3.5
Figure 3.5 Component structure in wireless pulse oximeter
3.4 Data Parsing
The first thing to do for data parsing is to classify the raw acceleration data and
confirm parameters for Hidden Markov Model.
A Hidden Markov Model is recognized by the following elements:
• N - the number of hidden states
• M - the number of symbols
• A - the state-transition probability matrix. Aij = P(qt+1 = j | qt = i) 1 i , j N
• B - observation probability distribution: Bj(k) = P(ot = k | qt = j) 1 k M
• - the initial state distribution: i= P(q1=i) 1 i N
• - the entire model = (A, B, )
We will discuss how to determine the value of M first.
3.4.1 Observation sequence classification
14
ADXL202E in MTS400 sensor board has x-axis and y-axis acceleration measurement.
Firstly, we need to get the resultant acceleration of X axis and Y axis:
2 2
xy x ya a a
The reason why we use resultant acceleration is that using either x-axis or y-axis data
alone will result in lower recognition rate for non-stationary human activities such as
climbing stairs, walking and running. In the other hand, we will use average value of x-
axis acceleration data when recognizing stationary activities as it will help to improve
recognition accuracy. Secondly, we group seven consecutive resultant acceleration data
together and calculate the value of following types of data in one group: a1, a2, and a3.
a1 means the maximum value minus the minimum value in one group for sensor one. a2
means the maximum value minus the minimum value in one group for sensor two. a3
means the maximum value minus the minimum value in one group for sensor three. But
a1, a2 and a3 are not the final observation symbols used in this thesis. There is one more
step to do for data processing. Thirdly, we will use a binary tree algorithm to classify a1,
a2, a3 values to different observation states.
The basic schema is shown in Figure 3.6:
16
Figure 3.6 Observation sequence classification
In Figure 3.6, a1MM, a2MM, a3MM are the same as a1, a2, and a3. th11,th12,th13
are different threshold values for sensor one, th21,th22,th23 are the threshold value for
sensor two, th31,th32,th33 are the threshold values for sensor three.
Input data are a1, a2, and a3 recorded by three sensor nodes at the same time.
Classification Program will firstly check whether the value of a1 is smaller than threshold
value th11. If it is the case, it will check whether a2 is smaller than threshold value th21.
If not, it will check whether a1 is between th11 and th12. Checking will not stop until all
the value intervals have been reached, and one special state will be assigned to each
individual checking route as shown in the Figure 3.6.
In this way, observed acceleration sequences are divided into 60 probable states. The
season why we do this is that large amount of raw acceleration data create a lot of noise
when they are brought into program running Baum-Welch algorithm and Viterbi
algorithm. After being processed by the binary tree algorithm, observation sequences
have been created, which greatly improve the system performance. Heart rate data is not
considered as one of the input observation sequence because of the non-stable status of
the subject.
3.4.2 Hidden States Definition
Hidden states are defined as different human activities. For example, we define
hidden states as standing(S9), walking(S2), running(S7), and transition states between
these stable states, which are standing to walking(S4), walking to standing(S6), standing
to running(S8), running to standing(S3), running to walking(S5), and walking to
17
running.(S1). As fall detection is a very important part in patient monitoring and
firefighter monitoring. We will add lying motion in later discussion because we regard
the transitional states between other activities and lying motion as the falling process.
When subject is lying on the ground, it is highly possible that he/she has fallen down to
the ground. Figure 3.7 shows a more detailed relation between those states.
S9
S4
S6
S2
S3
S8
S7S5
S1
Figure 3.7 States transition form for hidden states.
3.5 Hidden Markov Model Training Using Baum-Welch Algorithm
In our experiment, each subject carrying four sensors will perform human activities
shown in section 3.4.2 for training purpose, the order of his/her activities may vary, but
all the three stable states and six transition states must be included.
18
3.5.1 Baum-Welch Algorithm
Baum-Welch algorithm is a GEM algorithm. GEM means generalized expectation-
maximization. It can compute maximum likelihood estimations and posterior mode
estimations for the parameters (transition and emission probabilities) of an HMM, when
given emission as the training data only.
The algorithm has two steps:
Firstly, calculating the forward probability and the backward probability for each
HMM state.
Secondly, calculating the transition-emission pair values and multiplied by the
probabilities of the whole observation sequence.
In another word, given O = (O1, O2,…, OT ); Estimate = (A,B, ) to maximize
P(O| ). The procedures of Baum-Welch Algorithm will be:
• Let initial model be 0
• Compute new based on 0 and observation O.
• If log P(O| ) - log P(O| 0) < , then stop.
• Else set 0 ← and go to the second step.
In [18], Baum et. al give a detailed explanation on the mathematical procedure how
this algorithm works.
Baum-Welch: Preliminaries:
Define ( , )i j as the probability of being in state i at time t in state j at time t+1.
1 1 1 1
1 1
1 1
( ) ( ) ( ) ( ) ( ) ( )( , )
( | ) ( ) ( ) ( )
t ij j t t t ij j t t
N N
t ij j t t
i j
i a b o j i a b o ji j
P O i a b o j
19
Define ( )t i as the probability of being in state i at time t, given the observation
sequence.
1
( ) ( , )N
t t
j
i i j
1
( )T
t
t
i is the expected number of times state i is visited.
1
1
( , )T
t
t
i j
is the expected number of transitions from state i to state j.
Baum-Welch: Update rules:
i = expected frequency in state i at time (t = 1) = 1( )i
ija = (expected number of times in state j and observing symbol k)/(expected number
of times in state j.
1
11
1
( , ),1 ,1
( )
Tt
tij T
tt
i ja i N j N
i
( )jb k = (expected number of times in state j and observing symbol k)/( expected
number of times in state j ).
( )jb k ,( )
( )t
tt o k
t
j
j
In the training process, each Hidden Markov Model is trained for different subject.
Each subject needs to have his/her own trained file in order to improve the accuracy for
hidden states decoding.
The number of states N is set to be 9. For each training process, the value of M is not
fixed, as each training process may take different time, the value of M may vary.
20
3.6 Hidden States Decoding
In this section, Viterbi algorithm is introduced to find the most probable activity
series (hidden states of HMM) from observation series.
The Viterbi algorithm is a dynamic programming algorithm to find the Viterbi path
that is hidden in the observed sequence. The Viterbi path can also be regarded as the most
likely sequence of hidden states. Assume to be the probability of the most probable
path to the state. Then ( )t i is the maximum probability of all possible sequences ending
at state i when it is time t. The Viterbi path is the sequence which results in this maximal
probability. A is the matrix of state-transition probabilities with elements aij, B is the
matrix of observation probabilities with elements bij, and is the vector of initial state
probabilities with elements (i). In mathematical expression:
Find the path (q1,…, qt ) that maximizes the likelihood P(q1,…, qt |O, ).
Solution by Dynamic Programming:
Define: 1 2 1
1 2 t 1 2,..., t, ,...,
( ) max (q ,q ...,q =i, o ,o o | )t
tq q q
i P
By induction we have:
1 1( ) max[ ( ) ] ( )t t ij j ti
j i a b o
Initialization:
1( ) ( ),1i i ii b o i N
1( ) 0i
Recursion:
1
1
( ) ( ) ] ( )max[t t ij j t
i N
j i a b o
11
( ) max[ ( ) ], 2 ,1t t iji N
j i a t T j N
21
Termination:
1max[ ( )]T
i NP i
*
1arg max[ ( )]TT
i Nq i
Path (state sequence) backtracking:
* *1 1( ), 1, 2,...,1tt tq q t T T
In the end, the Viterbi path will be calculated and we will discuss the hidden states decoding in later chapter.
22
CHAPTER 4
EXTENDED KALMAN FILTER IN INDOOR LOCALIZATION 4.1 Kalman Filter
Kalman filter is an efficient tool in estimating the state of a linear dynamic noisy
system. In the world of engineering, Kalman Filter plays an important role in many areas,
such as computer version, aerospace, aircraft navigation, control systems and so on. Most
famously, Kalman filter solves the linear-quadratic-Gaussian control problem (LQG) in
control theory. Kalman filter is based on the paper published in 1960 by R.E.Kalman,
who introduced a linear filtering method for discrete data. Kalman filter provides a good
estimate of the coming state according to the value of the previous state, which is
different from many other engineering filters, such as the Williams filter which requests
values from all the previous states. Kalman Filter estimates the state nx R of a discrete-
time controlled process described in the stochastic difference equation below:
1k k k kx A x Bx w
A measurement matrix mz R is provided for the estimation.
k k kz Hx v
Matrix A is a square matrix which relates the current process state estimate with the
state in the previous time instant. Matrix B is not necessarily a square matrix, and relates
the current process state estimate with the system input at previous time instance. Matrix
23
H, which is a rectangular matrix, relates measurement to state estimate. Variables wk-1
and vk are random variables representing the process noise and measurement noise. They
are assumed to be mutually independent, white and with normal probability distributions.
p(w) ~ N(0,Q)
p(v) ~ N(0,R)
Where Q is the process noise covariance and R is measurement noise covariance.
Although they are assumed to be zero, in reality, they change with each time update step
or measurement update step.
Kalman Filter is executed in two steps, which are time update and measurement
update. Time update projects the state variable vector estimate ahead with consideration
of the system input, which means that the time update function predicts the new state of
the Kalman Filter. The measurement update adjusts the time update estimation with
consideration of measurements during the time interval.
Kalman Filter is written as follow:
.. (4.1)
.. (4.2)
.. (4.3)
.. (4.4)
.. (4.5)
Firstly, we need to estimate the initial value of the state variable vector and error
covariance matrix. Then we need to go through following steps:
Step one: Time Update:
24
Time update equations are 4.1 and 4.2. Equation 4.1 calculates the priori estimate of
the state variable vector (which is the estimation of state X at time k given measurements
up to time k-1). It is based on the previous time state and input to the system since the last
update. Equation 4.2 gives the update of the priori error covariance matrix P, which
represents the fact that the system will continuously decay over time due to the error in
the input and the system model which contributes uncertainty in the estimation.
Step two: Measurement Update:
The measurement update equations include equation (4.3), (4.4) and (4.5), which are
used to correct the time update state estimation in the previous step based on the sensor
reading. In our experiment, sensor reading is the acceleration data from X-axis, Y-axis
and Z-axis. They are used as the system input in the Kalman Filter. Equation 4.3
calculates the Kalman gain K to minimize the posteriori error covariance matrix P.
Equation 4.4 calculates the posteriori estimate of the state variable vector which is the
estimation of state X at time K given measurement data up to time K. This equation
updates the state variable estimation for the new measurement. In this process, the
equation compares the actual sensor measurement with sensor measurement predicted by
the state estimate (Hx) and calculates the ‘innovation’ or ‘residual’, which is the
difference between actual sensor measurement and estimated sensor data. Kalman gain is
calculated to determine how much to adjust the time update estimation.
Equation 4.5 gives the posteriori error covariance matrix P. If there is no system input,
then the Kalman Filter will have no measurement update, it will only have the time
update function.
25
4.2 Extended Kalman Filter
Kalman Filter assumes that the relation between measurements, input and state
variables are all linear. This is not always the case as sometimes the system being
modeled can be a non-linear system. In that case, Extended Kalman Filter (EKF) is
introduced to solve the problem. For Extended Kalman Filter, the basic idea is to linearise
the non-linear measurement of current state estimation so that we can introduce similar
stochastic equation to perform the recursive procedures defined in Kalman Filter. With
this in mind, we will show the equations of EKF below:
..(4.6)
..(4.7)
This non-linear equation can be expressed by the linear equation:
..(4.8)
..(4.9)
Where:
A is the Jacobian matrix of partial derivatives of f with respect to x
H is the Jacobian matrix of partial derivatives of h with respect to x
W is the Jacobian matrix of partial derivatives of f with respect to w
26
V is the Jacobian matrix of partial derivatives of h with respect to v
The time update (predict) and measurement update (correct) equations are shown
below.
..(4.10)
..(4.11)
..(4.12)
..(4.13)
..(4.14)
4.3 Application of Extended Kalman Filter in Indoor Localization
4.3.1 RSSI Based distance measurement
Many localization algorithms request distance measurement to estimate the position
of unknown subject. Our algorithm will perform in the same manner. We use received
signal strength (RSSI) of the incoming radio signal to estimate distance between target
nodes and stable nodes. In our experiment, one sensor node will send message to another
sensor node with the transmission power PTX, and the receiver will have a receiving
power PRX which is a signal indicator that can be used for calculating the distance
between the transmitter and the receiver. PRX is directly effected by PTX
27
The radio model we use is described as below:
[ ] ( ) ( )i loss i i iP dBm P d t
Where:
100
( ) 10 logi
loss i TXd
P d P Kd
K is a unitless constant that depends on the environment. D0 is a reference distance
for the antenna far field, and is the path loss coefficient. i denotes the random
attenuation due to shadowing, while ( )i t accounts for the fast fading effect [37].
Therefore, the channel model can be combined as following equation:
100
[ ] 10 logi
i TX id
P dBm P Kd
..(4.15)
In our experiment we choose parameter value as K+PTX=-45dBm, =1.6, d0=1m,
i =6.82dBm. Then we have the expression of distance between the transmitter node and
the receiver node.
45.5 6.82
1610RSSI
d
Here is a brief graph shows the relation between RSSI and distance according to the
experiment conducted in the lobby of computer science department.
28
Figure 4.1 Relation between distance and RSSI
4.3.2 System Model
The system consists of three stationary nodes deployed at pre-set coordinates. One
or two target nodes are carried by a human being. If acceleration data is not considered as
system input then only one target node is used. Otherwise, two nodes are used in order to
get x axis, y axis and z axis acceleration data as MTS400 sensor board can only measure
x axis and y axis acceleration data, we need to put one more sensor node to record z axis
acceleration data. In our experiment, direction of x axis is defined as pointing to the south
side, direction of y axis is defined as pointing to the west side and direction of z axis is
defined as pointing to ground. The target nodes will broadcast message to reference
nodes.
Assume that the coordinates of three stable nodes S1, S2 and S3 are (x1, y1, z1), (y2,
y2, z2) and (x3, y3, z3). The distance between stable nodes and target node are d1, d2, d3.
Then we can get the following equations:
29
2 2 2 21 1 1 1
2 2 2 22 2 2 2
2 2 2 23 3 3 3
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
d x x y y z z
d x x y y z z
d x x y y z z
..(4.23--25)
We can get the calculated coordinate of target node using equation 4.23---4.25. This
result has very low accuracy in our experiment. d1, d2, d3 are measured by RSSI based
distance measurement method.
We can have three different system models according to how many states we want to
define in the EKF. Generally, we list three EKF models which relate to our subject
tracking project:
Coordinate Model: The state vector of EKF only contains the coordinate of the
subject.
Coordinate-Velocity Model: The state vector of EKF contains the coordinate of
subject and the moving speed of subject as well.
Coordinate-Velocity-Acceleration Model: The state vector of EKF contains
coordinate, speed and acceleration information of the subject.
We will use the third model here.
Let coordinate estimation vector be
x
y
z
x
y
z
x
y
z
v
X v
v
a
a
a
30
Where x, y, and z are the coordinate of target node. Vx, Vy and Vz are the velocity in x,
y and z axis. xa , ya and za are the acceleration in x, y and z axis.
The state space model is given as:
1
xk
k k yk k
zk
u
X AX B u X
u
Where
1 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0
0 0 1 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0 0 1 0 0 0 0
0 0 0 0 0 1 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
T
T
T
A
and
2
2
2
/ 2 0 0
0 / 2 0
0 0 / 2
0 0
0 0
0 0
1 0 0
0 1 0
0 0 1
T
T
T
T
B T
T
xku , yku , and zku are the control forces in x, y and z axis. kX is the system process
white noise.
Measurement equation can be described as shown below:
31
1 1 1
2 2 2
3 3 3
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
k k k k k
d d d
x y z
d d dZ HX Z X Z
x y z
d d d
x y z
Where:
2 2 2( ) ( ) ( )
i i
i i i
d x x
x x x y y z z
..(4.16)
2 2 2( ) ( ) ( )
i i
i i i
d y y
y x x y y z z
..(4.17)
2 2 2( ) ( ) ( )
i i
i i i
d z z
z x x y y z z
..(4.18)
Then according to the EKF algorithm, we will do the following calculation:
Step 1. Calculate 1k k kX A X BU
and kZ according to 1kX
.
Step 2. The error covariance equation is:
..(4.19)
Step 3. Compute the Kalman gain:
..(4.20)
Step 4. Update estimation with measurement function:
..(4.21)
Step 5. Update the error covariance:
..(4.22)
Step 6. Go to step 1.
32
CHAPTER 5
CONCLUSION AND FUTURE WORK
5.1 Human Activity Recognition Results
Once the Hidden Markov Models are built, three different series of human activities
are performed to have the models tested and find the Viterbi paths. They are:
1. Standing-walking-standing-climbing stairs-standing-running-standing
2. Standing-sitting-lying
3. Standing-walking-sitting-running-lying-walking-climbing stairs-walking-standing
Result graph is show as below. These results are based on the human activities
conducted by author himself.
For proposition one: Each subject will perform the following activities: standing,
walking, standing, climbing stairs, standing, running and standing.
standing-walking-standing-climing_stairs-standing-running-standing
0
2
4
6
8
10
12
1 11 21 31 41 51 61 71 81 91 101111121131141 151161
time(s)
Acceleration(m/s2)
toe sensor
shoulder sensor
leg sensor
Figure 5.1 Reluctant acceleration data for Proposition 1
33
Hidden States Decoding
0123456789
1 11 21 31 41 51 61 71 81 91 101111121131141151161
time(s)
Hidden States
Viterbi Path
Figure 5.2 Hidden States Decoding for Proposition 1.
In Figure 5.2, the meanings of hidden states are: standing(S6), walking(S2), climbing
stairs(S8), and running(S3). Transitional states include standing to walking, walking to
standing and so on. Hidden state S1 appears when subject moves from walking to
standing, standing to climbing stairs, and standing to running. So S1 can be regarded as
the transitional state. As S1 is supposed to have only one definition, the transitional states
are not well recognized in this graph. Other transitional states in this figure include S4 and
S5. X-axis represents time, y-axis represents the hidden states.
Figure 5.1 shows the original reluctant sensor data a1, a2, and a3, which we
mentioned in section 3.4.1. Series 1 stands for the sensor node placed at the left toe.
Series 3 stands for the sensor node placed at the right leg. Series 2 stands for the sensor
node placed at the right side of waist. Human activities performed in the first graph are
clearly recognized as shown in the second graph. X-axis represents time and y-axis
represents acceleration with unit m/s^2. Time unit is 1 second.
For proposition 2, each subject will go through standing, sitting and lying activities.
34
standing-sitting-lying
0
0.5
1
1.5
2
2.5
1 13 25 37 49 61 73 85 97 109 121 133 145 157
time(s)
Acceleration(m/s2)
toe sensor
chest sensor
leg sensor
Figure 5.4 Reluctant acceleration data for Proposition 2.
Hidden States Decoding
0
1
2
3
4
5
6
7
1 13 25 37 49 61 73 85 97 109 121 133 145 157
time(s)
Hidden States
Viterbi Path
Figure 5.4 Hidden States Decoding for Proposition 2.
Proposition 2 is a group of stationary human activity series including standing, sitting,
and lying. Figure 5.3 shows the x-axis acceleration data of toe sensor, chest sensor and
leg sensor. As we can see, leg sensor data is very helpful for recognizing these human
activities. It provides a stable and reliable standard to classify these three stationary
activities. We can directly use these sensor data as the observation sequence in theHidden
35
Markov Model without running the classification algorithm. In Figure 5.3, as those
human activities are stationary, our accelerometers are supposed to have zero reading.
But we still get different sensor reading from accelerometers, which is because of the
drift of the accelerometers. We will not use the calibrated acceleration data here, instead,
we use the drift of accelerometers as a better indicator for stationary activities. When
considering about the combination of stationary and non-stationary human activities, we
will do the calibration in order to get the correct result. But the X-axis acceleration data
will also be taken into account in the classification algorithm when classifying combined
activities. X-axis represents time, with one second as the unit, Y-axis represents
acceleration, with m/s^2 as the unit. In Figure 5.4, Hidden states are defined as
standing(S3), sitting(S2), and lying(S1). We recall that standing is defined as S6 in
proposition 1, which is different from proposition 2. The reason is that we use different
Hidden Markov Model for the hidden states decoding. With different models, hidden
states are given different symbol numbers. The hidden states are well recognized in
Figure 5.4.
In proposition 3, each subject will go through standing, walking, sitting, running,
lying, climbing stairs, and standing which is a combination of stationary and non-
stationary human activities.
36
standing-walking-sitting-running-lying-climbing stairs-standing
012345678
1 16 31 46 61 76 91 106121 136151166181 196
time(s)
Acceleration(m/s2)
toe sensor
waist sensor
leg sensor
Figure 5.5 Reluctant acceleration data for Proposition 3
Hidden States Decoding
0
1
2
3
4
5
6
7
8
1 16 31 46 61 76 91 106121 136151 166181 196211
time(s)
Hidden States
Viterbi Path
Figure 5.6 Hidden States Decoding for Proposition 3.
In Figure 5.5, we can see that the reluctant data is 0 for stationary motion. So we will
use X-axis acceleration data from leg sensor in our recognition process. In Figure 5.6,
experiment shows the accuracy of recognizing climbing stairs is not as high as walking
and running, which can be observed from the Figure 5.5, acceleration data for climbing
stairs becomes fuzzy in the middle of the climbing activity.
37
Hidden states are defined to be different symbols in different experiments because of
different Hidden Markov Models being used. In proposition 3, they are defined as
standing(S3), walking(S6), sitting(S4), running(S1), lying(S2), and climbing stairs(S5).
In Figure 5.6, at the end of running activity, subject fell down and laid on the ground,
at the same time, hidden state goes from S2 to S5, then to S3. Fall detection can be
introduced by recognizing these sudden changes.
0
20
40
60
80
100
120
standing sitting lying walking running climbingstairs
Accuracy(%)
HMM
non-HMM
Figure 5.7 Accuracy of human activities recognition with HMM and without HMM
Figure 5.7 compares the accuracy of human activities recognition with HMM and
without HMM (using collected acceleration data directly). As we can see, HMM greatly
improve the accuracy of human activities recognition rate. For stationary activities
(standing, sitting, lying), we achieve 100 percent accuracy. For non-stationary activities,
we achieve comparably lower accuracy, as acceleration data are not stable and changes
from time to time. Even though, it is still much better than using the acceleration data
directly. Raw data can hardly be used for determining what the subject is doing at
38
specific moment. The graph is plotted based on subject mounted with three sensors which
are deployed at left toe, right leg and right waist.
Accuracy is calculated as following.
Assume that we have Viterbi path generated by the HMM like this:
0 0 0 1 4 1 4 1 1 2 2 5 2 5 2 2 3 3 3. And we have real human activity like this:
0 0 0 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3. Then we notice the differences between them are:
1 2 1 2
Let’s assume that symbol 0 represents standing, 1 represents walking, 2 represents
sitting, 3 represents running. We notice that there is no difference between Viterbi path
and real human activity for symbol 0 (standing), so the accuracy is marked as 100%. As
for symbol 1, which represents walking, we have two different sequences between
Viterbi path and real human activity, so the accuracy is calculated as 1-(2/6) =66.67%. In
the previous equation, we use 6 as denominator, which is the total sequence number for
walking. In the same manner, we can get the accuracy of sitting, which is 1-(2/7)
=71.43%. The accuracy of running is 1-(0/3) =100%.
0
20
40
60
80
100
120
standing sitting lying walking running climingstairs
Accuracy(%)
using own model
using others' model
Figure 5.8 Subject using his/her own model and Subject using other trainee’s model
39
When we apply the training file of one user to get the Viterbi path of another user,
result is shown in Figure 5.8. As we can see, recognition accuracy drops dramatically for
non-stationary human activities. So we need to train the HMM specifically for each
subject in order to get optimal recognition result.
0
20
40
60
80
100
120
standing sitting lying walking running climbingstairs
Accuracy(%)
one sensor
two sensors
three sensors
Figure 5.9 Comparison of accuracy for human activities with one sensor, two sensors
and three sensors mounted on each subject
We use three sensor nodes for data collection in each subject for most of our
experiments. Figure 5.9 shows the difference among one sensor node, two sensor nodes
and three sensor nodes for data collection in human activities recognition. In the first case,
the only sensor node is mounted at the right leg of the subject. In the second case, one
sensor node is mounted at the left toe and the other is mounted at the right leg. In the
third case, the first sensor node is mounted at the left toe, the second one is mounted at
the right leg, and the third one is mounted at the right waist of subject. Three sensor
nodes contribute the highest recognition rate in all the human activities recognition, while
two sensor nodes is in the middle and one sensor node gives the lowest recognition rate.
As sensor node is not cheap, in real world application, we can use two sensor nodes as
40
their recognition rate is not significant lower than three sensor nodes. But these two
sensor nodes must be mounted on toe and leg of the subject.
Another issue needs to be considered is where to wear those sensor nodes. We choose
to compare sensor nodes mounted on toe, leg, waist, shoulder and chest. In order to
recognize other activities such as typing, writing, eating, brushing teeth, we need to put
sensor nodes in the upper body of human being. Here is a simple comparison graph
between the accuracy and position of sensors mounted on human body.
0
20
40
60
80
100
120
toe,leg,shoulder toe,leg,chest toe,leg,waist toe,waist toe,chest leg,waist leg,shoulder
Accuracy(%) standing
sitting
lying
walking
running
climbing stairs
Figure 5.10 Comparison of accuracy among sensors mounted on
different parts of human body.
As we can see from Figure 5.10, three sensors are mounted on toe, leg, waist, or toe,
leg, chest, or toe, leg, shoulder of the subject. The highest recognition rate is given by
sensors mounted on toe, leg, and waist. If we choose two sensor nodes, the highest
recognition rate is given by sensors mounted on leg and waist. The accuracy differences
between three sensor nodes are not very big, so assume we have one sensor on toe, one
sensor on leg, it does not matter where the third sensor was mounted. They give similar
recognition rate. As for two sensor nodes, one senor must be mounted on leg, it does not
matter where the second sensor node is mounted, it gives similar recognition rate. If no
sensor node is mounted at leg, then the accuracy drops rapidly.
41
From the result we discussed before, it is clear that Hidden Markov Model gives an
outstanding performance in human activities recognition.
5.2 Indoor Localization Results
5.2.1 Experiment Set-up
Several experiments were conducted in computer science department.
First experiment: Three stable nodes were deployed at different places along the lobby
at computer science department. Their coordinates are given as (0, 0, 0), (0.9, 1.2, 0) and
(0, 1.8, 0.6). Test bed is set to be a 1.8m*2.1m*1.5m area. Two target nodes were
mounted together vertically on the subject. One target node recorded the acceleration of x
axis and y axis. The other target node recorded acceleration of z axis. Both of the target
nodes had been calibrated before they were used. The experiment result shows that the
longer the distance, the worse the accuracy will be. In the first experiment, one subject
will walk diagonally from one side to the other side of the area.
The mechanism of data collection is simple. Firstly, turn on the stable nodes. They
will wait for the message sent from target nodes. Secondly, turn on the target nodes.
Thirdly, send a TOS message from base station to target nodes. After receiving the
message, both of the target nodes will record the acceleration data in a fixed time interval.
At the same time, one of the target nodes will send another TOS message which contains
the transmission power value to the stable nodes. When stable nodes receive the message
from target node, they will record the RSSI value of received message and send a TOS
message containing RSSI value and transmission power to base station. A java program
running in the base station will process the raw data for distance calculation and EKF
42
based optimization.
Figure 5.11 Two target nodes deployed vertically to record three axis acceleration data
5.2.2 Experiment Results
First experiment: One subject walks through this 1.8m*2.1m*1.2m area diagonally.
Figure 5.12 Real route coordinate and EKF implemented coordinate in a
1.8m*2.1m*1.5m area (walk diagonally)
Figure 5.12 shows the differences among coordinates calculated by Extended Kalman
Filter (represented by the green round symbol), and coordinates of real route (represented
43
by the light blue rectangle). As we can see, EKF performs well in measuring the distance
that subject traveled in three axis. The longer the subject walks, the bigger the
measurement error will be. Figure 5.13, 5.14, and 5.15 shows the detailed measurement
data for EKF in three axis.
Figure 5.13 X-axis data for EKF coordinate, and real route coordinate
Figure 5.14 Y-axis data for EKF coordinate, real route coordinate
44
Figure 5.15 Z-axis data for EKF coordinate, and real route coordinate
Figure5.13 shows the differences of x-axis coordinate among EKF and real route.
Figure 5.14 shows the differences of y-axis coordinate among EKF and real route. Figure
5.15 shows the differences of z-axis coordinate among EKF and real route.
Error differences using Distance Formular and EKF
among x, y and z axis in the 1.8m*2.1m*1.5m area
(walking straightly)
0
0.05
0.1
0.15
0.2
0.25
EKF x-axis
average error
EKF y-axis
average error
EKF z-axis
average error
Error(m)
Figure 5.16 Average error for EKF in three axis in the 1.8m*2.1m*1.5m area
From Figure 5.16, we can see that EKF implementation achieves a good accuracy in
the 1.8m*2.1m*1.5m area. EKF overcomes the drawback of inaccurate measurement of
distance between target node and stable nodes with the advantages of computing kalman
gain and introducing error covariance to reduce the system noise in the iteration process.
Second Experiment: We did experiment in the same 1.8m*2.1m*1.2m area. This time
45
subject will walk straightly instead of walking diagonally. The result is shown as below.
Figure 5.17 Real route coordinate and EKF implemented coordinate
in a 1.8m*2.1m*1.5m area (walk straightly)
Figure 5.17 shows coordinate calculated by Extended Kalman Filter (represented by
the green round symbol), and coordinate of real route (represented by the light blue
rectangle). As we can see, EKF performs well in measuring the distance that subject
traveled in three axis. We achieve high accuracy in distance measuring in X and Y axis.
For the Y-axis, the longer the subject walks, the bigger the measurement error will be.
Figure 5.18, 5.19, and 5.20 show the detailed measurement data of EKF.
46
Figure 5.18 X-axis coordinate of EKF, and real route
Figure 5.19 Y-axis coordinate of EKF, and real route
47
Figure 5.20 Z-axis coordinate of EKF, and real route
From Figure 5.18, 5.19 and 5.20, we can see that EKF gives a precise estimation of
moving subject in X-axis and Z-axis, but in Y-axis, the estimation is not as precise as X-
axis and Y-axis. The experiment result shows that if the acceleration in one axis does not
change too much when the subject is moving, then the accuracy in certain axis can be
improved. In the second experiment, subject walks straightly on Y-axis, and the
environment noise is comparably larger on Y-axis. In the EKF model, we can adjust
different parameters in order to get better measurement result. We can use different
environment noise value to optimize the measurement result. When we applied it to the
real world application, we may need to figure out how to adjust parameters from time to
time in order to get better estimation for the subject localization.
Error differences using Distance Formular and EKF
among x, y and z axis in the 1.8m*2.1m*1.5m area
(walking straightly)
0
0.05
0.1
0.15
0.2
0.25
EKF x-axis
average error
EKF y-axis
average error
EKF z-axis
average error
Error(m)
Figure 5.21 Average error for EKF in three axis in the 1.8m*2.1m*1.5m area
Figure 5.21 shows the similar result with Figure 5.16, but X-axis error has been
greatly reduced.
Third Experiment: This time we did experiment in a larger area which is
1.8m*6m*1.5m. All the mechanism being used in the third experiment is the same as in
the first and second experiment.
48
Figure 5.22 Real route coordinate, EKF implemented coordinate,
DF coordinate in a 1.8m*6m*1.5m area (walk diagonally)
Figure 5.22 shows the differences between coordinate calculated by Extended
Kalman Filter and coordinate of real route in the third experiment. As we can see, the
longer the distance, the larger error we will get, especially in X-axis and Y-axis. Z-axis
can not be well recognized in this figure but will be shown later in Figure 5.25. On Y-
axis, EKF measurement is close to the real route in a distance less than three meters.
When the distance becomes larger than three meters, the result turns out to be unstable
and goes on the wrong track. X-axis increases slower than the real route, reaching 0.86m
in the final measurement while the real route for X-axis final measurement is 1.8m. The
experiment shows that the RSSI distance measurement based target tracking is better to
be applied in distance which is less than 3 meters if we use three stable nodes and two
target nodes. If we do not need the Z-axis data, then one target node is enough to measure
the X-axis and Y-axis coordinate.
49
Figure 5.23 X-axis coordinate of EKF, and real route
Figure 5.24 Y-axis coordinate of EKF, and real route
50
Figure 5.25 Z-axis coordinate of EKF, and real route
From Figure 5.23, 5.24 and 5.25, we get a better understanding about the
performance of EKF on X-axis, Y-axis and Z-axis measurement.
Error differences using Distance Formular and EKF
among x, y and z axis in the 1.8m*6m*1.5m area
(walking diagnally)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
EKF x-axis
average error
EKF y-axis
average error
EKF z-axis
average error
Error(m) s
Error(m)
Figure 5.26 Average error for EKF in three axis in the 1.8m*6m*1.5m area
As we can see from Figure 5.26, the accuracy of X-axis and Y-axis are decreasing
sharply compared to accuracy result shown on Figure 5.21. In order to get a better
estimation result, we need to deploy more stable nodes along the route of the subject .
From experiment one, two and three, we compare three different walking activities
which are walking diagonally, walking straightly, and walking diagonally in a larger area.
We can see that EKF gives a good estimation for indoor localization when the subject
moves in a 1.8m*2.1m*1.5m area. Further movement to a longer distance such as 6
meters in Y-axis will result in inaccurate measurement result. More research on
environment noise parameters of Extended Kalman Filter is needed in order to apply
EKF in a longer distance measurement and more human activities involved (as we only
did the walking experiment in this thesis). More sensor nodes are needed to provide
larger scale subject localization as well.
51
5.3 Conclusion and Future Work
In this thesis, three different human activity series have been tested. Hidden Markov
Model has been proved to be an effective tool in human activity recognition. We
achieved a high recognition rate on both stationary and non-stationary activities. We
discuss where to deploy the sensor nodes on human body and how many sensor nodes are
enough for monitoring the activities we performed in our lab. We also show that
individual training is compulsory in order to have a better recognition rate for the subject.
Fall detection has been briefly discussed, which takes the lying activity into account.
Future work may relate to recognize more human activities, such as reading, writing,
typing, eating, jumping and so on. A more flexible mechanism needs to be introduced for
real time monitoring which has more system restriction and system noise than the
experiment conducted in our lab.
Extended Kalman Filter (EKF) is introduced in indoor localization section.
Experiment result shows that EKF can effectively improve the accuracy of sensor
measurement in a 1.8m*2.1m*1.5m area. As for the 1.8m*6m*1.5m area, error rates
from EKF grows, which may be caused by the inaccurate measurement of distance
between target nodes and stable nodes based on RSSI. Future work may relate to
introduce more sensor nodes in the experiment for larger scale localization. Besides, the
experiment we did is to get absolute coordinate of the sensor node. In the real world
application, such as indoor localization for firefighters, when firefighters are on duty, we
can not deploy stable nodes in advance in the building where they are on mission. In that
case, we need to calculate the relative coordinate for each firefighter, so that once danger
happens to one firefighter, he/she can get help from his/her companion according to the
53
REFERENCES
[1] Rafael David Kaliski “Tinyos Laboratory Develepment”
[2] T. Zimmerman, “Personal area networks: near-field intrabody communication,” IBM
systems Journal, 1996, vol. 35, no. 3 & 4, pp. 609-617.
[3] D. Malan, T. Fulford-Jones, M. Welsh, and S. Moulton. “Codeblue: An ad hoc sensor
network infrastructure for emergency medical care.” In Proc. of the International
Workshop on Wearable and Implanatable Body Sensor Networks, April 2004.
[4] M. J. Mathie, A. C. F. Coster, N. H. Lovell, and B. G. Celler: “Accelerometry:
Providing an Integrated, Practical Method for Long-term, Ambulatory Monitoring of
Human Movement”, Physiological Measurement, 25. R1-R20. 2004.
[5] Harvard Sensor Network Lab http://fiji.eecs.harvard.edu/CodeBlue
[6] Ossama Younis and Sonia Fahmy, "An Experimental Study of Routing and Data
Aggregation in Sensor Networks," In Proceedings of the International Workshop on
Localized Communication and Topology Protocols for Ad hoc Networks (LOCAN), held
in conjunction with The 2nd IEEE International Conference on Mobile Ad Hoc and
Sensor Systems (MASS-2005), November 2005.
[7] Seon-Woo Lee and Kenji Mase. “Activity and location recognition using wearable
sensors.” IEEE Pervasive Computing, 1(3):24–32, 2002.
[8] K. Takenoshita, N. Shiozawa, J. Onishi and M. Makikawa, “Development of a
Portable Acceleration Monitor Device and its clinical application for the Quantitative
Gait Assessment of the Elderly”. Proceedings of the 2005 IEEE Engineering in Medicine
54
and Biology 27th Annual Conference Shanghai, China, September 1-4, 2005
[9] Michael Sung, Carl Marci and Alex Pentland, “Wearable feedback systems for
rehabilitation,” Journal of NeuroEngineering and Rehabilitation, Vol.2, No.17, June 2005
[10] J. Mantyjarvi, J. Himberg, and T. Seppanen. “Recognizing human motion with
multiple acceleration sensors,” In Proceedings of the IEEE International Conference
on Systems, Man, and Cybernetics, pages 747–52. IEEE Press, 2001.
[11] M. Makikawa, S. Kurata, Y. Higa, Y. Araki, and R. Tokue. “Ambulatory monitoring
of behavior in daily life by accelerometers set at both-near-sides of the joint,” In
Proceedings of Medinfo, volume 10(Pt) 1, pages 840–3. 2001.
[12] B. Clarkson and A. Pentland. “Predicting daily behavior via wearable sensors,”
Technical report, Massachusetts Institute of Technology Media Laboratory, July 2001.
[13] J. B. Bussmann, W. L. Martens, J. H. Tulen, F. C. Schasfoort, H. J. van den Berg-
Emons, and H. J. Stam. “Measuring daily behavior using ambulatory accelerometry: the
activity monitor,” Behavior Research Methods, Instruments, & Computers, 33(3):349–56,
2001.
[14] G. S. Chambers, S. Venkatesh, G. A. W.West, and H. H. Bui. “Hierarchical
recognition of intentional human gestures for sports video annotation,” In International
Conference on Pattern Recognition, pages 1082–5, Quebec City, 2002.
[15] L.E. Baum and T. Petrie, “Statistical Inference for Probabilistic Functions of Finite
State Markov Chains,” Annals of Math. Statistics, vol. 37, pp. 1,554-1,563, 1966.
[16] L.E. Baum and J.A. Egon, “An Inequality with Applications to Statistical
Estimation for Probabilistic Functions of a Markov Process and to a Model for Ecology,”
Bull. Amer. Meteorology Soc., vol. 73, pp. 360-363, 1967.
55
[17] L.E. Baum, G.R. Sell, “Growth Functions for Transformations on Manifolds,”
Pacific J. Math., vol. 27, no. 2, pp. 211-227, 1968.
[18] L.E. Baum, T. Petrie, G. Soules, and N. Weiss, “A Maximization Technique
Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains,”
Annals of Math. Statistics, vol. 41, no. 1, pp. 164-171, 1970.
[19] L. Baum, “An Inequality and Associated Maximization Technique in Statistical
Estimation for Probabilistic Functions of Markov Processes,” Inequalities, vol. 3, pp. 1-8,
1970.
[20] C.F.J. Wu, “On the Convergence Properties of the EM Algorithm,” Annals of
Statistics, vol. 11, no. 1, pp. 95-103, 1983.
[21] S.E. Levinson, L.R. Rabiner, and M.M. Sondhi, “An Introduction to the Application
of the Theory of Probabilistic Functions of Markov Process to Automatic Speech
Recognition,” Bell System Technical J., vol. 62, no. 4, pp. 1,035-1,074, 1983.
[22] G. Rigoll, A. Kosmala, J. Rottland, C.H. Neukirchen, “A Comparison between
Continuous and Discrete Density Hidden Markov Models for Cursive Handwriting
Recognition,” Proc. Int'l Conf. Pattern Recognition (ICPR '96), pp. 205-209, 1996.
[23] J. Hu, M.K. Brown, and W. Turin, “HMM Based On-Line Handwriting
Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 10, pp.
1,039-1,045, Nov. 1996.
[24] L. Rabiner and B.H. Juang, “Fundamentals of Speech Recognition.” Englewood
Cliffs, N.J.: Prentice Hall, 1993,
[25] L.R. Rabiner, “A Tutorial on Hidden Markov Models and Selected Applications in
Speech Recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257-286, 1989.
56
[26] L.R. Rabiner and S.E. Levinson, “A Speaker-Independent, Syntax-Directed,
Connected Word Recognition System Based on Hidden Markov Models and Level
Building,” IEEE Trans. Acoustic, Speech, and Signal Processing, vol. 33, no. 3, pp. 561-
573, 1985.
[27] L.R. Bahl, F. Jelinek, and R.L. Mercer, “A Maximum Likelihood Approach to
Continuous Speech Recognition,” IEEE Trans. Pattern Analysis and Machine
Intelligence, vol. 5, pp. 179-190, 1983.
[28] J. Cui and Z. Sun. “Visual hand motion capture for guiding a dextrous hand.” In
Proc. of the IEEE International conference on automatic face and gesture recognition,
pages 729-734. IEEE, May 2004.
[29] R.K.Ganti, P.Jayachandran, T.F.Abdelzaher, John A. Stankovic. “SATIRE: A
Software Architecture for Smart AtTIRE,” Proceedings of the 4th international
conference on Mobile systems, applications and services Uppsala, Sweden.
[30] Ling Bao, “Physical Activity Recognition from Acceleration Data under Semi-
Naturalistic Conditions,” Master thesis, MIT
[31] M. J. Mathie, B.G. Celler, N.H. Lovell, A.C.F. Coster, “Classification of basic daily
movements using a triaxial accelerometer,” Med. Biol. Eng. Comput., 2004, 42, 679-687
[32] Korhonen, I. PARKKA, and VAN GILS, M. (2003): “Health monitoring in the
home of the future,” IEEE Eng. Med. Biol. Mag., 22, pp. 66-73
[33] Mathie M. J., Coster, A. C. E, Lovell, N. H., Celler, B. G., Lord, S. R., and
Tiedernamm, A. (2004b): “A pilot study of longterm monitoring of human movements in
the home using accelerometry,” J. Telemed. Telecare, 10, pp. 144-151
57
[34] Michael Sung, Carl Marci and Alex Pentland, “Wearable feedback systems for
rehabilitation,” Journal of NeuroEngineering and Rehabilitation, Vol.2, No.17, June 2005
[35] Z. Rong, Z. Zhaoying, Y. Ting, and S. Xufeng, “Micro-measurement system for
tracking human body motion,” Qinghua Daxue Xuebao/Journal of Tsinghua university,
43:1464-1467, 2003.
[36] Hao Wu, Guohui Tian, Bin Huang, “Multi-Robot Collaborative Localization
Methods Based on Wireless Sensor Network,” Proceedings of the IEEE International
Conference on Automation and Logistics Qingdao, China September 2008.
[37] D. Moore, J. Leonard, D. Rus, and S. Teller, “Robust distributed network
localization with noisy range measurements,” in SenSys ’04: Proceedings of the 2nd
international conference on Embedded networked sensor systems. New York, NY, USA:
ACM Press, 2004, pp. 50–61.
[38] Erin-Ee-Lin Lau, Wan-Young Chung ,Enhanced RSSI-based Real-time User
Location Tracking System for Indoor and Outdoor Environments[C], 2007 International
Conference on Convergence Information Technology, 1213-1219
[39] C. Savarese, J. Rabaey, K. Langendoen, Robust positioning algorithms for
Distributed ad hoc wireless sensor networks[C], in: Proceedings of General Track:2002
USENIX Annual Technical Conference,2002,pp.317-327
[40] V. Ramadurai and M. Sichitiu, “Localization in wireless sensor networks: A
probabilistic approach,” 2003.
[41] A. Smith, H. Balakrishnan, M. Goraczko, and N. Priyantha, “Tracking moving
devices with the cricket location system,” 2004.
58
[42] Greg Welch and Gary Bishop, “An Introduction to the Kalman Filter”, Technical
Report TR 95-041
[43] Marko Helen, Juha Latvala, Hannu Ikonen, Jarkko Nittylahti, “Using Calibration in
RSSI-Based Location Tracking System”, Proceedings of the 5th World Multiconference
on Circuits, Systems, Communications & Computers (CSCC20001)
[44] Cesare Alippi, Giovanni Vanini, “A RSSI-based and calibrated centralized
localization technique for Wireless Sensor Networks,” Proceedings of the Fourth Annual
IEEE International Conference on Pervasive Computing and Communications
Workshops (PERCOMW’06).
[45] K. Whitehouse, C. Karlof, A. Woo, F. Jiang, and D. Culler. The effects of ranging
noise on multihop localization: an empirical study. In The Fourth International
Conference on Information Processing in Sensor Networks (IPSN ’05). Los Angeles,
California, Apr. 2005.
[46] J. Latvala, J. Syrjärinne, S. Niemi, J. Niittylahti , “Patient Tracking in a Hospital
Environment Using Wireless Stations and Extended Kalman Filtering”, Proceedings of
the 1999 Middle East Conference on Networking.
[47] Flavio Cabrera-Mora, and Jizhong Xiao, “Preprocessing Technique to Signal
Strength Data of Wireless Sensor Network for Real-Time Distance Estimation,” 2008
IEEE international Conference on Robotics and Automation Pasadena, CA, USA, May
19-23, 2008
[48] A. P. Jardosh, E. M. Belding-Royer, K. C. Almeroth and S. Suri, “Real-world
environment models for mobile network evaluation,” IEEE Journal on Selected Areas in
Communications, vol. 23, No. 3, pp. 622-632, March 2005.
59
[49] J. Blumenthal, F. Reichenbach and D. Timmermann, “Minimal transmission power
vs. signal strength as distance estimation for localization in wireless sensor networks,” In
Proceedings of the 3rd IEEE International Workshop on Wireless Ad-hoc and Sensor
Networks (IWWAN), New York, NY, USA. June 2006.
[50] K. Woyach, D. Puccinelli and M. Haenggi, “Sensorless sensing in wireless networks:
implementation and measurements,” In Proceedings of the Second International
Workshop on Wireless Network Measurement (WiNMee), Boston, MA, USA. April 2006.
[51] K. Lorincz and M. Welsh, "MoteTrack: A Robust, Decentralized Approach to RF-
Based Location Tracking," Proceedings of International Workshop on Location and
Context-Awareness (LoCA 2005), May 2005.
[52] L. Hu and D. Evans, “Localization for mobile sensor networks,” in roceedings of the
10th annual international conference on Mobile computing and networking, 2004, pp.
45–57.
[53] A. Galstyan, B. Krishnamachari, K. Lerman and S. Pattem, "Distributed Online
Localization in Sensor Networks Using a Moving Target," INTERNATIONAL
SYMPOSIUM ON INFORMATION PROCESSING IN SENSOR NETWORKS, vol. 3, pp.
61-70, 2004.
[54] R. R. Brooks, P. Ramanathan, and A. Sayeed, “Distributed target tracking and
classification in sensor network,” Proceedings of the IEEE, Vol. 91, No. 8, pp. 1163-
1171, August 2003.
[55] Pritpal Dang, Prasanna Ballal, Frank Lewis, Dan Popa, Real Time Relative and
Absolute Dynamic Localization of Air-Ground Wireless Sensor Networks.
60
[56] Ali Shareef, Yifeng Zhu, and Mohamad Musavi, “Localization Using Neural
Networks in Wireless Sensor Networks,” Mobilware’08, February 12-15, 2008,
Innsbruck, Austria.
VITA Bo Xu
Candidate for the Degree of
Master of Science or Arts Thesis: HUMAN ACTIVITY RECOGNITION IN WIRELESS SENSOR NETWORK Major Field: Computer Science Biographical:
Personal Data: Born in Dongying, Shandong, China on August, 27th, 1982. Education: Completed the requirements for the Master of Science or Arts in computer science at Oklahoma State University, Stillwater, Oklahoma in May, 2009. Received B.S Degree of Electronic Engineering from University of Science and Technology Beijing, Beijing, China, 2005. Experience: Worked for China Petroleum Engineering Co. Ltd as system administrator from October, 2005 to May, 2006.
ADVISER’S APPROVAL: Xiaolin Li
Name: Bo Xu Date of Degree: May, 2009 Institution: Oklahoma State University Location: Stillwater, Oklahoma Title of Study: HUMAN ACTIVITY RECOGNITION USING BODY AREA SENSOR
NETWORKS Pages in Study: 60 Candidate for the Degree of Master of Science.
Major Field: Computer Science Scope and Method of Study: In this thesis, we model the series of human activity as Markov process. Hidden Markov Model (HMM) is used for human activities recognition. Acceleration data which are continuously collected from wearable sensor mounted on human body are imported as observation sequence for HMM. HMM is established and applied to recover the hidden states in human activity recognition. In the second part of this thesis, we apply Extended Kalman Filter (EKF) for indoor target localization. We use Received Signal Strength Indicator (RSSI) to measure the direct distance between target node and anchor nodes. The measured distance is used as the measurement function in EKF. Besides, acceleration data of target node is recorded for system input in EKF as well. In this way, we demonstrate that EKF gives reasonably accurate estimation for tracking the target node in indoor environments. Findings and Conclusions: The experimental results show that Hidden Markov Model (HMM) outperforms using acceleration data directly in human activities recognition. We also demonstrate the accuracy of mounting one, two and three sensors on human body and the accuracy of mounting sensors in different parts of human body. Extended Kalman Filter (EKF) obtains reasonable accuracy in tracking target node within a relatively small indoor environment. HMM and EKF methods can be applied to many areas such as patient monitoring, firefighter monitoring and so on.