Precise Augmented Reality Enabled by

Carrier-Phase Differential GPS

Daniel P. Shepard, Kenneth M. Pesyna, Jr., and Todd E. HumphreysThe University of Texas at Austin

BIOGRAPHIES

Daniel P. Shepard is pursing a Ph.D. in the Depart-ment of Aerospace Engineering and Engineering Me-chanics at The University of Texas at Austin, wherehe also received his B.S. in 2011. He is a member ofthe UT Radionavigation Lab. His research interestsare in GNSS security, estimation and filtering, andguidance, navigation, and control.

Kenneth M. Pesyna, Jr. is pursuing a Ph.D. in theDepartment of Electrical and Computer Engineeringat The University of Texas at Austin where he re-ceived his M.S. in 2011. He received his B.S. in Elec-trical and Computer Engineering from Purdue Uni-versity in 2009. He is a member of the UT Radionav-igation Laboratory and the Wireless Networking andCommunications Group. His research interests lie inthe areas of indoor navigation and precise low-powermobile positioning.

Todd E. Humphreys is an assistant professor in thedepartment of Aerospace Engineering and Engineer-ing Mechanics at The University of Texas at Austin,and Director of the UT Radionavigation Laboratory.He received a B.S. and M.S. in Electrical and Com-puter Engineering from Utah State University and aPh.D. in Aerospace Engineering from Cornell Univer-sity. He specializes in applying optimal estimationand signal processing techniques to problems in ra-dionavigation. His recent focus is on radionavigationrobustness and security.

ABSTRACT

A prototype precise augmented reality (PAR) systemthat uses carrier phase differential GPS (CDGPS)and an inertial measurement unit (IMU) to obtainsub-centimeter level accurate positioning and degreelevel accurate attitude is presented. Several currentaugmented reality systems and applications are dis-cussed and distinguished from a PAR system. Thedistinction centers around the PAR system’s highly

accurate position estimate, which enables tight regis-tration, or alignment of the virtual renderings and thereal world. Results from static and dynamic tests ofthe PAR system are given. These tests demonstratethe positioning and orientation accuracy obtained bythe system and how this accuracy translates to re-markably low registration errors, even at short dis-tances from the virtual objects. A list of areas forimprovement necessary to create a fully capable PARsystem is presented.

I. INTRODUCTION

The concept of virtual reality has been around sinceshortly after the advent of the computer. People havebeen fascinated with the idea of designing a world ofentirely their own creation that can be visualized, ex-plored, and interacted with. Initially, attempts at cre-ating such environments were greatly limited by thetechnology of the time. Creating a realistic-looking,real-time representation of a virtual environment thata user can interact with requires fast processors, highscreen resolution, a significant amount of memory,and large communication bandwidths. Technologicaladvances in the last couple of decades have enabledmany applications of virtual reality including immer-sive training simulations and gaming.

In recent years, the concept of augmented reality hasgarnered significant attention. While virtual realityseeks to replace the real world with a simulated one,augmented reality blends together real world and vir-tual elements typically through context relevant visu-als. One can image a continuum of perception, withthe real world on one end and virtual reality on theother. Augmented reality seeks to place the user’sperception somewhere in the middle of the contin-uum with the goal of complementing the real worldwith virtual elements.

Augmented reality applications often require knowl-edge of the platform’s position and orientation dis-play context sensitive information. When positioning

Copyright c© 2012 by Daniel P. Shepard,

Kenneth M. Pesyna, Jr., and Todd E. Humphreys

Preprint of the 2012 ION GNSS Conference

Nashville, TN, September 19–21, 2012

in a global coordinate system is required, all currentconsumer platforms running augmented reality appli-cations are only capable of obtaining the user’s loca-tion to within a few meters. This level of position-ing uncertainty severely limits potential applicationsbecause accurate alignment of the real and virtualworlds cannot be obtained at close distances.

This paper presents a prototype augmented realitysystem that is capable of obtaining sub-centimeterlevel accurate positioning and degree level accurateattitude. The paper starts with an overview of cur-rent augmented reality applications and the tech-niques used to blend the real and virtual worlds to-gether. The concept for the prototype system is thenpresented. Next, a description of the prototype sys-tem and navigation filter are given. Finally, resultsfrom tests of the prototype system are presented andremaining technical challenges are discussed.

II. AUGMENTED REALITY OVERVIEW

Augmented reality has a presence in many fields in-cluding medicine, sports, art, architecture, tourism,entertainment, and marketing. In sports, the firstdown line on a live TV broadcast of a football gameis an example of augmented reality. This technol-ogy uses a combination of visual cues from the fieldas well as the location of the cameras producing thevideo feed [1]. In marketing, augmented reality isused to help consumers make shopping decisions, suchas the augmented reality system employed in Legokiosks where the Lego product is displayed fully as-sembled on top of the box when held up in frontof a smart-phone camera [2]. This application usesvisual tags on the box to orient the product cor-rectly. Augmented reality can also be used in tourism,where augmented reality applications, as an example,can be used to translate signs from one language toanother. Fig. 1 shows the smart-phone application“Word Lens” translating a sign from Spanish to En-glish.

Augmented reality applications use techniques thatcan be classified into two primary, yet overlapping,categories: image processing and so-called pose or po-sition and orientation. Image processing techniquesare those as described in the examples above, whichuse information from the video stream to properlyorient and position computer-rendered virtual objectsinto the video feed. In many cases, fiduciary mark-

Fig. 1. A smart-phone running the Word Lens applicationthat is translating a sign from Spanish to English [2].

ers, referred to as cues, are placed in the field of viewof the camera to allow for proper alignment of thevirtual and real objects. These markers are typi-cally of known size and shape. By identifying theobserved marker of known size and shape, the ARsystem can position and orient the camera relative tothe marker. In other cases, color palettes are usedto determine colors over which virtual objects can beplaced. The first-down line in football, which must beplaced on the field and avoid being overlayed on ref-erees or players, is a good example of this technique.Green screens are another.

Pose techniques are those that employ the use of po-sition and orientation to properly register the virtualinformation onto the real world. This technique usessensor information to obtain position and orientationand properly align or “register” virtual objects withthe real world. In some setups, orientation and posi-tion of the user is done using visual cues, as with fidu-ciary markers. However, this method requires somesetup: the visual cues must be pre-placed and posi-tioned in the real environment. Another techniqueis to use the absolute user position and orientationrather than the relative position and orientation pro-vided by visual cues. Systems employing absolutepositioning techniques require the absolute positionand orientation of the user, the virtual objects, andpossibly the real objects to properly create the aug-mented reality environment. The advantage of thisapproach is that visual markers are no longer needed.

Absolute positioning techniques require a shared co-ordinate system between the user and the objects inthe environment. This shared coordinate system isoften the Earth-centered, Earth-fixed (ECEF) coor-dinate system. Augmented reality applications em-

2

Fig. 2. The Star Walk application is being used to high-light constellations in the night sky [3].

ploying absolute positioning most often use GPS forpositioning and an inertial measurement unit (IMU)for orientation. Astronomy is one application whereabsolute-positioning-based augmented reality is used.Applications like “Star Walk,” shown in Fig. 2, cre-ated by Vito Technology allow users to point theirsmart-phone or tablet at the sky and highlight con-stellations. The application uses coarse position andorientation information provided by the on-board sen-sors to calculate the section of the sky at which thedevice is pointed [3].

In the case of the “Star Walk” application, the real-world objects, the stars, are far from the user. Con-sequintly, the 3-to-10 meter positioning accuracy pro-vided by pseudorange-based GPS is adaquite for thisapplication. However, there are many applicationswhere this positioning accuracy is far from adaquite.In the construction industry, ground excavation is of-ten performed when, for example, laying the founda-tion for a new building or building a new road. Beforeexcavation begins, workers must determine the loca-tions of pre-existing subsurface pipelines and powercables. Field operatives must determine the locationof these underground utilities from maps of the under-ground infrastructure and mark their location to en-sure excavation does not damage any pipes or cables.This process can be a cumbersome task. If an aug-mented reality system capable of determining a user’slocation and attitude with a high degree of accuracyis available, then this process can be simplified by vi-sually overlaying the subsurface infrastructure mapsonto a live video feed for the field operative. Figure 3demonstrates such a system developed by a group of

Fig. 3. An image produced by an augmented reality sys-tem developed by researchers at the University of Notting-ham. This image shows subsurface utilities visualized intheir correct location relative to the real world image inthe background [4].

researchers from the University of Nottingham [4].

In applications such as infrastructure overlay, meter-level errors in user positioning are unacceptable be-cause the user needs to determine the exact locationof the virtual objects by walking up to the objects.There exists, in these situations, a need for so-called“precise” augmented reality. In these systems, userpositioning and orientation is accurate at the sub-centimeter and degree levels respectively. This posi-tioning accuracy can be obtained by what is knownas carrier-phase differential GPS, and the orientationaccuracy can be obtained with a medium-grade IMU.

III. PRECISE AUGMENTED REALITY (PAR)

Precise augmented reality (PAR) can be defined asan augmented reality system exploiting highly ac-curate position and orientation information. Thispaper presents a PAR system that makes use ofsub-centimeter level accurate positioning, enabled bycarrier-phase differential GPS (CDGPS), and degreelevel orientation, provided by a medium-grade IMU.As discussed briefly in Sec. II, existing work in thisarea consists of a system built for subsurface datavisualization [4]. This system used a CDGPS re-ceiver built by Leica Geosystems to provide the posi-tion estimate. Additionally, a group from ColumbiaUniversity built a head-mounted augmented realitysystem, shown in Fig. 4, for the general purpose of

3

Fig. 4. An augmented reality system developed by re-searchers at Columbia University for providing users withinformation about their surroundings [5].

providing users information about their surround-ings as they navigate an unfamiliar urban environ-ment [5]. This system, however, used a weaker formof correction-based precise positioning called differ-ential GPS, which produced meter-level positioningerrors.

Neither of the previously mentioned systems take ad-vantage of the gains that can be obtained by couplingthe IMU and the GPS measurements together. Thesystem presented in this paper goes beyond this priorwork by integrating the CDGPS and IMU measure-ments within an extended Kalman filter. The result-ing accurate pose estimated permits tight registrationof virtual objects.

IV. PROTOTYPE PAR SYSTEM

A prototype PAR system was designed and built todemonstrate the potential of precise augmented re-ality. A picture of the inside of the system with la-bels for the various components is shown in Fig. 5.The system is comprised of a GPS receiver, an IMU,a webcam, a single-board computer (SBC), a GPSantenna, and a lithium-ion battery. Detailed descrip-tions of the GPS receiver, IMU, and webcam used aregiven in Sec. IV-A, IV-B, and IV-C respectively. Thissensor package is strapped to the back of a tablet PC,which collects the data from the GPS receiver andIMU and the webcam footage for post-processing. In

Fig. 5. The inside of the prototype PAR system withlabeled components.

a real-time version of the system, the tablet PC wouldtake the form of a “window” into the AR environ-ment; a user looking “through” the tablet would seean augmented representation of the real world on theother side of the tablet. A picture of the fully assem-bled system is shown in Fig. 6.

A. GPS Receiver

The GPS receiver used for the prototype PAR systemwas the FOTON software-defined GPS receiver. Thework-horse of the receiver is a digital signal processor(DSP) running the GRID software receiver developedby The University of Texas at Austin and CornellUniversity. This software receiver was originally de-veloped for Ionospheric monitoring. As such, it has ascintillation robust PLL and databit prediction capa-bility, which both help to prevent cycle slips [6]. Thereceiver is also dual-frequency and currently capableof tracking L1 C/A and L2C, but only the L1 C/Asignals were used in the prototype PAR system. Ob-servables and navigation solutions can be output at a

4

Fig. 6. The assembled prototype PAR system.

configurable rate which was set to 5 Hz. The receivercommunicates with the tablet over Ethernet via theSBC.

B. IMU

The IMU used for the prototype PAR system was theMTi developed by Xsens. This IMU is a completegyro-enhanced attitude and heading reference system(AHRS). It houses four sensors, (1) a 3D magnetome-ter, (2) a 3D gyro, (3) a 3D accelerometer, and (4)a thermometer. The MTi also has a DSP running aKalman filter (KF) that determines the attitude ofthe MTi relative to the north-west-up (NWU) coor-dinate system. This KF determines attitude by in-gesting temperature-calibrated (via the thermometerand high-fidelity temperature models) magnetometer,gyro, and accelerometer measurements to determinemagnetic north and the gravity vector, which is suffi-cient for full attitude determination. This estimate oforientation is accurate to better than 2o RMS, duringdynamic operation.

In addition to providing orientation, the MTi alsoprovides access to the highly-stable, temperature-calibrated measurements. The configurable outputrate of the MTi was set to 100 Hz. In order to obtaina time stamp for the MTi data, the MTi measure-ments were triggered by the FOTON receiver whichalso reported the GPS time the triggering pulse wassent.

C. Webcam

The webcam used for the prototype PAR systemwas the FV Touchcam N1. The Touchcam N1 is anHD webcam capable of outputting 720P at 22 fps orWVGA at 30 fps. The Touchcam N1 also has a wideangle lens with a 78.1o horizontal field of view.

V. PAR EKF

The goal of the navigation filter for this PAR sys-tem was to obtain sub-centimeter level accurate posi-tioning and degree level accurate attitude of the we-bcam. The IMU chosen for the prototype PAR sys-tem already provided attitude to the desired accuracythrough its own Kalman filter. Therefor, the filterdesigned here only needed to satisfy the positioningrequirement. For a first prototype, it was assumedthat the attitude and angular velocity provided bythe IMU were perfect for purposes of designing thefilter. Although this assumption is inaccurate, the ef-fect of errors in attitude and angular velocity on theestimate of position is negligibly small for this system.

The navigation filter designed for this PAR systemwas a tightly-coupled INS/GPS enhanced Kalman fil-ter (EKF) implemented as a square-root informationfilter. A block diagram of this filter is shown in Fig. 7.The filter can be divided into three primary compo-nents: (1) CDGPS measurement update, (2) iner-tial navigation system (INS) propagation step, and(3) augmented reality overlay. These primary com-ponents are described in detail in the sections thatfollow.

The state for the EKF is

~X =

[

~x~N

]

=

~r

~r~b~N

(1)

where ~X is the state vector, ~x is the real-valued partof the state vector, ~N is the carrier phase integer am-biguity vector, ~r and ~r are the ECEF position andvelocity vectors of the IMU respectively, and ~b is theaccelerometer bias vector. This state differs from thatof the GPS antenna and the webcam lens in positionand velocity due to the distance between the systemcomponents and the rigid-body rotation of the entire

5

Fig. 7. A block diagram of the PAR EKF. The blocks corresponding to the INS propagation step (red), CDGPSmeasurement update (blue), and augmented reality overlay (green) are highlighted in the figure.

system. This is accounted for through the followingtransformation for the GPS antenna state:

~rGPS = ~r +RIMUECEF~rGPS/IMU

~rGPS = ~r +RIMUECEF

(

~ω × ~rGPS/IMU

)(2)

where ~rGPS and ~rGPS are the ECEF position andvelocity vectors of the GPS antenna respectively,RIMU

ECEF is the rotation matrix from the IMU frameto the ECEF frame that is derived from the quater-nion output by the IMU and the current best positionestimate of the IMU, and ~rGPS/IMU is the positionvector of the GPS antenna relative to the IMU in theIMU frame. To obtain the webcam lens position andvelocity, replace ~rGPS , ~rGPS, and ~rGPS/IMU in Eq. 2

with ~rCAM , ~rCAM , and ~rCAM/IMU . In order for theEKF to be accurate to the sub-centimeter level, therelative position vectors of the components must beknown to millimeter-level accuracy.

A. CDGPS Measurement Update

The CDGPS measurement update is responsible fordetermining the accurate position of the system andcorrecting for IMU drift. The CDGPS algorithm re-quires observables measurements (carrier phase andpseudorange) from two separate GPS receivers. Thefirst receiver is a reference or base-station receiverthat is located at a pre-surveyed position. The sec-ond receiver is the mobile receiver, which is located onthe PAR system. The location of the base station re-ceiver’s antenna must be known with high accuracy,since the CDGPS algorithm only provides an accu-rate position of the mobile receiver’s antenna relativeto the base station receiver’s antenna.

A.1 Measurement Model

The CDGPS algorithm uses single frequency (L1C/A) double-differenced measurements of carrierphase and pseudorange to determine position to sub-

6

centimeter level accuracy. These double-differencedmeasurements are formed as follows:

single-differenced measurements: The pseudorange,ρi(k), and carrier phase, φi(k), measurements fromeach receiver taken at the same instant in time arematched up by satellite and differenced between thetwo receivers to obtained single-differenced pseudor-ange and carrier phase measurements. These single-differenced pseudorange and carrier phase measure-ments are denoted as

∆ρiAB(k) = ρiA(k)− ρiB(k)

∆φiAB(k) = φi

A(k)− φiB(k)

(3)

The subscripts A and B represent the base stationand mobile receivers respectively. Assuming the basestation and mobile receivers are relatively close to oneanother (within about 10 km) and the measurementswere taken at roughly the same time, these single-differenced measurements have removed the errors inthe measurements that are common to the satellite.These errors are: (1) Ionospheric and Troposphericdelays, (2) satellite clock errors, and (3) the initialbroadcast carrier phase of the signal.

double-differenced measurements: Of the satellitestracked by both receivers, one satellite is chosen asthe “reference” satellite (satellite 0). The single-differenced pseudorange and carrier phase measure-ments corresponding to the reference satellite are sub-tracted from all the other single-differenced pseu-dorange and carrier phase measurements to cre-ate double-differenced pseudorange and carrier phasemeasurements. These double-differenced pseudor-ange and carrier phase measurements are denoted as

∇∆ρi0AB(k) = ∆ρiAB(k)−∆ρ0AB(k)

∇∆φi0AB(k) = ∆φi

AB(k)−∆φ0

AB(k)(4)

These double-differenced measurements have re-moved the effect of receiver clock bias and, if the re-ceiver is designed properly, the ambiguities on the car-rier phase measurements have now become integers.These integer ambiguities can be determined muchfaster than their real-valued counterparts through theuse of the integer constraint.

A detailed derivation of the double-differenced mea-surement model for the carrier-phase can be found inRef. [7]. The double-differenced pseudorange model isderived in the same way with the ambiguity term re-moved. The double-differenced measurement modelsare given as

∇∆ρi0AB(k) = ∇∆ri0AB(k) +∇∆νρ

λL1∇∆φi0AB(k) = λL1N

i0AB +∇∆ri0AB(k) +∇∆νφ

(5)

where ∇∆νρ and ∇∆νφ are the white Gaussian mea-surement noises, λL1 is the wavelength of the GPS L1center frequency, N i0

AB is the integer ambiguity, and

∇∆ri0AB(k) =(

‖~r iSV (k)− ~rA‖ − ‖~r i

SV (k)− ~rB(k)‖)

−(

‖~r 0

SV (k)− ~rA‖ − ‖~r 0

SV (k)− ~rB(k)‖)

(6)

where ~r iSV (k) is the ECEF position of satellite i, ~rA is

the location of the base station receiver, and ~rB(k) =~rGPS(k).

The measurement model in Eq. 5 is linearized aboutthe a priori state to form the linearized measurements

zi0ρ (k) = ∇∆ρi0AB(k)−∇∆r i0AB(k)

=(

r0B(k)− riB(k))T

δ~r(k) +∇∆νρ

zi0φ (k) = λL1∇∆φi0AB(k)−∇∆r i0

AB(k)

=(

r0B(k)− riB(k))T

δ~r(k) + λL1Ni0AB

+∇∆νφ

(7)

where zi0ρ (k) and zi0φ (k) are the linearized double-differenced pseudorange and carrier phase measure-ments, ∇∆r i0

AB(k) is the a priori expected double-differenced range given by Eq. 6 using ~rB(k) in placeof ~rB(k), and riB(k) is the unit vector pointing fromthe mobile receiver to satellite i.

A.2 Matrix Equation Formation

If both receivers are simultaneously tracking the sameM+1 satellites, then M linearized double-differenced

7

measurements are obtained at each time step of theform given in Eq. 7. Gathering these M equationsinto matrix form yields

[

~zρ(k)~zφ(k)

]

=

[

Hρx(k) 0Hφx(k) HφN

] [

δ~x(k)~N

]

+

[

∇∆~νρ∇∆~νφ

](8)

where δ~x(k) is the a posteriori correction to the real-valued component of the state and

Hρx(k) = Hφx(k)

=

(

r0B(k)− r1B(k))T

0 0...

......

(

r0B(k)− rMB (k))T

0 0

(9)

HφN = λL1

1 0 · · · 00 1 · · · 0...

. . ....

0 · · · 1

(10)

Assuming that pseudorange and carrier phase mea-surement noise for each satellite and receiver is inde-pendent and identically distributed, the measurementcovariance matrices are easily derived as

Rρ = σ2

ρ

4 2 · · · 22 4 · · · 2...

. . ....

2 · · · 4

= R−Tρ R−1

ρ

Rφ = σ2

φ

4 2 · · · 22 4 · · · 2...

. . ....

2 · · · 4

= R−Tφ R−1

φ

(11)

where σρ and σφ are the standard deviation of thepseudorange and carrier phase measurements respec-tively and Rρ and Rφ are the cholesky factorizationsof the covariance matrices.

The state and state covariance are encoded as a set ofequations in a square-root information filter. Theseequations are given as

[

~zx(k)

~zN (k − 1)

]

=

[

Rxx(k) RxN (k)

0 RNN (k − 1)

]

×

[

~x(k)~N

]

+ ~wX

(12)

where ~wX is distributed as a multi-dimensional stan-dard normal distribution, the state is determinedfrom these equations through the solution processdescribed in Sec. V-A.3, and the state covariance isgiven by

P (k) =

[

Rxx(k) RxN (k)

0 RNN (k − 1)

]−T

×

[

Rxx(k) RxN (k)

0 RNN (k − 1)

]−1(13)

The square-root information equations from Eq. 12are appended to the measurement equations fromEq. 8 to incorporate the a priori information aboutthe state into the solution process. The augmentedupdate equations are normalized by the noise covari-ance matrix and given as

~zρ(k)

~zφ(k)δ~zx(k)~zN (k)

=

Rρ~zρ(k)

Rφ~zφ(k)~zx(k)− Rxx(k)~x(k)

~zN (k − 1)

=

Hρx(k) 0

Hφx(k) HφN

Rxx(k) RxN (k)0 RNN (k)

[

δ~x~N

]

+ ~w

=

RρHρx(k) 0

RφHφx(k) RφHφN

Rxx(k) RxN (k)

0 RNN (k − 1)

×

[

δ~x~N

]

+ ~w

(14)

where ~w is distributed as a multi-dimensional stan-dard normal distribution. The matrix coefficient ofthe state vector is then factorized into an orthonor-mal matrix and an upper-right triangular matrix viaQR factorization. This results in

8

Hρx(k) 0

Hφx(k) HφN

Rxx(k) RxN (k)0 RNN (k)

= Q(k)

Rxx(k) RxN (k)

0 RNN (k)0 0

(15)

The measurement equations are transformed usingthe orthonormal matrix from the factorization suchthat the measurement equations become

δ~zx(k)

~zN (k)~zr(k)

= QT (k)

~zρ(k)

~zφ(k)δ~zx(k)~zN (k)

=

Rxx(k) RxN (k)

0 RNN (k)0 0

[

δ~x~N

]

+ ~w

(16)

A.3 Solution Procedure

To determine the optimal set of integer ambiguities,an integer least-squares algorithm is applied. Integerleast-squares algorithms solve the problem of mini-mizing ‖~zN (k)−RNN (k) ~N‖ under the constraint thatthe solution, ~Nopt, be a set of integers. Two popu-lar algorithms for solving integer least-squares prob-lems are the LLL algorithm, described in Ref. [8], andthe LAMBDA method, described in Ref. [9]. TheLLL algorithm was chosen for two reasons. First, atight, easy-to-implement lower bound on the proba-bility that the integers were fixed correctly was read-ily available, as presented in Ref. [8]. Second, a soft-ware package to perform the LLL algorithm, calledMILES [10], was readily available.

Once the optimal integer ambiguity set has been de-termined using MILES, the a posteriori correction tothe real-valued component of the state is determinedas

δ~x(k) = R−1

xx (k)(

δ~zx(k)− RxN (k) ~Nopt

)

(17)

The a posteriori state covariance matrix can also becomputed as

P (k) =

[

Rxx(k) RxN (k)

0 RNN (k)

]−T

×

[

Rxx(k) RxN (k)

0 RNN (k)

]−1(18)

B. INS Propagation Step

The INS propagation step is responsible for propa-gating the state forward in time using the IMU’s ac-celerometer measurements and attitude estimate. Asmentioned previously, the attitude estimate providedby the IMU is assumed to be perfect for the purposesof this prototype system. This assumption was madeto simplify development of the EKF and does not sig-nificantly effect the performance of the system due tothe high accuracy of the IMU’s attitude estimate.

Since the IMU relies on a magnetometer to determineheading, the IMU must be given the magnetic decli-nation. This is accomplished by initializing the IMUwith a rough estimate of position, which it uses in amagnetic field model to determine declination. If thesystem stays within a few kilometers of the initial-ization point, then this value will remain reasonablyaccurate due to the slow spacial variation of the mag-netic field.

B.1 Dynamics and Measurement Models

The propagation step is essentially a double integra-tor of the accelerometer measurements after removingspecific forces or accelerations the IMU senses otherthan motion in the ECEF frame. These other sensedcomponents of the accelerometer measurements arethe specific normal force and the Coriolis accelera-tion due to rotation of the Earth. The accelerometermay also have small biases in its measurements dueto electrical biases in the circuitry. These biases willbe modeled as a random walk process. This model forthe accelerometer errors is similar to that presentedin Ref. [11]. This yields the following continuous-timedynamics and measurement model

9

d

dt~r = ~r

d

dt~r = RIMU

ECEF

(

~f −~b)

+RENUECEF

00−g

− 2~ωE × ~r + ~va

d

dt~b = ~vb

d

dt~N = 0

(19)

where ~f is the measured specific force vector (i.e.accelerometer measurement), RENU

ECEF is the rotationmatrix from the east-north-up (ENU) frame to theECEF frame that is derived from the a priori positionestimate of the IMU, g is the gravitational accelera-tion of Earth at the location of the IMU, ~ωE is theangular velocity vector of the Earth, and ~va and ~vbare zero-mean, white noise processes with standarddeviations of σa and σb respectively.

Equation 19 is discretized using an Euler integrationapproximation to determine the state transition ma-trix, which assumes a small time step between up-dates. Since the IMU reports measurements every0.01 s, this approximation is reasonable. The inte-ger ambiguity vector does not change with time (i.e.~N(k+1) = ~N(k)), so its propagation equation is triv-ial and thus eliminated from the propagation step.The discrete-time propagation model is given in ma-trix form as

~x(k + 1) =

~r(k + 1)

~r(k + 1)~b(k + 1)

=

I I∆t 00 I − 2 [~ωE×]∆t −RIMU

ECEF∆t

0 0 I

×

~r(k)

~r(k)~b(k)

+

0

RIMUECEF

~f + ~gECEF

0

∆t+ ~v′

= F (k)~x(k) + ~u(k) + ~v ′

(20)

where [~ωE×] is the cross-product-equivalent matrixfor the angular velocity of the Earth, ~gECEF =(

[

0 0 −g] (

RENUECEF

)T)T

, and ~v ′ is the process

noise vector. It can be shown that the process noisecovariance matrix is

Q =

1

3σ2a∆t3I 1

2σ2a∆t2I 0

1

2σ2a∆t2I I

(

σ2a∆t+ 1

2σ2

b∆t2)

−1

2RIMU

ECEFσ2

b∆t2

0 −1

2RIMU

ECEFσ2

b∆t2 σ2

b∆tI

(21)

B.2 Square-Root Information Propagation

For use in the EKF, the discrete-time propagationmodel in Eq. 20 is converted into square-root informa-tion form. Performing the propagation in this formaids in preserving the information relating the realand integer portions of the state through the square-root information equations given in Eq. 12. Start-ing with the square-root information equations, thediscrete-time propagation model is used to substitutethe state at time k+1 for the state at time k. The re-sulting equations, after augmenting with the square-root equation for the process noise, are

0~zx(k) +Rxx(k)F

−1(k)~u(k)~zN (k)

=

Rvv 0 0−Rxx(k)F

−1(k) Rxx(k)F−1(k) RxN (k)

0 0 RNN (k)

×

~v ′

~x(k + 1)~N

+ ~wv,X

(22)

where RTvv is the Cholesky factorization of the process

noise covariance matrix and ~wv,X is distributed as amulti-dimensional standard normal distribution. Aswith the CDGPS measurement update, the matrixcoefficient of the state vector is then factorized intoan orthonormal matrix and an upper-right triangularmatrix via QR factorization. This results in

10

Rvv 0 0−Rxx(k)F

−1(k) Rxx(k)F−1(k) RxN (k)

0 0 RNN (k)

= Q(k)

Rvv(k + 1) Rvx(k + 1) RvN (k + 1)0 Rxx(k + 1) RxN (k + 1)0 0 RNN (k + 1)

(23)

The propagation equations are transformed using theorthonormal matrix from the factorization such thatthey become

~zv(k + 1)~zx(k + 1)~zN (k + 1)

=

QT (k)

0~zx(k) +Rxx(k)F

−1(k)~u(k)~zN (k)

=

Rvv(k + 1) Rvx(k + 1) RvN (k + 1)0 Rxx(k + 1) RxN (k + 1)0 0 RNN (k + 1)

×

~v ′

~x(k + 1)~N

+ ~wv,X

(24)

The a priori state and state covariance can be de-termined in the same way as described in Sec. V-A.3. The propagated square-root information equa-tions are passed on to the next operation, which couldbe a CDGPS measurement update or another IMUpropagation step depending on what measurementsare next. Under the current configuration of the PARsystem, GPS observables are reported every 0.2 s andIMU measurements and attitude are reported every0.01 s. This leads to 20 sequential propagation stepsbeing performed between every measurement update.

C. Augmented Reality Overlay

The augmented reality overlay is responsible for tak-ing the precise position and attitude of the webcamprovided by the EKF, capturing the webcam’s view ina virtual world referenced to real world coordinates,and overlaying the view of the virtual world on topof the webcam footage, which constitutes the user’s

view of the real world. This combination of real worldvideo footage and virtual images represented in theirassigned real-world location and attitude comprisesthe augmented reality experience. The virtual worldcould contain any objects or text for any purpose,which makes the PAR system application ambivilous.

For the prototype PAR system, MATLAB’s SIMULINK3D Animation Toolbox was employed to create andvisualize the virtual world and perform the overlay ofthe view of the virtual world on top of the real worldfootage. The overlay was performed by adding anappropriately sized, positioned, and oriented box inthe virtual world that acted as a “projection screen”for the webcam footage, which was applied to thebox as a texture. While this method for performingthe image overlay has its drawbacks when it comesto adding lighting and shadowing effects to the vir-tual world, the method worked well for the prototypePAR system, which did not incorporate these effects.An example of this overlay is shown in Fig. 8. Theimage on the left side of the figure shows the view ofa virtual piston before the overlay, and the image onthe right side shows the result of the overlay.

VI. TEST RESULTS

The prototype augmented reality system was testedunder two different scenarios. The first test was astatic test that was meant to verify that the CDGPSalgorithm was performing as expected. The secondtest was a dynamic test that was meant to test thefull functionality of the system. For ease of testing,all processing was done after-the-fact by operatingon recordings of the IMU data, GPS observables, andwebcam video.

A. Static Test

Before the start of the static test, the baseline dis-tance between the base station antenna and the sec-ondary antenna used for the static test was measuredusing a tape measure, as seen in Fig. 9. This mea-surement provided the truth data for the static testto verify that the CDGPS algorithm was convergingcorrectly. The baseline distance measured was 21.155m.

30 minutes of data was collected and processed for thestatic test. The base station and secondary receiver

11

Fig. 8. An example of the overlay of the virtual world and real world footage. The image on the left side shows the viewof a virtual piston before the overlay. The image on the right side shows the result of the overlay.

tracked between 8 and 10 common satellites at vari-ous points during the 30 minute interval, as shownin Fig. 10. The dropping and adding of satellitesthroughout the dataset demonstrates the CDGPS al-gorithm’s capability to operate through changes inthe satellites tracked without any issues. This wasenabled by special modifications that the CDGPS al-gorithm makes to the square-root information equa-tions whenever a satellite is lost or acquired.

Figure 11 shows a plot of the lower bound on the prob-ability that the integer ambiguities have been fixedcorrectly during the first portion of the static test. Itcan be assumed with high certainty that the integershave converged correctly once this probability has ex-ceeded 0.999, which corresponds to less than a 0.1%probability of error. Using this metric, the correctinteger ambiguities were determined within the first11.2 s of the dataset.

Figure 12 shows the position estimate of the sec-ondary antenna in the ENU coordinate system cen-tered on the base station antenna from the time thatthe probability metric was exceeded to the end ofthe test. Almost all of the points in the plot fitwithin a centimeter range in both East and North di-rections. The baseline distance was calculated fromthese points and is shown in Fig. 13. The mean base-line distance was 21.1587 m, which is within a fewmillimeters of the value measured with a tape mea-sure. This demonstrates that the CDGPS algorithmis performing as expected.

B. Dynamic Test

The data for the dynamic test was collected over afew minutes. For the first 2–3 minutes, the prototypesystem was stationary on a ledge to allow the integerambiguities to converge. Then, the prototype systemwas walked toward and around a virtual object, whichwould later be visualized in post-processing. Finally,the system was returned to its starting position onthe ledge.

Figure 14 shows the satellites tracked by both thebase station and mobile receivers during the entiredataset. In this case, the same 7 satellites weretracked throughout the dataset due to the short du-ration of the test. Figure 15 shows the lower boundprobability on correct convergence of the integer am-biguities. In this case, the probability of correct res-olution exceeded the 0.999 metric after 31.8 s.

Figure 16 shows the position estimate of the mobileantenna in the ENU coordinate system centered onthe base station antenna from the time that the prob-ability metric was exceeded to the end of the test.The position estimates are good enough that the wob-bles in the trajectory as each step was taken can beclearly seen in the plot. It can also be seen that theposition estimate did not drift during the test, sincethe position estimate at the start and end of the testare nearly identical. The IMU’s orientation estimatethroughout the test is shown in Fig. 17 as a roll-pitch-yaw Euler angle sequence. Due to the orientation ofthe IMU in the prototype system, the roll-pitch-yawsequence if one were to hold the system level to theground and point the camera to the North would bea 90o roll, 0o pitch, and -90o yaw.

12

Fig. 9. Picture of measuring the baseline distance for thestatic test using a tape measure.

Fig. 10. The SVIDs of the GPS satellites tracked duringthe static test as a function of time.

Fig. 11. The lower bound on the probability that theinteger ambiguities have been fixed correctly as a functionof time for the static test.

Fig. 12. A plot of the East and North position relative tothe base station during the static test.

A virtual world was designed to demonstrate the aug-mented reality overlay portion of the system. Thisvirtual world contained a set of blue marble columnsthat the user passed through twice during the testand a model of a piston that the user walks up to,around, and then backs away from. Figure 18 shows asequence of frames taken from the augmented realityvideo that was produced using the described virtualworld, the webcam video, and the output position andattitude of the webcam from the EKF. The renderingof the virtual world was incredibly stable relative tothe real world and contained little so-called registra-tion error.

13

Fig. 18. A sequence of frames taken from the augmented reality video produced using the results from the dynamic test.The first frame (upper left) was taken from shortly after the system first started moving. The second frame (upper right)was taken from the moment the system reaches the southern-most point of its walk around the piston. The third frame(lower left) was taken from the moment the system reaches the northern-most point of its walk around the piston. Thefourth frame (lower right) was taken from near the end of the dataset.

Fig. 13. A plot of the measured baseline distance over the30 minute static test. The red line is the mean distanceand the green lines mark the ±1σ bounds.

Fig. 14. The SVIDs of the GPS satellites tracked duringthe dynamic test as a function of time.

14

Fig. 15. The lower bound on the probability that theinteger ambiguities have been fixed correctly as a functionof time for the dynamic test.

Fig. 16. A plot of the East and North position relative tothe base station during the dynamic test.

VII. CHALLENGES / FUTURE WORK

This section provides a brief description of some ofthe technical difficulties associated with translatingthis demonstration and prototype PAR system into aviable commercial product.

A. Carrier Phase Cycle Slips

The integer ambiguities solved for by the CDGPS al-gorithm are only good so long as the GPS receiver’sPLL is able to track the signal without slipping anycycles in the carrier phase. These cycle slips can occurdue to receiver dynamics, low carrier-to-noise ratio,

Fig. 17. A plot of the Euler angles reported by the IMUduring the dynamic test. The IMU coordinate system isaligned such that the x-axis points to the right and they-axis points up as seen by a user holding the system.

Ionospheric changes, and multipath. If the receiverslips a cycle, then the CDGPS algorithm would nolonger report the correct position unless this cycleslip is detected and mitigated. Properly designed re-ceivers will experience few cycle slips, but cycle slipscannot be completely eliminated. Research has beendone on this topic in Ref. [7], but further research iswarranted for this particular application of cycle slipdetection and mitigation.

B. Carrier Phase Wind-up

Carrier phase wind-up is a phenomenon caused bythe circular polarization of the signal. If the GPSantenna is rotated, then the received carrier phase ofeach signal will increase or decrease, depending onthe direction of rotation and polarization of the sig-nal, by an amount proportional to the angle rotated.This will lead to errors in the carrier phase measure-ments that will drift in time in the case of humanmotion. However, the IMU keeps track of the ori-entation of the PAR system, so, given a model forcarrier phase wind-up based on antenna orientation,the PAR system should be able to eliminate the ef-fects of carrier phase wind-up. One such model forcarrier phase wind-up is presented in Ref. [12].

15

C. GPS Reception

The problem of GPS signal reception encom-passes three different problems, antenna design, self-interference, and indoor and deep-urban navigation.Incorporating a GPS antenna into a small device withgood phase properties and reception gain pattern is adifficult challenge that poses a major obstacle to inte-grating CDGPS capability into a device like a smart-phone. One possible solution to this problem is to ac-cept that the system would have to be either a stand-alone device or a slightly larger attachable accessory.Self-interference refers to the problem of isolating theGPS antenna from any potential interference sourceswithin the PAR system itself. Oftentimes clocks, pro-cessors, and displays transmit noise in the GPS bandthat, in close proximity, can cause issues with GPSreception.

Indoor and deep-urban navigation also pose signifi-cant challenges to satellite positioning due to the nec-essarily weak signals [13]. The surveying communitydeals with deep-urban environments by using mul-tiple satellite constellations and frequencies, but thisfunctionality comes with a significantly elevated pricetag. Due to this problem, any PAR system wouldlikely only be able to function in areas with a clearview of the sky. This constraint, however, does notprevent use of a PAR system in many potential ap-plications.

VIII. CONCLUSIONS

Precise augmented reality (PAR) will open the doorfor a whole host of augmented reality applicationsby enabling alignment of the virtual and real worldsat short distances (< 100 m) without requiring vi-sual cues. Most current augmented reality systemsare only capable of positioning accuracies in the 3-to-10 meter range, while a PAR system is capable ofcentimeter and even sub-centimeter level accuracies.Only the absolute positioning accuracy of a PAR sys-tem is capable of reducing the alignment or “regis-tration” errors to the level required for small relativedistances between the user and virtual objects with-out visual cues.

A prototype PAR system was designed, built, andtested in both static and dynamic scenarios. Thestatic test demonstrated that the CDGPS algorithmwas providing sub-centimeter level accurate position-

ing, as expected. The dynamic test demonstrated theaccuracy of the system in both position and orienta-tion as it was walked around a virtual object. Therendering of the virtual object was remarkably stablein position and orientation relative to the real worldcamera feed.

While the results of these tests are promising, thereare still several issues that need to be resolved to makethe algorithm’s performance robust. These issues in-clude cycle slip detection and mitigation and carrierphase wind-up. GPS reception is also an obstaclewhich inhibits performance of the system. This issuemanifests itself in several ways including antenna de-sign, self-interference, and the difficulty of indoor anddeep-urban navigation. These issues are not likely tobe insurmountable, with the possible exception of in-door and deep-urban navigation. However, there arestill many potential applications where sky visibilityis not an issue that would benefit from a PAR system.

References

[1] Sportsvision, “1st and ten line system,” http://www.sportvision.com/foot-1st-and-ten-line-system.html, 2012,[Online; accessed 27-September-2012].

[2] Lego, “Lego augmented reality kiosks heading to shopsworldwide,” 2010.

[3] V. Technology, “Start walk,” http://vitotechnology.com/star-walk.html, 2012, [Online; accessed 28-September-2012].

[4] G. Roberts, A. Evans, A. Dodson, B. Denby, S. Cooper,R. Hollands et al., “The use of augmented reality, gps andins for subsurface data visualization,” in FIG XXII Inter-

national Congress, 2002, pp. 1–12.[5] S. Feiner, B. MacIntyre, T. Hollerer, and A. Webster, “A

touring machine: Prototyping 3d mobile augmented re-ality systems for exploring the urban environment,” Per-

sonal and Ubiquitous Computing, vol. 1, no. 4, pp. 208–217, 1997.

[6] B. O’Hanlon, M. Psiaki, S. Powell, J. Bhatti, T. E.Humphreys, G. Crowley, and G. Bust, “CASES: A smart,compact GPS software receiver for space weather monitor-ing,” in Proceedings of the ION GNSS Meeting. Portland,Oregon: Institute of Navigation, 2011.

[7] S. Mohiuddin and M. L. Psiaki, “High-altitude satellite rel-ative navigation using carrier-phase differential global po-sitioning system techniques,” Journal of Guidance, Con-

trol, and Dynamics, vol. 30, no. 5, pp. 1628–1639, Sept.-Oct. 2007.

[8] A. Hassibi and S. Boyd, “Integer parameter estimation inlinear models with applications to GPS,” IEEE Transac-

tions on Signal Processing, vol. 46, no. 11, 1998.[9] M. Psiaki and S. Mohiuddin, “Global positioning system

integer ambiguity resolution using factorized least-squarestechniques,” Journal of Guidance, Control, and Dynamics,vol. 30, no. 2, pp. 346–356, March-April 2007.

[10] X. W. Chang, X. Xie, and T. Zhou, MILES: MATLAB

package for solving Mixed Integer LEast Squares prob-

16

lems, 2nd ed., http://www.cs.mcgill.ca/chang/software.php, Oct. 2011.

[11] W. S. Flenniken IV, J. H. Wall, and D. M. Bevly, “Char-acterization of various imu error sources and the effect onnavigation performance,” in Proceedings of the ION ITM.Long Beach, California: Institute of Navigation, 2005.

[12] J. Wu, S. Wu, G. Hajj, W. Bertiguer, and S. Lichten,“Effects of antenna orientation on gps carrier phase mea-surements,” Manuscripta Geodaetica, pp. pp. 91–98, 1993.

[13] G. Seco-Granados, J. Lopez-Salcedo, D. Jimenez-Banos,and G. Lopez-Risueno, “Challenges in indoor global nav-igation satellite systems: Unveiling its core features insignal processing,” Signal Processing Magazine, IEEE,vol. 29, no. 2, pp. 108 –131, march 2012.

17