High-AssuranceSPIRAL

End-to-end Guarantees for Robotand Car Control

Franz Franchetti, Tze Meng Low, StefanMitsch, Juan Paolo Mendoza, Liangyan Gui,Amarin Phaosawasdi, David Padua, SoummyaKar, Jose Moura, Mike Franusich, Andre Platzer,Jeremy Johnson, Manuela Veloso

Cyber-physical systems (CPS) ranging fromcritical infrastructures such as power plants, tomodern (semi) autonomous vehicles are systemsthat use software to control physical processes.CPS are made up of many different computa-tional components. Each component runs itsown piece of software that implements its con-trol algorithms, based on its model of the envi-ronment. Every component then interacts withother components through the signals and valuesit sends out. Collectively, these components, andthe code they run, drives the complex behaviorsmodern society have come to expect and rely on.Due to these intricate interactions between com-ponents, managing the hundreds to millions linesof software to ensure that the system, as a whole,performs as desired can be often unwielding.

In this article, an approach towards tamingpart of the complexity is described. The ap-proach utilizes intrinsic multi-modal redundan-cies to detect brewing problems, provides formalguarantees for control algorithms, and automatesthe software production to implement these algo-rithmic ideas with guaranteed correctness.

Specifically, the approach addresses the prob-lem from three directions: 1) The desired behav-ior of the system and assumptions about the en-vironment is described formally (in differentialdynamic logic), and proven, over all valid execu-tions, to perform correctly based on the formal-ized assumptions. 2) High performance monitorsoftware that checks that the environment com-ply with the specified assumptions, and a proof

that its implementation is a faithful representa-tion of the mathematical specifications, are au-tomatically synthesized from the differential dy-namic logic model to reduce and even eliminatehuman coding error. 3) Side channel informationsuch statistical noises are fused with traditionalsensor inputs such as Global Positioning System(GPS), based on fundamental analytical redun-dancy, so as to establish that the inputs to thesystem (i.e. sensor readings) do not contradictthe known physics of the system.

This approach has been demonstrated on botha remote controlled unmanned research groundrobot, called the Landshark, and on an American-built car. In these demonstrations, the combina-tion of formal methods for hybrid systems, au-tomatic and provably correct code generation,and side-channel redundancy has been shown todetect and defend against GPS spoofing (ma-nipulating of the GPS signal to make the vic-tim believe to be at a different position), whileprotecting the car and robot from being driveninto known obstacles. More importantly, theseconcepts are applicable in the CPS arena be-yond unmanned ground vehicles or modern cars.Other domains for which the approach haveshown applicability includes system componentslike pumps in power plants, and control of un-manned aerial vehicles.

OverviewTwo formal systems are combined to provideend-to-end correctness guarantees from the con-trol algorithm/physical model level down to thedeployed implementation of the control algo-rithm. One or both systems formalize controlapproaches or self-consistency checks to ensuresystem safety. In either case, engineers then syn-thesize the final deployable software from a highlevel specification, and have guarantees that thesynthesized software is correct and efficient.

The top level system is KeYmaera X [1], atheorem prover for differential dynamic logic [2].With KeYmaera X it is possible to prove that afamily of controllers, applied to a cyber-physicalsystem with a given physical model will behavein a certain way. An example of a safety propertythat KeYmaera X can prove is that a robot witha controller based on the dynamic window controlapproach [3] will not hit an obstacle or anotherrobot [4] (which is called passive safety). The

1

Carnegie Mellon Carnegie Mellon Carnegie Mellon

HA SPIRAL Code Synthesis

KeYmaera X Hybrid Theorem

Prover

performance

QED.

QED.

Coq Proof Assistant

Physics model

KeYmaera X Hybrid System

Theorem Prover Monitor equation

Sφnx KeYmaera X to HA Spiral

HA SPIRAL formal compilation

HA SPIRAL backend compilation

Rewrite trace

C compiler

Rewrite trace Coq proof

Rewrite trace Rewrite trace Coq proof

Model validation

Rewrite trace Translation validation

Rewrite trace Translation validation

Figure 1: Overview of our end-to-end system (left) and the end-to-end proof argument (right).

dynamic window approach is suitable for robotsdriving circular trajectories. It computes admis-sible velocities that avoid collisions with obsta-cles, and from those it chooses a velocity thatcan be realized by the robot within a short timeframe (the dynamic window) and bring it closerto some goal.

After verification, using a technique calledModelPlex [5], KeYmaera X synthesizes a prov-ably correct mathematical condition (a monitor)with the following property. This generated mon-itor checks, at runtime, that the observed en-vironment fits the verified model environment.When the observed behavior fits to the verifiedmodel as validated by the monitor execution atruntime, then the system execution is safe. How-ever, when the monitor is violated, the systemmay have evolved beyond the model assumptions,which means that the system is potentially un-safe and will enter failsafe mode. This is similarto Simplex monitors [6] detecting when to switchbetween controllers.

Having derived a monitor condition that in-forms when the observed environment no longerfits to the assumed model, the remaining prob-lem is to translate this monitoring expression intoan efficient piece of software that performs thecheck at runtime. The SPIRAL system [7, 8, 9]is used to synthesize a software implementationfrom the monitoring expression. The core of SPI-RAL is a rewriting system that manipulates SPI-

RAL’s HCOL language (hybrid control opera-tor language) into an equivalent expression thatcan be translated into code. Key requirementsare that every HCOL expression has a mathe-matical interpretation, and each transformationperformed on the HCOL expression must returna mathematically equivalent HCOL expression.The requirements, together, guarantee that thefinal code (when executed over the real numbers)would be a mathematically equivalent expressionto the monitoring expression. Next, SPIRALuses interval arithmetic [10] to implement thisfinal code using floating point numbers availableon current architectures. SPIRAL utilizes perfor-mance enhancing computer architecture featureslike SIMD vector instructions as well as aggres-sive compiler techniques (all of which are cast asmathematical rewrite rules) to produce highly ef-ficient code.

Another component of the approach is to uti-lize statistical and analytical redundancy betweenmultiple sensors that measure different quantitiesto establish that the current state of the systemas understood by the controllers is self consis-tent, i.e., there is no intrinsic inconsistency in themeasurements given the accuracy of the sensors.Statistical tests and analytical redundancy estab-lish that location estimated through GPS andthrough a wheel encoder do not disagree morethan the intrinsic inaccuracy of the respectivesensors. This makes it possible to detect GPS

2

spoofing. Other analytical redundancy meth-ods for protecting againt compromised sensorsinclude the estimation of vehicle speed from mul-tiple sound channels obtained with microphonesplaced strategically on the car [11], estimation ofvehicle altitude from correlating barometric pres-sure with GPS [12], and determining the postureof a robot via its camera view. All these meth-ods protect against compromised sensors sincethey correlate measurements that have a compli-cated analytical relationship that cannot be eas-ily maintained by an attacker.

This approach has been demonstrated on anumber of manned and unmanned ground andair vehicles. The dynamic window monitor wasdeployed as an end-to-end example produced byKeYmaera X and SPIRAL. It was used bothon the Landshark and the American-built car toprevent a malicious operator from crashing thevehicle into an obstacle. In addition, resilienceto GPS spoofing while the monitor was runningwas demonstrated, by utilizing statistical and setbased inconsistency detectors. Further the de-tection of replay attacks was demonstrated us-ing a statistical test. In all these demonstrationthe critical code pieces were synthesized with theSPIRAL system. In addition very accurate speedestimation solely from vehicle sound was demon-strated. Finally, a quadcopter height controllerwith correctness guarantees was synthesized.

Proving ControllersCorrect–And Catching Them

If NotDue to their impact on the real world, cyber-physical systems need to be safe. That poses anontrivial but important challenge because it isnot easy to get the control decisions exactly rightto maintain safety of the physical system and itsresponse through actuation, especially in light ofthe interaction with other agents in the environ-ment. Formal verification has been identified as apowerful analysis technique to establish correct-ness guarantees about the behavior of the designor find issues as early as possible in the designprocess [13].

The development begins with a model of thesystem dynamics as a hybrid system [14, 15, 16,2], which are mathematical models that featureboth discrete and continuous dynamics. This

flexible combination of dynamics is important forunderstanding systems with computerized or em-bedded controllers for physical systems since thelatter are usually modeled continuously while theformer are discrete. The development also be-gins with a precise formal definition of the safetyproperty to be guaranteed.

The approach uses differential dynamic logic(dL [17, 18, 19, 2, 20, 21]) as the language inwhich both hybrid systems model and desiredcorrectness properties can be specified unambigu-ously. Differential dynamic logic also providesthe systematic way of proving that the hybridsystem satisfies such correctness properties and isimplemented in the theorem prover KeYmaera X[1]. Once the hybrid systems model is proved tosatisfy its desired correctness properties in KeY-maera X, the ModelPlex tactic [5], which is im-plemented in KeYmaera X, synthesizes provablycorrect monitor conditions that check complianceof the system with the verified model so thatsafety transfers to the real system implementa-tion.

Model. To illustrate the principles in action,consider a ground robot that has to avoid colli-sion with obstacles [4]. Let us consider a sim-ple setting where the robot drives on a flat, evensurface. It is equipped with a distance measure-ment sensor, such as radar or Lidar, so that therobot is able to detect drivable regions. Every-thing else is considered an obstacle (for example,walls or other robots), meaning that the robot isable to measure the distance to obstacles. Therobot does so periodically according to its sam-pling period (for example, every 20 ms) when itdecides on steering, acceleration and braking ac-cording. The decisions on acceleration and steer-ing are input into actuators, which turn these intophysical motion that is followed until the robotcontroller runs the next time (for example, 20 mslater). During that time, the decisions cannot bechanged. That way, the robot can stitch togetherits trajectory by following circular arcs of vary-ing radius, as in the dynamic window approach.The robot can avoid collisions with obstacles bystopping or by choosing appropriate values forsteering that let it drive around obstacles.

In principle, obstacles could do the same. How-ever, the number of constraints put on how ob-stacles will move should be kept low, so that themodel fits many different kinds of motion. Hence,

3

the model assumes a maximum velocity and oth-erwise allows any kind of motion (for example,walls stay put, while moving obstacles could evenmake sudden orientation changes and jumps inspeed).

The model in equations (1)–(5) describes thedecisions of obstacles obst, the control choices ofthe robot robot, and the entailed physical behav-ior dyn. It models the dynamic window approach[3] for collision avoidance and is described in de-tail in [4].

The model is shown in Fig. 2. The modelingidiom t := 0; dyn, t′ = 1 & t ≤ ε used in (1)describes the sampling period of the controller:the clock t with constant slope t′ = 1, togetherwith the condition t ≤ ε, ensures that at mosttime ε passes between controller runs.

The obstacle model obst is very liberal. Itonly guarantees that obstacles will not exceeda maximum speed V . Otherwise, any behav-ior is allowed by choosing velocity vo := ∗ non-deterministically, which even includes sudden ori-entation changes and jumps in speed.

The robot has three control choices. First, ifthe condition safe is satisfied it can choose its ac-celeration ar and a new curve described by therotational velocity ωr and the curve radius rc.Of course, not all choices are admissible, so thecontrol branch ends in a test that allows onlyaccelerations in the physical acceleration limits−b ≤ ar ≤ A between maximum braking −b andmaximum acceleration A. The condition furtherensures that the robot is not spinning rc > 0 andthat the curve preserves planar rigid body mo-tion: the curve preserves the robot’s longitudinalspeed ωrrc = vr. Second, the robot can staystopped ar := 0 without spinning wr := 0, if itis stopped already. Finally, the robot can chooseto just hit the emergency brakes ar := −b on itscurrent curve unconditionally at any time.

These control choices entail physical behavioras described in dyn: the robot’s position changesaccording to its speed and orientation (p′r =vrdr), with speed in turn determined by accel-eration (vr = ar), while orientation follows alongthe chosen curve (ω′r = ar

rcand d′r = −ωrd

⊥r ).

The obstacle’s position is modeled in a similarfashion. Note that v′r = ar may result in nega-tive speeds vr < 0 upon braking ar < 0, so thecondition vr ≥ 0 ensures that hitting the brakesdoes not make the robot drive backwards.

Safety Property. Next, a safety property isneeded in order to analyze the model dw from (1)formally. Intuitively, with only stationary obsta-cles around, at all times, everything the robotever does has to result in positions different fromobstacle positions, as captured in pr 6= po. Inthe presence of moving obstacles, however, thiscondition needs to be relaxed, since guaranteesare only possible about the robot at hand, notabout the behavior of obstacles, as elaborated in[4, 22]. Hence, the model will guarantee passivesafety vr 6= 0→ pr 6= po, which means that therewill be no collisions while the robot is driving. So,if a collision occurs at all, it is because a movingobstacle ran into the robot. Or if all agents aresafe, there will be no collisions.

Eq. (6) below defines the requirements on therobot in a dL formula of the form initial →[model] safety. This means that, when the systemstarts in any initial state meeting the conditionsinitial, then all runs of model will end up withthe safety condition safety being satisfied.

vr = 0 ∧A ≥ 0 ∧ b > 0 ∧ ε > 0 ∧ V ≥ 0

→ [dw] (vr 6= 0→ pr 6= po) (6)

The dL formula in (6) defines the starting con-dition initial as follows: the robot is stopped ini-tially vr = 0, and not malfunctioning, which in-cludes a proper engine A ≥ 0, functional brakesb > 0, a sampling period ε > 0, and it assumesthat obstacles will not exceed V ≥ 0. Whenstarted under these conditions, all executions ofthe model dw (denoted by the box operator [dw]in dL) guarantee passive safety (vr 6= 0 → pr 6=po). The logical formula (6) can be analyzed inthe hybrid system theorem prover KeYmaera X.

Verification. KeYmaera X applies sound ax-ioms and proof rules to decompose the formula(6) into easier formulas, until only conditions infirst-order real arithmetic remain. These condi-tions are finally checked for validity with a de-cision procedure for real arithmetic (quantifierelimination, for example, through cylindrical al-gebraic decomposition [23, 24]), resulting in aproof of the initial logical formula. While theverification of cyber-physical systems is certainlychallenging, as is their design, KeYmaera X andits predecessor KeYmaera [25] have already been

4

dw ≡ (obst; robot; t := 0; {dyn, t′ = 1 & t ≤ ε})∗ (1)

obst ≡ vo := ∗; ?vo ≤ V (2)

robot ≡

ar := ∗; ωr := ∗; rc := ∗; ? (−b ≤ ar ≤ A ∧ rc > 0 ∧ ωrrc = vr) if safe

ar := 0; ωr := 0 if vr = 0

ar := −b unconditionally

(3)

dyn ≡ p′r = vrdr, v′r = ar, ω

′r =

arrc, d′r = −ωd⊥r , p′o = vo & vr ≥ 0 (4)

safe ≡ ‖pr − po‖∞ >v2r

2b+ V

vrb

+

(A

b+ 1

)(A

2ε+ ε(vr + V )

)(5)

Figure 2: The hybrid system model of the joint discrete and continuous behavior of the robot andthe obstacle. Control decisions are modeled in obst and robot, the physical motion is captured usingdifferential equations in dyn.

used successfully to verify cars [26, 27], air-craft [28, 29], trains [30], robots [4], and surgi-cal robots [31], and to verify the usual controlschemes such as PID [32, 30]. For a tutorial onmodeling and proving safety with dL, see [33].

ModelPlex. Formal verification makes strongguarantees about the system behavior if ade-quate models of the system can be obtained. Inany CPS design process, models are essential;but any model necessarily deviates from the realworld. Faults may cause the system to func-tion improperly, sensors may deliver uncertainvalues, actuators may suffer from disturbance,or the model may have assumed simpler ideal-world dynamics for tractability reasons or madeunrealistically strong assumptions about the be-havior of other agents in the environment. Asa consequence, the verification results obtainedabout models of a CPS only apply to the actualCPS at runtime to the extent that the model ade-quately represents reality. A high-assurance CPSmust be aware of the limitations in its designand equipped with means to detect deviationsbetween design and reality.

The proofs so far formally show that a modelof the robot is safe. In other words, the mod-eled family of robot controllers provably guaran-tees passive safety. The remaining task is to val-idate whether the model is adequate, so that thesafety proof of the model transfers to the actualsystem implementation [34, 35]. ModelPlex [5]is a method to synthesize correct-by-construction

monitors for CPS by theorem proving automati-cally : ModelPlex is based on the sound axiomsand proof rules of dL [20, 21] to synthesize prov-ably correct monitors that validate complianceof system executions with a model. The difficultquestion answered by ModelPlex is what exactconditions need to be monitored at runtime toguarantee compliance with the models and thussafety. ModelPlex enables tradeoffs between ana-lytic power and accuracy of models while retain-ing strong safety guarantees.

At runtime, the ModelPlex monitors check thesystem behavior for model compliance. If the ob-served system execution fits to the verified model,then this execution is safe according to the of-fline verification result about the model. If itdoes not fit, then the system is potentially un-safe because it evolves outside the verified modeland no longer has an applicable safety proof, sothat a verified fail-safe action from the model isinitiated to avoid safety risks (cf. Simplex [6]).

Since failures may occur and software attacksmay happen, actual evolution must be moni-tored: the acceleration chosen by the controllermust fit to the current situation (for example, ac-celerate only when safe), the chosen curve mustfit to the current orientation, and no unintendedchange to the robot’s speed, position, orientation,or knowledge about the obstacles occurred. Thismeans, any variable that is allowed to change inthe model must be monitored. In the examplehere, these variables include the robot’s positionpr, longitudinal speed vr, rotational speed ωr,

5

acceleration ar, orientation dr, curve rc, and ob-stacle position po.

ModelPlex uses that the system is sampled pe-riodically: for each variable there will be twoobserved values, one from the previous sampletime (for example, positions pr) and one fromthe current sample time (for example, p+

r ). Itis not important for ModelPlex that the valuesare apart by exactly the sampling period, butmerely that there is an upper bound (ε). A Mod-elPlex monitor checks in a provably correct waywhether the evolution observed in the differenceof the sampled values can be explained by themodel. The verified hybrid system models them-selves are not helpful as fast executable models,because they involve nondeterminism and differ-ential equations. Hence, provably correct moni-tor expressions in real arithmetic are synthesizedfrom a hybrid system model using an offline proofin KeYmaera X. These expressions exhaustivelycapture the behavior of the hybrid system mod-els, projected onto the pairwise comparisons ofsampled values that are needed at runtime. Thefull process is described in detail in [5].

ModelPlex monitor. Here, let us focus ona controller monitor expression synthesized fromthe model in (1)–(4) above, which captures allpossible decisions of the robot that are consideredsafe. A controller monitor [5] checks the decisionsof an (unverified) controller implementation forbeing consistent with the discrete model. Mod-elPlex automatically obtains the discrete modelfrom model (1)–(4) with the ordinary differentialequation (ODE) being safely over-approximatedby its evolution domain. The resulting conditionmonitor in Fig. 3 (7), which is synthesized bya proof, follows the structure of the model: itcaptures the assumptions on the obstacle mono,the evolution domain from dynamics mondyn, aswell as the specification for each of the three con-troller branches (braking monb, staying stoppedmons, or accelerating mona).

The obstacle monitor part mono, see (8), saysthat the measured obstacle velocity d+

r must notexceed the assumptions made in the model aboutthe maximum velocity of obstacles. The dynam-ics monitor part mondyn, see (9), checks the evo-lution domain of the ODE and that the con-troller did reset its clock (t+ = 0). The brak-ing monitor monb, see (10) defines that in emer-gency braking the controller must only hit the

brakes and not change anything else (accelera-tion a+

r = −b, while everything else is of the formx+ = x meaning that no change is expected).1

When staying stopped mons, see (11), the cur-rent robot speed must be zero (vr = 0), and thecontroller must choose no acceleration and no ro-tation (ar = 0 and ωr = 0), while everything elseis unchanged. Finally, the acceleration monitormona, see (12)–(13), when the distance is safe therobot can choose any acceleration in the physicallimits −b ≤ a+

r ≤ A, a new non-spinning steeringc+r 6= 0 that fits to the current speed ω+

r c+r = vr;

position, orientation, and speed must not be setby the controller (those follow from the accelera-tion and steering choice).

The formula monitor in (7) synthesized withthis correct-by-construction proof approach is thebasis for code synthesis, as elaborated next.

Generating Code From aMathematical Specification

Given a provably correct monitor specificationthat guarantees the desired behavioral (e.g.safety) properties, it is important that the instan-tiation of the specification as code is faithfullyimplemented. This ensures that all proven be-havioral properties are preserved during the im-plementation process. In addition the implemen-tation must be conservative in the presence offloating point rounding errors. As many proofsprovided by formal methods are reasoned overreal numbers—as opposed to floating point num-bers found on computer systems—this differencein number representations, if not handled appro-priately, may cause undesirable deviations fromthe specified model.

The SPIRAL system [7, 8, 9] synthesizes aconservative and faithful implementation fromthe mathematical specification through the suc-cessive application of identity rewriting. Eachrewriting step replaces the input expression witha mathematically equivalent but more detailedexpression that is more aligned to code. Byensuring that mathematical equivalence is pre-served after each rewriting step, the correctnessof the final implementation is guaranteed. Fig. 4shows the overall flow. A multi-stage rewriting

1Note that unchanged obstacle position p+r = pr

means that the robot should not waste time measuringthe obstacle’s position, since braking is safe in any case.

6

monitor ≡ mono ∧mondyn ∧ (monb ∨mons ∨mona) (7)

mono ≡ ‖d+r ‖ ≤ V (8)

mondyn ≡ 0 ≤ ε ∧ vr ≥ 0 ∧ t+ = 0 (9)

monb ≡ p+o = po ∧ p+

r = pr ∧ d+r = dr ∧ v+

r = vr ∧ ω+r = ωr ∧ a+

r = −b ∧ c+r = cr (10)

mons ≡ vr = 0 ∧ p+o = po ∧ p+

r = pr ∧ d+r = dr ∧ v+

r = vr ∧ ω+r = 0 ∧ a+

r = 0 ∧ c+r = cr (11)

mona ≡ −b ≤ a+r ≤ A ∧ c+r 6= 0 ∧ ω+

r c+r = vr ∧ p+

r = pr ∧ d+r = dr ∧ v+

r = vr (12)

∧ ‖pr − p+o ‖∞ >

v2r

2b+ V

vrb

+

(A

b+ 1

)(A

2ε2 + ε(vr + V )

)(13)

Figure 3: Synthesized safety conditions. The generated monitor captures conditions on obstaclesmono, on dynamics mondyn, and on the robot controller’s decisions on braking monb, staying stoppedmons, and accelerating mona. The monitor distinguishes two observed values per variable, separatedby a controller run (for example, pr denotes the position before running the controller, whereas p+

r

denotes the position after running the controller).

system [36, 37] consisting of a backtracking andexpansion stage and multiple recursive descentand confluent term rewriting stages transformsan initial specification into a final piece of code,as explained in the remainder of this section.

Problem specification. Mathematical spec-ifications are specified using SPIRAL’s hybridcontrol operator language (HCOL). In HCOL, anoperator is a mathematical function that mapsone or more real vectors to a real vector. Realscalars are treated as vectors of dimension one,and higher dimensional objects such as matricesare linearized into vectors. The following discus-sion focuses on Eq. (13), which is part of the fullsafety condition summarized in Fig. 3. Eq. (13)is written as HCOL operator as

SafeDistV,A,b,ε : R× R2 × R2 → Z2;

(vr, pr, po) 7→(p(vr) < d∞(pr, po)

)(14)

with d∞(~x, ~y) = ‖~x − ~y‖∞, p(x) = a2x2 +

a1x + a0, a2 = 12b , a1 = V

b + ε(Ab + 1

), and

a0 =(Ab + 1

) (A2 ε

2 + εV). This is essentially the

same expression as (13) but makes explicit alldata types and free parameters and expresses thecomputation explicitly in terms of higher-levelmathematical objects such as polynomials andnorms.

Breakdown rules and basic operators.The first step for translating (14) into an equiv-alent high performance implementation is to de-rive a top level breakdown rule that explains (14)

in terms of SPIRAL’s library of known mathe-matical objects expressed in the HCOL language,summarized in Fig. 5. The rule expressing thistransformation for (14) is derived by KeYmaeraX as

SafeDistV,A,b,ε(., ., .)

→(P [a0, a1, a2](.) < d2

∞(., .))(., ., .). (15)

It closely mirrors the mathematical expression ofthe specification (14) and thus the original mon-itoring equation (13). As required, it expressesthe semantics of the safety distance in terms ofthe HCOL library shown in Fig. 5. This leveragesthe HCOL formalization of well known math-ematical objects such as infinity norm, Cheby-shev distance, scalar product, or evaluation of apolynomial that are part of SPIRAL’s library ofmathematical objects and identities.

The goal of the rewriting process is to breakHCOL operator specifications like (14) into ex-pressions of basic HCOL operators through re-peated applications of rules. The list of basicHCOL operators that are admissible in a fullyexpanded expression is shown in Fig. 6. Further,operations like ◦ and × (operator compositionand Cartesian product) are also allowed.

Consider the example of vector addi-tion, expressed through the basic operatorPointwisen×n,(a,b) 7→a+b . The Pointwise operatortakes two parameters (shown as subscripts),where n × n are the dimensions of the two

7

Pointwisen,fi : Rn → Rn; (xi) 7→(f0(x0), . . . , fn−1(xn−1)

)Pointwisen×n,fi : Rn × Rn → Rn;

((xi), (yi)

)7→(f0(x0, y0), . . . , fn−1(xn−1, yn−1)

)Reductionn,fi : Rn → R; (xi)i 7→ fn−1(xn−1, fn−2(xn−2, fn−3(. . . f0(x0, id()) . . . )

Inductionn,fi : R→ Rn+1; x 7→ (fn(x, fn−1(. . . ) . . . ), . . . , f2(x, f1(x, id)), f1(x, id), id())

Figure 6: Basic HCOL operators that have mathematical semantics but also can be seen as functionallanguage constructs.

Expansion + backtracking

Recursive descent

Confluent term rewriting

Recursive descent

Recursive descent

Abstract code

HCOL specification

HCOL (dataflow) expression

Optimized Ʃ-HCOL expression

Ʃ-HCOL (loop) expression

Optimized abstract code

C code

Confluent term rewriting

Figure 4: Code/proof co-synthesis as multi-stagerewriting system.

input vectors, and (a, b) 7→ a + b is the math-ematical operation that is performed on eachpair of scalar elements from the two inputvectors. Similarly, the Hadamard product (orelement-wise multiplication) can be defined asPointwisen×n,(a,b)7→ab . More complicated oper-ators can be defined through the compositionof simpler operators through HCOL operatorexpressions. For example, the scalar (or dot)

‖.‖n∞ : Rn → R; (xi) 7→ maxi=0,...,n−1 |xi|dn∞(., .) : Rn × Rn → R; (x, y) 7→ ‖x− y‖n∞

< ., . >n: Rn × Rn → R;((xi), (yi)

)7→

n−1∑i=0

xiyi

(xi)n : R→ Rn+1; x 7→(x0, x1, . . . , xn

)P [a0, . . . , an] : R→ R; x 7→

n∑i=0

aixi

Figure 5: Library of mathematical objects ex-pressed as HCOL operators.

product can be described as

< ., . >n→ Reductionn,(a,b) 7→a+b

◦ Pointwisen×n,(a,b) 7→ab . (16)

The recursive decomposition of higher levelHCOL operators into basic operators is capturedwithin the SPIRAL system as a library of break-down rules. Fig. 7 collects all the breakdownrules needed to fully expand the safety distancemonitor (14). By performing all the necessarysubstitutions as prescribed by the breakdownrules, the initial HCOL operator specification(14) is eventually translated into the finally ex-panded HCOL expression, shown in Fig. 8, con-sisting only of basic HCOL operators. This is thefinal result of the first stage in SPIRAL’s rewrit-ing system (backtracking and expansion, Fig. 4).

Code generation. The second stage in thecode generation process is the translation of anHCOL expression into highly efficient C code.This is performed by a sequence of rewritingstages performing either a recursive descend or aconfluent term rewriting phase. Logically, these

8

SafeDistV,A,b,ε = Pointwise(x,y)7→x<y

◦((

Reduction3,(x,y) 7→x+y ◦Pointwise3,x 7→aix ◦ Induction3,(a,b)7→ab,1

)×(

Reduction2,(x,y)7→max(|x|,|y|) ◦Pointwise2×2,(x,y)7→x−y))

Figure 8: The final HCOL expression derived for the monitoring expression (13).

dn∞(., .)→ ‖.‖n∞ ◦ Pointwisen×n,(a,b)7→a−b

‖.‖n∞ → Reductionn,(a,b)7→max(|a|,|b|)

< ., . >n→ Reductionn,(a,b) 7→a+b

◦ Pointwisen×n,(a,b)7→ab

P [a0, . . . , an]→< (a0, . . . , an), . > ◦(xi)n(xi)n → Inductionn,(a,b)7→ab,1

Figure 7: Breakdown rules that express HCOLmathematical objects as basic HCOL objects.

steps are grouped into two stages using two sep-arate sets of substitution rules.

In the first step, HCOL is translated into alower level mathematical representation called Σ-OL, where loops are made explicit. For instance,Pointwise is translated into the following expres-sion,

Pointwisen×n,fi →n−1∑i=0

eni ◦Pointwise1×1,fi

◦((eni )> × (eni )>

), (17)

where eni is a unit n-dimensional basis vector withthe 1 at the ith position and × is the cross prod-uct. (eni )

>represents a gather operation and eni

represents a scatter operation. Similarly, the re-duction operation is translated into Σ-OL by therule

Reductionn,(a,b)7→a+b →n−1∑i=0

(eni )>.

At the Σ-OL level, optimizations performed bya traditional optimizing compilers are performedthrough substitution rules such as

Pointwisen,fi ◦ ejn → ejn ◦Pointwise1,fj . (18)

# Hadamard Product

decl([i7], loopn(i7, n1,

assign(nth(Y, i7),

mul(nth(X, i7), nth(y1, i7)))))

# Reduction

decl([i4], chain(

assign(nth(Y, V(0)), V(0)),

loopn(i4, n1, decl([ s1 ], chain(

assign(s1, nth(X, i4)),

assign(nth(Y, V(0)), add(nth(Y, V(0)), s1))

)))))

# Scalar Product (optimized)

decl([i8], chain(

assign(nth(Y, V(0)), V(0)),

loopn(i8, n1, decl([ s2, s3 ], chain(

assign(s3, nth(X, i8)),

assign(s2, mul(s3, nth(y1, i8))),

assign(nth(Y, V(0)), add(nth(Y, V(0)), s2))

)))))

Figure 9: Spiral’s internal icode represen-tation for the (top) Hadamard product,Pointwisen×n,(a,b) 7→ab, (middle) Reduction,Reductionn,(a,b)7→a+b, and (bottom) ScalarProduct (after optimization). The icode isthen pretty-printed in the desired programminglanguage such as C. X is the input vector and Y

is the output vector.

The above rule turns a program fragment thatcopies n pieces of data into contiguous memoryaddresses before applying the function fi on eachelements, into a program fragment that appliesthe same function on the appropriate piece ofdata, copies it into contiguous storage, and re-peats for the remaining n − 1 pieces of data.While functionally equivalent, the optimized pro-gram is more efficient since it parses through thedata once.

Notice that the final Σ-OL expression is stilla mathematical expression, but can be seen ashighly optimized loop-based program that imple-ments a mathematical function. In addition, be-

9

cause traditional compiler optimizations are im-plemented within SPIRAL as substitution rules,the correctness of the optimizations is ensured.

The second translation step translates a Σ-OLexpression into an actual loop-based program, bymeans of a small set of compilation rules like

Code(y = (A ◦B)(x)

)→{decl(t),

Code(t = B(x)

),Code

(y = A(t)

)}. (19)

Strong guarantees about loops, conditionals,and array accesses are inherited and deducedfrom the original expression. All this togetherguarantees that the program over the real num-bers is a pure function that is mathematicallyequivalent to the original specification. Fig. 9shows the final generated code for the HadamardProduct, Reduction operator, and scalar productover real arithmetic represented in SPIRAL’s in-ternal code representation, called icode. Rewrit-ing Σ-HCOL to this internal code representationrequires five translation rules, which are sum-marized in Table 1. By repeated application ofthese five rules, the icode representation for (13)is shown in Fig. 10.

Code optimization. SPIRAL generates ver-ified code through a sequence of formally provedrewrite rules. The trace of the rewrite rules thatwere applied provides a certificate that a givenprogram is correct. However, the generated codemust be compiled. This last step should also beverified, and can be done through the use of acertified compiler such as CompCert[38]. As thegoal is correct and efficient code, it is necessaryto ensure the compiled code is optimized. Whilethe performance of CompCert has improved, itusually do not yield code with performance thatare comparable with those using state-of-the-artoptimizing compilers such as Intel’s C compiler.Nonetheless, it can be used since many of theoptimizations a good compiler performs are ac-complished through the transformations carriedout during the rewrite process, such as the loopmerging performed by (22).

Additional optimizations such as tiling, loopunrolling, and vectorization can be performed bysource to source transformations and verified atthe code level, and in many cases can be done ata higher mathematical level like the loop mergingexample. Even when the optimizations cannot bedone at the mathematical level, the fact that the

code is being generated allows various assump-tions, like dependence, to be guaranteed whichsimplifies proofs of their correctness. This is il-lustrated by further optimizations applied to thescalar product example. After loop merging thegenerated code looks like

for (s=0, j = 0; j < 2*N; ++j) {

s += x[j] * y[j];

}

This can be optimized by loop unrolling and vec-torized code can be obtained by combining theoperations in the unrolled loop.

s0 = 0; s1 = 0; s = 0;

for (i = 0; i < M; ++i)

for (j = 0; j < 2*N; j+=2) {

s0 += x[j] * y[j];

s1 += x[j+1] * y[j+1];

}

s = s0 + s1;

The equivalence of the unrolled code and theinitial code can be easily verified by induction.Alternatively, the vectorization can be derivedand verified through higher level transformations;namely the rule

〈., .〉2n → 〈., .〉2 ◦ 〈., .〉n ⊗ I2 (20)

which uses the tensor product [39, 40] to obtainvectorized code.

These simple transformations can lead to a sig-nificant performance gain. Timings on an In-tel Core i7-3770 processor running Ubuntu 14.04with CompCert 2.5 show a speedup of 3.5 fromjust the unrolling. In order to benefit from vec-torization it is necessary that CompCert be ableto generate code with vector instructions; how-ever, it is not required that CompCert performvectorization as this can be done as shown. Thisshows that a certified compiler can be used, with-out sacrificing performance, when combined withsource to source optimizations provided theiris good support for basic compiler functional-ity such as register allocation and instructionscheduling.

Floating-point arithmetic. Finally, the dif-ference between real and floating point numberrepresentation has to be tackled. A conserva-tive approximation is attained through the useof interval arithmetic [10]. Each real number a isrepresented by an interval [ainf, asup] where theboundaries ainf and asup are the floating point

10

Code(y = (A ◦B)(x)) → decl(t, chain(Code(t = B(x)),Code(y = A(t))))

Code

(y =

n−1∑i=0

Ai(x)

)→ chain(assign(y, 0),

loop(i, [0..n− 1],Code(y + = Ai(x))))

Code(y = (eni )>(x)

)→ assign(y, x[i])

Code(y = eni (x)) → assign(y[i], x)

Code(y = Pointwise1,f (x)) → assign(y, f(x))

Table 1: Rewrite rules for translating Σ-HCOLto Spiral’s icode representation.

numbers closest to a, such that ainf ≤ a ≤ asup.This ensures that the actual (true) value is al-ways bounded by ainf and asup. Interval arith-metic then computes using the boundary valuesas opposed to the true value. For instance,

[ainf, asup]+[binf, bsup] = [rounddown(ainf+binf),

roundup(asup + bsup)].

Similarly, the multiplication of two intervals isgiven by

[ainf, asup]× [binf, bsup] =[min

(rounddown(−ainf × binf),

rounddown(ainf × bsup),

rounddown(binf × asup),

rounddown(−asup × bsup)),

max(

roundup(ainf × binf),

roundup(−ainf × bsup),

roundup(−binf × asup),

roundup(asup × bsup))]

(21)

By using proper floating point rounding modes,operations on the intervals guarantee that the re-sult interval over floating point numbers includesthe result that is over the real numbers. Imple-menting interval arithmetic efficiently on modernprocessors can be challenging. However, the im-plementation of interval arithmetics within SPI-RAL leverages modern architecture features suchas the single instruction multiple data (SIMD)vector instruction set to reduce the number ofactual instructions executed by the processor.

Final monitor code. Introducing SPIRAL’sinterval arithmetics implementation to the icode

// icode implementation of Eq. (13) over the reals

func(TInt, "dwmonitor", [ X, D ],

decl([i3, i5, q3, q4, s1, s4,

s5, s6, s7, s8, w1, w2],

chain(

assign(s5, V(0.0)),

assign(s8, nth(X, V(0))),

assign(s7, V(1.0)),

loop(i5, [0..2], chain(

assign(s4, mul(s7, nth(D, i5))),

assign(s5, add(s5, s4)),

assign(s7, mul(s7, s8))

)),

assign(s1, V(0.0)),

loop(i3, [0..1], chain(

assign(q3, nth(X, add(i3, V(1)))),

assign(q4, nth(X, add(V(3), i3))),

assign(w1, sub(q3, q4)),

assign(s6, cond(geq(w1, V(0)),

w1, neg(w1))),

assign(s1, cond(geq(s1, s6),

s1, s6))

)),

assign(w2, geq(s1, s5)),

creturn(w2)

)))

Figure 10: The implementation of the dynamicwindow monitor in SPIRAL’s internal code rep-resentation in real arithmetic.

representation in Fig. 10 yields the C implemen-tation in Fig. 11. It is implemented using the In-tel C++ compiler’s intrinsic functions to explic-itly use the special vector instructions providedby the Intel SSE4 instruction set extension, andruns in approximately 100 processor cycles on an3.6 GHz Intel Core i7 processor. Notice the com-plexity of the code. If manually implemented, theprobability of an error being introduced would in-crease. However, this complexity is hidden fromthe programmer through the use of rewrite rulesthat are faithfully applied by SPIRAL. The faith-ful application of the rewrite rules ensure that theintroduction of interval arithmetics preserve theinput specifications (if compute with real num-bers). In addition, it is also guaranteed that thereal values is always bounded by the floating-point interval bounds, which ensures that the im-plementation is conservative.

Correctness proofs and guarantees. Sinceall transformations from specification to finalcode are rewrite rules that replace a mathemat-ical object (expression) with another equivalentexpression, the sequence of rule applications es-

11

// Final C/SSE 4.1 Implementation of Equation (13) for Intel Core i7 Processors

// This is a conservative high performance implementation using interval arithmetic

int dwmonitor(float *X, double *D) {

__m128d u1, u2, u3, u4, u5, u6, u7, u8 , x1, x10, x13,

x14, x17, x18, x19, x2, x3, x4, x6, x7, x8, x9;

int w1;

unsigned _xm = _mm_getcsr();

_mm_setcsr(_xm & 0xffff0000 | 0x0000dfc0);

u5 = _mm_set1_pd(0.0);

u2 = _mm_cvtps_pd(_mm_addsub_ps(_mm_set1_ps(FLT_MIN), _mm_set1_ps(X[0])));

u1 = _mm_set_pd(1.0, (-1.0));

for(int i5 = 0; i5 <= 2; i5++) {

x6 = _mm_addsub_pd(_mm_set1_pd((DBL_MIN + DBL_MIN)), _mm_loaddup_pd(&(D[i5])));

x1 = _mm_addsub_pd(_mm_set1_pd(0.0), u1);

x2 = _mm_mul_pd(x1, x6);

x3 = _mm_mul_pd(_mm_shuffle_pd(x1, x1, _MM_SHUFFLE2(0, 1)), x6);

x4 = _mm_sub_pd(_mm_set1_pd(0.0), _mm_min_pd(x3, x2));

u3 = _mm_add_pd(_mm_max_pd(_mm_shuffle_pd(x4, x4, _MM_SHUFFLE2(0, 1)),

_mm_max_pd(x3, x2)), _mm_set1_pd(DBL_MIN));

u5 = _mm_add_pd(u5, u3);

x7 = _mm_addsub_pd(_mm_set1_pd(0.0), u1);

x8 = _mm_mul_pd(x7, u2);

x9 = _mm_mul_pd(_mm_shuffle_pd(x7, x7, _MM_SHUFFLE2(0, 1)), u2);

x10 = _mm_sub_pd(_mm_set1_pd(0.0), _mm_min_pd(x9, x8));

u1 = _mm_add_pd(_mm_max_pd(_mm_shuffle_pd(x10, x10, _MM_SHUFFLE2(0, 1)),

_mm_max_pd(x9, x8)), _mm_set1_pd(DBL_MIN));

}

u6 = _mm_set1_pd(0.0);

for(int i3 = 0; i3 <= 1; i3++) {

u8 = _mm_cvtps_pd(_mm_addsub_ps(_mm_set1_ps(FLT_MIN), _mm_set1_ps(X[(i3 + 1)])));

u7 = _mm_cvtps_pd(_mm_addsub_ps(_mm_set1_ps(FLT_MIN), _mm_set1_ps(X[(3 + i3)])));

x14 = _mm_add_pd(u8, _mm_shuffle_pd(u7, u7, _MM_SHUFFLE2(0, 1)));

x13 = _mm_shuffle_pd(x14, x14, _MM_SHUFFLE2(0, 1));

u4 = _mm_shuffle_pd(_mm_min_pd(x14, x13), _mm_max_pd(x14, x13), _MM_SHUFFLE2(1, 0));

u6 = _mm_shuffle_pd(_mm_min_pd(u6, u4), _mm_max_pd(u6, u4), _MM_SHUFFLE2(1, 0));

}

x17 = _mm_addsub_pd(_mm_set1_pd(0.0), u6);

x18 = _mm_addsub_pd(_mm_set1_pd(0.0), u5);

x19 = _mm_cmpge_pd(x17, _mm_shuffle_pd(x18, x18, _MM_SHUFFLE2(0, 1)));

w1 = (_mm_testc_si128(_mm_castpd_si128(x19), _mm_set_epi32(0xffffffff, 0xffffffff,

0xffffffff, 0xffffffff)) ???

(_mm_testnzc_si128(_mm_castpd_si128(x19), _mm_set_epi32(0xffffffff, 0xffffffff,

0xffffffff, 0xffffffff))));

__asm nop;

if (_mm_getcsr() & 0x0d) {

_mm_setcsr(_xm);

return -1;

}

_mm_setcsr(_xm);

return w1;

}

Figure 11: The implementation of the dynamic window monitor using interval arithmetic in Intel’sSSE 4.1 instruction set. The shown monitor code runs in about 100 processor cycles on an 3.6 GHzIntel Core i7 processor.

12

tablishes mathematical equivalency of specifica-tion and final code. Over the real numbers thecode is mathematically identical to the originalspecification. Over floating point numbers, theuse of interval arithmetic in the resulting codeensures that the code is a conservative approxi-mation. Numerical results are sound as the trueanswer is guaranteed to be in the result interval.Logical answers are sound as the answer is con-servative: true/false/unknown. However, theseguarantees are only true if the rules themselveshave been implemented correctly.

Each rule that can be applied is formally veri-fied so that the transformed expressions are guar-anteed to be equivalent to the original expression.For example, the rewrite rule (16) is a special caseof the more general rule

Reductionn,f ◦Pointwisen×n,g

→ Reductionn×n,f◦g, (22)

which can be proven by induction on n. Alterna-tively, the validity of the special case in (16) canbe verified, using the property that the scalarproduct is bilinear, and checking that the twosides agree on a basis. Note that reduction withplus is the linear transformation given by the 1×nvector of ones, (1n)>, and the following compu-tation shows that the left and right hand sides of(16), applied to an arbitrary pair of standard ba-sis elements, are both equal to δi,j , the Kroneckerdelta.

(1n)>(eni · enj ) = (1n)>δi,jeni

= δi,j(1n)>eni

= δi,j = 〈eni , enj 〉

Similarly Rule 17 can be verified by applyingthe left and right hand sides to an arbitrary pairof vectors (x, y) and checking that the j-th ele-ment of the results are the same.

(enj )> ◦∑n−1

i=0

(eni ◦ Pointwise1×1,fi ◦(

(eni )> × (eni )>))(x, y)

=∑n−1

i=0

((enj )> ◦ eni ◦ Pointwise1×1,fi ◦((eni )> × (eni )>)

)(x, y)

=∑n−1

i=0

(δi,j ◦ Pointwise1,fi ◦((eni )> × (eni )>)

)(x, y)

= (enj )> ◦ Pointwisen×n,fi(x, y)

These calculations can be formalized andchecked with a proof assistant such as Is-abelle [41] or Coq [42]. Similar calculations al-low us to verify the rule that merges the reduc-tion and pointwise operators which optimizes thescalar product computation to use one instead oftwo loops.

When converting Σ-OL expressions to code wemust verify that the resulting code correctly pre-serves the mathematical semantics of the expres-sion. Once correctness is proven for the basic ex-pressions such as reduction and pointwise, thenan inductive proof can be obtained to prove thatthe code generated for arbitrary expressions builtup form higher level operators such as compo-sition and Cartesian product are correct. Sim-ilarly, optimizations that are traditionally per-formed by optimization compilers are formallywritten as rewrite rules in SPIRAL, thus provingthat the optimizations applied by SPIRAL forperformance reasons retain the correctness guar-entees of the input specifications. For more de-tails, see “Code Optimization as Rewrite Rules”.

Anomaly Detection asStatistical Deviation from

Nominal BehaviorThis section presents a set of statistical methodsfor anomaly detection based on two observations:(a) Robot sensors usually produce data that is re-dundant but noisy, and (b) It is often feasible tospecify a priori a model of nominal behavior forthese redundancies, but not to fully specify all theanomalies that may occur. Thus, the resulting al-gorithms first build statistical models of nominalbehavior, and then detect anomalies during exe-cution by finding sequences of observations thatdo not fit the model of nominal behavior.

Nominal models from redundancy

Robots often produce redundant informationabout the world from various sources. This re-dundancy can occur at various levels, such asworld state estimation, task completion time, ormotion properties. This section explores the ex-ample of monitoring the robot’s motion prop-erties, since it is applicable to many mobilerobots. Information about the robot’s motioncan be obtained from its wheel encoders, GPSsensors, IMUs, cameras, and the robot’s input

13

command, and localization algorithms that in-tegrate these sensors, among others. Generally,given two simultaneous observations x1

t and x2t

obtained from different sources at time t, the al-gorithms assume that it is possible to map themto two comparable observations x1

t = f1(x1),and x2

t = f2(x2) that are expected to have sim-ilar values during nominal execution. For exam-ple, the robot’s displacement between timestepscan be computed both from the robot’s wheelencoder values, and from consecutive outputs ofa sensor-fusing localization algorithm. Fig. 12ashows graphs of these two sources of informationin the CoBot mobile robots [43] during nomi-nal execution. The properties of the difference∆xt = x1

t − x2t can be extracted from data of

nominal execution. In particular, since many sen-sors have distributions that are approximatelynormal, the following examples will adhere tothat distribution. Thus, the algorithm first cre-ates a model θ0 of nominal execution:

P (∆xt|θ0) = N (µ, σ2) where µ ∈ [µ−, µ+](23)

That is, the difference between the two sources isnormally-distributed, with variance σ2 extractedfrom nominal execution, and mean µ allowed tobe within a small interval [µ−, µ+] around 0.Other sensors and sources of information mayhave different distributions, but this section fo-cuses on normal distributions as a useful examplein robotics.

Statistical testing for anomalies

Given that the model θ0 is given by a normal dis-tribution, the detection algorithms use a Z-testto determine the probability of observing a set ofobservations at least as unlikely as X given nom-inal execution; this section describes the Z-testfor one-dimensional observations, although ex-tension to higher dimensions is straightforward.

Given the set of observations X ={∆x1,∆x2, . . . ,∆xn}, the algorithm esti-mates the probability that the true meanµ of the underlying distribution lies within[µ−, µ+]. That is, it calculates the probabilityP (µ− ≤ µ ≤ µ+). First, define the standardized

sample mean Z:

Z(X) =X(X)− µ√σ2/|X|

where X(X) =1

n

n∑i=1

∆xi.

(24)

The standardized problem then becomes that ofcalculating P (Z− ≤ Z ≤ Z+), where Z− andZ+ are calculated analogously to Z, replacing µby µ+ and µ− respectively. Since these variablesare in standard form, the desired proability isobtained using the standard cumulative normaldistribution Φ(Z):

P (µ− ≤ µ ≤ µ+) = P (Z− ≤ Z ≤ Z+) (25)

= P (Z ≤ Z+)− P (Z ≤ Z−)

= Φ(Z+)− Φ(Z−)

This probability is then compared to a thresh-old Pmin to determine if the set X is too unlikelyto come from θ0.

A Multi-Scale window approach toanomaly detection

Depending on the type of anomaly to be detected,different sets of observations may be analyzed foranomalies. This section focuses on analyzing se-quences of observations to detect anomalies thatstart occurring at some time t0, and affect therobot at any time t ≥ t0, such as those illus-trated in Fig. 12; other work has analyzed sets ofnon-sequential but otherwise correlated observa-tions [44].

During each time step tk of execution, then,the algorithm searches for a time t0 such thatP (∆xt0 ,∆xt0+1, . . . ,∆xtk |θ0) is too low to beconsidered nominal. One approach used in re-lated work is to test every possible t0 ∈ [0, tk]for anomalies. However, this approach growslinearly with the number of observations, whichmay be restrictive for online monitoring of long-deployment robots. Instead, the algorithm pre-sented here uses an approach that tests windowsof time of various scales to find anomalies. Thus,the detector creates N sets X0, X1, . . . , XN ofmost recent observations on which to conduct aZ-test, where

Xi = {∆xk,∆xk−1, . . . ,∆xk−2i}. (26)

Then, the Z-test, previously discussed, is con-ducted on each of these windows of time.

14

0 5 10 15−0.02

0

0.02

0.04

0.06

0.08

0.1

Time (s)

Dis

pla

ce

me

nt

(m)

Source: localization

Source: encoders

(a) Nominal Execution

0 5 10 15−0.02

0

0.02

0.04

0.06

0.08

0.1

Time (s)D

isp

lace

me

nt

(m)

Source: localization

Source: encoders

(b) Subtle Anomaly

0 5 10 15−0.02

0

0.02

0.04

0.06

0.08

0.1

Time (s)

Dis

pla

ce

me

nt

(m)

Source: localization

Source: encoders

(c) Clear Anomaly

Figure 12: Motion data gathered from the CoBot robots [43]. For the anomalies, the wheel encodersdisplayed varying levels of malfunctioning.

Algorithm 1 Multi-window approach to statis-tical anomaly detection.Input: Sequence X of observations; Number ofwindows N ; Nominal model θ0

Output: true if an anomaly is detected, false oth-erwise.

function DetectAnom(X = [∆x0,∆x1, . . . ,∆xk],N , θ0 = {σ, µ−, µ+})

for i ∈ {0, 1, . . . , N,∞} doXi ← {∆xk,∆xk−1, . . . ,∆xk−2i} . Extract

data from window i

Z+(X) =X(Xi)−µ−√σ2/|Xi|

. Standardized deviations

Z−(X) =X(Xi)−µ+√σ2/|Xi|

P ← Φ(Z+)− Φ(Z−) . Probability thatµ ∈ [µ−, µ+]

if P < Pmin thenreturn true . Probability too low, return

failureend if

end forreturn false . No probability found at any time

scaleend function

Algorithm 1 summarizes the process of on-line statistical anomaly detection. The algorithmconducts the statistical Z-test on data comingfrom windows of N different sizes to find anoma-lies.

The time required to detect anomalies highlydepends on the nature of the subtlety of theanomaly. Fig. 13 illustrates this: anomalies ofdifferent magnitudes were injected into one of theCoBot robot’s wheel encoders: three of the wheelencoders work normally, but the fourth reports(1 − ε)d, where d is the displacement it wouldreport if working normally. Thus, by varying εform −0.5 to −0.1, the encoder reported half of

its displacement, to 90% of its displacement. Asε approaches 0 –i.e., no anomaly–, the detectiontime asymptotically approaches infinity. Fig. 12shows two anomalies: one with ε = 0.1 and onewith ε = 0.4.

Detecting SensorInconsistencies and Secure

State Estimation

This section focuses on malicious false-data-injection (FDI) attacks [45, 46, 47, 48] on thephysical sensing resources in which an adversarypotentially tampers (either remotely by hackinginto the sensor software interfaces or by phys-ically altering the sensing devices) the sensordata. Such attacks, if not detected promptly,might lead to inaccurate estimation of the ve-hicle state (such as its location and velocity) andtrigger incorrect control actions with potentiallydevastating consequences. This section reviewsa class of model-based approaches suited to thecurrent application that use sensor data in con-junction with physics-based information (knowl-edge of vehicle kinematics models and nominalmodels of the sensors) to perform attack detec-tion and secure state estimation. Model-basedapproaches, based on tight integration of sys-tem physics and sensor (data) characteristics,can be effective in terms of performance andimplementability when sensor measurements canbe linked to and represented in terms of phys-ical state variables such as vehicle position andvelocity. However, there might be other sens-ing modalities that may not be readily linkedto the physical characteristics: information from

15

−0.5 −0.4 −0.3 −0.2 −0.1 00

50

100

150

200

250

300

350

400

450

2.8 5.5 7.2 15.2

49.7

396.2

Encoder output error

Tim

e t

o f

au

lt d

ete

ctio

n (

s)

(a)

−0.5 −0.4 −0.3 −0.2 −0.1 00

10

20

30

40

50

60

2.8 5.5

7.2

15.2

49.7

Encoder output error

Tim

e t

o f

au

lt d

ete

ctio

n (

s)

(b)

Figure 13: Time to fault detection as a func-tion of the chosen fractional error ε. (a) showsall the experimental results, while (b) leaves outε = −0.05 for visualization of the remaining data.Error bars show one standard deviation.

these sensors might still contribute to the pri-mary task of inconsistency detection, however,through purely sensor data driven processing.The interested reader may wish to refer to theside bar “Multi Modal Consistency” for addi-tional details.

Overview. Model-based approaches are char-acterized by three crucial elements: dynamicalsystems (state-space) based representations ofthe vehicle kinematics, sensor models (both be-fore and after potential FDI attacks), and the in-consistency detection and secure state estimationmodule. The remainder of this section discussesthese three components in more detail and gives atheorem on detectable and undetectable attacks.The approach is based on linear models and pro-vides an inconsistency detection procedure.

State-space models. A very simplistic ab-straction of the vehicle kinematics may be ob-

tained as

p(t) = p(0) + tv(0) +

∫ t

0

∫ s

0

a(u)duds, (27)

where p(t) and v(t) denote the position and ve-locity vectors respectively at time t (collectivelythe state x(t)), and t = 0 corresponds to the ori-gin of motion with p(0) and v(0) denoting theinitial position and velocity respectively. Thevector a(·) corresponds to the instantaneous ac-celeration and, in control terminology, may beviewed as the input to the system. The acceler-ation may be assumed to be known up to a un-known but bounded (possibly disturbance) factor:in general, in this formulation it is assumed thatat all times t, the deviation between the actuala(t) and its known (predictable) part aknown(t)is norm-bounded by a known constant a. (In theworst case with no knowledge about the instan-taneous acceleration, this constant correspondsto the vehicle’s maximum possible accelerationin the given scenario.)

The important thing to note in the above isthat, assuming the initial state x(0) at time t = 0is known, the state uncertainty at any future timeinstant t may be captured by the relation

x(t) ∈ Ctp(a,x(0)), (28)

where Ctp(·) is a compact convex set dependingon x(0) and a only. In other words, the vehi-cle kinematics provides (predictive) informationabout the system’s state in terms of a boundedset of feasible states around the initial state; theassociated prediction uncertainty is quantified bythe size of Ctp(a,x(0)) which grows with t and a.

Sensor models. In the nominal no-attackscenario, the n-th sensor, n = 1, . . . , N , is as-sumed to measure a noisy linear function of thestate at each sampling instant k∆. Here k,k = 1, 2, . . . , denotes the discrete sampling in-dex and ∆ the sampling period. Formally, theobservation yn(k∆) at the n-th sensor at k∆ ismodeled as

yn(k∆) = Hnx(k∆) + wn(∆), (29)

where the matrix Hn specifies the sensing modal-ity (such as GPS, wheel encoder, or IMU) andwn(∆) the unknown sensing noise. The noisewn(·) is assumed to be norm-bounded but pos-sibly state-dependent, i.e., there exists a con-tinuous function wn(·) of the state such that

16

‖wn(k∆)‖ ≤ wn(x(k∆)) for all k. Commonlyused vehicle sensing resources which depend lin-early on the instantaneous position and veloc-ity may be cast in terms of (29), whereas, thebounded sensing noise is quite realistic for vehic-ular applications.

In the presence of FDI attacks, the sensormodel (29) assumes the following form:

yn(k∆) = Hnx(k∆) + wn(k∆) + bn(k∆), (30)

where bn(k∆) denotes the additional carefullycrafted false data injected by the attacker into thenominal sensor measurements which is unknownto the system operator. Thus, from the systemoperator’s viewpoint, both the sensor noise andthe FDI attack contribute to the uncertainty ofthe measurement. The goal of the operator atany instantK∆ is to use the sensor data collectedover all sensors at all times k∆, k = 1, . . . ,Kin conjunction with the knowledge of the vehiclekinematics to detect whether there has been anattack, i.e., bn(k∆) 6= 0 for some n and k, or not,and simultaneously obtain a feasible estimationof the vehicle state. This leads to inconsistency(attack) detector design discussed next.

Inconsistency detection and secure stateestimation. In the following an optimal (tobe discussed later) online recursive inconsistencydetection and state estimation algorithm is pre-sented. To this end, define for each n and kthe set of feasible vehicle states Xn(yn(k∆)) con-forming to the measurement yn(k∆), i.e.,

Xn(yn(k∆)) ={x : ‖yn(k∆)−Hnx(k∆)‖ ≤ kn(x(k∆))} .

(31)Now, consider the following recursive set mem-bership filtering (RSMF) procedure, which gen-erates recursively at each time instant k∆ a set-valued estimate T (k) of the vehicle’s state x(k∆):

• Initialization: Set T (0) = {x(0)}.

• Update: At each k ≥ 0, define the set

Tp(k + 1) =⋃

x∈T (k)

C1p(a, x), (32)

where the set C1p(·) corresponds to the set-

valued one-step state prediction as a func-tion of the acceleration-related norm-bound

a and past state information T (k) as intro-duced in (28). Now, update T (k) as

T (k+1) = Tp(k + 1)︸ ︷︷ ︸one-step

prediction

N⋂n=1

Xn(yn((k + 1)∆))︸ ︷︷ ︸innovation

.

(33)

• Detection, estimation and termination crite-ria: If T (k + 1) = ∅ declare an attack andterminate; otherwise, declare T (k+ 1) to bethe set of feasible vehicle states at time k+1(in particular, any x ∈ T (k+1) may be takento be an estimate of x((k + 1)∆)) and con-tinue the update step.

Note that, if in a given time horizon [0,K∆],T (k) 6= ∅ for all k = 1, . . . ,K, the test is in-conclusive as to whether or not there has beenno attack, i.e., bn(k∆) = 0 for all n, k: it mightbe possible that the attacker launched an unde-tectable attack trajectory {bn(k∆)}. In fact, un-detectable attacks constitute non-zero attack tra-jectories {bn(k∆)} that are carefully crafted suchthat they induce sensor observations that are fea-sible with respect to nominal or no-attack scenar-ios. The discussion on undetectable attacks willbe revisited, but note, depending on the sens-ing model (the Hn matrices) and the noise char-acteristics, such attacks may exist. These un-detectable attacks, when they exist, correspondto manipulating the sensor observations carefully(by the attacker) as a function of the geome-try of the sensing models and the noise proper-ties so as to induce tampered observations whichnonetheless conform to all physical and sensingconstraints. The following result presents impor-tant properties and optimality of the proposedRSMF algorithm (31)–(33).

Proposition 1 The RSMF procedure outlinedabove satisfies the following properties:

• The procedure is consistent, i.e., if, in agiven time horizon [0,K∆], there is no FDIattack, then T (k) 6= ∅ for all k = 1, . . . ,K.Further, in this case, the set T (k) exactlycorresponds to the set of all feasible sys-tem states x(k∆) (including the true butunknown state x(k∆)) that conform to thevehicle kinematics and (non-attacked) mea-surements in [0,K∆].

17

• The procedure is optimal in the class of con-sistent attack detectors under similar knowl-edge constraints, i.e., in a given time hori-zon [0,K∆], any non-zero attack sequence{bn(k∆)}n,k that is non-detectable by theRSMF procedure is also non-detectable byany other consistent attack detector undersimilar knowledge constraints.

• If the noise norm-bound functions kn(·),see (29), are concave, the sets T (k) are con-vex for all k.

• If the collective observation matrix H =[H>1 H>2 · · · H>N ]>, with > denoting matrixtranspose, has full (row)-rank, the diameterof the set-valued estimation sets T (k) staybounded, i.e., there exists a constant c > 0such that

supk

supx,´x∈T (k)

∥∥∥x− ´x∥∥∥ ≤ c. (34)

Discussion. Implications of Proposition 1are briefly described as follows. The consistencyshows, in particular, the proposed detector haszero false alarm rate. The optimality in the classof all consistent detectors is clearly desirable.The convexity of the T (k) for all k (together withthe fact that the sets stay bounded, see the fi-nal assertion of Proposition 1) implies that thedetection-estimation step at each k (see (33)) re-duces to a convex feasibility problem [49] andhence, may admit efficient numerical implemen-tations such as by using the method of alternateprojections. Finally, the (uniform) boundednessassertion implies that as long as the collectivesensing model is sufficiently informative (essen-tially, an observability condition), the state esti-mation error (obtained by selecting an arbitrarymember of T (k) as the estimate of x(k∆) at eachinstant k∆) under no-attack scenarios (respec-tively in scenarios involving detectable attacks)stays bounded at all times (respectively at alltimes till attack detection).

Undetectable attacks. Returning to the is-sue of attack undetectability, as noted earlier, theexistence of undetectable attacks (and the set ofall undetectable attacks) is, in general, jointly de-termined by the sensing models (the Hn matri-ces) and the noise characteristics. There is an im-portant(sub)class of fundamental undetectable

attacks are undetectable even in the limit of zeronoise. These attacks are solely determined by thegeometry of the sensing models. There is a richliterature on the characterization of such funda-mental undetectable conditions for general lin-ear time-invariant cyber-physical systems of theform studied in this paper [48, 50, 51, 52, 53, 54].More recently, geometric control techniques havebeen employed to characterize FDI attack detec-tion in cyber-physical systems in the presence ofside information and more refined classification ofattacks, for instance, characterizing attacks thatcan be sustained indefinitely without being de-tected and other related topics such as quickestdetection of attacks (see [55]).

Tool Chain and Live DemosThe applicability of the approach discussed inthis article was demonstrated on both the Land-shark robot (shown in Fig. 14), a small scale com-modity military robot, and an American-builtcar. In a series of demonstrations at the end ofPhases I and II of the DARPA HACMS program,the three thrusts of the approach and their inter-dependence were displayed.

Emergency brake monitor. The KeY-maera X and SPIRAL systems were used to gen-erate a monitor that ensures that the car/robotwill not hit an obstacle between the current andsubsequent execution of the monitor. In addi-tion, if the assumed model of the environment nolonger fits the observed environment, the moni-tor initiates an emergency stop. SPIRAL takesthe monitoring expression derived by KeYmaeraX as input and synthesizes a software implemen-tation and the accompanying proof that togetherensure whenever the software says the monitor-ing expression evaluates to false the true mon-itoring expression over the real numbers wouldhave evaluated to false. Thus, the software im-plementation is proven to be conservative. Thiscode is then deployed on the Landshark robotand the American-built car. Fig. 14 shows themoment when the KeYmarea-derived/SPIRAL-synthesized emergency monitor initiates an emer-gency stop of the Landshark robot to avoid hit-ting the obstacle.

This demonstration showed that a formal proofsystem, coupled with a provably correct methodof generating conservative and efficient softwareimplementation, can be used to generate quality

18

Figure 14: The Landshark robot stopping safely in front of an obstacle.

software that can be deployed on an actual pro-duction system. However, without ensuring thatthe inputs into the system are “reasonable” giventhe known operating environment, an adversarycan still fool the monitor into performing out-side of its operating assumptions by providingfalse/spoofed input signals. In the demonstra-tion, the adversary was able to fool the monitorwith false input signals (spoofed GPS that “tele-ported” the robot to a incorrect location), result-ing in the Landshark running over the cone.

Defense against sensor spoofing. To ad-dress this issue, side channel redundancy wasimplemented to detect sensor spoofing. Specif-ically, inputs from the GPS and wheel encoderson the vehicle were fused statistically to detectedwhen the mean of the difference between the twoinput signals deviated beyond a set threshold.These side channel redundancy algorithms weresimilarly generated by SPIRAL from their math-ematical specifications. With the addition ofside channel redundancy to the emergency brakemonitor, changes of GPS values that were incon-sistent with the inputs from the wheel encoderswere detected. The presence of unreliable (i.e.,spoofed) GPS inputs then triggered the emer-gency brakes, which stop the Landshark beforethe problem became catastrophic (i.e. the vehi-cle runs into an obstacle).

Tool chain. A cloud-hosted commercial gradetool chain with KeYmaera X and SPIRAL is ac-cessible through a browser-based IDE (shown inFig. 15). This makes the utilization of side chan-nel redundancy, formal verification, and provablycorrect code generation accessible to a broaderuser base. Using the interface a user can performa variety of tasks, such as studying and runningexamples, modifying existing projects, and build-ing new projects, while the IDE provides levels ofinteraction ranging from click-and-run scripts toa command line window for expert users. Multi-ple users can log into the same instance for col-laborative sessions, and users and projects aresupported by standard scheduling and versioningtools in the cloud environment.

Along with exposing the full functionality ofthe core tools the interface has many of the gen-eral features typical of an IDE, such as context-sensitive menus, multiple tabs, online help, a texteditor with language-specific syntax highlighting,and file downloads.

ConclusionThis article provides an overview of the High As-surance SPIRAL project, which is part of theDARPA HACMS program. The project bringstogether formal verification, code synthesis, andcompilation aspects to provide end-to-end guar-

19

Figure 15: The cloud computing interface to the integrated KeYmaera X and SPIRAL tool chain.The model and code generation of the dynamic window monitor is shown.

antees for control algorithms and safety moni-tors deployed on cyber-physical systems such asunmanned ground and air vehicles and state-of-the-art cars. In addition, the project leveragesrobotics and signal processing algorithms to de-tect attacks and establish trust in the availablesensor readings. Together, the combined ap-proach provides systematic and provable meth-ods for designing controllers for specified desir-able behaviors, generating implementations ofthe controllers with guarentees of correctness inthe presence of floating point errors, and tech-niques and algorithms for detecting inconsisten-cies that may indicate the presence of an at-tacker.

This approach is orthogonal to, and buildsupon traditional IT security defenses such ascommunication encryption and access controls.Most traditional security-in-defence techniquesfocuses on the securing the infrastructure and ap-plications to ensure confidentiality, integrity andavailability of the system. The presented ap-proach provided added assurance in the form ofguaranteed and provable behaviors, the absenceof unintended errors in programming, and highertrust-worthiness of the sensor inputs.

This project also demonstrates that formalmethod techniques can be used to generate pro-

duction quality code of significant complexitythat can be deployed on, and used to oper-ate actual cyber-physical systems. The feasi-bility and power of the presented approach wasdemonstrated at the final Phase I and PhaseII demonstrations of the DARPA HACMS pro-grams, where the team hardened the Landsharkrobot and an American built car to demonstratethe detection of GPS spoofing attacks and guar-anteed passive safety. All implementations ofalgorithms discussed in this article were gener-ated using an cloud-based tool front-end that in-tegrated KeYmaera X and SPIRAL. This pack-aging of formal methods and side channel redun-dancy methods in a user friendly format showsa way forward to deploy these techniques ona larger scale for critical cyberphysical systemsthat require an extra high level of assurance andsafety guarantees.

20

Figure 16: Landshark. The camera, the turret,and the paintball gun rotate.

Sidebar: Multi-ModalConsistency

A data-centric sensor fusion can be adopted todetect multi-modal sensor inconsistency, like in-consistency between a camera view and the ori-entation and posture of a robot. Based on thedata received from the sensors, a model of theworld is built and compared to the inputs from adifferent set of sensors. The model of the worldand the inputs from the second of sensors mustbe consistent or an alarm would be triggered.

An example of this approach is demonstratedon the Landshark ground vehicle. Specifically,the Landshark is equipped with an auxiliary cam-era system that is used to detect inconsistenciesin the values returned by the rotational sensorson the Landshark. It is important to note that,while the images captured by the camera (seedetails below) may not be readily linked to thevehicle physical kinematics as in the model-basedapproach discussed above, the image data can becompared with other invariants to detect incon-sistencies.

The LandShark has four rotation degrees offreedom: 1) camera rotations around the hor-izontal and the vertical rotation axes; 2) turretrotations around the vertical axis; and 3) paint-ball gun rotations around the horizontal rotation.The LandShark has sensors to detect these rota-tion parameters. The key idea is to check theconsistency between the data provided by thesensors and the images captured by the camera.At each time step, the rotation parameters re-turned by the sensors are used to generate car-toon images of what the camera should capture.

(a) (b)

(c) (d)

Figure 17: Two pairs of examples. Real image(17a) and cartoon image (17b) are consistent,showing that the sensors return the correct rota-tion parameters. Real image (17c) and cartoonimage (17d) are inconsistent, showing that thereis an e-attack.

The real images are captured by the camera inthe same time step, and used as reference images.The cartoon images are subsequently comparedwith these real images to check for consistency.If they are inconsistent, the attack is flagged. Ifthey are consistent, the sensors are assumed tobe reliable.

21

Biograpy

Franz Franchetti is an Associate Research Pro-fessor at the Department of Electrical and Com-puter Engineering at Carnegie Mellon Univer-sity. His research focusses on automatic per-formance tuning, program synthesis of math-ematical kernels, and hardware/algorithm co-design. More information can be found athttp://www.ece.cmu.edu/~franzf/.

Tze Meng Low is a Systems Scientist at theDepartment of Electrical and Computer Engi-neering at Carnegie Mellon University. Hisresearch focuses on high performance (hard-ware/software) implementations using analyticaltechniques and formal methods.

Stefan Mitsch is a Postdoctoral Researcher atthe Department of Cooperative Information Sys-tems at Johannes Kepler University and associ-ated with the Logical Systems Lab of CarnegieMellon University. His research focuses on mod-eling, formal verification, and runtime monitor-ing of cyber-physical systems.

Juan Pablo Mendoza is a PhD candidate in theRobotics Institute at Carnegie Mellon University.His research focuses on improving the robustnessof autonomous robots at execution time via exe-cution monitoring, anomaly detection and onlinelearning. See www.cs.cmu.edu/ jmendoza/ forfurther information about his projects and pub-lications.

Liangyan Gui is a PhD student in the De-partment of Electrical and Computer Engineer-ing at Carnegie Mellon University. Her researchfocusses on image processing for cyber-physicalsystems.

Amarin Phaosawasdi is a PhD student in theDepartment of Computer Science at the Univer-sity of Illinois at Urbana–Champaign. His re-search focus are program optimization and veri-fication.

Jason Larkin is a Senior Research Engineer atSpiralGen, Inc. His research interests includemodeling of complex phenomena, high perfor-mance and cloud computing, and open-sourcecollaboration.

David Padua is a Donald Biggar WillettProfessor of Engineering in the Departmentof Computer Science at the University of Illi-nois at Urbana-Champaign. His areas ofinterest include compilers and parallel com-

puting. He is a Fellow of the IEEE andthe ACM. More information can be found athttp://polaris.cs.uiuc.edu/~padua/.

Soummya Kar is an Assistant Research Pro-fessor of ECE at Carnegie Mellon University.His research interests include performance anal-ysis and inference in large-scale networked sys-tems, multi-agent decision theory and stochas-tic systems. More information can be found athttp://www.ece.cmu.edu/~soummyak/.

Jose M. F. Moura is the Philip L. and MarshaDowd University Professor at Carnegie MellonUniversity, with the Electrical and Computer En-gineering. Moura’s research interests are in sta-tistical signal and image processing. His researchinterests are in signal processing and data sci-ence. He is a Fellow from IEEE and AAAS, cor-responding member of the Academia das Cienciasof Portugal, and member of the US NationalAcademy of Engineering. More information canbe found at http://www.ece.cmu.edu/~moura/.

Mike Franusich is Vice President of Engineer-ing at SpiralGen, Inc. in Pittsburgh. His workhas focused on deploying emergent software tech-nologies into vertical markets. More informationcan be found at http://www.spiralgen.com/.

Andre Platzer is an Associate Professor ofComputer Science at Carnegie Mellon Univer-sity. He develops the logical foundations ofcyber-physical systems to characterize their fun-damental principles and to answer the questionhow we can trust a computer to control physi-cal processes. More information can be found athttp://symbolaris.com/andre.html.

Jeremy Johnson is a Professor of ComputerScience and Electrical and Computer Engineer-ing at Drexel University. His research fo-cuses on computer algebra, algebraic algorithms,program synthesis and verification, and high-performance computing and automated perfor-mance tuning. More information can be foundat https://www.cs.drexel.edu/~jjohnson/.

Manuela M. Veloso is the Herbert A. SimonUniversity Professor in the Computer Science De-partment at Carnegie Mellon University. She re-searches in Artificial Intelligence and Robotics,and is IEEE Fellow, AAAS Fellow, AAAI Fellow,and the past President of AAAI and RoboCup.See www.cs.cmu.edu/~mmv for further informa-tion, including publications.

22

References[1] N. Fulton, S. Mitsch, J. Quesel, M. Volp,

and A. Platzer, “Keymaera X: an ax-iomatic tactical theorem prover for hybridsystems,” in Automated Deduction - CADE-25 - 25th International Conference on Au-tomated Deduction, Berlin, Germany, Au-gust 1-7, 2015, Proceedings (A. P. Felty andA. Middeldorp, eds.), vol. 9195 of LNCS,pp. 527–538, Springer, 2015.

[2] A. Platzer, “Logics of dynamical systems,”in LICS, pp. 13–24, IEEE, 2012.

[3] D. Fox, W. Burgard, and S. Thrun, “Thedynamic window approach to collision avoid-ance,” IEEE Robot. Automat. Mag., vol. 4,no. 1, pp. 23–33, 1997.

[4] S. Mitsch, K. Ghorbal, and A. Platzer,“On provably safe obstacle avoidance forautonomous robotic ground vehicles,” inRobotics: Science and Systems (P. Newman,D. Fox, and D. Hsu, eds.), 2013.

[5] S. Mitsch and A. Platzer, “ModelPlex: Ver-ified runtime validation of verified cyber-physical system models,” in Runtime Ver-ification - 5th International Conference, RV2014, Toronto, ON, Canada, September 22-25, 2014. Proceedings (B. Bonakdarpour andS. A. Smolka, eds.), vol. 8734 of LNCS,pp. 199–214, Springer, 2014.

[6] D. Seto, B. Krogh, L. Sha, and A. Chuti-nan, “The Simplex architecture for safe on-line control system upgrades,” in AmericanControl Conference, pp. 3504–3508, 1998.

[7] M. Puschel, J. M. F. Moura, J. Johnson,D. Padua, M. Veloso, B. Singer, J. Xiong,F. Franchetti, A. Gacic, Y. Voronenko,K. Chen, R. W. Johnson, and N. Rizzolo,“SPIRAL: Code generation for DSP trans-forms,” Proceedings of the IEEE, special is-sue on “Program Generation, Optimization,and Adaptation”, vol. 93, no. 2, pp. 232– 275,2005.

[8] F. Franchetti, M. Puschel, Y. Voronenko,S. Chellappa, and J. M. F. Moura, “Dis-crete Fourier transform on multicore,” IEEESignal Processing Magazine, special issue on

“Signal Processing on Platforms with Multi-ple Cores”, vol. 26, no. 6, pp. 90–102, 2009.

[9] M. Puschel, F. Franchetti, and Y. Voro-nenko, Encyclopedia of Parallel Computing,ch. Spiral. Springer, 2011.

[10] R. E. Moore, R. B. Kearfott, and M. J.Cloud, Introduction to Interval Analysis.Philadelphia, PA, USA: Society for Indus-trial and Applied Mathematics, 2009.

[11] H. V. Koops and F. Franchetti, “An ensem-ble technique for estimating vehicle speedand geer position from acoustic data,” inInternational Conference on Digital SignalProcessing (DSP), 2015.

[12] V. Zaliva and F. Franchetti, “Baromet-ric and GPS altitude sensor fusion,” inIEEE International Conference on Acous-tics, Speech and Signal Processing, ICASSP2014, Florence, Italy, May 4-9, 2014,pp. 7525–7529, 2014.

[13] R. Alur, “Formal verification of hybrid sys-tems,” in EMSOFT (S. Chakraborty, A. Jer-raya, S. K. Baruah, and S. Fischmeister,eds.), pp. 273–278, ACM, 2011.

[14] R. Alur, C. Courcoubetis, N. Halbwachs,T. A. Henzinger, P.-H. Ho, X. Nicollin,A. Olivero, J. Sifakis, and S. Yovine, “Thealgorithmic analysis of hybrid systems,”Theor. Comput. Sci., vol. 138, no. 1, pp. 3–34, 1995.

[15] T. A. Henzinger, “The theory of hybrid au-tomata,” in LICS, pp. 278–292, IEEE Com-puter Society, 1996.

[16] J. M. Davoren and A. Nerode, “Logicsfor hybrid systems,” IEEE, vol. 88, no. 7,pp. 985–1010, 2000.

[17] A. Platzer, “Differential dynamic logic forverifying parametric hybrid systems.,” inTABLEAUX (N. Olivetti, ed.), vol. 4548 ofLNCS, pp. 216–232, Springer, 2007.

[18] A. Platzer, “Differential dynamic logic forhybrid systems.,” J. Autom. Reas., vol. 41,no. 2, pp. 143–189, 2008.

23

[19] A. Platzer, Logical Analysis of Hybrid Sys-tems: Proving Theorems for Complex Dy-namics. Heidelberg: Springer, 2010.

[20] A. Platzer, “The complete proof theoryof hybrid systems,” in LICS, pp. 541–550,IEEE, 2012.

[21] A. Platzer, “A uniform substitution calcu-lus for differential dynamic logic,” in CADE(A. P. Felty and A. Middeldorp, eds.),vol. 9195 of LNCS, pp. 467–481, Springer,2015.

[22] S. Mitsch, J.-D. Quesel, and A. Platzer,“From safety to guilty and from liveness toniceness,” in 5th Workshop on Formal Meth-ods for Robotics and Automation, 2014.

[23] G. E. Collins, “Hauptvortrag: Quantifierelimination for real closed fields by cylindri-cal algebraic decomposition,” in AutomataTheory and Formal Languages, 2nd GI Con-ference, Kaiserslautern, May 20-23, 1975(H. Barkhage, ed.), vol. 33 of Lecture Notesin Computer Science, pp. 134–183, Springer,1975.

[24] G. E. Collins and H. Hong, “Partial cylin-drical algebraic decomposition for quanti-fier elimination,” J. Symb. Comput., vol. 12,no. 3, pp. 299–328, 1991.

[25] A. Platzer and J.-D. Quesel, “KeYmaera: Ahybrid theorem prover for hybrid systems.,”in IJCAR (A. Armando, P. Baumgartner,and G. Dowek, eds.), vol. 5195 of LNCS,pp. 171–178, Springer, 2008.

[26] S. M. Loos, A. Platzer, and L. Nistor,“Adaptive cruise control: Hybrid, dis-tributed, and now formally verified,” in FM(M. Butler and W. Schulte, eds.), vol. 6664of LNCS, pp. 42–56, Springer, 2011.

[27] S. Mitsch, S. M. Loos, and A. Platzer, “To-wards formal verification of freeway trafficcontrol,” in ICCPS (C. Lu, ed.), pp. 171–180, IEEE, 2012.

[28] A. Platzer and E. M. Clarke, “Formal ver-ification of curved flight collision avoidancemaneuvers: A case study,” in FM (A. Caval-canti and D. Dams, eds.), vol. 5850 of LNCS,pp. 547–562, Springer, 2009.

[29] J.-B. Jeannin, K. Ghorbal, Y. Kouskoulas,R. Gardner, A. Schmidt, and E. Z. A.Platzer, “A formally verified hybrid sys-tem for the next-generation airborne colli-sion avoidance system,” in TACAS (C. Baierand C. Tinelli, eds.), LNCS, Springer, 2015.

[30] A. Platzer and J.-D. Quesel, “EuropeanTrain Control System: A case study in for-mal verification,” in ICFEM (K. Breitmanand A. Cavalcanti, eds.), vol. 5885 of LNCS,pp. 246–265, Springer, 2009.

[31] Y. Kouskoulas, D. W. Renshaw, A. Platzer,and P. Kazanzides, “Certifying the safe de-sign of a virtual fixture control algorithm fora surgical robot,” in HSCC (C. Belta andF. Ivancic, eds.), pp. 263–272, ACM, 2013.

[32] N. Arechiga, S. M. Loos, A. Platzer, andB. H. Krogh, “Using theorem provers toguarantee closed-loop system properties,” inACC (D. Tilbury, ed.), pp. 3573–3580, 2012.

[33] J.-D. Quesel, S. Mitsch, S. Loos,N. Arechiga, and A. Platzer, “How tomodel and prove hybrid systems withKeYmaera: A tutorial on safety,” STTT,2015.

[34] M. Leucker and C. Schallhart, “A brief ac-count of runtime verification,” J. Log. Al-gebr. Program., vol. 78, no. 5, pp. 293–303,2009.

[35] A. N. Srivastava and J. Schumann, “Soft-ware health management: a necessity forsafety critical systems,” ISSE, vol. 9, no. 4,pp. 219–233, 2013.

[36] N. Dershowitz and D. A. Plaisted, “Rewrit-ing,” in Handbook of Automated Reasoning(A. Robinson and A. Voronkov, eds.), vol. 1,ch. 9, pp. 535–610, Elsevier, 2001.

[37] J. W. Klop, “Handbook of logic in computerscience (vol. 2),” ch. Term Rewriting Sys-tems, pp. 1–116, 1992.

[38] X. Leroy, “Formal verification of a realis-tic compiler,” Communications of the ACM,vol. 52, no. 7, pp. 107–115, 2009.

24

[39] J. R. Johnson, R. W. Johnson, D. Ro-driguez, and R. Tolimieri, “A methodologyfor designing, modifying, and implementingFourier transform algorithms on various ar-chitectures,” Circuits, Systems, and SignalProcessing, vol. 9, no. 4, pp. 449–500, 1990.

[40] F. Franchetti, F. de Mesmay, D. Mc-Farlin, and M. Puschel, “Operator lan-guage: A program generation framework forfast kernels,” in IFIP Working Conferenceon Domain Specific Languages (DSL WC),vol. 5658 of Lecture Notes in Computer Sci-ence, pp. 385–410, Springer, 2009.

[41] T. Nipkow, M. Wenzel, and L. C. Paulson,Isabelle/HOL: a proof assistant for higher-order logic. Berlin, Heidelberg: Springer-Verlag, 2002.

[42] “The coq proof assistant reference manual,”2009.

[43] M. Veloso, J. Biswas, B. Coltin, S. Rosen-thal, and R. Ventura, “Cobots: Collabora-tive robots servicing multi-floor buildings,”in International Conference on IntelligentRobots and Systems (IROS), October 2012.

[44] J. P. Mendoza, M. Veloso, and R. Simmons,“Focused optimization for online detectionof anomalous regions,” in Proceedings of theInternational Conference on Robotics andAutomation (ICRA), (Hong Kong, China),June 2014.

[45] A. A. Cardenas, S. Amin, and S. Sastry,“Research challenges for the security of con-trol systems,” in Proceedings of the 3rd Con-ference on Hot Topics in Security, (San Jose,CA), pp. 1–6, July 2008.

[46] A. A. Cardenas, S. Amin, Z. Lin, Y. H.and. C. Huang, and S. Sastry, “Attacksagainst process control systems: Risk as-sessment, detection, and response,” in Pro-ceedings of the 6th ACM Symposium on In-formation, Computer and CommunicationsSecurity, (Hong Kong), pp. 355–366, Mar.2011.

[47] K. Koscher, A. Czeskis, F. Roesner, S. Pa-tel, T. Kohno, S. Checkoway, D. McCoy,B. Kantor, D. Anderson, H. Shacham, and

S. Savage, “Experimental security analysisof a modern automobile,” in Proceedings ofthe 2010 IEEE Symposium on Security andPrivacy, (Oakland, CA), pp. 447–462, May2010.

[48] A. Teixeira, D. Perez, H. Sandberg, andK. H. Johansson, “Attack models and sce-narios for networked control systems,” inProceedings of the 1st ACM InternationalConference on High Confidence NetworkedSystems, (Beijing, China), pp. 55–64, Apr.2012.

[49] H. H. Bauschke and J. M. Borwein, “Onprojection algorithms for solving convex fea-sibility problems,” SIAM review, vol. 38,no. 3, pp. 367–426, 1996.

[50] F. Pasqualetti, F. Dorfler, and F. Bullo, “At-tack detection and identification in cyber-physical systems,” IEEE Transactions onAutomatic Control, vol. 58, pp. 2715–2729,Nov. 2013.

[51] Y. Mo and B. Sinopoli, “Integrity attackson cyber-physical systems,” in Proceedingsof the 1st ACM International Conference onHigh Confidence Networked Systems, (Bei-jing, China), pp. 47–54, Apr. 2012.

[52] Y. Mo and B. Sinopoli, “False data injectionattacks in control systems,” in Proceedingsof the 1st Workshop on Secure Control Sys-tems, (Stockholm, Sweden), pp. 56–62, Apr.2010.

[53] A. Teixeira, I. Shames, H. Sandberg, andK. H. Johansson, “Revealing stealthy at-tacks in control systems,” in Proceedings ofthe 50th Annual Allerton Conference, (Mon-ticello, IL), pp. 1806–1813, Oct. 2012.

[54] Y. Chen, S. Kar, and J. M. F. Moura,“Cyber-physical systems: Dynamic sensorattacks and strong observability,” in Pro-ceedings of the 40th International Confer-ence on Acoustics, Speech and Signal Pro-cessing, (Brisbane, Australia), pp. 1752–1756, Apr. 2015.

[55] Y. Chen, S. Kar, and J. M. F. Moura,“Dynamic attack detection in cyber-physicalsystems with side initial state information.”

25

IEEE Transactions on Automatic Control.Submitted. Initial Submission: Mar. 2015.Revised: Dec. 2015. [Online]: http://

arxiv.org/pdf/1503.07125v1.pdf, Mar.2015.

[56] “Black-i Robotics LandShark UGV.”

26