Christoph M. Wintersteiger

Floating-point arithmetic

• Types• Half (16), Float(32), Double (64), Quadruple (128 bit)• double x = 1.5;

• Variables• E.g., float x; double y;

• Operations

• Special values:

• 5 Rounding Modes• Ties toward odd, Ties toward even, to negative, to positive, to zero

Floating-point arithmetic

• Operator semantics are defined on reals• Rounding when necessary• (predefined rounding modes only)

• E.g., (fp.add rm a b)

• Verification• Rounding introduces error• Unintuitive and unexpected results• Lacks `nice’ mathematical properties (e.g., associativity)• E.g.

FPA semantics

• Approximation of real numbers• Standards: IEEE754 vs SMT

• Special exponents (IEEE754)• = 0…0 for ±zero and denormal/subnormal numbers• = 1…1 for ±oo and NaN

FPA representation

sign exponent significand

(−1 ) sign ⋅significand ⋅2exponent

• TypesAny combination sbits/ebits: (_ FloatingPoint ebits sbits)Single: (_ FloatingPoint 8 24), also Float32Double: (_ FloatingPoint 11 53), also Float64

• Variables(define-fun x () (_ FloatingPoint 8 24))(define-fun y () Float32)

SMT floating-point arithmetic

(fp.add rm x y)(fp.sub rm x y)(fp.mul rm x y)(fp.div rm x y)(fp.sqrt rm x)(fp.rem x y)(fp.fma rm x y z)…

Fused Multiply-Add:

+ ,− ,∗ , ÷ ,% ,𝑎𝑏𝑠 ,√❑ , 𝑓𝑚𝑎 ,𝑚𝑖𝑛 ,𝑚𝑎𝑥 ≤<¿>≥SMT floating-point arithmetic

(fp.isNormal x)(fp.isSubnormal x)(fp.isZero x)(fp.isInfinite x)(fp.isNegative x)…

(fp.leq x y)(fp.lt x y)(fp.geq x y)(fp.gt x y)(fp.eq x y)

• Special values:

• Rounding modesRNE, roundNearestTiesToEvenRNA, roundNearestTiesToAwayRTP, roundTowardPositiveRTN, roundTowardNegativeRTZ, roundTowardZero

SMT floating-point arithmetic

(_ NaN ebits sbits)(_ -zero ebits sbits), (_ +zero ebits sbits)(_ -oo ebits sbits), (_ +oo ebits sbits)

double Kp=1.0, Ki=0.25;double set_point=20.0, integral=0.0;double error, in, out;

for (int i=0; i < N; i++) { in = read_input(); error = set_point – in; integral = integral + error; out = Kp*error + Ki*integral; set_output(out);}

An example

then out

If in

• Verification• Unwind loops• No quantifiers

• Convert• QF_FP -> QF_BV• QF_BV -> SAT

Example strategy

• Good approximate results• Real arithmetic• Fixed-point arithmetic• Reduced precision FPA• Interval analysis

(declare-fun Kp () (_ FloatingPoint 5 11))(declare-fun Kp () (_ FloatingPoint 5 11))(declare-fun …

(define-fun rm () RoundingMode RNE)

(assert (= c18 (fp #b0 #b10011 #b0010000000))) ;; = 18.0(assert (= c22 (fp #b0 #b10011 #b0110000000))) ;; = 22.0(assert (= cp3 (fp #b0 #b10000 #b1000000000))) ;; = +3.0(assert (= cn3 (fp #b1 #b10000 #b1000000000))) ;; = -3.0

(assert (= Kp (fp #b0 #b01111 #b0000000000))) ;; = 1.0(assert (= Ki (fp #b0 #b01101 #b0000000000))) ;; = 0.25(assert (= set_point (fp #b0 #b10011 #b0100000000))) ;; = 20.0(assert (= integral (_ +zero 5 11))) ;; = +0.0

An example in SMT2

(assert (not (=> (and(fp.leq c18 in)(fp.leq in c22)(fp.eq error_post (fp.sub rm set_point in))(fp.eq integral_post (fp.add rm integral error_post))(fp.eq out (fp.add rm (fp.mul rm Kp error_post)

(fp.mul rm Ki integral_post))))(and

(fp.leq cn3 out)(fp.leq out cp3))

)))

> z3 pi_controller_1.smt2unsat

An example in SMT2

• fp.add is commutative?(assert (not (fp.eq (fp.add RNE x y) (fp.add RNE y x))))

> z3 add_is_commutative.smt2unsat

• fp.add is associative?(assert (not (fp.eq (fp.add RNE (fp.add RNE x y) z) (fp.add RNE x (fp.add RNE y z)))))

> z3 add_is_associative.smt2satmodel (define-fun z () (_ FloatingPoint 5 11) (fp #b0 #b11010 #b1101111011)) ;; 1.8701171875 * 2^11…

Checking properties

• Approximations• Real arithmetic• Fixed-point arithmetic• Reduced precision FPA• Abstract interpretation-based

(Astrée)

• Precise• Translation to bit-vectors and/or

SAT(MathSAT, Z3, Sonolar, CBMC)

• Mixed abstractions (CBMC)• Abstract CDCL (MathSAT)• Translation to reals (Realizer)

Related techniques• Theorem prover-based (Gappa)• Proof assistants (Coq, HOL)• Often not sound or complete

1. Translate to QF_BV(build circuits)

2. Translate to SAT (e.g., Tseitin translation, bit-blasting)

3. Run SAT Solver4. Translate models/proofs

(back to QF_BV, then QF_FP)

Solving QF_FP via SAT

Circuit for and method of providing a floating-point adder

US 8463835 B1 (Xilinx)

• Conversion QF_FP -> QF_BV -> SAT

Example performance

Bad

1. Mixed abstractions (CBMC)2. Non-conservative approximations (Z3)3. Abstraction into interval arithmetic

(MathSAT)4. Translation into non-linear reals (Realizer)

Advanced QF_FP solving

Abstraction refinementSolve abstraction(s)

Check model

Refine abstraction(s)(model-guided)

Model

satRefine abstraction(s)(proof- or core-guided)

Proof

unsat

no refinement possiblefailed

• Abstract the problem• By reduction of significand precision• Adjust rounding modes• Replacing by fresh Boolean yields over-approximation• Restrict to non-rounded results yields under-approximation

• Over- and under-approximation• Solve them at the same time

• Refinement• Removal of under-approximation constraints• Increase of significand precision

Mixed abstractions

Brillout, Kroening, Wahl: Mixed Abstractions for Floating-Point Arithmetic. FMCAD 2009

Mixed abstractions

Brillout, Kroening, Wahl: Mixed Abstractions for Floating-Point Arithmetic. FMCAD 2009

sign exponent significand

sign exponent significand

ORMURM

RM

Small-float approximation

sign exponent significand

sign exponent significandRM

RM

Zeljic, Wintersteiger, Rümmer: Approximations for Model Construction. IJCAR 2014

For the theory of floating-point numbers:

Small-floats inclusion

𝑠≤𝑠 ′∧𝑒≤𝑒′⇒𝐹 𝑃 𝑠 ,𝑒⊆𝐹 𝑃𝑠 ′ ,𝑒′

Smaller bit-width makes subsets of numbers.

Zeljic, Wintersteiger, Rümmer: Approximations for Model Construction. IJCAR 2014

Approximation framework

𝑀⊨𝑇Φ

�̂�⊨𝑇Φ̂

liftreconstruct

No implication!May fail

Zeljic, Wintersteiger, Rümmer: Approximations for Model Construction. IJCAR 2014

Lifting

𝑦=𝑥+1.75∧𝑦>0∧(𝑥=2.0∨𝑥=−4.0)

∧

¿ ¿ ∨

𝑦 +¿ 0𝑝0

𝑥

𝑝1 ¿ ¿𝑦

1.75𝑝2 𝑥 2.0𝑝3 𝑥 −4.0𝑝4

Example precision terms

(fp.op a_1 … a_n)

(fp.op p a_1 … a_n)And

Refinement schemeSolve approximation

Reconstruct model

Refine approximation(model-guided)

Model

satRefine approximation(proof- or core-guided)

Proof

unsat

no refinement possiblefailed

• Search for• Approximate model• With all precisions assigned to values• (not necessarily the same)• Implementation: precision 0 =

• Could also• Optimize toward fewer rounding occurrences• Search for small values first• Use real numbers• And many more…

Approximate model construction

Model-guided refinementSolve approximation

Reconstruct model

Refine approximation(model-guided)

Model

satRefine approximation(proof- or core-guided)

Proof

unsat

no refinement possiblefailed

Precise model reconstruction

𝑦=𝑥+1.75∧𝑦>0∧(𝑥=2.0∨𝑥=−4.0)

∧

¿ ¿ ∨

𝑦 +¿ 00

𝑥

0 ¿ ¿𝑦

1.750 𝑥 2.00 𝑥 −4.00

2.0 3.5(

𝜔

𝜔

𝜔

𝜔 𝜔

𝑦≔3.5

3.75

• Choose terms to refine• Check which terms introduce imprecision•

• Increase precision of highest ranked term(s)• Error • propagated from a term’s children• or newly introduced by the term

Model-guided refinement

Proof-guided refinementSolve approximation

Reconstruct model

Refine approximation(model-guided)

Model

satRefine approximation(proof- or core-guided)

Proof

unsat

no refinement possiblefailed

• Various levels of granularity• Increase precision of all terms• … those that appear in the unsat core• … those that appear in the proof• … as expensive as you like• Trade-off between precision and efficiency

Proof-guided refinement

• Built upon existing decision procedure• Preserved: soundness, completeness

• Solving more but smaller problems often faster

• Candidate models provide hints for the refinement

• Also preserves termination• If is a well-founded relation• And there is progress in every iteration

Algorithm properties

Fröhlich, Kovásznai, Biere: More on the Complexity of Quantifier-Free Fixed-Size Bit-Vector Logics with Binary Encoding. CSR 2013Kovásznai, Fröhlich, Biere: On the Complexity of Fixed-Size Bit-Vector Logics with Binary Encoded Bit-Width. SMT 2012

A note on complexity

∃𝑥 𝑓∈𝐹𝑃𝐴 (11 ,53) .𝜙 (𝑥)

∃𝑥𝑏𝑣∈𝐵𝑉 64 .𝜙 ′ (𝑥)

∃𝑥0 ,…,𝑥31∈𝔹 .𝜙 ′ ′(𝑥0 ,…,𝑥31)

𝑛

2𝑛

𝑛

QF_BV: NEXPTIME

SAT: NP

QF_FP: (NEXPTIME)

• Crucial for success• “Close” models • Effective model reconstruction

• Incomplete solvers• Can be efficient when starting “close” to a solution• Can find solutions quickly (or we abort them early)• Examples:• Local-search, “cheap” optimization• Perhaps Horn or 2-CNF abstractions

Non-conservative approximation

Fröhlich, Biere, Wintersteiger, Hamadi: Stochastic Local Search for Satisfiability Modulo Theories. AAAI 2015

• Decision procedure framework• Concepts from abstract interpretation• Abstract values from abstract domains• Fixed-point computation over abstract values• Meet irreducibles (~ invertible constraints)• Very nice theory

• For FP-ACDCL• Half-open floating-point intervals, e.g., • Combined into intervals, e.g., • MathSAT

Abstract CDCL

Brain, D’Silva, Griggio, Haller, Kroening: Interpolation-Based Verification of Floating-Point Programs with Abstract CDCL. SAS 2013Brain, D’Silva, Griggio, Haller, Kroening: Deciding floating-point logic with abstract conflict driven clause learning. FMSD 45(2) 2014

FP-ACDCL

Decision Propagation

Analysis

𝑥 (…∧𝑥 )→ 𝑦

(… )→¬𝑥

𝑥∈[1.0 ,2.0] (… )→ 𝑦∈[4.0 ,5.0]

(… )→ 𝑥∈[2.0 ,3.0]

Realizer• Translates QF_FP into QF_NIRA• Satisfiability-equivalent• Eliminates all floating-point terms• Uses for real-valued • Normalization• Rounding

Leeser, Mukherjee, Ramachandran, Wahl: Make it real: Effective floating-point reasoning via exact arithmetic. DATE 2014

Mixed abstraction performance

Brillout, Kroening, Wahl: Mixed Abstractions for Floating-Point Arithmetic. FMCAD 2009

FP-ACDCL performance

Brain, D’Silva, Griggio, Haller, Kroening: Interpolation-Based Verification of Floating-Point Programs with Abstract CDCL. SAS 2013Brain, D’Silva, Griggio, Haller, Kroening: Deciding floating-point logic with abstract conflict driven clause learning. FMSD 45(2) 2014

• Comparison• Bit-blasting (in Z3)• FP-ACDCL (MathSAT)• Small-float approximation (in Z3)

• Benchmark sets• Verification of C programs• Randomly generated systems of

polynomial inequalities

Experimental evaluation

SAT UNSAT

Z3 76 56

MathSAT 76 76

Ours 86 46

Data (SAT)

Data (UNSAT)

Complexity of operations5.3 Homogeneous general-computational operations 5.3.0

5.3.1 General operations 5.3.1.0

Implementations shall provide the following homogeneous general-computational operations for all supported arithmetic formats; …

― sourceFormat remainder(source, source)When y ≠ 0, the remainder r = remainder(x, y) is defined for finite x and y regardless of the …

… FPREM1 … can reduce the exponent of ST(0) by no more than 63 in one executionof the instruction … Software can re-execute the instruction (using the partial remainder in ST(0) as the dividend) until …IEEE 754-2008, Sec 5.3.

Intel® 64 and IA-32 Architectures Software Developer’s Manual. May 2012

Conclusion• SMT2 Floating-point standard• Solvers are starting to support it• Various solving approaches

• Near future: challenging problems• Out-of-the-box performance• Theory combination• Quantifiers• Real error bounds• Non-linearity

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.