Constraint Based Synthesis · Synthesis: modern view = ∧ ∧ ∧ Space of programs Safety...

Constraint Based Synthesis

Armando Solar-Lezama

Synthesis: 1980s view

Complete Formal Specification

Synthesis: modern view

= ∧∧∧Space of programs

Safety Properties

Test Harnesses

Input/Output Examples

SketchC-like languagewith holes and

assertions

/* Average of x and y without overflow */int avg(int x, int y){int t = expr({x/2, y/2, x%2, y%2, 2 }, {PLUS, DIV});assert t == (x+y)/2;return t;

}

/* Operands */ /*Operators*/

/* Average of x and y without overflow */int avg(int x, int y){int t = y / 2 + x / 2 + ((x % 2 + y % 2) / 2);assert t == (x+y)/2;return t;

}

SketchC-like languagewith holes and

assertions

High-LevelLanguage Compiler

UnrollInlineEnumerate

AutomatedTutoring

Program+

Pre/Post cond+

Invariants

VCGenerator

SketchC-like languagewith holes and

assertions

UnrollInlineEnumerate

AutomatedTutoring

ProvablyCorrectSynthesis

ProgramOptimization

AutomatedTutoring

Program+

Properties

ProvablyCorrectSynthesis

ProgramOptimization

SketchC-like languagewith holes and

assertions

UnrollInlineEnumerate

AbstractDomainCompiler

VisualProgramming

AutomatedTutoring

Program+

Properties

ProvablyCorrectSynthesis

ProgramOptimization

SketchC-like languagewith holes and

assertions

UnrollInlineEnumerate

AbstractDomainCompiler

VisualProgramming

Automated TutoringWith Rishabh Singh and Sumit Gulwani

12

Pinpoint student errors

def computeDeriv(poly):deriv = []zero = 0if (len(poly) == 1):

return derivfor e in range(0, len(poly)):

if (poly[e] == 0):zero += 1

else:deriv.append(poly[e]*e)

return deriv

replace derive by [0]

Should be a 1

Get rid of this

def computeDeriv(poly):length = int(len(poly)-1)i = lengthderiv = range(1,length)

if len(poly) == 1:deriv = [0.0]

else:while i >= 0:

new = round((poly[i] * i),2)i -= 1deriv[i] = new

return deriv

Should be a 0

Should be >

This is a synthesis problem

SketchC-like languagewith holes and

assertions

High-LevelLanguage Compiler

UnrollInlineEnumerate

Python

SketchC-like languagewith holes and

assertions

CompilerUnrollInlineEnumerate

Python

Error Model

SketchPython CompilerUnrollInlineEnumerate

struct MultiType{int type;int ival;bit bval;MTString str; MTList lst;MTDict dict; MTTuple tup;

}

ival bval

lst…

Type

int bool

list

Dynamic Types

SketchCompilerUnrollInlineEnumerate

[1,2]

1 -

- -INT

int

list

bool

2 -

- -INT

int

list

bool

- -

len=2, lVals = {*, * } -

LISTint

list

bool

Python

SketchCompilerUnrollInlineEnumerate

Python

x + y addMT(x,y)

15 -

- -

INTint

list

bool

10 -

- -

INTint

list

bool

5 -

- -

INTint

list

bool

SketchCompilerUnrollInlineEnumerate

Python

x + y addMT(x,y)

- -

[1,2] -

LISTint

list

bool

- -

[3] -

LISTint

list

bool

- -

[1,2,3] -

LISTint

list

bool

SketchCompilerUnrollInlineEnumerate

Python

x + y addMT(x,y)

- -

[3] -

LISTint

list

bool

5 -

- -

INTint

list

bool

assert false;

SketchCompilerUnrollInlineEnumerate

Python

Error Model

• return a return [0]

• range(a1, a2) range(a1+1,a2)

• a0 == a1 False

SketchCompilerUnrollInlineEnumerate

Python

Error Model

range(0, len(poly))

range(a1,a2) range(a1+1, a2)

range({0 ,1}, len(poly))

default choice

SketchCompilerUnrollInlineEnumerate

Python

Error Model

{ , } modifyMT0( )

MultiType modifyMT0( ){if(??) returnelse

choice0 = truereturn

}

- -

- -

- -

- -

- -

- -

- -

- -

- -

- -

// default choice

// non-default choice- -

- -

Benchmark Test SetevalPoly-6.00 13

compBal-stdin-6.00 52compDeriv-6.00 103hangman2-6.00x 218prodBySum-6.00 268oddTuples-6.00 344hangman1-6.00x 351evalPoly-6.00x 541

compDeriv-6.00x 918oddTuples-6.00x 1756iterPower-6.00x 2875

recurPower-6.00x 2938iterGCD-6.00x 2988

Average Running Time (in s)

05

101520253035

Tim

e (in

s)

Feedback Generated (Percentage)

0%10%20%30%40%50%60%70%80%90%

Feed

back

Per

cent

age

Invariant Synthesis forOptimizationWork with Alvin Cheung and Shoaib Kamil

Optimization then and now

Naïve source code

Optimal executableKind-of-OK executable

ATLAS Pochoir

Domain specific problem description

Close to optimal implementation

Halide

What about existingsource code?

Source Code DSL ProgramProof ofEquivalence

Synthesis

Java to SQL

Application

SQL Queries

Database

Relations

ORMlibraries

Methods

Objects

Java to SQL

Application

SQL Queries

Database

Relations

ORMlibraries

Methods

Objects

Java to SQL

List getUsersWithRoles () {List users = User.getAllUsers();List roles = Role.getAllRoles();

List results = new ArrayList();

for (User u : users) {

for (Role r : roles) {

if (u.roleId == r.id)

results.add(u); }}

return results; }

SELECT * FROM user

SELECT * FROM role

List getUsersWithRoles () {return executeQuery(“SELECT u FROM user u, role r WHERE u.roleId == r.id

ORDER BY u.roleId, r.id”; }

convert to

Java to SQL

List getUsersWithRoles () {List users = User.getAllUsers();List roles = Role.getAllRoles();

List results = new ArrayList();

for (User u : users) {

for (Role r : roles) {

if (u.roleId == r.id)

results.add(u); }}

return results; }results = outputExpr(users, roles)

Verificationconditions

preConditionouterInvariant(users/query(…), results/[], …)

outerInvariant(…) ∧ outer loop terminatesresults = outputExpr(users, roles) …

outerInvariant(users, roles, u, results, …)

innerInvariant(users, roles, u, r, results, …)

Program+

Unkown Post cond+

Unknown Invariants

VCGenerator

SketchC-like languagewith holes and

assertions

UnrollInlineEnumerate

6/17/2013 PLDI 2013 34

Real-world Evaluation

Wilos (project management application) – 62k LOC

Operation type # Fragments found # Fragments converted

Projection 1 1

Selection 13 10

Join 7 7

Aggregation 11 10

Total 33 28

6/17/2013 PLDI 2013 35

Real-world Evaluation

iTracker (bug tracking system) – 61k LOC

Operation type # Fragments found # Fragments converted

Projection 3 2

Selection 3 2

Join 1 1

Aggregation 9 7

Total 16 12

6/17/2013 PLDI 2013 36

Selection Query

50% selectivity 10% selectivity

100

1K

10K

0 50K 100K

Page

load

tim

e (m

s)

Total number of users in DB

original

inferred

100

1K

10K

0 50K 100K

Page

load

tim

e (m

s)

Total number of users in DB

originalinferred

100

1K

10K

100K

1000K

0 20K 40K 60K 80K 100K

Page

load

tim

e (m

s)

Number of roles / users in DB

original inferred

6/17/2013 PLDI 2013 37

Join Query

Nested-loop join Hash join!O(n2) O(n)

What about HPC?

LegacyFortran/C++Code

Stencil DLS(Halide)

Proof ofEquivalence

Synthesis

Legacy to Halide

for (k=y_min-2;k<=y_max+2;k++) {for (j=x_min-2;j<=x_max+2;j++) {

post_vol[((x_max+5)*(k-(y_min-2))+(j)-(x_min-2))]=volume[((x_max+4)*(k-(y_min-2))+(j)-(x_min-2))]

+ vol_flux_y[((x_max+4)*(k+1 -(y_min-2))+(j)-(x_min-2))]- vol_flux_y[((x_max+4)*(k-(y_min-2))+(j)-(x_min-2))];

}}

∀ , ∈ post_vol[j,k] = volume[j,k]+ vol_flux[j, k+1] + vol_flux[j, k]

Invariants

∀ , ∈ ∶, = ( { , , , } )

for(int i=0; i<n-1; ++i)for(int j=0; j<m-1; ++j)

out[i,j] = sin(in[i,j])

Invariants

∀ , ∈ ∶, = ( { , , , } )

for(int i=0; i<n-1; ++i)for(int j=0; j<m-1; ++j)

out[i,j] = sin(in[i,j])

Invariants

∀ , ∈ ∶, = ( { , , , } )

Challenges

• Big invariants

• Complex floating point arithmetic

• Universal Quantifiers

Quantifiers

∃ ∀ ( )outerInvariant(…) ∧ outer loop terminates→ output= outputExpr(in)

if(outerInvariant(…) && cond()){assert out == outputExpr(in) ;

}

The ∀ leads to a ∃∀∃ constraint!

Quantifiers

∃ ∀ ( )if(outerInvariant(…) && cond()){

assert out == outputExpr(in) ;}

Always safe to weaken this condition (more true)

∀ , ∈ ∶ , = ( { , , , } ), = ( { , , , } ), ∈Let the synthesizer discover !

Does it work?

• Can compute invariants for all stencils in Cloverleaf

• Most complex one takes 20 hrs on 15 Cores

• Currently pushing new benchmarks

??SketchC-like languagewith holes and

assertions

UnrollInlineEnumerate

• There is more to synthesis than “synthesis”

• You can do this too!• All the sketch infrastructure is available in open source