Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2

Test-Driven Synthesis

Daniel Perelman1

Sumit Gulwani2 Dan Grossman1 Peter Provost3

1University of Washington

2Microsoft Research

3Microsoft Corporation

June 11, 2014

TDD example1

Test cases:

1. wrap("word", 5)

== "word"

2. wrap("foobar", 6)

== "foobar"

3. wrap("LongWord", 4)

== "Long\nWord"4. wrap("LongerWord", 6)

== "Longer\nWord"5. wrap("LongerWord", 2)

== "Lo\nng\ner\nWo\nrd"

Program:wrap(string s, int len) {// do nothing

throw new

NotImplementedException();

}

1http://blog.8thlight.com/uncle-bob/2013/05/27/

TheTransformationPriorityPremise.html2 / 49

http://blog.8thlight.com/uncle-bob/2013/05/27/TheTransformationPriorityPremise.html


TDD example1

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"





Program:wrap(string s, int len) {// return constant "word"

return "word";

}





TDD example1

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"





Program:wrap(string s, int len) {// return input string

return s;

}





TDD example1

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"


== "Long\nWord"

4. wrap("LongerWord", 6)



Program:wrap(string s, int len) {// return input if short

if(s.Len ≤ len)

return s;

else

return "Long\nWord";

}





TDD example1

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"



== "Longer\nWord"



Program:wrap(string s, int len) {// split string at len

if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}





TDD example1

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"





Program:wrap(string s, int len) {// wrap word to length len

if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + wrap(s[len:], len);

}





Google’s Real TDD2

2http://googletesting.blogspot.com/2014/04/

the-real-test-driven-development.html8 / 49

http://googletesting.blogspot.com/2014/04/the-real-test-driven-development.html


Google’s Real TDD2 (April Fools’ Day joke)

2http://googletesting.blogspot.com/2014/04/

the-real-test-driven-development.html9 / 49



Example domain: string transformations

“Spreadsheet Data Manipulation using Examples,CACM 2012,

Sumit Gulwani, William Harris, Rishabh Singh”⇓

“Spreadsheet Data Manipulation using Examples\nGulwani, S.; Harris, W.; Singh, R.\nCommunications of the ACM, 2012”

10 / 49

Example domain: table transformations

Qual 1 Qual 2 Qual 3

Andrew 01.02.2003 27.06.2008 06.04.2007

Ben 31.08.2001 05.07.2004

Carl 18.04.2003 09.12.2009

⇓Andrew Qual 1 01.02.2003

Andrew Qual 2 27.06.2008

Andrew Qual 3 06.04.2007

Ben Qual 1 31.08.2001

Ben Qual 3 05.07.2004

Carl Qual 2 18.04.2003

Carl Qual 3 09.12.2009

11 / 49

Example domain: XML transformations

<doc>

1

2

3

4

5

6

7

</doc>

⇒

<doc>

1

2

3

4

5

6

7

</doc>

12 / 49

Workflow

domain expert user

DSL(library+grammar)

test cases

synthesize program

program P

done?

generatenew test

case

output programnoyes

13 / 49

Workflow

domain expert user


test cases

synthesize program

program P

done?

generatenew test

case

output programnoyes

14 / 49

Workflow

domain expert user


test casessequence

sythesize change

program P

done?

generatenew test

case

output programnoyes

15 / 49

Contrast with genetic programming

domain expert user


fitnessfunction f

sythesize change

program P

f (P) ≥ α

generatenew test

case

output programnoyes

16 / 49

Workflow

domain expert user


test casessequence

sythesize change

program P

done?

generatenew test

case

output programnoyes

17 / 49

Workflow

domain expert user


test casessequence

sythesize change

program P

done?

generatenew test

case

output programnoyes

18 / 49

Modifying program

I Replace single subexpression onnew test case’s path

I Where to modify program

I What to replace with

19 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

20 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

21 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

22 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

23 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

24 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

25 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

26 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

27 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

28 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

29 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + s[len:];

}

30 / 49

Where to modify

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"






if(s.Len ≤ len)

return s;

else

return s[0:len] +

"\n" + wrap(s[len:], len);

}

31 / 49

What to put there

I DSL defines space of expressions

I Prefer expressions found inprevious program

I e.g., wrap(s[len:], len) containss[len:]

32 / 49

Conditionals

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"



== "Longer\nWord"



test caseprogram path 1 2 3 4

""

"word" X"foobar" X"Long\nWord" X"Longer\nWord" Xs X Xs[0:len] +

"\n" + s[:len]

X X

guard 1 2 3 4

s == "word" Tlen == 6 T Ts.Len == len Ts.Len ≤ len T T

33 / 49

Conditionals

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"



== "Longer\nWord"




""


"\n" + s[:len]

X X

guard 1 2 3 4


34 / 49

Conditionals

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"



== "Longer\nWord"




""


"\n" + s[:len]

X X

guard 1 2 3 4


35 / 49

Conditionals

Test cases:

1. wrap("word", 5)

== "word"


== "foobar"



== "Longer\nWord"




""


"\n" + s[:len]

X X

guard 1 2 3 4


36 / 49

Conditionals


if(s.Len ≤ len)

return s;

else

return s[0:len]

+ "\n"+ s[len:];

}


""


"\n" + s[:len]

X X

guard 1 2 3 4


37 / 49

Loops


if(s.Len ≤ len)

return s;

else

return s[0:len]

+ "\n"+ wrap(s[len:], len);

}


""


"\n" + s[:len]

X X

guard 1 2 3 4


38 / 49

Loops

I Recursion (if in DSL)

I Higher-order functions (if in DSL)

I Direct support for certain loops:see paper for details

39 / 49

Evaluation

I Three DSLs focused on end-user data transformation tasks.I String transformations (FlashFill+extensions)I Spreadsheet table transformationsI XML transformations

I Able to synthesize many examples from help forums in undera minute.3

I Good news: test case sequences are very short (easy for user)

I Bad news: test case sequences are very short (difficult toevaluate usefulness of sequences)

3https://homes.cs.washington.edu/˜perelman/publications/pldi14-tds.zip40 / 49

https://homes.cs.washington.edu/~perelman/publications/pldi14-tds.zip

Evaluation







Evaluation







Pex4Fun (now https://www.CodeHunt.com/)

domain expert

intro CS DSLtest cases

player modifies program

program

done?

generatenew test

case

win gamenoyes

43 / 49

https://www.CodeHunt.com/

Pex4Fun (now https://www.CodeHunt.com/)

domain expert

intro CS DSLtest casessequence

synthesize change

program

done?

generatenew test

case

win gamenoyes

44 / 49

https://www.CodeHunt.com/

Comparison to human players

0

20

40

60

80

100

0 20 40 60 80 100 120 140 160 180 200

Synth

esiz

er T

ime

(sec

onds)

User Time (seconds)

0

20

40

60

80

100

0 20 40 60 80 100 120 140 160 180 200

Synth

esiz

er T

ime

(sec

onds)

User Time (seconds)

Completion times comparableto human players

45 / 49

Research question

Hypothesis:I Test sequence order matters. . .I . . . but not too much.

46 / 49

Slowdown due to reordering test case sequence

1

10

100

0 0.2 0.4 0.6 0.8 1

Norm

ali

zed e

xecuti

on t

ime

Normalized inversions count

Robust to small reorderings,fails on larger reorderings

47 / 49

Contributions

I Synthesis workflow inspired byTDD

I Search strategy enabled by thisworkflow, parameterized by a DSL

48 / 49

Questions

49 / 49

Backup slides

50 / 49

Component-based synthesis example

Starting components:

I a+b

I 0

I x

I y

Generated components (1 step):

I 0+b

I x+b

I y+b

Generated components (2 steps):

I 0+0

I x+0

I y+0

I 0+x

I x+x

I y+x

I 0+y

I x+y

I y+y

51 / 49



I a+b

I 0

I x

I y


I 0+b

I x+b

I y+b


I 0+0=0

I x+0=x

I y+0=y

I 0+x=x

I x+x

I y+x

I 0+y=y

I x+y=y+x

I y+y

52 / 49



I a+b

I 0

I x

I y


I 0+b

I x+b

I y+b


I 0+0=0

I x+0=x

I y+0=y

I 0+x=x

I x+x

I y+x

I 0+y=y

I x+y=y+x

I y+y

53 / 49



I a+b

I 0=[0,0]

I x=[10,-31]

I y=[-10,31]


I 0+b

I x+b

I y+b


I 0+0=[0,0]

I x+0=[10,-31]

I y+0=[-10,31]

I 0+x=[10,-31]

I x+x=[20,-62]

I y+x=[0,0]

I 0+y=[-10,31]

I x+y=[0,0]

I y+y=[-20,62]

54 / 49



I a+b

I 0=[0,0]

I x=[10,-31]

I y=[-10,31]


I 0+b

I x+b

I y+b


I 0+0=[0,0]

I x+0=[10,-31]

I y+0=[-10,31]

I 0+x=[10,-31]

I x+x=[20,-62]

I y+x=[0,0]

I 0+y=[-10,31]

I x+y=[0,0]

I y+y=[-20,62]

55 / 49

Loops

I Rule-based generation of test cases for loop body from testcases for loop

I For loop example:Factorial(1)=1

Factorial(2)=2

Factorial(3)=6

Factorial(4)=24

⇒Factorial body(2,1)=2

Factorial body(3,2)=6


I Array by-element example:

CSum([6,2,8],[4,3,7])

=[10,15,30]⇒

CSum body(6,4,0,[])=10

CSum body(2,3,1,[10])=15

CSum body(8,7,2,[10,15])=30

56 / 49

Loops



Factorial(2)=2

Factorial(3)=6

Factorial(4)=24





CSum([6,2,8],[4,3,7])

=[10,15,30]⇒

CSum body(6,4,0,[])=10

CSum body(2,3,1,[10])=15

CSum body(8,7,2,[10,15])=30

57 / 49

Loops



Factorial(2)=2

Factorial(3)=6

Factorial(4)=24





CSum([6,2,8],[4,3,7])

=[10,15,30]⇒

CSum body(6,4,0,[])=10

CSum body(2,3,1,[10])=15

CSum body(8,7,2,[10,15])=30

58 / 49

Documents

Test-Driven Synthesis · Qual 1 Qual 2 Qual 3 Andrew 01.02.2003 27.06.2008 06.04.2007 Ben 31.08.2001 05.07.2004 Carl 18.04.2003 09.12.2009 + Andrew Qual 1 01.02.2003 Andrew Qual 2