Automated Unit Test Generation

Vassil Popovski, VMware

The Cost of Software Bugs?

$59.9 B per year in USA (NIST, 2002)*

(*) http://www.nist.gov/director/planning/upload/report02-3.pdf

$312 B per year worldwide (Cambridge University, 2012)**

(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?entryid=1606&command=core_download&PortalId=0&TabId=607

$312,000,000,000

The Cost of Software Bugs?

The Cost of Software Bugs?

Total market capitalization for all 3 companies(*):

!

$177.36 B

(*) Google finance - as of Nov 2nd, 2013

The Cost of World Hunger?

$40B per year(*)

(*) According to United Nations

The Cost of Fixing Bugs?

http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/0201485427?ie=UTF8&s=books&qid=1209056706&sr=1-1

Software Testing Pyramid

http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/

Unit Testing: 80-90%

Integration testing: 4-15%

E2E/GUI testing: 1-5%

Techniques/Tools

1. Random data generation 2. Search based 3. Symbolic/Concolic based

The goal: High (close to 100% code coverage*)

Anatomy of a Unit Test

Arrange Act Assert

Anatomy of a Unit Test

Arrange Data Arrange Mocks/Stubs Act Assert

Anatomy of a Unit Test

Evaluation of Tools

Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results

Techniques/Tools

x != 100 x == 100

x>=z x<z

Techniques/Tools

x>=z x<z

Random data generation: !

Feed x,y with random data x != 100 x == 100

Demo

Evaluation of Tools

Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results

yes - some

no

no

no

junit-quickcheck feed4JUnit

Techniques/Tools

1. Random Data Generation: • junit-quickcheck (Java) https://github.com/pholser/junit-quickcheck/ • Feed4JUnit & Feed4TestNG + Benerator (java) http://databene.org/feed4-tools.html http://databene.org/databene-benerator.html • Randoop (Java + .NET) https://code.google.com/p/randoop/ • T2 (Java) https://code.google.com/p/t2framework/

Techniques/Tools

x != 100 x == 100

x>=z x<z

Search based approach: !

1. Start with random/initial data (x=1, y=1)

Techniques/Toolsx=1, y=1

x != 100 x == 100

x>=z x<z

Search based approach: !

1. Start with random/initial data (x=1, y=1)

Techniques/Toolsx=1, y=1

x != 100 x == 100

x>=z x<z

Search based approach: !

1. Start with random/initial data (x=1, y=1)

Techniques/Tools

Search based approach: !

1. Start with random/initial data (x=1, y=1)

2. Based on coverage information - select new value(s) (x==100)

x=100, y=1

x != 100 x == 100

x>=z x<z

Techniques/Tools

Search based approach: !

1. Start with random/initial data (x=1, y=1)

2. Based on coverage information - select new value(s) (x==100) !

x=100, y=1

x != 100 x == 100

x>=z x<z

Demo

Evaluation of Tools

Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results

yes - some

no

no

no

EvoSuite

yes

no

yes

yes

junit-quickcheck feed4JUnit

Techniques/Tools

2. Search Based: • EvoSuite (Java)

http://www.evosuite.org/ • Testful (Java)

https://code.google.com/p/testful/

Techniques/Tools

x != 100 x == 100

x>=z x<z

Symbolic based approach: !1. Evaluate path conditions (PC)

on every branch PC: X!=100, Y PC: X == 100, Y

x=X, y=Y, z=2*Y

PC: X == 100, X>=2*Y

PC: X == 100, X<2*Y

1. X!=100, Y => (x=1, y=1) 2. X=100, X>=2*Y =>(x=100, y=1) 3. X=100, X<2*Y =>(x=100, y=51)

2. For each leaf - calculate X & Y using a constraints solver:

Techniques/Tools

x != 100 x == 100

x>=z x<z

Concolic based approach: !

Optimization of symbolic execution - but use real values for some of the branches

Demo

Evaluation of Tools

Arrange/Generate DataArrange/Generate Mocks/StubsAct/Invoke methodsAssert/Verify the results

yes - some

no

no

no

EvoSuite

yes

no

yes

yes

PEX (+ Moles)

yes

yes - some

yes

yes

junit-quickcheck feed4JUnit

Techniques/Tools

3. Symbolic/Concolic based: • Microsoft Pex (.NET) http://research.microsoft.com/en-us/projects/pex/ • jCUTE (Java) http://osl.cs.uiuc.edu/software/jcute/ • jalangi (Java Script) https://github.com/SRA-SiliconValley/jalangi • AgitarOne (Java, commercial) http://www.agitar.com/solutions/products/agitarone.html • Jtest (Java, commercial) http://www.parasoft.com/jsp/products/jtest.jsp

!

Best Usage of Auto Generated Unit Tests

1.Regression testing of legacy code !

2.To support your unit testing / TDD

Q&A

Thank you !

Vassil Popovski !

Sr. Director, R&D, VMware vpopovski@vmware.com