An exploratory study of the state of practice of performance testing in Java-based open source projects

An exploratory study of the state of An exploratory study of the state of practice of performance testing in Java-practice of performance testing in Java-

based open source projectsbased open source projects

Cor-Paul BezemerPhilipp Leitner

Industrial applications depend more and Industrial applications depend more and more on open source softwaremore on open source software

Is the dependency on open source software Is the dependency on open source software justified (or wise)? justified (or wise)?

We know (more or less) how to We know (more or less) how to assess quality of functionality...assess quality of functionality...

But there is no standardized way of But there is no standardized way of assessing performance!assessing performance!

What is the state of practice of What is the state of practice of performance testing in Java-based performance testing in Java-based

open source software?open source software?

We conducted an exploratory We conducted an exploratory study on 111 Java-based projectsstudy on 111 Java-based projects

We conducted an exploratory We conducted an exploratory study on 111 Java-based projectsstudy on 111 Java-based projects

‘bench’ or ‘perf’ in the src/test directory

0

20

40

60

<= 10

0

101 −

500

501 −

2000

2001

− 50

00> 5

000

# of Total Commits

# of

Pro

ject

s

Commits

0

20

40

60

12 −

56 −

10

11 −

25 > 25

# of Individual Contributors

# of

Pro

ject

s

Individual Contributors

0

20

40

60

<= 25

26 −

50

51 −

100

101 −

500

> 500

# of Stars

# of

Pro

ject

s

Stars

0

20

40

60

<= 25

26 −

50

51 −

100

101 −

500

> 500

# of Watchers

# of

Pro

ject

s

Watchers

We manually identified We manually identified (performance) tests(performance) tests

● Following process:– Search the test files for performance-related terms

● ‘perf’, ‘bench’, ‘fast’, ‘speed’, etc.

– Manually identify performance tests

We studied performance testing We studied performance testing from five perspectivesfrom five perspectives

1. The developers who are involved

2. The extent of performance testing

3. The organization of performance tests

4. The types of performance tests

5. The tools used

Perspective 1: DevelopersPerspective 1: Developers

Performance tests are usually done by a small group of developers (median 2 vs. median 9 developers)

Perspective 1: DevelopersPerspective 1: Developers

Performance tests are usually done by a small group of developers (median 2 vs. median 9 developers)

Performance testers are usually core developers of the project

Perspective 2: The extent of testingPerspective 2: The extent of testing

The performance test suite is usually small– A few hundred LOC compared to a few thousand LOC

for the functional test suite

– Exceptions are Hadoop and Deuce STM (Software Transactional Memory)

Perspective 2: The extent of testingPerspective 2: The extent of testing

The performance test suite is usually small– A few hundred LOC compared to a few thousand LOC

for the functional test suite

– Exceptions are Hadoop and Deuce STM (Software Transactional Memory)

Projects that claim to be the fastest/most efficient do not seem to take extra measures to support their claims

Perspective 3: OrganizationPerspective 3: Organization

There is no standardized way of organizing and conducting performance tests– Performance tests are scattered throughout the (test)

code

– Previous results and instructions for executing a performance test are included in code comments

Perspective 4: Types of performance testsPerspective 4: Types of performance tests

Type 1: Performance smoke test (50% of the projects)― Test a particular use case

Type 2: Microbenchmarks (32%)― Test smaller units of code

Type 3: One-shot performance tests (15%)― Test a known buggy case

Type 4: Performance assertions (5%)― ‘4x as fast as …’

Type 5: Implicit performance tests (5%)― ‘Test’ using data that is generated as a side-effect (e.g., execution time of a test)

























Perspective 5: ToolsPerspective 5: Tools

Approach 1: Unit testing (51% of the projects)– Usually using JUnit instead of a ‘dedicated’ performance

unit test framework

Approach 2: Stand-alone performance tests (50%)– Custom written for the project

Approach 3: Dedicated performance testing framework (16%)– Usually Caliper or JMH



unit test framework





unit test framework



ImplicationsImplications

There is a lack of a ‘killer app’ for performance testing

Writing performance tests is not a popular task in open source projects

Developers want support for quick-and-dirty performance testing

Performance testing is multi-faceted

Integration into standard CI-frameworks is key

























ConclusionConclusion

● Open source developers seem to be not very enthusiastic about writing performance tests

● There is a lack of a ‘killer app’ for performance testing

What we are doing as researchers, does not seem to reach practice!

Cor-Paul Bezemer, [email protected]

http://sailhome.cs.queensu.ca/~corpaul/

mailto:[email protected]




















