FOREPOST: A Tool For Finding Performance Problems Automatically with Feedback-

Directed Learning Software Testing

Qi Luo, Denys Poshyvanyk, Aswathy Nair*, Mark Grechanik*

College of William and Mary

*University of Illinois at Chicago

ICSE 2016

Austin, TX, U.S.

1

Performance Testing

Inputs Subjects

2

Profiling tools

Performance Testing

Inputs Subjects

3

Performance bottlenecks

Profiling tools

1. Agilefant.model.WidgetCollection.getName() 273.2s2. Agilefant.db.hibernate.UserTypeFilter.deepCopy() 213.5s3. Agilefant.model.Team.setId() 192.3s4. Agilefant.model.Backlog.setChildren() 123.9s5. ……

4

78 Million customer profiles

5

78 Million customer profiles

902 Days !!!

6

Facts:

1. California and Texas

7

Facts:

1. California and Texas2. Southwestern

8

Facts:

1. California and Texas2. Southwestern 3. Oklahoma

9

Facts:

1. California and Texas2. Southwestern 3. Oklahoma

WILDFIRE => bottlenecks

• Using machine learning algorithms to extract rules for selecting test input data

• Using Independent Component Analysis (ICA) to identify performance bottlenecks automatically

Feedback-ORiEnted PerfOrmanceSoftware Testing (FOREPOST)

10

• Using machine learning algorithms to extracts rules for selecting test input data

• Using Independent Component Analysis (ICA) to identify performance bottlenecks automatically

Input: test input data and binary codeOutput: a set of rules, a ranked list of methods

11

Feedback-ORiEnted PerfOrmanceSoftware Testing (FOREPOST)

FOREPOST Foundation

ICSE’12EMSE’16

12

13

FOREPOST

14

FOREPOST

15

FOREPOST

16

FOREPOST

17

FOREPOST

• Test input data comes from existing repositories or databases

• Test scripts

18

19

FOREPOST

The Eclipse Test & Performance Tools Platform (TPTP) Profiling tool

20

The Eclipse Test & Performance Tools Platform (TPTP) Profiling tool

21

22

FOREPOST

23

24

FOREPOST

Examples of rules:• (childOrAdultCareDetails.numberPersonsCaredForChild>3)

=> Good• (browse_title_ACADEMY_AFRICAN_2 > 5) => Bad• (viewPrdct_K9RT01 > 5) and (viewItem_EST16 > 5) => Bad• (storeStory3 > 100) and (storeProject6 < 25) => Bad

25

26

27

FOREPOST

28

FOREPOST

Bottlenecks => the most significant methods that occur in good traces but are not invoked or have little significance in bad traces

29

30

Traces

Elap

sed

Exe

cuti

on

Tim

e

Bad

Good

Bad Bad

Good

31

bottleneck

Bad

Good

Bad Bad

Good

Elap

sed

Exe

cuti

on

Tim

e

Traces

32

33

Experimental Design

JPetStore

Agilefant

Dell DVD Store

Renters

34

Experimental Design

• Effectively finding input test data • FOREPOST & Random

• Identifying performance bottlenecks effectively• Injected performance bottlenecks• Real performance bottlenecks

35

Experimental Results

• Effectively finding input test data that expose performance bottlenecks

• Identifying performance bottlenecks effectively

36

http://www.cs.wm.edu/semeru/data/ICSE16-FOREPOST/

Thank you!

Tool, source code, more materials:

37