Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,

Have Things Changed Now?- An Empirical Study of Bug Characteristics in Modern Open

Source Software

Zhenmin Li, Lin Tan, Xuanhui Wang, Shan LuYuanyuan Zhou and Chengxiang Zhai

University of Illinois, Urbana-Champaign

Motivation

Bugs account for 40% of system failures.

Designing debugging tools requires a good understanding of bug characteristics.

Many empirical studies have been performed.

Why another empirical study?

2

Things have changed

Many empirical studies have been performed over 10 years ago

3

Things have changed


More effective modern debugging tools - Valgrind, Purify, Coverity, etc

3

Things have changed



Fewer memory bugs in release code?

3

Things have changed




Rising security concerns

3

Things have changed




Rising security concerns

More security-related bugs?

3

Things have changed Emphasis on user friendly interfaces

4


Have GUI bugs become more pervasive?

4



Software architecture shift - Multithreaded/Multiprocessed

4




More concurrency bugs?

4




More concurrency bugs?

Need a new empirical study to answer these questions

4

Our Contribution Studied two large and popular OSS projects

5

Mozilla Apache


5

Mozilla ApacheProject Type Client Server


5


Major Language C & C++ C


5


Major Language C & C++ CCode Size (MLOC) 4 0.3


5



No. of Releases ~90 ~90


5



No. of Releases ~90 ~90Bug DB start time 1998 2001


5




Manually collected and classified 709 bugs


5




Manually collected and classified 709 bugs

Automatically classified around 29,000 bugs

Our Bug SourcesMozilla Apache

6


GeneralRandomly sampled

548 209

6



548 209

After filtering 264 98

6

After filtering: only fixed run-time bugs with known root causes



548 209


Security All 193 64

6




548 209


Security All 193 64

6


We use information retrieval techniques to collect concurrency bugs and manually classify 90 of them.



548 209


Security All 193 64

6


Total=709We use information retrieval techniques to collect concurrency bugs and manually classify 90 of them.

Classification Dimensions

Root CauseC

ompo

nent

Impact

7


Root CauseC

ompo

nent

Impact

7 Every bug is classified in three dimensions.


Root CauseC

ompo

nent

Impact

MemoryConcurrency

Semantic



Root CauseC

ompo

nent

Impact

MemoryConcurrency

Semantic

Performance DegradationIncorrect Functionality

Data CorruptionCrashHang



Root CauseC

ompo

nent

Impact

MemoryConcurrency

Semantic

CoreGUI

NetworkI/O

Performance DegradationIncorrect Functionality

Data CorruptionCrashHang


Major FindingsConcurrency

Semantic

Memory

8

1992 [Sullivan92]


Semantic

Memory

Memory

8

Memory bugs are decreasing.

1992 [Sullivan92] 2005


Semantic

Memory

Semantic

Memory

8


Semantic bugs are becoming increasingly dominant.

1992 [Sullivan92] 2005


Semantic

Memory

Concurrency

Semantic

Memory

8


Semantic bugs are becoming increasingly dominant.

A few concurrency bugs probably because of underreporting.

1992 [Sullivan92] 2005

Impact

9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%

75%Memory Concurrency Semantic

Apache

Results are similar for both

Mozilla and Apache

Impact

57.1% of crashes are caused by memory bugs.

9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%


Apache

14.3%


Mozilla and Apache

Impact


9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%


Apache

69.4%

14.3%


Mozilla and Apache

Impact


Incorrect functionality dominates.

9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%


Apache

69.4%

14.3%


Mozilla and Apache

Compare at 35%

Impact



9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%


Apache

69.4%

14.3%4.1% 1.0% 6.1% 1.0% 4.1%


Mozilla and Apache

Compare at 35%

Impact



9

Crash

FuncH

ang

Corru

ptPer

f

Oth

ers

unknown

0%

25%

50%


Apache

69.4%

14.3%4.1% 1.0% 6.1% 1.0% 4.1%


Mozilla and Apache

Compare at 35%

See our paper for numbers

with error range with 95%

confidence level

Component

10

Mozilla ApacheCore GUI Net I/O Others Core GUI Net I/O Others

0%

20%

40%

60%

80%

Component

10


0%

20%

40%

60%

80%

3.0%2.3%1.5%

52.7%

40.5%

Component

10


0%

20%

40%

60%

80%

3.0%2.3%1.5%

52.7%

40.5%

1.0%

12.2%8.2%2.0%

76.5%

Component

Client and server software have different bug characteristics.

10


0%

20%

40%

60%

80%

3.0%2.3%1.5%

52.7%

40.5%

1.0%

12.2%8.2%2.0%

76.5%

Security Bugs -Trend

11

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs


11

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs

Only contain half a year’s data


11

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs

99 00 01 02 03 04 050%

2%

4%

6%

8%

10%

12%

Mozilla Apache

Normalized Percentage



11

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs

99 00 01 02 03 04 050%

2%

4%

6%

8%

10%

12%

Mozilla Apache




11

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs

99 00 01 02 03 04 050%

2%

4%

6%

8%

10%

12%

Mozilla Apache




11

Security bugs are increasing.

0

20

40

60

80

100

99 00 01 02 03 04 05

Mozilla Apache

Number of bugs

99 00 01 02 03 04 050%

2%

4%

6%

8%

10%

12%

Mozilla Apache



Security Bugs

12

0%

25%

50%

75%

ApacheMem Sem Con Others

9.4%1.6%

71.9%

17.2%


Mozilla and Apache

Security Bugs

AGAINST the belief that buffer overflows are the most common form of security vulnerabilities

12

0%

25%

50%

75%

ApacheMem Sem Con Others

9.4%1.6%

71.9%

17.2%


Mozilla and Apache

Automatic Classification Do the previous distributions hold for all bugs?

Do we see the same trend across software evolution?

13



Require more bugs or all of the reported bugs, which could be around 29,000

13




Can NOT be done MANUALLY

13




Can NOT be done MANUALLY

Use machine learning techniques to automatically classify all 29,000 or so fixed run-time bugs in Mozilla (by June 2005)

13

Auto Classification Results Automatic classification confirmed the distribution results

of root causes and impacts (see our paper).

14



14

99 00 01 02 03 04 050%

20%

40%

60%

80%

100%

Semantic Memory

Mozilla



14

99 00 01 02 03 04 050%

20%

40%

60%

80%

100%

Semantic Memory

Mozilla



14

Semantic bugs increase over time

99 00 01 02 03 04 050%

20%

40%

60%

80%

100%

Semantic Memory

Mozilla



14


99 00 01 02 03 04 050%

20%

40%

60%

80%

100%

Semantic Memory

Mozilla



14


Memory bugs decrease over time

99 00 01 02 03 04 050%

20%

40%

60%

80%

100%

Semantic Memory

Mozilla

Conclusions Manually studied 709 bugs and automatically

classified 29,000 bugs

15



Memory bugs in release code decreased.

Detection tools have a positive impact.

15



Memory bugs in release code decreased.

Detection tools have a positive impact.

Semantic bugs are a major source of bugs, accounting for 81.1-86.7% of all classified bugs.

More effort should be applied to detecting and fixing semantic bugs.

15

Conclusions Most security vulnerabilities are caused by

semantic bugs

16


semantic bugs

Client and server have different bug characteristics.

Need different testing support

16


semantic bugs



Concurrency bugs are hard to reproduce.

Need tools, e.g. FDR and BugNet, to help replay

16


semantic bugs



Concurrency bugs are hard to reproduce.

Need tools, e.g. FDR and BugNet, to help replay

Correlation results and minor findings are shown in our paper.

16

Questions? Thank you!

Documents

Have Things Changed Now? - Purdue University€¦ · Things have changed! Many empirical studies have been performed over 10 years ago! More effective modern debugging tools - Valgrind,