How Do Software Engineers Understand Code Changes? FSE 2012

How Do Software Engineers Understand Code Changes? An Exploratory Study in Industry

Yida Tao1, Yingong Dang2, Tao Xie3

Dongmei Zhang2, Sunghun Kim1

1The Hong Kong University of Science & Technology

2Microsoft Research Asia 3North Carolina State University

2

3

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

4

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

>+ struct CIDEntry >+ { >+ const nsCID* cid; >+ bool service;

“What is this used for, I can’t spot it in

use anywhere. ”

5

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

>+ struct CIDEntry >+ { >+ const nsCID* cid; >+ bool service;

“What is this used for, I can’t spot it in

use anywhere. ”

> browser_hide_removing.js >+ browser_imageReload.js >+ image_Reload.html

“These files are missing from this

patch, aren’t they?”

6

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

>+ struct CIDEntry >+ { >+ const nsCID* cid; >+ bool service;

“What is this used for, I can’t spot it in

use anywhere. ”

> browser_hide_removing.js >+ browser_imageReload.js >+ image_Reload.html

“These files are missing from this

patch, aren’t they?”

>+ for (var i = aURL.length – 1; i >= 1; i--) { >+ var chPrev = aURL.charAt(i – 1) ; >+ var ch = aURL.charAt(i) ;

“I’m not sure why you walk this char

by char… ”

7

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

>+ struct CIDEntry >+ { >+ const nsCID* cid; >+ bool service;

“What is this used for, I can’t spot it in

use anywhere. ”

> browser_hide_removing.js >+ browser_imageReload.js >+ image_Reload.html

“These files are missing from this

patch, aren’t they?”

>+ for (var i = aURL.length – 1; i >= 1; i--) { >+ var chPrev = aURL.charAt(i – 1) ; >+ var ch = aURL.charAt(i) ;

“I’m not sure why you walk this char

by char… ”

8

> if (hudRef && hud) { > if (hudRef.consolePanel) { > + hudRef.consolePanel.hidePopup()

“Why this change here? This is the only one that doesn’t seem to make sense for me…”

>+ struct CIDEntry >+ { >+ const nsCID* cid; >+ bool service;

“What is this used for, I can’t spot it in

use anywhere. ”

> browser_hide_removing.js >+ browser_imageReload.js >+ image_Reload.html

“These files are missing from this

patch, aren’t they?”

>+ for (var i = aURL.length – 1; i >= 1; i--) { >+ var chPrev = aURL.charAt(i – 1) ; >+ var ch = aURL.charAt(i) ;

“I’m not sure why you walk this char

by char… ”

“…”

“…”

“…”

9

“How do software engineers understand code changes?”

Research Questions

10

RQ1: How frequently is code change understanding practiced and in which development tasks is it required?

RQ2: What are engineers’ information needs and difficulties for understanding code changes?

RQ3: How to improve the effectiveness and efficiency of the practices in understanding code changes?

Study Methodology

11

Literature Review

•Potential information needs

Questionnaire Design

•Investigate RQ1, RQ2

Pilot Interview

•Question is relevant & clear

Study Methodology

12

Literature Review

•Potential information needs

Questionnaire Design

•Investigate RQ1, RQ2

Pilot Interview

•Question is relevant & clear

Online Survey

•Over 1000 MS employees

Follow-up Interview

•Investigate RQ3

Analysis

•Answering RQs

Survey Participants

13

Dev 55%

Test 31%

PM 14%

Role Distribution Product Teams

OS

Desktop App

Web App

Mobile App

Service

Others

180 respondents (16% response rate)

14

RQ1 • Frequency ? • Development tasks ?

RQ2 • Information needs ? • Difficulty ?

RQ3 • Improvement ?

RQs

RQ1: Frequency of Understanding Code Changes

15

How often do you need to understand code changes?

o Several times each hour

o About once an hour

o Several times each day

o About once a day

o Several times each week

o About once a week

o Rarely

o Never

RQ1: Frequency of Understanding Code Changes

16

0%

5%

10%

15%

20%

25%

30%

35%

40%

Perc

enta

ge o

f res

pons

es

RQ1: Frequency of Understanding Code Changes

17

0%

5%

10%

15%

20%

25%

30%

35%

40%

Perc

enta

ge o

f res

pons

es 80%

RQ1: Tasks Requiring Code Change Understanding

18

“Select the top three tasks that most often require you to understand code changes” [Design/Planning] Refactoring [Implementation] Developing new feature [Implementation] Fixing bug [Integration] Resolving merge conflict [Verification] Reviewing others’ code changes [Verification] Reviewing my own code changes [Verification] Writing & updating test cases Other, please specify

RQ1: Tasks Requiring Code Change Understanding

19

121

100

89

73

48

34

30

0% 15% 30% 45% 60% 75%

Reviewing others' changes

Fixing bug

Developing new feature

Reviewing my own changes

Writing/updating test cases

Refactoring

Resolving merge conflict

Percentage of participants who selected the task

RQ1: Tasks Requiring Code Change Understanding

20

121

100

89

73

48

34

30

0% 15% 30% 45% 60% 75%

Reviewing others' changes

Fixing bug

Developing new feature

Reviewing my own changes

Writing/updating test cases

Refactoring

Resolving merge conflict

Percentage of participants who selected the task

21

RQ1 • Frequently practiced • Major development tasks

RQ2 • Information needs ? • Difficulty ?

RQ3 • Improvement ?

Answers to RQs

Potential Information Needs

22

Literature review (code-change analysis and management) 180 articles in 10 SE venues over the past decade

Potential Information Needs

23

Literature review (code-change analysis and management) 180 articles in 10 SE venues over the past decade

Reasoning & assessing the change • Clones •Design •…

Exploring the change’s context & impact • Risk • Consistency •…

Evaluating the change history • Change proneness •Defect proneness

Survey Questions

24

“Rate the importance & difficulty of each information need (formulated as question) in a change understanding task”

Very Important

Important

Somewhat Important

Not Important

3

2

1

0

Survey Questions

25

“Rate the importance & difficulty of each information need (formulated as question) in a change understanding task”

Very Important

Important

Somewhat Important

Not Important

Very Difficult

Difficult

Relatively Easy

Straightfor-ward

3

2

1

0

Survey Questions

26

“Rate the importance & difficulty of each information need (formulated as question) in a change understanding task”

Does this change introduce code clones?

Does this change break any code elsewhere?

Which tests should be run to verify this change?

Is this changed location a hotspot for past fixes? ……

Very Important

Important

Somewhat Important

Not Important

Very Difficult

Difficult

Relatively Easy

Straightfor-ward

3

2

1

0

RQ2: Information Needs

27

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

RQ2: Information Needs

28

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

RQ2: Information Needs

29

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Risk Consistency

Completeness

Design Correctness

30

RQ1 • Frequently practiced • Major development tasks

RQ2 • Risk & Quality are important but difficult to know

RQ3 • Improvement ?

Answers to RQs

RQ3: Interview Items

31

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Risk

RQ3: Interview Items

32

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Risk

Rationale

RQ3: Interview Items

33

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Risk

Rationale

Change proneness

Defect proneness

Assessing a Change’s Risk

34

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Risk

Current Practice on Assessing a Change’s Risk

35

Manual Code Review •Error prone •Cross-components •Unclear interface •Hidden assumptions •…

Unit & Regression Testing •Time consuming •Depends on how thorough the tests are •…

Support Assessing a Change’s Risk

36

Man

ual c

ode

revi

ew

•Navigation in diff: using code analysis tools (e.g., go to definition, find all references, caller/callee tree) on the code change

Support Assessing a Change’s Risk

37

Man

ual c

ode

revi

ew

•Navigation in diff: using code analysis tools (e.g., go to definition, find all references, caller/callee tree) on the code change

Diff

Code Analysis

Navigation in diff

“…miss a level of understanding object

relationships”

Support Assessing a Change’s Risk

38

Test

ing •which code must

be retested as it is dependent upon the change?

•who owns testing that dependency?

•which tests must be run?

“An ‘Intelli-sense’ for updating these (affected) tests would be nice as well.”

Discussion

39

0

1

2

3

0 1 2 3

Diff

icul

ty o

f acq

uiri

ng th

e in

form

atio

n

Importance

Rationale

Change proneness

Defect proneness

Discussion

40

• Availability & Quality of commit message • “It’s entirely up to the dev making the change as to how hard or

easy it is for someone else to figure out why the change was made.”

Why is understanding the rationale of a change easy?

• Developers • Here and now • Short-term issue • Own knowledge

• Testers & PMs • Historical metrics might be good to reflect bugginess and

complexity of a specific area

Why are historical metrics not that important?

Other Information Needs

41

“In addition to the information needs listed above, what else would you ask when you try to understand a code change? How difficult is it for you to answer?”

Other Information Needs

42

“Can this change be broken into smaller discreet changes?”

“In addition to the information needs listed above, what else would you ask when you try to understand a code change? How difficult is it for you to answer?”

Composite Code Change

43

Understanding a Composite Code Change

44

Decomposing a Composite Code Change

45

46

RQ1 • Frequently practiced • Major development tasks

RQ2 • Risk & Quality are important but difficult to know

RQ3 • Assessing the risk

of a change • Decomposing a

composite change

Answers to RQs

Summary

47

A large-scale exploratory study on industrial practice in understanding code changes Understanding code changes happens frequently in major

development tasks

Summary

48

A large-scale exploratory study on industrial practice in understanding code changes Understanding code changes happens frequently in major

development tasks

An extensive exploration on engineers’ information needs for understanding code changes Assessing a change’s risk, consistency & completeness

Summary

49

A large-scale exploratory study on industrial practice in understanding code changes Understanding code changes happens frequently in major

development tasks

An extensive exploration on engineers’ information needs for understanding code changes Assessing a change’s risk, consistency & completeness

A guideline for future research and tool design that aims at

supporting change-understanding tasks Navigation in diff Change decomposition

Acknowledgment

50

All participants of survey / interview

Miryung Kim, Robin Moeur, Thomas Zimmermann, Jacek Czerwonka, and Kathryn McKinley

Summary

51

A large-scale exploratory study on industrial practice in understanding code changes Understanding code changes happens frequently in major

development tasks

An extensive exploration on engineers’ information needs for understanding code changes Assessing a change’s risk, consistency & completeness

A guideline for future research and tool design that aims at

supporting change-understanding tasks Navigation in diff Change decomposition