Standard Grimoire Report

OPNFV Project

2016-Q4

January 11, 2017

Daniel Izquierdo CortazarChief Data Officerdizquierdo@bitergia.com

This report would not exist without the effort of the people involved inthe development of the Grimoire toolset.

(cc) 2016 Bitergia. Some rights reserved.This work licensed under Creative Commons Attribution-ShareAlike 4.0

Unported License.To view a copy of full license, see

http://creativecommons.org/licenses/by-sa/4.0,or write to Creative Commons, 559 Nathan Abbott Way, Stanford,

California 94305, USA.

1

Executive Summary

This report provides a quantitative analysis of the current and past situ-ation of the OPNFV project. All the data presented in it is based on infor-mation retrieved from the software development repositories of the project.The analysis includes a summary of the general situation of the project,and specific analysis of some of its development processes (issue tracking,code review) and communication channels (mailing lists, IRC, AskBot). Forcomparison with the past, most of the data is shown on a quarterly basis.

2

Contents

1 Project overview 4

2 Communication and support-related activities 72.1 Mailing Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Questions and Answers . . . . . . . . . . . . . . . . . . . . . . 92.3 IRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Details on OPNFV development community 133.1 Details of the project . . . . . . . . . . . . . . . . . . . . . . . 143.2 Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4 Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

A Metrics Definitions 19

B Source code and data sources 22

3

Chapter 1

Project overview

The report looks at activities across the OPNFV community during the lastquarter of 2016, comparing it to previous quarters.1.

Data source Activity 90 days Change (wrt to prev. 90 days)Gits 1590 commits -48%

Tickets 365 closed tickets -68%Mailing Lists 1754 sent emails -13%

Gerrit 1808 submitted reviews -36%Askbot 25 posted questions 13%

IRC 44332 messages -28%

Table 1.1: Activity during the last 90 days and its evolution

The overall development activity has decreased. Git and Gerrit activityhas decreased between a 36% and a 48%. This is also in line with the Jiraactivity with a decrease of 68%. Besides the communication channels keepdecreasing a 14% of mailing lists, 28% in the number of messages in the IRCchannels. On the other hand, Askbot shows an 13% of activity increase.If compared to previous periods, this quarter shows a similar activity levelthan the second quarter, while the first and third quarters of 2016 showhigher levels of activity in all areas.

1The analyzed data sources are available in appendixB

4

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

10

20

30

40

50

60

70

80

10.500

28.500

41.500

58.00062.000 62.500

75.500

61.500

Developers per monthPeriod Authors per month15-Q1 10.515-Q2 28.515-Q3 41.515-Q4 58.016-Q1 62.016-Q2 62.516-Q3 75.516-Q4 61.5

In this quarter of 2016 the mean number of developers active per monthhas reached a total of 61.5. It is a decrease when compared to the previousquarter, but in line with the second quarter of 2016.

The total number of contributors divided into three sets (core, regularand casual2) follow a similar pattern.

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

20

40

60

80

7

21

32

51 5255

63

46

8

16

25

3335

3337

30

5

16

24

29

34 33

46

41

Community structure

casualregularcore

Figure 1.1: Evolution during the last quarters of core, regular and casualdevelopers (based on git activity)

2Contributing developers are characterized as core, regular and casual depending ontheir activity in the git repositories. The classification is built by sorting contributors bytheir total number of commits; we sum the total commits per each individual contributors:the individuals whose commits sum up to 80% of the total number of commits in thequarter are the core contributors in that quarter. The regular contributors are thosewhose commits sum up to 95% of the total. The others are the casual contributors.

5

Period Core Regular Occasional15-Q1 5 8 715-Q2 16 16 2115-Q3 24 25 3215-Q4 29 33 5116-Q1 34 35 5216-Q2 33 33 5516-Q3 46 37 6316-Q4 41 30 46

Table 1.2: Characterization of developers by their total contribution to theproject

This report aims to provide some insight into the software developmentprocess of the OPNFV community measuring efficiency and process of thecommunity based on three metrics: the Review Efficiency Index (REI), theTime to Merge (TTM), and the Backlog Management Index (BMI). REI ismeasured as the number of closed (merged or abandoned) changesets out ofthe submitted changesets in a given period. TTM is measured as the timesince a review is submitted until this is closed. The BMI is measured as thenumber of closed tickets out of open tickets in a given period.

REI BMI TTM0.9 0.36 0.93 days

Table 1.3: Closed changesets out of opened changesets (REI), closed ticketout of opened tickets (BMI) and median time to merge in Gerrit (TTM)

6

Chapter 2

Communication andsupport-related activities

Analysis of the communication channels used for communication and support-related activities.

2.1 Mailing Lists

The following charts show activity in terms of emails sent, people sendingemails and people initiating threads per quarter. In addition, a table ispresented with the hot topics in the several analyzed mailing lists. Thisshows hot topics ordered by number of total posts in such thread.

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

500

1000

1500

2000

2500

1518

1735 17081803

23942269

2105

1759

Emails sentPeriod Emails15-Q1 151815-Q2 173515-Q3 170815-Q4 180316-Q1 239416-Q2 226916-Q3 210516-Q4 1759

7

2.1. MAILING LISTS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

50

100

150

200

250

300

162

185

207 204

229

267251 252

People sending emailsPeriod People15-Q1 16215-Q2 18515-Q3 20715-Q4 20416-Q1 22916-Q2 26716-Q3 25116-Q4 252

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

50

100

150

84

114 117124

133

157 159169

People initiating threadsPeriod People15-Q1 8415-Q2 11415-Q3 11715-Q4 12416-Q1 13316-Q2 15716-Q3 15916-Q4 169

8

2.2. QUESTIONS AND ANSWERS

Initial Author Numberand Date Subject Messagessw35882016-10-03 [opnfv-tsc] SDN World Congress 2016 presentation on

OPNFV EUAG19

myu2016-10-06 [opnfv-tech-discuss] OPNFV SPC Polestar WG Call

10/6/201617

Raymond Paik2016-12-15 [opnfv-tech-discuss] Starting to clean-up meetings

page on the wiki15

joehuang2016-12-14 [opnfv-tech-discuss] committer list maintainance 15Raymond Paik2016-10-09 [opnfv-tech-discuss] Announcing the 2016 TSC Chair

election result13

lukehinds2016-12-19 [Opnfv-security] Security checks at Gate 12Ryota MIBU2016-12-04 [opnfv-tech-discuss] [doctor] doctor CI now available 12Morgan Ri-chomme2016-10-27 [opnfv-tsc] sync before board/TSC meeting 11Yujun Zhang2016-12-15 [opnfv-tech-discuss] [all] TestPerf EcoSystem diagram

now editable11

Sindhur Malleni2016-11-22 [opnfv-users] OPNFV Colorado 2.0 with Triple0

RPMs11

2.2 Questions and Answers

The following charts show activity in the Ask site. Total number of ques-tions, number of answers, number of comments and people sending questionsare depicted. In addition two tables represent the hot topics activity in theAsk OPNFV site. These show information about the top visited questionsand questions with the highest number of people participating.

9

2.2. QUESTIONS AND ANSWERS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

10

20

30

40

0

3

23

36

30

45

24 25

QuestionsPeriod Questions15-Q1 015-Q2 315-Q3 2315-Q4 3616-Q1 3016-Q2 4516-Q3 2416-Q4 25

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

10

20

30

40

50

60

02

30

44

33

57

2118

AnswersPeriod Answers15-Q1 015-Q2 215-Q3 3015-Q4 4416-Q1 3316-Q2 5716-Q3 2116-Q4 18

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

10

20

30

40

1 1

18

22 21

41

6

1

CommentsPeriod Comments15-Q1 115-Q2 115-Q3 1815-Q4 2216-Q1 2116-Q2 4116-Q3 616-Q4 1

10

2.2. QUESTIONS AND ANSWERS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

5

10

15

20

25

0

3

14 14

20

23

19

14

People asking QuestionsPeriod People asking15-Q1 015-Q2 315-Q3 1415-Q4 1416-Q1 2016-Q2 2316-Q3 1916-Q4 14

• Top visited questions.

Question subject Visitscolorado release 10 arm iso not+ 18trying to install colorado on virtualbox+ 15opnfv deployment problem+ 13error installing openstack using fuel+ 9opnfv deployment in virtual environment+ 8linux kernel version used in colorado+ 7building colorado iso for armband+ 4deploy scenario os nosdn ovs ha+ 3build fail in single node build+ 2where exactly ie at what location+ 2

• Top questions with the highest number of people participating.

Question subject Peoplepartici-pating

opnfv deployment in virtual environment+ 2where exactly ie at what location+ 2error installing openstack using fuel+ 2colorado release 10 arm iso not+ 2trying to install colorado on virtualbox+ 2opnfv deployment problem+ 2error in opnfvfuel deployment error for+ 1error while installing colorado fuel from+ 1fuel not discovering nodes after pxe+ 1build fail in single node build+ 1

11

2.3. IRC

2.3 IRC

The community uses several IRC channels for asynchronous communication.This section shows information about the total number of messages sent inthe community during the last 7 quarters together with the number of peopleparticipating in such discussions.

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

10000

20000

30000

40000

50000

60000

70000

4584

1783520636

18963

49851

54879

62522

44415

Messages in IRC channelsPeriod Messages15-Q1 458415-Q2 1783515-Q3 2063615-Q4 1896316-Q1 4985116-Q2 5487916-Q3 6252216-Q4 44415

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-Q2 16-Q3 16-Q40

100

200

300

400

500

600

113

201

263293

461

544 534

486

People in IRC channelsPeriod People15-Q1 11315-Q2 20115-Q3 26315-Q4 29316-Q1 46116-Q2 54416-Q3 53416-Q4 486

12

Chapter 3

Details on OPNFVdevelopment community

Each breakdown is divided into three sections with information from thelast 6 quarters:

• activity: centered on the following metrics: commits from git activity,submitted, merge and abandoned reviews from the review system andopened and closed tickets from the issue tracking system.

• community: active core reviewers in gerrit, active authors in git andtop ten developers and top ten organizations contributing to the de-velopment in the last quarter.

• process: efficiency closing tickets, efficiency closing changesets, Timeto Merge (mean and median), number of patchsets (iterations) perchangeset and a study on the time waiting for a reviewer or submitteraction in the patchset review process.

13

3.1. DETAILS OF THE PROJECT

3.1 Details of the project

3.2 Activity

Commits in Git, submitted, merged and abandoned reviews in Gerrit andopened and closed issues in Jira.

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

500

1000

1500

2000

2500

3000

3500

4000

224

692905

1571

3098

2023

2902

1812

157

612791

1373

2869

1793

3114

1599

Commits and reviews: general

reviewscommits

Period Commits Reviews15-Q1 157 22415-Q2 612 69215-Q3 791 90515-Q4 1373 157116-Q1 2869 309816-02 1793 202316-Q3 3114 290216-Q4 1599 1812

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

200

400

600

800

1000

1200

1400

102

315

1103

634603

867 850

638

25

109

337

415481

515

1075

231

Opened and closed tickets: general

openedclosed

Period Closed Opened15-Q1 25 10215-Q2 109 31515-Q3 337 110315-Q4 415 63416-Q1 481 60316-02 515 86716-Q3 1075 85016-Q4 231 638

14

3.3. COMMUNITY

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

500

1000

1500

2000

2500

3000

3500

105

549711

1220

2673

1599

2583

1447

47 84 103188

330 314183 183

Merged and abandoned reviews: general

mergedabandoned

Period Merged Abandoned15-Q1 105 4715-Q2 549 8415-Q3 711 10315-Q4 1220 18816-Q1 2673 33016-02 1599 31416-Q3 2583 18316-Q4 1447 183

3.3 Community

Active core reviewers in Gerrit, active authors in Git, top authors and or-ganizations in the last quarter

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

20

40

60

80

14

34

59

6873

85 86

80

Active Core Reviewers: generalPeriod Active Core15-Q1 1415-Q2 3415-Q3 5915-Q4 6816-Q1 7316-02 8516-Q3 8616-Q4 80

15

3.4. PROCESS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

20

40

60

80

100

120

140

160

21

57

83

116124 125

151

123

Authors generalPeriod Authors15-Q1 2115-Q2 5715-Q3 8315-Q4 11616-Q1 12416-02 12516-Q3 15116-Q4 123

Commit (s) Author

143 Yujun Zhang

128 Jose Lausuch

115 Alexandru Avadanii

111 Morgan Richomme

78 Tim Rozet

71 matthew.lijun

65 Dan Radez

65 Fatih Degirmenci

65 zhihui wu

62 Serena Feng

Commit (s) Organizations

579 Huawei

326 ZTE Corporation

301 Ericsson

249 Red Hat

142 ENEA AB

141 Orange

87 Intel

75 Mirantis

42 Linux Foundation

25 NEC

23 ATT

16 SUSE

14 Dell

14 EMC

13 MontaVista

11 Juniper

7 Cisco

7 Nokia

7 Tata Consultancy

3.4 Process

Efficiency closing changesets and tickets, time to review (mean and median),number of patchests (iterations) per changeset and study on the time waitingfor a reviewer or submitter action in the patchset review process.

16

3.4. PROCESS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.250

0.3500.310

0.650

0.800

0.590

1.260

0.360

Efficiency closing tickets: generalPeriod Closed/Opened15-Q1 0.2515-Q2 0.3515-Q3 0.3115-Q4 0.6516-Q1 0.816-02 0.5916-Q3 1.2616-Q4 0.36

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0

0.2

0.4

0.6

0.8

1.0

0.680

0.910 0.900 0.900

0.970 0.950 0.9500.900

Changesets efficiency: generalPeriod (Aband. and Merg.)/Subm.15-Q1 0.6815-Q2 0.9115-Q3 0.915-Q4 0.916-Q1 0.9716-02 0.9516-Q3 0.9516-Q4 0.9

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

1

2

3

4

5

6

7

8

4.250 4.330

6.840

4.050

3.550

3.990

3.430

3.870

0.760

0.330 0.2700.590

0.180 0.3400.530

0.930

Time to merge (days): general

meanmedian

Period Median Mean15-Q1 0.76 4.2515-Q2 0.33 4.3315-Q3 0.27 6.8415-Q4 0.59 4.0516-Q1 0.18 3.5516-02 0.34 3.9916-Q3 0.53 3.4316-Q4 0.93 3.87

17

3.4. PROCESS

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40.0

0.5

1.0

1.5

2.0

2.5

3.0

1.840

2.100 2.110

2.310

1.930 1.920

2.100

2.460

1.000 1.000 1.000 1.000 1.000 1.000 1.000

2.000

Patchsets per Changeset: general

meanmedian

Period Median Mean15-Q1 1.0 1.8415-Q2 1.0 2.115-Q3 1.0 2.1115-Q4 1.0 2.3116-Q1 1.0 1.9316-02 1.0 1.9216-Q3 1.0 2.116-Q4 2.0 2.46

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

5

10

15

20

13.420

11.94013.110 13.430

12.070

18.790

9.790

12.680

4.990

0.3700.950 1.090

0.0500.670 0.300

1.360

Time waiting for the reviewer (days): general

avgmedian

Period Median Mean15-Q1 4.99 13.4215-Q2 0.37 11.9415-Q3 0.95 13.1115-Q4 1.09 13.4316-Q1 0.05 12.0716-02 0.67 18.7916-Q3 0.3 9.7916-Q4 1.36 12.68

15-Q1 15-Q2 15-Q3 15-Q4 16-Q1 16-02 16-Q3 16-Q40

5

10

15

20

25

30

35

40

24.200

29.000

15.690

29.020

31.810

23.300

18.890

16.470

32.550

3.490

0.920

7.030

1.420 0.970 0.730 1.500

Time waiting for the submitter (days): general

avgmedian

Period Median Mean15-Q1 32.55 24.215-Q2 3.49 29.015-Q3 0.92 15.6915-Q4 7.03 29.0216-Q1 1.42 31.8116-02 0.97 23.316-Q3 0.73 18.8916-Q4 1.5 16.47

18

Appendix A

Metrics Definitions

• Commit: this is defined as the action(s) that performes a change inthe source code. Bots, merges and other type of automatic activity isremoved from the records. In addition, when aggregating several gitrepositories, this metric only counts unique revisions (unique hashesfound in the git repositories). In addition, all branches are aggregatedto the analysis.

• Submitted changeset: a changeset is the process of peer reviewingsource code changes. A submitted code is not merged to the mastercode of a given project till this is approved for at least one core reviewerof such project. A submitted changeset is defined as any changesetsubmitted to the Gerrit system.

• Merged and abandoned changsets: a merge is defined as the patchsetthat was finally submitted to the source code. An abandoned change-set is a potential merge that was finally dismissed by developers asbeing part of the source code. This status is found in the status of thefinal patchset. However, although a patchset can be merged or aban-doned, this action can be reverted. If a patchset presents several ofthese changes in the same period of time, only one of them is counted(the very last one). On the other hand, if those changes take place indifferent periods of analysis, both status would be counted.

• Open and closed ticket: a ticket in Jira is counted as closed if thestatus of such ticket is defined as ’Closed’. The rest of the tickets arecounted as opened tickets.

• Active Core Reviewer: a core reviewer has the possibility to use +2 or

19

-2 actions when reviewing the code. However, if there are developersthat for some period do not use those actions, those can not be mea-sured as core reviewer. Thus, this metric provides information about’active’ core reviewers. This can be also defined as those developersthat actively have used the +2 or -2 review action. This metric is alsofiltered by branch of activity, only using ’master’. This helps to detectactual core reviewers in each of the projects.

• Authors: a developer is defined as author if she is the owner of thepatchset sent for reviewing and this is merged into the source code.As previously indicated, automatic commits such bot’s are removedfrom this analysis.

• Efficiency closing issues: this metric is a derivation of the BacklogManagement Index (BMI) that measures the number of closed ticketsout of the opened tickets in a period of time. Values under 1.0 indicatesthat the number of closing issues is lower than the number of openedissues arriving. On the contrary, higher charts would indicate bettermaintenance effort by the community.

• Efficiency closing changesets: this metric is a derivation of the BacklogManagement Index as it is named as Review efficiency index (REI).As similarly used in the BMI index, this metrics measures the numberof closed changesets (merged or abandoned) out of the total numberof new changesets.

• Time to Merge: this time consists of the time between the first uploadof the first patchset (as defined as a submitted changeset) till the lastpatchset of the changeset is merged into the code and this is indicatedin the comments side of the Gerrit tool. This metric is provided innumber of days.

• Patchsets per changeset: this metric calculates the total number ofiterations in a changeset till this is abandoned or merged.

• Time waiting for the reviewer or the submitter: a changeset is waitingfor a reviewer action if a new patchset upload or a new changset arrivesto the system. On the other hand, a submitter action is required whena specific negative verification or reviewing action takes place (Verified-1/-2 or Code-Review -1/-2). In addition, when a Code-Review +2action takes place, it is assumed that the changset is closing and no

20

more times are registered either for the reviewer or the submitter. Forthis analysis, those patchsets flagged as work in progress are ignored.

Metrics measured in the general overview:

• Community structure, core, regular and casual developers: developersare ordered in descendant order by the number of commits authoredfor a given period. Core developers are defined as the list of developersthat reach 80% of the total commits. Regular is the set of developersthat are between that 80% and 95% of the commits. Casual develop-ers are found in the rest of the 5%. Bots are ignored in this list ofdevelopers.

• Developer per month: average of developers per month ignoring bots.

• Emails sent: number of emails sent by people to the several mailinglists. Bots are not registered.

• People sending emails: number of people sending those emails ignoringbots.

• People initiating threads: a thread is defined as a list of emails thathas the same root. There may exist threads of one email.

• Top threads: this list provides the longest threads in terms of numberof emails that have a common root email.

• Questions, answers and comments in Askbot.

• People asking questions in Askbot: number of people sending a newquestion.

• Top visited questions.

• Messages and people in IRC: this analysis ignores as a message thoseentries in the IRC channels that provide information about peopleentering or leaving the system.

21

Appendix B

Source code and data sources

The source code of the scripts and templates used to produce this reportare available from the GrimoireReports repository1.

The databases used for the analysis can be obtained from the “Datasources” panel2 of the Grimoire Dashboard for the project3.

1https://github.com/VizGrimoire/GrimoireReports2http://projects.bitergia.com/opnfv/browser/data_sources.html3http://projects.bitergia.com/opnfv/

22