Modern Release Engineering in a Nutshell - Why Researchers should Care!

Modern Release Engineering in a Nutshell

Why Researchers should Care

Bram Adams Shane McIntosh,MC IS

Back in 2009 …

On average, we release new code fifty

times a day.

Release engineering aims to …

4


4

codechange


4

codechange


4

codechange


4

codechange

FAST

Release Engineering

Release Engineering

Release Engineering

integrating code changes

Release Engineering

integrating code changes building/testing (CI)

Release Engineering


deploying a new release

Release Engineering


releasing to the user deploying a new release

Nowadays …

Time-boxed releases

7

Time-boxed releases

6 months

7

Time-boxed releases

6 weeks

6 months

7

Time-boxed releases

2 weeks (mobile)

6 weeks

6 months

7

Time-boxed releases

twice/day (web)

2 weeks (mobile)

6 weeks

6 months

7

Time-boxed releases

twice/day (web)

daily

2 weeks (mobile)

6 weeks

6 months

7

http://www.informit.com/articles/article.aspx?p=1833567

Jez Humble


reduce the risk of releasing software

if it hurts, do it more frequently, and

bring the pain forward

Jez Humble

• How quickly can we ship a chemspill release?

• 4-6 weeks 11 hours

• How long to ship a “new feature” release?

• 12-18 months 6 weeks

• How many active code lines?

• 1 1/2 42

• How many checkins per day?

• ~15 per day 325 per day

Before & After

Thursday, June 6, 13

http://oduinn.com/images/2013/blog_2013_RelEngAsForceMultiplier.pdf 9






• 1 1/2 42



Before & After








• 1 1/2 42



Before & After



https://xebialabs.com/periodic-table-of-devops-tools/ & https://xebialabs.com/the-ultimate-devops-tool-chest/

11

industry Release engineering innovation

led by industry!

11

industry

academia

Release engineering innovation

led by industry!

12

What can you do for release engineering?

RELENG: International Workshop on Release Engineering

13


13

230 participants3 editions

dozens of industry

& academic talks


13


dozens of industry

& academic talks


13


next RELENG: in Fall 2016

http://releng.polymtl.ca

dozens of industry

& academic talks

How to predict integration effort?

14


14

git repo


14

git repo

myapp v.1


14

git repo

myapp v.1

?


14

git repo

myapp v.1


14

feature branch 1git repo

myapp v.1


14


myapp v.1


14


myapp v.1

master branch


14


myapp v.1

master branch


14


MERGE

myapp v.1

master branch


14


MERGE

myapp v.1

master branch


14


MERGE

myapp v.1

master branch


14


release branch

MERGE

myapp v.1

master branch


14


release branch

MERGE

myapp v.1

master branch


14


release branch

MERGE

myapp v.1

feature branch 2

master branch


14


release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch


14

3rd party dependency


release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch


14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

master branch


14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2sy

nc

15

https://xkcd.com/303/

user acceptance/system testsperformance tests

(selection of) unit tests

all unit testsintegration tests

time to run (hours)

full compilation

incremental compilation

`

Why do CI builds take so long?

16




time to run (hours)

full compilation


`

local developer build


16




time to run (hours)

full compilation


`



16




time to run (hours)

full compilation


` CI build of merged change



16




time to run (hours)

full compilation


CI build of merged change



16




time to run (hours)

full compilation


closer to release




16




time to run (hours)

full compilation


closer to release




16

Even worse …

17

Even worse …

17

build machinery run

on each commit!

Even worse …

17

build machinery run

on each commit!

Even worse …

17

build machinery run

on each commit!different feature configurations

Even worse …

17

build machinery run

on each commit!

not just build and test, also code quality builds, nightly builds, etc.

different feature configurations

Even worse …

17

build machinery run

on each commit!

not just build and test, also code quality builds, nightly builds, etc.

not all builds succeed

different feature configurations

What are deployment best practices?

18


18

v.1 v.1v.1


18

v.1v.2 v.1


18

v.2 v.1v.2


18

v.2 v.1v.1


18

v.1v.1v.1

Infrastructure code

19

# Install PostgreSQL server and client include_recipe "postgresql::server" include_recipe "postgresql::client"

# Make postgresql_database resource available include_recipe "database::postgresql"

# Create database for Rails app db = node["practicingruby"]["database"] postgresql_database db["name"] do connection( :host => db["host"], :port => node["postgresql"]["config"]["port"], :username => db["username"], :password => db["password"], ) end

Infrastructure code

19




Infrastructure code

19




Infrastructure code smells?

19

Is the release in good

shape?

Was the release a success?


shape?



shape?

The release is botched, how can we

roll back?


What’s the optimal cycle time for us?


shape?

The release is botched, how can we

roll back?

Chuck Rossi

21

Continuous delivery for mobile apps is a serious

challenge!

Chuck Rossi

21


22


22

What can release

engineering do for you?

When noble intentions meet reality

23

Harmful assumptions about release pipelines that can impact predictive modelling

24


1. All releases are equal

24

25

Look at Project X! It had a crazy number of

bugs in the typical six-month post-release window!

25

Look at Project X! It had a crazy number of

bugs in the typical six-month post-release window!

Well, Project X releases every 6 weeks, so

you’re counting several releases…

25

Release cycles vary amongpopular studied systems

Fig. 3: Integration delay is computed by counting the re-leases that occur between when an issue status changes toRESOLVED-FIXED and the the date of the release note thatlists that issue.

Fig. 4: Delays in days between releases of ArgoUML, Eclipse,and Firefox. The number shown over each boxplot is themedian interval

project. For example, Figure 4, shows that Firefox releasesconsistently every 42 days (six weeks), whereas the timesbetween ArgoUML releases vary from 50 to 220 days. Theconsistency of Firefox releases may lead to more delayedissues, since they rigidly adhere to a six-week release scheduledespite accumulating issues that could not be integrated.

34% to 60% of addressed issues in the traditionalrelease cycle systems were delayed by one or more releases.Figure 2 shows that 98% of the addressed issues in Firefoxare delayed by one or more releases. Firefox is expected tohave delayed issues due its rapid release cycles. However,98% is still a considerably large percentage. Furthermore, evenfor the systems that adopt a more traditional release cycle,34% (ArgoUML) to 60% (Eclipse) of the addressed issues aredelayed by one or more releases. This result indicates that eventhough an issue is addressed, integration could be delayed byone or more releases.

Many delayed issues were addressed well before releasesfrom which they were omitted. Addressed issues could bedelayed from integration because they were addressed latein the release cycle, e.g., one day or one week before theupcoming release date. In order to compare the rapid andtraditional release cycles regarding whether delayed issuesare addressed late in the release schedule, we computed the

Fig. 5: Distribution of days between when an issue wasaddressed and the next missed release divided by the releasewindow time.

Addressing Stage metric (AS) for each issue. The AS metricis calculated using the following equation: days to next release

release window

,where days to next release is the number of days when an issueis addressed before the next release (e.g., the time between t3to t4 in Figure 3), and the release window is the time in daysbetween the next upcoming release and the respective previousrelease (e.g., t4 to t2). An AS value close to 1 means that anissue was addressed too close to the next release, whereas avalue close to 0 means that an issue was addressed at thebeginning of a release cycle. Figure 5 shows the distributionof the AS metric for each project. The smallest AS medianis observed for Eclipse, which is 0.45. For ArgoUML andFirefox, the median is 0.52 and 0.53, respectively. The ASmedians are roughly in the middle of the release. Moreover,the boxes extend to cover between 0.25 and 0.75. The resultsuggests that, in the studied projects, delayed issues are usuallyaddressed 1

4 to 34 of the way through a release. Hence, it is

unlikely that most addressed issues miss the next release solelybecause they were addressed too close to an upcoming releasedate.

The integration of 34% to 60% of the addressed issuesin the traditionally releasing systems and 98% in therapidly releasing system were delayed by one or morereleases. Furthermore, we find that many delayed issueswere addressed well before releases from which they wereomitted from.

RQ2: Can we accurately predict when an addressed issuewill be integrated?Motivation. Several studies proposed approaches to inves-tigate the time required to address an issue [2–7]. Thesestudies could help to estimate when an issue will be addressed.However, we find that integration delays when an addressedissue will be delivered to users. Even though several issues areaddressed well before the next release date, their integration isdelayed. For users and contributors, however, knowing the re-lease in which an addressed issue will be integrated is of great

Day

s be

twee

n re

leas

es

An Empirical Study of Delays in the Integration of Addressed IssuesD. A. da Costa et al.

[ICSME 2014]

Release cycles can evenvary within systems!

1.02.02.53.03.54.05.06.07.08.09.0

10.00 200 400 600 800

Fire

fox

rele

ase

Days since prior release 27

The rapid release cycle of modern software systems

28

The rapid release cycle of modern software systems

Often release several timesin one day!

28

1. All releases are equal


29

1. All releases are equal2. All branches are equal


29

30

Weird… The size of this project fluctuates between

50k and 45k lines!

30

Weird… The size of this project fluctuates between

50k and 45k lines!

Hmm, did you select the relevant branch? Several

are developed in parallel!

30

An example of defect prediction

Feature developmentDefect repairingMerge

Commit types

31



Commit types

31


v1.0


Commit types

31


v1.0


Commit types

Pre-release

31


v1.0


Commit types

Pre-release

Post-release

31


v1.0


Commit types

Pre-release

Post-release

31

Handling the intricacies of a multi-branch release pipeline

Dev

Stable


Commit types

32


Dev

Stable


Commit types

32


Dev

Stable


Commit types

32


Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32


v1.0

Dev

Stable


Commit types

32

Branching can get really complex…

33



34


3. All files are equal


34

35

This popular web browser has an enormous codebase!

>30M lines!

35

This popular web browser has an enormous codebase!

>30M lines!

Yes, but the codebase contains several systems! The build configuration

decides which one is produced!

35

36

Many files are conditionally included in deliverables

Tracing Software Build Processes to Uncover License Compliance

InconsistenciesS. van der Berg et al.

[ASE 2014]

Aterm

Opkg

Bash

CUPS

Xalan

OpenSSL

FFmpeg

% Excluded Files

0% 20% 40% 60% 80%

36

Many files are conditionally included in deliverables

Fixes in these files may have a smaller impact (if any) on customers

Tracing Software Build Processes to Uncover License Compliance

InconsistenciesS. van der Berg et al.

[ASE 2014]

Aterm

Opkg

Bash

CUPS

Xalan

OpenSSL

FFmpeg

% Excluded Files

0% 20% 40% 60% 80%

Understanding conditionally included files using the build system

Design recovery and maintenance of build systemsB. Adams et al.

[ICSM 2007]

37

1. All releases are equal,2. All branches are equal,



38

but some are more equal than others




38

My nightmare

Amassing and indexing a large sample of version control systems

Audris Mockus [MSR 2009]

Boa: a language and infrastructure for analyzing ultra-large-scale software repositories

R. Dyer et al. [ICSE 2013]

The GHTorrent Dataset and Tool Suite

G. Gousios [MSR 2013]

My nightmare

Amassing and indexing a large sample of version control systems

Audris Mockus [MSR 2009]

Boa: a language and infrastructure for analyzing ultra-large-scale software repositories

R. Dyer et al. [ICSE 2013]

The GHTorrent Dataset and Tool Suite

G. Gousios [MSR 2013]

We collect all of the data in the world, but it’s meaningless without context!

Release Engineering



Release Engineering



Release Engineering







Jez Humble

Release Engineering


releasing to the user deploying a new releasehttp://www.informit.com/articles/article.aspx?p=1833567




Jez Humble

Release Engineering






Jez Humble

master branch

How to Predict Integration Effort?

14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2sy

nc

Release Engineering






Jez Humble

master branch


14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

sync

Release Engineering






Jez Humble

master branch


14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

syncbut some are more equal than others




Release Engineering






Jez Humble

master branch


14



release branch

MERGE CONFLICT?

myapp v.1

feature branch 2

sync

but some are more equal than others


