Always ship trunkManaging change in complex websites

Paul Hammond

Revision Control

version 1version 2version 3version 4

If youʼre not using any revision control system leave now and start using one

But...

Coordinating many people working on many things

at the same time is still hard

“Just use branching”

...which means merging

“Branching causes problems in Subversion because Subversion doesnʼt store enough information to make merging work. In Mercurial, merging is painless and easy, and so branching is commonplace and harmless.”

—Joel Spolsky, http://hginit.com/00.html

“Git will allow you to have multiple local branches ... the creation, merging and deletion of those lines of development take seconds ... Git makes this process incredibly easy and it changes the way most developers work when they learn it”

— Scott Chacon, http://whygitisbetterthanx.com/

Distributed branching & merging is awesome

But...

“What the !#*$ is going on?”“What the !#*$ just went wrong?”

“What the !#*$ is running on www34?”

All existing revision control systems were built by people who build installed software

1. Installed software2. Open source installed software3. Web applications & Software as a Service

Three kinds of software:

Web apps are not like installed apps

“In the online world, a software product drives a service to which users have access. There is, most often, a single copy of the actual software product in use. There is one consumer of the software: you. The users are consumers of the service built atop the software.”

– Theo Schlossnagehttp://omniti.com/seeds/online-application-

deployment-reducing-risk

Does your team admin every computer your software is installed on?

Each new release happens exactly once

Once an upgrade has happened, the old code will never be run again

1.0 1.1 1.2

1.0.1

1.1.1

1.0.2

r2301 r2302 r2306

Well, kind of...

Upgrades are not deployed to all servers simultaneously

Staging/QA environments

Beta environments

Upgrades are not deployed to all users simultaneously

A/B testing

Public betas

Configuration managementApplication code

Installed library dependenciesWeb service dependencies

Coordinating many people working on many things

and running them all at the same timeis hard

But nobody knows you just deployed(unless you tell them)

Idea one: separate feature launches from infrastructure launches

You can deploy the code for a feature long before you launch it

and nobody will know

You can completely rewrite your infrastructure and keep the UI the sameand nobody will know

Idea two: run multiple versions of your code at once

You can repeatedly switch between two backend systems and keep the UI the same

and nobody will know

You can deploy a non-user facing change to only a small percentage of servers

and nobody will know

You can offer different user interfacesto different users

and nobody will know

Need a revision control system that allows us to manage multiple parallel versions of the code and

switch between them at runtime

Branches?

Branches donʼt help you switch between versions at runtime

Branches donʼt help you managedependency changes that need to happen

on all versions at once

Need a revision control system that allows us to manage multiple parallel versions of the code

and switch between them at runtime

Need a revision control system that allows us to manage multiple parallel versions of the code

and switch between them at runtime

Need a revision control system that allows us to manage multiple parallel versions of the code

and switch between them at runtime

Manage the different versions within your application

Branch in code

if ($cfg['use_snowflake_ticket_server']) {

# new hotness $ticket = snowflake_ticket();

} else {

# old and boring $ticket = db_generate_ticket();

}

# hardcoded to not run$cfg['use_snowflake_ticket_server'] = false;

# only in certain environmentsif ($cfg['environment'] == 'dev') { $cfg['use_snowflake_ticket_server'] = true;} else { $cfg['use_snowflake_ticket_server'] = false;}

# only for a percentage of usersif (($user->id % 100) < 3) { $cfg['use_snowflake_ticket_server'] = true;} else { $cfg['use_snowflake_ticket_server'] = false;}

# only for a percentage of requests$rand = rand(0,99);if ($rand < 3) { $cfg['use_snowflake_ticket_server'] = true;} else { $cfg['use_snowflake_ticket_server'] = false;}

# only for a percentage of requestsif ($cfg['environment'] == 'dev') { $cfg['use_snowflake_percentage'] = 100;} else { $cfg['use_snowflake_percentage'] = 2;}

$rand = rand(0,99);if ($rand < $cfg['use_snowflake_percentage']) { $cfg['use_snowflake_ticket_server'] = true;} else { $cfg['use_snowflake_ticket_server'] = false;}

# done testing, let’s launch$cfg['use_snowflake_ticket_server'] = true;

# uh oh...$cfg['use_snowflake_ticket_server'] = false;

# ok, it works again$cfg['use_snowflake_ticket_server'] = true;

Feature testing on production servers

# team testing$team = array(41,287,5806,5930);if (in_array($user->id, $team)) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

# private staff alpha testingif ($user->is_admin()) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

# opt-in private staff alpha testing$has_cookie = isset($_COOKIE['new_context']);if ($user->is_admin() && $has_cookie) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

http://code.flickr.com/blog/2009/12/02/flipping-out/

# opt-in public betasif ($user->has_pref('new_context')) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

# opt-in public betas via groupsif ($user->in_group('new_context')) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

# user tagging (ala dopplr)if ($user->has_tag('context_widget')) { $cfg['use_new_context_widget'] = true;} else { $cfg['use_new_context_widget'] = false;}

Flexibility

# dark launches can be ramped upif ($cfg['front_page_dark_launch']) { # notice we're not keeping the data get_some_really_complex_data()}

1. Development on user facing features2. Development on infrastructure3. Kill-switches

Three types of feature flags:

# killswitchif ($cfg['disable_login']) { error('Sorry, login is unavailable'); exit;}

Usually need multiple flags at onceduring development and testing

# for development$cfg['disable_search_tests_all'] = false;$cfg['disable_search_ui_beta_test'] = false;$cfg['disable_search_darklaunch'] = false;

# for post launch$cfg['disable_search'] = false;

Complexity

Separate operational controls from development flags

Be disciplined about removing unused feature flags

Always deploy trunk to every server on every deploy

and manage versions through config

# integrate with configuration managementif (in_array('beta', $node['roles']) { # …} else { # …}

# or generate the application config file # from configuration management system…

“I canʼt tell you how many Subversion users have told me the following story: “We tried to branch our code, and that worked fine. But when it came time to merge back, it was a complete nightmare and we had to practically reapply every change by hand, and we swore never again and we developed a new way of developing software using if statements instead of branches.

“Sometimes theyʼre even kind of proud of this new, single-trunk invention of theirs. As if itʼs a virtue to work around the fact that your version control tool is not doing what itʼs meant to do.”

—Joel Spolsky, http://www.joelonsoftware.com/items/2010/03/17.html

Distributed branching & merging is awesome

Use branches for early development

Branches merged into trunk

Use flags for rollout of almost-finished code

“I canʼt tell you how many Subversion users have told me the following story: “We tried to branch our code, and that worked fine. But when it came time to merge back, it was a complete nightmare and we had to practically reapply every change by hand, and we swore never again and we developed a new way of developing software using if statements instead of branches.

“Sometimes theyʼre even kind of proud of this new, single-trunk invention of theirs. As if itʼs a virtue to work around the fact that your version control tool is not doing what itʼs meant to do.”

—Joel Spolsky, http://www.joelonsoftware.com/items/2010/03/17.html

“I canʼt tell you how many Subversion users have told me the following story: “We tried to branch our code, and that worked fine. But when it came time to merge back, it was a complete nightmare and we had to practically reapply every change by hand, and we swore never again and we developed a new way of developing software using if statements instead of branches.

“Sometimes theyʼre even kind of proud of this new, single-trunk invention of theirs. As if itʼs a virtue to work around the fact that your version control tool is not doing what itʼs meant to do.”

—Joel Spolsky, http://www.joelonsoftware.com/items/2010/03/17.html

All existing revision control systems were built by people who build installed software

Web apps are not like installed apps

What would a revision control system built for supporting deployed web applications be like?

Thank you!

paul@paulhammond.orghttp://www.paulhammond.org/2010/trunk