Upload
j2ch5en
View
881
Download
1
Embed Size (px)
DESCRIPTION
An presentation made by a software vendor.
Citation preview
© CMC Media 2008© CMC Media 2008
Challenges and Characteristics of
Enterprise Continuous Integration
Moderator
Patrick EganPublisher – CM Crossroads and Agile Journal
Speakers
Paul Julius – Willowbark Consulting
Anders Wallgren – Chief Technical Officer Electric Cloud
CM Crossroads Webcast Series
www.electric-cloud.com
Challenges and Characteristics of Enterprise
Continuous Integration
October 9, 2008 Slide 3
Introduction
In the past…
Teams spent weeks or longer verifying that applications
worked
Time spent „integrating‟ was tortuous, lead to integration
„storms‟ of broken builds
Developers would have to track down bugs introduced
months earlier
The answer to this painful process is Continuous Integration
October 9, 2008 Slide 4
CI: The Beginning
Continuous Integration (CI) dictates that, as soon
as changes are made, they are integrated and run
through a rigorous build and test cycle
Build engineers began using various CI tools,
sometimes without management approval
Once these tools became known, they were
embraced by developers who wanted and needed
the instant feedback on the health of their builds
As teams and applications grow over time, many
enterprises find themselves with CI servers
spread across the company
October 9, 2008 Slide 5
Challenges
With the proliferation of multiple standalone
systems, next step is to investigate the idea of
consolidation
Consolidated approach will help:
Teams to reuse procedures and tools
Better utilize hardware
Free personnel from manual, low value tasks
October 9, 2008 Slide 6
Challenges in Real World Enterprise CI
No centralized build-test-deploy infrastructure
Difficulty scaling the environment
Testing and resource administration increase in
complexity
Hardware Utilization difficulties
Barriers to sharing and reusing scripts
Difficulty tracking builds and cross-project
reporting
October 9, 2008 Slide 7
S.T.A.R
The challenges of Enterprise CI require a CI
solution that is:
Scalable
Turnkey
Automatic
Rapid
October 9, 2008 Slide 8
Scalable
Has many aspects
Must scale by:
Number of projects
Size of an individual project
Number of users of the system
Number of machines in the build cluster
Complexity of workflow
Enterprise CI is fundamentally about many projects and
many users
October 9, 2008 Slide 9
Turnkey
Enterprise CI has to be turnkey. The number of
builds, number of users, and the reality of 24/7
operations make this imperative
Turnkey implies that from the very beginning,
Enterprise CI must be useable. It must be trivial
to add scripts to the system, without requiring a
rewrite.
October 9, 2008 Slide 10
Automatic
CI system must be automatic, builds must be
launch-able with a single click, but it must also be
compatible with existing scripts
Three forms of automatic: On demand, On
stimulus and On the clock
On Demand: Builds should be a single click away
On Stimulus: Typically caused by changes in the
SCM system
On the Clock: Replacement for ‘cron’
October 9, 2008 Slide 11
Rapid
As the system scales to mulitple projects, and the
number of builds increase, builds have to be
rapid.
Hours long builds are a deal breaker
Reducing build times from hours to minutes
typically is done using parallelism
Enterprise CI systems should be able to work in
tandem with build parallelism software
October 9, 2008 Slide 12
Other Properties
Three other properties are important to
Enterprise CI:
Auditing
Virtualization
Reporting
October 9, 2008 Slide 13
Customer Example
Problem
Medium sized Product company
10‟s of disparate existing builds
Varied build infrastructure
Nearly impossible to identify a “good” build
Solution
Unified Dashboard for all builds
Consistent build infrastructure
Templated, resuable build infrastructure
October 9, 2008 Slide 14
Next Steps
It’s easy to get started with an open source CI
system
But as your needs change and your company
grows you have to evaluate the effectiveness of
this solution.
October 9, 2008 Slide 15
Build vs. Buy
Key question then, should we use an open source
tool, build our own, or buy a commercial tool?
October 9, 2008 Slide 16
Four Key Areas
As we mentioned four key areas to review
Fast Builds
Software keeps growing = longer builds
On the Clock Builds
Scheduled Builds
The classic „nightly build‟
On-demand Builds
Ad hoc, as needed builds
On-Stimuli Builds
Driven by changes in SCM
October 9, 2008 Slide 17Slide 17
Build Your Own?
Features
Investm
en
t
Trigger builds
after check-ins
Automated build
and test for one build tool, one
machine
Scheduled jobs
Seems pretty straightforward…
October 9, 2008 Slide 18Slide 18
Build Your Own?
Features
Investm
en
t
Trigger builds
after check-ins
Database of
build results
Automated build
and test for one build tool, one
machine
Notifications via e-mail/RSS
Scheduled jobs
Web-based
access to results
Just a few finishing touches
CruiseControl
gives you this
for Java/Ant
October 9, 2008 Slide 19Slide 19
Build Your Own?
Features
Investm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Multiple servers, concurrency
October 9, 2008 Slide 20Slide 20
Features
Build Your Own?In
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Better monitoring and control
October 9, 2008 Slide 21Slide 21
Features
Build Your Own?In
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Tools for
extracting
data from logs
Trend reports
Cross-product
reports
Resource
utilization
reports
Annotate builds
after completion
Customizable
reports
Better reporting
October 9, 2008 Slide 22Slide 22
Features
Build Your Own?In
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Tools for
extracting
data from logs
Trend reports
Cross-product
reports
Resource
utilization
reports
Annotate builds
after completion
Customizable
reports
Logging and
error reporting
Retry steps after
errors/ failures
Timeouts for
runaway job
steps
Detect resource
failures
Better error handling
October 9, 2008 Slide 23Slide 23
Features
Build Your Own?In
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Tools for
extracting
data from logs
Trend reports
Cross-product
reports
Resource
utilization
reports
Annotate builds
after completion
Customizable
reports
Logging and
error reporting
Retry steps after
errors/ failures
Timeouts for
runaway job
steps
Detect resource
failures
Developer access
to production
builds
LDAP
authentication
Role-based
access control
Shared access by
multiple teams
User
impersonation,
password
management
Priorities
Handle multiple
independent groups
October 9, 2008 Slide 24Slide 24
Features
Build Your Own?In
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Tools for
extracting
data from logs
Trend reports
Cross-product
reports
Resource
utilization
reports
Annotate builds
after completion
Customizable
reports
Logging and
error reporting
Retry steps after
errors/ failures
Timeouts for
runaway job
steps
Detect resource
failures
Developer access
to production
builds
LDAP
authentication
Role-based
access control
Shared access by
multiple teams
User
impersonation,
password
management
Priorities
Conditional steps
Modular,
composable
process steps
Extensibility
Parameters Control
environment
for jobs
More programming
features
October 9, 2008 Slide 25Slide 25
Features
Build Your Own? It Never EndsIn
vestm
en
t
Automated build
and test for one
build tool, one
machine
Scheduled jobs
Trigger builds
after check-ins
Web-based
access to
results
Notifications via e-mail/RSS
Database of
build resultsSingle job can use
multiple resources
Run job steps
in parallel
Multiple jobs run
on one resource
simultaneously
Run multiple
builds
simultaneously
Resource
pooling, load
balancing
Multiple servers,
remote invocation
Resource
selection criteria
Cancel jobs
View partial
results of
builds in
progress
Monitor system
status
On-demand job
invocation via
Web
Tools for
extracting
data from logs
Trend reports
Cross-product
reports
Resource
utilization
reports
Annotate builds
after completion
Customizable
reports
Logging and
error reporting
Retry steps after
errors/ failures
Timeouts for
runaway job
steps
Detect resource
failures
Developer access
to production
builds
LDAP
authentication
Role-based
access control
Shared access by
multiple teams
User
impersonation,
password
management
Priorities
Conditional steps
Modular,
composable
process steps
Extensibility
Parameters Control
environment
for jobs
Portable across
hardware/OS
platforms
SCM integration,
bill of materials
Support multiple
languages and
build tools
Web interface for
editing processes
Command-line
interface
October 9, 2008 Slide 26
Build Your Own
You could try that
Before you do think about…
How much time do you have to write all that code?
How you are going to manage 200+ builds per day?
How are you going to get the build down from 4 hours to 15
minutes?
How are you going to manage hardware failure in your build
system?
Build Managers now control the heartbeat of
software engineering
They need something more than a creaky Perl script
October 9, 2008 Slide 27
Build Your Own
Fast builds
Buy lots of SMP hardware and try out GNU Make parallelism
or manually parallelize the build.
Scheduled builds
Use cron for that
On demand builds
Build an intranet page, integrate it yourself with the current
build and source code management system
Stimuli builds
Build ad-hoc script attached to source code management
system
October 9, 2008 Slide 28
Open Source
Fast builds
distcc, ccache, GNU Make -j
Scheduled Builds
cron, CruiseControl
On-demand Builds
Apache Continuum
Stimuli Builds
CruiseControl has some support
SourceControl plug-in
October 9, 2008 Slide 29
Commercial
Build speed tools
Like ElectricAccelerator
Build visualization and troubleshooting tools
Like ElectricInsight
Build management tools
Like ElectricCommander or CruiseControl
Together these tools enable build managers to
build, package, test, and deploy software
And they enable Agile development
October 9, 2008 Slide 30
Stepping Stones
Builds up 18 months little by little
Four stepping stones
October 9, 2008 Slide 31
Stepping Stones (1)
Step 1: Put full builds in the hands of developers
Set up an internal web page that allows a developer to fire off a build
The developer should be able to select:Which branch to build…
… or the location of his personal source tree
Whether to run unit tests or not
The build runs and the engineer gets an email with the result
Start a new rule: developers run on demandbuilds before they commit
October 9, 2008 Slide 32
Stepping Stones (2)
Step 2: Buy those lava lamps
Set up the full build system to run more than
once per day
Set up a system the feedback to the entire team
whether the full build succeeded
red/green lava lamps or
a flat panel screen mounted high enough for engineering to
see and
a web page showing live build status
Scheduled builds start to be the heartbeat of
engineering
October 9, 2008 Slide 33
Stepping Stones (3)
Step 3: Speed up the build
Look into tools that can speed up your build
Set an under-60 minute goal for a full build and
smoke test
Announce to the team that multiple integrations
can now happen per day, during the day
These fast builds begin to enable agile
development
October 9, 2008 Slide 34
Stepping Stones (4)
Step 4: Fully automate
Integrate your fast builds with your source code
management system
When the source code changes run a stimuli
build and test it
Now the lava lamps really matter
Now the build is the heartbeat of engineering
Start referring to yourself as the Build Automator
not the Build Manager
October 9, 2008 Slide 35
When to buy
Look at $$$ build tools when
Existing builds are too slow; developers complain
Existing builds are frequently broken
Existing builds are blocking implementation of new
techniques (e.g. Agile/XP)
Development teams are spread across the country or the
globe
Other drivers are
Company mergers
Implementation of new target platforms
A rapidly growing code base
October 9, 2008 Slide 36
Summary
No one builds their own:
Editor
Compiler
Debugger
SCM
Most people don’t build their own:
Issue tracker
Test system
Why DIY the build system?
October 9, 2008 Slide 37
For More Information
Electric Cloud
408.419.4300
www.electric-cloud.com
Willowbark Consulting
312.560.5351
www.WillowbarkConsulting.com
© CMC Media Inc. 2008© CMC Media Inc. 2008
Questions and Answers
Please post your questions now using
the “Ask a Question” box
on left side of the screen
View other Webcasts in the Series at www.cmcrossroads.com/wc