Embed Size (px)
Citation preview
Optimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild
Europe, June 2015 Jonathann Zenou @zenoujohn
Footer
#jenkinsconf
Greetings fellow humans
• Hi, my name is Jonathann • Living in Tel Aviv, Israel • Originally from Marseille, France • Leading the DevOps effort at Ravello Systems • [email protected]
2
#jenkinsconf
Footer
What is Ravello Systems ?
#jenkinsconf
Footer
Ravello Systems
• 30 developers • 4 teams • 7 major projects • Kanban like
4
#jenkinsconf
Footer
A part of our stack
5
#jenkinsconf
Footer
What are we going to talk about ?
6
#jenkinsconf
Footer
Old problems
7
#jenkinsconf
Footer
Jobs Leftovers
• Causes full disk • Workspace Cleanup Plugin • Can pollute your maven repo or npm cache • Solution needed : a new clean environment for every job
8
Footer
#jenkinsconf
Better resource distribution
9
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
10
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
11
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
12
Backend Team Frontend Team
Footer
#jenkinsconf
Better resource distribution
13
Backend Team Frontend Team
“waiting for next available executor …”
Footer
#jenkinsconf
Better resource distribution
14
Backend Team/Frontend Team
Footer
#jenkinsconf
Better resource distribution
15
Backend Team/Frontend Team
Footer
#jenkinsconf
Better resource distribution
16
Backend Team/Frontend Team
Footer
#jenkinsconf
Frontend Team leader’s reaction
17
#jenkinsconf
Footer
Better resource distribution
• Solution needed : a rapid and scalable way to spawn dynamic environments
18
#jenkinsconf
Footer
Better resource distribution
19
#jenkinsconf
Footer
Using Docker in Jenkins
• Using Jenkins Docker plugin • Dedicated Docker host for each team • Hosts machine capabilities are a function of team size and
requirements • All images on all hosts • Problems solved : jobs leftovers and resource distribution
20
#jenkinsconf
Footer
Security
• No active directory or LDAP • Jim uses Dave’s account • Employee leaves the company
21
#jenkinsconf
Footer
Security
22
#jenkinsconf
Footer
Security
23
#jenkinsconf
Footer
Security
• Solution needed: third party dynamic authentication mechanism
24
#jenkinsconf
Footer
Security
• Using Jenkins Google Login plugin • Coupled with Matrix Authorization Strategy plugin • 2-Factor authentication • Open our Jenkins to the world
25
#jenkinsconf
Footer
Too much code in jobs
26
#jenkinsconf
Footer
Too much code in jobs
• Hard to maintain • Can be easily broken • “Who touched this ??”
27
#jenkinsconf
Footer
Too much code in jobs
• Solution: • Put your scripts in revision control (Github) • Using Jenkins Multiple SCM plugin
28
#jenkinsconf
Footer
Jobs Versioning
29
#jenkinsconf
Footer
Jobs Versioning • Solution:
• Using Jenkins JobConfigHistory plugin • Saves old configurations • Diff between job configurations • Revert job configuration
• Coupled with Jenkins SCM sync configuration plugin • Saves you whole Jenkins configuration in revision
control (Github)
30
#jenkinsconf
Footer
Jobs data sharing
31
Backend Job Frontend Job
Parameters: - Backend version: 1.1 - Frontend version: stable
Parameters: - Frontend version: 1.1 - Backend version: stable
#jenkinsconf
Footer
Jobs data sharing
• Solution needed: an interface/service that will store data that will be accessible from all the jobs
• Using Jenkins Global Variable String Parameter Plugin • Cannot alter them from a job • But you can using Groovy …
32
#jenkinsconf
Footer
Jobs data sharing
33
#jenkinsconf
Footer
Jobs data sharing
• Keep a global parameter for each of product stable version • Let all the other jobs use these values • Change the global parameter value of a product stable
version automatically when this product passes a green CI • Gist link: https://gist.github.com/johnyzed/2af71090419af2b20c5a
34
#jenkinsconf
Footer
CI at Ravello Systems
35
build +
unit tests
environment creation
code deployment test suites stop
environments
Artifactory
log collection
#jenkinsconf
Footer
CI at Ravello Systems
36
environment creation
#jenkinsconf
Footer
Before digging further
37
#jenkinsconf
Footer
The pieces are ready • What Do Developers Want ? • Blackbox • No duplication • C.D.R.S • Single form, with a lot of triggers • Using Multijob Plugin
38
#jenkinsconf
Footer
Jobs sandbox • Improving a job without:
• Breaking it • Stopping other people using it
• Solution needed: a parallel Jenkins environment almost identical to our production Jenkins environment to test the new changes on
39
#jenkinsconf
Footer
Jobs sandbox • Jenkins environment blueprint on Ravello Systems • Using Ansible:
• git clone • minor changes
• Running regular jobs • All that orchestrated from a Jenkins job
40
#jenkinsconf
Footer
Jenkins sandbox flow
41
changes on Jenkins script
branch
create dynamic Jenkins
environment run jobs stop dynamic
environment
#jenkinsconf
Footer
Jobs sandbox
42
#jenkinsconf
Footer
Comparing Jenkins environment
43
Old Jenkins New Jenkins
Number of jobs 137 56 Number of slaves 10 0-50 Build and test duration 70 43
#jenkinsconf
Footer
Comparing Jenkins environment
44
#jenkinsconf
Footer
Take-aways
• Ask your developers how to make their life easier • Don’t be afraid to break things • Culture • Follow @jenkins_release • Follow @devopsreactions
45
#jenkinsconf
Footer
Project Handover
46
#jenkinsconf
Footer
One last thing
• Ravello Repo • http://www.ravellosystems.com/repo/ • http://www.ravellosystems.com/repo/blueprints/60391436
47
#jenkinsconf
Footer 48
Questions ?
#jenkinsconf
Footer
Please Share Your Feedback
• Did you find this session valuable? • Please share your thoughts in the
Jenkins User Conference Mobile App. • Find the session in the app and click
on the feedback area.
49
#jenkinsconf
Thank you
50
