Upload
oleg-nenashev
View
87
Download
2
Embed Size (px)
Citation preview
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
PhD, Hardware Engineering
Background: R&D in Hardware & Embedded • Intel Labs, Sitronics, Synopsys• Large-scale automation infrastructures
Jenkins Developer, CloudBees• CloudBees Jenkins Platform • Customer support and consulting
Core Team Member, Jenkins Project• Maintainer of plugins • Meetup organizer, GSoC Org Admin
Contributor, LibreCores
About me
@oleg_nenashev
oleg-nenashev
2
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
The Butler is still young –applying modern Jenkins features to the Embedded areaOleg NenashevCloudBees, Inc.
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Do know about Jenkins?
About you
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
1. The most popular CI automation server in the world
2. It’s a framework• Swiss Army Knife of Automation
3. It’s open-source• Can be customized
4. Commercial support is available
Who is Mr. Jenkins?
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 6
Computinggrids
DocsHW
peripherals
Plugins Configs
Build agents
JobsJenkins-based Automation System
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Do you use Jenkins?
About you
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Jenkins is Popular
http://stats.jenkins-ci.org/jenkins-stats/
• Common: ~70% of teams use Jenkins
• Almost 150k installations• ~300 active committers
(over last 6 months)• ~3 new plugins every week• ~50 plugin releases every
week
Hudson Jenkins
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Many Embedded companies use Jenkins
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
LTS release linesBackward compatibility•Jenkins 1.x was compatible for 8+ years•Jenkins 2.x is almost compatible with Jenkins 1.x
It’s also conservative about continuous improvement
10
Jenkins is Conservative
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
NEW WEBSITE
NOW IT’S A CD AUTOMATION SERVER
FOCUS ON USER EXPERIENCE
UI IMPROVEMENTS
SECURITY IS ENABLED BY DEFAULT
PIPELINE AS CODE
11
Jenkins 2
http://bit.ly/jenkins20Stats:• Released on April 2016• LTS line – July 2016• 50% adoption over six months
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Jenkins 2 is a milestone, work continues
USER EXPERIENCE• Better stability of installations• Better UIs (Blue Ocean, etc.)
ARCHITECTURE• (?) Pluggable storage• Remoting changes• Jenkins in the Cloud – ”new” requirements
PROJECT INFRASTRUCTURE• Contributor experience• Better quality of plugins & core finally
12
What’s going on in Jenkins?
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
GOALS• Share info about new Jenkins
features• Share experience about
Pipeline usage in Embedded• Give stickers & T-shorts for
good questions ;)NO-GOALS
• Continuous Integration 101• Jenkins 101• “Classic” approaches to
Embedded use-cases
About this talk
JUC2015/London
More info:
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
What is an Embedded Product?
14
`Hardware
External IPFirmware / embedded software
External software
Development tools / EDAs
Demo software Demo hardware
Developer kits
Main Product
Documentation
Integ-ration
Boom!
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Almost everything needs to be automated
Multiple technologies and tools
Unstable hardware and networks
Legacy stuff is everywhere
Limited resources
Hardware engineers are not software experts
15
Embedded Area. Welcome to Automation Hell!
TechnicalNon-technical
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Approach: Use Free-style and Matrix:https://jenkins.io/solutions/embedded/
Classic Approaches in Jenkins
JUC2015/London
More info:
Keywords:
• Script steps to run the logic
• Custom Tools - manage tools
• Throttle Concurrent Builds and Lockable Resources - dispatch access to the shared hardware
• Naginator - restart builds on-demand
• Parameterized Trigger / Copy Artifact -parallelize tests and to collect results
16
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 17
Spaghetti Automation
• Many job dependencies• Difficult build flow
tracking• Complex configurations • Duplication
• Maintenance is expensive
Classic Approaches in Jenkins
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
ü Storage together with the projectü Tracking together with the projectü Tests together with the project
…. as code?Configuration…Documentation…Pipeline… Everything…
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Jenkins Pipeline
19
Configuration as Code for Jenkins jobsNot just a configuration as code
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline is a…
Ø Groovy DSL for automatic flow description *• Alternatives – Job DSL, Jenkins Job Builder
Ø Job type in Jenkins (Pipeline Job)• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations
Ø Ecosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2
// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"
} http://bit.ly/pipeline-tutorial
20
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Snippet generator
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Stored in SCM • Jenkinsfile
Jenkins can create projects from SCM• Multi-branch Pipeline• Folder with a project• Jobs for branches and pull requests
Works even on the organization level• GitHub, BitBucket, etc.
22
Pipeline…. as code
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 23
Multi-Branch Pipeline
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline. Useful Features
Robustness against restarts of Jenkins master
Robustness against network disconnects• sh() steps are based on the Durable Task plugin• Jenkins continues execution once the node reconnects
Parallelization across multiple nodes:• It can be achieved via combination of parallel() and node() steps• No need to use the trigger/copy chain (but you can do it)
Ability to store the shared logic in standalone Pipeline libraries
Visualization: Pipeline Stage View, Blue Ocean
24
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline. Stage View Plugin
25
New UI: Blue Ocean is coming soon!
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Example 2: Running tests in Parallel
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Embedded Components Testing Approaches
Software Hardware
SimulatorsEDA tools
FPGAsASIC prototypesHardware
Accelerators, etc.
x
27
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Example. Jenkins Node with an Attached Board
28
VM or bare-metal
PowerReset
when it hangs
Deploy bit-files,Blasting cables
Debugging cables,firmware upload
We don’t change them,
huh?
Jenkins node
Drivers
Tools
Peripherals
Attached peripherals
Runtime
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Issue #1. Hardware is Unstable
29
Nodes @ Site A Nodes @ Site CNodes @ Site B“We borrowed FPGAs here, somebody takes them randomly.”
“This stuff has Windows drivers only.”
“No infra team members here.”
“We run Jenkins nodes on developer laptops.”
Frankenboards“We built them from the garbage we had.”
L
L
L
L
L
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Node Failover
Pipeline allows re-running unreliable stages on other nodes.
System failure
Node #1 Node #2. . .
. . .
for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;
}} catch (Exception ex) {if (ex.message.contains
("exit code 255")) {// Fatal errorfail("Test run failed")
}}
}
Try all possible boards:
30
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
We want to…• minimize the amount of hardware we use• utilize the hardware as much as possible• share hardware between projects
Nice to have:• reuse the existing hardware▸Developer computers (nightly builds)▸Manual QA labs▸…
31
Issue #2. Hardware is Expensive!
M – x000$
L –x0000$M - http://www.xilinx.com/univ/xupv5-lx110t.htmL - HAPS-64, www.synopsys.com
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Access should be dispatched by Jenkins
Builds utilize the hardware during a single phase only
Scheduled availability of nodes
Teams fight for resources
Clean environment (again)
32
Issue #3. Hardware is Shared
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Resource Management
33
Dispatching parallel access:• Lockable Resources plugin is
integrated with Pipeline• Throttle Concurrent Builds
plugin is not integrated, license limitations may be a problem
Restricting access:• Job Restrictions Plugin• Node access schedule
echo 'Starting'node('board1') {
stage('Build')git url: 'http://github.com/myorg/myproject.git'withEnv(["PATH+GCC=${tool ’gcc4'}/bin"]) {
sh 'make clean build' }
}
stage(’Run tests')lock(env.NODE_NAME + "-board1") {
// Run something on the board// ...
}}
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Tools…
34
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
1. Local tool installation on a node2. External Tool Managers (CLI)3. Jenkins-based solutions
-Via plugins (if available)-Generic approaches: Custom Tools, EnvInject, etc.-Limited availability in Pipeline
4. Virtualization
Tool usage approaches
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Advantages:•Quick setup if you have ready infrastructure•Load balancing on physical hardware: vCenter, Docker Swarm•Works well with configuration as code (e.g. Puppet)•Clean environment•Dozens of plugins in Jenkins
Disadvantages:•Poor scalability when you have many tool combinations•Complicated for BIG tools (common case for EDAs)
Virtualization for tools
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Docker PluginDocker Pipeline PluginDocker Custom Build Environment Plugin…Yet Another Docker Plugin
Example. Docker plugins
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Example. Docker-based tools and Pipeline
Docker Pipeline Plugin:
• You can share workspaces• External Workspace Manager• https://github.com/jenkinsci/external-
workspace-manager-plugin• You can create build chains
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
You do NOT need to use a single node
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 40
Example
https://github.com/jenkinsci/external-workspace-manager-plugin
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Jenkins Pipelines are ComplexHow to reuse them?
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸Jenkins system-wide▸Folder▸Job
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline Library Definition [1/2]
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline Library Definition [2/2]
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Using Pipeline Libraries
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
More examples:https://github.com/jenkinsci/pipeline-
examples
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline. What’s next?
• Stability / Fault tolerance• UI/UX, Blue Ocean• Plugin integrations• Syntax improvements• (?) Freestyle => Pipeline Converter
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Blue Ocean
https://jenkins.io/projects/blueocean/
Ø New UI for Jenkins Ø New UX for common Jenkins usersØ Deep integration with Pipeline
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Blue Ocean
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Pipeline. Dev tools
Available: •Minimal IDE: Auto-completion•Minimal static analysis•Documentation browser for steps•Library manager
Missing:•Debugger•Real-time code deployment from IDE•Real-time approval of Script Security•Freestyle => Pipeline Converter
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 53
Summary. Embedded projects & Pipeline
SO…• We successfully adopted Jenkins Pipeline in several projects ▸E.g. LibreCores CI (http://librecores.org/)
• We got advantage in user experience: Templates, Blue Ocean, Pipeline Model Definitions
LESSONS LEARNT• Pipeline greatly simplifies task parallelization and failover within a single build
• Pipeline saves much time on automation system maintenance• Pipeline is a programming languages, high barrier to entry• Pipeline still lacks Dev Tools and integrations, but it evolves
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
There are many articles about Jenkins in the Embedded area•Most of them describe the “classic approaches” based on Freestyle jobs
Embedded solutions page:•https://jenkins.io/solutions/embedded/
Blogposts:•https://jenkins.io/node/
54
More Info Embedded Solutions Page
…
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM
Jenkins project:• Website: https://jenkins.io• Embedded Solutions page: https://jenkins.io/solutions/embedded/
Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples
CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform
55
Useful Links
© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees ,#CoDeSTHLM 56
Thank You!
Where to find me:
E-mail: [email protected]
GitHub: oleg-nenashev
Twitter: @oleg_nenashev
Want to get more from Jenkins? go.cloudbees.com