Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next Generation CI
Building Eclipse projects on the new CBI build farm
Frederic GurrEclipse Foundation
EclipseCon Europe - October 25, 2018
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History & Status quo
2
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History
3
2011-2012
More slaves
Dedicated master for performance tests
2013
Hudson instance per project
Part of the CBI initiative
Shared instance with slaves for Windows and MacOS
2009
Single Hudson instance
Maintained by the community
2010
1 Master, 2 Slaves
Maintained by Eclipse IT team.
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
History
4
2017
Mass migration to Jenkins
Migration tool https://github.com/eclipse-cbi/hipp2jipp
Custom scripts
2016
Hudson ecosystem not catching up
Started to evaluate Jenkins
2018
Jenkins instance per project
No more Hudson
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
5
• 220+ Jenkins instances
• Beefy machines(total ~250 cores, ~1.5TB RAM)
• A few dedicated slaves(e.g. for Windows and Mac)
https://ci.eclipse.org
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
6
• Managing 10s of Jenkins is “easy”, managing 100s of them is time consuming
• We want to spend our time on areas that add value: • Harden digital signature and packaging services• Ease publication/promotion (Maven central, Nexus, ...)• Better support of static analysis tools (SonarQube, …)• Better integration with GitHub
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Today’s infrastructure
7
• We also want to offer better performance• Pool our hardware to use it more efficiently• QoS - better isolation and control over resources usage• Be able to add additional resources more easily
• Either cloud-based or on-premise
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next generation CI
8
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
A duet solution
9
( )
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
10
Operation Center
JIPPS
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
11
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
12
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
Dedicated agents
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CloudBees Core
13
Operation Center
JIPPS
Kubernetes/OpenShift Cluster
Jobs run on temporary, dynamically provisioned agents
Dedicated agents
Shared agents
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Next generation status quo
14
Already running on new infra:• Every new provisioned JIPP
• Early adaptors like Xtext
• ~40 Jakarta EE projects
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
CBC and cluster problems
15
• We had some stability issues in the beginning
• Responsive and excellent support by CloudBees
• Cluster is now much more stable
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Migrating OSS Jenkins instances to CBC
16
• How?• No built-in solution for mass migration of jobs• => Custom solution
• What needs to be moved?• Job configurations• Views• Credentials• Maven settings
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Roadmap
17
June2018
Setup
OpenShift/CJE running on Foundation infrastructure
August 2018
Early adopters
New JIPPs are created on CJE and a couple of volunteer projects are migrated
Soon
Migration
All JIPPs are migrated to CJE
October2018
JakartaEE
All JakartaEE projects run on CJE
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
18
FAQ & Best practices
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
19
• Why does my build take so long and how can I make it faster?
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
20
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Jenkins Pipelines
21
• Store your job config in your repo
• Config linked to code
• Config history
• Scripted Pipeline vs Declarative Pipeline
• Recommended for docker based builds
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
22
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
• Can I finally run docker based builds?
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
FAQ
23
• Why does my build take so long and how can I make it faster?
• What are Jenkins pipelines and why/when should they be used?
• Can I finally run docker based builds?• What do I need to do for the migration?
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Best practices
24
• Modular builds and tests => Parallelization
• Play fair and don’t be a resource hog
• Build configuration in Code => Pipelines
• Docker
More: http://wiki.eclipse.org/CI_best_practiceshttps://wiki.eclipse.org/Jenkins#FAQ
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Questions?
25
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
Takeaways
26
• Less administration => more tools and services
• New infra => better utilization and scaling• Stateless build agents will require
adaptations
Copyright (c) 2018, Eclipse Foundation, Inc. | Made available under the Eclipse Public License 2.0 (EPL-2.0)
27
Thank you!
Please evaluate the talks on eclipsecon.org!