Elastic jenkins with mesos and dcos (2016 01-20)

© 2016 Mesosphere, Inc. All Rights Reserved.

Roger Ignazio | January 20, 2016

ELASTIC JENKINS WITH MESOS AND DCOS

© 2016 Mesosphere, Inc. All Rights Reserved.

Roger Ignazio

QE Tech Lead @ Mesosphere

@rogerignazio

ABOUT ME

© 2016 Mesosphere, Inc. All Rights Reserved.

mesosinaction.com

MESOS IN ACTION

© 2016 Mesosphere, Inc. All Rights Reserved.

● Conventional methods for scaling Jenkins● Elastic Jenkins on Mesos and Mesosphere DCOS● Demo● Future work

AGENDA

© 2016 Mesosphere, Inc. All Rights Reserved.

ALLOW ME TO INTRODUCE…

© 2016 Mesosphere, Inc. All Rights Reserved.

● General-purpose cluster resource manager● Represents many machines as a single entity● Advertises resources directly to applications● Works at scale: Apple, Twitter, Netflix, Airbnb, …

A BRIEF INTRO TO MESOS

© 2016 Mesosphere, Inc. All Rights Reserved.

A BRIEF INTRO TO MESOS

© 2015 Mesosphere, Inc. All Rights Reserved. 8

A BRIEF INTRO TO DCOS● Enterprise-grade

distributed operating system

● Mesos as a distributed systems kernel

● Built using open source and commercial tools

Existing Infrastructure

Mesosphere DCOS

Services & Containers

© 2016 Mesosphere, Inc. All Rights Reserved.

● Open source, distributed CI tool● Repeatable build/test of software projects● Large community, rich plugin ecosystem

A BRIEF INTRO TO JENKINS

© 2016 Mesosphere, Inc. All Rights Reserved.

SCALING JENKINS

© 2016 Mesosphere, Inc. All Rights Reserved.

SINGLE JENKINS MASTER

Team A . . .

Jenkins Agent 1

Jenkins Agent 4

Jenkins Agent N

Team B

Jenkins Agent 2

Jenkins Agent 5

Team X

Jenkins Agent 3

Jenkins Agent 6

. . .

Jenkins Master

© 2016 Mesosphere, Inc. All Rights Reserved.

SINGLE JENKINS MASTER → FAILED!

Team A . . .

Jenkins Agent 1

Jenkins Agent 4

Jenkins Agent N

Team B

Jenkins Agent 2

Jenkins Agent 5

Team X

Jenkins Agent 3

Jenkins Agent 6

. . .

Jenkins Master

© 2016 Mesosphere, Inc. All Rights Reserved.

MULTIPLE JENKINS MASTERS

Team A Jenkins

. . .Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team B Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team X Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

© 2016 Mesosphere, Inc. All Rights Reserved.

MULTIPLE JENKINS MASTERS

Team A Jenkins

. . .Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team B Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team X Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

© 2016 Mesosphere, Inc. All Rights Reserved.

MULTIPLE JENKINS MASTERS → STATIC PARTITIONING

Team A Jenkins

. . .Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team B Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

Team X Jenkins

Jenkins Agent 1

Jenkins Agent 2

. . .

Jenkins Agent N

90% Utilized(Normal)

140% Utilized(40 builds in queue)

0% Utilized(Idle)

© 2016 Mesosphere, Inc. All Rights Reserved.

JENKINS ON MESOS

Team A Jenkins . . .

Mesos Agent 1

Mesos Agent 4

Mesos Agent N

Team B Jenkins

Mesos Agent 2

Mesos Agent 5

Team X Jenkins

Mesos Agent 3

Mesos Agent 6

. . .

Mesos Master

© 2016 Mesosphere, Inc. All Rights Reserved.

DEMO

© 2015 Mesosphere, Inc. All Rights Reserved. 18

FUTURE WORK

Tighter CI/CD integration with DCOS

● App deployment● Service discovery

© 2016 Mesosphere, Inc. All Rights Reserved.

Q & A@rogerignazioroger@mesosphere.com

mesosphere.com/productmesos.apache.orggithub.com/jenkinsci/mesos-plugin