Are we there yet?

Continuous Delivery @Comcast

Brendan O’Bra, Principal Engineer (brendan_o’bra@cable.comcast.com)Twitter: @brendanobra LinkedIn: https://www.linkedin.com/in/brendanobra

September 17th, 2015Scala Bay Meetup

We are more than just cable TV…

Internet Phone On Demand

Home Security

Business Services (Internet)

Next Gen TV (IP Video)

Internet TV Apps (On

X1)

TV Content Production

(NBC)

Movies (Universal)

3

Almost everything at “National Scale”

• 10’s of Datacenters across US • Traffic can be “surge-ish”• “Carrier Grade” Customer Expectations• Software Upgrades • Millions of Devices• Cloud heterogeneity is important• 1000’s of VMs running at any time, across

all clouds

Continuous Delivery @Comcast

4

In the past… Many humans, some machines MUCH time

Continouos Delivery @Comcast

5

Now: One Human, many machines much LESS time

Continouos Delivery @Comcast

6

Let’s go FAST

Market was clearly changing, with competition delivering some cool stuff. Time to market became a stronger driver We did some light reading:

7

And then drank gallons of Continuous Delivery

8

And then… Epiphany!

Immutable Servers!

9

We arrived at a system that consisted of:

Puppet

App Config

Hiera

App Automation

Openstack

VMWare

Route53

Git

Cloud-init

10

Which can also be viewed like this…

11

Feedback from initial attempts at Continuous Delivery

• It needs a GUI

• Why is it so hard?

• What are all those moving parts there? WTF?!

• write the GUI with whatever you want!

12

Gumby!Your elastic friend in cloud

13

V1 : Play + Akka in Scala

• We made an app, in Play! GUI was in Angular, REST layer -> Biz Layer in Play/Akka (one dude want to learn Akka/Play/Scala, other dude wanted to learn AngularJS). Gumby was born!

• It was a great experiment, it worked in production, and actually solved a real problem(s)

14

Gumby: Spray + Akka to scale the Comcast Cloud

• Then.. The folks that that make X1 heard about this “gumby” thing and told us they wanted to use it.. That’s great, but it was only a science experiment….

• Time to rewrite in Spray, and Go Big• Spray allowed up to really clean things up, and because it was so fast, we

could DOS any cloud api we encontered ;)

15

Gumby: Current Architecture (Monolithic)

Workflow (Akka)

VSphere

Openstack

EC2

REST(Spray)Client

16

Fun Facts about Gumby

• Can deploy ~400 vms per dc / hour• Currently about 21000 lines of scala• Typically does 2-3 full national deploys/week• Deploys 60% of national X1 backend footprint

(4000 VMs)• Will deploy 100% of national X1 backend footprint

by Q1 2016• Is transitioning to microservices• Can deploy itself• David Bolene: “One Big Side Effect”