Elastic Streaming Spark Streaming + Dynamic Provisioning + Dynamic Allocation

Neelesh Shastry, ArchitectShaun Klopfenstein, CTO

The Vision

Requirements

Page 4Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Business Requirements

• Nearreal-timeactivityprocessing• Billons activitiespercustomerperday• Improve costefficiencyofoperationswhilescaling up• Globalenterprisegradesecurity andgovernance

Page 5Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

SAAS Requirements

• Customersareaddedandremoved• Fairnessandthrottlingpercustomer• Strictsequentialeventprocessingforsomeapplications• Temporarilysuspendacustomer,whenerrorsoccur

Technology Selection

Page 7Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Use Cases

• Reacttoactivities• Sendanemailwhensomeone visitsawebpage• Changethescorewhensomeone fillsaform

• Replicatedata• BuildSolrIndexes, near real-time• UpdateDataXChange– aninternal leadcache• Syncto/fromCRMSystems

• Analytics• Incrementallyupdateemailreports• Enrichactivitiesandfeed toDruidforadvancedemail/webreports

Page 8Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Why Spark Streaming?

• Micro-batchingprovidessink-sideefficiencies• GreatintegrationwithKafka• Nostrictrealtimeprocessingrequirements• Greatcommunity,industryadoption

Page 9Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Challenges with Spark + Kafka

• Nowaytoadd/removetopicsonthefly• NooutoftheboxsupportforsequencingRDDs• Nosupportforturningofftopicsundererrors• DoesnotplaywellwithscalingKafkapartitionsup/down,whenorderingisrequired

Page 10Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Challenges - Stragglers

• Abatchcan’tcompleteuntiltheslowestoperationfinishes

• Manyofourbatchesincludeslowoperations• Sometimesdon’tcompletewithinthebatchtime

• Batchesaremultitenant• onecustomersoperationcandelayprocessingforothercustomersinthesamebatch

• Severeimpactonutilization&batchdelay

Architecture & Design

Page 12Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Marketo Activity Architecture

Page 13Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Kafka Topics Organization

• Onetopicperusecase,datafromallcustomers• Easytomanage• Asinglecustomercancreatebacklogsforothersduringactivitystorms

• Fairness/throttlingishardtoimplement

• Onetopicperusecase,percustomer• Stormsareisolatedtothecustomer• Fairness/throttlingiseasytocontrol,bytweakingthetopic• PressureonKafkaZK– sofarnotaproblem

Solutions

Page 15Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Dynamic provisioning capacity

JobGenerator

DAGScheduler

Executor1

Executor2

MultitenantKafka

DStream

OffsetManager

ProvisioningFramework

CustomerRegistry

Add/Remove

Check & Pull Changes

compute#Get new offsets

Generate RDD

Submit Job

Schedule Tasks

Page 16Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Marketo Offset Manager

• Tracksmultitenancy• StreamingJobsprocessdataformanycustomers

• AccessingmultipleKafkatopicsandpartitions

• Addsnewtopics• Remove/Deactivate/Suspendtopics

Page 17Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

• EnablesefficientmultitenantRDDs• ControlledsequencingofRDDs• CoalesceKafkapartitions

• Bin-packingforefficiency

• Maintainspartitionlineageforoffsetmanagement

Multitenant DStream

Page 18Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Provisioning

• Managesallocatingcustomerstoasparkstreamingapplication

• roundrobin+resourceaffinity• Enablesrebalancingofcustomersacrosssparkstreamingjobs

• Oozie basedframework

Page 19Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Dynamic Resource Allocation

• SPARK-12133• Goal– “makeprocessing timeinfinitelyclosetoduration”• Assumes tasksareroughlysimilar

• Stragglersthrowthisgoaloff• Whatwereallywant:

• DRA+Safeconcurrent jobexecution

Page 20Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Results so far

• ~10differentusecases• >100SparkExecutors• >1000KafkaPartitions• Processinglatencies<5s(99th %)• Rolledoutto~20%customers

Future Work

Page 22Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Application Scheduling

• Schedulingwithinanapplicationtohandlestragglers• spark.streaming.concurrentJobs• Exploringschedulerpools• ChangestoStreamingJobScheduler,toexecutemultipleRDDssafely

Page 23Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Scaling Up Kafka Partitions

• Ourcustomersgrowinsizeoveraperiodoftime• Orderingrequirementsmeanwecannotaltertopiconthefly

• Coordinationrequiredonbothproducer&consumerfronts

• Enhanceprovisioner tomanagepartitionup/downscaling

Page 24Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Move to 2.x and Open Source!

We’re Hiring! Http://Marketo.Jobs

Q & A

Q & A

Page 27Marketo Proprietary andConfidential|©Marketo, Inc.10/30/16

Architecture Requirements

• Maximizeutilizationofhardware• Multitenancy supportwithfairness• Encryption,Authorization&Authentication• Applicationsmustscalehorizontally

Deploying It

Running It