The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015

  • View
    226

  • Download
    2

Embed Size (px)

Text of The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015

1. The$Direc)ons$Pipeline$at$Mapbox 2. Goal Always(have(the(freshest(map(data(available(for(rou6ng Allows(to(6mely(x(data(problems 3. 1. Get&OpenStreetMap&data 2. Pre1process&for&Direc8ons 3. Load&new&data&into&API&servers 4. Repeat Every'step'is'an'own'CloudForma6on'stack. 4. Overview 1. Get$OpenStreetMap$data 2. Pre'process,for,Direc0ons 3. Load,new,data,into,API,servers 4. Repeat Every'step'is'own'CloudForma6on'stack. 5. OpenStreetMap Planet.osm,is,all,OpenStreetMap,data,in,one,le New,version,release,every,week Big,le:,576,GB,uncompressed,,28,GB,as,compressed,protobuf Incremental,di,update,available,every,minute 6. OpenStreetMap 1#EC2#c3.2xlarge#instance Replays#latest#changesets#to#generate#a#new#planet#le New#planet#every#2#1/2#hours,#uploaded#to#S3 Updates#a#le#/latest#with#a#reference#to#most#recent#planet Old#planets#are#purged#via#S3#Lifecycle#Object#ExpiraHon 7. Overview 1. Get$OpenStreetMap$data 2. Pre'process,for,Direc0ons 3. Load,new,data,into,API,servers 4. Repeat 8. Overview 1. Get&OpenStreetMap&data 2. Pre$process)for)Direc-ons 3. Load&new&data&into&API&servers 4. Repeat 9. Direc&ons*pre,processing 1#EC2#r3.4xlarge#on#Spot#Pricing#per#prole Fetch#latest#OSM#planet#from#S3#(discovery#via#/latest) Run#preDprocessing#for#prole Car:#6#Hours Bicycle:#15#Hours Walk:#23#Hours 10. Direc&ons*pre,processing Upload(results(to(S3((and(update(/latest) Update(CloudForma6on(stack(of(API 11. Overview 1. Get&OpenStreetMap&data 2. Pre$process)for)Direc-ons 3. Load&new&data&into&API&servers 4. Repeat 12. Overview 1. Get&OpenStreetMap&data 2. Pre1process&for&Direc8ons 3. Load%new%data%into%API%servers 4. Repeat 13. Direc&ons*API One%CloudForma/on%stack%per%Prole Several%EC2s%r3.2xlarge ELB/AutoScaling On%start,%downloads%latest%Direc/ons%data%from%S3 EC2%instances%are%cycled%on%CloudForma/on%parameter%update 14. Direc&ons*pre,processing aws cloudformation update-stack--stack-name "$ApiDirectionsStack"--use-previous-template--capabilities "CAPABILITY_IAM"--parameters $params ParameterKey=LatestTimstamp,ParameterValue="$TimeStamp" 15. Direc&ons*API*CloudForma&on*Template { "Parameters": { "LatestTimstamp": { "Type": "String" } }, "LaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", "UserData": { { "Ref": "LatestTimstamp" } } }, "AutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", "UpdatePolicy": { "AutoScalingRollingUpdate": { } } } } 16. The$Good Few%moving%parts%and%easy%concept Upda5ng%stack%parameter%mirrors%our%exis5ng%deployment%ow Data