Breaking the MonolithOrganizing Your Team to Embrace Microservices

Paul Osman, 500px

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”

— Melvyn Conway, 1967

A Story

500px - Made of people

500px Team

• Approx 40 full-time employees• Approx 20 full-time engineers + 6 designers / ux researchers• Product Teams

- iOS & Android (3)- Prime (2)- 500px Web (4)- Platform Engineering (Operations + Dev) (7)- QA / Release Engineering (3)

Microservices

Our Journey

January 2014

• Classic web architecture (LB, App Servers, DB Servers).

• Many data stores, one large monolithic codebase.

• Scaling: all or nothing.

• Cascading failures. Lots of SPOFs.

• Large, unwieldy codebase.

• “Get in line” approach to deployments.

Search & Indexing

• Indexing is naturally asynchronous.

• Introduced dependency on RabbitMQ.

• Wrote new indexer in Go. First service and first Go project.

- Huge performance improvement. 20 hours to 20 minutes.

• Search service - HTTP service that translates query strings into ES queries.

• Eliminated major SPOF. ElasticSearch / Search outages no longer affect customers.

• Isolated knowledge of data store (ElasticSearch).

Uploads & Resizing

• Perfect for first service to move to EC2.

• Continued in Go, because it was working well for us.

• Performance improvement. Upload times dropped significantly.

- Bulk uploads via Lightroom plugin.

• Eliminated another SPOF. Next step: front-end work to gracefully communicate to user.

• Eliminated write-half of S3 knowledge. Read-half still to be done.

Activities & Notifications

• Ongoing.

• Replacing poorly designed service built on top of MongoDB.

• Will eliminate another SPOF. Fix a lot of reliability complaints.

• Get to revisit architecture, centre around data needs (both real-time and ETL workflows).

3 Lessons

Lesson # 1 - DevOps

12 Factor Apps *

* http://12factor.net/

Monitor Everything

On Call Rotation

Measure Incident Response

* MTTD * MTTR

Gameday Exercises

Automation

Lesson # 2 - Service Design

Where to Start

500px in January

Search & Indexing

Uploads & Resizing

Use Circuit Breakers

Use Circuit Breakers

https://github.com/500px/ruby-service-client

Where to Start

• Pick a business capability that is:

• Off the Critical Path

• Can be double-written to easily (i.e. indexing, aggregating, etc)

• Look out for natural integration points between disparate systems — also good candidates

Design for Business Capabilities

• Break your application into business capabilities (i.e. search, recommendations, uploads).

• Tempting to design around technology layers (front-end, back-end). Don’t.

• Even if you don’t (yet) have the numbers, treat each business capability as a team.

Lesson # 3 - Teams

API Facades

Team Structure

• Web App Team - Own front-end code, web-bff API facade

• Search Team - Own search infrastructure & service

• Media Team - Own upload service, resizing & converting services, watermarking, etc

• Mobile Team - Own mobile apps, mobile-bff API facade

• Platform Eng - Everything else, operations, devtools, QA / Release engineering

Cross Functional Teams

Move People Around

Thank you!

Paul Osman@paulosman

paul@500px.com

https://500px.com/jobs