Moz Local
• Primarily about transforming API data.
• Parallel IO —> node.js!
• Large pile of independent services.
Why not SoA• Complex Interdependencies.
• Reliability.
• Debugging.
• Toolchains.
• Integration testing.
• Deployment.
The descent into madness.🌀
• Circus.
• Collateral flapping.
• Deploys result in downtime.
• Service-level versioning.
Problem 1: Collateral Flap
• We sucked at node.
• Domains weren’t a thing.
• Crashes happened a lot.
Problem 1: Collateral Flap
• Multiple instances of a given backend.
• Service registry and an HTTP proxy.
We “solved”
it!
Problem 2: Deployment Downtime
• Fire up a new backend that registers itself.
• Cleanly reap old backends, they finish their tea, crumpets, and requests and go away.
We “solved”
it!
DMV + Seaport + Viaduct is the greatest thing.
• Shields crashes!
• Zero-downtime deploys!
• Lots of control!
DMV + Seaport + Viaduct is the worst thing ever.
• Client tooling is immature.
• Development is basicallytorture.
• Inter-service versioning is unsolved.
Service versioning• “Solved” problem.
• URL Versioning
• Query Params
• Timestamps
• Inter-service dependencies are real, and require our attention.
• Is there a better way?
Service versioning redux
• git commit -m “replaced cats with dogs, life’s better now”
• b8d825ddd9eb63336f1da4ec46b1a47eecab7427
• That looks suspiciously like a version.
Routing layer versioning• We’ve got a HTTP proxy already.
• Let’s extend it to route to git SHA’s as versions.
• We route domains to default SHAs.
production.foo.com 5e3… Service!
staging.foo.com 61a…A
different service!
Routing Layer Versioning• Advantages:
• As an engineer, I don’t have to build versioning.
• We can test multiple releases simultaneously.
• Disadvantages:
• Deployed services have to be SHA reflective.
• We have to run multiple instances of a given service.
• Inter-service versioning.
Inter-service versioning
• Service A is dependent on Service B.
• Think a business logic service (A) and a data layer service (B).
• They might need to change in lockstep.
A
B
Inter-service versioning• We build a “Release version” to encapsulate all
our services at a given revision.
A
B
V1
A
B’
V2
Inter-service Communication• When A communicates with B, it’s got to specify a
version.
• Three ways;
• I have to think about it. (Use API versioning or some such)
• Lock at deploy time.
• Pass a header.
Lessons learned
• Inter-service versioning is unsolved.
• Rolling deployments are required.
• Shielding engineers from complexity.
• Caveat: Without magic.
Let’s deploy better. 🌅
• Focus on engineer happiness.
• Better tooling.
• Minimize moving parts.
• 4 daemons + 2 persistence layers == wat
• Make it pluggable.