Upload
kenatindeed
View
144
Download
3
Embed Size (px)
DESCRIPTION
Increasing engineering velocity through architecture. This presentation was given at the Beyond Agile meetup group in Seattle on Sept 24, 2014.
Citation preview
Engineering Velocity
Building Great Software
Through Rapid Iteration
Matt Schemmel
Engineering
Resume Products
I help
people
get jobs.
Indeed
#1 Job Site Worldwide
More people find jobs on Indeed than anywhere
else
Over 140M monthly visitors
Over 3B monthly searches
(other)
Quality ProductsLate '90s
High Performance
Late '90s
Early '00s
Quality Products
High Performance
Late '90s
Early '00s
Quality Products
Late '00s High Scalability
High Performance
Late '90s
Early '00s
Quality Products
Late '00s High Scalability
Early '10s Rapid Iteration
Rapid Iteration:
Better Products
Faster
http://go.indeed.com/tdwi
http://go.indeed.com/ddpd
http://go.indeed.com/tdwi
Indeed Product Development Principles
Measure Everything
Test Everything (A/B Experiments)
Iterate Quickly
Build
MeasureLearn
Lean Startup Cycle
Indeed Product
Development Cycle
Deliver
MeasureLearn
Indeed Product
Development Cycle
Quickly!
Deliver
MeasureLearn
Indeed Product
Development Cycle
MeasureLearn MeasureLearn
Deliver
Engineering Velocity
MeasureLearn MeasureLearn
Deliver
Engineering Velocity (n)[ˌenjəˈni(ə)riNG vəˈläsətē]
Our ability to quickly deliver implementations of
ideas to job seekers and employers
Deliver Developer Productivity
Software Architecture
Processes and Tools
Risk Management
Deliver
Software Architecture
Processes and Tools
Risk Management
Deliver
Job Search
Web App
STOP
EVERY
THING
Job Search
Web App
Job Search
Web App
Service Oriented Architecture:
Technical Challenges
Challenge: High Performance
● Speed is a feature
● 100ms response time
clic
ks
time-to-render
Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics
http://go.indeed.com/boxcar
BoxcarHigh Performance SOA Framework
http://go.indeed.com/boxcar
Challenge: Compatibility over Time
● Any service can be updated at any time
● Any client can be updated at any time
● Any service or client can be rolled back
unexpectedly
Every service conversation must be forward
and backwards compatible
Represent complex data types
Extremely compact and performant
Forwards and backwards compatible
Boxcar: API Compatibility
Challenge: Deploy Independence
Forward/backward-compatible API is great.
But what about implementations?
Hard Scenarios
New API
Old server doesn't understand new client
Same API, Different Behavior
Old client doesn't understand new server
New client doesn't understand old server
ProctorEfficient A/B Experiment Management
http://indeedeng.github.io/proctor/
Proctor
● Visitor segmentation
● Population filters
● Contextual allocation
http://indeedeng.github.io/proctor/
Proctor: Filters
"rule" : "loggedIn"
"rule" : "smartphone &&
'JP' == country"
"rule" : "ua.os == 'ios' &&
ua.version >= '6.0'"
Sign Up Activate
Proctor: Visitor Allocation
10% 10%
Sign Up Activate
25% 25%
US
UK
Proctor
Manage Experiments → Manage Behavior
Job Search
Web App
Boxcar!
Decoupling
Web Front-Ends
Each page has
different features
different data
different operational requirements
different owners
Each page should be its own deliverable
Trends
Company Info
Forums
Salary
Job Search
Indeed
Indeed Web Properties Goals
Provide a consistent user experience
One implementation of shared functionality
Completely decoupled applications
Option: Shared Libraries?
Reduced flexibility of technology decisions
Requires a release of all projects
Occasional inconsistent user experience
Option: Shared Resources?
http://www.indeed.com/global/nav.js
http://www.indeed.com/global/nav.css
http://www.indeed.com/global/nav
On every app, include:
Option: Common Resource
Reduced flexibility of technology decisions
Requires a release of all projects
Occasional inconsistent user experience
No shared services
Solution: Navshell
Application-level Rewriting HTTP Proxy
Navshell
Request
Web App
HTTP
<head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>
<head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>
<head>
<title>Johnson & Johnson Careers</title>
<link type="text/css" href="common.css">
<link type="text/css" href="comp.css">
</head>
<body>
<div id="page-header">
......
</div>
<div id="page-body">
<div id="page-frame">
<div id="comp-header">
......
</div>
</div>
</div>
</body>
REQUEST HEADERS
Cookie, User-Agent, Referer, etcX-Indeed-Shell: 1X-Indeed-Logged-In: [01]X-Indeed-Email: [email protected]: en_USX-Indeed-Geo: US
Navshell
Web App
RESPONSE HEADERS
Set-CookieX-Indeed-Page-Title: Indeed CareersX-Indeed-Include-Js-Footer: /js/app.jsX-Indeed-Include-Css: /style/main.css
Navshell
Web App
Navshell
No duplicate code
Centralized navigation changes
Plain HTTP with strong contract
Decoupled front-ends
Navshell
Requests
Services Web Apps
Web-Based Delivery of
Mobile App Features
Mobile
Mobile Browser Mobile App
Mobile
Mobile Browser Mobile App
● One code base for mobile web, iOS, Android
● Simultaneous release of features across
platforms
● No waiting for App Store approval
● No waiting for job seekers to upgrade
Shared Code Across Platforms
Easy To Reuse Infrastructure
● A/B testing in native apps challenging
● Activity tracking challenging
● Performance monitoring challenging
We have battle-hardened solutions to all these.
Mobile
Browsers
Mobile
Apps
Desktop
Browsers
Job Search
App
Software Architecture
Processes and Tools
Risk Management
Deliver
Product
Eng QA
Continuous Collaboration
Continuous Deployment Toolchain
Integration git
Verification Jenkins
JUnit, TestNG, Selenium, JaCoCo, …
Deployment Rundeck
● Configuration changes
● Resource re-allocation
● Database schema updates
● Internationalization
But what about the hard stuff?
● Configuration changes
● Resource re-allocation
● Database schema updates
● Internationalization
But what about the hard stuff?
http://go.indeed.com/i18n
<Indeed map of the world>
one search. all jobs.
una búsqueda. todos los empleos.
Jedno vyhledávání. Všechna pracovní místa.
Ein Klick. Alle Jobs.
一站搜尋,工作齊全
tek arama. bütün işler.
uma busca. todos os empregos.
één klik. alle vacatures.
あらゆる仕事まとめて検索
una ricerca. tutti i lavori.
Implement Commit
Deploy to
QA
Deploy to
ProductionVerify
Before Internationalization
Translate
Implement Commit
Deploy to
ProductionVerifyDeploy to
QA
Identify
Internationalization Process Overhead
Commit
+1 day
+1 day (20%)
+1 day (minimum)
<Indeed map of the world>
<Indeed map of the world>
<but with only US in orange>
I help [English-speaking
Americans]
get jobs?
http://www.gnu.org/software/gettext/
gettext utilitiesGNU Translation Project Tool Set
Template
en.po
Extract
✓ Many text sources: jsp, ftl, soy, js, …
✓ Dialects: en_US, en_CA, en_GB, …
✓ Variable substitution
✓ Complex plural forms
gettext utilities (with some help)
Machine Translation
English
Portuguese
Machine Translation
Low quality translations are better than no
translations
But we needed something better
Human Translation Service
Extract Improve Release
Combined Translation Service
Translate
Implement Commit
Deploy to
ProductionVerifyDeploy to
QA
CommitIdentify
Localization before Translation Automation
Implement Commit
Deploy to
ProductionVerifyDeploy to
QA
Extract
Localization with Translation Automation
Improve
And if there just isn't enough time…
"rule" : "language != 'es'"
"rule" : "locale != 'en_GB'"
"rule" : "country != 'IT'"
I help
everyone
get jobs!
Software Architecture
Processes and Tools
Risk Management
Deliver
Severity of issue
Probability of issue
Number of users affected
Risk
Smaller Releases
More releases → less functionality per release
Probability of issue
Proctor Feature Toggles
Probability of issue
"rule" :
"endsWith(email,'@indeed.com')"
Manage Cross-Product Releases
Deploy products then activate features
Probability of ISsue
Decouple Features from Releases
Rollback features and releases independently
# of users affected
Number of users affected
Incremental allocation changes
1% → 5% → 25% → 50% → 100%
Embedded QA
Probability of issue
Severity of issue
Monitoring
# of users affected
Monitoring and Alerting
● Identify runtime issues
● Alert Ops when an issue arises
Healthcheck Framework
● Applications and services self-report
internal state & health
● Divert traffic away from unhealthy instances
● Disable features for failing dependencies
Measure Everything
Software Architecture
Processes and Tools
Risk Management
Deliver
Better Product
● Emphasize goals over features
● Fail faster, cheaper
● Incremental progress live in production
Better Life
● No deadline crunches
● There's always another release coming
● More transparent schedules = less stress
We built all of this
incrementally, to meet
our own challenges
Yours are different
Q&A
Visit
http://engineering.indeed.com/
and
http://go.indeed.com/jobs