Engineering Velocity @indeed eng presented on Sept 24 2014 at Beyond Agile

Engineering Velocity

Building Great Software

Through Rapid Iteration

Matt Schemmel

Engineering

Resume Products

I help

people

get jobs.

Indeed

#1 Job Site Worldwide

More people find jobs on Indeed than anywhere

else

Over 140M monthly visitors

Over 3B monthly searches

(other)

Quality ProductsLate '90s

High Performance

Late '90s

Early '00s

Quality Products

High Performance

Late '90s

Early '00s

Quality Products

Late '00s High Scalability

High Performance

Late '90s

Early '00s

Quality Products

Late '00s High Scalability

Early '10s Rapid Iteration

Rapid Iteration:

Better Products

Faster

http://go.indeed.com/tdwi

http://go.indeed.com/ddpd

http://go.indeed.com/tdwi

Indeed Product Development Principles

Measure Everything

Test Everything (A/B Experiments)

Iterate Quickly

Build

MeasureLearn

Lean Startup Cycle

Indeed Product

Development Cycle

Deliver

MeasureLearn

Indeed Product

Development Cycle

Quickly!

Deliver

MeasureLearn

Indeed Product

Development Cycle

MeasureLearn MeasureLearn

Deliver

Engineering Velocity

MeasureLearn MeasureLearn

Deliver

Engineering Velocity (n)[ˌenjəˈni(ə)riNG vəˈläsətē]

Our ability to quickly deliver implementations of

ideas to job seekers and employers

Deliver Developer Productivity

Software Architecture

Processes and Tools

Risk Management

Deliver

Software Architecture

Processes and Tools

Risk Management

Deliver

Job Search

Web App

STOP

EVERY

THING

Job Search

Web App

Job Search

Web App

Service Oriented Architecture:

Technical Challenges

Challenge: High Performance

● Speed is a feature

● 100ms response time

clic

ks

time-to-render

Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics

http://go.indeed.com/boxcar

BoxcarHigh Performance SOA Framework

BoxcarLoad Distribution without Middleware

http://go.indeed.com/boxcar

http://go.indeed.com/boxcar

Challenge: Compatibility over Time

● Any service can be updated at any time

● Any client can be updated at any time

● Any service or client can be rolled back

unexpectedly

Every service conversation must be forward

and backwards compatible

Represent complex data types

Extremely compact and performant

Forwards and backwards compatible

Boxcar: API Compatibility

Challenge: Deploy Independence

Forward/backward-compatible API is great.

But what about implementations?

Hard Scenarios

New API

Old server doesn't understand new client

Same API, Different Behavior

Old client doesn't understand new server

New client doesn't understand old server

ProctorEfficient A/B Experiment Management

http://indeedeng.github.io/proctor/

Proctor

● Visitor segmentation

● Population filters

● Contextual allocation

http://indeedeng.github.io/proctor/

Proctor: Filters

"rule" : "loggedIn"

"rule" : "smartphone &&

'JP' == country"

"rule" : "ua.os == 'ios' &&

ua.version >= '6.0'"

Sign Up Activate

Proctor: Visitor Allocation

10% 10%

Sign Up Activate

25% 25%

US

UK

Proctor

Manage Experiments → Manage Behavior

Job Search

Web App

Boxcar!

Decoupling

Web Front-Ends

Each page has

different features

different data

different operational requirements

different owners

Each page should be its own deliverable

Trends

Company Info

Forums

Salary

Job Search

Indeed

Indeed Web Properties Goals

Provide a consistent user experience

One implementation of shared functionality

Completely decoupled applications

Option: Shared Libraries?

Reduced flexibility of technology decisions

Requires a release of all projects

Occasional inconsistent user experience

Option: Shared Resources?

http://www.indeed.com/global/nav.js

http://www.indeed.com/global/nav.css

http://www.indeed.com/global/nav

On every app, include:

Option: Common Resource

Reduced flexibility of technology decisions

Requires a release of all projects

Occasional inconsistent user experience

No shared services

Solution: Navshell

Application-level Rewriting HTTP Proxy

Navshell

Request

Web App

HTTP

<head>

<title>Johnson & Johnson Careers</title>

<link type="text/css" href="common.css">

<link type="text/css" href="comp.css">

</head>

<body>

<div id="page-header">

......

</div>

<div id="page-body">

<div id="page-frame">

<div id="comp-header">

......

</div>

</div>

</div>

</body>

<head>

<title>Johnson & Johnson Careers</title>

<link type="text/css" href="common.css">

<link type="text/css" href="comp.css">

</head>

<body>

<div id="page-header">

......

</div>

<div id="page-body">

<div id="page-frame">

<div id="comp-header">

......

</div>

</div>

</div>

</body>

<head>

<title>Johnson & Johnson Careers</title>

<link type="text/css" href="common.css">

<link type="text/css" href="comp.css">

</head>

<body>

<div id="page-header">

......

</div>

<div id="page-body">

<div id="page-frame">

<div id="comp-header">

......

</div>

</div>

</div>

</body>

REQUEST HEADERS

Cookie, User-Agent, Referer, etcX-Indeed-Shell: 1X-Indeed-Logged-In: [01]X-Indeed-Email: chris@indeed.comX-Indeed-Locale: en_USX-Indeed-Geo: US

Navshell

Web App

RESPONSE HEADERS

Set-CookieX-Indeed-Page-Title: Indeed CareersX-Indeed-Include-Js-Footer: /js/app.jsX-Indeed-Include-Css: /style/main.css

Navshell

Web App

Navshell

No duplicate code

Centralized navigation changes

Plain HTTP with strong contract

Decoupled front-ends

Navshell

Requests

Services Web Apps

Web-Based Delivery of

Mobile App Features

Mobile

Mobile Browser Mobile App

Mobile

Mobile Browser Mobile App

● One code base for mobile web, iOS, Android

● Simultaneous release of features across

platforms

● No waiting for App Store approval

● No waiting for job seekers to upgrade

Shared Code Across Platforms

Easy To Reuse Infrastructure

● A/B testing in native apps challenging

● Activity tracking challenging

● Performance monitoring challenging

We have battle-hardened solutions to all these.

Mobile

Browsers

Mobile

Apps

Desktop

Browsers

Job Search

App

Software Architecture

Processes and Tools

Risk Management

Deliver

Product

Eng QA

Continuous Collaboration

Continuous Deployment Toolchain

Integration git

Verification Jenkins

JUnit, TestNG, Selenium, JaCoCo, …

Deployment Rundeck

● Configuration changes

● Resource re-allocation

● Database schema updates

● Internationalization

But what about the hard stuff?

● Configuration changes

● Resource re-allocation

● Database schema updates

● Internationalization

But what about the hard stuff?

http://go.indeed.com/i18n

<Indeed map of the world>

one search. all jobs.

una búsqueda. todos los empleos.

Jedno vyhledávání. Všechna pracovní místa.

Ein Klick. Alle Jobs.

一站搜尋，工作齊全

tek arama. bütün işler.

uma busca. todos os empregos.

één klik. alle vacatures.

あらゆる仕事まとめて検索

una ricerca. tutti i lavori.

Implement Commit

Deploy to

QA

Deploy to

ProductionVerify

Before Internationalization

Translate

Implement Commit

Deploy to

ProductionVerifyDeploy to

QA

Identify

Internationalization Process Overhead

Commit

+1 day

+1 day (20%)

+1 day (minimum)

<Indeed map of the world>

<Indeed map of the world>

<but with only US in orange>

I help [English-speaking

Americans]

get jobs?

http://www.gnu.org/software/gettext/

gettext utilitiesGNU Translation Project Tool Set

Template

en.po

Extract

✓ Many text sources: jsp, ftl, soy, js, …

✓ Dialects: en_US, en_CA, en_GB, …

✓ Variable substitution

✓ Complex plural forms

gettext utilities (with some help)

Machine Translation

English

Portuguese

Machine Translation

Low quality translations are better than no

translations

But we needed something better

Human Translation Service

Extract Improve Release

Combined Translation Service

Translate

Implement Commit

Deploy to

ProductionVerifyDeploy to

QA

CommitIdentify

Localization before Translation Automation

Implement Commit

Deploy to

ProductionVerifyDeploy to

QA

Extract

Localization with Translation Automation

Improve

And if there just isn't enough time…

"rule" : "language != 'es'"

"rule" : "locale != 'en_GB'"

"rule" : "country != 'IT'"

I help

everyone

get jobs!

Software Architecture

Processes and Tools

Risk Management

Deliver

Severity of issue

Probability of issue

Number of users affected

Risk

Smaller Releases

More releases → less functionality per release

Probability of issue

Proctor Feature Toggles

Probability of issue

"rule" :

"endsWith(email,'@indeed.com')"

Manage Cross-Product Releases

Deploy products then activate features

Probability of ISsue

Decouple Features from Releases

Rollback features and releases independently

# of users affected

Number of users affected

Incremental allocation changes

1% → 5% → 25% → 50% → 100%

Embedded QA

Probability of issue

Severity of issue

Monitoring

# of users affected

Monitoring and Alerting

● Identify runtime issues

● Alert Ops when an issue arises

Healthcheck Framework

● Applications and services self-report

internal state & health

● Divert traffic away from unhealthy instances

● Disable features for failing dependencies

Measure Everything

Software Architecture

Processes and Tools

Risk Management

Deliver

Better Product

● Emphasize goals over features

● Fail faster, cheaper

● Incremental progress live in production

Better Life

● No deadline crunches

● There's always another release coming

● More transparent schedules = less stress

We built all of this

incrementally, to meet

our own challenges

Yours are different

Q&A

Visit

http://engineering.indeed.com/

and

http://go.indeed.com/jobs