How did you know this Ad will be relevant for me?!

Behavioral Targeting @ Scale - How did we know that this Ad was relevant for you ?

Savin GoyalSivasankaran Chandrasekar

ADVERTISER

ROCKET FUEL

200+RTB

advertisingsupply

partners

50+ MnWebsites

50+ BnDaily impressions

3B WW CONSUMERS

100,000+ DEVICES

Exchanges

AdExchange

Rocket Fuel Platform

Auto Optimization

Real-Time Bidding

Agencies

Data Partners

Display Advertising Ecosystem

Bid on Ad

User Data

Bid Request

Rocket Fuel Winning AdAd Request

Ad Served to User

Page RequestWeb Browser

Rocket Fuel Platform

Smart Ad Servers

Response Prediction

Models

Calculate Propensity Score

5User Engagement Recorded

9 User Engages with Ad

Publishers

Refresh learning

Campaign & Audience

Qualify Campaign

Data Partners

Exchange Partners

Programmatic Buying

$2.38965$0.6782$1.7234

$0.09$1.78964$1.6782$1.7234$0.809$2.421.25

$2.11$1.26

$2.178$2.056$0.809$2.421.25

$2.11$1.26$2.78$1.56

$1.809$2.421.25

$2.11$1.26$2.78$0.56$2.421.25

$2.11$1.26$2.78

$0.756$0.809$2.421.25

$2.11$1.26$2.78

$1.256$1.809$2.421.25

$2.11$1.26$2.78

$0.586$2.009

1.25$2.11$1.26$2.78$1.56

Site/PageGeo/WeatherTime of DayBrand AffinityUser

[ + ][ + ]

Real Time Auction

Goal:Leads& sales

Goal:Coupondownloads

Goal:Brandawareness

Site/PageGeo/WeatherTime of DayBrand AffinityDemo

Impression ScorecardDemoBrand AffinityTime of DayGeo/WeatherSite/PageAd PositionIn-marketBehaviorResponse

Impression ScorecardDemoBrand AffinityTime of DayGeo/WeatherSite/PageAd PositionIn-MarketBehaviorResponse

+100+40-20+20+15+10+40+35

+40-70-20+10+15-25-40

-18+0.7

+10-10-20+20+10-35-25+10

+1.4% X✓

Real Time Auction

Scalable Predictive Models

Age/Gender

Occupation

IncomeEthnicity

Purchase Intent

OnlinePurchases

OfflinePurchases

BrowsingBehavior

Site Actions

Zip CodeCity/DMA

Search Sites

SearchCategories

Recency

Search Keywords

Web Site/Page

Referral URL

Site Category

Bizographics

Social

Interests Lifestyle

Positive Lift

Marginal Impact

Negative Lift

-7 +17

+17 +19

+8 +11

+8 +14X

+17 +8

+11X -12-10

+11-10+13

+28 +6

+13+19

Automated Feature Selection

▪ Infinite number of models

▪ Determine perfect model size

▪ Balance past data fit

and future generalization

Learn-Test-Refine

▪ Automatically learn from

each response

▪ Cross-validate - A / B testing

infrastructure

▪ Training pipeline

Throughput

Rocket Fuel Scale

▪ 34,474 CPU Processor Cores▪ 2655 servers▪ 187.4 Teraflops of computing

▪ 188 Terabytes of memory▪ 13X the memory of Jeopardy-

winning IBM Watson

▪ 42 Petabytes of storage▪ 106X the data volume of entire

Library of Congress

200 Servers 1400 Servers

1 Year

41 PB8x

Data Warehouse Growth

Behavioral Targeting

▪ Leverage online activities on the web to learn about user’s ▪ Long Term Interests

▪ User is interested in luxury cars▪ Short Term Interests

▪ User is looking for a pizza right now

▪ Expand user set beyond retargeting▪ Explore v/s Exploit

▪ Identify relevant users even if they have never been targeted previously

Behavioral Targeting @ Rocket Fuel

Label Data

Train Model

Back Test

Calibrate

TrainingEvents

Pixel Stream Ad Logs

BT Features (HBase)

Feature Generation

Score Profiles

Profile Generation

Scoring

Ad Serving Data Centers Model

Hadoop/HBase @ Rocket Fuel

▪ Cluster Highlights▪ 650+ Slaves (64 GB + 12 *3 TB)▪ 20 PB Storage▪ HA Name Node Set Up▪ 9k Map Slots + 5.5k Reduce Slots▪ Co-located to run HBase for offline processing

▪ HBase 0.94.15▪ 5 Node ZooKeeper quorum▪ Monitoring with OpenTSDB▪ Dual Master Setup

Behavioral Targeting @ Rocket Fuel

bmw.com 11:23

Cars 11:23

pizzahut.com 11:26

Food 11:26

honda.com 11:27

Cars 11:2730 minutes

honda.com

11:27 Recent 6 hours: 5

Between 6 and 12 hours: 3

Between 12 hours and …

Food 11:26 Recent 6 hours: 2

Between 6 and 12 hours: 7

Between 12 hours and …

Read events of last N days

Recency

Frequency

Others..Behavioral Targeting Profile

11:23 11:26 11:27

HBase Data Model

11:23ABCD06EFG

2014060416:site:bmw.com 2014060416:category:food

row_key: user_id

Single Column Family “u”

Column Qualifier:<date><hour>:<type>:<value>

Cell Value: [Protobuf]Most recent timestamp, Event details relative to timestamp

Event details relative to 11:23 Event details relative to 11:26

• Efficient look up for a given user

• Access range of events by event date, hour and type

Key Challenges

User Profile Freshness Scaling Issues Pipeline Failures

User Profile Freshness

▪ Strict latency requirements▪ Recent activity much better predictor

Solutions - ▪ Staggered Pipelines▪ Real Time Behavioral Targeting

Staggered Pipelines

Extract Score Filter Upload

Extract Score Filter UploadSource Data

Extract Score Filter Upload

Real Time Behavioral Targeting

Batched Profile

Blackbird – HBase instance tuned for 2ms latencies

Refreshed every N hours

Real Time Behavioral Targeting

Offline BT Pipeline

BT Profile

Ad Servers Merge Profiles

Blackbird

Online Profile

Record events for users in real time

Request

Response

Batched Updates vs. Real Time Updates

Event Granularity Aggregated over several hours/days

Raw recorded events appended for recent

N hours

Processing Load Requires minimal CPU processing

Needs aggregation on-the-fly

Disk FootprintCompact

representation captures several days

Strict limits to ensure read times are

acceptable

Coverage All interactions Only interactions at a data center

▪ Real Time Profile updated in milliseconds

▪ Batched Profile refreshed every N hours

Batched Profile Real Time Profile

Scaling Issues

▪ 3X growth in events processed/year▪ First Party Data▪ App Interactions▪ Geo-location Data▪ …

▪ Case Studies▪ HBase Region Hot-spotting▪ Network Bandwidth Troubles

HBase Region Hot Spotting

HBase Region

HBase Region Hot-spotting

High Write Load

HBase Region

Region Split (painful!)

Some users more active than othersNo control on user id’s generated

Still problematic

Non-uniform

distribution!

▪ Uneven write-load distribution▪ Non-Uniform Row Key Distribution

▪ Salt row key’s to ensure uniform distribution▪ Fixed length hashed prefix▪

Murmur hash based prefix

Original User ID

▪ Uniform pre-splits

▪ Don’t stop at salting▪ Map input splits configured for region boundaries

Region 1\x03\x85\x1E\xB8ZZZZZZ

Region 2\x07\x5C\xF5\xC2928ZZ

Region m\xFF\xAE\x14\xE1Z28ZZ

12345571234568123457912345831234594

..ZZAHT654ZZZGT934ZZZZNGA2ZZZZKLO1

Key Partitioner

‘k’ splits ‘m’ regions‘m’ splits

\x01\x85\x1E\xB811ZKL1\x01\x86\x1E\xB8129542

..\x03\x85\x1E\xB8ZZZKL1

\x05\x35\x9E\x18087KL1\x06\x86\x1E\xB8AHV24

..\x07\x5C\xF5\xC16534Z

\xEB\x27\x92\x1508RKL1\xFE\x86\x1E\xB8AHV24

..\xFF\xAE\x14\x126534Z

HBase Key Partitioner

▪ As many splits as regions to maximize parallelism

▪ Key Partitioner (MR) – ▪ Reads region boundaries of HBase table▪ Salts and sorts row key accordingly▪ Multiple Output Format to optimize reduce phase▪ Each generated split file corresponds to a single region

▪ Drastically reduces read latencies

Network Bandwidth Troubles

Data Center Expansion

Network Bandwidth Constraints

▪ Consistently overshot bandwidth limit during uploads▪ All sorts of delays (Redis, MySQL, Blackbird…)▪ Bidding hampered

Solutions

▪ Intelligent storage – protobufs everywhere

▪ Throttle writes

▪ Geo-splitting

Geo Splitting

Geo-splitting

▪ Tag user’s location history & predict future data center visits

▪ ⨍(dc, geo_history, bt_profile)

▪ A separate workflow periodically generates geo-split rules:▪ Clusters users & analyzes migration patterns▪ Ensures maximal look-up coverage of profiles▪ Minimizes total number of profiles stored

▪ Ensures efficient use of resources, with minimal impact on perf

Geo-splitting

Label Data

Train Model

Back Test

Calibrate

TrainingEvents

Pixel Stream Ad Logs

BT Features (HBase)

Feature Generation

Score Profiles

Profile Generation

Scoring

Ad Serving Data Centers Model

Cluster Users

Analyze Patterns

Generate Rules

Geo-split

Quick Recovery From Failures

▪ Break pipeline into short payloads▪ Fail fast, recover fast!▪ Actionable alerts, cut down noise

Quick Recovery From Failures

▪ Materialize data as frequently as possible▪ Cross system fault tolerance▪ Idempotency

▪ Backfill at EOD to plug holes if needed

Shout-outs!

We Are Hiring!

Questions ?

Thank You!

Sivasankaran Chandrasekarchandra@rocketfuel.com

Savin Goyalsavin@rocketfuel.com

We are hiring! (as always)

http://rocketfuel.com/careers

savin@rocketfuel.comchandra@rocketfuel.com

How did you know this Ad will be relevant for me?!

Technology

How to win with relevant digital ad?

Macquarie University ResearchOnline · As the Dettol ad ends, Jacqui inquires: “Did you see that? Ha ha! Did you see that ad, Mum?” She attempts to insert herself into the chat

Collaboration Reimagined - Cisco · Connecn gPeople$ Connecting people in more relevant, valuable ways People Occasional Messaging Scheduled Meetings Ad Hoc Meetings Furious Chat

gov.uk · Web viewThe nutritional intake of the players did meet the recommended nutritional status and food habits of players. Relevant Relevant 3 Spain N/A Jago, Baranowski, Yoo,

239 OPTICAL PROPERTIES COMPRESSIBLE INHOMOOENEOUS … · ad-ntis 239 optical properties of compressible inhomooeneous swear 1 layvers relevant to high powler laserscu) washington

AD-A251 475 - Defense Technical Information Center · AD-A251 475 HOW RELEVANT WAS ... IN WINNING THE COLD WAR? O John Lewis Gaddi~ _____ ... came to an end, your grandchildren are

The Value Of Southern California Radio To The Auto Industry Automotive... · were relevant to me. “ Among: Listened to the ad Q20. Thinking about the ad you've just heard for (dealer),

PowerPoint Presentationtsrmatters.com/wp-content/uploads/2014/06/Case-Study_University-of... · Ad Type Impressions Clicks CTR ... relevant but I received quick efficient replies!

Rethink ADRethink AD A Paradigm Shift for Identity Management. Free IT from the Constraints of Active Directory. Free IT from the Constraints of AD 2 How did Active Directory establish

Headline™ DID Headline™ DID Sign Sign 제안서20%C1%A6%BE%C8%BC%AD.pdf · Headline™ DID Headline™ DID Sign Sign 제안서 No.1 Total AD Platform CompanyHeadline™.. 헤드라인블루㈜의등록상표입니다

Lernmobilität von Einzelpersonen: KA1 KA102 - … · If relevant for your project, did you use Erasmus+ online platforms (e.g. EPALE, School Education Gateway, eTwinning) for the

CERTIFICATE - Rompetrolnewweb.rompetrol.com/cms/rominserv_valves_iaifo/AD2000.pdf · AD 2000-Merkblatt WO I AD 2000-Merkblatt A4 The scope of the audit and other relevant data are

1.What do you think this ad for butter is trying to communicate to consumers? 2.Did the advertiser explain all of this to you, or did you have to infer

6 Ways to Become Relevant Beyond An Ad

Gift Wrapping Holiday Shoppers with Relevant Digital Ad Solutions

What Did the Greeks Invent and is It Relevant To

Psychiatrische Störungen auf der Intensivstation · A regression analysis did not furnish statistically relevant predictors and/or factors of risk for developing a psy- chological

Increasing Podcast Downloads - Leverage Site/Case Studies...Step 1: Create a Facebook Ad strategy. Step 2: Create new, relevant, conversion-optimized ad copy and ad creative. Step

Monetizing Digital Content 3. What’s Next › wp-content › uploads › ...Mobile ad recall 0% 10% 20% 30% 40% 50% 60% 70% Test (viewed ad) Control (did not view ad) Native Mobile

Early European Exploration. What are the Americas?? AD 500-AD 1500 Mysterious stories told of an unknown land Vikings did visit it, though! –People of