Using RUM to deliver optimalPerformance in the Fast New MSN

Paul Roy, Microsoft

Buddy Brewer, SOASTA@bbrewer

New MSN•New UI design

• Entire stack on Azure

•New geo-distributed Content Store

• Responsive design

• Single web codebase for all verticals

Launched Fall 2014 in 55 countries

RUM Adoption

• Complement to synthetic measurements

•Adoption phases• Evaluation

• Pre-Launch

• Launch

• After Launch

• Futures

Played a key role in delivering optimal performance

Evaluation• Evaluated internal & external solutions

•MSN contacted SOASTA

• Proof of Concept working within 1 week

• Loved the dashboards & real-time

• Cost effective

• Secured buy-off from senior mgmt

Pre-Launch• Completed the integration

• Built custom RUM dashboards in preparation

for launch

Example

Launch •War Room – all hands on deck throughout the

night

• Perf guys were there to watch traffic ramp-up

and verify performance

After Launch

•Using data on ongoing basis to watch trends

and generate regular scorecards against

targets

• Spot regressions

•Verify improvements

• Reporting to executives

Steady state

Types of Problems

• Code regressions not detected by controlled

synthetic environment with limited data points

• Country-specific problems

• Browser-specific problems

• Device-specific problems

• Problems at high percentiles

• Problems at low bandwidths

• RUM waterfalls exposed anomalous script

behavior in the wild

Detected by RUM but not synthetic

Futures• Reduce time to detect regressions (alerting

and more frequent tracking)

•Mining waterfall and resource-level data to

detect anomalous patterns

• Further study of correlation between speed

and user engagement

Where do we go next

Correlation with Session Length

Correlation with Session Length

Visits: 100MCurrent pages/visit: 1.5Median load time: 2 secPages/visit @ 1.5 sec: 2

(2 – 1.5) * 100M = 50M PV

Operationalizing business metrics

See SOASTA at booth #801