FUTURESTACK13: Software analytics with Project Rubicon from Alex Kroman Engineering Manager at New...

Software Analytics with Project Rubicon

ALEX KROMANOCTOBER 25TH, 2013

Wednesday, November 6, 13

1. Learn how to write NRQL queries

2. Aggregating billions of events

3. Building features on top of Rubicon

Today’s agenda

Wednesday, November 6, 13

New Relic captures performance data from a bunch of places

Wednesday, November 6, 13

But performance data is a small part of application data

Clicks Log messages API callsTransactions

Wednesday, November 6, 13

Rubicon captures the raw events from your applications

Python AppiOS appRuby

App

Rubicon

Wednesday, November 6, 13

How many customers logged in today?

What is my average revenue per sale?

What is my most used feature?

How many people are live on the app now?

By capturing raw events you can ask your application complex questions

Wednesday, November 6, 13

An introduction to NRQL

Wednesday, November 6, 13

1. How many people used the site today?

2. How many people are using a new feature?

3. How many people are using a new feature over time?

4. What kind of customers are using this feature?

Four questions you can ask Rubicon

Wednesday, November 6, 13

1. How many people used the site today?

SELECT uniqueCount(user) FROM PageView SINCE 1 day ago

Wednesday, November 6, 13

Time windows in NRQL

SINCE 8 hours ago

SINCE 2 months ago

SINCE 1 week ago UNTIL 2 hours ago

Wednesday, November 6, 13

2. How many people are using a new feature?

SELECT uniqueCount(user) FROM PageView WHERE name LIKE '%x_rays%' SINCE 1 month ago

Wednesday, November 6, 13

3. How many people are using the feature over time?SELECT uniqueCount(user) FROM PageView WHERE name LIKE ‘%x_rays%’ SINCE 7 days ago TIMESERIES day

Wednesday, November 6, 13

4. What kind of customers are using this feature?

SELECT uniqueCount(user) FROM PageView WHERE name LIKE '%x_rays%' SINCE 7 days ago FACET apm_product

Wednesday, November 6, 13

•Plenty of aggregation functions

•Time windows

•Time series

•Facets

NRQL is designed to make it easy to aggregate data over time

Wednesday, November 6, 13

How do we aggregate a billion events?

Wednesday, November 6, 13

Build a wide cluster

Memory is cheap so use it

Build for failure

Send the cluster real traffic

Four things we’ve learned handling large amounts of data

Wednesday, November 6, 13

A brief tour of the architecture...

Router

Queries

Load balancer

RouterRouter

Worker A Worker B Worker C

Metadata

Backup Storage

Wednesday, November 6, 13

Building features on top of Rubicon

Wednesday, November 6, 13

Wednesday, November 6, 13

Wednesday, November 6, 13

Wednesday, November 6, 13

Making a histogram chart

Rubicon.nrql(“SELECT histogram(duration) FROM PageView SINCE 7 days ago”)

Wednesday, November 6, 13

JSON response{ "results": [ { "histogram": [ 1618, 4748, 3736, 4526,etc...

], "bucketSize": 0.25 }

Wednesday, November 6, 13

Making a percentile chart

Rubicon.nrql(“SELECT percentile(duration, 50, 95, 99) FROM PageView SINCE 7 days ago TIMESERIES hour”)

Wednesday, November 6, 13

JSON response{ "timeSeries": [ { "results": [ { "percentiles": { "50": 1.66, "95": 5.445, "99": 11.942 } } ], "beginTimeSeconds": 1382050194, "endTimeSeconds": 1382053794 },

Wednesday, November 6, 13

Adding a custom event

data = {‘salesTeam’: ‘Enterprise’, ‘amount’: 100000}

Rubicon.insert(‘RevenueChange’, data)

Wednesday, November 6, 13

Querying a custom event

SELECT count(*) FROM RevenueChange FACET salesTeam SINCE 7 days ago

Wednesday, November 6, 13

Learned a bit of NRQL

Overview of the architecture

Explored the Rubicon API’s

Recap

Wednesday, November 6, 13

Come visit us in the hacker lounge!

Wednesday, November 6, 13

Find out more!

newrelic.com/analytics

Wednesday, November 6, 13