AGILE MEAGILE ME

1

Science is Agile by Design

There is a thing called Gravity

Gravity warps space and time

Gravitational Waves are Real

Refinements of understanding takes place in

increments

AGILE MEAGILE ME

2

platinum sponsor

gold sponsor

silver sponsors

bronze sponsors

AGILE MEAGILE ME

ATIF ABDUL RAHMAN

Agile Analytics

www.About.Me/AtifAbdulRahman

I was like her according to Pearson-R;

We were both outliers

19th March, 2016, Dubai, UAE

AGILE MEAGILE ME

• Line 1

• Line 2

Title 1

4

Let’s address the elephant in the room

AGILE MEAGILE ME

• Line 1

• Line 2

Title 1

5

BI is Bureaucratic

Let’s address the elephant in the room

Data Warehouse Architectures are Fragmented by

Design

Vendor & Tools Lock-In created

artificial constraints

Manpower Outsourcing

Industry boomed and thrived upon this bureaucracy

AGILE MEAGILE ME

• Line 1

• Line 2

Title 1

6

BI is Bureaucratic

Let’s address the elephant in the room

Data Warehouse Architectures are Fragmented by

Design

Vendor & Tools Lock-In created

artificial constraints

Manpower Outsourcing

Industry boomed and thrived upon this bureaucracy

Relay Race: Everybody is Waiting

AGILE MEAGILE ME

7

Our ability to process data was always a step behind our capability to generate data, essentially data was always big. However, our technologies had eventually reached their shelf life of increments..

AGILE MEAGILE ME

8

Rise of Hadoop: Big Data Floodgates

AGILE MEAGILE ME

9

The Big in Data is not for the data being big, it’s the big disruption

AGILE MEAGILE ME

10

The Big in Data is not for the data being big, it’s the big disruption

Our ability to process data was always a step behind our capability to generate data, essentially data was always big. However, our technologies had eventually reached their shelf life of increments..

AGILE MEAGILE ME

11

Utility Hardware

can do more now

Open Source is

leading the technology

stack

Significant reduction in dependency

with IT

Democratization of Data Infrastructure

Big Data Technologies are removing barriers and constraints, its an enabler rather than a disruption in itself.

Arguably started by Hadoop but not the only player

Resources freed up for

Data Governance

AGILE MEAGILE ME

12

Data Doesn’t reveal its secrets very easily!

AGILE MEAGILE ME

13

Data Doesn’t reveal its secrets very easily!

AGILE MEAGILE ME

14

Data Doesn’t reveal its secrets very easily!

AGILE MEAGILE ME

15

Data Doesn’t reveal its secrets very easily!

Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it...

AGILE ME

Obs

Transactional

Declarative

(biggest in size

Difficult and misleading)

*In very specific environments, more data with simpler algorithms work better: Basic premise of VC Theorem: Machine Learning

“What information consumes is ratherobvious: it consumes the attention of itsrecipients. Hence, a wealth of informationcreates a poverty of attention.” Herbert Simon

Signal to Noise Ratio decreases with more data making it harder to find true signals in general

AGILE ME

Why is it taking it so long to predict the future?

17

Common complaints heard by data scientists?

AGILE ME

2014 2015

Gartner Hype Cycle

From Big Data To Analytics

Good News: Big Data hype has already peaked, now everyone wants value from it (Analytics).

AGILE MEAGILE ME

• CRISP-DM

• Analytics is inherently Agile

Learning & Empirical Process Control

19

This is the most adopted knowledge discovery approach, pretty much incremental in nature and focuses on feedback, improvements and learning empirically. This makes it well aligned with the Agile Manifesto.

Insights are Discovered, not

Designed

Russell Jurney

AGILE ME

THE NON-DATA DRIVEN PATTERN OF DATA DRIVEN INITIATIVES

20

With little investment, a lot of value can be gained (similar to an MVP)

80/20

Rule

AGILE MEAGILE ME

21

• Individuals & Interactions: Analytics is a Team Sport• Working Models: Models are Refined instead of Designed• Customer Collaboration: User Stories Emerge • Responding to Change: Models always have Expiry Dates

TDWI Survey 2013:

80% of practitioners reported improved success rates using Agile.

Agile Manifesto for Analytics

Andy Palmer CEO, Tamr

AGILE MEAGILE ME

22

Rise of the Data Scientist: An Agile Creature

These unicorns are rare, but teams of data scientists are common

AGILE MEAGILE ME

Agile Apps vs Agile Analytics

23

Features UXUser

StoriesValue

Differences between App vs Analytics User Stories

Applications: (Features Mostly) Analytics (Insights Mostly)

We need the top N recommendations with their ratings

We need to find similar books?We need to find books that the reader might purchase?

Differences must be addressed:

AGILE MEAGILE ME

24

Clear Not Clear

Ava

ilab

leN

ot

Ava

ilab

le

Rig

ht

Dat

aRequirements

Refinement

Dat

a En

rich

men

t

•Have as narrow a scope as possible;•Contain explicitly quantitative clauses;•Are ranked by relative value; and•Are potentially answerable given the available data.

Adapting for Analytics

User Stories emerge after

the fact

Data usefulness is discovered

after the fact

AGILE MEAGILE ME

25

Getting back to Science:

Most analytics problems are not linear problems like those in most application development. Analytics demand Agility on Steroids!

AGILE MEAGILE ME

26

Getting back to Science:

Most analytics problems are not linear problems like those in most application development. Analytics demand Agility on Steroids!

Remember Galileo’s Sad Story?

AGILE MEAGILE ME

Data Virtualization

27*TDWI

An Enabler to put Agile on Steroids and delivery awesome Analytics Projects

AGILE MEAGILE ME

28

Data Lakes

An Agile Data Architecture

*EMC

AGILE MEAGILE ME

29

Data Scientists are better at Statistics than most

Programmers and are better at Programming than most

Statisticians.

Choose your (Agile) Approach

Provision The Agile Data Architecture

Party Hard

Dear Agile Practitioners, Always Remember:

AGILE MEAGILE ME

Thank You

30