Bimodal IT and EDW Modernization

BIMODAL IT AND THE JOURNEY TO DATA WAREHOUSE MODERNIZATION

BY

ROB GLEAVE

VP ARCHITECTURE AND STRATEGY

SPORTS AUTHORITY

THE SHIFTING LANDSCAPE OF BUSINESS ANALYTICS

We are all facing this….

THE SHIFTING LANDSCAPE OF BUSINESS ANALYTICS Challenges on all fronts… •  "Traditional" IT-centric business intelligence strategies are no longer

sufficient to drive the use of analytics within an organization.

•  IT teams are struggling to cope with higher volume and diversity of demand.

•  Business is increasingly taking over control, ownership and responsibility for analytics — often underestimating the associated complexity and risks.

•  The almost infinite amount of data available offers great opportunity for new insights, but also makes it increasingly difficult for organizations to manage, secure and accurately interpret that data.


Without a roadmap to modernization, you will surely fail to keep

up with the demand for data…

What is your plan?

How do you start?


First and foremost… Don’t abandon what you have been doing… That is NOT necessary.

JUST

ENTER THE BIMODAL HIGHWAY You can take an alternate route…

BIMODAL IT… WHAT IS IT?

Source: Gartner (April 2015)

!

What is Bimodal IT?


Mode 1 Mode 2 Reliability, Incremental

Growth Goal Agility, Innovation

Price for Performance Value Revenue, Brand,

Customer Experience Waterfall,

High Ceremony Approach Agile, Low Ceremony

Plan Driven, Approval Based Governance Empirical, Adaptive

Enterprise Suppliers, Long-Term Deals Sourcing Small, New Vendors,

Short-Term Deals Good at Conventional

Process, Projects Talent Good at New and Uncertain Projects

Take the Order, Delight "Customers" Culture Innovate With

"Partners" Long (Months, Years) Cycle Times Short (Days, Weeks)

Think Ninja

Think Samurai

Gartner Presentation, Getting Real About Bimodal, Dave Aron, October 2015

Two Different mindsets and approaches… both essential.


! Source: Gartner (April 2015)

In a perfect world… we want a balanced mix of both capabilities.

BIMODAL IT… THE CHALLENGE

Most organizations are already executing Mode 1 well, delivering:

Reliability

Efficiency

Safety

Accuracy

The challenge is how to build a Mode 2 capability to deliver

SPEED & AGILITY

BIMODAL IT… THE CHALLENGE

Detractors sometimes discount the idea of Bimodal IT, saying…

“Why not just make every element of IT more agile?”

But that is not realistic…

Bimodal is about accepting deliberate trade-offs

ROADMAP TO MODERNIZATION

The road to modernization will be paved by BIMODAL thinkers

You need a viable plan…

Step #1: Embrace Self Service BI


“The users have won…”

-  CTO of IBM’s Lean Analytics Division Inventor of IBM Big Insights

Often, IT will resist self service….. WHY? •  Business users want it.. •  It is often more agile.. •  The work of BI is distributed across many hands, who are

all expert in the meaning of the data.. •  Self service sandboxes provide real value - they often

become areas of true innovation.. •  It helps the ‘underserved’. Most enterprises cannot fund

enough BI to feed the masses.

ROADMAP ROADMAP TO MODERNIZATION


Historically with self service, that fear is well founded.. When Sports Authority started the journey to data warehouse modernization, we had:

•  “Shadow IT” sandboxes everywhere •  Over 1900 MS Access databases on user desktops –

16,800 mdb files (purpose of most were unknown) •  One MS Access database supporting 130 users and

containing 50 different tables (many at max size). •  Most users queried transactional systems directly to

populate their data marts, impacting performance of those transaction systems dramatically.



HOST SYSTEM .

ApplicationsApplication

Application

Application

Data Source

Ad-Hoc Report Consumers

Data Sources

Data Source

Application

POS

eCommerce

EnterpriseData Warehouse

Corporate Report Consumers

Scales

Transactional Databases

Scales

Sports Authority - SA Data Landscape

11/1/2015 Page 1

A picture of our world….

Change is difficult, however, IT teams must resist the temptation to “own” everything. The truth is…

•  Overly centralized BI teams can't deliver the domain expertise, responsiveness and speed most organizations require.

•  While a centralized team does a good job in creating consistency and governance across certain key subject areas, it creates a bottleneck, causing most users to wait too long to get their requirements met.

•  The future of BI and analytics is about enabling both a centralized BI function as well as the decentralized analysis occurring within the company.

ROADMAP ROADMAP TO MODERNIZATION

It just takes the right technology…

ROADMAP

Plus, we all know that self-service works…



Stage 1

What does your technology look like today? Is this your data architecture?

Step #2: Pick your future-state data platform


What are kind of environment are we looking for? •  Secure •  Universally available •  Promotes data sharing •  Scalable •  Reliable •  Manageable •  Easy to use •  Cost-effective


Where can you find such an environment? Look to a cloud platform, like Google, Microsoft or Amazon…


What is BigQuery? •  A fully managed (Saas) data analytics service •  ‘Pay for what you use’ model -- very low cost •  Familiar SQL Interface •  Super-fast! Query against terabytes of data in seconds •  Elastic, auto-scaling, up to petabyte-scale databases •  Truly ‘Big Data’ – comparable to Hadoop/SQL or Spark •  Programmable - APIs, APIs, APIs….


How does BigQuery work? •  The largest columnar database on earth •  Optimized for selection, aggregation •  Provides superior data compression •  Stores data differently than a traditional RDBMS


Step #3: Offer Personal Sandboxes



Stage 2

Google BigQuery

Google Cloud Storage

Allow users to build personal sandboxes in the cloud environment…


Stage 2

At this point, the new sandboxes (Mode 2) sit beside the old EDW (Mode 1)…

Step #4: Experiment with “Citizen” Tools


We live in the era of the ‘Citizen’ knowledge worker…


Easy-to-use tools are easing business users into functions which have traditionally been handled by IT

•  Data blending (e.g. Alteryx, )

•  Data quality & MDM (e.g. Alteryx, Reltio, Dell Boomi )

•  Data modeling and virtualization (e.g. Looker, Denodo)

•  Data visualization (e.g. Tableau, Clikview, PowerBI, Microstrategy 10)

•  Lightweight integration and iPaas (e.g. Dell Boomi, Snaplogic)

•  Even…. Data Science! (e.g. R, Python Pandas)


Generally, these tools are only appropriate for SMEs or people who are currently your ‘super users’.

But, they are extremely powerful and give these business data analysts great freedom to explore, innovate and share valuable data assets.

These tools, extend the reach of IT and actually help eliminate some age-old problems, for example….



Stage 3

Where do these tools fit into the new data architecture?

ROADMAP TO MODERNIZATION Stage 3

Step #5: Build Core Data Sets in the Cloud



Google BigQuery


Core Datasets


Step #6: Migrate to the Data Warehouse of the Future


ROADMAP TO MODERNIZATION Stage 5 begins with a full Data Lake, because… •  The lake supports direct (Mode 2) discovery against new data sets. •  It provides a platform for Big Data – both structured and unstructured. •  Its scale-out architecture allows more data to be collected and retained. •  It can lower upfront costs, by delaying transformation and modeling until

needed.

Caveats: must define platform, security, data management, etc.

ROADMAP TO MODERNIZATION Notice the full Data Lake and cloud-based Hadoop for Big Data….

Stage 5


Google BigQuery

Google Cloud DataProc



Join SQL – over user data

set + core data set

BigQuery Sandbox Project

User Sandbox

User Sandbox

Core Subject A

rea

User Sandbox

User Sandbox

Data set secured to an individual user

Core Subject A

rea

User Sandbox

User Sandbox

User Sandbox

User Sandbox

Core Subject A

rea

Core Subject A

rea

Core Subject A

rea

Data set secured to an individual user

Core warehouse data set – read only

Plain SQL – only against user sandbox data set

Sports Authority - SA Data Landscape

11/2/2015 Page 1

Users enjoy a powerful data world consisting of personal sandboxes and core data sets in a new elastic data warehouse.

ROADMAP

In Review, the Steps to Data Warehouse Modernization

1.  Adopt a Mode 2 mindset - with a focus on Self Service. 2.  Pick a new platform for the future-state data world 3.  Offer personal sandboxes for exploration/discovery 4.  Introduce and promote “Citizen” data tools 5.  Build core data sets in the cloud 6.  Migrate completely to the Data Warehouse of the Future


QUESTIONS?

Rob Gleave

[email protected]

[email protected]

Documents

Bimodal IT and EDW Modernization