Practical BI

Tom Spetnagel

Practical BIStrategies that IT can use to

maximize its productivity and value to the business

Tom SpetnagelDirector of Business Intelligence, Valpak

Tom Spetnagel

2

Tom Spetnagel

Summary of Strategies

• Use agile practices for BI development

• Determine actual requirements and design to them

• Utilize appropriate requirements-gathering techniques

• Implement and wield a BI charter

My Background:

Tom SpetnagelTom Spetnagel

Good News for BI:It’s Taking Off!

Technology

Mobile

Social

Cloud

Data Access

Web Analytics

Semantic Search

Sentiment Monitoring

Data Mining

Unstructured Data

Mobile BI

BI Applications

Big DataReal-Time BI

Social Analytics

Self-Service

Inline BI

In-Database Analytics

Data Replication

In-Memory BI

Master-Data-Management

Web Services

Enterprise Search

Web Search

CPU Optimization

Public Data

Tom Spetnagel

5

Tom Spetnagel

The BI Explosion Is Scary, Too

• BI is getting bigger and more complex, but BI budgets aren’t keeping pace!

• Access to information is the root of recent evolution: Google, Facebook, mobile

• Self-Service BI is continuously on-the-rise

• IT is therefore becoming less central in BI!

Tom Spetnagel

6

Users Won’t Wait on IThttp://xkcd.com/

BI users can’t wait on IT;they create their own solutions, and they aren’t always good ones!

Tom Spetnagel

7

Tom Spetnagel

BI’s Biggest Challenge: Prioritization!

• Since BI supply can’t keep up with demand, continuously producing ‘something good at the right time’ is critical

Tom Spetnagel

8

Tom Spetnagel

BI Stakeholders:

-They’re not aligned-They want it all-They want it now-They always want something different

Managing BI stakeholders is a lot like trying to keep chickens under a blanket!

Tom Spetnagel

9

Tom Spetnagel

What’s A Solution?

Agile BI!(Practical BI, Strategy #1)

Tom Spetnagel

10

Tom Spetnagel

What is ‘Agile’?

• The application of common sense to software development*

• A set of concepts developed by people frustrated with the application of ‘traditional’ project management to software

* I wish I could trademark this!

Tom Spetnagel

11

Tom Spetnagel

Agile Evolution• Agile Manifesto conceived at an

informal drink-and-ski weekend in 2001

• Reaction to fundamental differences of building software and building physical items (like aircraft carriers)

X

Tom Spetnagel

12

Unlike with physical construction, since it’s only ‘zeros-and-ones’, software can be changed quickly!

Tom Spetnagel

13

Tom Spetnagel

4 Main Agile Principles

More Important• Individuals and

interactions• Working software• Customer

collaboration• Responding to change

Less Important• Processes and tools• Comprehensive

documentation• Contract negotiation• Following a plan

http://agilemanifesto.org/

Tom Spetnagel

14

Tom Spetnagel

Primary Intentions of Agile

• Deliver the most valuable thing at the right time

• Deliver working software quickly!• Embrace but manage change• Establish short-term predictability• Eliminate surprises from both the

IT and business sides

Tom Spetnagel

15

Tom Spetnagel

Some Agile ‘Methodologies’

Scrum

Unified Process (UP)

Crystal Clear

Feature Driven Development (FDD)

Extreme Programming (XP)

Lean Software Development

Tom Spetnagel

16

Tom Spetnagel

Agile at Valpak: “Scrum”

• Multiple scrum teams, each team having:– 1 Scrum Master, 1 Product Owner, 5 to 7 Team

Members• 2 week iterations, executing several

‘stories’ per team, bounded by:– Sprint planning (1st Monday)– Sprint demo and review (2nd Friday)

• Daily stand-up status meetings

Tom Spetnagel

17

Types of BI ‘Stories’ Include:

• ETLs• Metadata Mapping• Formal Reports /

Dashboards• Alerts• Automated Report

Distribution

• Performance• Data Quality• Security• Upgrade/patch

Tom Spetnagel

18

Tom Spetnagel

Why Is Agile Great for BI?

• Creates a practical method for handling crucial BI challenges which drive scope and affect success

• Gives ownership and flexibility to the business, not IT


Crucial Scope-Drivers in BI (1)

• “Data Quality”– A catch-all term for numerous different

problems:• Unclear definitions• Missing data / duplicated data• Unexpected data• Unreconciling data

• Performance/Speed– People expect reports to run as fast as business

‘transactions’ (create 1 order, save 1 order, etc.)• And it’s even worse with mobile devices!

Tom Spetnagel

20

Tom Spetnagel

Data Quality vs. Effort

Data quality is a function of effort; increasing effort has diminishing returns and it is never possible to reach 100% data quality


Crucial Scope-Drivers in BI (2)

• “Terminology”/Definitions

• Historical Data

The cultural hurdles that come with defining or redefining terms for BI take much time to overcome!

Stakeholders often want ‘history’, not just information from this point forward!

Tom Spetnagel

22

Tom Spetnagel

Agile Handling of BI Scope-Drivers

• Data Quality– Iterate to provide additional quality checks where

desired

• Performance– Iterate to achieve better performance where desired

• Terminology – Iterate to update definitions where needed; within an

iteration, make a decision and go!

• History– Load the history in a separate iteration after new

data collection has been activated

Tom Spetnagel

23

Tom Spetnagel

Business Accountability

• Let the business decide what they want most in the next iteration (based on what IT tells them it can get done in that timeframe)

Tom Spetnagel

24

Tom Spetnagel

1 More Reason Agile Is Great for BI

• It’s tough for BI stakeholders to know what something is worth!

• Example: What is it worth to you to have a timely, accurate bank balance?

Tom Spetnagel

25

Tom Spetnagel

1 Last Reason Agile Is Great for BI

“Walking on water and developing software from a specification are easy if both are frozen”

- Edward V. Berard, "Life-Cycle Approaches"

BI Stakeholders can rarely know what they really need (or need next) until they’re using it!

Tom Spetnagel

26

Tom Spetnagel

Challenges of Agile

• High ratio of planning & communicating time to coding time

• High amount of time discussing & refining the agile process; some danger of over-analysis

• High % of time collaborating; IT folks need to be good communicators

Tom Spetnagel

27

Tom Spetnagel

Agile ‘In Their Own Words’:

http://www.youtube.com/watch?v=A0As88akpXs



Tom Spetnagel

28

Tom Spetnagel

Practical BI Strategy #2:

Determine, and Deliver to,the Actual BI Requirements

• Don’t deliver just what is (initially) requested; scientifically deconstruct it into what is actually needed

• ‘Requirements’ and ‘design’ are different in BI, just as in application development

Tom Spetnagel

29

Tom Spetnagel

BI Requirements vs. Design:Example #1

Analyst Questions

1. Do users expect the new data to reconcile with anything existing?

2. How many people will need access to the same info at the same time? How often?

Designer Questions

1. What should the data sources be? Should the output have any built-in validations, reconciliations, or subtotals?

2. What mechanism is best for providing shared data (web page? email or text alert? printed poster?)?

Tom Spetnagel

30

Tom Spetnagel

BI Requirements vs. Design:Example #2

Analyst Questions

1. How recent/up-to-date does information need to be?

2. What is the acceptable timeframe for accessing information? What are the response-time requirements?

Designer Questions

1. Does the solution require access to real-time transaction data? Or can it be data warehouse data, updated/frozen on a schedule?

2. Should data be stored in-database or in-memory? What summarization or indexing is needed?

Tom Spetnagel

31

Tom Spetnagel

Mockups

• A report or dashboard mockup is nice but does not constitute either comprehensive requirements or design

• Mockups are a great starting point for a requirements conversation, though!

Tom Spetnagel

32

Tom Spetnagel


Use the Best Requirements-Gathering Technique for

the BI Assignment• There are a number of

different and effective ways to gather requirements

• Review, implement, and combine these however necessary

Tom Spetnagel

33

Requirements-Gathering Techniques

1. Interview2. Survey3. Focus Group4. Requirements

Workshop5. Observations

6. Reverse-Engineering

7. Document Analysis8. Prototyping9. Brainstorming10.Interface Analysis

Tom Spetnagel

34

Tom Spetnagel


Implement and Wield a BI Charter

• Gather a set of goals, principles, and strategies that IT and the business can agree on

• Use this to focus discussions and overcome objections to IT proposals and decisions

Tom Spetnagel

35

Tom Spetnagel

BI Charter: Example #1

• Goals– Limit confusion around ‘what numbers are right/best’

• Principles– Data in the data warehouse are the ‘official’ figures

unless specifically documented otherwise• Strategies

– Get ‘official’ figures into the data warehouse– Avoid storing both official and unofficial figures for

the same metric in the data warehouse– Restrict access to unofficial data in 3rd party tools

Tom Spetnagel

36

Tom Spetnagel

BI Charter: Example #2

• Goals– Minimize the amount of BI tool-training required

• Principles– IT will not support unofficial tools which users have self-

provided– Access to 3rd-party BI platforms will be supported on an

exception basis when unique value is provided• Strategies

– Minimize the number of tools users must know how to use

– Use a BI platform which scores highly on ease-of-use and which has multi-purpose tools

Tom Spetnagel

37

Tom Spetnagel

The End – Of The Beginning