36
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead Regression Testing OBIEE Without the Fuss Robin Moffatt Principal Consultant, Rittman Mead

Regression Testing OBIEE without the fuss

Embed Size (px)

DESCRIPTION

Just because OBIEE is presented to users through a web browser that doesn't mean the regression testing has to take place there too. This presentation will demonstrate how you can efficiently and effectively test the bulk of changes made to your system without going anywhere near the front end. Where front end testing is needing, it demonstrates the concept of visual regression testing using the latest tools. With hands-on examples of the tools involved this will be a practical presentation giving anyone interested in the topic a headstart in practical implementation. Video available here: http://ritt.md/regression-testing-video

Citation preview

Page 1: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Regression Testing OBIEE Without the Fuss Robin Moffatt Principal Consultant, Rittman Mead

Page 2: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Smart Regression Testing

Do better OBIEE development and deliver it more rapidly

Page 3: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Why Regression Test?

Confidence

Page 4: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Why?

Confidence

Agility in development

Happier users

Page 5: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

What?

•Regression Testing is not just a prolonged exercise on a Gantt chart, just before go live

•Smart Regression Testing can be run on demand, targeting the change made and assuring quality during development

Page 6: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Detect Problems Early

‣Fresh in the developer’s mind‣Easier to fix !

‣Fewer surprises during big test cycles‣More honest & realistic project status reports !

‣Better overall QA for the system‣Happier users

Page 7: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Save Time

•Manually checking for regressions•slow & error-prone•limited test coverage •boring! !

•Automated regression testing•fast & reliable•much greater test coverage

Page 8: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

OBIEE Conceptual Overview

Analysis Analysis

Data Sources

Dashboard

RPDBI Server

-------------------- Sending query to database named orcl (id: <<19108>>), connection pool named Connection Pool 01 !select sum(T117.FCAST_COST_AMT) as c1, T107.PROD_TYPE_DESC as c2, T127.MONTH_YYYYMM as c4 from GCBC_SALES.TIMES T127 /* Dim_TIMES */ , […]

Page 9: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

What Could Possibly Go Wrong?

•Data‣Are the numbers still correct?

•Functionality‣Do dashboards still run without error?‣Do dashboards still render in the same way?

•Security‣Data‣Visible Dashboards

Page 10: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Whoops… I broke it (1) - the RPD

•RPD may still pass consistency check - but do you know if you broke existing reports and dashboards?

•Cleanup / Refactoring•Developments•Bugfixes

Page 11: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Whoops… I broke it (2) - OBIEE front end

•Cleanup / Refactored calculations•Presentation Catalog changes•Application Role changes

Page 12: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Whoops… I broke it (3) - Database

•Refactored designs•View -> Table•Table -> MView• Indexes•Partitioning

Page 13: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

What Makes a Good Regression Test?

Easy

Better tests

Fast Flexible

TargetedScalable

Page 14: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

The OBIEE stack

Request

Response

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

User WebLogic Server OBIEE system components Data Source(s)

Net

wor

k

Net

wor

k

Not all components listed, eg cluster controller

-------------------- SQL Request, logical request hash: 810510fe SET VARIABLE QUERY_SRC_CD='Report',SAW_SRC_PATH='/shared/Test/test1 jan\/feb';SELECT 0 s_0, "Sales - Fact Sales"."Dim Products"."Product Type" s_1, "Sales - Fact Sales"."Dim Times"."Month" s_2, SORTKEY("Sales - Fact Sales"."Dim Times"."Month") s_3, "Sales - Fact Sales"."Fact Sales"."Cost" s_4 FROM "Sales - Fact Sales"

C1 C2 C3 C4 ---------- -------------------- -------------------- ---------- 4636.93 Bread Assortments FEB-2000 200002 4583.53 Bread Assortments JAN-2000 200001 3445.16 Bread Clubs FEB-2000 200002 3542.49 Bread Clubs JAN-2000 200001 4735.63 Cold Drinks FEB-2000 200002 4565.21 Cold Drinks JAN-2000 200001 4604.19 Gifts & Baskets FEB-2000 200002 […]

------------------------------------------------------------------------------------------- s_0 s_1 s_2 s_3 s_4 ------------------------------------------------------------------------------------------- 0 Bread Assortments JAN-2000 200001 4583.53 0 Bread Clubs JAN-2000 200001 3542.49 0 Cold Drinks JAN-2000 200001 4565.21 0 Gifts & Baskets JAN-2000 200001 4608.01 0 Hot Drinks JAN-2000 200001 5768.62 […]

Sending query to database named orcl (id: connection pool named Connection Pool 01 !select sum(T117.FCAST_COST_AMT) as c1, T107.PROD_TYPE_DESC as c2, T127.MONTH_DESC as c3, T127.MONTH_YYYYMM as c4 from GCBC_SALES.TIMES T127 /* Dim_TIMES */ , GCBC_SALES.PRODUCTS T107 /* Dim_PRODUCTS */ , GCBC_SALES.SALES T117 /* Fact_SALES */

Page 15: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Where Can We Detect Problems?

User WebLogic Server OBIEE system components Data Source(s)

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

Analysis request Logical SQL Physical Query x n

Query results x n

DatasetRendered Analysis

RPD

RPD

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

2

3

45

1

•Logical

•Physical

•Front End

41

3

5

2

Page 16: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Regression Testing Opportunities

•OBIEE Logical model‣Does the RPD still work and return the same results for a given logical query?

•Physical Database‣Does the Physical SQL that OBIEE generates still run?

•OBIEE front end‣From the point of view of an end user, does it look right?

Page 17: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

User WebLogic Server OBIEE system components Data Source(s)

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

RPD

RPD

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

Physical SQL

Dataset

Regression Testing Database Changes

Page 18: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Regression Testing Database Changes

•If only the DB has changed, then just test the DB. Don’t complicate things by involving the stack further up.

nqquery.log / Usage Tracking

SQL*Plus

•So long as the query still runs, and returns the same data, you’re guaranteed that nothing will change in OBIEE

Page 19: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Logical OBIEE Regression Testing

User WebLogic Server OBIEE system components Data Source(s)

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

RPD

RPD

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]

Logical SQL

Dataset

Page 20: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Logical SQL in OBIEE

Analysis

Logical SQL

Resultset via nqcmd

Analysis results

BI Server

RPD

Page 21: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

RPD Regression Testing in Action

1. Tool automagically extracts Logical SQL for all analyses in targeted dashboards or analyses

Page 22: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

RPD Regression Testing in Action

2. Tools runs captured Logical SQL to generate the baseline data

Page 23: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

RPD Regression Testing in Action

3. Developers make their RPD changes and deploy

Page 24: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

RPD Regression Testing in Action

4. Tool runs regression test to check both data and Logical SQL match the baseline

Page 25: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Regression Testing the OBIEE Front End

User WebLogic Server OBIEE system components Data Source(s)

BI Server

BI Server

Presentation Services

Managed Server

Managed Server

JavaHost

Presentation Services

Web Browser

Web Browser

BI Plug-in

BI Plug-in

RPD

RPD

DWH

[ ... ]

[ ... ]

DWH

[ ... ]

[ ... ]HTML, JS, CSS,

PNG

Page 26: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Regression Testing the OBIEE Front End

•Front End testing is complex!!•Two options to make sense of HTML, JavaScript, CSS, images: ‣Programatically - Parse server response‣Visually - Render server response

Page 27: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Front End test methods: Parsing vs Rendering

•Parsing‣Makes pass/fail tests easier-A string is present, or it is not!

‣Higher rate of false negatives-It’s easier to miss problems!

‣High development & maintenance cost-Need to reverse-engineer DOM-Often relies on non-deterministic internal IDs to accurately pinpoint assertions

Page 28: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Front End test methods: Parsing vs Rendering

•Rendering‣More accurate-Can detect differences not expected or explicitly coded for

‣More prone to false positives-Lots of things change on a page for valid reasons (dates, alerts, etc)

‣Still requires DOM knowledge but less reliance on internal IDs

Page 29: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Visual Regression Testing

Before

After Test Result: Fail

•Suitable for testing front-end changes, such as:‣Refactoring Analysis formulae into the RPD‣Presentation Catalog restructuring

Page 30: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

DEMO

Page 31: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Visual Regression Testing In Action

1. Tool takes automatic snapshot of dashboard pages. This is the baseline.

Page 32: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Visual Regression Testing In Action

2. Make Analysis/RPD changes

Page 33: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Visual Regression Testing In Action

Before After

Test Output

3. Tool captures new screenshots and compares against baseline. Any differences are reported.

Page 34: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Working smarter - summary

TestPhysical Logical Front End

What Changed?Report / Dashboard No No Yes

RPD No Yes YesDatabase Yes Yes Yes

Page 35: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

Smart Regression Testing

Do better OBIEE development and deliver it more rapidly

Page 36: Regression Testing OBIEE without the fuss

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead

#EOF✴ email: [email protected] ✴ web: http://ritt.md/rmoff ✴ twitter: @rmoff