Upload
robin-moffatt
View
125
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Just because OBIEE is presented to users through a web browser that doesn't mean the regression testing has to take place there too. This presentation will demonstrate how you can efficiently and effectively test the bulk of changes made to your system without going anywhere near the front end. Where front end testing is needing, it demonstrates the concept of visual regression testing using the latest tools. With hands-on examples of the tools involved this will be a practical presentation giving anyone interested in the topic a headstart in practical implementation. Video available here: http://ritt.md/regression-testing-video
Citation preview
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Regression Testing OBIEE Without the Fuss Robin Moffatt Principal Consultant, Rittman Mead
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Smart Regression Testing
Do better OBIEE development and deliver it more rapidly
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Why Regression Test?
Confidence
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Why?
Confidence
Agility in development
Happier users
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
What?
•Regression Testing is not just a prolonged exercise on a Gantt chart, just before go live
•Smart Regression Testing can be run on demand, targeting the change made and assuring quality during development
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Detect Problems Early
‣Fresh in the developer’s mind‣Easier to fix !
‣Fewer surprises during big test cycles‣More honest & realistic project status reports !
‣Better overall QA for the system‣Happier users
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Save Time
•Manually checking for regressions•slow & error-prone•limited test coverage •boring! !
•Automated regression testing•fast & reliable•much greater test coverage
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
OBIEE Conceptual Overview
Analysis Analysis
Data Sources
Dashboard
RPDBI Server
-------------------- Sending query to database named orcl (id: <<19108>>), connection pool named Connection Pool 01 !select sum(T117.FCAST_COST_AMT) as c1, T107.PROD_TYPE_DESC as c2, T127.MONTH_YYYYMM as c4 from GCBC_SALES.TIMES T127 /* Dim_TIMES */ , […]
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
What Could Possibly Go Wrong?
•Data‣Are the numbers still correct?
•Functionality‣Do dashboards still run without error?‣Do dashboards still render in the same way?
•Security‣Data‣Visible Dashboards
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Whoops… I broke it (1) - the RPD
•RPD may still pass consistency check - but do you know if you broke existing reports and dashboards?
•Cleanup / Refactoring•Developments•Bugfixes
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Whoops… I broke it (2) - OBIEE front end
•Cleanup / Refactored calculations•Presentation Catalog changes•Application Role changes
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Whoops… I broke it (3) - Database
•Refactored designs•View -> Table•Table -> MView• Indexes•Partitioning
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
What Makes a Good Regression Test?
Easy
Better tests
Fast Flexible
TargetedScalable
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
The OBIEE stack
Request
Response
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]
User WebLogic Server OBIEE system components Data Source(s)
Net
wor
k
Net
wor
k
Not all components listed, eg cluster controller
-------------------- SQL Request, logical request hash: 810510fe SET VARIABLE QUERY_SRC_CD='Report',SAW_SRC_PATH='/shared/Test/test1 jan\/feb';SELECT 0 s_0, "Sales - Fact Sales"."Dim Products"."Product Type" s_1, "Sales - Fact Sales"."Dim Times"."Month" s_2, SORTKEY("Sales - Fact Sales"."Dim Times"."Month") s_3, "Sales - Fact Sales"."Fact Sales"."Cost" s_4 FROM "Sales - Fact Sales"
C1 C2 C3 C4 ---------- -------------------- -------------------- ---------- 4636.93 Bread Assortments FEB-2000 200002 4583.53 Bread Assortments JAN-2000 200001 3445.16 Bread Clubs FEB-2000 200002 3542.49 Bread Clubs JAN-2000 200001 4735.63 Cold Drinks FEB-2000 200002 4565.21 Cold Drinks JAN-2000 200001 4604.19 Gifts & Baskets FEB-2000 200002 […]
------------------------------------------------------------------------------------------- s_0 s_1 s_2 s_3 s_4 ------------------------------------------------------------------------------------------- 0 Bread Assortments JAN-2000 200001 4583.53 0 Bread Clubs JAN-2000 200001 3542.49 0 Cold Drinks JAN-2000 200001 4565.21 0 Gifts & Baskets JAN-2000 200001 4608.01 0 Hot Drinks JAN-2000 200001 5768.62 […]
Sending query to database named orcl (id: connection pool named Connection Pool 01 !select sum(T117.FCAST_COST_AMT) as c1, T107.PROD_TYPE_DESC as c2, T127.MONTH_DESC as c3, T127.MONTH_YYYYMM as c4 from GCBC_SALES.TIMES T127 /* Dim_TIMES */ , GCBC_SALES.PRODUCTS T107 /* Dim_PRODUCTS */ , GCBC_SALES.SALES T117 /* Fact_SALES */
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Where Can We Detect Problems?
User WebLogic Server OBIEE system components Data Source(s)
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
Analysis request Logical SQL Physical Query x n
Query results x n
DatasetRendered Analysis
RPD
RPD
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]
2
3
45
1
•Logical
•Physical
•Front End
41
3
5
2
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Regression Testing Opportunities
•OBIEE Logical model‣Does the RPD still work and return the same results for a given logical query?
•Physical Database‣Does the Physical SQL that OBIEE generates still run?
•OBIEE front end‣From the point of view of an end user, does it look right?
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
User WebLogic Server OBIEE system components Data Source(s)
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
RPD
RPD
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]
Physical SQL
Dataset
Regression Testing Database Changes
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Regression Testing Database Changes
•If only the DB has changed, then just test the DB. Don’t complicate things by involving the stack further up.
nqquery.log / Usage Tracking
SQL*Plus
•So long as the query still runs, and returns the same data, you’re guaranteed that nothing will change in OBIEE
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Logical OBIEE Regression Testing
User WebLogic Server OBIEE system components Data Source(s)
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
RPD
RPD
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]
Logical SQL
Dataset
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Logical SQL in OBIEE
Analysis
Logical SQL
Resultset via nqcmd
Analysis results
BI Server
RPD
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
RPD Regression Testing in Action
1. Tool automagically extracts Logical SQL for all analyses in targeted dashboards or analyses
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
RPD Regression Testing in Action
2. Tools runs captured Logical SQL to generate the baseline data
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
RPD Regression Testing in Action
3. Developers make their RPD changes and deploy
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
RPD Regression Testing in Action
4. Tool runs regression test to check both data and Logical SQL match the baseline
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Regression Testing the OBIEE Front End
User WebLogic Server OBIEE system components Data Source(s)
BI Server
BI Server
Presentation Services
Managed Server
Managed Server
JavaHost
Presentation Services
Web Browser
Web Browser
BI Plug-in
BI Plug-in
RPD
RPD
DWH
[ ... ]
[ ... ]
DWH
[ ... ]
[ ... ]HTML, JS, CSS,
PNG
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Regression Testing the OBIEE Front End
•Front End testing is complex!!•Two options to make sense of HTML, JavaScript, CSS, images: ‣Programatically - Parse server response‣Visually - Render server response
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Front End test methods: Parsing vs Rendering
•Parsing‣Makes pass/fail tests easier-A string is present, or it is not!
‣Higher rate of false negatives-It’s easier to miss problems!
‣High development & maintenance cost-Need to reverse-engineer DOM-Often relies on non-deterministic internal IDs to accurately pinpoint assertions
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Front End test methods: Parsing vs Rendering
•Rendering‣More accurate-Can detect differences not expected or explicitly coded for
‣More prone to false positives-Lots of things change on a page for valid reasons (dates, alerts, etc)
‣Still requires DOM knowledge but less reliance on internal IDs
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Visual Regression Testing
Before
After Test Result: Fail
•Suitable for testing front-end changes, such as:‣Refactoring Analysis formulae into the RPD‣Presentation Catalog restructuring
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
DEMO
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Visual Regression Testing In Action
1. Tool takes automatic snapshot of dashboard pages. This is the baseline.
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Visual Regression Testing In Action
2. Make Analysis/RPD changes
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Visual Regression Testing In Action
Before After
Test Output
3. Tool captures new screenshots and compares against baseline. Any differences are reported.
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Working smarter - summary
TestPhysical Logical Front End
What Changed?Report / Dashboard No No Yes
RPD No Yes YesDatabase Yes Yes Yes
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
Smart Regression Testing
Do better OBIEE development and deliver it more rapidly
www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
#EOF✴ email: [email protected] ✴ web: http://ritt.md/rmoff ✴ twitter: @rmoff