Upload
mark-rittman
View
120
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Oracle Exalytics is Oracle’s hardware/software platform for “speed of thought” analytics. This presentation, delivered by a speaker from an Oracle partner that was the first to buy Oracle Exalytics in the U.K., looks at the architecture of Oracle Exalytics and goes through best practices for the install and configuration of the product and for the use of Oracle In-Memory Database Cache together with the other caching and optimization tools available with the platform. The session also considers alternative approaches to loading and populating the in-memory cache, including the use of Oracle GoldenGate, and discusses what types of performance problems are best solved by Oracle Exalytics.
Citation preview
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Mark Rittman, Technical Director, Rittman MeadOracle Openworld 2012, San Francisco, September 2012
Oracle Exalytics and TimesTen Best Practices
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Mark Rittman
•Mark Rittman, Co-Founder of Rittman Mead•Oracle ACE Director, specialising in Oracle BI&DW•14 Years Experience with Oracle Technology•Regular columnist for Oracle Magazine•Author of two Oracle Press Oracle BI books
‣Oracle Business Intelligence Developers Guide‣Oracle Exalytics Revealed
•Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog
•Email : [email protected]•Twitter : @markrittman
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
About Rittman Mead
•Oracle BI and DW gold partner•World leading specialist partner for technical excellence, solutions delivery and
innovation in Oracle BI•Approximately 30 consultants worldwide•All expert in Oracle BI and DW•UK based•Offices in US, Europe (Belgium) and India•Skills in broad range of supporting Oracle tools:
‣OBIEE‣OBIA‣ODIEE‣Essbase, Oracle OLAP‣GoldenGate‣Exadata
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
What is Oracle Exalytics?
•Hardware Element‣Sun Fire X4470 M2 server‣1TB RAM, 40 Cores, 3.6TB HDD
•Software Element‣OBIEE 11.1.1.6 with Exalytics Enhancements‣Oracle Essbase 11.1.2 with Exalytics Enhancements‣Oracle TimesTen 11.2.2.2 for Exalytics‣New in v1.1 : Support for Oracle Endeca
Information Discovery, Golden Gate, ODI‣Runs on 64-bit Oracle Linux
(Exalogic distribution)•OBIEE and Essbase are licensed as
Oracle BI Foundation•Exalytics features can only be used in
conjunction with Exalytics hardware
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Rittman Mead and Oracle Exalytics
•Rittman Mead are the first Oracle Partner in the UK to buy an Exalytics box•Ordered February 2012 through Arrow (reseller), delivered end-April 2012•Hosted in a secure datacenter in London Docklands, access via VPN•To be used for customer PoCs, training, testing and possible cloud hosting
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Installation and Delivery Process
•Installed by Oracle ACS over two days•Day 1 involved unpacking, racking, connecting to network etc•Day 2 was for installing the software (only ships with Oracle Linux)
‣Oracle WebLogic‣Oracle BI EE 11g‣Oracle TimesTen for Exalytics
•Post-install config from ACS for‣Creating initial TimesTen DB‣Configuring BI EE for Exalytics‣Configuring BI Server > TimesTen connection
•Post-install config from Rittman Mead included:‣Configuring RPD for Usage Tracking &
Summary Statistics‣Connecting RPD to TimesTen DB
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exalytics as the Exa-Machine for BI
•Runs the BI layer on a high-performance, multi-core, 1TB server•In-memory cache used to accelerate the BI part of the stack•If Exadata addresses 80% of the query performance,
Exalytics addresses the remaining 20%‣Consistent response times for queries‣In-memory caching of aggregates‣40 cores for high concurrency‣Re-engineered BI and OLAP software
that assumes 40 cores and 1TB RAM
ERP/Apps DW
Oracle BI
In-Memory DB/Cache
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Also Supports Essbase, and Endeca Information Discovery
•In-Memory Essbase for planning, budgeting and sales analysis-style OLAP applications•Endeca Information Discovery for search/analytic applications against diverse data
OracleExalytics
In-Memory Machine
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Enables High-Density Analysis of Big Data
•BI tier is rarely the bottleneck, but it can be if very dense visualizations are used‣Sparklines, grid of charts etc
•Exalytics’ 40 cores and 1TB RAM make higher density presentation viable‣Single query sent to the database‣Exalytics breaks data up to create microcharts
•Also helps support high numbers of concurrent users (100+)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exalytics Architecture
•Developer tools used for definingand maintaining aggregate definitions
•Client tools for OBIEE repository,TimesTen and Oracle Database
•Exalytics layer contains OBIEE,TimesTen and Essbase(not pictured)
•Database tier contains FMWdatabase repository, includingusage tracking and summary statistics
Developer Workstation
Oracle BI Administration Tool
Summary Advisor Wizard
nqcmd
TimesTen Client
SQL Developer
Oracle Database 11g Client
Oracle Business IntelligenceOracle TimesTen for Exalytics
WLS AdminServer
WLS ManagedServer
Oracle BISystem Comp.
BI Server
Aggregate Schema
Database for RCU Schemas
Enhanced Usage TrackingSummary Statistics
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exalytics Under the Covers - How Does it Work?
• Exalytics brings together different technologies, which are still standalone products in their own right
• To harmonise and optimise their use within Exalytics, it utilises the following techniques:‣ In-Memory Adaptive Data Mart‣ In-Memory Intelligent Result Cache‣ In-Memory Cubes
• Some of these are genuine "secret sauce"‣ New functionality and algorithms‣ You can only get them through licensing Exalytics
• Others are descriptions of DW/BI strategies, or existing product functionality, extended to take advantage of the capacity for processing in memory that Exalytics has
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
In-Memory Adaptive Data Mart
•Commonly-used aggregates are copied into Oracle TimesTen for Exalytics•Past query patterns are analyzed and suitable aggregates recommended•Oracle BI Server then uses these aggregates to make queries run faster•Aggregates change over time in
response to changes in query patterns•Tools are provided for managing
and populating these aggregates
TimesTen BI Server
Exa
lyti
cs
Aggregates
Data Warehouse
Detail-levelData
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
In-Memory Intelligent Results Cache
•OBIEE has an advanced result cache, which can support:‣Queries matching previous ones‣Queries matching a subset of the results of a cached query resultset ‣Queries aggregating the results of a cached query resultset
•Stored in cache files on the OBIEE server filesystem•With Exalytics, the 1TB of RAM typically means
that these files are held in memory•RAMdisk can also be created for
BI Server & Presentation Serverworking area files
TimesTen BI Server
Exa
lyti
cs
In-Memory Result Cache
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Oracle TimesTen for Exalytics
•New version of TimesTen specifically for Exalytics (and only available with Exalytics)•Support for analytic functions
‣Perform all the processing at source‣Combine with being in-memory = should be very fast
•Column compression‣Whitepaper cites 5x‣Given the hardware capacity, we could seriously contemplate loading the whole Data
Warehouse into memory‣Opens up lots of interesting design potential
•We can load aggregates into TimesTen, leave base data at source, and use OBIEE’s Vertical Federation capability to seamlessly report across both‣All hidden from the end-user, all they will know is that their reports run fast!
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
TimesTen and OBIEE Architecture
•Oracle BI Server communicates with TimesTen through TimesTen Client•Summary Advisor, and nqcmd use Oracle BI Server to access TimesTen•Typical single TimesTen database per Exalyics machine
‣Max TimesTen database size around 300MB- Due to need to set aside equal
Temp size for the Perm size selected•Clustered Exalytics boxes can be daisy-chained
together using InfinBand connections‣For HA scenarios, does not increase
available RAM‣Summary advisor scripts write to both TimesTen
databases, replicating aggregates‣TimesTen databases can be “wired together”
for failover/HA purposes
TimesTenMemory-ResidentDatabase
CheckpointFiles
LogFiles
ODBC
Oracle BIServer
nqcmdSummaryAdvisor
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Summary Advisor for Aggregate Recommendation & Creation
•Utility within Oracle BI Administrator tool that recommends aggregates•Bases recommendations on usage tracking and summary statistics data•Captured based on past activity•Runs an iterative algorithm that searches,
each iteration, for the best aggregate
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Recommendations Based on Enhanced Usage Tracking Data
•Historically, usage tracking has been tracked using S_NQ_ACCT‣Holds basic usage tracking statistics + logical SQL query
•Now supplemented by S_NQ_DB_ACCT ‣Extra usage tracking information, includes physical SQL
•Exalytics Summary Advisor uses S_NQ_SUMMARY_ADVISOR‣Contains summary statistics, execution time etc‣Gathered at same time as usage tracking when
Exalytics is enabed‣Contents can be derived from usage tracking if needed
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Oracle Exalytics Best Practices
•Installation, Configuration for Multiple Environments, Management•Choosing What Goes in the Adaptive Data Mart •Where Does Exalytics Add Value?•Measuring Exalytics Performance and Metrics•Maintaining and Refreshing the Adaptive Data Mart
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Configuring Multiple Environments (Consolidation)
•Exalytics main value proposition is around performance‣In-memory aggregates speeding up query performance
•However many customers wish to use Exalytics for consolidation‣Run multiple environments (test, dev, prod) on a single server‣Consolidate multiple applications on a single server
•Desire is usually down to cost (multiple physical servers, per-processor licensing)
DEV DEV TEST PROD
vs.
MWHOME1
DEV
MWHOME2
DEV
MWHOME3 MWHOME1 MWHOME1 MWHOME1
Single Exalytics Server Exalytics Svr 1 Exalytics Svr 2 Exalytics Svr 3
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Common Consolidation Approach - Multiple Installs
•Up until recently, customers wishing to consolidate usually performed multipleMiddleware Home installs
•OBIEE element usually works fine (installer detects existing ports use, configures OK)•However TimesTen element may not work as expected with multiple installs
‣Checkpointing clashes between TT installs, single TT install vs. multiple installs etc‣No resource caging between MWHome environments
•Approach is therefore not recommended for consolidation use-cases
DEV
MWHOME1
DEV
MWHOME2
DEV
MWHOME3
Single Exalytics Server
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Best Practice for Consolidation : Oracle Virtual Machine
•Exalytics is now certified for use with Oracle Virtual Machine, Oracle’s hard-partitioning virtualization platform
•OVM is installed first, on Exalytics “bare metal”, with OVM Repository on external DB server (as with BIPLATFORM, MDS)
•Exalytics VM templates then used to create required environments (dev, test etc)
•Also allows for licensing in smaller per-processor license blocks (5, 10, 15 etc)
•OVM provides for resource caging, isolating environments from each other
DEV
MWHOME1
Single Exalytics Server
OVM Hypervisor (Oracle VM Server)
OVM Guest 1
DEV
MWHOME1
OVM Guest 2
DEV
MWHOME1
OVM Guest 3
Oracle VMManager
Oracle VMRepository
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
High Availability using Clustered Exalytics Servers
•Multiple Exalytics servers can be clustered for HA and performance (load balancing)‣Requires hardware load balancer above clustered servers
•Linked together via InfiniBand (one for clustering, one for connection to Exadata)•Note: does not double available RAM
- aggs are replicated in each TT database‣Note that Summary Advisor / nqcmd
does not replicate agg tables -manual replication required
•HA is between TT instances, and OBIEEmanaged server + system components
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Management Best Practices
•Exalytics is primarily an OBIEE system, and usual best practices apply•May want to consider cloning tool (CloneZilla etc) for cloning “clean” server build•New : Oracle Enterprise Manager 12cR2 and BI Management Pack
‣OEM 12cR2 prerequisite for sub-capacity licensing using OVM
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
OEM 12cR2 Additional Capabilities vs. Fusion Middleware Control
•Ability to report on DAC runs for BI Apps, plus BI configuration settings etc
•Usage tracking reporting of dashboard usage, analysis run-times etc
•Manage Essbase and other BI / DB / Middleware components
•Long-term persistence of metrics, SLAs, alerts•Middleware cloning/
management
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
What Should the TimesTen Adaptive Data Mart be Used For?
•Standard approach is to store aggregates in the TimesTen datamart‣Aggregated by the source DB, aggregates then cached in TT database
•Other approaches could be used, however‣Store whole detail-level dataset in the TT database‣Store just recent detail-level data in TT, and use OBIEE’s fragmentation feature‣Use Essbase to hold the aggregates instead‣Use Essbase to hold detail-level + aggregates
•Where does Exalytics add value? •What should go in the TimesTen database?
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Potential TimesTen Use Case : In-Memory Database Cache
•Automatically replicate “hot” transactional data from Oracle into TimesTen•Use OBIEE fragmentation to enable automatic navigation between sources•Aggregation performed by both TimesTen, and by source DB (as appropriate)
In-Memory DataBase Cache
Transactional data
DATE = TODAY
DATE < TODAY
Transactional data
DATE = TODAY
BI Server
OBIEE chooses the correct source based on date
range
LTS Fragmentation in the RPD
specifies the data cutoff
Cache Group is defined on TimesTen
No change to Oracle source server, except minor schema config
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Conclusions and Best Practices Around Detail/Aggregate Storage
•Best use of OBIEE + TT is how the product was intended to be used‣Use TimesTen for storing (caching) pre-created aggregates‣Aggregates created by source DB (Oracle, SQL Server etc)‣Aggregation not performed by TimesTen (as it would be if it held detail-level data)
•IMDB interesting approach, but relies on TT performing aggregation (slow)•Same goes for holding whole detail-level source in TimesTen•Also, bear in mind that majority of performance benefit may come from re-hosting
OBIEE application on the Exalytics hardware‣Particularly with Exadata sources, source DB may even be fastest host for aggs‣Test for your data, do not pre-suppose outcome
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Testing Exalytics
•Need to quantify the performance benefit of various Exalytics deployment options•Take a baseline, and then repeat the same test for incremental changes•Several test framework options
‣Manual‣Homebrew - Logical SQL + nqcmd‣Performance test tool - JMeter, OATS, etc
Presentation Services
BI Server
TimesTen
nqcmdLogical
SQL
JMeter
Run dashboards manually, and time with a stopwatch
Fully automated web user performance test tool
Extract Logical SQL, run with nqcmd
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Plan your testing
•Tempting to go in all guns blazing‣40 cores!! 1TB memory!! w00t!11!
•Decide on the aim of the test. For example: ‣Validate a configuration option‣Determine the response time for a dashboard‣Test the performance under concurrent load
•For configuration option testing, emphasis should be on absolute repeatability•If testing to determine system capacity, make sure the workload model is representative
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Setting up JMeter
•JMeter can run on Windows or *nix•No installation, just unzip•Record and customize test script•Execute from JMeter running from suitable location
‣Consider network, and host server size
TimesTen
BI Server JMeter
TimesTen
BI Server
JMeter
TimesTen
BI Server
Virtual Machine
JMeter
Run locally on Exalytics
Run within a VM on Exalytics
Run remotely, across appropriate network link to Exalytics
Option 1 Option 2 Option 3
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Analyzing the test output
•What’s the response time on a dashboard?
•How does the response time change if there are concurrent users?
How long did the dashboard take to run?
Response TimevsNumber of users
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Other metrics
•“Will the system support xxxx users?” is a common question
•JMeter is good at simulating multiple users‣Be careful with your workload model
though•For system capacity in particular, monitor
OS and OBIEE performance metrics•Consider EM12cR2 for long-term metrics OS metrics
OBI metrics
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Refreshing Aggregates Within the TimesTen Data Mart
•Aggregates are built in TimesTen to support sub-second response times•Summary Advisor tool suggests aggregate dimensionality and grain, generates script•BI Server’s Aggregate Persistence executes script:
1. Create TimesTen aggregate table2. Populate TimesTen aggregate3. Update RPD online with new aggregate metadata
•Handles supporting dimensions too
Base data Aggregates
TimesTen
OBIEE Aggregate Persistence
RPD
Aggregate tables created
RPD updated with new aggregate mappings
Aggregate tables loaded from base data
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Aggregate Persistence limitations
•To refresh an aggregate, it is deleted and rebuilt from scratch•The RPD is edited directly on the BI Server each time an aggregate is created or rebuilt•Build failures can be difficult to debug•If a rebuild fails it can leave the RPD in an inconsistent state with TimesTen
After a failed build, the aggregates are still in the RPD, but no longer exist in TimesTen
nqquery.log suggests possible errors but no clear root cause
This is that the error the user sees
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Alternatives to using Aggregate Persistence
•However aggregates are refreshed, they must be included in the RPD‣Can be done manually‣Aggregate Persistence is useful for this
•Complete refresh of aggregate data‣Extract the SQL that OBIEE generates in Aggregate
Persistence, run this through ODI‣Write bespoke aggregate refresh code in ODI
•Incremental refresh using GoldenGate and ODI‣Instead of rebuilding aggregates in their entirety each
time, only update the part of the aggregate that has changed on the base data
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Incremental refresh of aggregates using GoldenGate and ODI
•Changes to base data are replicated by GoldenGate onto Staging database•ODI Interfaces apply deltas to the TimesTen aggregates
Aggregates
TimesTen
CDC staging data
Oracle
Base data
(any supported DB)
GoldenGate ODI
ODI agentGoldenGate GoldenGate
TimesTen JDBC client
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Considerations if using Alternatives to Aggregate Persistence
•Exalytics “Out of the Box” is designed around using Aggregate Persistence‣Fair assumption that future developments and optimizations will be around this
model•If rate of change in aggregate requirements is high, then manual RPD changes can be
time-consuming•Hybrid approach - use Summary Advisor and Aggregate Persistence as an accelerator
for the initial RPD and Aggregate build, and then handle refreshes manually•There is no one answer -- It Depends!
‣Do you have time & skills available to write bespoke code?‣Rate of change in your aggregate requirements‣Existing deployment landscape and technologies available‣Risk of missing out on Exalytics optimizations if too far from OOTB deployment?
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Summary
•Exalytics is a powerful platform for hosting OBIEE, optimized for a BI workload•New supported options for hosting multiple environments on a single server•Oracle VM now permitted for multiple environments and sub-capacity licensing•TimesTen is best used as a cache store for aggregate tables•Typically, use source DBs for initial aggregation and storage of detail-level data•Use JMeter or other metrics capture tools to measure performance before and after•Manage Exalytics (+ OVM) using EM12cR2 Cloud Control•Options to use ODI + GoldenGate as alternatives to Summary Advisor for agg refreshes
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
More Information
•Thank you for attending this presentation•More information can be found at http://www.rittmanmead.com•Contact us at [email protected] or [email protected]•Look out for our book, “Oracle Business Intelligence Developers Guide” due Q3 2012•Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Mark Rittman, Technical Director, Rittman MeadOracle Openworld 2012, San Francisco, September 2012
Oracle Exalytics and TimesTen Best Practices