Top 10 Tricks and Tools of an Oracle EPM Administrator

Top 10 Tricks and Tools of an Oracle EPM AdministratorEffective EPM Administration

Nicholas KingDatabase Administrator

Oct 2014

Purpose and background

Agenda: Tips & Tricks for...● Improving uptime by building a troubleshooting toolkit● Automation / working effectively / less toil● Security / auditing

Nicholas King● 6 years as Unix sysadmin and software development● 8 years Hyperion infrastructure consulting

=14 years administering complex enterprise systems

Tip 1: Manual TroubleshootingFocus on Uptime

Search for *.log on one server

Types of Logs● Domains: WebLogic Logs

user_projects\domains\EPMSystem\servers● Diagnostics: Application Logs

user_projects\epmsystem1\diagnostics\logs● Windows Event Viewer● HFM Error Log Viewer● FDM Outbox

Too Many Logs!

Too many logs Tip #1

Effective troubleshooting

depends on getting useful error messages

from logs.

Demo

Demo: Manual Troubleshooting

Tip #1

Tip 2: Log Analysis UtilityFind Problems Fast

What if you could scan hundreds of logs at once?● Step 1: Search folder < 60 min old

● Step 2: Search both “domains” and “diagnostics”

● Step 3: Run against multiple servers using UNC path

Using the Log Analysis Utility

loganalysis.bat -system -tmin 60 -d search_directory -o output_filename

loganalysis.bat ... -d d:\Oracle\Middleware\user_projects\ domains -o domains_reportloganalysis.bat ... -d d:\Oracle\Middleware\\user_projects\epmsystem1\ diagnostics -o diagnostics_report

loganalysis.bat ... \\svr1\d$\Oracle\Middleware\user_projects\epmsystem1 \diagnostics -o node1_diagnostics

Tip #2

Log Analysis Utility on multiple servers

start cmd /C "loganalysis.bat -system -tmin 60 -d

\\hsssvr\d$\Oracle\Middleware\user_projects\domains -o

web1_domains"


\\hsssvr\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics

-o web1_diagnostics "

start cmd /C "loganalysis.bat -system -tmin 60

-d \\hfmsvr\d$\Oracle\Middleware\user_projects\domains -o

hfm1_domains"


\\hfmsvr\d$\Oracle\Middleware\user_projects\epmsystem1\diagnostics

-o hfm1_diagnostics "

Server 2Server 1

Server xServer y

Server z...

Tip #2

Each browser tab contains a different report hss_diagnostics, hss_domains ra_diagnostics, ra_domains hfm_diagnostics, hfm_domains

Log Analysis Utility output Tip #2

Demo

Demo: Log Analysis Utility

Tip #2

Log Analysis Utility comparison

Advantages● Targeted search for just errors● No experience necessary● Simple, first pass at log analysis● Hundreds of files scanned in

seconds● Easy to read HTML report● Detailed messages can lead

to root cause

Disadvantages● Scans Oracle log format only ● Still need logs to get context

about activities before and after error message

Tip #2

Tip 3: Validate UtilityTiny Testing Tool

What does it do?● A custom test suite with hundreds of intricate tests

Invocation: user_projects\epmsystem1\bin\validate.bat

Tip #3Introduction to Validate Utility

user_projects\epmsystem1\config\validation\11.1.2.0\vtconfig.xml

Validate Utility internals

hat_tmp\hatInput.vbs > diagnostics\logs\validation\hat.output ----STARTING VALIDATION SCRIPTS---- LOGGING IN HFM.... CREATING APPLICATION.... OPENING APPLICATION.... LOADING SECURITY.... LOADING METADATA....

Tip #3

Validate Utility comparison

Advantages● Specialized test suite● Intricate tests - much more

than just a connectivity test● HTML based output is

easy to read● Pinpoint detailed internal

errors

Disadvantages● Errors can be missed if

they do not fail a specific test

● Failed tests may not identify the root cause

● Must be run on each node

Tip #3

Tip 4: LCM AutomationDuct Tape for Quick Repairs

Lifecycle Management automation

Nightly Automated LCM Exports to Date Stamped Folder (versioning)● Partial restore without downtime

● Financial Reports● Shared Services Security ● Internal HSS registry ● Application artifacts - planning forms, EPMA hierarchy...etc

How to Invoke● Location: D:\Oracle\Middleware\user_projects\epmsystem1\bin● Utility.bat XML_File

● Every LCM export in SS contains a template import.xml and export.xml. Just add username, password to XML file.

Simple: only need an XML file

Tip #4

Demo

Demo: Versioned Financial Reports

Tip #4

Tip 5: Copy ProvisioningEstablish Security Fast

Clone/mirror Shared Services security from one app to another

Why Copy Provisioning?● Cloning security for Essbase cubes● Quickly apply security to a test app● Migrating security from an old app to newer version● Pushing/Staging large scale security changes

Copy provisioning introduction

FMAPP3

Provisioning

Tip #5

user1: Consol Adminuser2: Post Journals

FMAPP2

Sync security example

Example:● Clone FMTEST2 Security to FMTEST3● FMTEST3 is not provisioned

Initial State:Provisioning Report for FMTEST2

Provisioning Report for FMTEST3

Tip #5

Example: Syncing security

FMTEST3 security established in a few mouse clicks

Shared Services Application Groups Source: FMTEST2

Tip #5

Tip 6: Password ChangesKeep the System Secure

Database accounts in Config ToolSteps to Change Typical DB Accounts1. Change password for product in the Config Tool

2. Redeploy the application server 2. Change directly in WebLogicHome>Summary of Security Realms>Summary of JDBC Data Sources> eas_datasource

Tip #6

Tip 7: Working with Oracle SupportHow Detailed SRs Save You Time

General Oracle Support tips

Adding Details in the SR Saves You Time● Incomplete info will result in lost hours/days● Provide overview, context, step by step process, screenshots● Reproduce the problem - prove within a shadow of a doubt● Simply the problem - can you reproduce with a sample app, basic report,

etc.● Flip side - how can Oracle reproduce the problem outside of your env?● Ask for a WebEx - demonstrate the problem first hand

Tip #7

Tip 8: Oracle SR ToolsTools Required for Working with SRs

Creating a SR - deployment report

Deployment report - Layout of environment

Location: user_projects\epmsystem1\bin

Invocation: epmsys_registry report deployment

Output: diagnostics\reports\deployment_report.html

Product / Server Layout

Database Connections

Tip #8

Creating a SR - versions and patch levels

Opatch lsinventory - Exact versions of all products on serverLocation: D:\oracle\Middleware\EPMSystem11R1\OPatchInvocation: opatch lsinventory -oh D:\Oracle\Middleware\EPMSystem11R1 -jre D:\Oracle\Middleware\jdk160_29 Output: EPMSystem11R1\cfgtoollogs\opatch\lsinv

Patch 13905088 : applied on Mon Oct 29 16:14:16 PDT 2012Patch 13936087 : applied on Mon Oct 29 15:51:44 PDT 2012Patch 14031912 : applied on Mon Oct 29 15:07:44 PDT 2012

support.oracle.com Patch Search

Tip #8

Creating a SR - more tools

Zip logs - Collect all relevant logs on system Invocation: ziplogs Output: diagnostics\ziplogs\EPM_logs_hss-node_11.1.2.2.0.zipRegistry report - HTML view of internal EPM HSS Registry Invocation:epmsys_registry (no arguments) Output: diagnostics\reports\registry.htmlValidate (shown earlier) Invocation: validate Output: diagnostics\reports\instance_report*.html

Script Locations: user_projects\epmsystem1\bin

Tip #8

Tip 9: Auto Restart Failed ServicesHands Free Problem Solving

Restart failed Windows services automatically

● Error causes service stop/crash

● Not frequent, but may save some headaches

● Avoid repeated restarts● Custom settings reset every

time you rerun config tool

Command linesc \\localhost failure HyS9FRReports actions= restart/60000/restart/60000/""/60000 reset= 0

Tip #9

Tip 10: Exploring AuditIdentifying System Usage

Goal: View the number of logins v. time per product from the audit logs

Enable Audit in Shared Services: Administration - Configure Auditing

Exploring Audit Tip #10

Audit tables: basic query Tip #10

Basic query for Essbase logins v. time

To Do: Add unified time series for graphing

Audit tables: building a useful query (1) Tip #10

Create uniform time sequence for graphing


Join time series with original query


Results show uniform time with number of logins

Visualizing the results Tip #10

EssbaseWorkspace

Logins vs. Time by Product

Recap Tip #10

Tips and Tricks:1. Manual Troubleshooting2. Log Analysis Utility3. Validate Utility4. LCM Automation5. Copy Provisioning6. Password Changes7. Working with Oracle Support8. Oracle SR Tools9. Auto restart failed services

10. Exploring Audit

Objectives:● Uptime● Less toil / Effective EPM

Administration● Security

Thank You!Q&A

Nicholas [email protected]

Technology

Top 10 Tricks and Tools of an Oracle EPM Administrator