Tools, not only for Oracle RAC

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Tools, not only for RAC databases.. OraCHK, CHM, OSWatcher and TFA

Markus FlechtnerPrincipal Consultant

Our company.

Tools, not only for RAC databases2 24.12.16

Trivadis is a market leader in IT consulting, system integration, solution engineeringand the provision of IT services focusing on and technologiesin Switzerland, Germany, Austria and Denmark. We offer our services in the following strategic business fields:

Trivadis Services takes over the interactive operation of your IT systems.

O P E R A T I O N

COPENHAGEN

MUNICH

LAUSANNEBERN

ZURICHBRUGG

GENEVA

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASLE

VIENNA

With over 600 specialists and IT experts in your region.


14 Trivadis branches and more than600 employees

200 Service Level Agreements

Over 4,000 training participants

Research and development budget:CHF 5.0 / EUR 4 million

Financially self-supporting and sustainably profitable

Experience from more than 1,900 projects per year at over 800customers

About Markus Flechtner

Principal Consultant, Trivadis, Duesseldorf/Germany, since April 2008Discipline Manager Infrastructure Database @TrivadisWorking with Oracle since the 1990’s– Development (Forms, Reports, PL/SQL)– Support– Database Administration

Focus– Oracle Real Application Clusters– Database Migration Projects

Teacher– O-RAC – Oracle Real Application Clusters– O-NF12CDBA – Oracle 12c New Features for the DBA

Blog:http://markusdba.de/

@markusdba


Our database doctors ..

Dr. ORAchk– Regular screening examination

Dr. CHM & OSWatcher– Electrocardiogram (ECG)

Dr. TFA– In case of emergency


Oracle Support Tools Bundle

Collection of database and RAC support tools

Includes– ORAchk– ExaChk (*) – like OraChk, but for Engineered Systems– OSWatcher– ProcWatcher (*) – tool to examine and monitor Oracle database and/or clusterware

processes– ORATOP (*) - near real-time monitoring of databases– SQLT (*) – helps in tuning SQL statements– DARDA (*) - Diagnostic Assistant - interface for other diagnostic tools– .. And many more

Integrated in TFA collector since release 12.1.2.3.0 (*) not covered by this talk


Running other tools from TFA collectororacle@walhall:~/ [rdbms12102] tfactl toolstatus.--------------------------------------.| External Support Tools |+---------+--------------+-------------+| Host | Tool | Status |+---------+--------------+-------------+| walhall | alertsummary | DEPLOYED || walhall | exachk | DEPLOYED || walhall | ls | DEPLOYED || walhall | pstack | DEPLOYED || walhall | orachk | DEPLOYED |[..]| walhall | events | DEPLOYED || walhall | ps | DEPLOYED || walhall | srdc | DEPLOYED |'---------+--------------+-------------'

oracle> /u00/app/oracle/tools/tfa/bin/tfactl run alertsummary


Agenda


1. ORAchk

2. Cluster Health Monitor (CHM) (RAC only)

3. OSWatcher

4. Trace File Analyzer (TFA) Collector


OraChk

ORAchk – Purpose & History

Available since July 2011Current version 12.1.0.2.6_20160207Formerly known as "RACCheck"Supported on Unix, Linux and Windows (Cygwin/Standalone version)Checks your installation against more than 1.000 Oracle Best Practices– Audit_Checks_Report_Orachk.html contains a list of all checks– Additional user defined checks are possibleExaChk is a similar tool for Exadata


ORAchk – Not a RAC or database tool only

ORAchk includes checks for – Oracle Database (Single Instance + RAC)– MAA Validation– Upgrade Readiness– Golden Gate– Enterprise Manager 12c Cloud Control– Peoplesoft– Siebel– Oracle Sun Server


ORAchk – Interfaces

CLI tool– Daemon possibleHTML- and ZIP-outputResults can be stored in a database– è "configuration management lite"GUI – Collection Manager (APEX)– Enterprise Manager Plugin


ORAchk - Installation

Clusterware 11.2.0.4 and 12.1.0.2

– Installed with the software (into $ORACLE_HOME/suptools/orachk)

– So far not updated with the PSUs L

For older versions

– Install TFA Collector 12.1.2.3.0 or higher

– Download ORAchk via MOS 1268927.2

Recommendation:

– Use newest standalone version


ORAchk – Basic Command Line Options

Option Meaning-a Run all Checks-b Best Practice Check only-p Patch Check Only-u –o pre|post Pre or Post Upgrade Checks

-dbnames run for a subset of databases only-clusternodes run for a subset of nodes only

-h Help on all available parameters (long list)


ORAchk – Sample Output (1) – at runtime

ORAchk checks O/S, clusterware and databases on all nodes

Result: ZIP-File and HTML-Report


ORAchk – Sample Output (2) – Header


ORAchk – Sample Output (3) – Overview


ORAchk – Sample Output (4) – Details


ORAchk – Advanced Command Line Options

Option Meaning-diff Compare 2 reports

-d Manage ORAchk daemon

-profile Run for specific components or applications like:• ASM• Clusterware• EBS• MAA• Goldengate• Enterprise Manager 12c .. And more


ORAchk – Collection Manager (1)

ORAchk results can be stored in a repository database

Collection Manager is a GUI for the repository database

APEX application (4.2.0 or higher)

– Import.sql is delivered with ORAchk software

Installation

– Create database user for ORAchk

– create 3 tables (see Appendix F of the OraChk Users Guide)

– Install APEX application


ORAchk – Collection Manager (2)

Set environment

Run ORAchk

– If the environment is set, then the data will be inserted into the repository database

export RAT_UPLOAD_CONNECT_STRING="(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbserver)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=EMREP)))"export RAT_UPLOAD_TABLE=auditcheck_resultexport RAT_PATCH_UPLOAD_TABLE=auditcheck_patch_resultexport RAT_ZIP_UPLOAD_TABLE=RCA13_DOCSexport RAT_UPLOAD_USER=orachkexport RAT_UPLOAD_PASSWORD=orachkexport RAT_UPLOAD_ORACLE_HOME=/u00/app/oracle/product/11.2.0.4


ORAchk – Collection Manager (3) – some screenshots









Cluster Health Monitor

Cluster Health Monitor (CHM)

Available since Oracle Grid Infrastructure 11.2.0.2

Collects OS information of the cluster nodes– CPU load– Memory – Top Processes– File Systems– System information

Components– sysmond (on every cluster node)– loggerd

Cluster Resource crf


Cluster Health Monitor (CHM) – CLI oclumon

grid@rac1node1:~/ oclumon –hFor help in interactive mode : <verb> -hCurrently supported verbs are :dumpnodeview, manage, version, debug, analyze, quit, exit, and help

OptionDumpnodeview Shows collected data (for specific nodes and/or a specific timewindow

Manage Manages the CHM repository and showVersion Shows version informationDebug Debugs CHM componentsAnalyze Deprecated, will be ignored


Cluster Health Monitor (CHM) – CLI show data

grid@rac1node1:~/ [grid12102] oclumon dumpnodeviewdumpnodeview: Node name not given. Querying for the local host----------------------------------------Node: rac1node1 Clock: '15-02-22 18.05.43 ' SerialNo:1440----------------------------------------SYSTEM:#pcpus: 1 #vcpus: 2 cpuht: N chipname: Intel(R) cpu: 20.59 cpuq: 0 physmemfree: 393676 physmemtotal: 4958228 mcache: 2506540 swapfree: 3956548 swaptotal: 3964924 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 156 iow: 78 ios: 32 swpin: 0 swpout: 0 pgin: 155 pgout: 59 netr: 102.554 netw: 75.683 procs: 323 procsoncpu: 2 rtprocs: 13 rtprocsoncpu: N/A #fds: 20704 #sysfdlimit: 6815744 #disks: 9 #nics: 4 nicErrors: 0TOP CONSUMERS:topcpu: 'mdb_vktm_-mgmtd(5402) 4.39' topprivmem: 'java(2046) 171088' topshm: 'ora_mman_raccdb(5479) 300808' topfd: 'oraagent.bin(4891) 251' topthread: 'console-kit-dae(3254) 64'[..]


Cluster Health Monitor (CHM) – -MGMTDB (1)

In Oracle 12c CHM data is stored in the Grid Infrastructure Management Repository (GIMR), SID=-MGMTDB

– Mandatory with 12.1.0.2

– Single instance database, CDB database with 12.1.0.2 (PDB-name = clustername)

– No additional license required

– Basic installation needs about 5 GB in the diskgroup with OCR and voting files

– Additional listener MGMTLSNR

Required size depends on number of nodes and retention time

– About 1,3 GB + 500 MB/node

– Check and configure with "oclumon"


Cluster Health Monitor (CHM) – -MGMTDB (2) - Tools

mgmtca (for initial configuration only)Srvctloclumon

– Oracle recommends a retention time of 72 h ( = 259200 seconds)

grid@rac1node2:~/ oclumon manage -hManage verb usage=================

manage -repos {checkretentiontime <time> | changerepossize <memsize>} | -get {<key1> [<key2> ...] | alllogger [-details] | mylogger [-details]}..grid@rac1node2:~/ oclumon manage -repos checkretentiontime 259200The Cluster Health Monitor repository is too small for the desiredretention. Please first resize the repository to 5844 MB


Cluster Health Monitor (CHM) – EM 12c Cloud Control

CHM data can be displayed in EM 12c Cloud Control


Cluster Health Monitor (CHM) – Memory Guard

Evaluates the memory usage on the cluster nodes based on data collected by Cluster Health Monitor (CHM)

Automatically stops database services (transactional) in case of memory pressure on a cluster node

– .. or even kills database sessions

.. and automatically reactivates the services when enough memory is available

Starting with Oracle12.1.0.2 Memory Guard is automatically activated



OSWatcher

OSWatcher (1)

Collects OS statistics in the background (hourly) using OS commands

– CPU

– Memory

– Disk I/O

Installed and activated with TFA collector

Can generate graphics

OSWatcher vs. CHM

– CHM CPU overhead lower

– OSWatcher runs with user priority (CHM: Realtime)

– OSWatcher collects more information


OSWatcher (2) – Generate graphics

oracle> /u00/app/oracle/tools/tfa/bin/tfactl run oswbbStarting OSW Analyzer V7.3.1OSWatcher Analyzer Written by Oracle Center of ExpertiseCopyright (c) 2014 by Oracle CorporationParsing Data. Please Wait...Scanning file headers for version and platform info...Parsing file dbserver.markusflechtner.vm_iostat_15.02.22.0800.dat ...Parsing file dbserver.markusflechtner.vm_iostat_15.02.22.0900.dat ...[..]Parsing Completed.

Enter 1 to Display CPU Process Queue GraphsEnter 2 to Display CPU Utilization GraphsEnter 3 to Display CPU Other GraphsEnter 4 to Display Memory GraphsEnter 5 to Display Disk IO Graphs[..]Enter Q to Quit ProgramPlease Select an Option:


OSWatcher (3) - Examples


OSWatcher (4) - Housekeeping

Files are written to $TFA_HOME/repository/suptools/walhall/oswbb/oracle/archive/

No automatic housekeeping L

Create a cronjob

– Replace $TFA_HOME with your directory


find $TFA_HOME/repository/suptools/walhall/oswbb/oracle/archive -name "*.*" -mtime +10 -exec rm -f {} \;


Trace File Analyzer (TFA)Collector

Real life experience ..

26 node cluster– 5 databasesStrange ASM issueOracle Support requested – Clusterware logs – ASM alert.logs– Database alert.logs

For each of the26 servers!!


Trace File Analyzer Collector

Initial release in January 2013, current version 12.1.2.7.0 (March 2016)

Collects trace and log files and system information from all nodes into a cluster with a single command initiated on one cluster node

Integrates a lot of other tools with one single CLI

Centralized output

Real-time scanning for specific error messages possible è Automatic Collection

Included in Clusterware 11.2.0.4 and 12.1.0.2

For other versions (10.2 or higher):– Download from MOS: 1513912.1– RAC and DB Support Tools Bundle is included in current TFA package


TFA Collector – Installation

For Clusterware 11.2.0.4 and 12.1.0.2: No additional installation required

For older versions:[root@rac1node1 tmp]# ./installTFALite.shStarting TFA installationEnter a location for installing TFA [/tmp]: /u00/app/oracleChecking for available space in /u00/app/oracleEnter a Java Home that contains Java 1.6 or later : /usr/java/jre1.7.0_13Running Auto Setup for TFA as user root…Would you like to do a [L]ocal only or [C]lusterwide installation ? [L|l|C|c] [C] : CThe following installation requires temporary use of SSH.If SSH is not configured already then we will remove SSH when complete.Do you wish to Continue ? [Y|y|N|n] [N] y

Installing TFA at /u00/app/oracle in all hostsDiscovering Nodes and Oracle resourcesChecking whether CRS is up and running..


TFA Collector – Update

TFA updates are not part of the PSUs

– è TFA installed with Oracle Clusterware is not updated automatically

Manual updates

– Running TFA is detected automatically

– TFA is updated in the correct directory

Recommendation:

– Disable the version delivered with Oracle Clusterware

– Install the newest standalone version


TFA Collector – Architecture

JAVA-based tool

TFA-daemon “TFAMain” running on all cluster nodes

Data Storage

– File-Repository for Diagnostic Information

– Berkeley Database for metadata, file inventory, event history, etc.

Command Line Interface

– tfactl (perl)

– Communication with daemon using secure sockets

oracle@rac1node1:~/ [rdbms12102] ps -ef |grep tfa |grep –v greproot 2325 1 0 10:14 ? 00:00:03 /bin/sh /etc/init.d/init.tfa runroot 3631 1 0 10:16 ? 00:05:10 /u00/app/grid/product/12.1.0.2/jdk/jre/bin/java –[..] oracle.rat.tfa.TFAMain /u00/app/grid/product/12.1.0.2/tfa/rac1node1/tfa_home


TFA Collector – Commands (1) – Command Overview

oracle@rac1node1:/home/grid/ tfactlUsage : /u00/app/grid/product/12.1.0.2/bin/tfactl <command> [options]<command> =

print Print requested detailsanalyze List events summary and search strings in alert logs.diagcollect Collect logs from across nodes in clustercollection Manage TFA collectionsdirectory Add or Remove or Modify directory in TFAtoolstatus Prints the status of TFA Support Toolsrun <tool> Run the desired support toolstart <tool> Starts the desired support toolstop <tool> Stops the desired support toolrestart <tool> Restarts the desired support tool

For help with a command: /oracle/u00/app/oracle/tools/tfa/bin/tfactl <command> -help


TFA Collector – Commands (2) – commands for root

Configuration tasks must be done by root

The following additional commands are available:<command> =

start Starts TFAstop Stops TFAenable Enable TFA Auto restartdisable Disable TFA Auto restartaccess Add or Remove or List TFA Users and Groupspurge Delete collections from TFA repositorydirectory Add or Remove or Modify directory in TFAhost Add or Remove host in TFAset Turn ON/OFF or Modify various TFA featuresuninstall Uninstall TFA from this nodediagnosetfa Collect TFA Diagnostics

..


TFA Collector – Commands (3) – print config

root@rac1node1:/home/grid/ tfactl print config+--------------------------------------------+------------+| Configuration Parameter | Value |+---------------------------------------------+------------+| TFA version | 12.1.2.3.1 || Automatic diagnostic collection | OFF || Trimming of files during diagcollection | ON || Repository current size (MB) | 7 || Repository maximum size (MB) | 10240 || Inventory Trace level | 1 || Collection Trace level | 1 || Scan Trace level | 1 || Other Trace level | 1 || Max Size of TFA Log (MB) | 50 || Max Number of TFA Logs | 10 || Max Size of Core File (MB) | 20 || Max Collection Size of Core Files (MB) | 200 || Automatic Purging | ON || Minimum Age of Collections to Purge (Hours) | 12 |'---------------------------------------------+------------'


TFA Collector – Commands (4) – diagcollect (1)

Collects trace and log files from the cluster nodesoracle@green:/tmp/ [grid12102] tfactl diagcollectCollecting data for the last 4 hours for all components...Collecting data for all nodesCollection Id : 20160503103456greenRepository Location in green : /u00/app/oracle/tfa/repositoryCollection monitor will wait up to 30 seconds for collections to start2016/05/03 10:35:01 CEST : Collection Name : tfa_Tue_May_3_10_34_56_CEST_2016.zip2016/05/03 10:35:01 CEST : Sending diagcollect request to host : red2016/05/03 10:35:01 CEST : Sending diagcollect request to host : white2016/05/03 10:35:01 CEST : Scanning of files for Collection in progress...[..]Logs are being collected to: /u00/app/oracle/tfa/repository/collection_Tue_May_3_10_34_56_CEST_2016_node_all/u00/app/oracle/tfa/repository/collection_Tue_May_3_10_34_56_CEST_2016_node_all/green.tfa_Tue_May_3_10_34_56_CEST_2016.zip/u00/app/oracle/tfa/repository/collection_Tue_May_3_10_34_56_CEST_2016_node_all/white.tfa_Tue_May_3_10_34_56_CEST_2016.zip/u00/app/oracle/tfa/repository/collection_Tue_May_3_10_34_56_CEST_2016_node_all/red.tfa_Tue_May_3_10_34_56_CEST_2016.zip


TFA Collector – Commands (5) – diagcollect (2)

Which data is collected by default?– alert.log from all databases - Patch Information– ASM log files - CHM information– listener.log files - Clusterware logs

Components, node list and time window can be specified

Data is "trimmed" to the relevant time window2016/05/03 10:36:56 CEST : Total Size of all Files Checked : 3GB2016/05/03 10:36:56 CEST : Number of files containing required range : 2712016/05/03 10:36:56 CEST : Total Size of Files containing required range : 175MB2016/05/03 10:36:56 CEST : Number of files trimmed : 182016/05/03 10:36:56 CEST : Total Size of data prior to zip : 93MB2016/05/03 10:36:56 CEST : Saved 112MB by trimming files2016/05/03 10:36:56 CEST : Zip file size : 7MB


TFA Collector – Commands (6) – autodiagcollect

Automatic diagnostic collection

– Tfa scans the alert.log files and runs "diagcollect" automatically

– Collection triggered by ORA-600, ORA-7445, ORA-4031, ..

– Trimming interval +/- 600 seconds

root@rac1node1:~/ tfactl set autodiagcollect=<ON|OFF> [-c]


TFA Collector – Commands (7) - analyze

Checks system log files and Oracle log files on all nodesroot@rac1node1:~/ [grid12102] tfactl analyzeINFO: analyzing all (Alert and Unix System Logs) logs for the last 60 minutes... Please wait...INFO: analyzing host: rac1node1

Report title: Analysis of Alert,System LogsReport date range: last ~1 hour(s)

Report (default) time zone: CET - Central European TimeAnalysis started at: 21-Feb-2015 09:02:34 PM CET

[..] Message types for last ~1 hour(s)

Occurrences percent server name type----------- ------- -------------------- -----

2 66.7% rac1node1 WARNING1 33.3% rac1node1 generic

[..]


TFA Collector – other tools (1)


Command in TFACTL Explanation

Alertsummary Event summary from all alert.log files

Changes Lists changes of OS and instance configuration

Oratop "top" for Oracle Databases, Linux client required

Events List important events

Pstack Stack trace for a process (across the cluster)

Darda Diagnostic assistant, , common interface for various tools

Prw (ProcWatcher) Capture diagnostic output for perfomance issues and session hangs

(partial) list of the tools which are integrated in TFA (incl. "Support Tools Bundle"):

Please see TFA documentation for a complete list incl. documentation for each tool

TFA collector – other tools (2) – example output: events


tfactl> events

Output from host : walhall------------------------------May/02/2016 14:29:28 :

[db.emrep.EMREP] : Starting ORACLE instance (normal) (OS id: 9557)May/02/2016 14:31:17 :

[db.emrep.EMREP] : TNS-12537: TNS:connection closed[db.emrep.EMREP] : opiodr aborting process unknown ospid (15621) as a result of

ORA-609May/02/2016 14:29:28 :

[db.tvd12_site1.TVD12] : Starting ORACLE instance (normal) (OS id: 9527)May/02/2016 14:29:28 :

[db.tvd12cdb.TVD12CDB] : Starting ORACLE instance (normal) (OS id: 9537)

TFA collector – other tools (3) – example output: oratop



Summary

Summary

Oracle provides a lot of tools to keep a database in a healthy state

There are multiple ways to install the same tool

The toolset is not complete integrated in the PSU lifecycle so far

Overlapping functionality

– Healthchecks: OraChk vs. cluvfy

– System performance data: CHM vs. OSWatcher

Σ



Further Information

• Some MOS-Notes:• TFA Collector - Tool for Enhanced Diagnostic Gathering (Doc ID 1513912.1)• ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)• oratop - Utility for Near Real-time Monitoring of Databases (Doc ID 1500864.1)• SQLT Diagnostic Tool (Doc ID 215187.1)• Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware (Doc ID 459694.1)

Questions and AnswersMarkus FlechtnerPrincipal Consultant

Phone +49 211 5866 [email protected]

@markusdba http://markusdba.de

Download the slides from http://www.slideshare.net/markusdbaPlease don‘t forget the session evaluation – Thank you!

24.12.16 Tools, not only for RAC databases58

Technology

Tools, not only for Oracle RAC