Monitoring system performance and health of i CEC 2012

IBM Power Systems

© 2012 IBM Corporation

Monitoring System Performance and Health of IBM i

Dawn May - [email protected]

© 2012 IBM Corporation2

IBM Power Systems

Goals Do you need to understand system performance in real-time? Do you need to know who is consuming the system CPU? Do you want to notify an operator when a partition is not performing as

expected? Do you want to know immediately when an inquiry message is sent or when

a critical job ends?

This session will show you how to automate monitoring so you can focus on other aspects of your job.

By the end of this session, you will be able to:– Create and use management central monitors– Understand monitoring with IBM Systems Director– Understand Health Indicators with IBM Systems Director Navigator


IBM Power Systems

Agenda

Three Toolsets

Monitoring with Management Central

Monitoring with IBM Systems Director

Performance Health Indicators


IBM Power Systems

Monitoring Interfaces

Three different management tools

– System i Navigator• Windows client application

“iSeries Navigator”, “Operations Navigator”• Management Central Monitors

– IBM Systems Director• Cross-platform systems management solution• Platform management, HW alerts• Real-time performance monitoring

– IBM Systems Director Navigator• Browser-based interface to manage a single IBM i partition• IBM i Performance Tasks - Performance Data Investigator• Health Indicators


IBM Power Systems

Notes: Monitoring Interfaces - where to get them? System i Navigator, is included with IBM iTM at no additional cost. The IBM i function is integrated into the base of the operating

system. The client function is shipped as part of System i Access for Windows.

Management Central is a technology integrated into System i Navigator and is not directly installed. When installing System i Access for Windows, choose 'Custom Install'. Expand the System i Navigator option tree and select the appropriate components such as Monitors, Commands.....

The general rule for Management Central functions for connectivity is that N-2 and N+2 releases are supported. However, for the best performance and most functions available it is strongly recommended that your IBM System i Navigator and your Central System be at the highest release you have available.

Your endpoint systems can then be at a mix of previous releases.

Also, note that the functions available to you are only as current as the client and Central System combination. (i.e. if System i Navigator is at 6.1 and the Central System is at 5.4, only 5.4 functions will be available). Also, in almost all cases, function new in a certain release can’t be run to endpoints at older releases.

IBM Systems Director Navigator is the Web console available beginning with IBM i 6.1. It is included with IBM i; no installation is necessary, all you need in a Web browser.

IBM Systems Director is a browser-based management tool. You need to download and install the management server (it doesn't run on i) as well as installing the agents required on each endpoint. For monitoring on i, you need to install the 5770-UME licensed program product and 5733-SC1, Option 1 (SSH) on each IBM i partition you want to monitor.IBM Systems Director can be downloaded from http://www-03.ibm.com/systems/software/director/downloads/index.html


IBM Power Systems

Management Central Monitors


IBM Power Systems

Know your Tools

IBM System i Navigator for Windows -- monitoring-- historical trending

Central System

Endpoint Systems

System Group

Firewall and I nternet

Web application

server

IBM Systems Director Navigator for i – Performance tasks


IBM Power Systems Notes - Terminology

IBM System i Navigator provides a graphical user interface to IBM i® . It comes in 2 complimentary options: Windows and Web.System i Navigator for Windows is in installed on the PC so the user can have a rich graphical interface to interact with their systems.System i Navigator tasks on the Web perform a subset of Navigator tasks through an Internet Web browser. These are URL-addressable links only.

IBM Systems Director Navigator for i is the browser-based console that has much of the function of System i Navigator; monitors however, are not available through this interface. The Performance tasks is only available through this Web console.

IBM® i® integrated Web application server - The i integrated Web application server (5761-SS1) integrates an OSGi-based Web-servlet container with the i operating system. (5.4 and later)

Central System helps to manager your other systems (called endpoints) and stores most management information (inventory, command, package, product, and user definitions, etc).

Endpoints are the systems which your PC does not need to be in direct contact with in order to "manage".

Source System is the system from which objects, files and information are sent using Management Central's send tasks. The Source System is the source of the objects, files and information being sent.

Model System has all and only desired fixes installed or has all system values set properly for the targets.

Target System is where objects, files and information are sent within Management Central's send tasks. The Target Systems are the destinations of the objects, files and information being sent. Target Systems (and more generally, endpoint systems) are often grouped into System Groups.


IBM Power Systems

Real-time performance graphs with System Monitors


IBM Power Systems

System Monitors

Monitor system performance– Predefined metrics

● CPU Utilization, Disk Arm Utilization...

Enabled with...– Event logging– Trigger/Reset notification– CL Command Automation– System Actions

● Job actions (hold, release...)


IBM Power Systems

System MonitorsSelect 'New Monitor...' and specify General properties


IBM Power Systems

Notes: Select 'New Monitor...' & specify General properties

You need to name your monitor – specific names work well. You can select one or multiple metrics in a single monitor

Question: How many metrics do you put in a single monitor? Answer: It depends 1. do you like to see all monitors on a single screen? 2. do you prefer to have more granular monitor notifications

There is no limit on the number of endpoint systems that a monitor can be started on. However you do get into usability issues when displaying the graph, too many systems on a graph and it might get to difficult to view.

General - The General page for New Monitor or Monitor Properties allows you to view and change general information about the monitor. The general information includes the name of the monitor, a brief description of the monitor.

Name - The unique name of the monitor. You can change the name, using up to 64 characters for the new name. Do not use any of the following characters: asterisk (*), backslash (\), colon (:), greater than (>), less than (<), question mark (?), quotation mark (“), slash (/), or vertical bar (|).

Description - A brief description to help you identify this monitor in a list of monitors. You can change the description, using up to 64 characters for the new description.


IBM Power Systems

System Monitors

What

Vertical axis

How often

Horizontal axis

Select the 'Metrics to monitor‘, when done press OK to create


IBM Power Systems Notes: Define A Monitor

The Metrics page for New Monitor or Monitor Properties allows you to select the metrics that you want to monitor. You can view and change information about the collection interval, the maximum graphing value, and the display time for each metric. You can also click Threshold 1 or Threshold 2 to specify information about the thresholds for each metric.

Metrics is the piece of information to collect. Possible values are:CPU Utilization (Average) Communications IOP Utilization (Average)CPU Utilization (Interactive Jobs) Communications IOP Utilization (Maximum)CPU Utilization (Interactive Feature) Communications Line Utilization (Average)CPU Utilization Basic (Average) Communications Line Utilization (Maximum)CPU Utilization (Secondary Workloads) LAN Utilization (Average)CPU Utilization (Database Capability) LAN Utilization (Maximum)Interactive Response Time (Average) Machine Pool FaultsInteractive Response Time (Maximum) User Pool Faults (Average)Transaction Rate (Average) User Pool Faults (Maximum)Transaction Rate (Interactive) Disk Storage (Average) Batch Logical Database I/O Disk Storage (Maximum) Disk Arm Utilization (Average) Disk IOP Utilization (Average)Disk Arm Utilization (Maximum) Disk IOP Utilization (Maximum)

Collection Interval is the time to wait in-between each collection of data.Maximum graphing value is the highest value to be displayed on the vertical axis of the graph.Display time is how many minutes you want displayed on the horizontal axis of the graph.


IBM Power Systems

System Monitors

Condition

What

Automation

Setting Thresholds


IBM Power Systems Notes: Setting ThresholdsThreshold - A threshold is a setting for a metric that is being collected by a monitor.

This setting allows you to specify actions to be taken when a specified value (called the trigger value) is reached. You can also specify actions to be taken when a second value (called the reset value) is reached. For example, you can specify a CL command that stops any new jobs from starting when CPU utilization reaches 90% and another command that allows new jobs to start when CPU utilization falls to less than 70%. You can also choose to add an event to the Event Log whenever the trigger value or the reset value is reached. You can set up to two thresholds for each metric that the monitor is collecting. Trigger - considered bad (usually high but can be low), reset - consider good (opposite of trigger)

The two Thresholds tabs on the metrics page provide a place for you to specify whether or not you want to monitor this metric for a particular threshold. You must check the Enable threshold box before you can specify the conditions to trigger and to reset this threshold. You can also specify the action to be taken when the threshold is triggered and when it is reset. The action that you specify must be a CL command. When you click OK, this metric will be actively monitored for this threshold if the monitor is currently running. If the monitor is not currently running, this metric will be monitored for this threshold the next time the monitor is started

You can specify the following conditions and commands for Threshold trigger and for Threshold reset:Value - Specifies the condition that must be met to trigger or to reset this threshold.Duration - Specifies the number of consecutive collection intervals that the value must meet the

criterion to cause a threshold trigger or reset event. Specifying a higher number of collection intervals for Duration helps to avoid unnecessary threshold activity due to frequent spiking of values.

i command - Specifies the command to be run on the i endpoint system when the threshold is triggered or reset on that endpoint. This command can be as simple as sending a message, or as complex as submitting or calling a program.


IBM Power Systems Notes: Threshold variables

System Monitor Replacement Variables:

Parameter Passed Data&DATE The Date the monitor triggered or reset&INTVL Collection interval: How often the monitor collected data (in seconds)&MON The Monitor name&RDUR Reset duration: How many intervals does the reset value have to be met before the monitor resets.&RVAL Reset value: The value that the metric was monitoring for when the monitor reset&SEQ Sequence number: A unique, incrementing number assigned to each collection interval. Can be used in a

program to compare when triggers happened and in what sequence.&TDUR Trigger duration: How many intervals does the trigger value have to be met before the monitor triggers&TIME The time the monitor triggered or reset&TVAL Trigger value: The value that the metric was monitoring for when the monitor triggered&VAL Current value: The actual value of the metric when the monitor triggered (2)

Note: A couple of things to note about system monitor replacement parameters:- The dollar sign ($) that was available in previous releases is still supported, for example, $TIME.- The wording is a bit different on some metrics and values:- Batch I/O is shown as I/O operations rather than transactions per second.- Transaction rates are shown as transactions rather than transactions per second.- Interactive response times (both average and maximum) are shown in milliseconds rather than seconds.


IBM Power Systems

System MonitorsSelect the monitor, then the start button


IBM Power Systems Notes: Start A MonitorThe Start Monitor dialog allows you to select the endpoint systems and system groups

on which you want to start the monitor (if no endpoint systems or system groups have been previously selected for this monitor).

To add a system or group to the Selected systems and groups list, select it in the Available systems and groups list, and then click Add. If a monitor is started and then a system is added, the monitor will be started on that endpoint system automatically.

To remove a system or group from the Selected systems and groups list, select it in the list, and then click Remove. If a monitor is started and then a system is removed, the monitor will be stopped on that endpoint system automatically.

Available systems and groups - A list of endpoint systems and system groups from which you can select a system or group. Click the plus sign (+) next to any group to see the systems that are included in the group.

Monitor data is collected and stored on the endpoint system. A minimum amount of data is actually sent back to the client when viewing the graph, The more specific, detailed data is only sent to the client when the graphs are open

PC is not required to be connected once monitor is started. The graph window can also be minimized and the monitor will still be active.

The data shown in the graph is obtained from Collection Services. Collection Services houses the data in management collection objects. This data is used by system monitors, job monitors and other performance tools.


IBM Power Systems

System Monitors

Overall status

View the status


IBM Power Systems Notes: View Status for a MonitorThe Status dialog allows you to see the current status of each endpoint system and system group

associated with a monitor. The status of each system and group is updated automatically as changes occur. You can expand any group in the System or Group list to see the status of individual systems in the group. By clicking the Restart button, you can restart the monitor on any systems on which it has failed.

Overall status - The current status of the monitor. Possible values are:x thresholds triggered - The number of thresholds that are currently active for the monitor (that is, x

represents the number of thresholds that have been triggered but have not been reset). Started on x of y systems - The monitor is collecting data on x of y endpoint systems, where x represents

the number of systems where the monitor is running and y represents the number of systems where you requested to start the monitor. The monitor is in the process of starting on the remaining systems.

Started - The monitor is collecting data on all endpoint systems where you requested to start the monitor.Starting - The monitor is in the process of starting. Stopping - The monitor is in the process of stopping.Stopped - The monitor is no longer collecting data.Failed - An attempt was made to start the monitor on the specified systems or groups, but the monitor was

not started on any systems. The failure may have occurred because the systems were not running when you tried to start the monitor, or it may be because a connection was lost or a server was not started. Click Restart to try starting this monitor again.

Failed on x of y systems - The monitor has failed to start or unexpectedly stopped working on x of y systems (where x is the number of systems on which work has stopped and y is the total number of systems on which the monitor is to be run). The monitor is starting or started on the remaining systems. The failure may have occurred because the systems were not running when you tried to start the monitor, or it may be because a connection was lost or a server was not started. Click Restart to try starting this monitor again. See the System or Group Status for a list of the endpoint systems and system groups associated with the monitor and the current status of each system and group.


IBM Power Systems

System Monitors

Drill down with Actions

Threshold IndicatorsViewing the thresholds


IBM Power Systems

Properties

Active Control

Menu

Changing Thresholds


IBM Power Systems

You can change the thresholds several ways. PropertiesActive graphical controlMenu items

You can change thresholds while a monitor is started, e.g., you do not need to stop the monitor to change the thresholds. The general properties of the monitor can be accessed view the toolbar or menu items for making any changes or additions to the thresholds and values.

To change the trigger value or the reset value for a threshold using the active graphical control, place the mouse pointer on the threshold indicator. When the ToolTip indicates Trigger, hold the mouse button down and move up or down to change the trigger value. The changing values are shown in the ToolTip. When the ToolTip indicates Reset, hold the mouse button down and move up or down to change the reset value . Click any collection point on a Monitor graph line to see Details of the data associated with the collection point.

By accessing the menu items, you will taken directly to the thresholds page in properties to make any changes.There are several visual indicators when a threshold occurs:

Status in the toolbar area.Upper Left corner icon will change.Line in the graph will change to red.Metric graph title will change to red with icon indicator

Notes: Changing Thresholds


IBM Power Systems

IBM i

PC Client

Threshold Actions


IBM Power Systems

The Actions page for Monitor Properties allows you to specify the actions to occur when a threshold is triggered and when a threshold is reset which apply to all metrics.

Log event - Adds an entry to the Event Log on the central system indicating that the threshold was triggered. The entry also includes the date and time the event occurred, the endpoint system being monitored, the metric being collected, and the monitor that logged the event.

Open Event Log - Displays the Event Log, which is a list of threshold trigger and reset events that have occurred.

Open monitor - Displays a graphical view of the metrics as they are being collected.

Sound alarm - Sounds an alarm on the PC.

Threshold commands will be run under the monitor's owner's user profile.

When a threshold gets triggered/reset, your PC client does not need to be up and running to run the Operating System command. However, if it is not up the corresponding PC action will not happen.

Notes: Threshold Actions


IBM Power Systems

Viewing Events From Thresholds


IBM Power Systems

The Event Log window displays a list of threshold trigger and reset events for all of your monitors. You can specify on the Properties page for each monitor whether or not you want events added to the Event Log. To see the Properties page for any monitor, select the monitor in the Monitors window and then select Properties from the File menu. The list of events is arranged in order by date and time by default, but you can change the order by clicking on any column heading. For example, to sort the list by the endpoint system where the event occurred, click on System.

An icon to the left of each event indicates the type of event::A red circle with white x - indicates that this event is a trigger event for which you did not specify a host command to be run when the threshold was triggered.A yellow circle with red x - Indicates that this event is a trigger event for which you specified a host command to be run when the threshold was triggered.A white check with a check mark -indicates that this event is a threshold reset event.

You can customize the list of events to include only those that meet specific criteria by selecting Options from the menu bar and then selecting Include. You can have more than one Event Log window open at the same time, and you can work with other windows while the Event Log windows are open. Event Log windows are updated continuously as events occur.

Notes: Event Logs


IBM Power Systems

Customize Event Log Information


IBM Power Systems

Options menu choices

Click Options on the menu bar to display the actions you can perform to change what information is displayed. The possible choices are:

Include...

Displays the Include dialog, which allows you to specify which events you want to display in the list.

Columns...

Displays the Columns dialog, which allows you to specify which columns of information you want to display in the list. You can also specify the order in which you want the columns to be displayed.

Notes: Options – Include


IBM Power Systems

Event Properties - Trigger (reset similar)


IBM Power Systems

The Trigger/Reset page for Event Properties allows you to view additional information about the event. This information includes the value, the duration, the Operating System command and the sequence number of the event.

Trigger/Reset value - The value specified in the monitor properties. Actual value - The actual value that exceeded the trigger value and caused the trigger event.Duration - The number of collection intervals specified for the duration in the monitor properties. Operating System command - The command that was run on the endpoint system when the event occurred.

The General page for Event Properties allows you to view general information about the event. The general information includes the type of event (trigger or reset), the date and time the event occurred, the endpoint system that the event occurred on, the metric that was being collected, and the name of the monitor that logged the event. For more information, select the following:

Event type SystemDate TimeMonitor Metric

Notes: Event Properties


IBM Power Systems

Management Central Job Monitors

Monitor the system by selecting ...– Criteria to subset data

● Job criteria - subsystem, job name, job type or user● Server name - web server, ftp server...

– Predefined metrics● Job Count, Thread count, CPU Utilization, …


● Job actions such hold, release...


IBM Power Systems

Job MonitorsSelect 'New Monitor...' and specify General properties


IBM Power Systems

Job Monitors

Different Metrics Types Levels

Problem Condition

Automation

Select 'Metrics to monitor' and thresholds then press OK to create


IBM Power Systems

Notes: Define A Monitor

The Metrics page for New Monitor or Monitor Properties allows you to select the metrics that you want to monitor. You can view and change information for each metric. You can also click Threshold 1 or Threshold 2 to specify information about the thresholds for each metric.

Metrics are the pieces of information to collect. Possible values are:Job Count, Job Log Message and Job StatusJob Numeric Values:CPU Percent UtilizationLogical I/O RateDisk I/O RateCommunications I/O RateTransaction RateTransaction TimeThread CountPage Fault RateSummary Numeric Values (same a job level)


IBM Power Systems

Notes: Threshold variables

Job Monitor Replacement Variables:

Parameter Passed Data &DATE The Date the monitor triggered or reset&INTVL Collection interval: How often the monitor collected data (in seconds)&MON The monitor name&TIME The time the monitor triggered or reset&ENDPOINT The endpoint system name&EVENTTYPE Event type: The type of trigger or reset that is happening, defined as follows:

Triggered Event = 1Auto Reset Event = 2Manual Reset Event = 3

&JOBNAME The job name of the job causing the trigger/reset&JOBNUMBER The job number of the job causing the trigger/reset&JOBSTATUS The job status causing a trigger/reset &JOBTYPE The job type of the job causing the trigger/reset&JOBUSER The job user of the job causing the trigger/reset


IBM Power Systems


Job Monitor Replacement Variables (continued):

Parameter Passed Data &METRICTYPE The category of the metric. For a Job monitor, the categories are as follows:

Status Metric = 10010Message Metric = 10020Numeric Metric = 10030

&METRIC Metric that has triggered/reset, defined as follows:Job CPU Utilization = 1010 Summary Comm I/O = 2040Job Logical I/O = 1020 Summary Trans. Rate = 2050Job Disk I/ = 1030 Summary Trans. Time = 2060Job Comm I/O = 1040 Summary Thread Cnt = 2070Job Transaction Rate = 1050 Summary Page Faults = 2080Job Transaction Time = 1060 Job Status = 3010Job Thread Count = 1070 Job Log Messages = 3020Job Page Faults = 1080 Summary Job Count = 4010Summary CPU Utilization = 2010Summary Logical I/O = 2020Summary Disk I/O = 2030


IBM Power Systems


Job Monitor Replacement Variables (continued):

Parameter Passed Data&NUMCURRENT Current numeric value &NUMRESET Threshold value to cause auto-reset of numeric metric

&NUMTRIGGER Threshold value to cause trigger of a numeric metric

&OWNER Monitor owner&RDUR Reset duration, in intervals, as set in the threshold&RESETTYPE Reset type and defined as follows:

Manual reset = 1Automatic reset = 2

&SBS Subsystem of the job causing the trigger/reset&SERVER Server type of the job causing the trigger/reset. Note: Not supported for summary metrics.&TDUR Trigger duration, in intervals, as set in the threshold

&THRESHOLD Threshold number causing the trigger&MSGID Message ID causing the trigger/reset &MSGSEV Message severity causing the trigger/reset

&MSGTYPE Message type causing the trigger/reset


IBM Power Systems


Invalid Job Monitor Replacement Variable Combinations:

Job Monitor substitution parameter notes: • If a monitor is triggered and the user performs a manual reset ("Reset with Commands" or "Reset Only") , there is no

substitution value for the Parm &NUMRESET, &RDUR. It will only have a value if the reset is automated.• &MSGID, &MSGSEV, or &MSGTYPE you need to be monitoring the 'Job Log Message' metric - otherwise there is no

substitution value for these. Additionally, these are only valid in the trigger and reset commands of Job Log Messages thresholds.

• &RESETTYPE only has a valid substitution value on a reset command. Constant values are used to determine whether the reset type is manual or automated.

• &EVENTTYPE is valid for all substitution and has constant values that are used to determine the type of monitor event that occurred (automated trigger, automated reset, or manual reset). In a trigger command, the value is always the trigger constant; in a reset command, it can either be the automated reset or manual reset constant.

• &TDUR, &NUMTRIGGER, and &NUMCURRENT only have valid substitution when a trigger occurs, in the trigger command.

• &NUMTRIGGER, &NUMCURRENT, and &NUMRESET only have valid substitution when a "numeric" metric is being monitored, in the trigger and reset commands of numeric metric thresholds.

• &JOBSTATUS only has valid substitution when the Job Status metric is monitored, in the trigger and reset commands of Job Status thresholds.

• Job Count metric not valid with: &JOBNAME, &JOBUSER, &JOBNUMBER, &JOBTYPE, &SBS, &SERVER, &MSGID, &MSGSEV, &MSGTYPE, AND &JOBSTATUS

• Job Log Message metric not valid with: &RDUR, &NUMRESET, &TDUR, &NUMTRIGGER, &NUMCURRENT, and &JOBSTATUS

• Job Status metric not valid with: &NUMRESET, &NUMTRIGGER, &NUMCURRENT, &MSGID, &MSGSEV, AND &MSGTYPE

• The 'Job Numeric Values' metrics of CPU Percent Utilization, Logical I/) Rate, Disk I/O Rate, Communications I/) Rate, Transaction Rate, Transaction Time, Thread Count, and Page Fault Rate are not valid with: &MSGID, &MSGSEV, &MSGTYPE AND &JOBSTATUS

• The 'Summary Numeric Values' metrics of CPU Percent Utilization, Logical I/) Rate, Disk I/O Rate, Communications I/) Rate, Transaction Rate, Transaction Time, Thread Count, and Page Fault Rate are not valid with: &JOBNAME, &JOBUSER, &JOBNUMBER, &JOBTYPE, &SBS, &SERVER &MSGID, &MSGSEV, &MSGTYPE AND &JOBSTATUS


IBM Power Systems

Job MonitorsHow often?

5,15, 30 minutes or 1 hour

Tuning options

Caution

Uses system resources!

Setting collection interval


IBM Power Systems

Notes: Job Monitors and System Resources

Job monitors connect to a QZRCSRVS job for each job that is being monitored for the Job Log Messages

and the Job Status metrics.

QZRCSRVS jobs are not Management Central jobs. They are IBM i TCP Remote Command Server jobs

that the Management Central Java server uses for calling commands and APIs. In order to process the

API calls for the Job Log Messages and Job Status metrics in a timely fashion within the job monitor’s

interval length, the APIs are called for each job concurrently at interval time.

When both metrics are specified on the same monitor, two QZRCSRVS jobs are started for each job. For

example, if 5 jobs are monitored for Job Log Messages, 5 QZRCSRVS jobs are started to support the

monitor. If 5 jobs are monitored for Job Log Messages and Job Status, then 10 QZRCSRVS jobs are started.

Thus, it is recommended that for standard systems, when you are using the Job Log Message and Job

Status metrics, you limit the number of jobs monitored on a small system to 40 jobs or less.


IBM Power Systems

Job Monitors

PC Client

Server

Data will collect without

thresholds and actions

Job, Message and File Monitors

Actions


IBM Power Systems

Job MonitorsSelect the monitor, then the start button


IBM Power Systems

Job Monitors

Restart on failed systems

Overall status

View the status


IBM Power Systems

Job Monitors

Status

Actions

DetailedInformation

Viewing the system and job information


IBM Power Systems

Message monitors can be used to view messages across systems that match monitor criteria. You can work with the messages listed in the monitor (display details, reply, and delete).

Message Monitors

Monitor the system by selecting ...– A single message queue– Message criteria

● ID, severity or type– Predefined metrics

● Count...


● Message reply, delete...


IBM Power Systems

Message MonitorsSelect 'New Monitor...' and specify General properties


IBM Power Systems

Select 'Metrics to monitor' and thresholds then press OK to create

Message Monitors

Messages

Automation

Condition


IBM Power Systems Notes: Threshold variables

Message Monitor Replacement Variables:

Parameter Passed Data&DATE Date&MON Monitor name&INTVL Collection interval length in seconds&TIME Time&ENDPOINT Endpoint system name&EVENTTYPE Event type and defined as follows:

Triggered Event = 1Manual Reset Event = 3

&FRMJOBNUMBER Job number for the job causing the triggering message&FRMJOBNAME Job name for the job causing the triggering message&FRMUSER User owning the job causing the triggering message&FRMPROGRAM Name of the program causing the triggering message &MSGKEY 4-byte message key for the message causing the trigger (as a hex string)&MSGID Message ID causing the trigger&MSGSEV Message severity causing the trigger&MSGTYPE Message type causing the trigger&MSGCOUNT Current message count (that caused the trigger)&OWNER Monitor owner&THRESHOLD Threshold number causing the trigger&TOLIB Message queue's library to which this message was sent (the library of the

queue being monitored)&TOMSGQ Message queue name to which this message was sent (the queue being

monitored)


IBM Power Systems

Message Monitors

Actions

Viewing the system and message information

Status


IBM Power Systems

Monitoring with Watches

Watches can be use to automate the actions taken when the following occur:– Message– Licensed Internal Code Log (LIC Log)– Problem Activity Log Entry (PAL entry)

Start Watch (STRWCH) command or API (QSCSWCH) End Watch (ENDWCH) command or API (QSCEWCH)

When the condition being watched occurs, your program gets control and you can take any action you want

http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/topic/rzahb/rzahb_eventfunction.htm

http://ibmsystemsmag.blogs.com/i_can/2010/01/i-can-automate-monitoring-with-watches.html


IBM Power Systems

Watches

Low Overhead– Watches are an exit– Almost no overhead until the watched condition occurs

– Your program gets control to determine what action to take

– For message watches• Can watch for messages sent to any message queue, including

QSYSOPR, History Log• Can watch for messages sent to any job log

Can specify generic job name Can specify *ALL to watch for a message to all job logs


IBM Power Systems

You can use a file monitor to notify you whenever a selected file has changed, reached a specified size, or for specified text strings.

File Monitors

Monitor the system by selecting ...– History Log (QHST) or

specific files– File criteria

● File location– Predefined metrics

● Status, size and text

Enabled with...– Event logging– Trigger/Reset notification– CL Command Automation


IBM Power Systems

File MonitorsYou can select to monitor all system log files or selected files.


IBM Power Systems

File Monitors

Metrics– Text– File Status– Size


IBM Power Systems

Automatic Reset for Message and File Monitor Triggers

Some metrics can be reset automatically after a trigger command runs– Only available for Message and File monitors


IBM Power Systems

Sharing of Monitors

Monitors can be shared

Owner– User that created the monitor

None– No one else can see it

Read-only– Others can see it

● i.e. view properties & copy it Controlled

– Other can perform actions● i.e. start and stop


IBM Power Systems

Notes: Sharing

The owner has specified one of the following levels of sharing:

NoneOther users cannot view this item.

Read-OnlyOther users can view this item and use it (but can not start or stop it). Other users can create a new item based on this one and make changes to the new one as needed. However, other users cannot delete or change this item in any way. If you are the owner of a monitor and have specified actions (such as opening the event log window or sounding an alarm on the PC), these actions occur for all users of the monitor whenever a threshold is triggered or reset. The other users cannot change these actions.

ControlledOther users can start and stop this item. Only the owner can change the level of sharing. Other users can also view this item and use it to create a new item based on this one. If you are the owner of a monitor and have specified actions (such as opening the event log window or sounding an alarm on the PC), these actions occur for all users of the monitor whenever a threshold is triggered or reset. The other users cannot change these actions. Actions are run under the profile of owner!


IBM Power Systems

Monitoring with IBM Systems Director


AIX® and PowerVM™ Workshop

Cross-platform Management

6666

VM

VM VM

VIO VIO

Microsoft Windows™

VMVMware ESX™ VM

AIX Administrator

Windows Administrator Linux

Administrator

IBM i Administrator

Managed SystemsCommon agent, Platform agent, No agent

Upward Integration modules supportingTivoli, Computer Associates, Hewlett Packard, Microsoft


IBM Power Systems

62

IBM i Prerequisites for IBM Systems Director 6.3

Agentless - SSH is required to discover IBM i– SSH, 5733-SC1 Option 1

Platform agent – IBM Universal Management Enablement– For CIM capabilities (for example, IBM i Monitors)– 5770-UME V1R3

– Operating System and UME fixes are requiredhttp://www-912.ibm.com/s_dir/slkbase.NSF/DocNumber/618224537

Common Agent (CAS)– Download from IBM Systems Director web site– Install manually on i or through the IBM Systems Director UI

http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.tbs.helps.doc/fqm0_tbs_ibm_i_endpoints.html


IBM Power Systems

63

Collection Services Prerequisites for Monitoring

Collection Services must be started

Consider the collection interval for frequency of updates through System Director

– Default is every 15 minutes

Data must be in Collection Services DB2 files to be displayed on IBM i Monitors in Systems Director

– CFGPFRCOL – CRTDBF parameter must be *YES

“No Data Available”– Symptom of possible Collection Services problem when viewing IBM i Monitors


IBM Power Systems

Performance Summary


IBM Power Systems

Monitor Your i


IBM Power Systems

Common CIM Monitors


IBM Power Systems

IBM i Monitors


IBM Power Systems

68

IBM i Monitors 34 Metrics for common monitoring scenarios


IBM Power Systems

69


IBM Power Systems

70

Create Your Own Monitor


IBM Power Systems

71

Director Agent Monitors (requires CAS agent)


IBM Power Systems

72


IBM Power Systems

73


IBM Power Systems

74

CIM MonitorsThe are many metrics that you can monitorwith CIM, but it can be difficult to figure outwhat's available.


IBM Power Systems

75

Monitor Your IOA Cache Batteries

http://ibmsystemsmag.blogs.com/i_can/2012/04/monitoring-cache-battery-status-with-ibm-systems-director-63.html


IBM Power Systems

76

IOA Cache Battery Monitor Metrics


IBM Power Systems

77

Manage Processes … aka, Work with Active Jobs(Requires CAS Agent)


IBM Power Systems

78

Monitoring IBM i Processes

Create a process monitor from the Manage Processes task

Once a process monitor is createdadditional metrics can be monitored for that processusing the Create monitortask

Requires CAS agent


IBM Power Systems

Monitor Thresholds

The process of setting threshold and enabling automation is the same regardless of the metric type

For any metric that you want to monitor, you can set thresholds

Once thresholds are set, you then can create an event filter for the event that occurs when the threshold setting is hit

Automation plans complete the set up to enable automatic notification that the threshold was hit


IBM Power Systems

Monitor Thresholds


IBM Power Systems

Monitor Thresholds


IBM Power Systems

Create Event Filters


IBM Power Systems

Event Automation Plans


IBM Power Systems


IBM Power Systems

Event Actions


IBM Power Systems

Graphs and Dashboard


IBM Power Systems

Health Summary


IBM Power Systems

Monitoring Messages with IBM Systems Director Events


IBM Power Systems

Event Filters for QSYSOPR messages


IBM Power Systems

Event Filters for QSYSOPR message


IBM Power Systems

Event Filters for QSYSOPR message


IBM Power Systems

Monitoring QSYSOPR .... Event Automation Plan

After you establish what you want to monitor by creating the event filter for IBM i messages ...

You must also create an event automation plan for the action Director is to take when those events occur

Even though these events are generated by their respective operating systems (or an optional layer that is installed on the operating system), IBM Systems Director does not process these events unless you create an event automation plan to do so.


IBM Power Systems

Monitoring System Performance with IBM Systems Director Navigator


IBM Power Systems

IBM Systems Director Navigator for iPerformance Tasks


IBM Power Systems

Health IndicatorsManually Monitor your System Performance


IBM Power Systems

Health IndicatorsCustomize Health Indicator Thresholds


IBM Power Systems

Continuing Education

A Few More Things....


IBM Power Systems

Monitor Restart Options


IBM Power Systems

Notes: Changing the Monitor Restart Options

Restarting Monitors

Monitor Restart was added to provide a way to automatically restart monitors when the Management Central servers have been interrupted. These interruptions could be as simple as the MC Central Server or MC Endpoint Server being restarted, or something more dramatic such as the temporary loss of communications between the Central Server and an Endpoint Server or a system being IPLed.

If you select to have the system automatically attempt to restart your monitors, you may also specify how long you want the central system to keep trying to restart the monitors and how often you want the system to try during that time period.

For example, if you want the system to try to restart monitors every five minutes for a period of 3 hours, you select 'Automatically restart monitors on failed systems' and then specify 180 minutes for 'How long to attempt restart' and 5 minutes for 'How often to attempt restart'.A change to this setting takes effect the next time the Management Central servers are restarted.All Monitors support the restart option.

Default behavior is OFF.

Documentation on how to automatically restart Management Central Monitors:https://www-304.ibm.com/support/entdocview.wss?uid=nas16e5b0871315547a68625729e004737ce


IBM Power Systems

References

IBM i Home Page– http://www-03.ibm.com/systems/power/software/i/

Information Center– http://publib.boulder.ibm.com/iseries/

IBM i Systems Management– www.ibm.com/systems/i/solutions/management/– www.ibm.com/systems/i/software/navigator/index.html

http://www.redbooks.ibm.com/redbooks/pdfs/sg246226.pdf

http://publib.boulder.ibm.com/iseries/

http://www-03.ibm.com/systems/i/solutions/management/

http://www-03.ibm.com/systems/i/solutions/management/

http://www.ibm.com/systems/i/software/navigator/index.html


IBM Power Systems

This document was developed for IBM offerings in the United States as of the date of publication. IBM may not make these offerings available in other countries, and the information is subject to change without notice. Consult your local IBM business contact for information on the IBM offerings available in your area.Information in this document concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive, Armonk, NY 10504-1785 USA. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied.All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions.IBM Global Financing offerings are provided through IBM Credit Corporation in the United States and other IBM subsidiaries and divisions worldwide to qualified commercial and government clients. Rates are based on a client's credit rating, financing terms, offering type, equipment type and options, and may vary by country. Other restrictions may apply. Rates and offerings are subject to change, extension or withdrawal without notice.IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies.All prices shown are IBM's United States suggested list prices and are subject to change without notice; reseller prices may vary.IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in this document may have been made on development-level systems. There is no guarantee these measurements will be the same on generally-available systems. Some measurements quoted in this document may have been estimated through extrapolation. Users of this document should verify the applicable data for their specific environment.

Revised September 26, 2006

Special notices


IBM Power Systems

IBM, the IBM logo, ibm.com AIX, AIX (logo), AIX 5L, AIX 6 (logo), AS/400, BladeCenter, Blue Gene, ClusterProven, DB2, ESCON, i5/OS, i5/OS (logo), IBM Business Partner (logo), IntelliStation, LoadLeveler, Lotus, Lotus Notes, Notes, Operating System/400, OS/400, PartnerLink, PartnerWorld, PowerPC, pSeries, Rational, RISC System/6000, RS/6000, THINK, Tivoli, Tivoli (logo), Tivoli Management Environment, WebSphere, xSeries, z/OS, zSeries, Active Memory, Balanced Warehouse, CacheFlow, Cool Blue, IBM Systems Director VMControl, pureScale, TurboCore, Chiphopper, Cloudscape, DB2 Universal Database, DS4000, DS6000, DS8000, EnergyScale, Enterprise Workload Manager, General Parallel File System, , GPFS, HACMP, HACMP/6000, HASM, IBM Systems Director Active Energy Manager, iSeries, Micro-Partitioning, POWER, PowerExecutive, PowerVM, PowerVM (logo), PowerHA, Power Architecture, Power Everywhere, Power Family, POWER Hypervisor, Power Systems, Power Systems (logo), Power Systems Software, Power Systems Software (logo), POWER2, POWER3, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER6+, POWER7, System i, System p, System p5, System Storage, System z, TME 10, Workload Partitions Manager and X-Architecture are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries.

A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.AltiVec is a trademark of Freescale Semiconductor, Inc.AMD Opteron is a trademark of Advanced Micro Devices, Inc.InfiniBand, InfiniBand Trade Association and the InfiniBand design marks are trademarks and/or service marks of the InfiniBand Trade Association. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries or both.Microsoft, Windows and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other countries or both.NetBench is a registered trademark of Ziff Davis Media in the United States, other countries or both.SPECint, SPECfp, SPECjbb, SPECweb, SPECjAppServer, SPEC OMP, SPECviewperf, SPECapc, SPEChpc, SPECjvm, SPECmail, SPECimap and SPECsfs are trademarks of the Standard Performance Evaluation Corp (SPEC).The Power Architecture and Power.org wordmarks and the Power and Power.org logos and related marks are trademarks and service marks licensed by Power.org.TPC-C and TPC-H are trademarks of the Transaction Performance Processing Council (TPPC).UNIX is a registered trademark of The Open Group in the United States, other countries or both.

Other company, product and service names may be trademarks or service marks of others.

Revised December 2, 2010

Special notices (cont.)

Technology

Monitoring system performance and health of i CEC 2012