® IBM Software Group © 2008 IBM Corporation Tivoli Workload Scheduler Distributed Managing Events in a Tivoli Workload Scheduler Environment

®

IBM Software Group

© 2008 IBM Corporation

Tivoli Workload Scheduler Distributed

Managing Events in a Tivoli Workload Scheduler Environment

IBM Software Group | Tivoli software


IntroductionUsing the Event-Driven Workload Automation you can:

Solve many workload automation scenarios

Perform on-demand workload automation

Carry out a predefined set of actions in response to events

Event- driven workload automation performs on-demand workload automation and plan-based job scheduling. This automation defines rules that can trigger on-demand workload automation.

The object of event-driven workload automation in Tivoli Workload Scheduler is to carry out a predefined set of actions in response to events that occur in the environment. The focus of this session will be to demonstrate how you can solve workload automation scenarios by using event-driven workload automation.

2


© 2008 IBM Corporation3

An Event-Driven Workload Automation Rule is a composition of: Events

E.g. “file x is created”, “message xyz issued in a log-file“, “Job x abended with RC=12” , “TWS agent unlinked”,

“An email is received”, “Event xyz issued on SAP systemx”

Events Correlation Conditions E.g. Filter, Events Sequence, Events Set

Actions E.g. “Submit a TWS jobStream” “Send an e-mail”, “Send An event to T/EC”, Write a message in the

message log

Event-Driven Workload Automation basic concepts

Correlation rule

Action Action

Event

Sample scenarios

Submit a Job Stream when a defined set of events is received When file /abc/file1 is FTP’ed on machine-x and at least one of the TWS job* running

on the Workstation machine-x is in successful status, then start the TWS jobstream-y

Scenario #2 – Trigger a credit transfer transaction and notify result If msg “EGS0243 Transaction A terminated” is issued in logfile /var/log1 on machine-

x, start TWS job_exec_tr1 If event “job_exec_tr1=succesfull” is received in 1 hour send an Informational email to

[email protected] otherwise send an Error email to the same user

EventEvent



The main concept of the event-driven scheduling/notification architecture is the Event Rule

The Event Rule object is a composition of: Events

e.g. “The file x on machine y is created”

Events Correlation Conditions e.g. “event_x and event_y are received in a specific or random order”,

Actions e.g “submit job_x on machine_y”,

“send an email to [email protected]”,

The Event Rule can be associated to a:

Validity time interval

Status: draft or complete

Event Rule Concepts

Correlation rule

Action Action

Event Event Event





Event Rule modeling

The supported Events are:

File created, file modification completed, file deleted,log message written

A message is issued on a log-file (e.g. may be an application log file)

TWS specific monitors Job/jobstream status changed and return code condition Status of TWS agents (linked/unlinked, started/stopped) Application server status changed Prompt status changed

Generic event. May also be an event generated via a Command Line Interface that will be provided by TWS

and that can be used by customer to generate custom events

SAP batch related events (using TWS 8.4 for Applications)

Correlation rule

Action Action

Event Event Event



Event Rule modeling

The supported Actions are:

Submit a TWS Job, submit a TWS Job Stream, submit ad-hoc TWS Job

Answer a TWS Prompt

Send an e-mail

Send an EIF event to T/EC, ITM or NetCool

Write a message in the message log

Correlation rule

Action Action

Event Event Event



Starts the monitoring activity

Activates the events receiving

Activates the correlation and actions triggering

Activation & Deploy

Listen &

Correlate

Events

EDWA life cycle

Event-Rules: the Life Cycle big picture

Create, Modify, Delete, List Modeling

Query and browse event rule instances

Drill down on actions executed for a specific event rule

Monitoring & Management

Jobs status notification

send Email

send T/EC event

Ah-Hoc Job

submission

PerformActions

Events, Correlation Rule,

Actions

Event Rule(not built yet)Rule

Editor

Create

Deploys

Rules

Builder

Event-Rules

Engine

Builds

Monitors

Deploy (Manual/

Scheduled)

Event Rule (built)



User Interfaces Can be done using the Tivoli Dynamic

Workload Console (aka TWS Web UI) and the TWS Command Line Interface (composer)

Documented APIs are provided Modeling

Activation

&

Deploy

It can be automated using the option manager (optman)

A new customization of the planman CLI is provided for manual or scheduled activation

Monitoring

&

Management

Can be done both using the Tivoli Dynamic Workload Console (aka TWS WebUI) and the TWS Command Line Interface (aka TWS composer CLI)

Documented APIs are provided



Creating an event rule: General information

10

While in draft status, an event rule is not deployed in Tivoli Workload Scheduler. When draft status is changed to complete, the event rule is compiled and deployed.



Adding events



Actions

Tivoli Workload Scheduler action:

Submit predefined job, job stream or ad hoc jobReply to prompt

Notification action:Send an e-mailForward an event to Tivoli Enterprise ConsoleLog a message to file

Custom action:Run custom commands on the event processor server



Specifying actions



Scenario #1 – Simple notification

You define the following Event Rule When jobs ABC*#* terminates in error, then send an e-mail to the [email protected]. In the

subject and the body of the e-mail the text includes the key of the job ended in error.

The Event Rule is valid from Dec 1st to Dec 31th, in the time-window 18:00-22:00 EST

You save the Event Rule that is automatically activated by the TWS

TWS starts monitoring all the specified jobs, and as soon as one of the specified jobs ends in error, it send the notification e-mail

You receive the e-mail and check the status of the jobs.

Possible customer scenarios



Scenario #2 – Trigger TWS Agents status

You define the following Event Rule When any the Workstation CPU1 is unlinked and the Workstation CPU1 is not linked back within

10 minutes, send an Error email to [email protected]


TWS starts monitoring the status of the Workstation CPU1

As soon as the Workstation status is unlinked and the TWS starts the 10 minutes timeout

If the event “CPU1 linked” is not received in 10 minutes, TWS sends the Error email

Bill Miller receives the Error email, goes to the TWS Event Rule management interface, queries the actions/rules that was triggered in the last 10 minutes and from there navigates to the CPU1 instance where can perform a first problem analysis




Scenario #3 – Submit a Job Stream when FTP completed

You define the following Event Rule When a file is created in directory /abc on machine_1 and the same file is no more modified, then

start the TWS jobstream-y

The Event Rule is valid from Dec 1st to Dec 31th, in the time-window 18:00-22:00 EST


TWS starts monitoring file /abc/file1 on machine_1, and as soon as the file is created and it is not anymore in use, it submits the jobstream-y

You check if the Event Rule was triggered in the last two hours and see its status




Scenario #4 – Trigger long duration Jobs based on timeout

You define the following Event Rule When the event “job-x=exec” and the event “job-x=succ/abend” are received in 5 minutes TWS

should reply Yes to prompt-1 and starts the jobstream-z otherwise it should send an Informational email to the [email protected] alerting for a Job in late

You save the Event Rule in draft status. After few days the user reedits the rule, changes the email recipient and save it as not-draft

The Event Rule is automatically activated

Once the job-x status is exec, TWS starts the 5 minutes timeout

If the event “job-y successful or abend” is not received in 5 minutes, TWS sends the Informational email otherwise it replies Yes to prompt-1 and submits the jobstream-z




FileMonitor Events Workflow

1. Create the rule with TWS

2. TWS creates the cfg file

3. CFG file is deployed to SSM

4. SSM reads the file and monitors the system

5. When the event is matched the SSM generates a trap

6. EIF listens to the trap and sends event to TWS

7. TWS executes the action correlated with rule

Add Add FileMonitor FileMonitor rulerule

Create .cfg Create .cfg file for SSM file for SSM agentagent

CFG fileCFG file

SSM reads SSM reads CFG and CFG and monitors the monitors the systemsystem

EIF reads EIF reads trap and trap and sends event sends event to TWSto TWS

no

yesEvent is Event is

matched?matched?



Composer new features

Is possible to specify in “composer new” command which kind of object you want to add.

For example:

-composer new eventRule

Composer is able to manage eventRule objects with commands:

create – extract

delete

display

list - print

lock - unlock

modify

new

rename eventRule definition is completely written in XML unlikeother TWS objects written in Scheduling Language

Local encoding is used by default For XML encoding



Planman command line: event rule activation

Is possible to perform the build and deployment of the event rules using directly the planman CLI:

planman deploy [-scratch]

where:

-scratch option builds again all rules defined in the database.

This is an example of the output running “planman deploy”.



Planman command line : event rule activation

planman command line activates all valid event rules and deploys the monitoring configuration to the specific agents.

The event rules that are not in draft status and are in the valid time interval are periodically (the frequency interval can be customized) checked for changes by TWS that automatically builds or refreshes the internal correlation and monitoring information for those rules.

To get this result a new optman keyword has been created:

deploymentFrequency / df = 5

Note: The default value is 5 minutes.



Monitor triggered actions



Monitoring Events



Conman new features (1/2)

The enhancements are provided by new commands that you can use to:

Download the latest monitoring configuration for the event monitoring engine on the

workstation, using deployconf command (permission to “start” action on the specified

workstation is required in the security file):

deployconf [domain!]workstation

Manage the monitoring process using startmon and stopmon commands (permissions

to “start” and “stop” actions on the specified workstation are required in the security

file):

startmon [domain!]workstation [;noask]

stopmon [domain!]workstation [;wait] [;noask]



Conman new features (2/2)

Start and stop the WebSphere Application Server process using startappserver and

stopappserver commands (permissions to “start” and “stop” actions on the specified

workstation are required in the security file):

startappserver [domain!]workstation [;wait]

stopappserver [domain!]workstation [;wait]

Manage the new event processing server used in event-driven workload automation

using starteventprocessor, stopeventprocessor and switcheventprocessor commands

(permissions to “start” and “stop” actions on the specified workstation are required in

the security file):

startevtproc [domain!]workstation

stopevtproc [domain!]workstation

switchevtproc [domain!]workstation

®

IBM Software Group


EDWA Using the TDWC Web Console

Creating or editing an Event Rule



Event Rule Editor: Overview

Common attributes correlation

Timeout conditionsEvent correlation types

Draft/Complete status

Validity interval

Timeout actions

Events

Actions

The Event Rule Editor is the tool you use to create and edit Event Rules

You open the Rule Editor directly from the TDWC portfolio: - Workload Definition - Event Management - New Event Rule

Or you open it by selecting an Event Rule from a query table and click “Edit”

Engine on which the Rule is created/edited



Event Rule Editor: OverviewThe Event Rule Editor connects to a specific engine to store or retrieve the Event Rule.

As soon as you select an engine, the Editor connects to the engine and loads the list of event and action plug-ins available on it.

When the connection is successful, the Editor opens in its entirety.The Event Rule, whether existing or a new one, is locked on the TWS database and is kept locked until the Editor is “Closed”.

You can then save the progresses on the Rule multiple times while you are working on it. The Rule will be kept locked all the time.

Select an Engine Connection and click Go



Event Rule Editor: General InformationThis section contains general information about the Event Rule and its validity period.

A “Draft” Rule will not be activated or deployed, so it will not generate events and will not run actions.

A “Non Draft”, or “Complete” Rule will be activated and deployed at next “Planman –deploy” (either scheduled or manual) and will be active within itsValidity Period.

The changes you make to a Rule (including changing the “Draft” flag) will be effective only after the next deploying of the Rule.

The Rule will be valid every day between the specified dates, but only within the hours specified here.

An empty value for one or both extremes means midnight.

A Daily end prior to Daily start is a valid range and represents a range that crosses midnight. In this case refer to the manuals for what are the effects on the first and last validity day.

The Rule will be valid between the specified dates.

An empty value for one or both extremes means indefinite.

Dates and times expressed here will be referred to the TWS engine Time zone unless otherwise specified here



Event Rule Editor: Events

Select the type of correlation: Single event, Unordered Set of events, Ordered Sequence of events

Click here to change

Add an Event by clicking here

Complete Information available clicking here

To expand/collapse a section click the title bar

Remove an Event by clicking here



Event Rule Editor: Timeout and Correlation

Enable Timeout and specify a value here

Select properties common to all events here.They must all match for the rule to be satisfied



Event Properties specify a filtering condition for events.

A condition on a property is specified with a comparison operator and one or more values. The values are in OR condition.Wildcards may or not be allowed.

Filtering on some properties may be mandatory. You can add optional properties from the drop-down list.

A property can be added multiple times. These are in AND condition.

Event Rule Editor: Event Properties

Add optional properties from this list

Wildcards are allowed for this property

A non-empty value is required

Specify a comparison operator

Select an Event by clicking it to specify its properties

The Properties are displayed below

Multiple, semi-colon-separated, values not allowed

Remove Optional properties clicking here

A name is generated and assigned to each Event automatically. You can

change it here



Event Rule Editor: Actions and Variables

A variable chooser helps inserting variables.

Variables are used to take values from Event properties to use into Action properties

You can use Variables for any Action Property

Use Machine-readable format to use variables as input to scripts, programs, etc…

Select an Event from the list of events currently added to the rule, and then one of its Properties.

A Variable pointing to the selected property will be inserted at the end of the current input field

Actions Properties do not specify a comparison operator, because they are actually parameters for the Action

Some properties are mandatory. You can add optional properties from the drop-down list.

A property can be specified only once.



Event Rule Editor: Lookup of TWS objectsWhen specifying properties related to TWS objects, a Lookup facility helps selecting an existing object in the TWS database.

Lookup is available for Jobs, Job Streams and Workstations.

Looking-up an object will fill-in all the object properties in the Action, not only the property next to which the button appears.

Specify search criteria and press Search.

The list of results appears below.

Click on an item to select it, or change the criteria and search again.

Press the Lookup button here



Event Rule Editor: Timeout Actions

If you enabled Timeout, switch to the Timeout Actions tab to work with these

Switch by clicking here



Event Rule Editor: User Input Validation

An Event or Action box turns RED if there are outstanding errors with it.

Click the box to resolve them in the Properties section

Errors are detected while you are typing and are shown below the

corresponding field



Event Rule Editor: User Input Validation

Errors related to invalid ranges are outlined with a box around the

corresponding fields



Event Rule lifecycle management Event Rules

Associate Event filters with response and timeout Actions

Can be listed using a query task

On the resulting table, you can drill-down to see a rule’s properties in read-only, and perform administrative actions on it

®

IBM Software Group


Event Monitoring using the TDWC Web Console



Event Rule monitoring Event Rule Instances

Represent event rules that have been matched An Event Rule Instance includes information like event rule name, date/time when it

was matched and the list of actions that were triggered, whether the rule was satisfied or timed-out and other

An Event-Rule Instance is created each time the event conditions contained in the an event-rule are matched or you have defined timeout actions and the rule times-out



Triggered Actions Represent actions that have been run as result of an event rule being matched A triggered action instance includes information like action type, action name, date/time when it

was executed and if it was executed successfully or not A triggered action instance is created for each action that is executed when an event rule is

matched

Event Rule monitoring



Operator Messages An Operator Message is a particular type of action instance that includes a textual field

containing the message text Operator Messages represent another view on the Triggered Actions list, specific for actions of

the “Log Message” type This view shows each message that was logged as result of a Log Message action with a more

event console-like look&feel You will be able to see for each Operator Message a corresponding record also in the Triggered

Actions view




Workstations The “workstations in plan” view has been enhanced to show event monitoring information You can see whether a workstation is the Event Processor Server You can see whether the SSM Monitoring Agent is running You can see whether the workstation has received the latest monitoring configuration You can perform administrative actions on the selected workstation(s)


®

IBM Software Group


Troubleshooting EDWA



Global Options General options for EDWA:

enEventDrivenWorkloadAutomation (ed)enables or disables the event driven workload automation featuredefault = YES (EDWA is enabled)

deploymentFrequency (df)specifies how often event rule updates are applied on the event processor and deployed to the TWS agentsdefault = 5 (minutes)

enEventProcessorHttpsProtocol (eh)enables or disables use of the HTTPS protocol to connect to the event processor serverdefault = YES (HTTPS is enabled)

eventProcessorEIFPort (ee)port of the Tivoli Event Integration Facility (EIF) for communication with the event processor serverdefault = 31131

logHistory (lh)number of days for which history of triggered rule instances, action runs and logged messages are kept in the TWS databasedefault = 10

logCleanupFrequency (lc)specifies how often to look for history data older than logHistory days, which must be deleteddefault = 5 (minutes)



Global Options Options used by MailSender action plugin:

smtpServerName (sn)hostname or IP address of the SMTP server through which outgoing e-mails are delivereddefault = localhost

smtpServerPort (sp)port of the SMTP server through which outgoing e-mails are delivered default = 25

smtpUseSSL (us)enables or disables use of Secure Sockets Layer (SSL) to secure the connection to the SMTP server default = NO (SSL is disabled)

smtpUseTLS (tl)enables or disables use of Transport Layer Security (TLS) to secure the connection to the SMTP server default = NO (TLS is disabled)

smtpUseAuthentication (ua)enables or disables user/password authentication against the SMTP serverdefault = NO (authentication is disabled)

smtpUserName (un)name of the user to be used to authenticate with the SMTP server (if authetication is enabled)default = name of the tws_user

smtpUserPassword (up)password of the user to be used to authenticate with the SMTP server (if authetication is enabled)default = not set

mailSenderName (ms)name of the mail sender to be used for e-mails sent by TWSdefault = TWS



Local Options Options added to the localopts file for EDWA:

AUTOSTART MONMAN Tells whether or not the monman process for monitoring of generated events

should be automatically started at the startup of this CPU

the monman process can be also controlled manually by using the “conman startmon” and “conman stopmon” commands

default is YES

CAN BE EVENT PROCESSOR Tells whether or not this CPU can take responsibility of being an event processor

(e.g. as a result of a “conman switcheventprocessor” command)

only the master domain manager and the backup master can be event processors

default is YES for master DM and backup master DM, NO for other CPUs



Troubleshooting/Diagnostics (EDWA)Scenario: An event rule does not trigger the required action.

Check if the event management feature is enabled Use the optman command line to verify :

enEventDrivenWorkloadAutomation / ed = YES

Action: If the property is set to NO, run the command: optman chg ed=YES and run JnextPlan –for 0000

Check if the event processor is installed, up and running, and correctly configured

Run conman showcpus

CPUID RUN NODE LIMIT FENCE DATE TIME STATE METHOD DOMAIN CPU_MASTER 11 *WNT MASTER 0 0 09/03/07 09:51 I JW MDEA

MASTERDM FTA1 11 WNT FTA 0 0 LT MASTERDM



The STATE field has a lower-case eIf the STATE field has a lower case e, the event processor is installed but not running. Action: Start the event processor by conman:

conman startevtproc

The STATE field has no MIf the STATE field has no M, monman is not running. Action: Start monman process by conman:conman startmon

The STATE field has no DIf the STATE field has no D, the current monitoring package configuration is not deployed. Action: Force the deployment by conman:

conman deploy

Note: If there is neither an upper-case E nor a lower-case e, the event processor is not installed. The event processor is installed by default on the master domain manager and backup master domain manager. If you are working on either, then the installation did not complete correctly.

Troubleshooting/Diagnostics (EDWA)



Check if the rule has been added to the monitoring configuration on the workstation.

Check if the rule is present in the workstation monitoring configuration by running the command:conman <workstation>;getmon

Monitoring configuration for CPU_MASTER: ******************************************************* Package date : 2007/09/03 07:48 GMT *** ****************************************************TEST1::FileMonitor#FileCreated:C:\TEMP\FILE5.TXT ON CPU_MASTER; TEST1::TWSObjectsMonitor#JobSubmit:* # * . TEST*;

Check in the <TWS-home>/monconf if the configuration is present

Check if the rule is active:

Use Web UI interface or composer command line to verify the state of the rule: For example by composer

-list er=@ Event Rule Name Type Draft Status Updated On Locked By ------------------ --------- ----- ------- ---------- --------------

TEST1 filter N active 09/03/2007 –




Check if the event is correctly received: Check in the SystemOut of the server whether the event has been received. The output is different, depending on the type of event: FileMonitorPlugIn event

a. This is an example of the output of a FileMonitorPlugIn event

event: [9/3/07 9:55:05:078 CEST] 00000035 EventProcessor A com.ibm.tws.event.EventProcessorManager processEvent(IEvent) AWSEVP001I The

following event has been received: event type = "FILECREATED"; event provider = "FileMonitor"; event scope = "c:\temp\file5.txt on CPU_MASTER". FILECREATED FileMonitor c:\temp\file5.txt on CPU_MASTER

Additional check: If the event has not been received check if it has been created by looking in the <TWS_home>\ssm\Log\traps.log for the message that indicates that the event has been created:

.1.3.6.1.4.1.1977.47.1.1.4.25 OCTET STRING FileCreatedEvent event

b. This is an example of the output of a TWSObjectMonitorPlugIn event

event: [9/3/07 12:28:38:843 CEST] 00000042 EventProcesso A com.ibm.tws.event.EventProcessorManager processEvent(IEvent) AWSEVP001I The following event has been received: event type = "JOBSUBMIT"; event provider = ""TWSObjectsMonitor""; event scope = "CPU_MASTER # JOBS . (CPU_MASTER #) TEST". JOBSUBMIT "TWSObjectsMonitor" CPU_MASTER # JOBS . (CPU_MASTER #) TEST

Additional check :If the TWSObjectMonitorPlugIn event has been received, check in the same log that the EIF event has been sent.




Scenario: Actions involving the automatic sending of an e-mail fail An event rule is created, including as the required action the sending of an e-mail.

When the event occurs, the action fails with the following message:

AWSMSP104E The mail "<mailID>" has not been successfully delivered to "<recipient>".

Action: Specify the domain name of the SMTP server (mailSenderName (ms))

Example: optman chg ms [email protected]

Scenario: An event is lost If the event queue is not big enough, it is possible that the most recent event or events are

missing.

Action: increase the size of the event processor queue.

1. At the workstation running the event processor, locate the file: <TWS_home>/appserver/profiles/twsprofile/temp/TWS/EIFListener/eif.templ

2. Edit the file and locate the keyword: BufEvtMaxSize

3. Increase the value of this keyword, according to your requirements.

4. Stop and restart the WebSphere Application Server using the conman stopappserver and conman startappserver commands




Where to get help Event Management Problems

The Tivoli Workload Scheduler, Version 8.5.1 Event management problems Troubleshooting Guide contains a section, which describes problems that might occur with processing of events. The topics are as follows: Troubleshooting an event rule that do

Troubleshooting an event rule that does not trigger the required action

Actions involving the automatic sending of an e-mail fail

An event is lost

Event rules not deployed after switching event processor

Event LogMessageWritten is not triggered

Deploy (D) flag not set after ResetPlan command used

Missing or empty event monitoring configuration file

Events not processed in correct order

The stopeventprocessor or switcheventprocessor commands do not work

Information Center http://www.ibm.com/developerworks/wikis/display/tivolidoccentral/Tivoli+Workload+Scheduler

53

Documents

® IBM Software Group © 2008 IBM Corporation Tivoli Workload Scheduler Distributed Managing Events in a Tivoli Workload Scheduler Environment