Zen and the Art of REQADM Maintenance Steve Willoughby PMD Engineering Computing 12 April 2000 Copyright © 2000 Intel Corporation

Zen and the Art of REQADM Maintenance

Steve Willoughby

PMD Engineering Computing

12 April 2000Copyright © 2000 Intel Corporation.

What This Course Is

• Introduction to the REQADM system components for the REQADM Administrator.

• Tips for configuring and maintaining the system.

• Provides familiarity with system to help you troubleshoot your system.

• A peek ahead at REQADM 2.0, and how it’ll change what you do.

What This Course Is Not

• User-level introduction to REQADM clients for either systems administrators or customers.

• In-depth look at report generation.

• How to organize a help center.

• There are separate presentations available for each of those.

Agenda

• What is REQADM, inside and out?

• How does it interact with external services?

• How do I set up and configure REQADM?

• What standard maintenance is required?

• What can I do for special cases or problems?

• How can I automate tasks with REQMGR?

• How can I stay informed about changes?

Agenda

• What is REQADM, inside and out?– System Architecture

• Directory Layout

– Ticket States and Statuses– Overview of Clients– Server Design and Components

System Architecture

• Server stores and manages tickets.– All access to ticket data goes through server.

• Ensures reliable data, lock management, (minimal) security, sanity-checking, common interface.

• Clients provide various interfaces to server.– New interfaces added independently, don’t

require server or other client changes.– API library

ASCII files for closed tickets stored in directories of 1,000 tickets each.

Directory Layout/opt/reqadm*

bin sbinlibdb

Backup

spoolarchive

Purged tickets archived here for later retrieval (compressed archives)

queue1

queue2

Fixed

D0 D1000

Supplimental scripts and compiled programs, not “core” REQADM services.

Database files for REQADM server’s DBMS module.

Nightly cron script makes database backup copies and recovery files here.

Configuration files, *.pid files, etc.“Core” REQADM system programs (server, clients, maintenance scripts).

ASCII ticket data files stored in directories named for the REQADM queues

*default name; YMMV

Agenda




Ticket StatesNew Ticket

Assigned

Prioritized

Work in Progress

On Hold

Categorized

Resolved

Closed

Ticket Status

• REQADM rates each ticket’s status:– Invalid:

• Malformed ASCII file or other anomaly. Normally, you wouldn’t even see these, as the clients generally filter them from view.

– Proto:• A new ticket still being built by REQADM. Not

really a request yet. Again, clients tend not to show these to anyone until they’re full-blown tickets.

Ticket Status

• REQADM rates each ticket’s status:– Re-queued:

• Ticket has been moved to a different queue; this number is the old ticket number it used to have.

• Kept for historical references.

• Read-only copy; can’t be changed further.

• REQADM 2.0’s flat numbering space will render this obsolete.

Ticket Status

• REQADM rates each ticket’s status:– Resolved:

• Work completed. SLA “clock” stopped. Referred to customer for approval or denial.

– Closed:• Work completed and approved by customer.

– Pending:• “On hold” pending action by 3rd-party or customer.

SLA “clock” stopped while on hold, unless explicitly committed to a specific date and time.

Ticket Status

• REQADM rates each ticket’s status:– New:

• Submitted but not yet assigned to an owner.

– Due:• Due (by SLA or commitment) within one hour.

– Stale• Not modified within grace period (site configurable).

– Active:• Any ticket not matching any of the above.

Agenda




Overview of Clients

• REQADM includes several standard clients which provide user interfaces to the system.

• Also maintenance clients (discussed later).

• Custom clients may be written locally in C++ or Perl.

ASCII Tools

• reqadm– If ticket number is already known, can

interactively operate on ticket, doing just about anything the X interface can do.

– Batch-capable: can do all operations using command line arguments in non-interactive mode.

– Can perform multiple operations on many tickets in one step.

ASCII Tools

• reqadm-find– Displays queue list (one ticket per line).– Use to search for tickets or monitor queues.– Options to look for specific ranges of tickets.

• Phonetic subject search.

• Search by date ranges, people involved, etc.

ASCII Tools

• reqadm-create– Creates new ticket for customer.– Support may invoke on behalf of customer.– ASCII input field editing.– Spawns external text editor for multi-line fields

(user option) or uses line-at-a-time entry system.

GUI Tools

• xreqadm– Integrated tool with actively-updated queue

display and “editor” windows to work with individual tickets.

– Can perform multiple operations on a single ticket, or a single operation on multiple tickets.

– Arguably the most useful way for support personnel to use REQADM.

GUI Tools

• reqadm-create– If a $DISPLAY variable is set in the

environment, reqadm-create will attempt to shift to X mode rather than ASCII mode.

– Displays an interactive GUI form to fill out.

GUI Tools

• reqadm-prefs– Allows users (or support on behalf of a user) to

customize their personal settings.– “Look and feel” of all REQADM clients

affected.– E-mail sent to user (from whatever tickets)

customized to their own preferences.

Web Tools

• Most of the GUI functionality also available on web server (clients respond to being run as CGI rather than stand-alone).

• CGI environment seriously hinders what tools can do, so this is not the most desirable way to access REQADM, especially for support personnel.

Mail Tools

• reqadm-mail-receiver– Invoked to handle all incoming mail to

REQADM, for new tickets and notes added to old ones.

– Written in Perl, so lots of customization possible.

– Invokes the batch “reqadm” commands to do the actual work.

Report Tools

• reqadm-report– General-purpose reporting package for

REQADM.– Pulls collection of tickets from server, analyzes

them for various metrics.– Produces Structured ASCII, PostScript or

Comma-Separated ASCII outputs, suitable for printing or importing into spreadsheets and other programs.

Agenda




Server Design

• Server is the central manager of all activity within REQADM:– Storage and retrieval of tickets.

– Management of simultaneous access between users (lock management, update management).

– Tracks user customization and site configuration information.

– Interacts with external services such as CDIS and LDAP.

Server Design

...

Client Session Management layerServer Command InterpreterPrimitive Operations layer

DBMSLock

ManagerConfiguration

ManagerPersonnelManager

Individual Client Connections

Connections to External Services

Server Subsystems

• DBMS– Performs database management services.– REQADM 1.x uses this to store “overview”

data (essential attributes of tickets) for fast queries and displays. Details still stored in ASCII files.

– REQADM 2.0 will move to all ticket data being stored in the database, for increased speed, and enhanced search capabilities.

Server Subsystems

• Lock Manager– Arbitrates who gets to access tickets.– Currently supports both read (simultaneous) and

write (exclusive) locks, but most common clients expect to update ticket, so always get write locks.

• This means anyone viewing a ticket will lock out all other users from touching it.

– Locks expire after a site-configurable length of time, or the client disconnects from the server.

Server Subsystems

• Lock Manager, continued– Locks will be broken only if already expired

and another client wants the same lock.– Even if lock broken, original writer allowed to

update if the intervening locker(s) didn’t modify the ticket.

– REQADM 2.0 will introduce more advanced locking, allowing simultaneous reads/writes.

Server Subsystems

• Configuration Manager– Serves site configuration to clients. Discussed

in detail later.

• Personnel Manager– Handles user information in the database.– Serves user info to clients, allows clients and

automated processes to update database from external sources (e.g., Human Resources).

Agenda








External Services vs. REQADM

• REQADM does not operate in a vacuum.

• Gets information about customers and support personnel from external personnel information servers and/or files.

• Interacts with mail system to send and receive tickets and updates.

• Web interface interacts with web server and authentication systems.

User Identification Infrastructure

• User ID needed for tracking and security.

• Initially loaded from static files– /etc/passwd

– site “user table”

• Dynamic lookup possible if you use LDAP and/or CDIS servers, which REQADM will talk to.

RequestServer

Usertable* passwd

CDIS

LDAP

*and CDIS and stdusers files.

RequestServer

usertable passwd

CDIS

LDAP

User Identification Infrastructure

• All users are identified by “world-wide ID” number.– Any number between 1 and 134,217,727.

• (That’s 0x00000001 and 0x07ffffff.)

– Allows usernames, full names, e-mail addresses, user ID numbers, etc. to change without losing ticket tracking history.

• But how do we map a person to a WWID in the first place, and maintain that mapping?

RequestServer

usertable passwd

CDIS

LDAP

Static User ID Files

• The baseline REQADM system reads four static files to get this user information:– /etc/passwd (or passwd NIS map)

• Defines full name, user ID number and username (therefore E-mail address) for each local user.

– Site “usertable” file• Associates username with WWID number.

RequestServer

usertable passwd

CDIS

LDAP


• Usertable is a simple ASCII file, one line per user, to map WWID to UNIX username:1042:steve

1100:scaggs

WWID username

RequestServer

usertable passwd

CDIS

LDAP


• The baseline REQADM system reads four static files to get this user information:– /opt/reqadm/lib/stdusers

• Defines WWIDs (with names and mail addresses) for users not in the password file (including administrative accounts which wouldn’t be in the usertable file).

– Site “CDIS” data file• Defines personal information (phone numbers, office

location, etc.) for each WWID. Optional.

RequestServer

usertable passwd

CDIS

LDAP


• /opt/reqadm/stdusers is formatted just like usertable, but is for defining non-user IDs within REQADM. May include additional fields:

100000000:reqadmad:[email protected]:REQADM Administrator

WWID username E-mail Full name


RequestServer

usertable passwd

CDIS

LDAP

• /opt/reqadm/stdusers could be used for:– The REQADM administrator account (which

probably won’t be in CDIS)– Other “non-human” accounts which may send

or manipulate tickets– Accounts for customers or vendors who aren’t

in your password file

RequestServer

usertable passwd

CDIS

LDAP


• CDIS file is supposed to be imported on a regular basis from some corporate HR database, customer database, or other appropriate source

• Tab-delimited ASCII file with one line per user, identified by WWID number.

• REQADM only uses certain fields from this file:

RequestServer

usertable passwd

CDIS

LDAP


• 00 Full name of user• 06 E-mail address• 16 Mailstop• 19 Phone number for messages (secretary?)• 21 Office location (room/cubicle number)• 24 Pager number• 25 Office phone number

RequestServer

usertable passwd

CDIS

LDAP


• 32 WWID number

• 33 Cellular phone number

• 39 Date and time entry was last updated– In format such as mm/dd/yyyy hh:mm. See Intro.txt

for description of other legal formats.– This date field is used by REQADM to determine

whether CDIS data is more recent than REQADM’s own database (triggering an update from this file into the REQADM database).

RequestServer

usertable passwd

CDIS

LDAP


• Nightly process updates REQADM database from these files.

• This may not be timely enough, or contain all users who want to use REQADM.

• Dynamic updates of user information possible using external services:– LDAP– CDIS

RequestServer

usertable passwd

CDIS

LDAP

E-Mail Address Resolution

• When REQADM needs to identify a user based only on E-mail address (such as when processing incoming mail messages):– If address looks like “domain\username” (NT

style), we assume the local NT domains are in sync with the UNIX password file, so we drop the domain\ part and search for username:

• In REQADM’s database of known usernames

• Using a CDIS server, if one is available.

RequestServer

usertable passwd

CDIS

LDAP


• Otherwise, if the address is only a string of digits with no domain or punctuation, we assume it is a WWID number, and look it up as explained below.– We have off-site users who authenticate to the

web using their WWID number and a password, so this is how they appear to the web-based REQADM clients.

RequestServer

usertable passwd

CDIS

LDAP


• Otherwise, we check REQADM’s database of known E-mail addresses, with and without canonizing the mail address (we’ll try adding the local mail domain to unqualified addresses).

RequestServer

usertable passwd

CDIS

LDAP


• If all the above fails, we contact an LDAP directory server (if one is available) which is configured to translate mail addresses to WWID numbers for our organization.

• If no such user could be found, the client may elect to generate a temporary WWID to use for this user.

RequestServer

usertable passwd

CDIS

LDAP

WWID Lookups

• If we are given a WWID number for a user, we look them up in the system by:– If the WWID is a group number, we retrieve the

group information from the REQADM database (which comes from workgroups.conf).

– Otherwise, if the WWID isn’t found in REQADM’s database already, a CDIS server is contacted (if available) to provide supplimental info for the user, and that is stored in REQADM’s database for future use.

RequestServer

usertable passwd

CDIS

LDAP

User ID Resolution

• Most clients look at the user ID they are running under, and resolve that to a REQADM WWID number:– REQADM’s database of user ID’s is searched.– getpwuid() called to translate user ID to username.– Site “usertable” file opened and searched to get a

WWID number for that username.– CDIS is contacted to get supplimental info and a

new record is created in the database.

RequestServer

usertable passwd

CDIS

LDAP

LDAP Server

• The basic idea here is that you can set up an LDAP sever which contains all the valid E-mail addresses known within your organization, and associates them with WWID numbers.

• Useful for large organizations like Intel which have lots of users who may have accounts at multiple company sites.

RequestServer

usertable passwd

CDIS

LDAP

LDAP Server

• The LDAP server used by REQADM should accept queries of the general form:– o=Organization-name– (mail=address-being-searched-for)

• The response should include an attribute called “employeenumber” which contains the WWID for that address.

• All of the above query components are site configurable to match your server settings.

RequestServer

usertable passwd

CDIS

LDAP

CDIS Server

• This is a simple server which serves corporate personnel information in response to a query in one of the following formats:– iname– wnumber

• If the specified local username or WWID number is not known, CDIS responds:– Key ’name’ not found.

RequestServer

usertable passwd

CDIS

LDAP

CDIS Server

• If a record was found, data lines are returned:WWID= [email protected]= JOHNLname= RANDOMMailstop= X1-111MI= XOfficeLoc= X1-1-11MsgNum= 5551111

RequestServer

usertable passwd

CDIS

LDAP

CDIS Server

• Easy to implement if you have the corporate data somewhere you can get to it (on-line database, static file, etc.).

• Can be written in Perl without much effort.

• We intend to include a small sample CDIS server in REQADM 2.0’s distribution.

Mail Server Interaction

MailServer

New Ticket Receipts

Update NoticesIncoming Ticket

Response to Notice

Approval/Denial

RequestServer

DB

MailServer

New Ticket Receipts


Response to Notice

Approval/Denial

RequestServer

DB


• Tightly integrated with mail services

• “Use the tool you’re currently using”– Customer gets E-mail notice of ticket activity– Can simply “reply” to mail—reply goes back to

REQADM, is noted in ticket history.

• New commentary at top of each mail message—gives feel of casual E-mail conversation between parties.

MailServer

New Ticket Receipts


Response to Notice

Approval/Denial

RequestServer

DB


• Each client sends its own outgoing mail by contacting the SMTP server.

• Return mail is spooled and fed into the REQADM server by reqadm-mail-receiver script– Generally run every 5 minutes via cron.

Web Server Interaction

WebServer

Browser

Browser

Browser

Browser

htpasswd

NT AuthServer

RequestServer

WebServer

Browser

Browser

Browser

Browser

htpasswd

NT AuthServer

RequestServer


• Most REQADM functions available via CGI.

• Problem of authenticating user to REQADM.– What functions to allow (security)?– Whom to log as taking action on tickets?

• Easiest is to make CGI directory password-protected, and make sure all users have entries in the web server password file.

WebServer

Browser

Browser

Browser

Browser

htpasswd

NT AuthServer

RequestServer


• We assume the web username is the same as:– UNIX username,– Numeric-only WWID number (no “P” prefix),– E-mail address, or– NT-style username (domain\username)

• It’s the web server’s responsibility to authenticate the user by whatever means necessary (NT auth server, .htpasswd file, etc.)

Agenda








Installation

• We won’t go through a tutorial of the installation process here.

• But please refer to all these documents before installing:– README file– RELEASE-NOTES file

• What’s in this release, known issues, etc.

– Intro.txt file• Lots and lots of infrastructure details, etc.

Installation

• However, there are some setup parameters you need to figure out before you install.– These will be “burned in” to the REQADM binaries

when installed. Changing them will require a re-install.

– Why? Mostly because it’s difficult to get so many customers to set environment variables or dotfiles to point REQADM clients to the local server and configuration directories. So we burn that info into REQADM directly.

Installation Parameters

• REQADM server hostname and TCP port– Clients will connect sockets there to talk to server.

• Pathname to Perl 5 interpreter– Written into #! lines of Perl scripts.

• Pathname to GNU Tar utility– Written into some scripts which use tar, also used

by the install program to unpack files.


• REQADM home directory– Called “/opt/reqadm” here; server files will be

installed there.

• Pathname REQADM web images, HTML dir– Setup will install files there.

• Pathname to client directory– Clients will look here to find help files.


• Assigned site codes “A” and “B”– Together, these form a 64-bit number which clients

send to assure the server they are the correct clients for that server. (Not really air-tight security, but helps keep accidents from happening.)

– Each can be any number from 99,999,999 to 2,147,483,647. Just be consistent throughout all your installations on all platforms.


• REQADM account user ID– Numeric UNIX user ID number for the account set

aside to run the REQADM server. The setup program will chown all files to this account.

• Web server account user ID– Numeric UNIX user ID number for the account

which runs your web server. If a REQADM client is run by this account, it assumes it’s a CGI client and reacts accordingly.


• Other parameters are configured at runtime and may be changed at will.

• Caveat: web pages contain many of these runtime configuration values (e.g., your organization name and phone number). – These are static HTML files, so you must

configure these items before installing the web support. Changing these fields requires re-installing the web support.


• The fields affected are listed in Intro.txt.

• In REQADM 2.(mumble), the static pages will all be replaced by a dynamic document server, which can get these values in real time from the REQADM server, so this won’t be a problem anymore.

Runtime Configuration

• The parameters used to configure REQADM are stored in several files:– reqadm.conf– catcodes.conf– hours.conf– queues.conf– systems.conf– workgroups.conf

Configuration: reqadm.conf

• This is the master configuration file for the REQADM system.

• Values here are available to any client.

• Sets most of the policies and behavior of REQADM.


• REQADM is very touchy about mistakes:– A missing field will crash the server and/or

client which was trying to access that field.– Be very careful to follow the file’s syntax rules.– Since this controls so much of REQADM’s

internal code, be very sure you know what you are doing before changing these values.


• ASCII file, defining one “field” (i.e., parameter) per line.

• Comments begin with a “#” as the first character on the line (no whitespace allowed to the left).– “#” elsewhere on the line has no special meaning.

• Lots of comments in the file, explaining what the fields do. Pay close attention to them.


• All fields have a four-character name.– Can actually be any four bytes, including non-

printable characters (but we recommend sticking to printable ASCII).

– Case-sensitive.– REQADM will never define fields with names

beginning with “$” (e.g., “$ABC”), so you are free to use those names for custom clients which need central configuration fields.


• Text fields defined as:SUPN=Acme Widgets, Inc. Support

Field name

Field value


• Multi-line text fields defined as:CAT@=E-mail problem

CAT@=Workstation crashed

CAT@=Need software installed

Field name

Field value

It’s a multi-line (array) field if the field’s name ends with “@”


• Numeric fields defined as:LTO#=3600

Field name

Field value

It’s a numeric field if the field name ends with a “#”


• Numeric fields may be in different bases:LTO#=3600

LTO#=$3FF

LTO#=@277

LTO#=%010111111

Decimal

Hex

Octal

Binary


• This file sets policies enforced by REQADM:– Grace period before resolved tickets

automatically close w/o customer approval,– Lock timeout interval,– Definition of “stale” ticket (how old),– Account to notify for all ticket resolutions and

commitments


• This file sets identity information about your group:– Support center name and phone number,– Escalation mail address if customer not satisfied,– E-mail domains local to your group,– Return address and ID name for REQADM mail,– Your local area code


• This file sets resource location information:– Pathnames of other configuration files,– URLs of web components,– Host info for SMTP, CDIS, LDAP, etc.,– Commands for printing, paging, etc.,– Pathname to client help files,– Pathnames to static user ID files


• This file defines custom content:– HTML code to add to stock REQADM home page,

– Text of error messages to be mailed out,

– Which web graphics are used• Very skilled (and brave) admins can use these to change

the appearance of the web forms used by REQADM.

• REQADM 2.(mumble) will extend this to include GUI tools’ graphic images and colors as well.


• This file controls REQADM behavior:– Mail receiver parameters, alias names, etc.,– Category to use when merging tickets,– What queues customers can submit tickets into,– Input problem type list,– Required and/or edit-locked input fields,– Standard pre-define queue views for xreqadm


• Changes made to this file will be seen as soon as the reqadm server daemon is sent a HUP (1) signal.

• Clients may have these values cached; they will need to be restarted to load fresh values again.– REQADM 2.0 will include code to notify the

clients when these values change, so that will no longer be necessary.

Configuration: catcodes.conf

• Defines all root-cause categories available to categorize tickets.– Different than the problem type (CAT@) in

reqadm.conf, which was the simple “what kind of complaint I have” data point supplied by the customer.

– This is the more detailed “what actually caused the problem in the first place” data point supplied by the support tech resolving the ticket.


• These may be used to analyze the major issues and problem trends.

• Hierarchial list of categories; for example:– Software Problems

• Need upgrade

• Bug in application

– Workstation Problems• Dead hardware

• Misconfigured system


• File defines one category per line.– Comments as with reqadm.conf– Use “/” to show hierarchy (like a pathname):

sw = Software Products

sw/bug = Bug in application

sw/upgrade = Upgrade needed

ws = Workstation Problems

ws/dead = Dead hardware

ws/config = Mis-configured system


• File defines one category per line.– Comments as with reqadm.conf– Use “/” to show hierarchy (like a pathname):

sw = Software Products

sw/bug = Bug in application

sw/upgrade = Upgrade needed

ws = Workstation Problems

ws/dead = Dead hardware

ws/config = Mis-configured system

Category code (used internally and with command-line options

Obsolete 2nd field (can be any non-space text)

Category description (multiple words okay)


• Don’t forget to include the category defined in reqadm.conf’s CDUP field.

• Changes to this file may not be noticed until the server is restarted. This will eventually be fixed so that sending a HUP signal will suffice.

Configuration: hours.conf

• Defines the hours your center is “open”.

• Used for calculating SLA-based due dates for tickets.

• A single line containing hour numbers for “open” and “close” time, Sunday through Saturday.

• Will probably be moved into reqadm.conf at some point.

Configuration: hours.conf

0 0 8 17 8 17 8 17 … 8 17 0 0

Closed Sunday

Open Monday 8:00-17:00

Open Tuesday 8:00-17:00

Open Friday 8:00-17:00

Closed Saturday

Configuration: queues.conf

• Defines the ticket queues used by REQADM.

• File defines one queue per line.– Comments as with reqadm.conf.– First queue is the default queue for all clients.

general 005 010 040 400 0

restore 010 010 010 000 0

project 000 000 000 000 0


• Defines the ticket queues used by REQADM.

• File defines one queue per line.– Comments as with reqadm.conf.– First queue is the default queue for all clients.

general 005 010 040 400 0

restore 010 010 010 000 0

project 000 000 000 000 0Queue name

SLA hours (urgent)

SLA hours (low)

Not used


• Removing a queue from this file will not delete it from REQADM. Once a queue is set up (especially if it contains tickets), REQADM is too paranoid to delete it.– You could, however, scratch and reload your database

and just happen to fail to load that queue back in.– But beware if that (deleted) queue had tickets

requeued to other active queues, or vice versa. REQADM will still need to have it around.


• Changes to this file will not be noticed by REQADM until the server is restarted.

• This will eventually be fixed so sending a HUP signal will suffice.

Configuration: systems.conf

• Defines an initial set of host names, locations and platform types to associate with tickets.

• File defines one system per line.Intel::Linux|lenny|John’s Office

Intel::Linux|laverne|Mary’s Office

Sun::3/60|balthmus|Joey’s Office

Intel::Windows::NT 4|pc123|Accounting

Intel::Windows::98|demo27|Sales Laptop


• Defines an initial set of host names, locations and platform types to associate with tickets.

• File defines one system per line.Intel::Linux|lenny|John’s Office

Intel::Linux|laverne|Mary’s Office

Sun::3/60|balthmus|Joey’s Office

Intel::Windows::NT 4|pc123|Accounting

Intel::Windows::98|demo27|Sales Laptop

Platform name. Note hierarchy with “::” separating each level.

Hostname(unique in database)

Location


• Removing a system from this file will not remove it from the database. This file is just used to load into (merge with) the current database.

• The contents of this file are re-read when the reqadm server is started or receives a HUP signal.

Configuration: workgroups.conf

• Defines privileged users: those who can own tickets and have “administrator” access to the REQADM tools.– I.e., your systems administration staff.

• Also defines what group(s) each user belongs to.– Can then see a queue view for all tickets owned by

the group (those owned by any members of the group), to simplify managing the group.


• Also defines “rotations”– “Fake” groups corresponding to areas of

ownership (accounts, postmaster, etc.).– Tickets may be assigned to the rotation group,

then given to the actual owner who is in the rotation at that point in time.

• Manually by a dispatcher or the recipient themselves

• Or automatically via tools like REQMGR.


• One group/rotation/user per line.• Users must be defined after the group they

belong to.– Multiple entries for the same user indicate they

belong to multiple groups.

• Groups may be nested.• Each group is assigned a number in the same

range as WWID’s (but separate numbering space; will not conflict)


• Sample file:group 1 N support

member 1042 steve

group 2 N mailteam 1

member 1157 joey

member 1291 beth

group 100 Y postmaster

group 101 Y accounts



member 1042 steve


member 1157 joey

member 1291 beth



Start defining a group

This is group #1

The group may not own tickets (not a rotation)

Group name (shouldn’t be the same as an existing user)



member 1042 steve


member 1157 joey

member 1291 beth



Start defining a member of the support group

WWID

UNIX username (must match with WWID in database)



member 1042 steve


member 1157 joey

member 1291 beth




This is group #2

This group is directly under group #1 in the hierarchy (so “support” includes everyone in “mailteam” too).



member 1042 steve


member 1157 joey

member 1291 beth




This is group #100

This group may own tickets, so this is actually defining a “rotation”.


• Changes to this file will be reflected when the REQADM server receives a HUP signal.

• You’d usually do this when hiring a new support person, adding them to the list of privileged users.

Agenda








Maintenance

• Normally, REQADM should run fairly smoothly on its own, with little or no supervision.

• Maintenance tasks are automated and run from cron.

• The only usual exceptions are reconfiguring or reacting to a problem.

Nightly Cron Script

• /opt/reqadm/sbin/reqadm-nightly runs out of cron when users are not expected to need the system.– Closes requests which have not been approved– Suspends request server– Backs up database (only n days worth kept)– Updates personnel info from all static user ID files.– Allows server to continue running again.

Nightly Cron Script

• Watch the mailed output from this script to make sure it’s not indicating that disk space is running low or any other problems are occurring.

Agenda








Suspending the Server

• Some of the following actions require suspending the REQADM server while they are performed (e.g., any external access to the database files).

• This is done with the reqadm-control command.• Like everything else we’ll discuss, this is in

/opt/reqadm/sbin. Examples assume you’re in that directory.

reqadm-control

• ./reqadm-control -locks– Prints a list of all tickets currently locked:

G ID- TIME- QUEUE----- NUMBER----

W 037 00:05 request 130000

HOSTNAME/IP-NUM USERNAME COMMENT---

192.168.1.1 steve REQADM

reqadm-control



W 037 00:05 request 130000


192.168.1.1 steve REQADMLock grade (write)

Session ID

Lock Age

Ticket

reqadm-control



W 037 00:05 request 130000


192.168.1.1 steve REQADM

System connected from Username

Program running

reqadm-control

• Use to see who’s getting in the way of your shutting down the server.

• Also useful if people complain that they have requests locked: find the system/process where their client is still running.

reqadm-control

• ./reqadm-control -suspend– Suspend the server when remaining locks

released, and do something according to other options specified:-verbose

– Print informational messages showing what’s happening.

-time mm:ss– Abort attempt to suspend server if the specified time

expires before locks are released.

reqadm-control

• ./reqadm-control -suspend– Following the options is one of the following:

mm:ss– Suspend for the specified length of time, then resume

server operations.

commands– Execute the shell commands specified. When they

complete, resume server operations.

Checking Database Integrity

• To check to see if the database is intact:./dbchk ../db/reqadm-db.sc – Verifies that database is internally consistent

and valid for the DBMS to use.– Does not check that the data inside the database

actually make sense.– Database password is “lao:ban0”.

• SUSPEND SERVER BEFORE RUNNING!

Clearing Database Lockfiles

• If something died and left a lockfile behind, clear it with:./dbchk -u ../db/reqadm-db.sc – Be very certain the lockfile is not valid (i.e.,

that the database server or other program isn’t really running, accessing the database now).

– Database password is “lao:ban0”.


Recovering Trashed Databases

• If something catastrophic happened, you can recover the database from backups, to quickly get the system back up and running.

• This is tricky (we need a script to automate this), and if you don’t have a huge number of tickets, it’s probably better to clear and reload the database instead.

• In fact, I’m not going to teach it here until we do get a script for it...

Reloading Database From Scratch

• Delete the entire contents of the database:./dbinit ../db/reqadm-db.sc – Database password is “lao:ban0”.– If this won’t work because the database is

locked, you can use a -u option just like with dbchk (and with the same cautions).



• If the dbinit failed because the database is too corrupt even for that, you’ll have to recover an old reqadm-db.sc file. These are saved nightly as ../db/Backup/REQADM_ DBS.*.*.A. Copy one of these to the ../db directory under the name reqadm-db.sc and try the dbinit again.


• Now that the database is empty, load all your personnel information from the static files:./reqadm-update-db -people -verbose

• For speed, this blasts the data directly into the database itself, bypassing the REQADM server (which must not be running).



• Now we need to load all the tickets from the ASCII files into the REQADM database.

• If you have time to spare, leave the server down and use this program to bulk-load all the tickets directly into the database (bypassing the server, which must not be running):./reqadm-update-request-db



• If time is not on your side, and the server must be started ASAP, you can load only the most critical tickets by naming them as explicit arguments to the update program (this loads only those tickets, and prevents the normal queue scan):./reqadm-update-request-db queue/n ...



• What’s most critical? Maybe you want to load all the open tickets but not old, closed ones:cd ../spool

foreach queue (*)

../sbin/reqadm-update-request-db \

$queue/[0-9]*

end



• If you only load partial tickets, there is one important set of tickets you must load before bringing up the server: always load the highest-numbered ticket in each queue!– These might be closed already, so hunt in the Fixed/D*

directories for them.– Load them explicitly as explained already, but note they are

specified as queue/number even if stored in the Fixed directories.



• The reason for this is that if a new ticket is received, REQADM will assign a new ticket number higher than what it currently sees in the queue. If the highest existing ticket was not loaded, a duplicate ticket number will be issued by REQADM.

• Once (at least) the highest ticket in each queue is loaded, you may restart the server.

Bulk-Loading Tickets

• If you wish to bulk-load tickets while the server is up and running (perhaps to update status of all tickets, or maybe to finish loading a set of tickets partially loaded as noted a moment ago), run:./bulk-loader– This scans all your queues, invoking reqadm-bulk-load-request-db to load the tickets into the server.

Bulk-Loading Tickets

• This takes longer, but can be done while the server’s up, and people are accessing the request system.

Adding/Correcting Individual Tickets

• If you want to update the database to reflect what’s in a ticket (perhaps after editing the ASCII ticket file directly):./reqadm-bulk-load-request-db q/no– This is done with the server up and running

normally.

Agenda








Reqmgr

• Request queue manager– Monitors open requests– Automatically takes action based on state

• Highly customizeable– Modular design– Uses Perl API

REQMGR material by John Mechalas, PMD EC

Reqmgr

• Useful tasks– notify people when a request is submitted– auto-assign request from “rotations” to people– escalate “ignored” requests (extended time w/o

a human owner)– auto-assign requests with special “Subject:”

tags

Getting Started with reqmgr

• reqmgr.conf– defines “trigger” conditions– module called when request state matches a

rule– modules actually do the work

reqmgr.conf

• Directives– “AddModule” specifies a module to be loaded– “Match” determines whether or not multiple

rules can match a single request– see the man page

• Directive syntaxAddModule <modname>

Match <single|multiple>

reqmgr.conf

• Macros– substitue one string for another– expanded in rulesets only!– no quoting: value string taken literally– can be nested…– …but there’s no recursion protection

• Macro syntaxDefine macroname value string …

reqmgr.conf

• Variables– Sets a variable name to a text string– No quoting: value string taken literally– Visible to all reqmgr modules

• Variable syntaxSet varname value string …

reqmgr.conf

• Rulesets– define “trigger” conditions that activate a

module– processed in the order they are defined

• Ruleset syntaxRule ruleid:

field ::= target value

Module modname

Rulesets

• Rule expression– possible request fields defined in Reqmgr(3p)– ::= is an equality, expression match, or list

lookup operator (can be negated)– value is the data that field is compared to– see reqmgr.conf(5) for details

• modname must be declared via AddModule

Sample config

AddModule rotation

Set rot_config rotation.confSet rot_count_queues request incident restoreSet rot_assign_msg (reqmgr assignment)

Define ROTATION unixreq postmaster ntreq

Rule 1:owner =@ ROTATIONModule rotation

Sample Config

• Activates module “rotation” if owner one of:

unixreq

postmaster

ntreq

• Sets variables to be used by “rotation” module

Sample Config

• Module “rotation”– distributed with REQADM– uses config file to define rotation membership

• day of week or date

• time of day

• rotation name

– assigns requests to rotation members based on request load

reqmgr Modules

• Module API– written in Perl– documented in ReqmgrModuleAPI(3p)– see sample modules for development hints

Agenda








Keep In Touch

• Subscribe to the reqadm-users-l mailing list:– Send mail to [email protected]– In the message body place the text:

subscribe reqadm-users-l

end

– You’ll receive announcements of new releases, problems discovered, enhancements planned.

– Also a discussion forum for asking questions of interest to all REQADM users around the world.

Read the Files

• When you download a new release of REQADM, be sure to read the RELEASE-NOTES file to see what has changed since the last release.

Further Reading

• The REQADM distribution includes an unadvertised technical documentation section.

• Assuming you installed REQADM’s web files to /reqadm on your server, go to:http://.../reqadm/reqadm-tech.html

• Includes design information, database schemas, client/server protocol documentation, etc.

Copyright © 2000 Intel Corporation. All Rights Reserved. May be distributed under same terms and conditions as the REQADM distributions.

Documents

Zen and the Art of REQADM Maintenance Steve Willoughby PMD Engineering Computing 12 April 2000 Copyright © 2000 Intel Corporation