Building an MXG PDB from NTSMF Data

Building an MXG PDB from NTSMF Data

Chuck Hopf

chuck@chopf.com

www.chopf.com/geekland.shtml

Building an MXG PDB from NTSMF Data

• ‘Classic’ MXG PDB Structure

• Why it doesn’t work

• What needed to change

• How it is implemented for NTSMF

• Using BLDNTPDB

Classic MXG PDB Structure

• Read ALL the data and place it in a PDB with possibly CICSTRAN on the side

• Run daily - daily contains detail of ALL data read

• WEEKBLD consolidates dailies into weeklies

• MONTHBLD consolidates weeklies into monthlies

Classic MXG PDB Structure

• DASD storage required:– 8 daily PDB– 5 weekly PDB– enough WORK space for the largest

dataset (STEPS or TYPE74)

Why It Doesn’t Work

• 40 GB of SMF data daily

• Needs 160GB of WORK space (SORTWK and WORK)

• 320GB of space for daily datasets

• 1+ Terror Bytes for weekly datasets

• Time - Daily jobs that run for more than a day are not practical

What Needed to Change

• Multiple Parallel Threads

• Run week-to-date

• Run month-to-date

• Eliminate weekly/monthly jobs

• Exploit summarization macros where needed to reduce volume

How Is It Implemented for NTSMF?

• BLDNTPDB - a very large and complex collection of SAS MACROs that allows the user to choose how the PDB should be structured.

• Can use the classic MXG structure or...

• Can use virtually any structure the user chooses

Using BLDNTPDB

• Parameters and Meanings

• Examples of use

• Future Directions

• ?????????????????????

Using BLDNTPDB - Parameters and Meanings

• NTSMF=C:\MXG\NTSMF\NTSMF.SMF - where is your RAW NTSMF data?

• WEEKSTRT=MON - on what day of the week does your week start? The normal MXG default is Monday. This controls when weekly processing takes place.

• RUNNTINT=YES - should we run the NTINTRV summarization?

Parameters and Meanings

• ASUMDUR=HOUR - what level of summarization should we apply? Valid values are QTRHOUR, HALFHOUR, HOUR, SHIFT, DATE, WEEK, MONTH

• KEEPERS=null - A list of datasets to be ‘kept’ (usually not used)

• DROPPERS=null - A list of datasets to be ‘dropped’ (built but not kept in the output PDB.)

Parameters and Meanings

• ZEROOBS=null - Similar to DROPPERS but datasets are created with 0 observations

• MAXDRVS=26 - Maximum number of logical drives/physical drive (limits and array size.)

• RERUN=null - If YES is specified forces the rewriting of an old PDB USE WITH CAUTION

Parameters and Meanings

• MON=C:\MXG\MON• TUE=C:\MXG\TUE• WED=C:\MXG\WED• THU=C:\MXG\THU• FRI=C:\MXG\FRI• SAT=C:\MXG\SAT• SUN=C:\MXG\SUN• Dataset placement for DAILY PDBs. If a null string is

used, the dataset WILL NOT BE BUILT.

Parameters and Meanings

• WEEK=C:\MXG\WEEK

• WEEK2=C:\MXG\WEEK2

• WEEK3=C:\MXG\WEEK3

• WEEK4=C:\MXG\WEEK4

• WEEK5=C:\MXG\WEEK5

• Placement for weekly datasets. If a null string is used, the dataset WILL NOT BE BUILT.

Parameters and Meanings

• TREND=C:\MXG\TREND

• MONTH=C:\MXG\MONTH

• MONTH1=null

• Placement for monthly and TREND datasets. A null string results in the dataset NOT BEING BUILT.

Parameters and Meanings

• RUNDAY=YES - Run the BUILD process daily and build the specified daily databases

• RUNWEEK=NO - YES means run the weekly BUILD process if it is the day specified by WEEKDATE and WTD means build WTD datasets daily

• RUNMONTH=NO - YES means run the monthly BUILD process on the 2nd day of the month and MTD means build MTD datasets daily

• RUNTREND=WEEKLY - Build the TREND datasets when the WEEKLY BUILD is run. If DAILY is specified, build the TREND datasets when the DAILY datasets are built.

Parameters and Meanings

• RPTRAW=NO - Build a table of the distribution of the RAW NTSMF record types

• RPTDAY=PDB - Create reports for the most recent PDB at the hourly

• RPTWEEK=NO - Create reports for the WEEKLY PDB if it has just been

• RPTMONTH=NO - Create reports for the MONTHLY database if it has just been created

• RPTTREND=NO - Create reports for the TREND database if it has just been updated

Parameters and Meanings

• GRAFDAY=PDB - Create reports for the most recent PDB at the hourly

• GRAFWEEK=NO - Create reports for the WEEKLY PDB if it has just been

• GRAFMNTH=NO - Create reports for the MONTHLY database if it has just been created

• GRAFTRND=NO - Create reports for the TREND database if it has just been updated

Parameters and Meanings

• Any combination of RUNxxxx, RPTxxxx, and GRAFxxxx specifications may be used. You can do DAILY, WEEKLY, and MONTHLY runs in a single execution as well as creating reports and graphs at each level.

Parameters and Meanings

• Null strings passed as some parameters means ‘dont do this!’

Examples of Use

• Classic MXG Structure

• PDB for each day of week, 5 weeks, monthly with week run on Monday– %BLDNTPDB(RUNDAY=YES,

RUNWEEK=YES, RUNMNTH=YES, RUNTRND=WEEKLY);

Examples of Use

• Single daily PDB no WEEKLY/MONTHLY/TREND– %BLDNTPDB(RUNDAY=YES,MON=,

TUE=, WED=, THU=, FRI=, SAT=, SUN=);

Examples of Use

• Single daily PDB with week-to-date, month-to-date, and trending done daily– %BLDNTPDB(RUNDAY=YES,

RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MON=,TUE=,WED=, THU=, FRI=, SAT=, SUN=);

A Few Words About MXG 16.04 and Later

• Each dataset now has an up to 6 character mnemonic identifier. In the case of NTSMF data it starts with NT and then has 4 unique characters.

• This is used to identify what datasets to KEEP DROP or set to ZEROOBS.

• YOU MUST REVIEW THE 16.04 CHANGES and DOCUMENTATION!

Examples of Use

• Build and keep only the datasets needed for NTINTRV– %BLDNTPDB(KEEPERS=NTBROW

NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

Examples of Use

• Build only the datasets needed for NTINTRV and DROP after NTINTRV is created: – %BLDNTPDB(DROPPERS=NTBROW

NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

Examples of Use

• Build the datasets needed for NTINTRV but build the PROCESS dataset with zero observations:– %BLDNTPDB(KEEPERS=NTBROW

NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROR NTRDIR NTSERV NTSYST NTTCP, ZEROOBS=NTPROC);

Examples of Use

• Reporting can be added to any execution:– %BLDNTPDB(xxxxxxxxxxxxxxx,

RPTDAY=YES);

Examples of Use

• Reporting can be done against any PDB:– %BLDNTPDB(xxxxxxxxxxxxxxx,

RPTDAY=MON);

Examples of Use

• GRAPHS can be created for daily, weekly and trend datasets (CPU usage only at this time - more to come.)– %BLDNTPDB(xxxxxxxxxxxxx,

GRAFDAY=YES, GRAFWEEK=YES);

Examples of Use - A Case Study

Suppose you want the following:Run the PDB DAILY keeping all datasetsMaximum of 5 logical drives/physical driveStart the week on SundayKeep a WTD/MTD/and TREND datasetsRun reports of Daily usageCreate Graphs of Daily usage

Examples of Use- A Case Study

%BLDNTPDB(MON=,TUE=,WED=,THU=,FRI=,SAT=,SUN=,WEEK2=,WEEK3=,WEEK4=,WEEK5=,RUNDAY=YES,WEEKSTRT=SUN,RUNWEEK=WTD,RUNMNTH=MTD,RUNTRND=DAILY,MAXDRVS=5,RPTDAY=PDB,GRAFDAY=PDB);

Future Directions

• Largely up to YOU!

• My knowledge of the data is limited

• My experience with the data is limited

• I need to know what YOU want/need

Future Directions

• What reports are needed– Data sources– Data elements– Levels of summarization

Future Directions

• What graphs are needed– What data sources– What kinds of graphs– What levels of summarization

Future Directions

• Currently on the development list:– Selection of datasets to place in the

WEEKLY/MONTHLY databases - a WEEKKEEP= operand or something along those lines

– More automated summarization (this is a tough one because it supposes knowledge of the datasets and variables that is difficult to have programmatically.)

Future Directions

• Currently on the Development List– Ability to limit variables in WTD/MTD

datasets using DROP=– Ability to output reports as HTML

???????????