DB-8: Jump Starting Your OpenEdge ® Auditing Solution Stephen Ferguson Progress Software

DB-8: Jump Starting Your OpenEdge® Auditing Solution

Stephen FergusonProgress Software

© 2007 Progress Software Corporation2 DB-8: Jump Starting Your OpenEdge Auditing Solution

Agenda

OpenEdge Auditing Overview Getting Started with Auditing Staying in Control Creative Reporting

This presentation includes annotations with additional complementary information


What is Auditing?

“The process of evaluating an organization’s practices for safeguarding electronic

information from loss, damage, unintended disclosure, or denial of availability.”

The OpenEdge Auditing Core Service gathers, records, and securely maintains the information necessary to perform the

auditing process:

• Who was the client

• What action took place

• When did it happen

• Where did it happen


What is a Core Service?

Non-domain specific related functions that provide the common infrastructure for a modern application

Standard behavior, features and functionality independent of any specific application requirements

Typically pre-started and always available

Definition


What Can OpenEdge Auditing Do Out of the Box?

ABL & SQL • Database connections• Security administration• User login/logout (needs OpenEdge security)

OpenEdge DB • Default record level events • Schema changes• Database and _User administration

Audit policy and events administration Ease of reporting

Use the OpenEdge supplied policies and reports


From Schema-Trigger Based Auditing A

BL

Clie

nt

Audit PolicyTools

Application Code

Ap

pli

cati

on

D

ata

App DB

Audit EventManager(schema triggers)

Audit Data

Audit Data Manager

Audit Policy ManagerA

PI

Policy Data

Sec

uri

ty M

anag

er

SQ

L C

lien

t

Application Code R

epo

rtM

anag

erAudit

Report

Audit Data

Archive DB

ArchiveDaemon

Arc

hiv

eM

anag

erOfflineAuditData


To Auditing in OpenEdgeA

BL

Clie

nt

Database Tools and Utilities

Open Tools

Audit Policy Tools (APMT)

Application Code

SQ

L C

lien

t

Application Code

Audit Data A

pp

lica

tio

n

Dat

a

Policy Data

App DB

Audit Data

Archive DB

Audit EventSubsystem

Dat

abas

e

Inte

rnal

Ap

pli

cati

on

Sec

uri

ty S

ub

syst

emAudit Data Subsystem

Audit Policy SubsystemA

PI

ArchiveDaemon A

rch

ivin

g S

ub

syst

em

Rep

ort

ing

Su

bsy

stem

AuditReport

OfflineAuditData


No Thanks, I Already Got One

Flexible, scalable Core Service• Common built-in auditing for both SQL/ABL clients

• Performance, performance, performance

• Security

Audit system events • Utilities, schema changes, etc

Flexible, secure reporting Archiving Multi-database, multi-application

Why use OpenEdge Auditing over your own solution?


Agenda



Step 1: Before you Begin

Upgrade Databases AND Clients to 10.1+

Audit Data A

pp

lica

tio

n

Dat

a

Policy Data

App DB

Add Type II Storage Areas for Auditing Enable Auditing (prepares for auditing) Set database options Assign audit permissions Import shipped audit

policies


Step 2: Define Your Own Audit Policies

Through Audit Policies you control• What audit information is recorded

• Where to store audited information

• How to store audited information

• How much audited information to store

• Context information to query audit information

• Security of audit information

“An Audit Policy is the configuration that controls the recording of audit data into an OpenEdge database”


Audit Policy Attributes

Stored in audit-enabled OpenEdge databases• Contain any number of policies

Apply only to the database they are stored in Can have active or inactive state ( on/off ) Active policies are merged at load-time Can be changed and reloaded on-line Has a unique GUID identifier Policies contain event records


Agenda



Auditing Policy Designs

Many possible designs

No single right design for every application

Every application has one best design

The policy design is driven by• Who generates and runs the reports• Who generates and manages the policies• Who consumes the reports

What type of policy design do I use?


Audit Policy Design Goals

1. Record enough to generate the reports

2. NOT abuse disk space & performance

3. Simplify auditing administration

My Audit Policy design needs to


Choosing an Audit Policy Strategy

Do nothing Customer 100% responsible for generating policies

Supply audit policies as templates Development supplies 80% knowledge in templates Customer customizes remaining 20% of templates Are there any liability issues?

Sell audit administration as a service Developer does remote policy creation and administration

Supply turn-key audit policies Developer supplies 100% knowledge Customer site uses UI tool to manage auditing Are there any liability issues?

What are my audit policy deployment strategy choices ?


What is an OpenEdge Auditing Event?

Each Event definition is a unique action or operation

Audit Events fall into three types• Database CUD ( OpenEdge )

• Internal ( OpenEdge )

• Application ( ABL or SQL )

Each Event definition has a• Unique positive integer value ( 1 to max integer )

• “name” ( “customer.create” )

• “description” ( “create customer record” )

“Audit Events represent the WHAT in auditing.”


Audit Event Types

Used for• Recording a table’s row operations

– Create, Update, and Delete• Optionally recording selected field values

Recorded only in the local database

Query by table name OR table and selected field values

No automatic “application context” relating the record operation to application operation

Database Record Events


Audit Event Types

Used for• Recording business level, coarse grained, events• Events with no corresponding database operation• “Read auditing”• Applying “application context” to [record] audit events• Grouping related audit events for easy queries

Triggered by ABL language statements• ABL or SQL application code

Coded into the application• Event number• Audit record’s Event Context format and content• Audit record’s event detail format and content

Application Events


Application Events and Multiple Databases

Application Events are propagated to all databases• Allows for immediate query of events in any database

• Same Audit record UUID primary index in each database (duplicate)

• Duplicates removed by archive utility load operation

Minimize performance overhead• Enable only one database’s Event policy to record the

event if immediate queries are not required

What happens?


Setting Audit Context and Scope

Event ID & Context are the primary query filters

Used to simplify queries for specific• Record changes

• Application operation or action

• OpenEdge operations or actions

WARNING: avoid format changes at production sites(or you make queries very complex)

“Audit Event Context defines a specific instance of an audit event”


Event Context Strategy

Record Event context

• Query table changes by [index] field values “PUB.Customer” “PUB.Customerpluto” “PUB.Customerpluto•56 Bone Dr.”

Application Event context

• Use multiple fields of context information c1 [ .c2 [ .c3 … ] ]

– More context fields yields smaller record subsets“print”“print.audit”“print.audit.users.dduck”

Simplify queries for one or more instances of an Audit Event


Assigning Record Operation Audit Events

Each table has a block of 10 event numbers Related tables occupy sequential blocks Each table’s events

• CREATE - record create (table-base + 0)• UPDATE - record update (table-base + 1)• DELETE - record delete (table-base + 2)• VIEW - viewed by terminal user * (table-base + 3)• IMPORT - electronic transfer in * (table-base + 4)• EXPORT - electronic transfer out * (table-base + 5)• PRINT - paper copy made * (table-base + 6)• REPLICA - electronic copy made * (table-base + 7)

Controlled in table policies Controlled in event policies *

Suggested File Policy Event strategy


Audit Event Types

Regulations audit the “human” data access

Only application knows the “human” access• OpenEdge reads many records in a query

• Filtered record set returned to application

Read is not the only “human” access• Printed reports

• Electronic copy to removable media

• Network transport to external application

“Read” Audit Events


Keeping the long-term storage under control

Audit Archiving

Audit Archiver(s)

Audit Archive

DB

Short Term StoragePurposed,

Long Term Storage

ApplicationDB

Audit Data

.abd file(s)Audit

ArchiveAudit

ArchiveLoader(s)

AuditReports

Offline Storage


Auditing Archive Strategy

Consider application database as short term storage for audit data• Do not enable audit indexes• Use separate storage area for audit data• Archive often!

Use purposed database for audit archive / reporting• Enable all indexes

Plan for off-line storage


Agenda



Generating the required reports

Tables need audit policies• Which record operations need auditing• Fields values need to be recorded• Field values need to be indexed

Application events are needed and where

Application event context formats and values to use

Application-context and Audit-event-group to use• Where in the application code• Spanning which procedures and classes

The audit reports drive which


Querying Audit Data

Secure access to audit data• Separation of duty

Exposed as standard database tables for ease of reporting

Requires knowledge of the implementation• Schema and meta-schema

• Identifying fields

• How context is formatted (Varies by event id)

Audit data searchable by• User id, event id, date, context, transaction, audit

group, DB connection, client session

Reporting Subsystem

Reporting Subsystem


Querying Audit Transactional Data

Only record what you need to report

Use structured event names

• _sys.tbl.create• _sys.tbl.trig.update

Use reporting database

• Avoids SHARE-LOCK Stringed values always in

American format

• SESSION:DATE-FORMAT = “mdy”

• SESSION:NUMERIC-FORMAT = “American”

is the group for

supplies context to

consists of

created

_aud-audit-data

_Audit-data-guid

_Database-connection-id (IE1.1)_Client-session-uuid (FK) (IE1.2)_User-id (IE2.1)_Audit-date-time (IE5.1)_Audit-event-group (FK) (IE3.1)_Db-guid (FK) (IE3.2)_Transaction-id (IE3.3)_Transaction-sequence (IE3.4)_Event-id (FK) (IE4.1)_Event-context (IE6.1)_Application-context-id (FK) (IE7.1)_Event-detail_Audit-custom-detail_Audit-data-security-level_Data-seal

_aud-audit-data-value

_Audit-data-guid (FK)_Field-name (IE1.1)_Continuation-sequence

_Data-type-code_Old-string-value_New-string-value_Old-blob-value_New-blob-value_Old-clob-value_New-clob-value_Audit-data-security-level_Data-seal

_client-session

_Client-session-uuid

_Client-name_User-id (IE1.1)_Authentication-date-time (IE2.1)_Server-uuid_Authentication-domain-type_Authentication-domain-name_Db-guid (FK) (IE3.1)_Session-custom-detail_Audit-data-security-level_Data-seal

Client

Session

Information

Audit

Transaction

DataModified

Values

Per field

AuditReport

Recursive Join


What information is recorded?

supplies context to

is the group for

_aud-audit-data

_Audit-data-guid

_Database-connection-id (IE1.1)_Client-session-uuid (FK) (IE1.2)_User-id (IE2.1)_Audit-date-time (IE5.1)_Audit-event-group (FK) (IE3.1)_Db-guid (FK) (IE3.2)_Transaction-id (IE3.3)_Transaction-sequence (IE3.4)_Event-id (FK) (IE4.1)_Event-context (IE6.1)_Application-context-id (FK) (IE7.1)_Event-detail_Audit-custom-detail_Audit-data-security-level_Data-seal

Who did it?

When did it happen?

What event caused it?

What was the event on?

What was going on at the time?

Any other relevant info?


Reporting onApplication Context and Event Groups

Are a form of application audit event Normalize applying “application context” to

• Database record audit events• Other application audit events

Group related audit records across multiple databases

Application-Context and Audit-event-groups

UUID AB627H8Event 31998Application-context-idEvent context “Record visit”

UUID AB627H8Event 31998Application-context-idEvent context “Record visit”

UUID G78456UEvent 34600Application-context-id AB627H8Event context “Visit OK Btn”

UUID G78456UEvent 34600Application-context-id AB627H8Event context “Visit OK Btn”

UUID Q2395NLEvent 34002Application-context-id AB627H8Event context “PUB.T1:Jones”

UUID Q2395NLEvent 34002Application-context-id AB627H8Event context “PUB.T1:Jones”

Application-Context eventApplication events

Record events


Auditing Best Practices

Only audit what is absolutely necessary – tune with audit policy maintenance

Plan for reporting• Group event ids into ranges• Structure context consistently• Leverage audit event groups

Coding style even more important (assigns, record scope, transaction scope)


In Summary

Auditing is a Core Service One of many new features in

OpenEdge 10 Spend time planning your

implementation


Relevant Exchange Sessions

DB-19: OpenEdge Authentication Without the _User Table

DB-14: OpenEdge run-time database security revealed


Education / Documentation References

Education • What's New In OpenEdge 10.1: Auditing

Documentation• Core Business Services

PSDN• http://psdn.progress.com/index.ssp


Questions?


Thank you foryour time



Preparing for Auditing

Upgrade Databases AND Clients to 10.1A+ Add Type II Storage Areas for Auditing

• prostrct add <db> addaudit.st

Enable Auditing (prepares for auditing)

Preparing for auditing

d "Audit_Data":20,32;512 . f 40960

d "Audit_Data":20,32;512 .

d "Audit_Index":21,1;64 . f 5120

d "Audit_Index":21,1;64 .

proutil <db> -C enableauditing area “Audit_Data” indexarea “Audit_Index” [deactivateidx]

Audit Data A

pp

lica

tio

n

Dat

a

Policy Data

App DB


Database Options and Audit Permissions

Sec

uri

ty S

ub

syst

em


Application Context and Audit Event Groups

Example usage

DEFINE VARIABLE ctxID AS CHARACTER.

DEFINE VARIABLE grpID AS CHARACTER.

ctxID = AUDIT-CONTROL:SET-APPL-CONTEXT

(PROGRAM-NAME(1) + “:Create Order",

cOrderData,cExtraStuff).

…

grpID = AUDIT-CONTROL:BEGIN-EVENT-GROUP

(PROGRAM-NAME(1) + “:Create Order Line",

cLineData,cExtraStuff).

…

AUDIT-CONTROL:END-EVENT-GROUP.

AUDIT-CONTROL:CLEAR-APPL-CONTEXT.

Indexed

Indexed


Audit Event Types

Default Database Record Events

Demonstration and development purposes

Recommend using application defined event IDs for Production auditing

Name Event-id Description Type

_sys.db.rec.create 5100 “Create record” “schema”

_sys.db.rec.update 5101 “Update record” “schema”

_sys.db.rec.delete 5102 “Delete record” “schema”


Recording Field Values

Streamed (default)• Modified values stored in _Event-detail field of the

primary _aud-audit-data record• Minimizes performance impact• Limited by max record length – auto overflows • Arbitrary field order / content

Selectable via table / field policy

<fld-nam> + CHR(6) + <data-typ> + CHR(6) + [<old-val> +] CHR(6) + <new-val> + CHR(7)

• CHR(8) is used to delimit array elements

One Record per Field• Query for specific field value changes


Controlling the Storage of Audited Field Values

Audit Data Subsystem

Audit Data Subsystem

Au

dit

Eve

nt

Su

bsy

stem

Au

dit

Eve

nt

Su

bsy

stem

Dat

abas

e

Audit Policy Subsystem

Audit Policy Subsystem

_aud-audit-data_aud-audit-data

_aud-audit-data_aud-audit-data_aud-audit-data_aud-audit-data

_aud-audit-data_aud-audit-data_aud-audit-data-value_aud-audit-data-value

_Event-detail

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14Database record

“f1:old/new, f3:old/new, f10:old/new”“Streamed Field Values”(f2, f6, f9, f14)

“1 Field/Record”

_aud-file-policy_aud-field-policy

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 f14

Audited Fields


Application Event Examples

…

/* 32800 = Run Menu Option */

AppID = AUDIT-CONTROL:LOG-AUDIT-EVENT

(32800, cMenuCode,

cDetail, cMore).

…

/* READ auditing 32003 = Customer Enquiry */

AppID = AUDIT-CONTROL:LOG-AUDIT-EVENT

(32003, STRING(Customer.CustNum),

cCustomerDetail, cMore).

…

Indexed

Indexed


Audit Event Types

Are a form of application audit event

Could not be captured by an application’s bespoke auditing system

Are triggered by internal OpenEdge operations• ABL & SQL database clients• Database utilities

Ids are predefined by OpenEdge• In OpenEdge controlled event-id space [ 0 – 31,999 ]

_pvm.user.login.pass #10510_sys.audit.data.dump #10310_sys.tbl.create #5000_sql.dba.create #210_sys.area.truncate #10209

Internal Audit Events


Locating Specific Audit Data

DEFINE VARIABLE cKey AS CHARACTER NO-UNDO.

ASSIGN cKey = "PUB.orderline" + CHR(6) + STRING(orderline.ordernum) + CHR(7) + STRING(orderline.linenum).

IF CAN-FIND(FIRST _aud-audit-data NO-LOCK

WHERE _aud-audit-data._event-context = cKey)

THEN MESSAGE "Audit data exists for " + cKey.

Event context field _aud-audit-data._event-context

<owner>.<table>CHR(6)<id-fld-val>[CHR(7)<id-fld-val>.. ]

CHR(8) is used to delimit array elements

By default uses Primary

Key Fields

Documents

DB-8: Jump Starting Your OpenEdge ® Auditing Solution Stephen Ferguson Progress Software