A Binary Technology for COTS Software Integrity

Anant AgarwalRichard Schooler

InCert Software

Active-X, Java, and increasing reliance on “commercial off-the-shelf” technology help infiltrators make unknowing accomplices of legitimate users

CACM, July ‘99Durst, Champion, Witten, Miller, Spagnuolo, AFRL, on mission critical computer systems

The Mission Critical Environment

Output

Input OperatingSystem

SAP

COTSBinary

The developmentenvironment The deployment

environment

COTSBinary

“Hostility” in The Mission Critical Environment

COTSBinary

Input

Output

OperatingSystem

SAP

Virus attacks

Bad data (remember Y2K)

Invalid/Nullarguments

Untrusted third party data

C2 securityrequirements

Untrusted third party software

Objective

COTSBinary

Input

Output

OperatingSystem

SAP

To improve the integrity of thedeployment environmentfor COTS softwarein the presence of"hostilities"

Some Current Approaches

•Applied at source level during developmente.g., type based safety; work of Lee et al.

•Applied at link time with special object formatse.g., software fault isolation; work of Pandey et al.

•Applied through interpretore.g., safe Java interpreters

•Applied during program executione.g., middlewareThis approach works with COTS packages, PC, Mainframe, etc. -- hence it is a widely adopted commercial approach

•Modify OS, like middleware integrated into OS e.g., wrap OS layer to intercept calls for services

COTS Integrity Approach through Binary Augmentation

The developmentenvironment

The deploymentenvironment

COTSNewMissing

sourceLegacy

COTS Binary

COTSNewMissing

sourceLegacy BAS

Recovery logsAccess constraintsLogging requirementsArgument rangesRare code execution(defaults for fault tol.,policy specs for security)

COTSNewMissing

sourceLegacy

UserSpecified

The Current Commercial Solution: Middleware

COTSBinary

Input

Output

OperatingSystem

SAP

Middleware

•Slow•Maintenance nightmare•Cannot handle untrusted software•Cannot deal with Viruses•Cannot improve fault tolerance of COTS package itself

Why other solutions (source, link, interpreter) do not often apply to COTS software

Source

Out-sourceddevlp.

Missingsource

COTSbin objpkgs

PackagedCOTS executable

Legacyobjects

COTSNewMissing

sourceLegacy

Consider this Vendor’s development environment

Why other solutions (source, link, interpreter) do not often apply to COTS software

•Vendor wants to supply generic COTS (tryconvincing m to customize word for you)

•User wants to customize security policy

•Impossible to take a security approach involving “writing-all-code-afresh”

•Near impossible for user to arm-twist vendor into adding security features (note the difficulty beingfaced by ARM like apps mgmt standards)

Needed: An Approach to Integrity that

•Works with COTS binaries, even legacy codes

•Allows a user to establish desired security levelsand to some extent modify policy on the fly

•Works completely at the user’s deployment site

Our’s is a systems level approach that attempts to satisfy the above goals

COTS Integrity Approach through Binary Augmentation

The developmentenvironment

The deploymentenvironment

COTSNewMissing

sourceLegacy

COTS Binary

COTSNewMissing

sourceLegacy BAS

Recovery logsAccess constraintsLogging requirementsArgument rangesRare code execution(defaults for fault tol.,policy specs for security)

COTSNewMissing

sourceLegacy

UserSpecified

Three Major Components in the Prototype,Three Major Tasks

•Core technology for customizable agent insertion into PC/NT, PC/Linux

•Anomaly detection

•Rapid recovery technology

Three Major Components in the Prototype

•Core technology for customizable agent insertion•Develop basic instrumentation technology for NT•Hard-to-find relocations -- use incremental control

and dataflow analysis to create control-flow graph•Dynamic methods from binary translation to

augment static analysis•Evaluate on-the-fly binary rewriting versus table

driven approaches for augmenting agent function•Optimize performance of on-the-fly instrumentation,

and that of the instrumented COTS binary during itsproduction run

Three Major Components in the Prototype

•Anomaly detection•Several defaults -- open to other ideas

•Rare code exec (application path signatures, andtest path signatures if available)

•address ranges, null ptrs, historical value ranges etc. •User specified -- need help here

•We want to leverage an existing spec•Training phase to relate user function to code

Develop training instrumentation agentse.g., fire bad transaction, agents record code path,arguments, etc, and cause alert in production run

Three Major Components in the Prototype

•Rapid recovery technologyOne form of agents that records where programhas been, so trace can enable fast recovery from crashor alert following an actual or suspected attack

•Record program path in circular trace buffers•Maintain time stamp info at entry and exit points into

program to enable stitching together multithreadedtraces

•Record values upon user request (as a side effect, cancreate logs for various security requirements, e.g., C2)

•Cause alarm into theater wide console (e.g. unicenter TNG) upon alert/crash, and write buffers to file•Implement mechanisms for trace compression and

execution speed

User Model•Import COTS software package•User specifies policies and constraints, but

fault tolerance mechanisms, logging rapid recovery and other useful defaults exist

•Our binary augmentation compiler will stitch in appropriate code (agents) into binary; hard to analyze code to be augmented with on-the-fly code rewriting

•Run COTS software. Stitched in code or agents performlogging, checks, and apply constraints

•On a crash or user-specified alert the dynamic sequencingof instructions and data values can be retrieved fromtrace logs, and will enable rapid recovery

Measures of Success

•We will build a prototype system, work with real users, and measure•Core technology for agent insertion into binary:

Can we handle all binaries, even dusty decks?Performance degradation to be under 1 percent

•Anomaly detectionWhat fraction of injected problems can we detect

- automatically- with user spec

•Rapid recovery technologyPerformance degradation to be under 1 percentCan we cut recovery time significantly? We will measure

recovery time with and withoutAs a bonus, can we catch problems before system|goes down?

Some Challenges

•Core technology for agent insertion into binary•Creation of a general instruction framework for

multiple ISAs -- how much work is it to go to another platform?•How to deal with unknown relocations, e.g., for dusty decks -

an integrated static and dynamic method?•Anomaly detection

•How to relate user function to binary checks - learning phase to obtain execution path signatures?

•How to minimize runtime overhead - use compileroptimization technologies for agents? (e.g., stealregisters, inline code, sampling, multilevel checks)

•Rapid recovery technology•Runtime overhead issue - especially for data values

use dataflow analysis and offline simulation toobtain intermediate data values?

•Logfile size issue - use logfile compression methods?

What do we Need?•A partner from a defense site with PC/NTs and

ES390’s, so we can get help on perceivedsources of attack, test our planned systemin a realistic environment and obtain accuratemeasures of performance, generality, and ease of use.

•Suggestions for a theater wide “console,” sowe can integrate our alerts within that standardAPI. e.g., any defense site using Unicenter TNG?

•Suggestions on a spec for user-specified security•Is there a “attack scenarios” benchmark set? If not,

we must create one.

Technology Transition

•Build prototype and make available to a defense installation

•Integrate our system with its alerts into a commercialoff the shelf theater wide console

•Our COTS based approach will make our technology interesting to both military and commercial IT organizations with mission critical enterprisesoftware systems (remember, eTrade’s disastrous shutdown!), so we will look moveexperimental system to a commercial product

Summary

•A systems approach to COTS Integrity•User specifies policies and constraints, but useful

defaults exist •Approach based on binary agent insertion•Integrity technology will work even with

legacy binaries, requires no new formats,or language modifications