15
Copyright © 2001, T-VEC Technologies, Inc. All rights reserved. Model-based Approach to Security Test Automation Mark Blackburn, Robert Busser, Aaron Nauman, T-VEC Ramaswamy Chandramouli, National Institute of Standards and Technology Security functional testing is a costly activity typically performed by security evaluation laboratories. These laboratories have struggled to keep pace with increasing demand to test numerous product variations. This paper summarizes the results of applying a model-based approach to automate functional security testing. The approach involves developing models of security requirements as the basis for automatic test vector and test driver generation. In the application, security properties were modeled and the resulting tests were executed against Oracle and Interbase database engines through a fully automated process. The findings indicate the approach, proven successful in a variety of other application domains, provides a cost-effective solution to functional security testing. 1 Introduction Software security is a software quality issue that continues to grow in importance as software systems are used to manage continually increasing amounts of critical corporate and personal information. The use of the Internet to manage and exchange this data on a daily basis has heightened the need for software architectures, especially internet-based architectures, which are secure. At the same time, the shortened development and deployment cycles for software make it difficult to conduct adequate security functional testing to verify whether software systems exhibit the expected security behavior. Presently, developing and executing security functional tests is time-consuming and costly. Security evaluation laboratories are struggling to meet demands to test many product variations produced in short release cycles. The situation calls for improving the economics of security functional testing. As a result, the National Institute of Standards and Technology (NIST) initiated a program to develop methods and tools for automating security functional testing [Cha99]. Security Functional Testing verifies whether the behavior of a product or system conforms to the security features claimed by the manufacturer (i.e., the product does what it is supposed to do). NIST and its sponsors initiated a multi-phase investigation to assess the use of a model-based approach to automate security functional testing. Several model-based approaches were accessed as part of the investigation. The approach described in this paper succeeded where others failed to provide end-to-end support including model development, model analysis, automated test generation, automated test execution in multiple environments, and results analysis. The assessment of this approach has demonstrated the feasibility of modeling security requirements to automate testing for various products and target platforms. NIST believes this should improve the economics of security functional testing for security evaluation laboratories, as well as commercial organizations that perform security testing. 1.1 Organization of Paper Section 2 details NIST’s vision for a methodology and toolkit to support automated security functional testing. Section 3 provides an overview of a methodology and toolkit that have been

Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

  • Upload
    ngobao

  • View
    221

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

Model-based Approach to Security Test Automation Mark Blackburn, Robert Busser, Aaron Nauman, T-VEC

Ramaswamy Chandramouli, National Institute of Standards and Technology

Security functional testing is a costly activity typically performed by security evaluation laboratories. These laboratories have struggled to keep pace with increasing demand to test numerous product variations. This paper summarizes the results of applying a model-based approach to automate functional security testing. The approach involves developing models of security requirements as the basis for automatic test vector and test driver generation. In the application, security properties were modeled and the resulting tests were executed against Oracle and Interbase database engines through a fully automated process. The findings indicate the approach, proven successful in a variety of other application domains, provides a cost-effective solution to functional security testing.

1 Introduction

Software security is a software quality issue that continues to grow in importance as software systems are used to manage continually increasing amounts of critical corporate and personal information. The use of the Internet to manage and exchange this data on a daily basis has heightened the need for software architectures, especially internet-based architectures, which are secure. At the same time, the shortened development and deployment cycles for software make it difficult to conduct adequate security functional testing to verify whether software systems exhibit the expected security behavior.

Presently, developing and executing security functional tests is time-consuming and costly. Security evaluation laboratories are struggling to meet demands to test many product variations produced in short release cycles. The situation calls for improving the economics of security functional testing. As a result, the National Institute of Standards and Technology (NIST) initiated a program to develop methods and tools for automating security functional testing [Cha99]. Security Functional Testing verifies whether the behavior of a product or system conforms to the security features claimed by the manufacturer (i.e., the product does what it is supposed to do).

NIST and its sponsors initiated a multi-phase investigation to assess the use of a model-based approach to automate security functional testing. Several model-based approaches were accessed as part of the investigation. The approach described in this paper succeeded where others failed to provide end-to-end support including model development, model analysis, automated test generation, automated test execution in multiple environments, and results analysis. The assessment of this approach has demonstrated the feasibility of modeling security requirements to automate testing for various products and target platforms. NIST believes this should improve the economics of security functional testing for security evaluation laboratories, as well as commercial organizations that perform security testing.

1.1 Organization of Paper

Section 2 details NIST’s vision for a methodology and toolkit to support automated security functional testing. Section 3 provides an overview of a methodology and toolkit that have been

Page 2: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

2

effective in satisfying NIST’s objectives and that form the basis of this report. Section 4 uses an example to illustrate the development of Security Verification Models to support test automation. Section 5 summarizes the activity of model analysis and test vector generation. Section 6 briefly discusses aspects of test driver generation and test execution.

2 NIST Requirements For Automated Security Functional Testing

NIST wishes to develop a methodology and a supporting toolkit to automate the process of Security Functional Testing. This automation will help security evaluation laboratories meet the demand for product testing. The automation approach is based on expressing a product’s security functional requirements in a model and using the supporting toolkit to automatically generate tests needed to verify security properties. A model of system security properties is referred to as a Security Verification Model. The supporting toolkit processes these models to:

• Check the specification for contradictions, requirement defects, feature interaction problems, and circular definitions. This analysis ensures that the underlying security functional requirements are consistent and reasonable as a basis for testing.

• Generate test cases from the security requirements specifications expressed in the models. These test cases must be effective in demonstrating an implementation satisfies the security requirements. Ideally, the test cases should include test inputs, expected behavior or outputs, and an association between each test and the specification from which it was derived. Test cases of this form are referred to as test vectors to distinguish them from generated tests cases that include only test inputs.

• Check for requirement-to-test traceability and report whether each requirement has an associated test.

As a single fault in security functionality can annul the entire system’s security behavior, it is critical that the model representation of the security requirements be complete. The techniques for developing tests to verify the security properties must also provide 100 percent test coverage of the security properties. As system security behavior is often a product of both trusted and untrusted system component, complete testing minimizes the risk of using untrusted components in a system. This risk minimization is an additional objective of the NIST effort.

3 Methodology and Toolkit for Automating Security Functional Testing

The basis for the methodology and toolkit described in this paper is a model-based test automation approach used successfully in various application domains since1996. The approach is referred to as the Test Automation Framework (TAF). The TAF integrates various modeling tools, like the SCRtool for modeling system and software requirements with the test automation tool T-VEC.* In this work, that TAF approach was tailored to automate security functional testing through Security Verification Models. The result is a set of guidelines for modeling security requirements. The assessment was based on modeling security requirements in order to automate testing in three distinct environments, as shown in Figure 1. The specific activities carried out in the assessment include:

* The Software Productivity Consortium develops TAF translators and methods. The Software Cost Reduction

(SCR) method and associated modeling tool, SCRtool, were developed by the Naval Research Laboratory [HJL96]. The T-VEC Test Vector Generation System is commercially available from T-VEC Technologies, Inc.

Page 3: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

3

• Model security requirements in SCR specifications using the SCRtool

• Translate SCR specifications into T-VEC test specification using an existing SCR-to-T-VEC model translator [BBF97; Bla98]

• Generate test vectors from the transformed SCR specification

• Develop test driver schemas for various target test environments

• Generate test drivers for a Java-based application

• Generate Perl test drivers for an SQL database using an ODBC database interface

• Generate Java test drivers for an SQL database using a JDBC database interface

ISO/IEC 15408 Security Target

scr2tvecModel

Translator SCR Security

Verification Model

T-VEC TestSpecification

T-VEC Test Vector Generator

Test Vectors

T-VEC Test Driver Generator

Test DriverSchemas

Figure 1. Process Flow Through the Tools

Figure 1 illustrates the process for automated security functional testing used in the assessment. First, security properties from ISO/IEC 15408 Security Target⊥ specification for Oracle 8 Database Server were modeled in SCR with the SCRtool. An SCR-to-T-VEC translator, developed by the Software Productivity Consortium and T-VEC, was used to translate the SCR model to a T-VEC test specification. T-VEC tools were then used on the T-VEC representation of the security properties to automatically generate test vectors (i.e., test cases with test input values, expected output values and traceability information) and requirement-to-test coverage metrics. The T-VEC test driver generator was used in the assessment to automatically generate test drivers to execute tests against a Java application designed to demonstrate the security properties, an Interbase 6.0 database server and an Oracle 8i database server. These tests were executed and the

⊥ An ISO/IEC 15408 Security Target is a document that contains a set of Security Functional Requirements,

corresponding implementation features and a set of Security Assurance Requirements written in a format that corresponds to an international standard.

Page 4: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

4

results were compared with the expected results from the test vectors to determine each product’s compliance to the security properties.1

The primary effort in customizing the TAF approach to support security functional testing involved developing heuristics for modeling security properties with SCR and finding techniques for developing test driver schemas to automate execution of SQL statements.

4 Security Verification Model

This section describes the development of a security verification model using the SCRtool through a process of requirement clarification. First, basic SCR modeling concepts are described. This is followed by a description of a security requirement that is then refined into a verification model.

4.1 SCR Modeling Concepts

SCR is a table-based modeling approach, as shown in Figure 2 that models system and software requirements. SCR represents system inputs as monitored variables, system outputs as controlled variables and intermediate values as term variables. Variables are defined as primitive types (e.g., Integers, Float, Boolean, Enumeration) or as user-defined types. Behavior is defined using a tabular approach relating four model elements: modes, conditions, events, and terms. A mode class is a state machine, where system states are called system modes and the transitions of the state machine are characterized by guarded events. A condition is a predicate characterizing a system state. An event occurs when any system entity changes value. Terms and controlled variables are functions of input variables, modes, or other terms. Their values are defined in the model through event or condition tables.

4.2 Security Specifications

The security requirements used in the assessment are defined in the Oracle8 Security Target document [Ora00]. This document describes the security functionality (behavior) claimed by Oracle and is submitted along with the product for security evaluation. A subset of the security requirements, referred to as Granting and Revoking Privileges and Roles, was modeled in the assessment. The test vectors derived from the model were used to generate test drivers for two different database servers, Interbase 6.0 and the Oracle 8.0.5.

The following sections describe the process of modeling the Granting Object Privilege (GOP) requirement, which is a part of the Granting and Revoking Privileges and Roles functionality. The GOP is defined in the Oracle8 Security Target as:

Granting Object Privilege Capability (GOP) - A normal user (the grantor) can grant an object privilege to another user, role or PUBLIC (the grantee) only if:

a) the grantor is the owner of the object; or

b) the grantor has been granted the object privilege with the GRANT OPTION.

A role represents a group of related users. The keyword PUBLIC represents all users. 1 The process of SCR model translation, test vector generation, test driver generation, and execution against the

Interbase database using Perl and ODBC completed in 2 minutes and 54 second running on a 400 MHz Windows NT machine with 256 KB of memory.

Page 5: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

5

Data Types

Requirement Modeling andClarification

Variables

ISO/IEC 15408 Security Target

BehaviorState Machines(Mode Table) Events Conditions

Figure 2. SCR Modeling Constructs

4.3 Requirement Analysis

Developing SCR models requires identifying the system monitored (input) and controlled (output) variables, and defining the relationships between them. This process is typically iterative. It involves defining the variables, the data types associated with the variables, and the tables that define relationships between the variables. A useful guideline for developing SCR models is to work backwards from each output to make the process goal-oriented. The value of each output is defined in terms of the system inputs. Term variables are introduced whenever intermediate values are necessary or useful. The relationships between the inputs and outputs are refined until complete enough to support both manual review and automated analysis. Manual review processes can validate the correctness of the model and completeness with respect to the textual requirements, while automated analysis can identify inconsistencies in the model.

Breaking the GOP requirement into clauses supports identifying variables and relationships. Table 1 contains elaboration and clarification of the GOP requirements to support modeling. In addition, it identifies the variables and relationships associated with each clause.

Page 6: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

6

Table 1. Variables and Relations

Requirement Statement/Clause Variables Relationsgrantorgranteeobjectprivilegegrantee typegrantorgranteeobjectprivilegegrantorgranteeobjectprivilegeobject owner GRANT OPTION granted object

A normal user (the grantor) can grant an object privilege to another user, role or PUBLIC (the grantee)

GOP (b) – a grantor (that does not own the object) can grant object privileges to the grantee if the object owner previously granted object privilege to the grantor with the GRANT OPTION

GOP (a) - a grantor can grant an object privilege to a grantee if the grantor owns the object

grantee constraints (user, role or PUBLIC)

grantor owns object

granted object privilege

From the analysis above, the monitored (input) variables identified in the system can be refined into the following set:

• privName – type of object privilege that can be granted (ALL, SELECT, INSERT, UPDATE, DELETE, etc)

• grantor – user granting an object privilege

• grantee – user being granted an object privilege

• granteeType – type of grantee for a particular grant operation as defined in the first sentence of the GOP textual requirement; grantee is a user, role, or PUBLIC

• selectedObj – object selected for a particular grant operation

• grantedObject – object for which grant privileges have previously been granted (identified through GRANT OPTION)

• objOwner – owner of the object

Two other variables are related to the concept of a role; a role is a type of grantee as defined in the first sentence of the GOP textual requirement. The related variables include:

• roleID – role being granted an object privilege

• granteeRoleID – role of the grantee (if any) being granted an object privilege

There can be one or more roles defined and known by the database system. The variable roleID is used to refer to a specific role known within the system, and used in various test cases. The granteeRoleID is a specific role assigned to the grantee.

The GOP requirements specify the conditions when privileges are granted for an object. An SCR model of these requirements should ensure that when all model conditions are satisfied, the output indicates the privilege is granted. This output is modeled as the Boolean controlled variable:

• grantedObjPriv– the grant operation executes successfully (TRUE) or fails (FALSE)

Page 7: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

7

4.3.1 Modeling Variables and Data Types

Variables are modeled in the SCRtool through the Variable Dictionary as shown in Figure 3. For example, the grantee is a monitored (input) variable (MON) of type userIDType.

Figure 3. Variables Modeled in SCR

User-defined types are model through the Type Dictionary. Data types can be numeric (Integer and Float), Boolean or Enumerated. Figure 4 shows some of the data types used in the GOP model. The type objectPrivType is an enumerated type whose values define valid privileges associated with an object. The type objectIDType is defined as an Integer with a range of 0 to 5. The SCRtool also has a Constant Dictionary for defining constants.

Figure 4. Data Types Modeled in SCR

4.4 Modeling Security Functional Requirements

Once the system’s data is defined, its behavior can be modeled. In SCR, this involves defining the values of the controlled (output) variables through condition, event, or mode tables. These tables define the value of a variable in terms of monitored (input) variables, terms (intermediate) variables, and mode (state) machines. Figure 5 provides a representation of the GOP model. The output value, grantObjPriv, is defined by a condition table referencing three other terms. The requirement GOP(a) is directly associated with the term grantor_owns_object and requirement GOP(b) is directly associated with the term granted_object_privileges. The term

Page 8: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

8

grantee_constraints is derived from the first sentence in GOP that defines a grantee as a user, role or PUBLIC.

grantObjPriv

Monitored(Input)

Variables

grantObjPriv

Controlled(Output)Variables

grantee_constraints

granted_object_privileges

grantor_owns_object

objOwner

grantee ,selectedObj, grantedObject

grantor

granteeType, granteeRoleID, roleID

privName

Key

ConditionTable(relation)

Figure 5. Model Structure for Grant Object Privilege

A value of a term variable is defined through a condition or event table as an intermediate value. Terms can be referenced as part of the constraints or value calculations of other terms or controlled variables. They reduce the complexity of the model by simplifying expressions and eliminating redundancies. The following sections describe the terms used in defining the value of grantObjPriv.

4.4.1 Modeling Relation grantor_owns_object

The term grantor_owns_object defines the conditions under which the grantor owns the object for which privileges are being granted. When these conditions are satisfied, the value of grantor_owns_object is TRUE. The condition table for grantor_owns_object is shown in Table 2. It specifies that the term is TRUE (grantor owns the object) when grantor = objOwner, otherwise, the term is FALSE.

Table 2. Table for Relation grantor_owns_object

Table Namegrantor = objOwner NOT(grantor = objOwner)

grantor_owns_object = TRUE FALSE

Condition

The conditions within a condition table can include:

- input or term variables - arithmetic operators (+,-,*,etc.) - relational operators (=, !=, >, <, etc.) - logical operators (AND, OR, or NOT)

Page 9: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

9

4.4.2 Modeling Relation grantee_constraints

The first clause of the GOP requirement (See Table 1) specifies that a user, role, or PUBLIC can be granted privileges. These classes of grantees are defined by granteeType. If a role is being granted privileges, the role is identified by roleID. A user can be associated with a particular role, which is represented by the monitored variable granteeRoleID. Table 3 shows the term grantee_constraints that defines the relationships between the granteeType, granteeRoleID, and roleID. There are three cases:

1. If the granteeType is user, then the grantee is a user. To ensure that the grantee is granted privileges as a user and not through the grantee’s role, the model specifies that the roleID must not equal the granteeRoleID.

2. If the granteeType is role, then the roleID must be valid, and the granteeRoleID must equal the roleID.

3. If the granteeType is PUBLIC, then the other variables can take on any value (i.e., don’t care situation)

Table 3. Table for Relation grantee_constraints

Table Name(granteeType = user AND granteeRoleID != roleID)OR( granteeType = role AND roleID != NULL AND granteeRoleID = roleID )OR(granteeType = PUBLIC) FALSE

grantee_constraints = TRUE FALSE

Condition

The grantee_constraints defines condition on variables that must be TRUE for any grant operation to succeed; therefore, conditions for grantee_constraints are defined when the output is TRUE.

4.4.3 Modeling Relation granted_object_privileges

The GOP(b) requirement states that if a user wishes to grant a privilege to an object and does not own the object, the user must have been granted the privilege with the GRANT OPTION. The term granted_object_privileges shown in Table 4 defines these conditions. The term is TRUE when:

1. the selected object is the object for which the privilege was granted (i.e., the selectedObj is the grantedObject).

2. the privilege was granted with the option to grant others the privilege (GRANT_OPTION is TRUE)

3. the owner of the object is not the grantor 4. the owner of the object is not the grantee

Page 10: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

10

Table 4. Table for Relation granted_object_privilege

Table NameselectedObj = grantedObject AND GRANT_OPTION AND objOwner != grantor AND objOwner != grantee

selectedObj = grantedObject AND NOT(GRANT_OPTION) AND objOwner != grantor AND objOwner != grantee

granted_object_privileges = TRUE FALSE

Condition

The FALSE condition for granted_object_privilege requires similar conditions to be TRUE to establish the relationships between the selectedObj, grantedObj, grantor, and grantee, but forces the GRANT_OPTION to be FALSE, because the GRANT_OPTION is the distinguishing condition between these cases.

4.4.4 Modeling Relation grantObjPriv

The definition of grantObjPriv, shown in Table 5, completes the model for the GOP requirement. Its definition includes references to the term tables previously described, as well as additional constraints on monitored variables. The two potential values for grantObjPriv include:

• grantObjPriv = TRUE – test case conditions are such that the privilege should be granted

• grantObjPriv = FALSE - test case conditions are such that the privilege should not be granted

Table 5. Condition Table for Grant Object Privilege (grantObjPriv)

GOP(b)

GOP(a)

Test Constraints

Table Name( grantor_owns_object OR (granted_object_privileges AND grantee_constraints)) AND (grantor != grantee) AND ( granteeType = user OR granteeType = role OR granteeType = PUBLIC) AND ( Priv_Name = ALL OR Priv_Name = UPDATE OR Priv_Name = SELECT OR Priv_Name = INSERT OR Priv_Name = DELETE)

NOT(grantor_owns_object) AND(NOT(granted_object_privileges) AND grantee_constraints) AND (grantor != grantee) AND ( granteeType = user OR granteeType = role OR granteeType = PUBLIC) AND ( Priv_Name = ALL OR Priv_Name = UPDATE OR Priv_Name = SELECT OR Priv_Name = INSERT OR Priv_Name = DELETE)

grantObjPriv = TRUE FALSE

Condition

Page 11: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

11

The conditions are divided into three groups to support explanation. The groups include:

1. GOP(a) – grantor can grant privilege to a grantee because the grantor owns the object 2. GOP(b) – grantor can grant privilege to a grantee because the grantor has been granted

object privileges with GRANT OPTION 3. Test Constraints – additional conditions that ensure that the GOP(a) and GOP(b)

conditions are fully exercised during test generation. The conditions ensure the following situations are tested:

- grantor is not the grantee - all possible combinations of the granteeType (user, role, or PUBLIC) - all possible privileges on operations (ALL, UPDATE, SELECT, etc.)

The differences between the TRUE and FALSE case for grantObjPriv is that the TRUE case establishes the required conditions:

1. the grantor_owns_object relationship that is associated with GOP(a), where the grantor owns the object, or

2. granted_object_privileges and grantee_constraints – that is associated with GOP(b) 3. Test constraints force all combinations to be applied

The FALSE case establishes the conditions under which the grant operation fails:

1. grantor is not the object owner (i.e., NOT(grantor_owns_object)) 2. grantor has not been granted object privilege (i.e., NOT(granted_object_privilege)) 3. the Test Constraints force complete test coverage of the grant types and privileges

5 Model Analysis and Test Vector Generation

Modeling and test vector generation is typically performed iteratively as the model is developed. The SCRtool provides a number of checks on the model to ensure that individual tables are consistent and complete. The SCR-to-T-VEC model translator and T-VEC tools perform additional checks that identify cross-table inconsistencies and contradictions. These model analysis capabilities support refining the model by identifying and correcting model defects.

The SCR-to-T-VEC model translator transforms each SCR table into a T-VEC subsystem. The T-VEC compiler converts each subsystem into a set of primitive test specifications that are used as the basis of test vector generation [BBF97]. The translated and compiled version of the grantObjPriv requirement includes 20 test specifications. The test vector generator attempts to determine two test vectors for each test specification based on a test selection strategy derived from the concept of domain testing theory2. Table 6 shows a tabular representation of the 40 test vectors produced for grantObjPriv. The test vectors include 12 monitored variables and 6 term variables (not shown in the table). The test values shown in Table 6 reflect how the test generator systematically selects low-bound and high-bound test points at the domain boundaries. The input 2 White and Cohen [WC80] proposed domain testing theory as a strategy for selecting test points to reveal domain

errors. It is based on the premise that if there is no coincidental correctness, then test cases that localize the boundaries of domains with arbitrarily high precision are sufficient to test all the points in the domain. This approach produces test input values that satisfy the conditions of the test specification and that localize the decisions in the specification to maximize defect detection. Once a set of test inputs are selected that satisfy the specification constraints, these inputs are used to derive the value of the output.

Page 12: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

12

values ranges and constraints (e.g., relational operators) of the specification define the domain boundaries. For example, vector # 1, grantor has id = 1, grantee has id = 2, is based on low-bound values of the data type range of userIDType, while vector # 2, grantor has id = 4, grantee has id = 3, is based on the high-bound for the data type range. In addition, the test generator creates a test for each value of privName and granteeType.

Table 6. Test Vectors for grantObjPriv

Vector # DCP grantObjPriv grantor grantee privNamegrantee

Type objOwnerselected

ObjgrantedObject

GRANT_OPTION

granteeRoleID roleID

1 1 TRUE 1 2 ALL user 1 4 4 TRUE 2 22 1 TRUE 4 3 ALL user 4 1 1 FALSE 0 03 2 TRUE 1 2 UPDATE user 1 4 4 TRUE 2 24 2 TRUE 4 3 UPDATE user 4 1 1 FALSE 0 05 3 TRUE 1 2 SELECT user 1 4 4 TRUE 2 26 3 TRUE 4 3 SELECT user 4 1 1 FALSE 0 07 4 TRUE 1 2 INSERT user 1 4 4 TRUE 2 28 4 TRUE 4 3 INSERT user 4 1 1 FALSE 0 09 5 TRUE 1 2 DELETE user 1 4 4 TRUE 2 210 5 TRUE 4 3 DELETE user 4 1 1 FALSE 0 0

�����37 19 FALSE 1 2 DELETE user 3 1 1 FALSE 0 138 19 FALSE 4 3 DELETE user 2 4 4 FALSE 2 139 20 FALSE 1 2 DELETE role 3 1 1 FALSE 1 140 20 FALSE 4 3 DELETE role 2 4 4 FALSE 2 2

Vector # DCP grantObjPriv grantor grantee privNamegrantee

Type objOwnerselected

ObjgrantedObject

GRANT_OPTION

granteeRoleID roleID

1 1 TRUE 1 2 ALL user 1 4 4 TRUE 2 22 1 TRUE 4 3 ALL user 4 1 1 FALSE 0 03 2 TRUE 1 2 UPDATE user 1 4 4 TRUE 2 24 2 TRUE 4 3 UPDATE user 4 1 1 FALSE 0 05 3 TRUE 1 2 SELECT user 1 4 4 TRUE 2 26 3 TRUE 4 3 SELECT user 4 1 1 FALSE 0 07 4 TRUE 1 2 INSERT user 1 4 4 TRUE 2 28 4 TRUE 4 3 INSERT user 4 1 1 FALSE 0 09 5 TRUE 1 2 DELETE user 1 4 4 TRUE 2 210 5 TRUE 4 3 DELETE user 4 1 1 FALSE 0 0

�����37 19 FALSE 1 2 DELETE user 3 1 1 FALSE 0 138 19 FALSE 4 3 DELETE user 2 4 4 FALSE 2 139 20 FALSE 1 2 DELETE role 3 1 1 FALSE 1 140 20 FALSE 4 3 DELETE role 2 4 4 FALSE 2 2

The number of vectors generated and the specific test values depend on the test vector generation mode, test input selection heuristics, and the satisfiability of the test specification conditions. A test specification is considered satisfiable, if a set of input values exist that satisfy all conditions and result in a valid expected output value. Unsatisfiable test specifications typically result from specification errors (e.g., requirement defects).

6 Test Driver Generation and Execution

The last step in the process involves transforming the tests into a test driver that can be executed against a security target, like the Oracle database. NIST stated that the capability to transform models into test drivers for a variety of platforms is an important discriminating capability of this toolset.

The test driver generator combines test driver schemas, user-defined object mappings and test vectors to produce test drivers as illustrated in Figure 6. The test driver schema encodes generic descriptions for test execution based on an algorithmic pattern that is applicable to the specific test environment. The object mappings relate objects in the model to the objects in the implementation or component interfaces. The test driver generator creates test drivers by repeating the execution steps defined in the schema for each test vector. There are typically four primary steps for executing each test case:

• Set the value of the test output to some value other than what is expected

• Set the values of the test inputs

• Cause execution of the test

• Retrieve and save the results of the test execution

Page 13: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

13

Test driver schemas provide a description of how to accomplish each of these steps for a specific testing environment using a small language that can access information about the specification model, data objects, types, ranges, test values, and user customizable information. A schema is also used to describe the form of expected outputs to support test execution and results analysis.

Database System

RetrieveOutputs

LoadInputs

Test Vectors

Algorithmic pattern

TestInputs

Global init;Forall tests

init target;set inputs;execute SUT;get outputs;store output;

endforall

CapabilityUnder Test

Implementationto Specification

Mapping

Implementation must suit the general algorithmic pattern(s) used

to automate driver generation

TestDriver

Test Driver Schema

ActualOutputs

SQL/ODBC/JDBC

Figure 6. Elements of a Test Driver

Three different test driver schemas and object mapping descriptions were used with the grantObjPriv model to test three different applications. First, a GUI-based Java application was developed to illustrate how test drivers could be injected into an application that has a graphical user interface. Next test drivers were generated for the InterBase 6.0, and Oracle 8 database engine. The Interbase test driver was developed in Perl using ODBC interface to issue SQL commands. The Oracle test driver was developed in both Perl and Java. The Java test drivers used JDBC to communicate to the database.

7 Summary and Future Work

The TAF approach, customized with specific guidelines for modeling security properties and developing test drivers for databases, satisfies NIST’s requirements for an automated model-based approach to automated Security Functional Testing. In the assessment of the approach, security requirements for the Oracle8 Security Target were modeled using the SCRtool. These models were then used as the basis of automated test vector and test driver generation with the T-VEC toolset for multiple product applications and test environments. This approach reduces the time and effort associated with security testing, while increasing the level of test coverage. NIST cited the approach’s ability to support driver generation for a variety of platforms as a key discriminator. These results demonstrate the feasibility of using model-based test automation to improve the economies of security functional testing. Specifically, the TAF approach is applicable

Page 14: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

14

to security evaluation laboratories and other commercial organizations that need a cost-effective approach for performing security functional testing.

7.1 Other Applications and Results

The core capabilities underlying this approach were developed in the late 1980s and proven through use in support of FAA certifications for flight critical avionics systems [BB96]. Statezni described how the approach supports requirement-based test coverage mandated by the FAA with significant life cycle cost savings [Sta99; Sta2000]. Safford presented results stating the approach reduced cost, effort, and cycle-time by eliminating requirement defects and automating testing [Saf2000]. Safford’s presentation summarized the benefits:

• Better quality requirements for design and implementation help eliminate rework in those phases as well as during test

• Verification modeling can reduce the time normally spent in verification test planning by up to 50 percent

• Test generation from a verification model can eliminate up to 90 percent of the manual test creation and debugging effort

• Both the number of test cases and the phasing of their execution can be optimized, eliminating test redundancy

• A known level of requirements coverage can be planned, and measured during test execution

The approach and tools described in this paper have been used for modeling and testing system, software integration, software unit, and some hardware/software integration functionality. It has been applied to critical applications like telemetry communication for heart monitors, flight navigation, guidance, autopilot logic, display systems, flight management and control laws, airborne traffic and collision avoidance. In addition, it has been applied to non-critical applications such as workstation-based Java applications with GUI user interfaces and database applications. The approach supports automated test driver generation in a variety of open languages (e.g., C, C++, Java, Ada, Perl, PL/I, SQL), as well as, proprietary languages, COTS test injection products, and test environments.

7.2 Future Work

The development team continues to evolve the model translation capabilities to support functional, object-oriented, control system and hybrid modeling approaches. In addition, the team is involved in the Object Management Group, UML Action Language Semantics formalization. The team is also involved in the development of modeling guidelines and training material that help integrate commercial modeling approaches with verification tools.

As continued support for NIST, additional models for the Oracle Security Target are being modeled to address the capabilities of: audit generation, security management, identification, authentication, and session management.

Page 15: Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional

Copyright © 2001, T-VEC Technologies, Inc. All rights reserved.

15

8 References

[BB96] Blackburn, M.R., R.D. Busser, T-VEC: A Tool for Developing Critical System. In Proceeding of the Eleventh International Conference on Computer Assurance, Gaithersburg, Maryland, pages 237-249, June, 1996.

[BBF97] Blackburn, M.R., R.D. Busser, J.S. Fontaine, Automatic Generation of Test Vectors for SCR-Style Specifications, In Proceeding of the 12th Annual Conference on Computer Assurance, Gaithersburg, Maryland, pages 54-67, June, 1997.

[Bla98] Blackburn, M. R., Using Models For Test Generation And Analysis, Digital Avionics System Conference, October, 1998.

[Cha99] Chandramouli R., Methodology for Automated Security Testing”, NIST Request for Proposal, Nov 1999.

[HJL96] Heitmeyer, C., R. Jeffords, B. Labaw, Automated Consistency Checking of Requirements Specifications. ACM TOSEM, 5(3):231-261, 1996.

[Ora00] Oracle Corporation, Oracle8 Security Target Release 8.0.5, April, 2000.

[Sta99] Statezni, David, Industrial Application of Model-Based Testing, 16th International Conference and Exposition on Testing Computer Software, June 14-18, 1999.

[Sta00] Statezni, David. Test Automation Framework, State-based and Signal Flow Examples, Twelfth Annual Software Technology Conference, 30 April - 5 May 2000.

[Saf00] Safford, Ed, L. Test Automation Framework, State-based and Signal Flow Examples, Twelfth Annual Software Technology Conference, 30 April - 5 May 2000.

[WC80] White, L.J., E.I. Cohen, A Domain Strategy for Computer Program Testing. IEEE Transactions on Software Engineering, 6(3):247-257,May, 1980.