Model-based Approach to Security Test Automation to functional security testing. 1 Introduction Software security is a ... For Automated Security Functional Testing ... functional testing through Security

  • Published on

  • View

  • Download

Embed Size (px)


  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.

    Model-based Approach to Security Test Automation Mark Blackburn, Robert Busser, Aaron Nauman, T-VEC

    Ramaswamy Chandramouli, National Institute of Standards and Technology

    Security functional testing is a costly activity typically performed by security evaluation laboratories. These laboratories have struggled to keep pace with increasing demand to test numerous product variations. This paper summarizes the results of applying a model-based approach to automate functional security testing. The approach involves developing models of security requirements as the basis for automatic test vector and test driver generation. In the application, security properties were modeled and the resulting tests were executed against Oracle and Interbase database engines through a fully automated process. The findings indicate the approach, proven successful in a variety of other application domains, provides a cost-effective solution to functional security testing.

    1 Introduction

    Software security is a software quality issue that continues to grow in importance as software systems are used to manage continually increasing amounts of critical corporate and personal information. The use of the Internet to manage and exchange this data on a daily basis has heightened the need for software architectures, especially internet-based architectures, which are secure. At the same time, the shortened development and deployment cycles for software make it difficult to conduct adequate security functional testing to verify whether software systems exhibit the expected security behavior.

    Presently, developing and executing security functional tests is time-consuming and costly. Security evaluation laboratories are struggling to meet demands to test many product variations produced in short release cycles. The situation calls for improving the economics of security functional testing. As a result, the National Institute of Standards and Technology (NIST) initiated a program to develop methods and tools for automating security functional testing [Cha99]. Security Functional Testing verifies whether the behavior of a product or system conforms to the security features claimed by the manufacturer (i.e., the product does what it is supposed to do).

    NIST and its sponsors initiated a multi-phase investigation to assess the use of a model-based approach to automate security functional testing. Several model-based approaches were accessed as part of the investigation. The approach described in this paper succeeded where others failed to provide end-to-end support including model development, model analysis, automated test generation, automated test execution in multiple environments, and results analysis. The assessment of this approach has demonstrated the feasibility of modeling security requirements to automate testing for various products and target platforms. NIST believes this should improve the economics of security functional testing for security evaluation laboratories, as well as commercial organizations that perform security testing.

    1.1 Organization of Paper

    Section 2 details NISTs vision for a methodology and toolkit to support automated security functional testing. Section 3 provides an overview of a methodology and toolkit that have been

  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.


    effective in satisfying NISTs objectives and that form the basis of this report. Section 4 uses an example to illustrate the development of Security Verification Models to support test automation. Section 5 summarizes the activity of model analysis and test vector generation. Section 6 briefly discusses aspects of test driver generation and test execution.

    2 NIST Requirements For Automated Security Functional Testing

    NIST wishes to develop a methodology and a supporting toolkit to automate the process of Security Functional Testing. This automation will help security evaluation laboratories meet the demand for product testing. The automation approach is based on expressing a products security functional requirements in a model and using the supporting toolkit to automatically generate tests needed to verify security properties. A model of system security properties is referred to as a Security Verification Model. The supporting toolkit processes these models to:

    Check the specification for contradictions, requirement defects, feature interaction problems, and circular definitions. This analysis ensures that the underlying security functional requirements are consistent and reasonable as a basis for testing.

    Generate test cases from the security requirements specifications expressed in the models. These test cases must be effective in demonstrating an implementation satisfies the security requirements. Ideally, the test cases should include test inputs, expected behavior or outputs, and an association between each test and the specification from which it was derived. Test cases of this form are referred to as test vectors to distinguish them from generated tests cases that include only test inputs.

    Check for requirement-to-test traceability and report whether each requirement has an associated test.

    As a single fault in security functionality can annul the entire systems security behavior, it is critical that the model representation of the security requirements be complete. The techniques for developing tests to verify the security properties must also provide 100 percent test coverage of the security properties. As system security behavior is often a product of both trusted and untrusted system component, complete testing minimizes the risk of using untrusted components in a system. This risk minimization is an additional objective of the NIST effort.

    3 Methodology and Toolkit for Automating Security Functional Testing

    The basis for the methodology and toolkit described in this paper is a model-based test automation approach used successfully in various application domains since1996. The approach is referred to as the Test Automation Framework (TAF). The TAF integrates various modeling tools, like the SCRtool for modeling system and software requirements with the test automation tool T-VEC.* In this work, that TAF approach was tailored to automate security functional testing through Security Verification Models. The result is a set of guidelines for modeling security requirements. The assessment was based on modeling security requirements in order to automate testing in three distinct environments, as shown in Figure 1. The specific activities carried out in the assessment include:

    * The Software Productivity Consortium develops TAF translators and methods. The Software Cost Reduction

    (SCR) method and associated modeling tool, SCRtool, were developed by the Naval Research Laboratory [HJL96]. The T-VEC Test Vector Generation System is commercially available from T-VEC Technologies, Inc.

  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.


    Model security requirements in SCR specifications using the SCRtool Translate SCR specifications into T-VEC test specification using an existing SCR-to-

    T-VEC model translator [BBF97; Bla98]

    Generate test vectors from the transformed SCR specification Develop test driver schemas for various target test environments Generate test drivers for a Java-based application Generate Perl test drivers for an SQL database using an ODBC database interface Generate Java test drivers for an SQL database using a JDBC database interface

    ISO/IEC 15408 Security Target


    Translator SCR Security

    Verification Model

    T-VEC TestSpecification

    T-VEC Test Vector Generator

    Test Vectors

    T-VEC Test Driver Generator

    Test DriverSchemas

    Figure 1. Process Flow Through the Tools

    Figure 1 illustrates the process for automated security functional testing used in the assessment. First, security properties from ISO/IEC 15408 Security Target specification for Oracle 8 Database Server were modeled in SCR with the SCRtool. An SCR-to-T-VEC translator, developed by the Software Productivity Consortium and T-VEC, was used to translate the SCR model to a T-VEC test specification. T-VEC tools were then used on the T-VEC representation of the security properties to automatically generate test vectors (i.e., test cases with test input values, expected output values and traceability information) and requirement-to-test coverage metrics. The T-VEC test driver generator was used in the assessment to automatically generate test drivers to execute tests against a Java application designed to demonstrate the security properties, an Interbase 6.0 database server and an Oracle 8i database server. These tests were executed and the

    An ISO/IEC 15408 Security Target is a document that contains a set of Security Functional Requirements,

    corresponding implementation features and a set of Security Assurance Requirements written in a format that corresponds to an international standard.

  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.


    results were compared with the expected results from the test vectors to determine each products compliance to the security properties.1

    The primary effort in customizing the TAF approach to support security functional testing involved developing heuristics for modeling security properties with SCR and finding techniques for developing test driver schemas to automate execution of SQL statements.

    4 Security Verification Model

    This section describes the development of a security verification model using the SCRtool through a process of requirement clarification. First, basic SCR modeling concepts are described. This is followed by a description of a security requirement that is then refined into a verification model.

    4.1 SCR Modeling Concepts

    SCR is a table-based modeling approach, as shown in Figure 2 that models system and software requirements. SCR represents system inputs as monitored variables, system outputs as controlled variables and intermediate values as term variables. Variables are defined as primitive types (e.g., Integers, Float, Boolean, Enumeration) or as user-defined types. Behavior is defined using a tabular approach relating four model elements: modes, conditions, events, and terms. A mode class is a state machine, where system states are called system modes and the transitions of the state machine are characterized by guarded events. A condition is a predicate characterizing a system state. An event occurs when any system entity changes value. Terms and controlled variables are functions of input variables, modes, or other terms. Their values are defined in the model through event or condition tables.

    4.2 Security Specifications

    The security requirements used in the assessment are defined in the Oracle8 Security Target document [Ora00]. This document describes the security functionality (behavior) claimed by Oracle and is submitted along with the product for security evaluation. A subset of the security requirements, referred to as Granting and Revoking Privileges and Roles, was modeled in the assessment. The test vectors derived from the model were used to generate test drivers for two different database servers, Interbase 6.0 and the Oracle 8.0.5.

    The following sections describe the process of modeling the Granting Object Privilege (GOP) requirement, which is a part of the Granting and Revoking Privileges and Roles functionality. The GOP is defined in the Oracle8 Security Target as:

    Granting Object Privilege Capability (GOP) - A normal user (the grantor) can grant an object privilege to another user, role or PUBLIC (the grantee) only if:

    a) the grantor is the owner of the object; or

    b) the grantor has been granted the object privilege with the GRANT OPTION.

    A role represents a group of related users. The keyword PUBLIC represents all users. 1 The process of SCR model translation, test vector generation, test driver generation, and execution against the

    Interbase database using Perl and ODBC completed in 2 minutes and 54 second running on a 400 MHz Windows NT machine with 256 KB of memory.

  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.


    Data Types

    Requirement Modeling andClarification


    ISO/IEC 15408 Security Target

    BehaviorState Machines(Mode Table) Events Conditions

    Figure 2. SCR Modeling Constructs

    4.3 Requirement Analysis

    Developing SCR models requires identifying the system monitored (input) and controlled (output) variables, and defining the relationships between them. This process is typically iterative. It involves defining the variables, the data types associated with the variables, and the tables that define relationships between the variables. A useful guideline for developing SCR models is to work backwards from each output to make the process goal-oriented. The value of each output is defined in terms of the system inputs. Term variables are introduced whenever intermediate values are necessary or useful. The relationships between the inputs and outputs are refined until complete enough to support both manual review and automated analysis. Manual review processes can validate the correctness of the model and completeness with respect to the textual requirements, while automated analysis can identify inconsistencies in the model.

    Breaking the GOP requirement into clauses supports identifying variables and relationships. Table 1 contains elaboration and clarification of the GOP requirements to support modeling. In addition, it identifies the variables and relationships associated with each clause.

  • Copyright 2001, T-VEC Technologies, Inc. All rights reserved.


    Table 1. Variables and Relations

    Requirement Statement/Clause Variables Relationsgrantorgranteeobjectprivilegegrantee typegrantorgranteeobjectprivilegegrantorgranteeobjectprivilegeobject owner GRANT OPTION granted object

    A normal user (the grantor) can grant an object privilege to another user, role or PUBLIC (the grantee)

    GOP (b) a grantor (that does not own the object) can grant object privileges to the grantee if the object owner previously granted object privilege to the grantor with the GRANT OPTION

    GOP (a) - a grantor can grant an object privilege to a grantee if the grantor owns the object

    grantee constraints (user, role or PUBLIC)

    grantor owns object

    granted object privilege

    From the analysis above, the monitored (input) variables identified in the system can be refined into the following set:

    privName type of object privilege that can be granted (ALL, SELECT, INSERT, UPDATE, DELETE, etc)

    grantor user granting an object privilege grantee user being granted an object privilege granteeType type of grantee for a particular grant operation as defined in the first



View more >