Client and Server Verification for Web Services Using Interface Grammars

Graham Huges, Tevfik Bultan, Muath Alkhalaf

Department of Computer ScienceUniversity of California, Santa Barbara

Outline

• Motivation• Interface Grammars • Interface Compiler• Interface Grammars for Web Services• A Case Study• Conclusions

Model Checking Software

• Model checking– An automated software verification technique– Exhaustive exploration of the state space of a program

to find bugs

• Systematically explore all possible behaviors of a program– look for violations of the properties of interest

• assertion violations, deadlock

• Software model checkers: Verisoft, Java PathFinder (JPF), SLAM, BLAST, CBMC

Two Challenges in Model Checking

• State space explosion– Exponential increase in the state space with increasing

number of variables and threads• State space includes everything: threads, variables,

control stack, heap

• Environment generation– Finding models for parts of software that are

• either not available for analysis, or• are outside the scope of the model checker

Modular Verification

• Modularity is key to scalability of any verification technique– Moreover, it can help isolating the behavior you wish

to focus on, removing the parts that are beyond the scope of your verification technique

• Modularity is also a key concept for successful software design– The question is finding effective ways of exploiting the

modularity in software during verification

Interfaces for Modularity

• How do we do modular verification?– Divide the software to a set of modules– Check each module in isolation

• How do we isolate a module during verification/testing?– Provide stubs representing other modules (environment)

• How do we get the stubs representing other modules?– Write interfaces

• Interfaces specify the behavior of a module from the viewpoint of other modules

• Generate stubs from the interfaces

Our Approach: Interface Grammars

Here is the basic use case for our approach:

1. User writes an interface grammar specifying the interface of a component

2. Our interface compiler automatically generates a stub from the interface grammar

3. Automatically generated stub provides the environment during modular verification

Interface Grammars

Component A

Component B

Inte

rfac

e G

ram

mar

InterfaceCompiler

Component B Stub

Component A

ModelChecker

Inte

rfac

e G

ram

mar

An Example

• An interface grammar for transactions– Specifies the appropriate ordering for calls to a

transaction manager

– Calls are the terminal symbols of the interface grammar

SStarttart →→ BaseBaseBaseBase →→ beginbegin TailTail BaseBase

|| εεTailTail →→ commitcommit

|| rollbackrollback

An Example

• Consider the call sequence

begin rollback begin commit

• Here is a derivation:

Start Base begin Tail Base begin rollback Base begin rollback begin Tail Base begin rollback begin commit Base begin rollback begin commit

SStarttart →→ BaseBaseBaseBase →→ beginbegin TailTail BaseBase

|| εεTailTail →→ commitcommit

|| rollbackrollback

Another Example

• This example can also be specified as a Finite State Machine (FSM)

• However, the following grammar which specifies nested transactions cannot be specified as a FSM

StartStart →→ BaseBaseBaseBase →→ beginbegin BaseBase TailTail BaseBase

|| εεTailTail →→ commitcommit

|| rollbackrollback

begin

commit rollback

Yet Another Example

• Let’s add another operation called setsetrollbackonlyrollbackonly which forces all the pending transactions to finish with which forces all the pending transactions to finish with rollbackrollback instead of instead of commitcommit

• We achieve this by extending the interface grammars with We achieve this by extending the interface grammars with semantic predicates and semantic actionssemantic predicates and semantic actions

StartStart →→ «r:=false; l:=0»«r:=false; l:=0» BaseBaseBaseBase →→ beginbegin «l:=l+1»«l:=l+1» BaseBase Tail Tail «l:=l-1; if l=0 then r:=false»«l:=l-1; if l=0 then r:=false» BaseBase

|| setsetrollbackonly rollbackonly «r:=true»«r:=true» BaseBase || εε

TailTail →→ «r=false»«r=false» commitcommit || rollbackrollback

Interface Grammar Translation

• Our interface compiler translates interface grammars to executable code: – the generated code is the stub for the component

• The generated code is a parser that – parses the incoming calls– while making sure that the incoming calls conform to the

interface grammar specification

Verification with Interface GrammarsIn

terf

ace

Gra

mm

ar

InterfaceCompiler

Program

ModelChecker

Top-downparser

parsetable

semanticpredicates

andsemanticactions

pars

er s

tack

Com

pone

nt S

tub

invocation(lookahead)

Interface Grammars: Client vs. Server

• Interface grammars can be used for – Client verification: Generate a stub for the server– Server verification: Generate a driver for the server

InterfaceCompiler

Inte

rfac

e

ClientStub

ServerDriver

Parser

SentenceGenerator

Interface Grammars and Data

• A crucial part of the interface specification is specifying the allowable values for the arguments and generating allowable return values

• Approach 1: These can be specified in the semantic actions and semantic predicates of the interface grammars

• Approach 2: Can we specify the constraints about the arguments and return values using the grammar rules?– Yes, grammar productions can be used to specify the

structure of most recursive data structures.

Modular Verification of Web Services

• We applied our modular verification approach based on interface grammars to client and server side verification of Web services

Interface Grammars for Web Services

Our approach:

1. A WSDL-to-interface grammar translator automatically generates grammar productions that generate and/or validate XML arguments and return values

2. User adds control flow constraints by modifying the grammar

3. Interface compiler automatically generates a stub for client side verification and a driver for server-side verification

Interface Grammars for Web Services

A Case Study: AWS-ECS

• We performed both client and server side verification for the Amazon E-Commerce Web Service (AWS-ECS) using our approach

• AWS-ECS WSDL specification lists 40 operations – that provide differing ways of searching Amazon’s

product database

• We focused on the following core operations:– ItemSearch, CartCreate, CartAdd, CartModify,

CartGet, CartClear

Client Verification

• For client verification we used a demonstration client provided by Amazon: AWS-ECS Java Sample

• This client does not check any constraints such as– You should not try to insert an item to a shopping cart

before creating a shopping cart• When such requests are sent to AWS-ECS they would

return an error message• Using our approach we can easily check if the client allows

such erroneous requests• We used the Java PathFinder (JPF) model checker

– Falsification time changes with the type of faults we are looking for (data or control errors), changes from 10 to 60 seconds

Web Service Client

Server Stub

AutomaticallyGenerated Stub/Parser

TestDriver

AWS-ECS Java Sample

Hand writtenDriver

Client Verification Setup

• The AWS-ECS Client uses Apache Axis SOAP library• Given a WSDL specification, Apache Axis automatically

generates Java classes as wrappers for SOAP calls• We replaced the Java classes generated by Apache Axis

and forwarded them to our automatically generated parser/server stub

• We wrote a driver which non-deterministically generates input events for the Client, simulating user behavior

• JPF model checker exhaustively explores all non-deterministic choices (both in the driver and the stub)

Input Validation Check

• We checked if the client performs input validation upon receiving an input from the user before passing the corresponding request to the server

• Possible erroneous inputs by the user– Type errors

• For example: Entering a string for a numeric field– Nonsensical data

• For example: Adding a nonexistent item to a nonexisting shopping cart

– Uncorrelated data• For example: Search for an item but then try to insert

another nonexisting one

Input Validation Experiments with JPF

Type Time (sec.) Memory (MB)

Type error 12.5 25

Nonsensical data 11.1 25

Uncorrelated data 20.8 43

Control-Flow Validation Check

• Call sequences that can lead to errors– Such as modifying the contents of a cart after clearing it

• We wrote a driver that:– First initializes the cart and – Then generates a sequence of user events non-

deterministically

• We experimented for different event sequence sizes

Control-Flow Validation Experiments

Type Depth Time (sec.)

Memory (MB)

Errors

First error 2 31.8 43 0

First error 3 64.2 62 1

First error 4 49.6 73 1

First error 5 57.3 73 1

All errors 2 31.8 43 0

All errors 3 77.3 62 2

All errors 4 266.8 112 15

All errors 5 862.6 230 68

AWS-ECS: Server Verification

• Our interface compiler automatically generates a driver that sends sequences of requests to AWS-ECS server and checks that the return values conform to the interface specification

• The driver is a sentence generator– It generates sequences of SOAP requests based on the

interface specification

• We used two algorithms for sentence generation:– A random sentence generation algorithm– Purdom’s algorithm: A directed sentence generation

algorithm that guarantees production coverage

Directed Sentence Generation

• Number of sentences generated: 5

• Average derivation length: 24

• Average number of SOAP requests/responses: 3.8

• Verification time: 20 seconds

Random Sentence Algorithm

0 5 10 15 20 25 30 35 40 45 500%

20%

40%

60%

80%

100%

Percentage of covered productions

Percentage of covered nonterminals

Number of exceutions0 20 40 60 80 100 120

0

50

100

150

200

250

300

350

time in seconds

Number of executions

• Number of sentences generated: 100

• Average derivation length: 17.5

• Average number of SOAP requests/responses: 3.2

Server verification

• We found two errors during server side verification– Errors were discovered within 10 seconds

• These errors indicate a mismatch between the interface specification and the server implementation

• It may mean that we misunderstood the description of the Web service interface

• It may also mean that there is an error in the service implementation

Conclusions

• Modular verification is a necessity

• Interfaces are crucial for modular verification

• Interface grammars provide a new specification mechanism for interfaces

• Interface grammars can be used for automated stub generation leading to modular verification for both client and server for verification of Web Services

THE END

Related Work: Interfaces

• L. de Alfaro and T. A. Henzinger. Interface automata.

• O. Tkachuk, M. B. Dwyer, and C. Pasareanu. Automated environment generation for software model checking.

• A. Betin-Can and T. Bultan. Verifiable concurrent programming using concurrency controllers.

• T. Ball and S. K. Rajamani. SLAM interface specification language.

• G. T. Leavens et al.: JML

Related: Grammar-based Testing

• A. G. Duncan, J. S. Hurchinson: Using attributed grammars to test designs and implementations

• P. M. Maurer: Generating test data with enhanced context free grammars

• P. M. Maurer: The design and implementation of a grammar-based data generator

• E. G. Sirer and B. N. Bershad: Using production grammars in software testing