Supporting Data Consistency in Concurrent Process Execution with Assurance Points and Invariants*

Andrew Courter

Masters Thesis DefenseDepartment of Computer Science

08/24/2010

Committee Members: Dr. Susan Urban (Chair) Dr. Michael Shin Dr. Susan Mengel

*This research is supported by NSF Grant No. CCF-0820152.

Overview of Presentation

Motivation Research Objectives Related Work Background Research for the Use of Invariants Overview of the Invariant Monitoring System A Prototype of the Invariant Monitoring System Testing and Evaluation of the Invariant Monitoring

System Summary and Contributions Future Research

2

Motivation

Service-oriented computing and Web services are becoming a standard approach for developing distributed applications

Traditional distributed transactions have relied on ACID properties to protect data Two-Phase Commit

Serializability

It is not always practical for distributed processes composed of Web services to lock data in service-oriented computing

Need to relax ACID properties and depend on user-defined constraints

Serviceoperation1

Serviceoperation2

Serviceoperation3

Serviceoperation4

Serviceoperation5

Serviceoperation2

Serviceoperation4

Serviceoperation5

……. Serviceoperationm

……. Serviceoperationn

Process1

Process2

Service Provider1

Service Provider2

Service Provider3

3

Assurance Points

Shrestha, R. Using Assurance Points and Integration Rules for Recovery in Service Composition. M.S. Thesis, 2010.

Part of the Decentralized Data Dependency Analysis Project Focused on enhancing a BPEL-like process with user-defined constraints Supports user defined checkpoints at locations within a process Three techniques for variable levels of recovery (APRollback, APRetry, APCC)

Service Composition with AP

Cg1

Cg2

Cg3

AP1

AP2

IRpost

F

T

F

Conditional OperationT

FAP Data

Recovery Actions

APRetry

APRollback

APCC

Alternative Process

T

AP2

IRpre

IRcond

4

S. D. Urban, L. Gao, R. Shrestha,And A. Courter, AchievingRecovery in Service CompositionThrough Assurance Points and Integration Rules, to appear in Proceedings of the InternationalConference on CooperativeInformation Systems (COOPIS), Crete, Greece, October, 2010.

The Invariant Concept

• APs provide checkpoints but cannot monitor conditions between APs

• Enhancing APs with the concept of monitored invariants

• Invariants allow stronger constraint monitoring for a specific duration of execution without locking data

5

Process Specification

Process A...

AP1...

AP2...

Invariant

data changed

condition checked

violated?invoke

recoveryAction

datamonitoring

YN

DEGSinterface

Process B...

Update operations

.

.

.

Process Specification

Research Objectives

Develop the concept of invariants to monitor data in a concurrent environment that needs to enforce user-defined constraints

Extend the AP model to support the expression of invariants. Design a prototype system for registration of invariants and

analysis/extraction of data items to be monitored. Design and prototype a system for monitoring data changes

together with any necessary revisions to the DEGS system to support monitoring capabilities.

Design and prototype a notification and invariant re-evaluation system in response to data changes.

Test and evaluate the invariant re-evaluation system.6

Related Work

Related Work

• Transactional Workflows

(Sheth, 1997)

• Transactional Issues for Web Services

(Papazoglou, 2003)

• Tentative Holding

(Limthanmaphon& Zhang, 2004)

• Promises Approach

(Chen, Fekete, Greenfield, & Jang, 2009)

• Transactional Attitudes

(Mikalsen, Tai, &Rouvellou, 2002)

8

Related Work

• Reservation-Based Approach (Zhao, Moser, &Melliar-Smith, 2009)

• Monitoring Extension to BPEL (Baresi& Guinea, 2005)

• Aspect-Oriented Workflows (A. Charfi&Mezini, 2006) (Wu, et al., 2010)

• The Invariant Concept allows users to define constraints and how long to monitor these constraints

• Express user-defined constraints

• Monitor the constraint conditions

• Specify the monitoring time frame

• Not reservation or promise based

• Optimistic approach to concurrent execution

9

Background Research for the Use of Invariants

Delta-Enabled Grid Services (DEGS)

Deltas

Invoke

Deltas

DeltaGrid

Oracle Database

Delta Repository

OGSA-DAI

Delta-Enabled

Grid Service

Process Execution

Event Processor

Process History Capture SystemFailure

Recovery System

Delta Notification

Key Components◦ Delta

Repository◦ OGSA-DAI◦ Delta

Notification◦ Delta-Enabled

Grid Service◦ Delta Grid

Urban, S. D., Xiao, Y., Blake, L., & Dietrich, S. W. (2009). Monitoring Data Dependencies in Concurrent Process Execution through Delta-Enabled Grid Services. 5(1), pp85-106.

11

Service Composition withAssurance Points (APs)

Basic Use of AP and Integration Rules 12

Service Composition with AP

cg1

cg2

cg3

AP1

AP2

IRpost

F

T

F

Conditional OperationT

FAP Data

APRetry

APRollback

APCC

Alternative Process

T

AP2

IRpre

IRcond

Recovery Actions

Note:

• Condition (c) is always expressed in a negative form (not(C)).

• The expression of a pre-condition, post-condition or any additional condition is optional.

Overview of the Invariant Monitoring System

The Invariant Monitoring System

14

Process1...

AP1 ( Tuple1.A+Tuple2.B > 10::Boolean)

.

.

.

AP2 ( Tuple1.A+Tuple2.B > 10::Boolean)

Process2

.

.

.

Operation 2: Tuple1.A = 5;

Operation 3: Tuple2.B = 2;

.

.

.

T1

TN

T2

T3

Invariant Agent

Delta Analysis Agent

Delta-Enable Grid Service

SourceDatabase Table1

A B ...

.

.

.

.

.

.

Tuple1Tuple2

52

1

2

3

4

5

6

7

9

10

11

12

13

15

DeltaRepository

Relation ObjectID Attribute Old_Val Attribute P_ID OP_ID TimeTable1Table1

Tuple1Tuple2

AB

1012

52

P2P2

OP2OP3

T2T3

. . . . . .8

14

.

.

.

Challenges for the Invariant Monitoring System

Register, Activate, and Deactivate Invariants

Capture and Filter Deltas Against Invariant Conditions

Re-evaluation of Invariants

15

Invariant Specification

CREATE RULE: Invariant_identifier::inv

EVENT: startAP ( endAP, Parameter1, Parameter2….)

CONDITION: rule condition specification

ACTION: recovery procedures

[ON RETRY]: additional/alternative recovery procedures

16

IRinv is expressed as an Event-Condition-Action (ECA) rule.

The action specification is executed if the condition evaluates to true.

During retry action, there is possibility to execute same integration rule for a second time invoke ON RETRY

Hotel Room Monitoring Example

17

Subprocess

BeginTravelPlanning (hotelID, price)

.

.

.

ReadyToBook (...)

create rule HotelRoomMonitoring::invevent: BeginTravelPlanning (ReadyToBook, hotelID, price)condition: (Not exists(select * from Rooms R where R.roomPrice < ‘” + price + “’ and R.roomType = ‘seaview’ and R.hid = “ + hotelID))recoveryAction1: APRetryrecoveryAction2: APRollBack

AP1

AP2

.

.

.

.

.

.

.

.

.

Bank Loan Monitoring Example

18

Subprocess

LoanAppCreation (amount, customerId)

.

.

.

LoanCompletion (customerId)

create rule LoanAmountMonitoring::invevent: LoanAppCreation (LoanCompletion, customerId)

condition: (Not exists(select * from loan where loan.applicantID = ‘” + customerId + “’ and loan.status = ‘pre-qualified’ and loan.amount < (select 10*balance from

account where account.customerID = ‘” + customerId + “’))

recoveryAction1: APRetryrecoveryAction2: APRollBack

AP1

AP2

.

.

.

.

.

.

.

.

.

A Prototype of the Invariant Monitoring System

Monitored Objects

Monitored Objects are acquired from the SQL condition by extracting the table names together with the attributes and relevant conditions.

• Invariant: select * from Rooms R where R.roomPrice < ‘” + price + “’ and R.roomType = ‘seaview’ and R.hid = “ + hotelID

• Monitored Objects:

• (Rooms, roomPrice, <, price)

• (Rooms, roomType, =, ‘seaview’)

• (Rooms, hid, =, hotelID)

• Invariant: select * from loan where loan.applicantID = ‘” + customerId + “’ and loan.status = ‘pre-qualified’ and loan.amount < (select 10*balance from account where account.customerID = ” + customerId + “’)

• Monitored Objects:

• (loan, applicantID, =, customerID)

• (loan, status, =, ‘pre-qualified’)

• (loan, amount, <, ‘calc’)

• (account, balance, =, ‘calc’)

• (account, customerID, =, customerID)20

XML Representation of Invariants

21

<rules><event type="ap" ap="loanAppCreation"> <inv> <ecaRule> <inputs> <input invariantName="loanAmountMonitoringInv" startAP="loanAppCreation" endAP="loanCompletion"/> </inputs> <parameters> <parameter name="customerId"/> <parameter name="amount"/> </parameters> <condition name="loanAmountMonitoring"> <invoke serviceName="SQLEvaluation" serviceLocation="http://localhost:8080/wsrf/services/examples/core/sqlEval/sqlEvalService" operation="main" SQLVariable="(Not exists(select * from loan where loan.applicantID = ‘” + customerId + “’ and loan.status = ‘pre-qualified’ and loan.amount < (select 10*balance from account where account.customerID =‘” + customerId + “’))" curRecovVariable="currentRecovery" outputVariable="result"> <monitoredObjects tableName="loan"> <attributes> <attribute attributeName="status" setValue="pre-qualified" relation="equals"/> <attribute attributeName="applicantID" setValue=customerId relation="equals"/> <attribute attributeName="amount" setValue=”calc” relation="lessthan"/> </attributes> </monitoredObjects> <monitoredObjects tableName="account"> <attributes> <attribute attributeName="applicantID" setValue=customerId relation="equals"/> <attribute attributeName="balance" setValue=”calc” relation="equals"/> </attributes> </monitoredObjects> </invoke> </condition> <actions> <action name="APRetry" targetAP= "startLoan"/> <action name="APRollback"/> </actions> </ecaRule> </inv> </event><rules/>

Invariant Registration

MonitoredObject

-siteID-className

-setValue-propertyName

Invariants

-InvariantIdentifier

-InvariantCondition

* *

-Mid

-InvariantAction

-relation

-siteID-SQLEvalWSLocation

InvariantCondition, MonitoredObjects, siteID, SQLEvalLocation, InvariantAction received in Invariant Agent

SQL Eval WS receives InvariantIdentifer and InvariantCondition and checks condition.

If there are no tuples, process is notified and Invariant is removed. Otherwise, MonitoredObjects are forwarded to Delta Analysis Agent.

Invariant Database Storage

22

Evaluation Web Service

Invid, SQL, list of log tables

InvariantAgent

Create logs Create View

Select Count(*) from

View

Invariant Evaluation Web Service

Tuple Count

Do logs exist?

Does view exist?

N

YY

N

CREATE MATERIALIZED VIEW inv123REFRESH FORCE ON COMMIT ENABLE QUERY REWRITE AS <select statement for invariant>

Materialized ViewEvaluation Web Service

Functionality

23

Extensions to DEGS

No change needed in delta representation for Insert and Delete operations

For Update operation, the DEGS was modified to capture all attributes of a tuple in a delta.

24

Invariant Storage Container

Two hashtables are used, one containing the tablename and attribute, the other containing each Invariant and its related MonitoredObjects (MObj)

25

Overview of the Filtering Process

Check Attributes of Monitored Objects

• Checking Insert/Delete requires conditions in a delta to satisfy all Monitored Object constraints in an Invariant

• Checking Update requires conditions in a delta to satisfy most Monitored Object constraint and violate at least one Monitored Object constraint

• The setValue and Relation values in monitored objects are checked against deltas to determine violations

Filtering Single and Multiple Tables

• Single tables use the tupleCount and numViolations variables in each Invariant and when these are equal to each other the Invariant is re-evaluated

• Multiple tables use the tupleCount and numViolations in addition to a threshold value to determine when a significant number of potential violations have been made.

26

Single Table Filtering

Invariant: “select r.price from room r where r.price < ‘30’ and r.roomType = ‘seaview’ and r.hotelid = ‘234’“

Monitored Objects: [(room, price, <, ‘30’), (room, roomType, =, ‘seaview’), (room, hotelid, =, ‘234’)]

Number of Satisfying Tuples: 1

Case 1: Single table insert

• A tuple satisfying all of the monitored object conditions is inserted into the room table, so the number of tuples is incremented by one.

Case 2: Single table delete

• The only satisfying tuple is deleted by an external process.

• The number of violations are incremented and the number of tuples will equal the number of violations.

Case 3: Single table update (similar to single table delete)

27

Multiple Table Filtering

Invariant: “select r.price from room r, hotel h where r.price < ‘30’ and r.roomType = ‘seaview’ and r.hotelid = h.hotelid and h.state = ‘Texas’“

Monitored Objects: [(room, price, <, ‘30’), (room, roomType, =, ‘seaview’), (hotel, state, =, ‘Texas’)]

Number of Satisfying Tuples: 25 Threshold: 25% 

Case 4: Multiple table insert

• All inserts into multiple table invariants are ignored.

• Inserting tuples can potentially increase the size of the number of tuples that satisfy the invariant condition, but will not cause a violation.

Case 5: Multiple table delete

• 7 tuples from the room table satisfying the invariant condition are deleted one after another by an external process.

• The number of violations are incremented each time and after the seventh deletion the number of violations will be greater than the threshold(7 > .25*25).

Case 5: Multiple table update (similar to multiple table delete) 28

Evaluation of the Invariant Monitoring System

Test Structure

Hotel

-Hotelname

-City

Room

-Roomid-Hotelid

1 *

-Hotelid

-Roomtype-Price

-State

-Hoteltype

Primary focus for evaluation was on execution of the Invariant Evaluation Web Service

The Hotel example was used to demonstrate differences between execution times of the Evaluation Web Service for multiple table monitoring.

The database structure is shown above where multiple Rooms can be related to one Hotel

Hotel Example Database Structure

30

Evaluation of the Invariant Evaluation WSTime Comparisons

Measurement Description Avg Time (Microsec) with

Invariant Removal

Avg Time (Microsec) without

Invariant Removal

Creating Materialized View 249492 269974

EvalWS Total Time 371234 371025

Evaluating from Invariant

Agent

575150 603457

Select from Existing

Materialized View

452 443

EvalWS Total Time

(without creating materialized

view)

107844 117455

Evaluating from Delta Analysis

Agent

618913 292364

Executing Select Query 2143 207931

4x as long

Twice as long to remove

Evaluation of the Invariant Evaluation WS

There exists some overhead when creating the materialized view and when removing the Invariant monitoring.

Using a materialized view for SQL evaluation is more efficient when re-evaluation will potentially be frequent.

32

SUMMARY AND CONTRIBUTIONS

Enhanced Assurance Points with the Invariant Monitoring System Invariant Agent to register Invariants and handle activation and deactivation of multiple

Invariant conditions. Delta Analysis Agent to filter through deltas and determine if monitored object conditions

have been violated. Filtering distinguishes between single and multiple table filters and uses a threshold value to

minimize execution of the invariant evaluation web service.

Designed and Evaluated an Invariant Evaluation Web Service to make use of Materialized Views to handle condition re-evaluations.

Invariants provide a way to monitor data consistency in an environment where the coordinated locking of data items across multiple service executions is not possible, thus providing better support for reliability and maintenance of user-defined correctness conditions among concurrent processes.

33

FUTURE RESEARCH

Using DEGS to monitor the created materialized views

Statistical analysis to determine more accurate threshold values for different invariants.

Enhance the Invariant Evaluation Web Service to push changes to the Invariant Agent

Testing concurrent processes that use APs with Invariant monitoring capabilities

34

INVARIANT MONITORING

SYSTEM

Questions?

Committee Members:

Dr. Susan D. Urban (Chair) Dr. Michael Shin Dr. Susan Mengel

Le Gao, Mary Shuman

ACKNOWLEDGEMENTS

The End

36

Update Algorithm

public boolean updateMonitoring(String MObjectList){

temporary MObj and MObjContainer are createdtokenize input MObjListwhile( there are more tokens ){

if( first token ){

set invID in temp MObjset other attributes to emptyinsert temp MObj into temp MObjContainerincrement token counter

}

set tablename, attribute name, relation and setvalue in temp MObjinsert temp MObj into temp MObjContainer

//Note: all keys are lower case using the .toLowerCase() function//Check if we are already monitoring this table/attr combinationif(table/attribute hashtable contains tablename and attribute name in temp MObj){

get the container containing the invariant IDsif( the invID is not already in this container){

add the invID to the containerupdate tablename/attribute key and container in table/attribute hashtable

}}else{

create new integer containeradd the invID to the containerinsert tablename/attribute key and container into table/attribute hashtable

}increment token counter by 4

}//Finished getting all the monitored objectsinsert tempMObjContainer into the Invariant hashtable

if( table/attribute hashtable's size is 0 ){

return false; //no objects to monitor}else{

return true; //objects to monitor}

} 37

Remove Algorithm

public boolean removeMonitoring(String MObjectList){

temporary InvContainer is createdtokenize input MObjListwhile( there are more tokens ){

if( first token ){

get MObjectList invID increment token counter

}

//Note: all keys are lower case using the .toLowerCase() function//look for this table/attr combinationif(table/attribute hashtable contains tablename and attribute name in temp MObj){

get the container containing the invariant IDsfor( each invID in the container ){

if( current invID == MObjectList invID ){

remove the invID from the container}

if( InvContainer.size() > 0){

update container in table/attribute hashtable}else{

remove key from table/attribute hashtable}

}}else{

return error; //invID not found}increment token counter by 4

}//Now remove the invID from the invToMObjContainerremove invID from Invariant hashtable

return true; //remove successfull} 38

Insert/Delete Filtering

for( each delta ){

for(int i=1;i<=numberColumnsInDelta;i++){

//check if we are monitoring this column of the delta if(hashtable1 contains "tablename/column(i)"){

//get invariant container from hashtable1for( each invariant in the invariant container ){

if( we have already checked this invariant)//exit, we do not want to check an invariant for the same delta twice

//get the MObjContainer for the invariant from hashtable2for( each MObj in the MObjContainer (j) ){

for( each column in the delta (k) ){

if( MObj (j) contains attrib (k) ){

if( attribute (k) violates the condition being monitored by MObj (j) and is not a calculated value )

{//stop checking this MObjContainer because a filter has

been violated}

}}

}if( entire MObjContainer satisfies all object constaints && operation is a delete){

//increment number of violations//calculate threshold (number of tuples * threshold percentage)if(numViolations > threshold && invariant is monitoring multiple tables)

//forward invID to Invariant Agent else if( numViolations >= numTuples && invariant is monitoring one table )

//forward invID to Invariant Agent }else if(entire MObjContainer satisfies all object constaints && operation is a insert &&

invariant monitoring one table)//add one to tupleCount

}}

}}

39

Update Filteringfor( each delta ){

for(int i=1;i<=numberColumnsInDelta;i++){

//check if we are monitoring this column of the delta if(hashtable1 contains "tablename/column(i)"){

//get invariant container from hashtable1for( each invariant in the invariant container ){

if( we have already checked this invariant)//exit, we do not want to check an invariant for the same delta twice

//get the MObjContainer for the invariant from hashtable2for( each MObj in the MObjContainer (j) ){

for( each column in the delta (k) ){

if( MObj (j) contains attrib (k) ){

if( attribute (k) has been changed in this delta ){

if( attribute (k) is a calculated value ){

//flag for violation}else if( the change in attribute (k) violates the condition

being monitored by MObj (j) ){

//flag for violation}

}else //attribute (k) has not been changed in this delta{

if( attribute (k) violates the condition being monitored by MObj (j) )

{//stop checking this MObjContainer because a filter

has been violated}

}}

}}if( entire MObjContainer satisfies all object constaints and violates at least one condition){

//increment number of violations//calculate threshold (number of tuples * threshold percentage)if(numViolations > threshold && invariant is monitoring multiple tables)

//forward invID to Invariant Agent else if( numViolations >= numTuples && invariant is monitoring one table )

//forward invID to Invariant Agent }

}}

}} 40

Update Tuple Count

public boolean updateInvTupleCount(String invariantInfo){

//create tempMObjContainer

//tokenize invariantInfo, extracting the invID and the number of tuples

if( we are monitoring this invID){

//get the MObjContainer of this invID and put it into tempMObjContainer//update the tupleCount with the received number of tuples value//set the number of violations to zero

if( there are no more tuples ) //we want to remove the monitoring{

for( each monitored object in tempMObjContainer ){

//add monitored object to string to send to removeMonitoring function}//send entire string of monitored objects to the removeMonitoring function

}}else{

return false;}return true;

}

41