IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 8, NO. 1 ...ajanta.cs.umn.edu/papers/tsc-2015.pdf · Scalable Transaction Management with Snapshot Isolation for NoSQL Data Storage Systems

Scalable Transaction Management withSnapshot Isolation for NoSQL Data

Storage SystemsVinit Padhye and Anand Tripathi, Fellow, IEEE

Abstract—We address the problem of building scalable transaction management mechanisms for multi-row transactions onkey-value storage systems, which are commonly termed as NoSQL systems. We develop scalable techniques for transactionmanagement utilizing the snapshot isolation (SI) model. Because the SI model can lead to non-serializable transaction executions,we investigate two conflict detection techniques for ensuring serializability. To support scalability, we investigate systemarchitectures and mechanisms in which the transaction management functions are decoupled from the storage system andintegrated with the application-level processes. We present two system architectures and demonstrate their scalability under thescale-out model of cloud computing platforms. In the first system architecture all transaction management functions are executedin a fully decentralized manner by the application processes. The second architecture is based on a hybrid approach in whichthe conflict detection functions are performed by a dedicated service. We perform a comparative evaluation of these architecturesusing the TPC-C benchmark and demonstrate their scalability.

Index Terms—Transaction management, scalable services, cloud data management systems

Ç

1 INTRODUCTION

THE cloud computing platforms enable building scalableservices through the scale-out model by utilizing the

elastic pool of computing resources provided by suchplatforms. Typically, such services require scalable man-agement of large volumes of data. It has been widelyrecognized that the traditional database systems based onthe relational model and SQL do not scale well [1], [2]. TheNoSQL databases based on the key-value model such asBigtable [1] and HBase [3], have been shown to be scalablein large scale applications. However, unlike traditionalrelational databases, these systems typically do not providemulti-row serializable transactions, or provide such trans-actions with certain limitations. For example, HBase andBigtable provide only single-row transactions, whereassystems such as Google Megastore [4] and G-store [5]provide transactions only over a particular group ofentities. These two classes of systems, relational andNoSQL based systems, represent two opposite points inthe scalability versus functionality space. For certainapplications, such as web search, email, and socialnetworking, such limited support for transactions in key-value based storage models has been found to be adequate.However, many applications such as online shoppingstores, online auction services, financial services, whilerequiring high scalability and availability, still need certain

strong transactional consistency guarantees. For example,an online shopping service may require ACID (atomicity,consistency, isolation, and durability) [6] guarantees forperforming payment operations. Thus, providing ACIDtransactions for NoSQL data storage system is an importantproblem.

We present here scalable architecture models forsupporting multi-row serializable transactions for key-value based NoSQL data storage systems. Our approach isbased on decentralized and decoupled transaction man-agement where transaction management functions aredecoupled from the storage system and performed by theapplication-level processes themselves, in decentralizedmanner. Fig. 1 illustrates the seminal elements of thisapproach. A service hosted in a cloud datacenter environ-ment is accessed by clients over the Internet. The servicecreates application level processes for performing servicefunctions. These processes belong to the trusted domain ofthe deployed service. The application-specific data of theservice is stored in a key-value based storage system in thedatacenter. In our approach, all transaction managementfunctionsVsuch as concurrency control, conflict detectionand atomically committing the transaction updatesVareperformed by the application processes themselves indecentralized manner. These functions are provided tothe application in the form of library functions. Themetadata necessary for transaction management is storedin the underlying key-value based storage.

In realizing the transaction management model de-scribed above, the following issues need to be addressed.These issues are related to the correctness and robustnessof the decentralized transaction management protocol. Inour approach, the commit protocol is performed in varioussteps by individual application processes, and the entire

. The authors are with the Department of Computer Science, University ofMinnesota, Minneapolis, MN 55455 USA.

Manuscript received 19 June 2013; revised 31 Aug. 2013; accepted 11 Sept.2013. Date of publication 30 Sept. 2013; date of current version 6 Feb. 2015.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TSC.2013.47

1939-1374 � 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 8, NO. 1, JANUARY-FEBRUARY 2015 121

sequence of steps is not performed as a single atomicaction. Not performing all steps of the commit protocol asone atomic action raises a number of issues. The transactionmanagement protocol should ensure the transactionalconsistency when multiple processes execute the protocolsteps concurrently. Any of these steps may get interrupteddue to process crashes or delayed due to slow execution. Toaddress this problem, the transaction management proto-col should support a model of cooperative recovery; anyprocess should be able to complete any partially executedsequence of commit/abort actions on behalf of anotherprocess that is suspected to be failed. Any number ofprocesses may initiate the recovery of a failed transaction,and such concurrent recovery actions should not cause anyinconsistencies.

Based on the decoupled and decentralized transactionmanagement model, we develop two system architecturesand demonstrate their scalability using the TPC-C bench-mark [7]. The first architecture is fully decentralized, inwhich all the transaction management functions areperformed in decentralized manner. The second architec-ture is a hybrid model in which only the conflict detectionfunctions are performed using a dedicated service and allother transaction management functions are performed indecentralized manner. We refer to this as service-basedarchitecture. In developing these architectures we utilize thesnapshot isolation (SI) [8] model. The SI model is attractivefor scalability, as noted in [8], since transactions read from asnapshot, the reads are never blocked due to write locks,thereby providing more concurrency. However, the snap-shot isolation model does not guarantee serializability [8],[9]. Our work addresses this issue and develops techniquesfor ensuring serializability of SI-based transactions forNoSQL data storage systems.

Various approaches [10], [11], [12] have been proposedin the past, in the context of relational database manage-ment systems (RDBMS), to ensure transaction serializabil-ity under the SI model. Some of these approaches [10], [11]are preventive in nature as they prevent potential conflict

dependency cycles by aborting certain transactions, but theymay abort transactions that may not necessarily lead toserialization anomalies. On the other hand, the approachpresented in [12] detects dependency cycles and abortsonly the transactions necessary to eliminate a cycle.However this approach requires tracking of conflictdependencies among all transactions and checking fordependency cycles, and hence it can be expensive. Wepresent here results of our investigation of these ap-proaches in the context of key-value based NoSQL datastorage systems.

The major contributions of our work are the following.We present and evaluate two system architectures forproviding multi-row transactions using snapshot isolation(SI) on NoSQL databases. Furthermore, we extend the SIbased transaction model to support serializable transac-tions. We demonstrate the scalability of our approach usingthe TPC-C benchmark. Our work demonstrates thattransaction serializability guarantees can be supported ina scalable manner on key-value based storage systems.Using the transaction management techniques presentedhere, the utility of key-value based cloud data managementsystems can be extended to applications requiring strongtransactional consistency.

The rest of the paper is organized as follows. In the nextsection we discuss the related work to present the contextand significance of our contributions. In Section 3, weprovide an overview of the snapshot isolation model.Section 4 presents the framework that we have developedfor transaction management and highlight the variousdesign issues in this framework. Section 5 presents ourdecentralized design for supporting the basic SI basedtransactions. In Section 6, we discuss how the basic SI modelis extended to provide serializability. Section 7 discussesthe service-based architecture. Section 8 presents theevaluations of the scalability of the proposed techniques.Section 9 evaluates the performance of the cooperativerecovery mechanisms. The conclusions are presented in thelast section.

Fig. 1. Decentralized and decoupled transaction management model.

IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 8, NO. 1, JANUARY-FEBRUARY 2015122

2 RELATED WORK

In recent years, researchers have recognized the scalabilitylimitations of relational databases, and to address thescalability and availability requirements, various systemsbased on the key-value data model have been developed[1], [13], [14], [4]. Systems such as SimpleDB [14] andCassandra [15] provide weak consistency. Bigtable [1] andits open-source counterpart HBase [3] provide strong dataconsistency but provide only single-row transactions.Other systems provide transactions over multiple rowswith certain constraints. For example, Megastore [4], andG-store [5] provide transactions over a group of entities. InElasTraS [16] and VoltDB [17] ACID transactions aresupported only over a single partition. Sinfonia [18] andGranola [19] systems provide restricted forms of storagelevel multi-item transactions. The Calvin system [20]supports ACID transactions by using deterministic order-ing of transactions, but this approach requires priorknowledge of a transaction’s read/write sets and it ismainly aimed for in-memory databases. CloudTPS [21]provides a design based on a replicated transactionmanagement layer which provides ACID transactionsover multiple partitions. An approach based on decouplingtransaction management from data storage using a centraltransaction component is proposed in [22]. In contrast tothese approaches, our design presents a decentralizedtransaction management protocol, wherein the transactionmanagement functions are performed by the applicationprocess itself.

Other researchers have proposed techniques for pro-viding multi-row snapshot isolation transactions [23], [24].The Percolator system [23] addresses the problem ofproviding SI-based transaction for Bigtable [1]. However,it does not address the problem of ensuring serializabilityof transactions. Moreover, it is intended for offlineprocessing of data. The work presented in [24] does notadequately address issues related to recovery and robust-ness when some transaction fails.

The problem of transaction serializability in snapshot-isolation model has been extensively studied [9], [10], [11],[12], [25]. The work in [9] characterizes the conditionsnecessary for non-serializable transaction executions in theSI model. Based on this theory, many approaches havebeen suggested to avoid serialization anomalies in SI.These approaches include static analysis of programs [26]as well as runtime detection of anomalies [10], [11], [12],[25]. Technique presented in [10], [11], [25] tend to bepessimistic and can lead to unnecessary aborts. PSSI

approach [12] avoids such problems and aborts only thetransactions that lead to serialization anomalies. However,these approaches were developed in the context oftraditional relational databases and, except in the case of[25], provided solutions only for centralized databases.

3 BACKGROUND: SNAPSHOT ISOLATION MODELAND SERIALIZABILITY ISSUES

Snapshot isolation (SI) based transaction execution modelis a multi-version based approach utilizing optimisticconcurrency control [27]. In this model, when a transactionTi commits, it is assigned a commit timestamp TSic, which isa monotonically increasing sequence number. The committimestamps of transactions represent the logical order oftheir commit points. For each data item modified by acommitted transaction, a new version is created with thetimestamp value equal to the commit timestamp of thetransaction.

When a transaction Ti’s execution starts, it obtains thetimestamp of the most recently committed transaction. Thisrepresents the snapshot timestampTSis of the transaction. Aread operation by the transaction returns the most recentcommitted version up to this snapshot timestamp, there-fore, a transaction never gets blocked due to any writelocks.

A transaction Ti commits only if there exists nocommitted transaction concurrent with Ti which hasmodified any of the items in Ti’s write-set. That meansthere exists no committed transaction Tj such thatTSis G TS

jc G TS

ic and Tj has modified any of the items in

Ti’s write-set. Thus, if two or more concurrent transactionshave a write-write conflict, then only one of them isallowed to commit. It is possible that a data item in theread-set of a transaction is modified by another concurrenttransaction, and both are able to commit. An anti-dependency [28] between two concurrent transactions Tiand Tj is a read-write (rw) dependency, denoted by Ti !

rwTj,

implying that some item in the read-set of Ti is modified byTj. This dependency can be considered as an incomingdependency for Tj and an outgoing dependency for Ti. Thisis the only kind of dependency that can arise in the SImodel between two concurrent transactions. There areother kinds of dependencies, namely write-read (wr) andwrite-write (ww), that can exist between two non-concurrenttransactions.

Fekete et al. [9] have shown that a non-serializableexecution must always involve a cycle in which there aretwo consecutive anti-dependency edges of the formTi !

rwTj !

rwTk. In such situations, there exists a pivot

transaction [9] with both incoming and outgoing anti-dependencies. Fig. 2 shows an example of a pivottransaction. In this example, T2 is the pivot transaction. Inthe context of traditional RDBMS, several techniques [10],[11], [12], [25] have been developed utilizing this fact toensure serializability. We investigated the following twoapproaches for implementing serializable transactions onkey-value based storage systems.

. Cycle Prevention Approach: When two concurrenttransactions Ti and Tj have an anti-dependency, one

Fig. 2. Pivot transaction.

PADHYE AND TRIPATHI: SCALABLE TRANSACTION MANAGEMENT WITH SNAPSHOT ISOLATION 123

of them is aborted. This ensures that there can neverbe a pivot transaction, thus guaranteeing serial-izability. In the context of RDBMS, this approachwas investigated in [10].

. Cycle Detection Approach: A transaction is abortedonly when a dependency cycle is detected involvingthat transaction during its commit protocol. Thisapproach is conceptually similar to the technique[12] investigated in the context of RDBMS.

The conflict dependency checks in the above two ap-proaches are performed in addition to the check for write-write conflicts required for the basic SI model. Weimplemented and evaluated the above approaches inboth the fully decentralized model and the service-basedmodel. The cycle prevention approach can sometimes aborttransactions that may not lead to serialization anomalies.The cycle detection approach aborts only the transactionsthat can cause serialization anomalies but it requirestracking of all dependencies for every transaction andmaintaining a dependency graph to check for cycles.

4 A FRAMEWORK FOR DECENTRALIZEDTRANSACTION MANAGEMENT

We present here a framework for decentralized transactionmanagement using snapshot isolation in key-value baseddata storage systems. Implementing SI based transactionsrequires mechanisms for performing the following actions:

1. reading from a consistent committed snapshot;2. allocating commit timestamps using a global se-

quencer for ordering of transactions;3. detecting write-write conflicts among concurrent

transactions; and4. committing the updates atomically and making

them durable. Additionally, to ensure serializabilitywe also need to detect or prevent serializationanomalies as discussed above.

In our approach of decoupled and decentralized trans-action management, the transaction management functionsdescribed above are decoupled from the data storagesystems and performed by the application processesthemselves in decentralized manner. The transactionmanagement metadata required for performing thesefunctions is also stored in the key-value storage system.This is to ensure the reliability of this metadata andscalability of transaction management functions whichrequire concurrent access to this metadata.

A transaction execution goes through a series of phasesas shown in Fig. 3. In the active phase, it performs read/

write operations on data items. The subsequent phases arepart of the commit protocol of the transaction. Forscalability, our goal is to design the commit protocol suchthat it can be executed in highly concurrent manner by theapplication processes. We also want to ensure that after thecommit timestamp is issued to a transaction, the timerequired for commit be bounded, since a long commitphase of the transaction can potentially block the progressof other conflicting transactions with higher timestamps.Thus, our goal is to perform as many commit protocolphases as possible before acquiring the commit timestamp.We discuss below the various issues that arise in utilizingthis approach.

4.1 Timestamps ManagementIn the decentralized model the steps in the commit protocolare executed concurrently by the application processes.Because these steps cannot be performed as a single atomicaction, a number of design issues arise as discussed below.There can be situations where several transactions haveacquired commit timestamps but their commitment statusis not yet known. We also need to make sure that even if atransaction has made its update to the storage system, theseupdates should not be made visible to other transactionsuntil the transaction is committed. Therefore, we need tomaintain two timestamp counters: GTS (global timestamp)which is the latest commit timestamp assigned to atransaction, and STS (stable timestamp), which is thelargest timestamp such that all transactions with committimestamp up to this value are either committed or abortedand all the updates of the committed transactions arewritten to the global storage. An example shown in Fig. 4illustrates the notion of GTS and STS. In this example, STSis advanced only up to sequence number 16 because thecommit status of all the transactions up to sequencenumber 16 is known, however, the commit status of thetransaction with sequence number 17 is not yet known.When a new transaction is started, it uses the current STSvalue as its snapshot timestamp. We first experimentedwith using the key-value storage itself to store thesecounter values. However, we found this approach to beslow, and therefore we use a dedicated service which werefer to as TimestampService for maintaining these countervalues.

4.2 Eager vs Lazy Update ModelAn important design issue that arises is when should atransaction write its updates to the global key-valuestorage. We find two distinct approaches with differentperformance tradeoffs as discussed below. We characterizethem as eager and lazy update models. In the eager updatemodel, a transaction writes its updates to the globalstorage during its active phase, before acquiring its commit

Fig. 3. Transaction protocol phases.Fig. 4. STS and GTS counters.


timestamp, whereas in the lazy approach all writes areperformed after acquiring the commit timestamp.

In the lazy update approach, the time for executing thecommit protocol can become arbitrarily long depending onthe size of the data-items to be written. A long commitphase of a transaction would potentially delay the commitdecisions of other concurrent and conflicting transactionsthat have higher commit timestamps. This may affecttransaction throughput and system scalability, but it hasthe advantage that the writes are performed only when thetransaction is certain to commit.

In the eager update approach the data is written to theglobal storage during the active phase, i.e. prior to thecommit protocol execution, thereby reducing the executiontime for the commit protocol. Also, the transactions canperform their writes overlapped with computation duringthe active phase. The eager update scheme is attractivebecause its commit protocol execution time does notsignificantly depend on the size of the write-set of thetransaction. Also, it facilitates the roll-forward of atransaction that fails during its commit, since its updateswould be already present in the key-value datastore. Dueto these advantages we choose to adopt the eager updatemodel instead of the lazy update model.

Implementing the eager update model requires main-taining uncommitted data versions in the storage. For suchdata versions, we cannot use the transaction’s committimestamp as the version number because it is not knownduring the active phase. Therefore, in the commit protocolthese data versions need to be mapped to the transaction’scommit timestamp. Moreover, ensuring the isolationproperty requires that such uncommitted versions shouldnot be visible until the transaction commits.

4.3 Transaction ValidationThe SI model requires checking for write-write conflictsamong concurrent transactions. This requires a mechanismto detect such conflicts and a method to resolve conflicts byallowing only one of the conflicting transactions to commit.When two or more concurrent transactions conflict, thereare two approaches to decide which transaction should beallowed to commit. The first approach is called first-committer-wins (FCW)[27], in which the transaction with thesmallest commit timestamp is allowed to commit. In thisapproach, conflict checking can only be performed by atransaction after acquiring its commit timestamp. Thisenforces a sequential ordering on conflict checking basedon the commit timestamps. This would force a youngertransaction to wait for the progress of all the oldertransactions, thereby limiting concurrency. In contrast, inthe second approach, which is called first-updater-wins(FUW) [9], conflict detection is performed by acquiringlocks on write-set items and in case of conflictingtransactions the one that acquires the locks first is allowedto commit. The FUW approach appears more desirablebecause the conflict detection and resolution can beperformed before acquiring the commit timestamp, there-by reducing any sequential ordering based on committimestamps and reducing the time required for executingthe commit protocol. Therefore, we chose to adopt the FUWapproach for conflict detection.

4.4 Cooperative RecoveryThere are two problems that arise due to transactionfailures. A failed transaction can block progress of otherconflicting transactions. A failure of a transaction afteracquiring commit timestamp stalls advancement of the STScounter, thereby forcing the new transactions to use oldsnapshot time, which may likely result in greater abortsdue to write-write conflicts. Thus, an appropriate timeoutmechanism is needed to detect stalled or failed transactionsand initiate their recovery. The cooperative recoveryactions for a failed transaction are triggered in twosituations: 1) a conflicting transaction is waiting for thecommit of a failed transaction, and 2) the STS advancementhas stalled due to a failed transaction that has acquired acommit timestamp. The recovery actions in the firstsituation are performed by any of the conflicting transac-tions, whereas the failures of the second kind are detectedand recovery actions are performed by any applicationlevel process or by a dedicated system level process. If atransaction fails before acquiring a commit timestamp, thenit is aborted, otherwise the transaction is committed androlled-forward to complete its commit protocol.

5 IMPLEMENTATION OF THE BASIC SI MODEL

We first describe the metadata that needs to be maintainedin the global storage for transaction management. We thendescribe the various steps in transaction managementprotocol performed by the application processes. We alsodescribe the cooperative recovery protocol for dealing withtransaction failures.

5.1 Storage System RequirementsWe first identify the features of the key-value data storagesystem that are required for realizing the transactionmanagement mechanisms presented here. The storagesystem should provide support for tables and multiplecolumns per data item (row), and primitives for managingmultiple versions of data items with application-definedtimestamps. It should provide strong consistency forupdates [29], i.e., when a data item is updated, anysubsequent reads should see the updated value. Moreover,for the decentralized architecture, we require mechanismsfor performing row-level transactions involving any num-ber of columns. Our implementation is based on HBase [3],which meets these requirements.

5.2 Transaction Data Management ModelFor each transaction, we maintain in the global storage thefollowing information: transaction-id ðtidÞ, snapshot time-stamp ðTSsÞ, commit timestamps TSc, write-set informa-tion, and current status. This information is maintained in atable named TransactionTable in the global storage, asshown in Fig. 5. In this table, tid is the row-key of the tableand other items are maintained as columns. The column‘out-edges’ is used to record information related tooutgoing dependency edges, which is required only inthe cycle detection approach. To ensure that the Transac-tionTable does not become the bottleneck, we set the tableconfiguration to partition it across all the HBase servers.The data distribution scheme for HBase is based on


sequential range partitioning. Therefore, if we generatetransaction ids sequentially it creates a load balancingproblem since all the rows in TransactionTablecorresponding the currently running transactions will bestored only at one or few HBase servers. Therefore, to avoidthis problem we generate transaction ids randomly.

For each application data table, hereby referred asStorageTable, we maintain the information related to thecommitted versions of application data items and lockinformation, as shown in Fig. 6. An application may havemultiple such storage tables. Since we adopt the eagerupdate model, uncommitted versions of data items alsoneed to be maintained in the global storage. A transactionwrites a new version of a data item with its tid as theversion timestamp. These version timestamps then need tobe mapped to the transaction commit timestamp TSc whentransaction commits. This mapping is stored by writing tidin a column named committed-version with version time-stamp as TSc. The column ‘wlock’ in the StorageTable isused to detect write-write conflicts, whereas columns‘rlock,’ ‘read-ts,’ and ‘readers’ are used in detecting read-write conflicts for serializability, as discussed in the nextsection.

5.3 Transaction Management Protocol for BasicSI Model

A transaction Ti begins with the execution of the startphase protocol shown in Algorithm 1. It obtains itstransaction-id ðtidÞ and snapshot timestamp ðTSsÞ fromTimestampService. It then inserts in the TransactionTable anentry: htid; TSs; status ¼ activei and proceeds to the activephase. For a write operation on an item (specified by rowand column keys), following the eager update model, thetransaction creates a new version in the StorageTable usingtid as the version timestamp. The transaction also main-tains its own writes in a local buffer to support read-your-

own-writes consistency. A read operation for the data itemsnot contained in the write-set is performed by firstobtaining, for that data item, the latest version of thecommitted-version column in the range ½0; TSs�. This givesthe tid of the transaction that wrote the latest version of thedata item according to the transaction’s snapshot. Thetransaction then reads data specific columns using this tidas the version timestamp.

Algorithm 1 Execution Phase for transaction Ti:

Start Phase:1: tidi get a unique tid from TimestampService2: TSis get current STS value from TimestampService3: insert tid, TSis information in TransactionTable.

Active Phase:Read item:/� item is a row-key �/

1: tidR read value of the latest version in the range½0; TSis�) of the ‘‘committed version’’ column for item

2: read item data with version tidR3: add item to the read-set of Ti

Write item:1: write item to StorageTable with version timestamp ¼ tidi2: add item to the write-set of Ti

A transaction executes the commit protocol as shown inAlgorithm 2. At the start of each phase in the commitprotocol it updates its status in the TransactionTable toindicate its progress. All status change operations areperformed atomically and conditionally, i.e., permittingonly the state transitions shown in Fig. 3. The transactionfirst updates its status in the TransactionTable to validationand records its write-set information, i.e. only the item-identifiers (row keys) for items in its write-set. Thisinformation is recorded for facilitating the roll-forward ofa failed transaction during its recovery. The transactionperforms conflict checking by attempting to acquire writelocks on items in its write-set as described below. If acommitted newer version of the data item is alreadypresent, then it aborts immediately. If some transaction Tjhas already acquired a write lock on the item, then Ti abortsif tidj G tidi, else it waits for Tj to either commit or abort.This wait/die scheme is used to avoid deadlocks andlivelocks. The conflict checking operations for a singleitem, shown by lines 7-12 in the algorithm, are performedas a single atomic action using the row level transaction

Fig. 6. StorageTable structure.

Fig. 5. TransactionTable structure.


feature provided by HBase. On acquiring all locks, thetransaction proceeds to the commit-incomplete phase.

Algorithm 2 Commit protocol executed by transaction Tifor Basic SI model.

Validation phase:1: if status ¼ active then2: status validation3: end if4: insert write-set information in TransactionTable5: for all item 2 write-set of Ti do6: [ begin row level transaction:7: if any committed newer version for item is created

then abort8: if item is locked then9: if lock-holder’s tid G tidi, then abort else wait

10: else11: acquire lock on item by writing tidi in lock column12: end if13: :end row level transaction ]14: end for

Commit-Incomplete phase:1: if status ¼ validation then2: status commit-incomplete3: else abort4: TSic get commit timestamp from TimestampService5: for all item 2 write-set of Ti do6: insert TSic ! tidi mapping in the StorageTable and

release lock on item7: end for8: status commit-complete9: notify completion and provide TSic to TimestampService

to advance STSAbort phase:

1: for all item2 write-set of Ti do2: if Ti has acquired lock on item, then release the lock.3: delete the temporary version created for item by Ti4: end for

Once Ti updates its status to commit-incomplete, anyfailure after that point would result in its roll-forward. Thetransaction now inserts the ts! tid mappings in thecommitted-version column in the StorageTable for the itemsin its write-set and changes its status to commit-complete. Atthis point the transaction is committed. It then notifies itscompletion to TimestampService and provides its committimestamp TSic to advance the STS counter. The updatesmade by Ti become visible to any subsequent transaction,after the STS counter is advanced up to TSic. If thetransaction is aborted, then it releases all the acquiredlocks and deletes the versions it has created.

5.4 Cooperative Recovery ProtocolWhen a transaction Ti is waiting for the resolution of thecommit status of some other transaction Tj, it periodicallychecks Tj’s progress. If the status of Tj is not changedwithin a specific timeout value, Ti suspects Tj has failed. IfTj has reached commit-incomplete phase, then Ti performsroll-forward of Tj by completing the commit-incompletephase of Tj using the write-set information recorded by Tj.

Otherwise, Ti marks Tj as aborted, acquires the conflictinglock, and proceeds further with the next step in its owncommit protocol. In this case, the cleanup actions, such asreleasing other locks held by the aborted transaction anddeletion of temporary versions created by the transactions,can be performed lazily if if does not block any othertransaction. The cooperative recovery actions are alsotriggered when the STS counter cannot be advancedbecause of a gap created due to a failed transaction. Inthis case, the recovery is triggered if the gap between STSand GTS exceeds beyond some limit. These recoveryactions are triggered by the TimestampService itself basedon the gap between STS and GTS.

In the above mechanism, setting the proper timeoutvalue is crucial. Setting a high timeout value will causedelays in detecting failures and thus potentially blockingconflicting transactions for a long time. When a transactionfails after acquiring commit timestamp, its timely recoveryis crucial since it can block the advancement of STS. On theother hand, setting a low timeout value is also notdesirable, since it can cause aborts of transactions thathave not actually failed. We refer to these as false aborts. Theappropriate timeout value depends on the average timetaken by a transaction to complete its commit protocol. InSection 9, we present detailed evaluation of this aspect.

6 DECENTRALIZED MODEL FOR SERIALIZABLESI TRANSACTIONS

We now describe how the decentralized model for the basicsnapshot isolation is extended to support serializabletransaction execution using the cycle prevention and cycledetection approaches discussed in Section 3.

6.1 Implementation of the CyclePrevention Approach

The cycle prevention approach aborts a transaction whenan anti-dependency among two concurrent transactions isobserved. This prevents a transaction from becoming apivot. One way of doing this is to record for each itemversion the tids of the transactions that read that versionand track the read-write dependencies. However, this can beexpensive as we need to maintain a list of tids per item anddetect anti-dependencies for all such transactions. To avoidthis, we detect the read-write conflicts using a lockingapproach. During the validation phase, a transactionacquires a read lock for each item in its read-set. Read lockson an item are acquired in shared mode. A transactionacquires (releases) a read lock by incrementing (decrement-ing) the value in a column named rlock in the StorageTable.

The commit protocol algorithm for the cycle-preventionapproach is presented in Algorithm 3. An anti-dependencybetween two concurrent transactions can be detected eitherby the writer transaction or the reader transaction. We firstdescribe how a writer transaction can detect a read-writeconflict with any other concurrent reader transaction.During the validation phase, a writer transaction checksfor the presence of a read lock for an item in its write-set atthe time of attempting to acquire a write lock on that item.The transaction is aborted if the item is already read locked.Note that we need to detect read-write conflicts only among


concurrent transactions to detect anti-dependencies. Thisraises an issue that a concurrent writer may miss detectinga read-write conflict if it attempts to acquire a write lockafter the conflicting reader transaction has committed andits read lock has been released. To avoid this problem, areader transaction records its commit timestamp, in acolumn named ‘read-ts’ in the StorageTable, while releasinga read lock acquired on an item. A writer checks whetherthe timestamp value written in the ‘read-ts’ column isgreater than its snapshot timestamp, which indicates thatthe writer is concurrent with a committed reader transac-tion. A reader transaction checks for the presence of a writelock or a newer committed version for an item in its read-set to detect read-write conflicts. Otherwise, it acquires aread lock on the item.

Algorithm 3 Commit protocol for cycle prevention approach.

Validation phase:1: for all item 2 write-set of Ti do2: [ begin row-level transaction:3: read the ‘committed version,’ ‘wlock,’ ‘rlock,’ and

‘read-ts’ columns for item4: if any committed newer version is present, then abort5: else if item is already locked in read or write mode,

then abort6: else if ‘read-ts’ value is greater than TSis, then abort.7: else acquire write lock on item8: :end row-level transaction]9: end for

10: for all item 2 read-set of Ti do11: [ begin row-level transaction:12: read the ‘committed version’ and ‘wlock’ columns for

item13: if any committed newer version is created, then abort14: if item is already locked in write mode, then abort.15: else acquire read lock by incrementing ‘rlock’ column

for item.16: :end row-level transaction]17: end for18: execute commit-incomplete phase shown in Algorithm 219: for all item 2 read-set of Ti do20: [ begin row-level transaction:21: release read lock on item by decrementing ‘rlock’

column22: if read-ts G TSic then23: read-ts TSic24: end if25: :end row-level transaction]26: end for27: status commit-complete28: notify completion and provide TSic to TimestampService

to advance STS

During the commit-incomplete phase, Ti releases theacquired read locks and records its commit timestamps inthe ‘read-ts’ column for the items in its read-set. Since therecan be more than one reader transactions for a particulardata item version, it is possible that some transaction hasalready recorded a value in the ‘read-ts’ column. In this

case, Ti updates the currently recorded value only if it isless than TSic. The rationale behind the logic for updatingthe ‘read-ts’ value is as follows. For committed transactionsT1; T2; . . . ; Tn that have read a particular data item version,the ‘read-ts’ column value for that item version wouldcontain the commit timestamp of transaction Tk ðk � nÞ,such that Tk is the transaction with the largest committimestamp in this set of transactions. An uncommittedwriter transaction Tj that is concurrent with any transactionin the set T1; T2; . . . ; Tn must also be concurrent with Tk i.e.,TSjs G TS

kc , since Tk has the largest commit timestamp. Thus

Tj will detect the read-write conflict by observing that the‘read-ts’ value is larger than its snapshot timestamp.

6.2 Implementation of the CycleDetection Approach

The cycle detection approach requires tracking all depen-dencies among transactions, i.e., anti-dependencies (bothincoming and outgoing) among concurrent transactions,and write-read and write-write dependencies among non-concurrent transactions. We maintain this information inthe form of a dependency serialization graph (DSG) [9], in theglobal storage. Since an active transaction may form depen-dencies with a certain committed transaction, we need toretain information about such transactions in the DSG.

A challenge in this approach is to maintain thedependency graph as small as possible by frequentlypruning to remove those committed transactions that cannever lead to any cycle in the future. For detectingdependencies, we record in StorageTable (in a columnnamed ‘readers’), for each version of an item, a list oftransaction-ids that have read that item version. Moreover,for each transaction Ti, we maintain its outgoing depen-dency edges as the list of tids of the transactions for whichTi has an outgoing dependency edge. This informationcaptures the DSG structure.

In the transaction protocol, we include an additionalphase called DSGupdate, which is performed before thevalidation phase. In the DSGupdate phase, along with thebasic write-write conflict check using the locking techniquediscussed in Section 5, a transaction also detects depen-dencies with other transactions based on its read-write sets.If a transaction Ti has outgoing dependency of any kindwith transaction Tj, it records Tj’s transaction-id in the ‘out-edges’ column in its TransactionTable entry. In thevalidation phase, the transaction checks for a dependencycycle involving itself, by traversing the outgoing edgesstarting from itself. If a cycle is detected, then thetransaction aborts itself to break the cycle.

7 SERVICE-BASED MODEL

We observed that the decentralized approach inducesperformance overheads due to the additional read andwrite requests to the global storage for acquiring andreleasing locks. Therefore, we evaluated an alternativeapproach of using a dedicated service for conflict detection.In the service-based approach, the conflict detection servicemaintains in its primary memory the information requiredfor conflict detection. A transaction in its commit phase sendsits read/write sets information and snapshot timestamp


value to the conflict detection service. We designed thisservice to support conflict detection for the basic-SI modeland the cycle prevention/detection approaches for serial-izable transaction. Based on the particular conflict detectionapproach, the service checks if the requesting transactionconflicts with any previously committed transaction or not. Ifno conflict is found, the service obtains a commit timestampfor the transaction and sends a ‘commit’ response to thetransaction process along with the commit timestamp,otherwise it sends ‘abort’. Before sending the response, theservice updates the transaction’s commit status in theTransactionTable in the global storage.

The transaction commit phase executed using thisapproach is presented below in Algorithm 4. Note thatthis dedicated service is used only for the purpose ofconflict detection and not for the entire transactionmanagement, as done in [21], [22]. The other transactionmanagement functions, such as getting the appropriatesnapshot, maintaining uncommitted versions, and ensur-ing the atomicity and durability of updates when atransaction commits are performed by the application levelprocesses. For scalability and availability, we designed thisservice as a replicated service as described below.

Algorithm 4 Commit algorithm executed by Ti in Service-Based approach.

1: update status to Validation in TransactionTable providedstatus ¼ Active

2: insert write-set information in TransactionTable3: send request to conflict detection service with write-set

information and TSis4: if response ¼ commit then5: TSic commit timestamp returned by the service6: execute CommitIncomplete phase as in Algorithm 2

except for step 27: else8: execute Abort phase as in Algorithm 29: end if

7.1 Replicated Service DesignThe conflict detection service is implemented as a group ofprocesses. The data item space is partitioned across thereplicas using a hashing based partitioning scheme. Thus, aservice replica is responsible for detecting conflicts for a setof data items. A service replica stores, for each data item it isresponsible for, the information necessary to detect conflictsfor that data item. For conflict detection, each replicamaintains an in-memory table called as ConflictTable, whichcontains following information for each data item:

1. commit timestamp of the latest committed transac-tion that has modified the item (write-ts),

2. commit timestamp of the latest committed transac-tion that has read the item (read-ts),

3. lock-mode (read-mode, write-mode, or unlocked),and

4. writer-tid (in case of write lock) and list of reader-tids (in case of read lock).

The validation for a transaction is performed byreplica(s) responsible for the data items in the transaction’sread/write sets. When a transaction’s read/write sets spanacross more than one replica, the validation is performedby coordinating with other replicas, as described below. Aclient, i.e., the application process executing a transaction,contacts any of the service replicas to validate thetransaction by providing its read-write set and snapshottimestamp information. The contacted service replica thenacts as the coordinator in executing the protocol fortransaction validation. The coordinator determines thereplicas, called participants, that are responsible for the dataitems in the transaction’s read/write sets. It is possible thatthe coordinator itself is one of the participants. It thenperforms a two-phase coordination protocol with theparticipants, as described below.

In the first phase, the coordinator sends acquire-locksrequest to all the participants. Each participant then checksread-write and write-write conflicts for the items it isresponsible for in the following way. For a write-set item,if the write-ts value is greater than the transaction’ssnapshot timestamp then it indicates a write-write conflict.Similarly, if the read-ts value is greater than the snapshottimestamp then it indicates a read-write conflict. For a read-set item, there is a read-write conflict if the write-ts value isgreater than transaction’s snapshot timestamp. If anyconflict is found, the participant sends ‘failed’ response tothe coordinator. If there are no conflicts, then an attempt ismade to acquire read/write locks on the items. We use adeadlock-prevention scheme very similar to the one usedin [30]. If the participant acquires all the locks, it sends a‘success’ response to the coordinator.

In the second phase, if the coordinator has received a‘success’ response from all the participants, then thetransaction is committed, otherwise it is aborted. The statusof the transaction is updated in the TransactionTable. In case oftransaction commit the coordinator obtains a commit time-stamp from the TimestampService and sends a ‘commit’message to all participants along with the commit timestamp.Each participant then updates the write-ts and read-ts valuesfor the corresponding items and releases the locks. Anyconflicting transaction waiting for the commit decision of thistransaction is aborted. In case of abort, each participantreleases the locks acquired by the aborted transaction.

For tracking the validation requests, each replicamaintains two in-memory tables: a ParticipantTable to storeinformation related to the validation requests for which thereplica is a participant, and CoordinatorTable to storeinformation for the validation requests for which thereplica is the coordinator. The ParticipantTable maintains,for each validation request, the transaction-id, the part ofthe transaction’s read/write sets pertaining to this partic-ipant, and lock status of each item in this set. TheCoordinatorTable contains, for each request, the participantreplicas, the read/write sets of the transaction and lockstatus of each item in the set, and responses received fromdifferent participants.

7.2 Service Fault ToleranceThe failure-detection and the group membership manage-ment is performed by the service replicas in decentralized


manner. Failure-detection is performed using a heart-beatbased mechanism, and for group membership manage-ment we use the protocol we developed in [31]. When areplica crashes, a new replica is started which takes overthe role of the failed replica. The new replica then performsthe recovery protocol as described below.

7.2.1 Recovering Service StateWhen a replica fails, there may be uncompleted validationrequests for which the replica is either a coordinator or aparticipant. The new replica needs to recover the informa-tion maintained in the CoordinatorTable and ParticpantTable.This information is soft-state and can be recovered fromother replicas. The new replica contacts all other existingreplicas in the group and obtains information regarding thepending requests for which it was either a coordinator or aparticipant and the lock status for the items involved inthese requests. The reconstruction of ConflictTable is doneusing the read/write sets information stored in theTransactionTable. However, scanning the entire Transaction-Table can become expensive, and hence to reduce thisoverhead the ConflictTable is periodically checkpointed instable storage. Thus, only the transactions committed afterthe checkpoint start time need to be scanned.

7.2.2 Failure CasesEnsuring the correctness of the two-phase coordinationprotocol under replica crashes is crucial. With respect to aparticular validation request, the crashed replica can eitherbe a coordinator or a participant. In case of the coordinatorcrash, the client will timeout and retry the request bycontacting the new replica or any other replica. There arethree failure cases to consider: 1) failure before initiatingthe first phase, 2) failure before recording the commitstatus, and 3) failure after recording the commit status. Inthe first failure case, the client will timeout and retry therequest, since none of the replicas would have anyinformation for this request. In the second case, the newcoordinator will resend the lock requests. It may happenthat some locks have been already acquired, however, lockoperations are idempotent, so resending the lock requestsdoes not cause any inconsistencies. When failure occursafter recording the commit status, the new coordinator willfirst check the commit status and send commit or abortrequests to participants accordingly. In case of the partic-ipant crash, the validation request cannot be processeduntil the recovery of the crashed participant is complete.

8 SCALABILITY EVALUATIONS

In our evaluations of the proposed approaches the focuswas on evaluating the following aspects:

1. the scalability of different approaches under thescale-out model,

2. comparison of the service-based model and thedecentralized model in terms of transactionthroughput and scalability,

3. comparison of the basic SI and the transaction serial-izability approaches based on the cycle-preventionand the cycle-detection techniques,

4. transaction response times for various approaches,and

5. execution times of different protocol phases.

During the initial phase of our work, we performed apreliminary evaluation of the proposed approaches todetermine which approaches are more scalable and whichof these need to be investigated further on a large scalecluster. This evaluation was done using a testbed cluster of40 nodes on which the number of cores on the cluster nodesvaried from 2 to 8 cores, each with 2.3 GHz, and thememory capacity ranged from 4 GB to 8 GB. The finalevaluations in the later phase were conducted using amuch larger cluster provided by the Minnesota Super-computing Institute (MSI). Each node in this cluster had8 CPU cores with 2.8 GHz capacity, and 22 GB mainmemory.

8.1 TPC-C BenchmarkWe used TPC-C benchmark to perform evaluations under arealistic workload. However, our implementation of thebenchmark workload differs from TPC-C specifications inthe following ways. Since our primary purpose is tomeasure the transaction throughput we did not emulateterminal I/O. Since HBase does not support compositeprimary keys, we created the row-keys as concatenation ofthe specified primary keys. This eliminated the need of joinoperations, typically required in SQL-based implementa-tion of TPC-C. Predicate reads were implemented usingscan and filtering operations provided by HBase. Since thetransactions specified in TPC-C benchmark do not createserialization anomalies under SI, as observed in [9], weimplemented the modifications suggested in [11]. In ourexperiments we observed that on average a TPC-Ctransaction performed 8 read operations and 6 writeoperations.

8.2 Preliminary EvaluationsDuring each experiment, the first phase involved loadingdata in HBase servers, which also ensured that the data wasin the memory of HBase servers when the experimentstarted. Before starting the measurements, we ran thesystem for five minutes with initial transaction rate ofabout 1000 transactions per minute. The purpose of thiswas to ‘warm-up’ the TransactionTable partitions inHBase servers’ memory. The measurement period wasset to 30 minutes, in which we gradually increased thetransaction load to measure the maximum throughput. Fordifferent cluster sizes, we measured the maximum trans-action throughput (in terms of committed transactions perminute (tpmC)) and response times. In our experiments, weused one timestamp server and for the service-based modelwe used one validation server process.

Fig. 7 shows the maximum throughput achieved fordifferent transaction management approaches for differentcluster sizes. Since there is significant node heterogeneityin our testbed cluster, we indicate the cluster size in termsof the number of cores instead of the number of nodes. Thisfigure shows the throughput for basic-SI model to under-stand the cost of supporting serializability. We can observefrom Fig. 7 that scalability of throughput is achieved in


both the service-based as well as the decentralized model.However, the service-based approach gives higher trans-action throughput than the decentralized approach. Asexpected, the basic SI model achieves higher throughputcompared to approaches for ensuring serializability. This isbecause of the overhead for performing additional checksrequired for ensuring serializability. The cycle-preventionapproach provides higher throughput than the cycle-detection approach. This is because in the decentralizedmodel the overhead of the cycle-detection approach issignificant due to the overhead of maintaining dependencyinformation in the global storage. We also compared thecycle-prevention and the cycle-detection approaches in thecontext of the service-based model. However, we did notobserve any significant difference in the transactionthroughput.

Fig. 8 shows the average transaction response times forvarious approaches. As expected, the service-based ap-proach gives smaller response times than other ap-proaches. The cycle-detection approach has significantoverhead. In the largest configuration, the average re-sponse time for the cycle-detection approach is more thandouble of the same for the cycle-prevention approach.Also, the cycle-detection approach does not scale well interms of response times for large clusters. Therefore, weconclude that if serializability is required, it is better to use

the cycle-prevention approach than the cycle-detectionapproach.

We also measured and compared the time taken toexecute the various phases of the transaction protocol fordifferent approaches. Fig. 9 shows the average executiontimes for different phases. This data is shown for theevaluations conducted with the largest (96 cores) configu-ration. The validation phase for the cycle-preventionapproach takes more time (approximately by a factor oftwo) than the validation phase for the basic SI approach. Inthe cycle-detection approach the DSGupdate phase inducesa significant overhead.

The preliminary evaluations indicated that the service-based and the decentralized cycle-prevention approachesare scalable for supporting serializable transactions.Among these two approaches the service-based approachperforms better. We found that the decentralized cycle-detection approach does not scale well.

8.3 Scalability Validations on a Large ClusterBased on the above observations, we selected the service-based approach and decentralized cycle-prevention ap-proach for further evaluations over a large scale cluster tovalidate their scalability. These evaluations were per-formed using the MSI cluster resources. Using this cluster,we measured maximum transaction throughput achievedfor different cluster sizes. The results of these evaluationsare presented in Figs. 10 and 11.

Fig. 10 presents the throughput scalability, and Fig. 11shows average response times for various cluster sizes. The

Fig. 8. Average transaction response times under the scale-out model.

Fig. 9. Execution time for different protocol phases.

Fig. 7. Transaction throughput under the scale-out model.

Fig. 10. Transaction throughput under the scale-out model.


largest cluster size used in these experiments correspondsto close to 100 nodes (800 cores). In these evaluations weenabled the synchronous logging option for HBase toensure that the writes are durable even under crashes ofHBase servers. Hence, the response times are generallyhigher under this setting compared to the response timesshown in Fig. 8 where synchronous logging was disabled.We can observe that both the approaches provide incre-mental scalability for large cluster sizes. The service-basedarchitecture provides higher transaction throughput andlower response times compared to the decentralizedmodel, confirming our earlier observations from thepreliminary evaluations.

8.4 Scalability of Conflict Detection ServiceWe also evaluated the scalability of the replicated conflictdetection service. In this evaluation, we were mainlyinterested in measuring the throughput of validationrequests. For this purpose, we generated a synthetic work-load as follows. A pool of clients generated validationrequests for randomly selected read/write sets from anitem space of 1 million items. For each request, the size of theread/write sets was randomly selected between 4 to 20 items,with half of the items being read items and half being writeitems. We measured the throughput as the number ofrequests handled by the service per second, irrespective ofthe commit/abort decision, since we are mainly interested inmeasuring the request handling capacity of the service.Fig. 12 shows the saturation throughput of the service fordifferent number of replicas. We can see that increasing thenumber of replicas provides sub-linear increase in through-put, for example, increasing replica size from 4 to 8 providesthroughput increase by a factor of 1.35. An important thing tonote here is that the saturation throughput of the conflictdetection service, even with small number of replicas, issignificantly higher than the overall transaction throughputof the system. For example, from Figs. 12 and 10, we can seethat the saturation throughput of the service with 8 replicas isapproximately 24,000 requests per second whereas thesaturation transaction throughput with 100 nodes is approx-imately 5000 transactions per second. Thus, a small numberof replicas for conflict detection service can suffice to handlethe workload requirement of a large cluster.

8.5 Impact of Transaction SizeAnother aspect that we were interested in evaluating is theimpact of transaction size, i.e. the number of reads andwrites in a transaction, on various performance measures.To evaluate this impact, we created a custom benchmark asfollows. We created a single table with 1 million items. Thebenchmark included three classes of transactions: small sizetransactions accessing 10 items each, medium size transac-tions accessing 100 items each, and large size transactionsaccessing 1000 items each. In all the three classes, half of theaccessed items were read-set items and half were write-setitems. The items to read and write were randomly selectedbased on uniform distribution. We performed separateevaluations for each class of transactions by generatingtransaction load for that class and measured the maximumthroughput, average response times, and number of abortsfor that transaction class. Table 1 shows the results of theseevaluations. These evaluations were performed using acluster of 24 nodes on the MSI platform and thedecentralized model with cycle prevention approach. Nofailures were injected during these evaluations.

From Table 1, we can observe that as we increase thetransaction size the maximum throughput decreases andthe average response time increases. This is expected sincethe maximum throughput and the response time aredirectly proportion to the number of read/write operationsin a transaction. The percentage of aborts increase with theincrease in transaction size. This is primarily because undera fixed database size with increase in transaction size thelikelihood of transaction conflicts increases.

9 FAULT TOLERANCE EVALUATIONS

Our goal was to measure the performance of the cooper-ative recovery model. In these evaluations, our focus wason observing the following aspects: 1) impact of failuretimeout values, 2) time taken to detect and recover failed

Fig. 11. Average transaction response times under the scale-out model.

Fig. 12. Scalability of the conflict detection service.

TABLE 1Impact of Transaction Size


transactions, and 3) impact of failures on abort rate andSTS advancement.

9.1 Experiment SetupWe performed these evaluations on a cluster of 24 nodes onthe MSI platform. We induced a moderate transaction loadof approximately 50,000 transactions per minute, which islower than the saturation load observed in Fig. 10 forcluster size of 24 nodes. The injection of faults wasperformed as follows. A transaction randomly stalls duringthe commit protocol execution with certain probabilitycalled as failure probability. The failure probability iscalculated based on the desired failure rate. The failure isinjected either in the validation phase or the commit-incomplete phase (after acquiring commit timestamp). Weexperimented with a setting of 50 percent failures invalidation and 50 percent failures in the commit-incompletephase as well as an extreme case with all failures invalidation phase. For every injected failure, we measuredthe delay in detection of that failed transaction as well astime required to perform recovery actions. We alsorecorded information about the number of transactionsthat were wrongly suspected to be failed and aborted dueto timeouts. We refer to such aborts as false aborts. Onewould expect that for smaller timeouts the percentage ofsuch false aborts would be higher. The percentage of validtimeout-based aborts depends on the number of injectedfailures. We performed these experiments for failure ratesof 0.1 percent, 1 percent, and 10 percent and timeout valuesof 50, 100, 300, and 500 milliseconds. In these evaluations,we measured following performance measures:

1. percentage of aborts due to concurrency conflictssuch as read-write and write-write conflicts, referredto as self aborts,

2. percentage of valid timeout-based aborts and falseaborts,

3. average time for failure detection and recovery, and4. average gap between STS and GTS under failures.

9.2 Evaluation ResultsFig. 13 shows the abort statistics for various timeout valuesand failure rates. This data correspond to the setting of

50 percent failures in the validation and 50 percent failures inthe commit-incomplete phase. We can see that, as expected,the percentage of false aborts increases as we decrease thetimeout values. The percentage of valid timeout-basedaborts depends on the failure rate. Note that, even thoughwe decrease the timeout values from 500 ms to 100 ms, thepercentage of total aborts increase only by approximately afactor of two (from 15 percent to 30 percent). This is becausethe transactions that do not conflict with any other transac-tions are unaffected by the timeout values. If a transactiondoes not conflict with any other concurrent transaction, itwould not be aborted by any transaction irrespective of thetimeout values. The only problem that will arise due tofailure of a non-conflicting transaction is the blocking of STSadvancement if it has acquired the commit timestamp.However, in that case the transaction would be rolled-forward instead of aborting. We also performed thisevaluation with the setting of all failures in the validationphase. We observed that under this setting also the falseaborts increase with decrease in timeout values, confirmingour earlier observation. Thus, the appropriate timeout valuemust be large enough so that the number of false aborts iskept minimum. This largely depends on the transactionresponse time. Therefore, the appropriate timeout value canbe chosen by observing the transaction response times. Onecan also include autonomic mechanisms to set the timeoutvalues by continually observing the response time values atruntime.

Fig. 14 shows the data regarding average delays indetection of the failure of transactions that were failed inthe validation phase. Fig. 15 shows this data for transactionsthat were failed in the commit-incomplete phase. Since wewere interested in measuring the delays in detecting validfailures, we measured this data only for the failures thatwere injected by the failure injection mechanism and notfor the transactions that were wrongly suspected to befailed. From Figs. 14 and 15, we can see that the detectiondelays for transactions failed in the validation phase aresignificantly higher than that for the transaction failed inthe commit-incomplete phase. This is expected because, failureof a transaction failed in the validation phase will only bedetected when some other transaction encounters a read-writeor write-write conflict with the failed transaction, whereas

Fig. 13. Abort statistics for different failure rates (timeout values rangingfrom 50 ms to 500 ms).

Fig. 14. Average time to detect failures in validation phase.


failure of a transaction failed in the commit-incomplete phasewill be detected more promptly due to non-advancement ofSTS. We can also observe that the failure detection delay fortransactions failed in the commit-incomplete phase is mainlydominated by the timeout value: as we decrease the timeoutvalue the failure detection delay decreases. We observed thatthe time required to perform recovery actions is independentof whether the transaction failed in the validation phase or thecommit-incomplete phase. The average time to performrecovery typically ranged between 55 ms to 90 ms. We alsomeasured the average gap between STS andGTS for variousfailure rates and timeout values. This data is shown in Fig. 16.We can see that the average STS gap does not depend on thetimeout values. However, as we increase the failure rate theaverage gap value typically increases due to more numbertransactions blocking the advancement of STS.

10 CONCLUSION

We have presented here a fully decentralized transactionmanagement model and a service-based architecture forsupporting snapshot isolation as well as serializabletransactions for key-value based cloud storage systems.We investigated here two approaches for ensuring serial-izability. We find that both the decentralized and service-based models achieve throughput scalability under thescale-out model. The service-based model performs betterthan the decentralized model. To ensure the scalability ofthe service-based approach we developed a replication-based architecture for the conflict detection service. Thedecentralized model has no centralized component that canbecome a bottleneck, therefore, its scalability only dependson the underlying storage system. We also observe that thecycle detection approach has significant overhead com-pared to the cycle prevention approach. We conclude that ifserializability of transaction is required then using the cycleprevention approach is desirable. We also demonstratedhere the effectiveness of the cooperative recovery mechan-isms used in our approach. In summary, our workdemonstrates that serializable transactions can be sup-ported in a scalable manner in NoSQL data storagesystems.

ACKNOWLEDGMENT

This work was carried out in part using computingresources at the University of Minnesota SupercomputingInstitute. This work was supported by US National ScienceFoundation Grant 1319333 and Grant-in-Aid program ofthe University of Minnesota.

REFERENCES

[1] F. Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D.A. Wallach,M. Burrows, T. Chandra, A. Fikes, and R.E. Gruber, ‘‘Bigtable: ADistributed Storage System for Structured Data,’’ ACM Trans.Comput. Syst., vol. 26, no. 2, pp. 1-26, June 2008.

[2] B.F. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein,P. Bohannon, H.-A. Jacobsen, N. Puz, D. Weaver, and R. Yerneni,‘‘Pnuts: Yahoo!’s Hosted Data Serving Platform,’’ Proc. VLDBEndowment, vol. 1, no. 2, pp. 1277-1288, Aug. 2008.

[3] Apache, Hbase. [Online]. Available: http://hbase.apache.org/.[4] J. Baker, C. Bond, J. Corbett, J.J. Furman, A. Khorlin, J. Larson,

J.-M. Leon, Y. Li, A. Lloyd, and V. Yushprakh, ‘‘Megastore:Providing Scalable, Highly Available Storage for InteractiveServices,’’ in Proc. CIDR, 2011, pp. 223-234.

[5] S. Das, D. Agrawal, and A.E. Abbadi, ‘‘G-Store: A Scalable DataStore for Transactional Multi Key Access in the Cloud,’’ in Proc.ACM Symp. Cloud Comput., 2010, pp. 163-174.

[6] T. Haerder and A. Reuter, ‘‘Principles of Transaction-OrientedDatabase Recovery,’’ ACM Comput. Survey, vol. 15, no. 4, pp. 287-317, Dec. 1983.

[7] T.P. Council, San Francisco, CA, USATPC-C Benchmark.[Online]. Available: http://www.tpc.org/tpcc.

[8] H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O’Neil, andP. O’Neil, ‘‘A Critique of ANSI SQL Isolation Levels,’’ in Proc.ACM SIGMOD, 1995, pp. 1-10.

[9] A. Fekete, D. Liarokapis, E. O’Neil, P. O’Neil, and D. Shasha,‘‘Making Snapshot Isolation Serializable,’’ ACM Trans. DatabaseSyst., vol. 30, no. 2, pp. 492-528, June 2005.

[10] M. Bornea, O. Hodson, S. Elnikety, and A. Fekete, ‘‘One-CopySerializability With Snapshot Isolation Under the Hood,’’ in Proc.IEEE ICDE, Apr. 2011, pp. 625-636.

[11] M.J. Cahill, U. Rohm, and A.D. Fekete, ‘‘Serializable Isolation forSnapshot Databases,’’ ACM Trans. Database Syst., vol. 34, no. 4,pp. 20:1-20:42, Dec. 2009.

[12] S. Revilak, P. O’Neil, and E. O’Neil, ‘‘Precisely SerializableSnapshot Isolation (PSSI),’’ in Proc. IEEE ICDE, 2011, pp. 482-493.

[13] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati,A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, andW. Vogels, ‘‘Dynamo: Amazon’s Highly Available Key-ValueStore,’’ SIGOPS Oper. Syst. Rev., vol. 41, no. 6, pp. 205-220,Dec. 2007.

[14] Amazon, Amazon Simpledb. [Online]. Available: http://aws.amazon.com/simpledb/.

Fig. 16. Average gap between STS and GTS for different failure ratesand timeout values.

Fig. 15. Average time to detect failures in CommitIncomplete phase.


[15] A. Lakshman and P. Malik, ‘‘Cassandra: A DecentralizedStructured Storage System,’’ SIGOPS Oper. Syst. Rev., vol. 44,no. 2, pp. 35-40, Apr. 2010.

[16] S. Das, D. Agrawal, and A. El Abbadi, ‘‘ElasTraS: An ElasticTransactional Data Store in the Cloud,’’ in Proc. HotCloud, 2009,pp. 1-5.

[17] E.P. Jones, D.J. Abadi, and S. Madden, ‘‘Low OverheadConcurrency Control for Partitioned Main Memory Databases,’’in Proc. ACM SIGMOD Int’l Conf. Manage. Data, 2010, pp. 603-614.

[18] M.K. Aguilera, A. Merchant, M.A. Shah, A.C. Veitch, andC.T. Karamanolis, ‘‘Sinfonia: A New Paradigm for BuildingScalable Distributed Systems,’’ ACM Trans. Comput. Syst., vol. 27,no. 3, pp. 5:1-5:48, Dec. 2009.

[19] J. Cowling and B. Liskov, ‘‘Granola: Low-Overhead DistributedTransaction Coordination,’’ in Proc. USENIX Conf. ATC, 2012, pp. 1-13.

[20] A. Thomson, T. Diamond, S.-C. Weng, K. Ren, P. Shao, andD.J. Abadi, ‘‘Calvin: Fast Distributed Transactions for PartitionedDatabase Systems,’’ in Proc. ACM SIGMOD Int’l Conf. Manage.Data, 2012, pp. 1-12.

[21] Z. Wei, G. Pierre, and C.-H. Chi, ‘‘CloudTPS: Scalable Transac-tions for Web Applications in the Cloud,’’ IEEE Trans. Serv.Comput., vol. 5, no. 4, pp. 525-539, Apr. 2011.

[22] D.B. Lomet, A. Fekete, G. Weikum, and M.J. Zwilling, ‘‘UnbundlingTransaction Services in the Cloud,’’ in Proc. CIDR, 2009, pp. 1-10.

[23] D. Peng and F. Dabek, ‘‘Large-Scale Incremental ProcessingUsing Distributed Transactions and Notifications,’’ in Proc.USENIX OSDI, 2010, pp. 1-15.

[24] C. Zhang and H.D. Sterck, ‘‘Supporting Multi-Row DistributedTransactions With Global Snapshot Isolation Using Bare-BonesHBase,’’ in Proc. IEEE/ACM GRID, 2010, pp. 177-184.

[25] H. Jung, H. Han, A. Fekete, and U. Roehm, ‘‘Serializable SnapshotIsolation for Replicated Databases in High-Update Scenarios,’’ Proc.VLDB Endowment, vol. 4, no. 11, pp. 783-794, Aug. 2011.

[26] S. Jorwekar, A. Fekete, K. Ramamritham, and S. Sudarshan,‘‘Automating the Detection of Snapshot Isolation Anomalies,’’ inProc. VLDB, 2007, pp. 1263-1274.

[27] H.T. Kung and J.T. Robinson, ‘‘On Optimistic Methods forConcurrency Control,’’ ACM Trans. Database Syst., vol. 6, no. 2,pp. 213-226, June 1981.

[28] A. Adya, B. Liskov, and P.E. O’Neil, ‘‘Generalized Isolation LevelDefinitions,’’ in Proc. ICDE, 2000, pp. 67-78.

[29] W. Vogels, ‘‘Eventually Consistent,’’ Commun. ACM, vol. 52,no. 1, pp. 40-44, Jan. 2009.

[30] M. Maekawa, ‘‘An Algorithm for Mutual Exclusion in Decen-tralized Systems,’’ ACM Trans. Comput. Syst., vol. 3, no. 2,pp. 145-159, May 1985.

[31] V. Padhye and A. Tripathi, ‘‘Building Autonomically ScalableServices on Wide-Area Shared Computing Platforms,’’ in Proc.IEEE Symp. Netw. Comput. Appl., Aug. 2011, pp. 314-319.

Vinit Padhye received the BE degree from theUniversity of Mumbai, India, in 2005 and the MSdegree in computer science from the Universityof Minnesota, in 2011. He is currently pursuingthe PhD degree in the Department of ComputerScience & Enginnering, University of Minnesota,Minneapolis. His current research interests arein distributed systems, fault-tolerant computing,scalable and highly available systems, and cloudcomputing.

Anand Tripathi received the BTech degree inelectrical engineering from the Indian Institute ofTechnology, Bombay, in 1972 and the MS andPhD degrees in electrical engineering from theUniversity of Texas at Austin, in 1978 and 1980,respectively. His research interests are indistributed systems, fault-tolerant computing,system security, and pervasive computing. Heis a professor of computer science at theUniversity of Minnesota, Minneapolis. Heworked as a Scientific Officer at Bhabha Atomic

Research Center, India, during 1973-75. During 1981-84 he worked asa Senior Principal Research Scientist at Honeywell Computer ScienceCenter, Minneapolis. He joined the University of Minnesota in 1984.During 1995-97, he served as a Program Director in the Division ofComputer and Communications Research at the National ScienceFoundation, Arlington, Virginia. He is a Fellow of IEEE and a member ofthe ACM. He served as an at-large member of the IEEE ComputerSociety Publications Board (2001-2005). He has served on the editorialboards of IEEE Transactions on Computers, IEEE Distributed SystemsOnline, Elsevier Journal on Pervasive and Mobile Computing, and IEEEPervasive Computing. He was the Program Chair for the IEEESymposium on Reliable Distributed Systems (SRDS) in 2001 and forthe Second IEEE International Conference on Pervasive Computingand Communications (PerCom) in 2004. He was one of the organizersof two ECOOP Workshops on exception handling held in 2000 and2003, and co-editor for two Springer LNCS volumes on exceptionhandling, published in 2002 and 2006.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.


Documents

IEEE TRANSACTIONS ON SERVICES COMPUTING, VOL. 8, NO. 1 ...ajanta.cs.umn.edu/papers/tsc-2015.pdf · Scalable Transaction Management with Snapshot Isolation for NoSQL Data Storage Systems