Learning Greenplum Distributed Transaction

Learning Greenplum Distributed TransactionCode Analysis & Call flow

Transaction Framework

Begin Transaction

ClientServer

EndTransaction

UPDATE

Transaction Block

Begin Transaction Block

End Transaction Block

Low level Transaction Layer

Execute SQL

PostgreSQLTransactionStateData-TBlockState ： high-level stat-TransState ： low-level state

QE Distributed Transaction

Transaction Framework

Begin Transaction

客户端Master

EndTransaction

UPDATE

Transaction Block




Greenplum

Segment 0

Transaction Block




Execute SQL

QD DTM

Execute SQL

QE Distributed Transaction

Segment 0

Transaction Block




Execute SQL

Dtm Command

Dtm Command

TMGXACT-DtxState

TransactionStateData-TBlockState ： high-level stat-TransState ： low-level state

DtxContext

TransactionStateData-TBlockState ： high-level stat-TransState ： low-level state

DtxContext

Main Call Flow (Normal)Master(GP_ROLE_DISPATCH) Seg-N

exec_simple_query

start_xact_command

ProcessUtility

finish_xact_command

doQEDistributedExplicitBegin

setupQEDtxContext

performDtxProtocolCommand

exec_mpp_dtx_protocol_command

firstchar = ‘M’ only for “BEGIN”

firstchar = ‘T'

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND

Middle Layer function

low level function


low level function

Main Call Flow (Retry)Master(GP_ROLE_DISPATCH) Seg-N

doDtxPhase2Retry

StartTransactionCommand

doDispatchDtxProtocolCommand

CommitTransactionCommand

DTX_CONTEXT_QD_RETRY_PHASE_2

When DTX_STATE_RETRY_COMMIT_PREPARED

DTX_STATE_RETRY_COMMIT_PREPAREDDTX_STATE_RETRY_ABORT_PREPARED

doInsertForgetCommitted



firstchar = ‘T'

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND


• Normal system's calls• StartTransactionCommand• CommitTransactionCommand• AbortCurrentTransaction

• User change calls by issue “BEGIN, COMMIT, ROLLBACK, SAVEPOINT, ROLLBACK TO or RELEASE”• BeginTransactionBlock• EndTransactionBlock• UserAbortTransactionBlock• PrepareTransactionBlock

• DefineSavepoint• RollbackToSavepoint• ReleaseSavepoint

Low level calls

• StartTransaction• CommitTransaction• AbortTransaction• CleanupTransaction• PrepareTransaction

• StartSubTransaction• CommitSubTransaction• AbortSubTransaction• CleanupSubTransaction

Postgres Normal middle level call flow

/ StartTransactionCommand;

/ StartTransaction;

1) < ProcessUtility; << BEGIN

\ BeginTransactionBlock;

\ CommitTransactionCommand;


2) / ProcessQuery; << SELECT ...


\ CommandCounterIncrement;


3) / ProcessQuery; << INSERT ...


\ CommandCounterIncrement;


/ ProcessUtility; << COMMIT

4) < EndTransactionBlock;


\ CommitTransaction;

1) BEGIN2) SELECT * FROM foo3) INSERT INTO foo VALUES (...)4) COMMIT

High Level State transfer

TBLOCK_DEFAULT

TBLOCK_START

TBLOCK_ABORT_PENDING

TBLOCK_BEGIN

TBLOCK_INPROGRESS

TBLOCK_ABORT

TBLOCK_END

TBLOCK_ABORT_END

BeginTransactionBlock


AbortCurrentTransaction

UserAbortTransactionBlock

EndTransactionBlock

CommitTransactionCommandAbortCurrentTransaction



EndTransactionBlock



AbortCurrentTransactionCommitTransactionCommand

PostgreSQL

TBLOCK_PREPARE

PrepareTransactionBlock


failed xact failed xact

AbortOutOfAnyTransaction



High Level State transfer

TBLOCK_SUBEND

TBLOCK_SUBBEGIN

TBLOCK_SUBABORT_PENDING

TBLOCK_SUBINPROGRESS

TBLOCK_SUBABORT

TBLOCK_SUBABORT_RESTARTTBLOCK_SUBABORT_END

TBLOCK_SUBRESTART

PostgreSQL


PushTransaction

ReleaseSavepoint

EndTransactionBlock



EndTransactionBlock

EndTransactionBlock


RollbackToSavepoint

RollbackToSavepoint

RollbackToSavepoint

RollbackToSavepoint

failed xact

failed xactfailed xact

Detailed transfer

TBLOCK_DEFAULT

TBLOCK_START


StartTransactionCommand AbortCurrentTransaction





Single Query Transaction

Error Happened

Rollback issued

Detailed transfer

TBLOCK_DEFAULT

TBLOCK_START

TBLOCK_BEGIN

TBLOCK_INPROGRESS


StartTransactionCommandCommitTransactionCommand

Explicit Begin


Error Happened

Detailed transfer

TBLOCK_DEFAULTTBLOCK_INPROGRESS

TBLOCK_END

EndTransactionBlock


TBLOCK_PREPARE




Commit/End

Error Happened

Detailed transfer

TBLOCK_INPROGRESS

TBLOCK_ABORT


failed xact

SQL Error Happened in Transaction

Detailed transfer

TBLOCK_DEFAULT

TBLOCK_INPROGRESS

TBLOCK_ABORTTBLOCK_ABORT_END


EndTransactionBlock


failed xactfailed xact


Rollback




Low Level State transfer

TRANS_DEFAULT

TRANS_START

TRANS_COMMITTRANS_INPROGRESS

TRANS_ABORT

TRANS_PREPARE

StartTransactionStartTransaction

CommitTransaction

PrepareTransaction PrepareTransaction

CommitTransaction

AbortTransaction CleanupTransaction

TM Control Block - TMGXACT

char gid[TMGIDSIZE];

DistributedTransactionId gxid;

DtxState state;

int sessionId;

TransactionId localXid;

LocalDistribXactRef localDistribXactRef;

bool explicitBeginRemembered;

DistributedTransactionId xminDistributedSnapshot;

bool bumpedPhase1Count;

bool badPrepareGangs;

bool retryPhase2RecursionStop;

int debugIndex;

bool directTransaction;

uint16 directTransactionContentId;

TM Actions to RM

• Normal• doPrepareTransaction

• doDispatchDtxProtocolCommand(DTX_PROTOCOL_COMMAND_PREPARE)• doInsertForgetCommitted

• Insert FORGET COMMITTED into the xlog.• doNotifyingCommitPrepared

• doDispatchDtxProtocolCommand(DTX_PROTOCOL_COMMAND_COMMIT_PREPARED)• doNotifyingAbort

• doDispatchDtxProtocolCommand(DTX_PROTOCOL_COMMAND_ABORT_PREPARED)

• Crash Recovery• doNotifyCommittedInDoubt

• doDispatchDtxProtocolCommand(DTX_PROTOCOL_COMMAND_RECOVERY_COMMIT_PREPARED)

• doAbortInDoubt• doDispatchDtxProtocolCommand(DTX_PROTOCOL_COMMAND_RECOVERY_ABORT_PREP

ARED)

TM Actions to RM

• Others• sendDtxExplicitBegin

• dispatchDtxCommand(“BEGIN ISOLATION LEVEL %s, READ %s”)

• DefineDispatchSavepoint• dispatchDtxCommand(“savepoint”)

• ReleaseSavepoint• dispatchDtxCommand(“release savepoint”)

• DispatchRollbackToSavepoint• dispatchDtxCommand(“rollback to savepoint”)

TM dispatch command to RM

• doDispatchDtxProtocolCommand• cdbdisp_dispatchDtxProtocolCommand

• cdbdisp_dispatchToGang(GP_DISPATCH_COMMAND_TYPE_DTX_PROTOCOL)

• CdbCheckDispatchResult(&ds, DISPATCH_WAIT_NONE);• ----------------wait for QE------------------• cdbdisp_returnResults• cdbdisp_destroyDispatchThreads

DTM to RM protocol commands

• DTX_PROTOCOL_COMMAND_NONE

• DTX_PROTOCOL_COMMAND_STAY_AT_OR_BECOME_IMPLIED_WRITER = 1,

• DTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED,

• DTX_PROTOCOL_COMMAND_PREPARE,

• DTX_PROTOCOL_COMMAND_ABORT_SOME_PREPARED,

• DTX_PROTOCOL_COMMAND_COMMIT_PREPARED,

• DTX_PROTOCOL_COMMAND_ABORT_PREPARED,

• DTX_PROTOCOL_COMMAND_RETRY_COMMIT_PREPARED,

• DTX_PROTOCOL_COMMAND_RETRY_ABORT_PREPARED,

• DTX_PROTOCOL_COMMAND_RECOVERY_COMMIT_PREPARED,

• DTX_PROTOCOL_COMMAND_RECOVERY_ABORT_PREPARED,

• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_BEGIN_INTERNAL,

• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_ROLLBACK_INTERNAL,

• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_RELEASE_INTERNAL,

TM State transfer

DTX_STATE_NONE

DTX_STATE_ACTIVE_DISTRIBUTED

DTX_STATE_PREPARING

DTX_STATE_INSERTING_COMMITTEDDTX_STATE_INSERTED_COMMITTED

DTX_STATE_FORCED_COMMITTED DTX_STATE_INSERTING_FORGET_COMMITTEDDTX_STATE_INSERTED_FORGET_COMMITTED

DTX_STATE_NOTIFYING_COMMIT_PREPARED

doPrepareTransaction

DTX_STATE_PREPARED


DTX_STATE_NOTIFYING_ABORT_SOME_PREPARED

DTX_STATE_NOTIFYING_ABORT_NO_PREPARED

DTX_STATE_NOTIFYING_ABORT_PREPARED

RecordTransactionCommit/ forcedDistributedCommitted

doNotifyingCommitPrepared doInsertForgetCommitted

RecordTransactionCommit/insertingDistributedCommitted

rollbackDtxTransaction

DTX_STATE_RETRY_ABORT_PREPAREDrollbackDtxTransaction(badPrepareGangs)

rollbackDtxTransaction(badPrepare)

rollbackDtxTransaction(xa rollback??)

DTX_STATE_RETRY_COMMIT_PREPARED

doNotifyingCommitPrepared(commit failed)doDtxPhase2Retry/doInsertForgetCommitted

createDtx/initGxact

startTransaction/createDtx

DTX_STATE_ACTIVE_NOT_DISTRIBUTEDdtmPreCmd

DTM State

• Uninitialized transaction• DTX_STATE_NONE = 0

• The distributed transaction is active• DTX_STATE_ACTIVE_NOT_DISTRIBUTED //auto-commit on the QEs

• DTX_STATE_ACTIVE_DISTRIBUTED //it is explicit or an implicit writer transaction

DTM State

• For two-phase commit, the first phase is about to run• DTX_STATE_PREPARING

• For two-phase commit, the first phase has completed• DTX_STATE_PREPARED

• DTX_STATE_INSERTING_COMMITTED

• DTX_STATE_INSERTED_COMMITTED

• DTX_STATE_FORCED_COMMITTED

• DTX_STATE_NOTIFYING_COMMIT_PREPARED

• DTX_STATE_INSERTING_FORGET_COMMITTED

• DTX_STATE_INSERTED_FORGET_COMMITTED

DTM State

• Transaction rollback has been requested and QD is notifying all QD processes• no QEs have started work on the first phase of two-phase commit

• DTX_STATE_NOTIFYING_ABORT_NO_PREPARED

• at least one QE has done the first phase of two-phase commit• DTX_STATE_NOTIFYING_ABORT_SOME_PREPARED

• the QE processes have done the first phase of two-phase commit• DTX_STATE_NOTIFYING_ABORT_PREPARED,• DTX_STATE_RETRY_COMMIT_PREPARED,• DTX_STATE_RETRY_ABORT_PREPARED,• DTX_STATE_CRASH_COMMITTED

DTX Context

• DTX_CONTEXT_LOCAL_ONLY • There is no distributed transaction. This is the initial state and, for

utility mode connections or master-only queries, the only state. It is also the state to which the QD and QE return to between transactions.

• DTX_CONTEXT_QD_DISTRIBUTED_CAPABLE,• On QD: the process is currently part of a distributed transaction.

Whether or not the transaction has been started on a QE is not a part of the QD state -- that is tracked by assigning one of the DTX_CONTEXT_QE* values on the QE process, and by updating the state field of the currentGxact.

• DTX_CONTEXT_QD_RETRY_PHASE_2,• On QD: indicates that the second-phase of the two-phase commit is

being retried.

DTX Context

• DTX_CONTEXT_QE_ENTRY_DB_SINGLETON,• TODO: how is something an Entry db? fix this documentation

• DTX_CONTEXT_QE_AUTO_COMMIT_IMPLICIT,• On a QE that is the root of a query, this context means that the distributed transaction was opened implicitly by a non-writing query in an auto-commit

transaction.

• This is essentially the same as _TWO_PHASE_IMPLICIT_WRITER except we know the query won't dirty anything and so we don't need a two-phase commit.

• DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_WRITER,• On a QE, this context means that the distributed transaction was opened explicitly by a BEGIN query statement Note that this state can happen even if there

are no writer queries (because this state is entered when the BEGIN statement is processed).

• DTX_CONTEXT_QE_TWO_PHASE_IMPLICIT_WRITER,• On a QE that is the root of a query, this context means that the distributed transaction was opened implicitly by a writing query in an auto-commit transaction,

and this is the writing QE process Same as _AUTO_COMMIT_IMPLICIT except that we need a two-phase of the dirty data.

• DTX_CONTEXT_QE_READER,• On a QE that is not the root of a query, this context is used. This can happen regardless of whether the overall transaction for the query is implicit or explicit,

and auto-commit vs two-phase

• DTX_CONTEXT_QE_PREPARED,• On the QE, the two-phase commit has been prepared (first phase complete).

• DTX_CONTEXT_QE_FINISH_PREPARED• On the QE, the two-phase commit has been committed or aborted (second phase complete).

TM: ProcessUtility

• case nodeTag(parsetree) = T_TransactionStmt• Case Stmt->kind:

• -----normal-----• TRANS_STMT_BEGIN or TRANS_STMT_START

• BeginTransactionBlock

• sendDtxExplicitBegin

• TRANS_STMT_COMMIT• EndTransactionBlock

• TRANS_STMT_ROLLBACK• UserAbortTransactionBlock

TM/RM : ProcessUtility


• -----savepoint-----• TRANS_STMT_SAVEPOINT

• RequireTransactionChain• sendDtxExplicitBegin• DefineDispatchSavepoint

• TRANS_STMT_RELEASE• RequireTransactionChain• ReleaseSavepoint

• TRANS_STMT_ROLLBACK_TO• RequireTransactionChain• RollbackToSavepoint

RM: ProcessUtility ??? XA???


• -----I think it’s XA feature, not know when called in RM----• TRANS_STMT_PREPARE

• PrepareTransactionBlock

• TRANS_STMT_COMMIT_PREPARED • PreventTransactionChain

• FinishPreparedTransaction

• TRANS_STMT_ROLLBACK_PREPARED • PreventTransactionChain

• FinishPreparedTransaction

RM: doQEDistributedExplicitBegin

• Handle ExplicitBegin• StartTransactionCommand();• BeginTransactionBlock();• CommitTransactionCommand();

RM: performDtxProtocolCommand

• Case dtxProtocolCommand• DTX_PROTOCOL_COMMAND_STAY_AT_OR_BECOME_IMPLIED_WRITER

• Case DistributedTransactionContext:• DTX_CONTEXT_LOCAL_ONLY

• setupQEDtxContext• StartTransactionCommand

• DTX_CONTEXT_QE_TWO_PHASE_IMPLICIT_WRITER

• DTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED• AbortOutOfAnyTransaction

• DTX_PROTOCOL_COMMAND_PREPARE• Case DistributedTransactionContext:

• DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_WRITER or DTX_CONTEXT_QE_TWO_PHASE_IMPLICIT_WRITER• performDtxProtocolPrepare

• DTX_PROTOCOL_COMMAND_ABORT_SOME_PREPARED• Case DistributedTransactionContext:

• DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_WRITER or DTX_CONTEXT_QE_TWO_PHASE_IMPLICIT_WRITER• AbortOutOfAnyTransaction

• DTX_CONTEXT_QE_PREPARED• performDtxProtocolAbortPrepared

• DTX_PROTOCOL_COMMAND_COMMIT_PREPARED• performDtxProtocolCommitPrepared

• DTX_PROTOCOL_COMMAND_ABORT_PREPARED• performDtxProtocolAbortPrepared

RM: performDtxProtocolCommand

• Case dtxProtocolCommand• DTX_PROTOCOL_COMMAND_RETRY_COMMIT_PREPARED

• performDtxProtocolCommitPrepared• DTX_PROTOCOL_COMMAND_RETRY_ABORT_PREPARED

• performDtxProtocolAbortPrepared• DTX_PROTOCOL_COMMAND_RECOVERY_COMMIT_PREPARED

• performDtxProtocolCommitPrepared• DTX_PROTOCOL_COMMAND_RECOVERY_ABORT_PREPARED

• performDtxProtocolAbortPrepared

• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_BEGIN_INTERNAL• Case DistributedTransactionContext:

• DTX_CONTEXT_LOCAL_ONLY• setupQEDtxContext• StartTransactionCommand

• BeginInternalSubTransaction• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_ROLLBACK_INTERNAL

• RollbackAndReleaseCurrentSubTransaction• DTX_PROTOCOL_COMMAND_SUBTRANSACTION_RELEASE_INTERNAL

• ReleaseCurrentSubTransaction

Log Type

• Nomal log:• #define XLOG_XACT_COMMIT 0x00• #define XLOG_XACT_ABORT 0x20

• 2PC Log• RM Log

• #define XLOG_XACT_PREPARE 0x10• #define XLOG_XACT_COMMIT_PREPARED 0x30• #define XLOG_XACT_ABORT_PREPARED 0x40

• TM log• #define XLOG_XACT_DISTRIBUTED_COMMIT 0x50• #define XLOG_XACT_DISTRIBUTED_FORGET 0x60

TM Log write

• RecordTransactionCommitif (isDtxPrepared)

{

XLogInsert(RM_XACT_ID, XLOG_XACT_DISTRIBUTED_COMMIT)

} else {

XLogInsert(RM_XACT_ID, XLOG_XACT_COMMIT);

}

XLogFlush()

• RecordDistributedForgetCommittedXLogInsert(RM_XACT_ID, XLOG_XACT_DISTRIBUTED_FORGET | XLOG_NO_TRAN)

TM Log write

• XLOG_XACT_DISTRIBUTED_COMMIT• XLogRecData rdata[4];

0

1

2

3

XLogRecData

Data: xl_xact_commitLen: MinSizeOfXactCommit

Data: persistentCommitBufferLen: persistentCommitSerializeLen

Data: childrenLen: nchildren * sizeof(TransactionId)

Data: TMGXACT_LOGLen: sizeof(TMGXACT_LOG)

persistentCommitObjectCount > 0

nchildren > 0

isDtxPrepared

TM Log write

• XLOG_XACT_DISTRIBUTED_FORGET• XLogRecData rdata[4];

0

XLogRecData

Data: xl_xact_distributed_forgetLen: sizeof(xl_xact_distributed_forget)

TMGXACT_LOG

RM Log write

• Prepare LOG• Log state format

1. TwoPhaseFileHeader2. TransactionId[] (subtransactions)3. RelFileNode[] (files to be deleted at commit)4. RelFileNode[] (files to be deleted at abort)5. TwoPhaseRecordOnDisk6. ...7. TwoPhaseRecordOnDisk (end sentinel, rmid == TWOPHASE_RM_END_ID)8. CRC32

• XLog

RM Log write

• Log state format

RM Log write

• XLOG_XACT_PREPARE• XLogRecData rdata[n];

0

1

2

XLogRecDataData: TwoPhaseFileHeader

Len: TwoPhaseFileHeader or 512



nchildren > 0


3Data: TwoPhaseRecordOnDisk

（TWOPHASE_RM_NOTIFY_ID）Len: sizeof(TwoPhaseRecordOnDisk)

3Data: relname

Len: sizeof(relname)


（TWOPHASE_RM_FLATFILES_ID）Len: sizeof(TwoPhaseRecordOnDisk)

info != 0


（TWOPHASE_RM_INVAL_ID）Len: sizeof(TwoPhaseRecordOnDisk)

RelcacheInitFileInval != NULL

6 x nData: TwoPhaseRecordOnDisk（TWOPHASE_RM_LOCK_ID）

Len: sizeof(TwoPhaseRecordOnDisk)

Has locks

6 x nData: TwoPhaseLockRecord

Len: sizeof(TwoPhaseLockRecord)

Has locks

7 x nData: TwoPhaseRecordOnDisk

（TWOPHASE_RM_PGSTAT_ID）Len: sizeof(TwoPhaseRecordOnDisk)

7 x nData: TwoPhasePgStatRecord

Len: sizeof(TwoPhasePgStatRecord)

xact_state != NULL

8Data: TwoPhaseRecordOnDisk（TWOPHASE_RM_END_ID）

Len: sizeof(TwoPhaseRecordOnDisk)

RM Log write

• XLOG_XACT_DISTRIBUTED_COMMIT• XLogRecData rdata[4]

0

1

2

XLogRecData

Data: xl_xact_commit_preparedLen: MinSizeOfXactCommitPrepared



persistentPrepareObjectCount > 0

nsubxacts > 0

RM Log write

• XLOG_XACT_DISTRIBUTED_COMMIT• XLogRecData rdata[3];

0

1

2

XLogRecData

Data: xl_xact_abort_preparedLen: MinSizeOfXactAbortPrepared




nchildren > 0

RM Log write

• EndPrepareXLogInsert(RM_XACT_ID, XLOG_XACT_PREPARE)XLogFlush()

• RecordTransactionAbortXLogInsert(RM_XACT_ID, XLOG_XACT_ABORT)XLogFlush() only when we are deleting files

• RecordTransactionCommitPreparedXLogInsert(RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED | XLOG_NO_TRAN)XLogFlush()

• RecordTransactionAbortPreparedXLogInsert(RM_XACT_ID, XLOG_XACT_ABORT_PREPARED | XLOG_NO_TRAN)XLogFlush()

Log Write Seq when CommitMaster(GP_ROLE_DISPATCH) Seg-N

XLogFlush

XLogInsert (RM_XACT_ID, XLOG_XACT_PREPARE)

XLogInsert (RM_XACT_ID, XLOG_XACT_DISTRIBUTED_COMMIT)

XLogFlush

XLogInsert (RM_XACT_ID, XLOG_XACT_DISTRIBUTED_FORGET | XLOG_NO_TRAN)

XLogFlushXLogInsert (RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED | XLOG_NO_TRAN)

XLogInsert(RM_XACT_ID, XLOG_XACT_ABORT)

XLogFlushXLogInsert (RM_XACT_ID, XLOG_XACT_ABORT_PREPARED | XLOG_NO_TRAN)


XLogFlushXLogInsert (RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED | XLOG_NO_TRAN)

Commit

Rollback

Retry Commit

Retry Rollback

Test Details

• BEGIN• INSERT

• END• MANUNALLY ROLLBACK• INSERT FAILED WITH AUTO ROLLBACK

• Kill RM-0 before master write log, RM-1 Prepared• Kill RM-0 when master send commit to RMs, RM all Prepared• Error happened when master send commit to RMs, no Gangs

errors

• Crash recovery

PostgresMain

Master(GP_ROLE_DISPATCH) Seg-N

PostgresMainfirstchar = 'Q'

exec_simple_query

start_xact_command


StartTransactionBlock state: TBLOCK_DEFAULT -> TBLOCK_STARTED

State: TRANS_DEFAULT -> TRANS_START

PortalRun

ProcessUtilityTRANS_STMT_START

BeginTransactionBlockBlock state: TBLOCK_STARTED -> TBLOCK_BEGIN

sendDtxExplicitBegin

DTX_CONTEXT_QD_DISTRIBUTED_CAPABLE

createDtxGxact->state : DTX_STATE_NONE -> DTX_STATE_ACTIVE_NOT_DISTRIBUTED

dispatchDtxCommand

cdbdisp_dispatchRMCommand

cdbdisp_dispatchCommand

cdbdisp_dispatchToGang

CdbCheckDispatchResult

cdbdisp_returnResults

cdbdisp_destroyDispatchThreads


finish_xact_command

Block state: TBLOCK_BEGIN -> TBLOCK_INPROGRESS

doQEDistributedExplicitBegin




setupQEDtxContext

needTwoPhase = false; explicitBegin = true

Block state: TBLOCK_DEFAULT -> TBLOCK_STARTED


StartTransaction

firstchar = ‘M'

Block state: TBLOCK_STARTED -> TBLOCK_BEGIN

Block state: TBLOCK_BEGIN -> TBLOCK_INPROGRESS

State: TRANS_START -> TRANS_INPROGRESS


pq_flush

ReadyForQuery_QEWriter

DTX_CONTEXT_LOCAL_ONLY

DistributedTransactionContext: DTX_CONTEXT_LOCAL_ONLY -> DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_

BEGIN;

--------------------wait----------------------

dtmPreCommandGxact->state: DTX_STATE_ACTIVE_NOT_DISTRIBUTED -> DTX_STATE_ACTIVE_DISTRIBUTED

PostgresMain


PostgresMain

firstchar = 'Q'

exec_simple_query

start_xact_command

StartTransactionCommandTBLOCK_INPROGRESS

PortalRun

PortalRunMulti

ProcessQueryExecutorStart

start_xact_command

exec_mpp_query

firstchar = ‘M'

dtmPreCommand


INSERT ;


finish_xact_command

--------------------wait----------------------

PortalRun

PortalRunMulti

ProcessQueryExecutorRun

ExecutePlan

ExecInsert

heap_insertheap_log_tuple_insert

heap_log_tuple_insertXLogInsert (RM_HEAP_ID)


finish_xact_command

ExecutorEndmppExecutorFinishup

CdbCheckDispatchResult

CommandId++

PostgresMain


PostgresMain

firstchar = 'Q'

exec_simple_query

start_xact_command


PortalRun

ProcessUtilityTRANS_STMT_COMMIT

EndTransactionBlockBlock state: TBLOCK_INPROGRESS -> TBLOCK_END


finish_xact_command

BeginCommand



State: TRANS_BEGIN -> TRANS_PREPARE

firstchar = ‘T'

CommitTransaction

prepareDtxTransaction

dtxProtocolCommand = DTX_PROTOCOL_COMMAND_PREPARE

DistributedTransactionContext == DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_WRITER

performDtxProtocolPrepare




DTX_CONTEXT_QE_TWO_PHASE_EXPLICIT_WRITER -> DTX_CONTEXT_QE_PREPARED

EndTransactionBlockBlock state: TBLOCK_INPROGRESS -> TBLOCK_END

Block state: TBLOCK_END -> TBLOCK_PREPARE

PrepareTransaction

XLogFlush

Block state: TBLOCK_PREPARE -> TBLOCK_DEFAULT

pq_flush


END/COMMIT 1;

doPrepareTransactionGxactstate: DTX_STATE_ACTIVE_DISTRIBUTED ->DTX_STATE_PREPARING

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND_PREPARE

cdbdisp_dispatchDtxProtocolCommand

Gxactstate: DTX_STATE_PREPARING -> DTX_STATE_PREPARED


EndPrepare

--------------------wait---------------------- XLogInsert (RM_XACT_ID, XLOG_XACT_PREPARE)

State: TRANS_INPROGRESS -> TRANS_COMMIT

State: TRANS_PREPARE -> TRANS_DEFAULT

StartPreparegxact = MarkAsPreparing()


PostgresMain

BeginCommandperformDtxProtocolCommand

exec_mpp_dtx_protocol_commandfirstchar = ‘T'

dtxProtocolCommand = DTX_PROTOCOL_COMMAND_COMMIT_PREPARED

performDtxProtocolCommitPreparedStartTransactionCommand

FinishPreparedTransaction


DTX_CONTEXT_QE_PREPARED -> DTX_CONTEXT_LOCAL_ONLY


Block state: TBLOCK_DEFAULT -> TBLOCK_STARTED

Block state: TBLOCK_STARTED -> TBLOCK_DEFAULT


StartTransaction

DTX_CONTEXT_QE_PREPARED -> DTX_CONTEXT_QE_FINISH_PREPARED


State:TRANS_INPROGRESS ->TRANS_COMMIT

CommitTransaction

RecordTransactionCommitXLogFlush

END/COMMIT 2;

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND_COMMIT_PREPARED



doNotifyingCommitPrepared

notifyCommittedDtxTransaction

Dtx state : DTX_STATE_FORCED_COMMITTED -> DTX_STATE_NOTIFYING_COMMIT_PREPARED

doInsertForgetCommittedDtx state : DTX_STATE_NOTIFYING_COMMIT_PREPARED -> DTX_STATE_INSERTING_FORGET_COMMITTED

Dtx state : DTX_STATE_NOTIFYING_COMMIT_PREPARED -> DTX_STATE_INSERTED_FORGET_COMMITTED


RecordDistributedForgetCommitted

RecordTransactionCommit

forcedDistributedCommittedDtx state : DTX_STATE_INSERTED_COMMITTED-> DTX_STATE_FORCED_COMMITTED

--------------------wait----------------------

insertingDistributedCommittedDtx state : DTX_STATE_PREPARED -> DTX_STATE_INSERTING_COMMITTED

XLogInsert (RM_XACT_ID, XLOG_XACT_DISTRIBUTED_COMMIT)

insertedDistributedCommittedDtx state : DTX_STATE_INSERTING_COMMITTED -> DTX_STATE_INSERTED_COMMITTED

RecordTransactionCommitPrepared

XLogFlush


finish_xact_command

CommitTransaction

……

……

XLogFlush

……

XLogInsert (RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED


Block state: TBLOCK_END -> TBLOCK_DEFAULT

State:TRANS_COMMIT ->TRANS_DEFAULT

State:TRANS_COMMIT ->TRANS_DEFAULT

PostgresMain


PostgresMain

firstchar = 'Q'

exec_simple_query

start_xact_command


PortalRun

ProcessUtilityTRANS_STMT_ROLLBACK

UserAbortTransactionBlockBlock state: TBLOCK_INPROGRESS -> TBLOCK_ABORT_PENDING


finish_xact_command

BeginCommand



firstchar = ‘T'

AbortTransaction


dtxProtocolCommand = DTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED



pq_flush


ROLLBACK;

Gxactstate: DTX_STATE_ACTIVE_DISTRIBUTED ->DTX_STATE_NOTIFYING_ABORT_NO_PREPARED

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED



--------------------wait----------------------

CleanupTransaction

Block state: TBLOCK_ABORT_PENDING -> TBLOCK_DEFAULT

State: TRANS_INPROGRESS -> TRANS_ABORT

doNotifyingAbort

AbortTransaction

CleanupTransaction



State: TRANS_ABORT -> TRANS_DEFAULT


RecordTransactionAbortXLogInsert(RM_XACT_ID, XLOG_XACT_ABORT)

PostgresMain


PostgresMain

firstchar = 'Q'


AbortTransaction BeginCommand



firstchar = ‘T'

rollbackDtxTransactiondtxProtocolCommand = DTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED



pq_flush


INSERT FAILED;

Gxactstate: DTX_STATE_ACTIVE_DISTRIBUTED ->DTX_STATE_NOTIFYING_ABORT_NO_PREPARED

doDispatchDtxProtocolCommandDTX_PROTOCOL_COMMAND_ABORT_NO_PREPARED



--------------------wait----------------------

CleanupTransaction

Block state: TBLOCK_ABORT_END -> TBLOCK_DEFAULT


doNotifyingAbort

AbortTransaction

CleanupTransaction





RecordTransactionAbortXLogInsert(RM_XACT_ID, XLOG_XACT_ABORT)

Need to call rollback or end manually here!!!PostgresMain

firstchar = 'Q'

exec_simple_querystart_xact_command


Block state: TBLOCK_INPROGRESS -> TBLOCK_ABORT

PortalRunProcessUtility

TRANS_STMT_ROLLBACK

UserAbortTransactionBlockBlock state: TBLOCK_ABORT -> TBLOCK_ABORT_END

CommitTransactionCommandfinish_xact_command


Kill RM-0 before master write log, RM-1 Prepared

--------------------some RM Failed----------------------

PostgresMain


finish_xact_command

CommitTransaction




TRANS_INPROGRESSTBLOCK_ENDDTX_STATE_PREPARING

AbortTransaction


Gxactstate: DTX_STATE_PREPARING -> DTX_STATE_RETRY_ABORT_PREPARED


disconnectAndDestroyAllGangs

GP Kill RM-1

doDtxPhase2Retry


DTX_STATE_RETRY_COMMIT_PREPARED

StartTransaction

TRANS_DEFAULT -> TRANS_START


GP Renew all RMs

PostgresMain

BeginCommand



firstchar = ‘T'

dtxProtocolCommand = DTX_PROTOCOL_COMMAND_ABORT_PREPARED

performDtxProtocolAbortPrepared




DTX_CONTEXT_LOCAL_ONLY -> DTX_CONTEXT_QD_RETRY_PHASE_2

DTX_CONTEXT_QD_DISTRIBUTED_CAPABLE -> DTX_CONTEXT_LOCAL_ONLY


DTX_CONTEXT_QD_RETRY_PHASE_2 -> DTX_CONTEXT_LOCAL_ONLY

finishDistributedTransactionContext

Do nothing on RM-0

Do Abort Prepared on RM-1


StartTransactionCommandFinishPreparedTransaction


RecordTransactionAbortPrepared

XLogFlushXLogInsert (RM_XACT_ID, XLOG_XACT_ABORT_PREPARED

GP Kill all RMs then



Kill RM-0 when master send commit to RMs

--------------------some RM Failed----------------------

PostgresMain

disconnectAndDestroyAllGangs

GP Kill RM-1

doDtxPhase2Retry

StartTransactionCommandDTX_STATE_RETRY_COMMIT_PREPARED

StartTransactionTRANS_DEFAULT -> TRANS_START

DTX_PROTOCOL_COMMAND_COMMIT_PREPARED


GP Renew all RMs

DTX_CONTEXT_LOCAL_ONLY -> DTX_CONTEXT_QD_RETRY_PHASE_2

DTX_CONTEXT_QD_DISTRIBUTED_CAPABLE -> DTX_CONTEXT_LOCAL_ONLY

CommitTransactionCommandDTX_CONTEXT_QD_RETRY_PHASE_2 -> DTX_CONTEXT_LOCAL_ONLY

Do nothing on RM-0

Do Commit Prepared on RM-1

performDtxProtocolCommitPrepared



RecordTransactionCommitPrepared

XLogFlushXLogInsert (RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED

doInsertForgetCommittedDtx state : DTX_STATE_RETRY_COMMIT_PREPARED -> DTX_STATE_INSERTING_FORGET_COMMITTED

Dtx state : DTX_STATE_INSERTING_FORGET_COMMITTED -> DTX_STATE_INSERTED_FORGET_COMMITTED






RecordTransactionCommitCommitTransaction

XLogFlush

Dtx state : DTX_STATE_NOTIFYING_COMMIT_PREPARED-> DTX_STATE_RETRY_COMMIT_PREPARED


Kill all after RMs write prepare log

---------------------------- ---------------------------- ------------------- Kill All TM/RMs and restart ------------------- ---------------------------- --

PostgresMain

DTX_PROTOCOL_COMMAND_PREPARE


PostgresMain


firstchar = ‘T'

XLogFlush


Gp_role == GP_ROLE_DISPATCH

initTMinitTM_recover_as_needed

recoverTMrecoverInDoubtTransactions

gatherRMInDoubtTransactions

cdbdisp_dispatchRMCommand(select gid from pg_prepared_xacts)

First char: 'M';Return indoubt transactions

abortRMInDoubtTransactionsdoAbortInDoubt




RecordTransactionAbortPreparedXLogInsert (RM_XACT_ID, XLOG_XACT_ABORT_PREPARED | XLOG_NO_TRAN)


segmentMaxDistributedTransactionId


cdbdisp_dispatchRMCommand(select gp_max_distributed_xid())

First char: 'M';Return gp_max_distributed_xid

First char: ‘T';

DTX_PROTOCOL_COMMAND_RECOVERY_ABORT_PREPARED

XLogFlush

shmNumGxacts = 0

ReplayRedoFromUtilityModeredoDistributedCommitRecord


Kill all after TMs write commit log

---------------------------- ---------------------------- ------------------- Kill All TM/RMs and restart ------------------- ---------------------------- --

PostgresMain

DTX_PROTOCOL_COMMAND_COMMIT_PREPARED


PostgresMain


firstchar = ‘T'

XLogFlush


Gp_role == GP_ROLE_DISPATCH

initTMinitTM_recover_as_needed

recoverTMrecoverInDoubtTransactions

shmNumGxacts = 1

gatherRMInDoubtTransactionscdbdisp_dispatchRMCommand(select gid from pg_prepared_xacts)

segmentMaxDistributedTransactionId


cdbdisp_dispatchRMCommand(select gp_max_distributed_xid())

First char: 'M';Return gp_max_distributed_xid

ReplayRedoFromUtilityModeredoDistributedCommitRecord

dispatchDtxProtocolCommand

doNotifyCommittedInDoubt

performDtxProtocolCommitPrepared


RecordTransactionAbortPreparedXLogInsert (RM_XACT_ID, XLOG_XACT_COMMIT_PREPARED | XLOG_NO_TRAN)


First char: ‘T';

XLogFlush




First char: 'M';Return 0 indoubt transactions


savepoint s1; PostgresMainfirstchar = ‘M'

exec_simple_query

start_xact_command

PortalRun

PortalRunMulti


finish_xact_command

state: TRANS_DEFAULT-> TRANS_START-

StartSubTransaction

Block state: TBLOCK_SUBBEGIN -> TBLOCK_SUBINPROGRESS

state: TRANS_START-> TRANS_INPROGRESS

dispatchDtxCommand (SAVEPOINT s1)


finish_xact_command


StartSubTransaction

Block state: TBLOCK_SUBBEGIN -> TBLOCK_SUBINPROGRESS


ProcessUtilityPortalRunUtility

DefineDispatchSavepoint

DefineSavepoint

PushTransaction

TRANS_STMT_SAVEPOINT

TRANS_DEFAULTTBLOCK_SUBBEGIN

ProcessUtility

DefineSavepoint

sendDtxExplicitBegin();


rollback to savepoint s2 PostgresMain

firstchar = ‘M'

exec_simple_query

start_xact_command

PortalRun

PortalRunMulti


finish_xact_command


StartSubTransaction


dispatchDtxCommand (rollback to savepoint s2)


finish_xact_command

AbortSubTransaction

state: TRANS_INPROGRESS -> TRANS_ABORT

ProcessUtility

PortalRunUtility

RollbackToSavepoint

TRANS_STMT_ROLLBACK_TO

ProcessUtility

RollbackToSavepointTRANS_STMT_ROLLBACK_TO

Sub state: TBLOCK_SUBINPROGRESS -> TBLOCK_SUBABORT_PENDING

State: TBLOCK_SUBINPROGRESS -> TBLOCK_SUBRESTART

CleanupSubTransaction

state: TRANS_ABORT -> TRANS_DEFAULT

DispatchRollbackToSavepoint

Sub state: TBLOCK_SUBINPROGRESS -> TBLOCK_SUBABORT_PENDING

State: TBLOCK_SUBINPROGRESS -> TBLOCK_SUBRESTART

AbortSubTransactionstate: TRANS_INPROGRESS -> TRANS_ABORT

CleanupSubTransactionstate: TRANS_ABORT -> TRANS_DEFAULT

RecordSubTransactionAbort

XLogInsert(RM_XACT_ID, XLOG_XACT_ABORT)

DefineSavepoint

PushTransactionTRANS_DEFAULTTBLOCK_SUBBEGIN

StartSubTransactionState: TBLOCK_SUBBEGIN -> TBLOCK_SUBINPROGRESS

DefineSavepoint

PushTransactionTRANS_DEFAULTTBLOCK_SUBBEGIN

StartSubTransactionState: TBLOCK_SUBBEGIN -> TBLOCK_SUBINPROGRESS


release savepoint s2 PostgresMain

firstchar = ‘M'

exec_simple_query


finish_xact_command

state: TRANS_INPROGRESS -> TRANS_COMMIT

CommitSubTransaction


finish_xact_command

CommitSubTransaction

state: TRANS_INPROGRESS -> TRANS_COMMIT

ProcessUtilityTRANS_STMT_RELEASE

ProcessUtility

ReleaseSavepointTRANS_STMT_RELEASE

dispatchDtxCommand

While Sub blockState: TBLOCK_SUBEND

If blockState: TBLOCK_END

CommitTransactionState: TBLOCK_END -> TBLOCK_DEFAULT

If blockState: TBLOCK_PREPARE

PopTransaction

Commit with spState: TRANS_BEGIN -> TRANS_PREPARE

PrepareTransaction

XLogFlush

EndPrepare


State: TRANS_PREPARE -> TRANS_DEFAULT


......

Debug tips

• regres01 55749 55656 0 08:50 ? 00:00:00 postgres: port 5432, regres01 postgres [local] con6 [local] cmd5 idle in transaction

• regres01 56475 55619 0 09:28 ? 00:00:00 postgres: port 40000, regres01 postgres localhost(6655) con6 seg0 idle in transaction

• regres01 56478 55620 0 09:28 ? 00:00:00 postgres: port 40001, regres01 postgres localhost(14862) con6 seg1 idle in transaction

• gp_vmem_idle_resource_timeout is for idle mem time in case the resource will be released

• gpconfig -c gp_vmem_idle_resource_timeout –v 300

Debug tips• b StartTransactionCommand

• b CommitTransactionCommand

• b AbortCurrentTransaction

• b BeginTransactionBlock

• b EndTransactionBlock

• b UserAbortTransactionBlock

• b DefineSavepoint

• b RollbackToSavepoint

• b ReleaseSavepoint

• b StartTransaction

• b CommitTransaction

• b AbortTransaction

• b CleanupTransaction

• b StartSubTransaction

• b CommitSubTransaction

• b AbortSubTransaction

• b CleanupSubTransaction

• b performDtxProtocolCommand

• b exec_mpp_dtx_protocol_command

• b recoverTM

• b recoverInDoubtTransactions

• b gatherRMInDoubtTransactions

• b abortRMInDoubtTransactions

• b cleanRMResultSets

• b resolveInDoubtDtx

• b doDispatchDtxProtocolCommand

• b doPrepareTransaction

• b doInsertForgetCommitted

• b doNotifyingCommitPrepared

• b doNotifyingAbort

• b doNotifyCommittedInDoubt

• b doAbortInDoubt

• b doQEDistributedExplicitBegin

• b UtilityModeSaveRedo

• b ReplayRedoFromUtilityMode

• b RemoveRedoUtilityModeFile

• b performDtxProtocolCommitPrepared

• b performDtxProtocolAbortPrepared

• b cdbdisp_debugDispatchResult

• b ProcessUtility

• b prepareDtxTransaction

• b PrepareTransaction

• b XLogFlush

• b ReadyForQuery_QEWriter

• b RecordDistributedForgetCommitted

• b RecordTransactionCommit

• b RecordCrashTransactionAbortRecord

• b XLogInsert

• b XLogFlush

• b forcedDistributedCommitted

• b rollbackDtxTransaction

• b dtmPreCommand

• b PrepareTransaction

• b disconnectAndDestroyAllGangs

61

Software

Learning Greenplum Distributed Transaction