Sample for Building a DataStage Job Using Change Data ...files. capture...Sample for Building a DataStage Job Using Change Data ... the DataStage Job used in the certification ... Certification Report

  • View
    214

  • Download
    1

Embed Size (px)

Transcript

  • SampleforBuildingaDataStageJobUsingChangeDataCapture1.5AccessingASIQ12.4.0

    ThissampleprovidesadescriptionoftheDataStageJobusedinthecertificationprocesstocapturethelatestreplicatedtransactionsfromOracleintoASIQusingCDCwithReplicationServerandRAO.

    Thestepstocreatethisprocessfollowandassumethefollowing:

    Allproductsareinstalledandconfiguredproperly(ArdentDataStageandCDC,ASIQ,ODBC,ASE,RAO,Repserver).Thereplicationenvironmentwasconfiguredandinproductionalready.ODBCconnectivitynecessaryforDataStagetocommunicatewithASIQserverisconfigured.ThetablesexistinASIQ.Inthisparticularcaseweareusingatablewithmultipledatatypes.

    ThefollowingscreendisplaystheDataStageJobthathandlesthelogicofreplicatedevents(INSERT,DELETE,UPDATE)comingfromOraclethroughReplicationServerandCDCtopassintoASIQ.RememberthatASIQserversdonotsupportUPDATEstatements.ThesetransactionswillbehandledasaDELETE/INSERTtransaction.AbriefdescriptionofthisJobcouldbeasfollows:

    1. ThereplicatedtransactionscomingfromRepserver(ReplicationServer)andcapturedbyCDC(ChangeDataCapture)arereadbytheDataStageCDCPlugin.

    2. Thenthegoalistocreatetwodifferentfiles:onesequentialfile(DeleteKeys)thatwillholdalltheDELETEandUPDATEtransactionsandaHashedFile(LastChangeStage)thatwillholdthelasttransactionforeachreplicatedrow.

    3. Havingthesetwofiles,loadtheSequentialfileintoatemporaltablewithinASIQanddefinetherightlogicprocess(insidetheASIQPluginStage)tofirstdeletetherowsfromthetargettablethatarepartofthatSequentialfile.

    4. LoadthedatastoredintheHashedfilepassingonlytheINSERTandUPDATEtransactionstoASIQ.

  • 1.CDCPluginStage(ReadChanges)

    Inthefollowingsections,themostimportantdescriptionforeachmodulewillbediscussed.Thestageforeachmodulewillbemarkedasshowninthefollowingscreen.WewillstartthedescriptionofCDCPluginStage.

  • 1a)IntheCDCPluginmodule,thefollowingpanelsshowtheinformationrequired.IntheGeneralpanelaprojectnameisrequired.WithinProperties,wewillusethesubscriptiondefinitionthatwasdefinedusingthesamestepsasshowninthesection"SetupandConfiguration"ofReport154.

    1b)TheoptionLoad(showninthefollowingscreen)isusedtogetthecolumninformation,whichwaspreviouslycreatedwhenthesubscriptionwasdefinedwithCDCConfigurationutility.

    2.TransformationStage(T1)Inthisstagewewillsplitthetransactionsdependingontype(UPDATE,DELETESorINSERT)forfurtherprocessing.

  • 2a)Createthelinksbetweenmodules,asshowninthefollowingscreen.

  • 2b)DoubleclickintheConstraintareatoaddit.Inthiscaseanyoperationmarkedas2(UPDATE)and3(DELETE)willbepassedtolinkDeletesToAgg.

  • 3.AggregationStage(AggIDs)

    3a)Descriptionofthisstage.

  • 3b)DefinethecolumnswithintheInputspanel.Note:checkhowDisplayvalueisgreaterthanLength,weranintoproblemstryingtousethesamevaluesotheonlywaywewereabletoCompiletheJobwastoincreasethevalueforDisplay.

    3c)Asdescribedbefore,thecountoperationisusedtogivetheAggregatorafunctiontoperformwithintheOutputspanel.

  • 4.TransformationStage(T3)

    4a)Createthederivationbetweenstagesandmakesurethatforfillercolumnyoudefineadouble".

  • 5.SequentialFileStage(DeleteKeys)

    5a)FortheStagepaneldefinethedirectoryinwhichtostorethesequentialfile.

  • 5b)NamethesequentialfileusingtheOverwriteoptionwithinInputspanel.

    5c)Followingaretheformatoptionsusedinthesequentialfile.

  • 5d)Definetheoutputcolumnsthatwillbepartofthisfile.

    6.HashedFileStage(LastChangeStage)

  • 6a)Descriptionofstage.

    6b)Definethefilenameusing"Clearfilebeforewriting"option.

  • 6c)LoadthecolumndefinitionsfromMetadata.Note:checkhowtheorderinthisfilechanges.Thefirstcolumnis"row_number_new"insteadof"Timestamp"andtheKeyfieldismarkedwith"Yes".ThereasonisthisisourkeytoselectthelastchangeforaspecificrowandaTimestampfieldwouldnotbeusefulforthispurpose.AsmalltipforthisoperationistousethesameCDCmetadatafilethatwassavedatsubscriptiondefinitionwithintheCDCConfigurationutilitythenjustremovethefirsttwocolumns(TimestampandOperation)andreloadthesamemetadata,sincetherestofthecolumnshavebeenalreadyloaded.Thetwomissingcolumnswillbeinsertedattheend.

    6d)DefinethesamefilenameontheOutputspanel.

  • 6e)Loadthecolumndefinitionsusingtheprocess,asdoneintheInputspanel.

    7.TransformationStage(T2)

  • 7a)Createtherightderivationforeachcolumn.Inthiscase,thenewcolumnvalueswillbepassedtotheInsertsToASIQLinkasshowninthenextfigure.

    7b)TheConstraintdefinedonthisstagewillallowonlythoseoperationsmarkedas1(INSERT)and2(UPDATE)tobepassed.InotherwordstheDELETEstatementsareignoredinthisstage.

  • 8.ASIQPluginStage(Insert)

    8a)IntheStagePanelPropertieswedefinetheallthenecessaryinformationtomaketheconnectionforASIQusingtheODBCdriver.

  • 8b)ThisInputs>PropertiespanelhandlesthemostimportantpieceofthisJob.DefinedinthePreInsertCommandoptionistheLOADoftheDeleteTableusingtheDeleteSequentialfile(previouslydefined).Note:theDeleteTablehastobecreatedwithinASIQserverbeforeyourunthisjob.

    8c)AftertheLOADofdataforDeleteTableexecutesthenthenextDELETEsentencewillexecute(whichdeletesfromthetabletheDELETE,UPDATEtransactions)previoustothefinalLOADthatcontainstheINSERT,UPDATEtransactions.

  • 8d)LoadthecolumndefinitionsusingthemetadatacreatedwiththeODBCoption.

    8e)SaveandcompiletheJob,andlaterrunitusingDataStageDirector(toseemoredetailsonthisyoucanchecktheCertificationReport#138thatdescribesinmoredetailtheprocessofcreating,compiling,verifyingandrunningaJobwithDataStage).Inordertogiveyouanidea,yourSQLfile(theonecreatedforASIQPlugin)shouldlookprettysimilartotheoneusedinourtests.KeepinmindthattherecouldbeotherwaystocreateaJobthatwilluseadifferentlogic,butattheendtheresultsshouldbeidentical.

  • ThisconcludestheDataStageJobdescriptiontocapturereplicatedeventsintoASIQusingArdent'sChangeDataCapturePlugin.