clearMDM - QuickStart Guide 2 - Normalisation · 2017. 10. 8. · Appendix C – Troubleshooting ... on completion, unless Auto Normalisation is applied where the record level flag

clearMDM®isaregisteredtrademarkofAudit9LtdacompanyregisteredintheUnitedKingdomwithcompanynumber08088394. 1

QuickStartGuide2-NormalisationDocumentVersion:v1.3ProductVersion:v2.11Date:8thOctober2017ThisdocumentprovidesanoverviewandStep-by-StepimplementationinstructionsfortheclearMDMNormalisationMDMoperation.ThedocumentAppendicesalsoprovideadditionalreferencematerials.ForpractitionerguidanceinrespecttotheimplementationofclearMDMpleaserefertotheImplementationModeldocumentationprovidedonthewebsite,oruponrequest.

TableofContents

QuickStartGuide2-Normalisation...................................................................................1MDMProcessOverview.............................................................................................................2NormalisationIntroduction........................................................................................................3

Definition.......................................................................................................................................3KeyConcepts.................................................................................................................................3NormalisationRuleTypes..............................................................................................................5NormalisationMethods.................................................................................................................6

Step1–ConfigureApplicationSettings......................................................................................7Step2–ConfigureTargetObjectSettings...................................................................................8Step3–ConfigureaBasicNormalisationRule............................................................................9Step4–ConfigureaLookupNormalisationRule......................................................................10Step5–ConfiguretheBlockingKey..........................................................................................11Step6–RuntheNormalisationJob..........................................................................................12Step7–ViewNormalisationResults........................................................................................13AppendixA-NormalisationSettingsReference........................................................................14AppendixB–BlockingKeyStatistics.........................................................................................15AppendixC–Troubleshooting..................................................................................................16

NormalisationLogTypes.............................................................................................................16


MDMProcessOverviewThediagrambelowprovidesanoverviewofthecoreMDMoperationsinsequence.ThisdocumentrelatestoSTEP1–RECORDNORMALISATION.

STEP 1 - RECORD NORMALISATION

Target ObjectERP_Company__c

STEP 2 - RECORD MATCHING

Target ObjectAccount

Source ObjectERP_Company__c

External Data Source: ERP CompaniesTarget Object = AccountSource Object = ERP_Company__c

Matched Record Pair

ObjectMatching

RECORD MATCHING

Matched Record Pair

ObjectMerge



RECORD MATCHING

Reparenting



Child ObjectInvoice__c Existing Parent

Relationship

Identifiers

DataSource.Acompa)bleobject(orpar))on)thatprovidesSourceRecordstoMDMopera)ons.

SourceRecord.SourceRecordsaretheinputtoMDMopera)onsandmayberelatedtoaMasterRecord.

STEP 3 - RECORD MERGESTEP 4 - DATA CONSOLIDATION

Normalisation


Normalisation

Inputs >ERP Company Name = Barista Coffee LimitedCountry__c = UK

Outputs >ISOCountryCode__c = GBStandardisedName__c = BARISTA COFFEEBlockingKey__c = GBBARISTA

Inputs >Name = Barista Coffee LtdBillingCountry = Great Britain

Outputs >ISOCountryCode__c = GBStandardisedName__c = BARISTA COFFEEBlockingKey__c = GBBARISTA

TargetObject.Acompa)bleobjectthatisdefinedasthetargetforNormalisa)onopera)onsandwhereMasterRecordswillbecreatedorupdated.AllMDMenabledobjectsmustbenormalised.

Internal Data Source: AccountsTarget Object = AccountSource Object = Account

Matching Engine Pass 1:Gather records where the Blocking Key starts with “GBBARI”. Assuming a Blocking Key Match Length setting value of 6.

Pass 2:Compare each unique record pairing within the “GBBARI” group using the Matching Rules defined for the Target Object. Store matches in the Matched Record Pair Object. Matched Record Groups relate Matched Record Pairs within a transitive group.

Master Record Relationship

Matched Record Pair :Record1Id = ‘001….’Record2Id = ‘00X….’RecordGroupId = ‘001….’Target Object = AccountRecord1DataSource = ‘Accounts’Record2DataSource = ‘ERP Companies’Match Score% = 72%Match Status = Candidate

Each Matched Record Group is processed into a single Master Record Create or Update operation.Trust rules control the flow of field values to the Master Record.

Identifiers can be copied from the Source Records to the Master Record and/or Source Records can be related to the Master Record via relationship field.

Source Records can be deleted or retained.

Output >Name = Barista Coffee LtdMDM Status = Merge Master

Output >Name = Barista Coffee LimitedMDM Status = Merge SourceMaster Record = [Account: Barista Coffee Ltd]

MasterRecord.TheMasterRecordistheoutputoftheConversionandMergeMDMopera)onsandisop)onallyrelatedtotheunderlyingSourceRecordsviarela)onshipfieldorsimplythroughconcatena)onofiden)fiersintoafieldontheMasterRecord.

New Parent Relationship

Input >Name = Barista Coffee LtdMDM Status = Merge Master

Input >Name = Barista Coffee LimitedMDM Status = Merge SourceMaster Record = [Account: Barista Coffee Ltd]

Apply Normalisation Rules to Standardise Field Values.

Construct the Blocking Key used to group records by the first pass of the Matching Engine.

A relationship is established between the Master Record and Source Record Related Data. Filters can be applied to ensure only relevant objects and records are reparented.

Once Related Data is re-parented Custom Rollups can aggregate KPI to the Master Record across the consolidated dataset.

RelatedData.ChildrecordsrelatedtoaSourceRecordwherearela)onshiptotheMasterRecordisrequiredtoestablishaconsolidatedview.

Master Record Relationship


NormalisationIntroductionDefinitionNormalisation is the first step of the MDM process and performs two key functions; fieldstandardisationandBlockingKeypopulation. Informercaserecordmatchingaccuracy is improvedthroughthepre-standardisationofkeyinputfields.Examplesbeingcountrynamestandardisationtothe ISO Country Code (United Kingdom, Great Britain etc. to GB), or removal of spurious specialcharactersorirregularlyoccurringsuffixes(Ltd,LLP,Sretc.).Inthelattercasethefirst-phaseoftheMatchingenginegroupsrecordsbyacommonBlockingKeyvaluebeforefiner-grainedMatchingRulesareapplied.TheBlockingKeyisconstructedfromcharacterstakenfromupto3inputfields.RecordswithoutacompleteBlockingKeyarenotmatchedatall,recordsmustshareacommonBlockingKeyto be matched together; the Blocking Key structure is therefore a critical decision point whenconfiguring clearMDM that requires understanding of the population characteristics of the targetdataset.KeyConceptsConcept DefinitionNormalisationSettings NormalisationisconfiguredperTargetObjectontheTarget

ObjectSettingspage.EveryobjectthatexposesdataformatchingmusthaveaTargetObjectdefinitionandNormalisationsettings.Wherecross-objectmatchingisimplementedtheBlockingKeystructuremustbeuniformacrosstheTargetObjectsettingsforallobjects.Asetting-by-settingdefinitionfortheNormalisationSettingsisprovidedinAppendixA.

NormalisationState AgivenrecordiseitherinaNormalisedstateornot.The[IsNormalised?]record-levelflagistypicallyconfiguredtorecordthiskeystatevalue.TheNormalisationMDMoperationwillsetthisflagtotrueoncompletion,unlessAutoNormalisationisappliedwheretherecordlevelflagisignored.WhereAutoNormalisationisnotusedtherecord-levelflagisusedtoisolaterecordsthatrequirenormalisation,i.e.onlyrecordswheretheflagisfalseareprocessed.clearMDMcanbeconfigured(“CheckNormalisationStateonSave?”)tomanagetheflagvalue;recordmodificationsareevaluatedtodeterminewhetheraNormalisationsignificantchangehasoccurred,i.e.anyfieldreferencedinthesettingsorruledefinitionshaschanged.Ifsuchachangeoccursthe


flagissettoFalse,i.e.therecordisnolongerconsideredtobeinanormalisedstate.

NormalisationRules NormalisationRulestakeaninputfield,applyaruleandpopulateanoutputfieldwiththeproduct.TheoutputfieldscanthenbeutilisedinthecompositionoftheBlockingKeyandwithinMatchingRules.ThesupportedNormalisationRuletypesaredefinedinthefollowingsubsection.

BlockingKey TheBlockingKey(orMatchingKey)providestheinitialgroupingofrecordstowhichMatchingRulesareapplied.TheBlockingKeyisconstructedfromcharacterstakenfromupto3inputfields–theselectedfieldsmusthaveahighpopulationpercentage,bestaticdatai.e.notsubjecttofrequentchange(non-volatile)andincombinationcovernomorethanapproximately250recordsacrossthedataset.RecordswithoutacompleteBlockingKeyarenotmatchedatall,recordsmustshareacommonBlockingKeytobematchedtogether;theBlockingKeystructureisthereforeacriticaldecisionpointwhenconfiguringclearMDMthatrequiresunderstandingofthepopulationcharacteristicsofthetargetdataset.

BlockingKeyStatistics ArobustBlockingKeyStructureiscriticaltotheabilitytoperformcompleteandaccuraterecordmatching.GiventhestateddependencyonthepopulationcharacteristicsofthedatasetanempiricalmechanismtoprojectthedistributionofrecordsacrossBlockingKeysisrequiredtosupportaniterativeprocessforthedeterminationofanoptimalBlockingKeystructure.clearMDMprovidesaBlockingKeyStatisticstoolforthispurpose,furtherdetailisprovidedinAppendixB.


NormalisationRuleTypesNormalisationRulestakeaninputfield,applyaruleandpopulateanoutputfieldwiththeproduct.Thesupportedruletypesareoutlinedinthetablebelow.RuleType DefinitionEmailDomain {[email protected]=>contosio.com}

Stripthedomainfromanemailaddress.

Format {Johnsmith=>JOHNSMITH}Removewhitespaceandspecialcharacterssuchas*}[>.

Lookup {UnitedKingdom,UK=>GB}{Will,Bill=>William}ACustomSettingcanbereferenced(e.g.ISOCountryCodes)thatmatchestheinputfieldvalue(e.g.BillingCountry=“UnitedKingdom”)toastandarisedvalue(“GB”).TheCustomSettingshouldhavea“MatchList”fieldofcomma-separated,non-standardisedvaluesrelatingtoastandardisedequivalentrecordedintheNamefield.TheclearMDMpackageprovidestheISOCountryCodesettingasanexample.Note,allclearMDMNormalisationrelatedCustomSettingsareinstalledwithasmallnumberofexamplerecordsonly,beforeusesuchsettingsshouldbefullypopulated.Note,inadditiontotheexampleCustomSettings,anyCustomSettingdefineddirectlyinthecustomerorgcanbereferencedbyNormalisationRules.

RemoveEnd {JohnSmithSnr=>JohnSmith}ACustomSettingcanbereferenced(e.g.LastNameSuffixes)thatreturnsalistofvalues(e.g.“Ltd”or“Esq”)toremovefromtheinputfieldvalue.

Title {Laterassa=LaTerrassa}EnsureeachwordinthetextisTitle-cased.

TitleHyphen {Laterassa=La-Terrassa}EnsureeachwordinthetextisTitle-casedandwordsareseparatedbyhyphensnotspaces.

URL {www.contosio.com=>contosio.com}Stripthedomainfromawebaddress(orfully-formedUrl).


NormalisationMethodsThetablebelowoutlinesthesupportedmethodsforinvocationoftheNormalisationMDMOperation.SettingreferencesrefertotheTargetObjectNormalisationsettings.Method DefinitionBatchJob TheclearMDMJobspagecanbeusedtoscheduletheNormalisationMDM

operationtorunforagivenTargetObjectimmediatelyoronascheduledbasis.The[AutoNormalise?]settingcontrolswhetherallrecordsarenormalised,irrespectiveofNormalisationState,orwhetherjustrecordsthatrequirenormalisationareprocessed.WhereadailybatchprocessingmodelisimplementedforMDMprocessing,theNormalisationMDMoperationwilltypicallybethefirstjobandwillinvokethenextjobinthesequenceusingthejobchainingsettingse.g.MatchingSettingssection,IsInvokedbyNormalisationJob?flag.Note,wherethevolumeofrecordstonormaliseexceeds1million,theprocesscountmustbesetto1.Inthismodetheprocesspartitioningjobphaseisskippedandthehardplatformlimitof1millionqueriedrecordsisavoided.

OnRecordCreate RecordscanbeNormalisedoncreationtoensurestandardisedfieldvaluesandtheBlockingKeyarepopulatedatthetimeofrecordcreation.Thismethodrequiresthe[NormaliseonCreate?]settingtobesettoTrueandtheApplicationSettings–IsTriggersActive?settingtobesettoTruealso.

OnRecordSave RecordscanbeNormalisedonsavetoensurestandardisedfieldvaluesandtheBlockingKeyarealwaysupdatedinlinewithfieldvaluemodifications.Thismethodrequiresthefieldspecifiedbythe[NormaliseonSaveFieldName]settingtobesettoTrueoneachrecordandtheApplicationSettings–IsTriggersActive?settingtobesettoTrue.

Action RecordscanbeNormalisedbyProcessBuilderaspartofacustomprocessautomation.ToconfigureaProcessBuilderActionforthispurposeaddanActionwithpropertiessetasbelow.

1. ActionType=Apex2. ApexClass="NormaliseRecordsAction”3. RecordIDparameter=Reference[ObjectIdfield].

TheActioncanalsobeimplementedwithinVisualWorkflow.

API RecordscanbeNormalisedbyacustomactionexposedviathestandardForce.comRESTAPI;endpointbelow./services/data/v38.0/actions/custom/apex/clearmdm__ RecordNormalisationAction TheAPIoperationtakesasinglerecordIdparameter.FurtherdetailscanbefoundintheclearMDMAPIGuide.


Step1–ConfigureApplicationSettingsPre-requisite:clearMDMmustbesettoActiveviatheApplicationSettingspage.Pre-requisite:clearMDMmustbesetwithTriggersActiveviatheApplicationSettingspage.Tocompletethisstep,firstnavigatetotheMDMApp,opentheSettingstab,tickthetwofieldshighlightedbelowandclicktheSavebutton.


Step2–ConfigureTargetObjectSettingsTocompletethisstep,firstnavigatetotheMDMApp,opentheSettingstab,selecttheTargetObjectstabandclicktheEditlinknexttotherequiredTargetObject.Next,setthe[IsActive?]flagequaltotrueandensuretherelevantsettingsareconfiguredcorrectly.AppendixAprovidesaSettingsreference.


Step3–ConfigureaBasicNormalisationRuleTocompletethisstep,firstnavigatetotheMDMApp,opentheSettingstab,selecttheTargetObjectstabandclicktheEditlinknexttotherequiredTargetObject.OntheTargetObjectsettingspageclicktheFieldstab.Next,setthe[IsNormalised?]flagequaltotruenexttotherelevantfieldandselecttherequiredruletype(i.e.Format).Finally,selecttheTargetFieldandclicktheSavebutton.In theexamplebelow theFirstName is the source field, theRuleType is Format (which removesspecialcharactersandstandardisescasing)andtheTargetFieldfortheruleistheContactNormalisedFirstNamefield.


Step4–ConfigureaLookupNormalisationRuleTocompletethisstep,firstnavigatetotheMDMApp,opentheSettingstab,selecttheTargetObjectstabandclicktheEditlinknexttotherequiredTargetObject.OntheTargetObjectsettingspageclicktheFieldstab.Next,setthe[IsNormalised?]flagequaltotruenexttotherelevantfieldandselecttherequiredruletype(i.e.Lookup).Next, set the Reference Setting, Lookup field (Match List), Reference Field (Standard Value FieldName),andtheTargetFieldandclicktheSavebutton.IntheexamplebelowtheMailingCountryistheSourceField,theRuleTypeisLookupandtheTargetFieldfortheruleistheISOCountryCodefield.


Step5–ConfiguretheBlockingKeyTocompletethisstep,firstnavigatetotheMDMApp,opentheSettingstab,selecttheTargetObjectstabandclicktheEditlinknexttotherequiredTargetObject.Next,settherequiredBlockingKeysettingsashighlightedbelow.AppendixAprovidesaSettingsreference.FinallyclickSavetoapplytheconfigurationchange.


Step6–RuntheNormalisationJobTocompletethisstep,firstnavigatetotheMDMApp,opentheJobstab,selecttheNormalisationJobtypeandtherequiredTargetObject.FinallyselecttherequiredJobScheduleandclickthe“ScheduleJob”button.


Step7–ViewNormalisationResultsThe screenshot below shows the Normalisation MDM fields displayed on an example record.Placement of MDM fields on layouts for administrators, data stewards and end-users is a keyimplementationdecision.


AppendixA-NormalisationSettingsReferenceNormalisationsettingsareconfiguredperTargetObjectontheTargetObjectsettingspage.Setting DefinitionIsActive? IfsettoFalsetheTargetObjectwillbeexcludedfromall

NormalisationMDMoperation.NormaliseonCreate? IfsettoTrue,newrecordswillbeNormalisedautomaticallyatthe

timeofcreationirrespectiveofhowtherecordiscreated.AutoNormaliseRecords? IfsettoTrue,allrecordswillbeprocessedbytheNormalisationMDM

operationwhenexecutedasaJob.IfsettoFalse,onlyrecordswheretherecord-level[IsNormalised?]flagissettoFalsewillbeprocessed.

AutoNormaliseRecentlyModified? Where[AutoNormaliseRecords?]issettoTrueprocessingcanbelimitedtorecordswherethelastmodifieddateistodayoryesterday.

IsNormalisedFieldName ThefieldontheTargetObjectinwhichtherecord-levelNormalisationStateflagisheld(TrueorFalse).

CheckNormalisationStateonSave? IfsettoTrue,recordsmodificationswillbeevaluatedtodeterminewhetheraNormalisationsignificantchangehasoccurred,i.e.anyfieldreferencedinthesettingsorruledefinitionshaschanged.IfsuchachangeoccurstheNormalisationStateissettoFalse,i.e.therecordisnolongerconsideredtobeinanormalisedstate.

NormaliseonSaveFieldName AcheckboxfieldontheTargetObjectwhichwhensettoTrueonarecordforcesnormalisationtorunforthatrecordwhensaved.OncenormalisationhascompletedthefieldwillberesettoFalse.

LastNormalisedDateFieldName ThedatetimefieldontheTargetObjectinwhichthelastnormaliseddateandtimewillbeset.

BlockingKeyFieldName ThefieldontheTargetObjectinwhichtheBlockingKeyispopulated.BlockingKeyGroupFieldName ThefieldontheTargetObjectinwhichtheBlockingKeyGroupis

populated.BlockingKeyInput1FieldName AfieldontheTargetObjectwhichsuppliesthe1stBlockingKeyInput.BlockingKeyInput1Length Thenumberofcharacterstotakenfrominput1.BlockingKeyInput2FieldName AfieldontheTargetObjectwhichsuppliesthe2ndBlockingKeyInput.BlockingKeyInput2Length Thenumberofcharacterstotakenfrominput2.BlockingKeyInput3FieldName AfieldontheTargetObjectwhichsuppliesthe3rdBlockingKeyInput.BlockingKeyInput4Length Thenumberofcharacterstotakenfrominput3.IsBlockingKeyCompleteFieldName ThefieldontheTargetObjectinwhichtherecord-levelIsBlocking

KeyCompleteflagisheld(TrueorFalse).Onlyrecordswithacompleteblockingkeyarematched.Recordscanhaveanincompleteblockingkeywhereinputfieldsarenotpopulatedorthelengthislessthanthedefinedinputlengthandpaddingisnotapplied.

BlockingKeyCompleteStrict? IfsettoTrue,allBlockingKeyInputfieldvaluesmustbepopulatedandbeatleastthedefinedinputlength,otherwisetheblockingkeyisnotconstructed.

BlockingKeyInputPadding? IfsettoTrue,paddingcharacters‘_’areaddedwhereBlockingKeyInputfieldvaluesarepopulatedbutlessthanthedefinedinputlength.Paddingallowsablockingkeytoconstructedincaseswhereminoradjustmentsaresufficient.Paddingwillnotaddresscaseswheretheinputfieldisblank.


AppendixB–BlockingKeyStatisticsTheBlockingKeyStatisticsjobcalculatesadistributionofrecordsacrosstheBlockingKeyMatchvaluesgeneratedbythecurrentNormalisationandMatchingsettings.ThepurposeofthejobistoenableanalysisoftheBlockingKeyMatchvaluedistribution.Keyscenariostoidentifyarelistedbelow.

1. HighSelectivity.EachBKMVgroupingcontainsaverysmallnumberofrecordsandthereforematchingwillbelimited.

2. LowSelectivity.BKMVgroupingsthatcontainsahighnumberofrecords(morethan500).Suchgroupscan’tbeprocessedbyclearMDMwiththedefaultsettingsandwillbeskippedbytheMatchingengine.

TheBlockingKeyStatisticsjobrecordsthedistributiondataintheBlockingKeyStatisticsobjectaccessiblebyTaborbystandardSalesforcereporting.Thekeyfieldsofthisobjectarelistedbelow.

• TargetObject• DataSource• BlockingKeyGroupValue• BlockingKeyMatchValue• MasterRecords• AllRecords

Whereselectivityissuesareidentified,anincrementalapproachcanbetakenwhereBlockingKeystructureandBlockingKeyMatchValuelengthchangescanbeappliedandthestatisticsjobre-run.


AppendixC–TroubleshootingAllclearMDMMDMoperationslogactivity(StartandEndtimesetc.)anderrorstotheAuditLogEntryobject.TheMDMapplication includesanAuditLogEntry tabtoprovideconvenientaccess to thisdata.EachAuditLogEntryrecordistime-stampedandrelatedtotheparentBatchJobRunrecord:allMDMoperationsthatrunviatheJobMethodcreatedaBatchJobRunrecordthatrecordsjobstatisticsandstatus.MonitoringoftheAuditLogshouldbeafrequentactivityperformedbytheAdministratororDataStewardresponsiblefortheclearMDMimplementation.BestPractice:SalesforceReportingNotificationsprovideaproactivemeansofreportingonerrorsgeneratedbyclearMDMoperations.InthismodelastandardreportisusedtoreturndatafromtheAuditLogEntryobjectwhererequiredconditionsaremet.OnlywhenrecordsmeetthecriteriaisareportsenttotheAdministratororDataStewardresponsible.NormalisationLogTypesRuleType DefinitionRecordCountExceedsLimit ThenumberofrecordsintheTargetObjectexceedsthe

editionlimit.

NormalisationRuleLookupFailed TheCustomSetting(ISOCountryCodesetc.)referencedbyaLookupruledoesnotcontainastandardisedvaluefornon-standardisedvalueencountered(onarecord).Eachinstanceofamissingvalueisloggedonceonly.Administratorscanusethisinformationtoidentifycaseswhereadditionalstandardisedvaluesarenecessary.

MissingorInactiveTargetObjectNormalisationsettings

ThiserrorcanoccurwheretheTargetObjectNormalisationsettingsareinactive,deletedortheSalesforceUserdoesnothavepermissionstotheobjectorfieldsreferenced.

Documents

clearMDM - QuickStart Guide 2 - Normalisation · 2017. 10. 8. · Appendix C – Troubleshooting ... on completion, unless Auto Normalisation is applied where the record level flag