CDW- Customer Data Warehouse_Tech

8/7/2019 CDW- Customer Data Warehouse_Tech

http://slidepdf.com/reader/full/cdw-customer-data-warehousetech 1/72

CDW- Customer DataWarehouse



� Onc or is a regulated ele c tric distributio n a ndtra nsmissio n busi ness that uses superior assetma nageme n t skills to provide reliable ele c tric itydelivery to consumers. Onc or operates thelargest distributio n a nd tra nsmissio n system i n Texas, deliveri ng power to approximately 3million homes a nd busi nesses a nd operati ngmore tha n 117,000 miles of distributio n a ndtra nsmissio n lines i n Texas. Onc or is ow ned byEnergy Future Holdi ngs corp.



CDW

� A r chite c ture O f CDWFu nc tional view of CDW

� S our ce S ystems To CDWTe chn ical Overview of CDW

� S upport a c tivities of CDW

Commo n Issues i n CDW



CDW Ar chite c ture



CDW Ar chite c ture

The e n tire CDW ar chite c ture is bei ngdivided i n to 3 stages.

� S TAG ING area.� O DS area.

CDW area (DWH).

The loadi ng S trategy is as below:

S tagi ng ODS CDW

Error Ha ndling..



S tagi ng Area..

There are 27 S tagi ng Mappi ngs a nd are prese n tin the ONC OR_S TG folder .In this area, we commu n icate to various

sour ces. The sour ces for CDW are:� L CIS : Lega cy Customer Informatio n S ystem.ERCOT: E lec tric Reliability Cou nc il of TexasePM: e -P ortfolio Ma nageme n t

� S TACKE R database.

These mappi ngs will extra c t data from these sour cesystems a nd load them to correspo nding S tagi ng tables.These tables are Tru nc ate a nd Load Daily.



ODS Area..

� Onc e the data loadi ng in to the stagi ng tables isdo ne, the correspo nding ODS jobs will ki ck start.In ODS , we have UP S ERT kind of mappi ngs.We have 65 ODS mappi ngs. These mappi ngswill be pulli ng the data from the stagi ng tablesa nd UP S ERTing the data to the ODS tables.

These mappi ngs are prese n t in ONC OR_O DS folder.



CDW Area..

The data used for reporti ng will be loadedin to the Data ware house. Cog nos pullsthe data from the tables available i n thisarea for reporti ng.There are 19 DWH mappi ngs. Thebusi ness logi c applied data will be loaded

in to these tables, via these mappi ngs.These mappi ngs are prese n t in ONC OR_ CDW folder.



CDW- Fu nc tional Overview

Why CDW?Onc or was havi ng Polaris database, but as thesize of database was i nc reasi ng o nc or teamproposed solutio n of havi ng data warehouse.And there the CDW came i n Pic ture.Curre n tly there are both the systems Polaris

a nd CDW, but as a nd whe n CDW be comescompletely reliable Polaris will not be i n thes ce ne.



CDW Fu nc tional view

CDW supports 5 appli catio ns-MCDW _ Daily

MCDW _ Hourly _ EDIMCDW _SQ_ Data _Load _ ThriceMCDW _ Daily _ Trigger

MCDW _ Hourly



MCDW _ Daily

This appli catio n runs o nc e i n a day at 9:00am C S T a nd gets completed arou nd 3:00am C S T.This appli catio n pro cesses LCIS flat filesa nd E RCOT flat files.



MCDW _ Hourly _ EDI

This appli catio n keeps o n runn ingcon tinuously.

It pro cesses the EDI tra nsa c tion data(ePM)� S oo n EDI gateway is goi ng to be

impleme n ted. And sour ce files would becomi ng dire c tly to CDW i nstead of comi ngthrough Polaris.



MCDW _SQ_ Data _Load _ Thrice

This appli catio n runs thri ce i n a day i.e4:00am, 8:00 am, 12:00 pm C S T

It pro cesses sour ce data comi ng fromLCIS in form of SQ feeds.



MCDW _ Daily _ Trigger

This appli catio n ge nerates 3 reve nuesummary reports usi ng LCIS DB2 tables.

It ru ns 5 days a week.(Mo n -Fri)



MCDW _ Hourly

This appli catio n runs everyday, everyhour.

It triggers SQ data load e ngine a nd pullsdata from M Q series a nd loads i n to SQ tables.In this o ne job is added as a n effe c t of Tamperi ng cha nge.



S our ces For CDW

CDW re ceives data from 4 differe n tsour ces. They are:

� L CISERCOTePM

� S TACKE R



From LCIS ..CDW re ceives two types of data from LCIS .

Data through Flat files.� SQ data feeds.

These are the flat files what CDW re ceives from LCIS .a nal_ mstr.datidr _ esiids.datCB LTJ _R EP.datam _ daily_ premise.datsoqu.datFMIS_ METE R .DATFMIS_ MTR_ TE S T.TXTMSAO .DATTOWNC ODE.TXT

The SQ data feeds are dire c tly fed to their correspo ndingHIS T tables, from here i nc reme n tally pull the re cordsa nd load them i n to ODS tables.



Co n td..

� SQ data feeds from LCIS providesin formatio n regardi ng S ervi ce O rders,Premises, Te na n ts i n formatio n which arenot re ceived from ePM.These feeds a c ts as a n update to thefeeds what we get from ePM.



From ePM..

CDW dire c tly pull EDIs from ePMdatabase.CDW pulls 650, 814, 867, 810, 997tra nsa c tion data.This will be loaded i n to the correspo ndingEXTRACT tables a nd subseque n tly ODS

jobs pull the data from the EXT RACT tablea nd load them i n to correspo nding mai n tra nsa c tion table.



ePM Data..Ele c tro n ic Data I n ter cha nge is the new la nguage for thenew deregulated market. It is the primary medium for informatio n ex cha nge betwee n all market parti c ipa n ts i n the new deregulated market a nd is esse n tial for marketoperatio n .

650 ± S ervi ce O rders ±> Installatio n of new meter /poll810 ± Invoices Payme n t details from TD S P to C R814 ± Customer I nformatio n (S witches) premise wa n t

to move to other C R .820 ± Payme n t O rder/ Respo nse or Co nfirmatio n Advicese nd to TD S P for more details / TD S P Fi na nc ials

867 ± Usage Power usage



TRANSACT ION LEGEND

650 - Service Orders

810 - Invoices

814 - Customer Inf ormatio n (Switches)

820 - Payme n t Order/Respo n se

or Co nf irmatio n Advice

824 - Applicatio n Advice

867 - Usage

*

l¡ ¢ rin£ ¤

¥

¦ s ¡ Tr ¢ ns ¢ cti¥ ns** §

¥ int t ¥ §

¥ int Tr ¢ ns ¢ cti¥ ns

814*

824*

867*

TDSP814*

824*

867*

CR

824**

810**

650**

820**

ePM Data..



From E RCOT..CDW re ceives a zip file from E RCOT. This zip filecon tains .C SV files. S ome of them are listed below:

0001039940674000-E S IID-05- AUG -09. csv0001039940674000-E S IIDS ERV ICEHI S T-05- AUG -09. csv0001039940674000-E S IIDS ERV ICEHI S T_ DELETE-05- AUG-09. csv

0001039940674000-E S IIDUSAG E-05- AUG -09. csv0001039940674000-E S IIDUSAG E_ DE LETE-05- AUG-09. csv0001039940674000- LS CH ANNE LCUTD ATA-05- AUG-09. csv0001039940674000- LS CH ANNE LCUTHE ADER -05- AUG -09. csv0001039940674000- LS CH ANNE LCUTHE ADER_ DELETE-05- AUG -09. csv0001039940674000.E S IID_ EXTRACT.C OUNT S_ 0836. csvMRE-05- AUG -09. csv

P GC-05- AUG -09. csv� R EP-05- AUG -09. csv� S TATION-05- AUG-09. csv

The data i n these . csv files will be loaded to their correspo nding E RCOT stagi ng tables.



From S ta cker..

Un like ePM, CDW dire c tly pulls data fromS TACKE R database.

CDW re ceives data related to 650outbou nd tra nsa c tion , meter a nd meter readi ng related i nformatio n .



LCIS : a nal_ mstr.dat

CDW re ceives Cy c le i n formatio n for aparti cular customer from this file.

This customer is ide n tified by Cust _n mfrom the file.� S tagi ng table: CI S_ ES I_ ID_ CYC LE_ CROSSR EF� O DS table: E S I_ ID_ CYC LE_ CROSSR EF

This table will be used as lookup table for variousmappi ngs. E.g.: ODS_S ERV_OR D_ IU



LCIS : idr _ esiids.dat

We will be getti ng the ID R_ DT for thosecyc le re ceived for that parti cular customer.



LCIS : CB LTJ _R EP.dat

We will be getti ng the i n formatio n regardi ng whether ACTIVE C Rs.



LCIS : am _ daily _ premise.dat

CDW re ceives the i n formatio n , whether aparti cular premise is valid or not.



LCIS : soqu.dat

CDW will be re ceiving the tra nsa c tionsmissed out i n ePM, with their error des criptio ns from LCIS through is file.



LCIS : FMIS_ METE R .DAT

Meter related i nformatio n will be re ceivedfrom this flat file. Meter ma nufa c turingcode, Meter i nstalled date, S tatus of themeter for a parti cular premise will bere ceived.



LCIS : FMIS_ MTR_ TE S T.TXT

Here also, CDW re ceives meter relatedin formatio n for a parti cular premise.

If the meter is u nder i nspe c tion , the testresults will be re ceived through this flatfile.



LCIS : MSAO .DAT

TDS P a nd C R related i nformatio n will bere ceived.

The total number of C R , for a parti cular TDS P will be re ceived from this flat file.



LCIS : TOWNC ODE.TXT

TOWN in formatio n will be re ceived fromthis flat file.

Informatio n related to tow n a nd its tow n code will be re ceived.



These files will be dire c tly loaded i n thestagi ng tables.These files from LCIS will be ar chived i n UNIX server as a zip file.

If any of these stagi ng jobs fail, just u nzipthe file a nd pla ce the file i n the SR Clocatio n a nd ru n the mappi ng.Till now we have n¶t fa ced these ki nd of issues.



SQ data.

From LCIS , CDW also re ceives datathrough M Q series. Whe n a parti cular tra nsa c tion occ urs, they will be queued upin queues. CDW pulls these data a ndloads them i n to its correspo nding SQ tables.

From these SQ data tables, it will beloaded i n to ODS HIS T tables. From there,it will be loaded to mai n tables.



S ervi ce rder a nd its remark, P remise,Te na n t, Work rder a nd its remark relatedin formatio n will be re ceived from LCIS asSQ data feeds.



CDW SQ data tables are«CIS_CS RA 0 P_SQ : Se r i e Or e r iti n l in fo r ti o n.CISRAALP_SQ : Add itio n l Te n nt in fo r ti o n.CISRANAP_SQ : Add itio n l Te n nt in fo r ti o n ( CUST ).CISRAABP_SQ : Te n nt in fo r ti o n.CISRAATP_SQ : Te n nt r ema rk in fo r ma tio n.CISRAPMP_SQ : P r em i e in fo r ma tio n.CIS_CS ORAR OP_SQ : Se r i e Or de r in fo r ma tio n.CISRABPP_SQ : B ill a r ame te r in fo r ma tio n.CIS_CS ORAEHP_SQ : Wo rk o r de r eve nt in fo r ma tio n.CIS_CS ORAUAP_SQ : Un a tt ached Se r vice Or de r in fo r ma tio n.CIS_CS ORAWRP_SQ : Wo rk o r de r r ema rk in fo r ma tio n.CIS_CS ORAW 0 P_SQ : Wo rk o r de r in fo r ma tio n.CIS_CS ORATXP_SQ : Se r vice Or de r r ema rk in fo r ma tio n.



ePM data

CDW dire c tly pulls all the 650, 814, 810,867, 997 tra nsa c tions from ePM ¶s O ra c ledatabase. These tra nsa c tions are dumpedin to their correspo nding EXT RACT tables.



S TACKE R : 650 S tate crossref

� S TACKE R a c ts like a moderator betwee n ePM a nd LCIS . While routi ng re cords fromePM to LCIS , S TACKE R runs somevalidatio ns o n ePM data. Thesespe c ificatio ns are provided i n the sta cker state xref table.



� R egardi ng the other 650 sta cker tables,one deals with the 650 tra nsa c tion , theother deals with meter i n formatio n a nd theother deals with the meter readi ng.



ERCOT : * ES IID* .zip

CDW re ceives a zip file from E RCOT. Thiszip file will be pla ced i n the sr cfiles lo catio n in UNIX server.This file con tains . csv files.This file will be ar chived a nd after unzippi ng it, the correspo nding E RCOTstagi ng tables will be loaded.



ERCOT : E S IID.csv

� A ll the E S IIDs will be loaded i n to CDWfrom this flat file.

� A long with the E S IIDs, their created datewill also loaded.� S tagi ng table: E RC_ ES IID_ACTY



ERCOT : E S IIDS ERV ICEHI S T. csv

For a parti cular E S IID, its history will beloaded i n to this table.

� S tagi ng table: E RC_ ES IID_SRV C_ HIS T_ACTY

First E S IIDs will be loaded a nd for thoseES IIDs, their history will be loadedsubseque n tly.



ERCOT :ES IIDS ERV ICEHI S T_ DE LETE. csv

This file con tains already loaded E S IIDin formatio n , with its start time. If aparti cular E S I_ ID is already prese n t in theES IIDS ERV ICEHI S T table, that parti cular ES IID will be deleted from theES IIDS ERV ICEHI S T table a nd new data

from the file be loaded.



ERCOT : MRE. csv, REP. csv,TDS P. csv

� A ll the C Rs, TD S Ps a nd all the M REs(MRE inc ludes E RCOT a nd TXU ONC OR )will be loaded i n to correspo nding stagi ngtables.These e n tities will be UP S ERTED i n totheir correspo nding ODS tables.

� A ll these ODS ERCOT tables will be usedas lookup tables, while loadi ng in formatio n from SR C to D S T.



Te chn ical Overview

Espresso- Job s cheduli ng (184 jobs)Dime nsio nal Model- S tar Sc hema

Impleme n tatio n- S CD Type2Database size - 3 Terabytes



Te chn ical OverviewFolder S tru c ture-± ONC OR_ALL_ COMMON_O BJECT S± ONC OR_ CDW± ONC OR_ ERR± ONC OR_ HIS T± ONC OR_O DS± ONC OR_ PATCH± ONC OR_S TG



S tagi ng LCIS to ODS LCIS «

The ODS CIS mappi ngs are broadly c lassifiedin to 2 categories.

Flat file mappi ngs ± 11 mappi ngs.� SQ data mappi ngs ± 24 mappi ngs.

These mappi ngs are fitted i n the folder ONC OR_O DS alo ng with other ODS mappi ngs.



SQ data load i n to ODS «

� A ll the stagi ng * _SQ * tables will be populated fromthe M Q series queue.From these * _SQ * tables, ODS HIS T tables will bepopulated usi ng the ODS HIS T mappi ngs.From these HI S T mappi ngs, the correspo ndingmai n CIS tra nsa c tion table will be populated.



SQ data load«� SQ data will be i nc reme n tally extra c ted from SQ

tables o n the basis of IN S ERT_ TIMES TAMP a nd itwill be populated i n HIS T tables.Further the ODS tables will be populated fromthese HI S T tables whi ch is i nc reme n tal UP S ERTload.



� A ll the CI S HIS T SQ mappi ngs are i nsert o n ly. It isa dire c t insert i n to the HI S T tables from _SQ tables.

� A lso S P is made to ru n as SO URCE P RE LOAD.

This will be che cking if the previous ru n wassu cc essful or not.



HIS T table to ODS mai n tableload«

� G e nerally while loadi ng from ODS HIS T(SQ ) tables to the mai n tra nsa c tion tables, it will be che cking for P REM_ IDsa nd TEN _ IDs i n TEN a nd P REM tables. If prese n t, the n the re cords are routed to

mai n tables else routed to E RROR tables.



Flat file i n to ODS load«For re cords that are re ceived from LCIS flat file aredire c tly pulled from the stagi ng table a nd inserted

in to the ODS mai n tra nsa c tion table.E.g: ODS_ CIS_A M_ DAILY_ P REMIS E_ I± This mappi ng will be dire c tly extra c ting re cords from CI S

stagi ng table a nd will be appe nding C REATE _ DT a nd

UPDT _ DT as S YS DATE to re cords that are bei nginserted.

� S ame is the case for all the LCIS flat file ODS mappi ngs.



ODS ERCOT jobs«

� A ll the E RCOT ODS jobs are IN S ERT/UPD ATEtype.



ODS ePM jobs«For ea ch a nd every EDI tra nsa c tion , we have o neODS mappi ng.

Tr a n ac tio ncode Mapp in n ame Wo rk f low n ame

650 _ 01 ODS_SRV C_OR D_RQS T_ CHN G_RS N_ P OLE_ MI wf _O DS_SRV C_OR D_RQS T_ CHN G_RS N_ P OLE

650 _ 02 ODS_SRV C_OR D_R ES P _R JCT _RS N_ MI wf _O DS_SRV C_OR D_R ES P _R JCT _RS N

650 _ 04 ODS_S US PN _ DS_ NOTIF_ CANCL_ MTR_ NUM_ MI wf _O DS_S US PN _ DS_ NOTIF_ CANCL_ MTR_ NUM

650 _ 05 ODS_S US PN _ DS_R EJ _R ES P _ MI wf _O DS_S US PN _ DS_R EJ _R ES P

814 _ 20 ODS_ MNTN_ ES I_ ID_RQS T_ MI wf _O DS_ MNTN_ ES I_ ID_RQS T

814 _ 21 ODS_ MNTN_ ES I_ ID_R ES P _R JCT _RS N_ MI wf _O DS_ MNTN_ ES I_ ID_R ES P _R JCT _RS N

814 _ 24 ODS_ MV_O UT_RQS T_ I wf _O DS_ MV_O UT_RQS T

814 _ PC ODS_ MNTN_ CUS T_ INFO_RQS T_ MI wf _O DS_ MNTN_ CUS T_ INFO_RQS T

814 _ PD ODS_ MNTN_ CUS T_ INFO_R ES P _R JCT _RS N_ MI wf _O DS_ MNTN_ CUS T_ INFO_R ES P _R JCT _RS N



ODS : 650 _ 01 load..

� S ele c t all the re cords from HE ADE R a ndall the 650 child tables, where tra nsa c tion code like 650 _ 01.DUN S_ NUM will be calculated as acombi natio n of 2 fields. i.e., It will beche cking whether DUN S_ NUM comi ngfrom the SR C is prese n t in theMRKT_ P RTCPNT table or not.



For all the re cords havi ng DUN S_ NUM willbe routed to HE ADE R ,SRV C_OR D_RQS T,SRV C_OR D_RQS T_ CHN G_RS N,SRV C_OR D_RQS T_ P OLE_ NUM,CU S T_ CONTACT table. Else, will be

routed to error table.� S o in similar way all EDI data will getloaded i n correspo nding tables.



ODS sta cker mappi ngs«

There are 4 ODS sta cker mappi ngs.

Mapp in n ame Wo rk f low n ame

ODS_S TATE _ XREF _ IU wf _O DS_S TATE _ XREF

ODS_ EDI_ 650 _O UTBND _ IU wf _O DS_ EDI_ 650 _O UTBND

ODS_ EDI_ 650 _O UTBND _ MTR_ IU wf _O DS_ EDI_ 650 _O UTBND _ MTR

ODS_ EDI_ 650 _O UTBND _ MTR_R DG_ IU wf _O DS_ EDI_ 650 _O UTBND _ MTR_R DG



ODS sta cker mappi ngs«

These mappi ngs are alsoINS ERT/UPD ATE.It will be looki ng for DUN S_ NUM in theMRKT_ P RTCPNT table. If there are nore cords mat ching, it will be routed to error table.



S upport Ac tivities i n CDW

Day to day issue ha ndlingMon thly Pro cessi ng

Holiday Pro cessi ng



Day to Day issues-

Un ique constrai n t error/ PK violatio nUnable to exte nd table spa ce segme n t

� S pa ce issueCa n not insert null

� R ollba ck segme n t too small

Fatal Error In tegrity Co nstrai n t





Mon thly Pro cessi ng- AMR File Load.



AMR File

The file for the mo n th is usually re ceivedduri ng e nd of the mo n th.

This con tains the premise, meter, a cc ou n tnumber, meter type, i nstall datein formatio n of a parti cular premise.This i nformatio n will be se n t to CDW AMteam i n a .zip file whi ch will have . Lst file.



Tables.

This i nformatio n is loaded i n to theµCDW_ USR_ CR ITER IA¶every mo n th.In order to load these i nformatio n in to theCDW table, we need to create ³amr.dat´flat file. This is the sour ce for theµCDW_ USR_ CR ITER IA¶table.

This i nformatio n will be appe nded to thetarget table.



Mappi ng used.

Mappi ng Name: ODS_ US ER_ CR ITER IA_ IWork flow Name: wf _O DS_ US ER_ CR ITER IA

� S our ce: amr.dat (Flat File)

Target: CDW _ USR_ CR ITER IA (O ra c le table)This mappi ng pulls dire c tly the re cords from theflat file a nd loads it i n to the targetµCDW_ USR_ CR ITER IA¶, by appe nding the

CREATE _ DT, C REATE _ US ER , UPDT _ DT a ndUPDT _ US ER to the re cords.



Create amr.dat

Extra c t the P REMIS E (o n ly 7 digits) from.LS T file extra c ted from the .zip folder.

� A ppe nd the P REMIS E in formatio n to theamr.dat file.Find the sample amr.dat file for the mo n thDe cember, 2008.

amr.dat



� A fter creati ng the flat file, pla ce the file i n the lo catio n ³/cdwappprd01/pm/CDW/I n fa_S hared/ S r cFiles/CI S ´.Before ru nn ing the mappi ng, make a noteof the cou n t, max( create _ dt), max(updt _ dt)of the table µ CDW _ USR_ CR ITER IA¶ .



How to ru n the mappi ng?

There are 2 ways to ru n the mappi ng:� On e way is to dire c tly ru n the workflow

³wf _O DS_ US ER_ CR ITER IA´ in the ONC OR_O DS

folder.The other way is to ru n the s c ript³exe c_ usr _ unexe c_ info.sh´ from UNIX server s c ripts lo catio n ³/cdwappprd01/pm/CDW/I nfa_S hared/ Sc ripts´.This s c ript will be i n ter nally calling the workflow.



After ru nn ing the mappi ng.

� V alidate the pro cess by compari ng thecou n t before a nd after ru n . The cou n t in the . LS T file should mat ch the differe nc e.Make sure that there are no dupli cates i n .LS T file.Commu n icate the results to the c lien t tovalidate the pro cess.



EDI - E lec tron ic Data In ter cha ngeePM - e -P ortfolio Ma nageme n tCIS - Customer Informatio n S ystemERCOT - E lec tric Reliability Cou nc il of TexasTDS P - Tra nsmissio n a nd Distributio n S ervi ceP roviders

CR - Competitive RetailersTexas S ET - Texas S ta ndard E lec tron ic Tra nsa c tion

Ac ronyms Used..



Any Questio ns..?



Tha nk You..

Documents

CDW- Customer Data Warehouse_Tech