Data migration plan - UNOFFICIAL - EDIEL Migration... · 2019-05-10 · Data migration plan - UNOFFICIAL 27.9.2018 Updated 10.5.2019 4 (34) Fingrid Datahub Oy Katuosoite Postiosoite

Data migration plan - UNOFFICIAL

27.9.2018


27.9.2018

Updated

10.5.2019 2 (34)

Fingrid Datahub Oy

Katuosoite Postiosoite Puhelin Faksi Y-tunnus 2745543-5, ALV rek.

Läkkisepäntie 21 PL 530 [email protected]

00620 HELSINKI 00101 HELSINKI 030 395 5000 030 395 5196 www.fingrid.fi

Table of Contents

1 Executive Summary .............................................................................................................................. 5

2 Parties and responsibilities .................................................................................................................. 7

2.1 Market parties’ responsibilities for delivery of source data .............................................................. 7

2.1.1 Master data delivery responsibility and rules concerning data selection .................................... 7

2.1.2 Metering data delivery responsibility and rules concerning data selection ............................... 10

2.2 Responsibilities of Fingrid Datahub Oy ......................................................................................... 13

2.3 Responsibilities of the data migration service provider ................................................................. 13

2.4 Responsibilities of the datahub system supplier ........................................................................... 14

2.5 RACI matrix ................................................................................................................................... 14

3 Data migration project ........................................................................................................................ 15

3.1 General description and schedule of the data migration project ................................................... 16

3.2 Requirements for source data quality ............................................................................................ 16

3.3 Requirements for the duration of migration file production and delivery ....................................... 19

3.4 Targets of the project stages and acceptance criteria for checkpoints ......................................... 20

4 Instructions for data migration work ................................................................................................. 23

4.1 Titta service ................................................................................................................................... 23

4.2 Description of data migration process ........................................................................................... 23

4.3 Market party tasks during the data migration process ................................................................... 24

4.3.1 Production of data migration files .............................................................................................. 25

4.3.2 Import and checking of master data .......................................................................................... 26

4.3.2.1 File check ............................................................................................................................... 26

4.3.2.2 Integrity check ........................................................................................................................ 27

4.3.2.3 Source data consistency check ............................................................................................. 27

4.3.3 Processing of observations produced in the Titta service ......................................................... 28

4.3.4 Import and checking of metering data ....................................................................................... 28

4.3.5 The transfer of data to the datahub system ............................................................................... 28

4.3.6 Data migration check ................................................................................................................. 30

4.4 Data check against external data sources .................................................................................... 30

4.5 The introduction of new IDs ........................................................................................................... 31

5 Data security for the data migration service .................................................................................... 31


27.9.2018

Updated

10.5.2019 3 (34)

Fingrid Datahub Oy




5.1 General information about data security for server rooms ............................................................ 31

5.2 Data ownership .............................................................................................................................. 31

5.3 Maintenance of data security ......................................................................................................... 32

6 Data protection for the data migration service ................................................................................ 32

7 Definitions ............................................................................................................................................ 32

8 Appendices .......................................................................................................................................... 33


27.9.2018

Updated

10.5.2019 4 (34)

Fingrid Datahub Oy




Change history

Date Version Change

27.9.2018 1.0 First official version

17.12.2018 1.1 The time for metering data migration file production and delivery to Titta has been revised. The word “preliminary” has been removed in Table 6.

22.2.2019 1.2 The rules for selecting customer information data are specified in table 2

30.4.2019 1.3

Applied validation checks for data quality in every stage added to Table 5

UNOFFICIAL text added to the title

Few reference errors fixed in the document

10.5.2019 1.4

Responsibilities for gird operators has been specified in table 2 and in section 2.1.2

Market party definitions has been specified in table 1


27.9.2018

Updated

10.5.2019 5 (34)

Fingrid Datahub Oy




1 Executive Summary

Datahub data migration means quality assurance and initial download of master data and

metering data of electricity retail market business processes from source systems to the

new datahub system before its commissioning.

The source systems are primarily the business applications of distribution system

operators and electricity suppliers, in which customer, accounting point, contract and

metering data are kept. In addition, data is also required from the systems of main grid

and regional grid operators.

The data migration work has two main goals:

Harmonisation of the data delivered from the source systems

Implementing the initial load for the datahub system

The datahub system operations require consistency of the data used in the system.

Fingrid Datahub Oy offers electricity retail market parties and service providers a data

migration service (Titta service), which checks data consistency and via which the data

will be transferred to the datahub system. The market parties can implement the Titta

service by signing a service agreement with Fingrid Datahub Oy.

The datahub data migration project is divided into five stages, and the following

preliminary schedules and objectives have been specified for the first three stages. The

schedules will be updated if necessary once the new Electricity Market Act takes effect:

Stage 1: 01/2019 - 05/2019

Objectives:

The market parties can deliver all up-to-date master data in the format

specified in the migration file instructions.

o All master data has been delivered to Titta in the migration files.

Metering data is not master data.

The quality of master data that is critical to datahub is at a sufficient level for

data integrity checks and checking the accuracy of customer information and

addresses.


27.9.2018

Updated

10.5.2019 6 (34)

Fingrid Datahub Oy




Stage 2: 6/2019 - 10/2019

Objectives:

The quality of data critical to market functionality is at a sufficient level for

consistency checks that cover the entire market.

Mass checks of the accuracy of customer information and addresses can be

performed against external registers.

Stage 3: 11/2019 - 05/2020

Objectives:

The use of new GS1 and EIC IDs will begin in the data migration.

o The GLN party IDs and GSRN accounting point IDs being introduced

are compliant with the GS1 system.

The quality of data transferred in the data migration is sufficient to allow a

transition to preparations for datahub commissioning.

The data migration performance meets the requirements for datahub

commissioning.

After stage 3, the data migration work will progress to support preparations for

commissioning of the datahub system and the actual commissioning and the quality of

data will continue to be maintained and improved.

The data migration plan helps the electricity retail market parties and their system

suppliers to prepare for commissioning of the datahub system, and supports the parties

during possible system changes and improvement of the quality of data being delivered

from the source systems. The plan has been compiled in cooperation with the Data

migration work group, which consists of representatives from companies in the industry.

The Electricity Market Act that becomes valid in 2019 will oblige distribution system

operators and electricity suppliers to participate in the datahub data migration project and

implement the migration of their company’s data to datahub in accordance with this plan.

In addition, regional grid operators and the transmission system operator are obliged to

transfer data related to exchange point metering to datahub in accordance with this plan.

The parties participating in the data migration project and their responsibilities are

described in chapter 2. The stages of the datahub data migration project and the

acceptance criteria associated with them are described in chapter 3. Chapter 4 includes

instructions for the practical data migration work.


27.9.2018

Updated

10.5.2019 7 (34)

Fingrid Datahub Oy




2 Parties and responsibilities

2.1 Market parties’ responsibilities for delivery of source data

In conjunction with the datahub data migration, Market party refers to all companies that

have singed a data migration service agreement with Fingrid Datahub Oy, including

companies that do not have an electricity retail market ID. Signature of the agreement is

a prerequisite for use of the data migration system. The market party can operate in one

or more of the roles described in Table 1.

TABLE 1 MARKET PARTIES

Party Definition

Distribution system operator Distribution system operators are companies that practise electricity grid operations in a 0.4 - 70 kV distribution grid and have an electricity grid permit granted by the Finnish Energy Authority.

Closed distribution system operator

Closed distribution system operators are companies that practise electricity grid operations in a closed distribution system and have an electricity grid permit granted by the Finnish Energy Authority. A closed distribution system operator is subject to the same rules concerning the responsibility to deliver data as a distribution system operator.

Electricity supplier company Electricity supplier companies are companies registered in Finland as electricity suppliers and which have an agreement with a balance-responsible party or serve as a balance-responsible party.

Transmission system operator

The transmission system operator with system responsibility (Fingrid Oyj).

Regional grid operator

Regional grid operators, also known as high voltage distribution grid operators, are companies that practise electricity grid operations in high voltage distribution grids and have an electricity grid permit granted by the Finnish Energy Authority.

Industrial grid Grid operator that is not balance responsible and grid operator has agreed with DSO and eSett that the grid area that grid area is named as a separate MGA.

Production grid Over 1 MVA production plants will be handled as a production grids

Service provider

Service providers that need to use the data in datahub within the scope of authorisations can deliver authorisation information en masse in conjunction with data migration.

A market party can also be a service provider that delivers source data to the Titta service on behalf of another party.

2.1.1 Master data delivery responsibility and rules concerning data selection

The market parties deliver to datahub the data for which they have maintenance

responsibility. Data for which maintenance responsibility lies with another market party

can also be stored in the parties’ information systems. With regard to accounting point


27.9.2018

Updated

10.5.2019 8 (34)

Fingrid Datahub Oy




data, for example, each distribution system operator is responsible for maintaining the

accounting point data in its own region, but accounting point data can also be contained

in the electricity suppliers’ systems. In this case, only the distribution system operator

delivers the data to datahub during the data migration work. The table below shows

which market party delivers the data, and the period of time covered by historical data

that is delivered in data migration.

TABLE 2 DATA DELIVERY RESPONSIBILITY AND RULES CONCERNING DATA SELECTION

Migration file Data owner (delivers data to datahub)

Rules Data to be delivered

Customer information

Distribution system operator (grid agreement)

Electricity supplier (sales agreement)

If the information that different market parties have about the same customer differ, only the customer information of the seller with the most recent (valid latest) contract with the customer will be delivered to the datahub.

Customers with valid agreements. Customers with agreements that have ended within the past six weeks. Customers with future agreements (See Agreement information)

Accounting point data

Distribution system operator

If a DSO’s system contains accounting points outside its metering grid area, the DSO only supplies the accounting points for its own area. Only main metering points are transferred.

Accounting points in use are imported. (Under construction, connected or disconnected)

Accounting points removed from use are imported if an agreement specified for transfer is associated with them (see Agreement information).

Agreement information.

Distribution system operator (grid agreement)

Electricity supplier (sales agreement)

-

Valid agreements and agreements that have ended within the past six weeks and confirmed agreements that will begin within the next 90 days.

Party information

All

Market parties deliver their own information.

Regional, industrial and production grid operators as well as the transmission system operator deliver their own party information, if the metering grid area has at least one exchange point with a DSO or closed DSO where the party is a metering responsible for this exchange point

Only valid information.


27.9.2018

Updated

10.5.2019 9 (34)

Fingrid Datahub Oy






Product information

Distribution system operator (grid product)

Electricity supplier (sales product)

- Valid products and prices connected to transferable agreements.

Authorisation information

Service provider (third party)

Authorisations are based on third-party rights to process data for a specified customer and accounting point.

The service provider specifies the valid authorisation information needed for the migration file.

Metering grid area data

All grid operators

The DSO delivers metering grid area information for the distribution grid.

The TSO delivers metering grid area information for the main grid.

The regional grid operator delivers metering grid area information for the regional grid.

Industrial and production grid operators deliver metering grid area information for the industrial and production grids.

Only valid data.


27.9.2018

Updated

10.5.2019 10 (34)

Fingrid Datahub Oy






Exchange point data

All grid operators

Only the metering responsible party delivers data for an exchange point between two distribution grids.

The TSO delivers exchange point data for the main grid and distribution grid if the TSO is the metering responsible party for the exchange point.

The regional grid operator delivers exchange point data for the regional grid and distribution grid if the regional grid operator is the metering responsible party for the exchange point.

The industrial and production grid operators deliver exchange point data for the industrial and production grids if the grid operator is the metering responsible party for the exchange point.

Only valid data.

Production unit data


- Only valid data.

Additional accounting point addresses


Additional addresses can be reported if necessary for those accounting points that are imported in the Accounting point data.

See Accounting point data.

Accounting point supplier data


The valid supplier data is reported for accounting points that are in Accounting point data

Valid agreements and agreements that have ended in the past six weeks. Confirmed agreements that will begin within the next 90 days.

2.1.2 Metering data delivery responsibility and rules concerning data selection

Delivery responsibility for metering data:

The DSO delivers metering data for accounting points, production units and

exchange points in its own metering grid areas.


27.9.2018

Updated

10.5.2019 11 (34)

Fingrid Datahub Oy




Only the metering responsible party delivers metering data for an exchange point

between two distribution grids.

The regional grid operator delivers metering data for an exchange point in

metering grid areas between a regional grid and a distribution grid when it is the

metering responsible party for a regional grid and distribution grid exchange point.

The industrial grid operator delivers metering data for an exchange point in

metering grid areas between a industrial grid and a distribution grid when it is the

metering responsible party for a industrail grid and distribution grid exchange

point.

The production grid operator delivers metering data for an exchange point in

metering grid areas between a production grid and a distribution grid when it is the

metering responsible party for a production grid and distribution grid exchange

point.

The TSO delivers metering data for an exchange point between the main grid and

a distribution grid when it is the metering responsible party for an exchange point

in metering grid areas of the main grid and distribution grid.

The figure below presents the rules concerning the transfer of metering data history. The

red vertical line describes the time prior to which data is transferred to datahub. During

the data migration project, data is uploaded several times, so a red vertical line may also

depict the moment when data is extracted from the system for data checking.


27.9.2018

Updated

10.5.2019 12 (34)

Fingrid Datahub Oy




FIGURE 1. RULES RELATED TO THE TRANSFER OF METERING DATA HISTORY DATA

The items presented in figure 1 refer to the following cases:

1. The accounting point grid agreement has been valid for a shorter period than six

years, so metering data is imported from the whole agreement period.

2. The accounting point grid agreement has been valid throughout the six-year period.

Metering data is transferred for the entire six-year period. Metering data for a longer

period is not imported to datahub.

3. If an accounting point grid agreement has expired no more than six weeks before the

data was extracted from the source system, the metering data from these accounting

points is also taken. Metering data is collected for the validity period of the agreement

after it ends, but only for a maximum of six years from the time of picking.

4. Only metering data within the scope of hourly time series metering is taken into

consideration in metering data. If the metering method has changed during the

agreement validity period, only the hourly time series metered data is included in the

metering data.


27.9.2018

Updated

10.5.2019 13 (34)

Fingrid Datahub Oy




5. If accounting point consumption is metered in some other way than hourly time series,

use of these accounting points must be balanced via balancing calculation and then

switched to use of hourly time series or hourly time series modelling. This must take

place in conjunction with datahub commissioning at the latest.

Metering data for production units and exchange points is imported for a six-year period

or for the operating period of the metering point if the metering point has been in use for

less than six years.

The data migration process for metering data will be clarified as planning of

commissioning progresses.

2.2 Responsibilities of Fingrid Datahub Oy

Fingrid Datahub Oy’s responsibilities and obligations for the entire datahub project and

system are specified in the Electricity Market Act and decrees.

Fingrid Datahub Oy is responsible for the progress and schedule of the data migration

project. Monitoring of progress will be carried out by monitoring the amounts of data

delivered to the Titta system by market parties and the quality of material and comparing

these observations to the stages presented in chapter 3. As part of monitoring, surveys

can be sent to the market parties, which can be used to assess the market party situation

in more detail.

Fingrid Datahub Oy is responsible for coordinating, scheduling and communicating about

the simultaneous deliveries of source datas for the entire industry.

Fingrid Datahub Oy is responsible for the quality, development and maintenance of the

Titta service.

Fingrid Datahub Oy is responsible for communication and instructions related to the data

migration project. The instructions are centralised in the Titta service and the Ediel.fi

portal. Communication will be carried out as needed using different channels.

Fingrid Datahub Oy is actively monitoring the industry’s preparations for changes brought

by datahub as part of the data migration project. Monitoring is carried out using a range of

surveys targeting the industry Monitoring the preparation of the industry is vitally

important in order to carry out the planning and commissioning of the whole datahub

project in an efficient way.

2.3 Responsibilities of the data migration service provider

The data migration service provider (Solteq Oyj) is responsible for the technical

development and maintenance of the Titta service. The service provider is also

responsible for technical user support provided to the market parties.


27.9.2018

Updated

10.5.2019 14 (34)

Fingrid Datahub Oy




2.4 Responsibilities of the datahub system supplier

The datahub system supplier (CGI) is responsible for compliance of the datahub system

with the Titta service and for uploading data to the datahub system and producing the

data migration reports.

2.5 RACI matrix

The tasks and responsibilities of data migration work are described in the accompanying

RACI matrix (Responsibility assignment matrix). The roles are divided as follows in the

matrix:

R = responsible

A = accountable

C = consulted

I = informed

TABLE 3 DATA MIGRATION TASKS AND RESPONSIBILITIES

Task Market party

Data migration service supplier

(Solteq Oyj)

datahub system supplier

(CGI)

Fingrid

Datahub Oy

Communications

Communications and contact directed at data migration project market parties

C/I C C/I A/R

Communications directed at data migration project system suppliers

C/I C A/R

Communications and reporting directed at the data migration project pilot group

C/I R C/I A/R

Communications for other external stakeholders in the data migration project

C C A/R

Training and technical support

Instructions for using the Titta service

C/I R A

Training for market parties and system suppliers

C/I C/I A/R

Data migration project technical support

I R A

Titta service


27.9.2018

Updated

10.5.2019 15 (34)

Fingrid Datahub Oy




Task Market party

Data migration service supplier

(Solteq Oyj)

datahub system supplier

(CGI)

Fingrid

Datahub Oy

The commissioning of source data checking rules based on the data standard

C/I R

A

Maintenance and management of the Titta service

R A

Establishment of master data in the Titta service

R A

Maintenance of user information in the Titta service

A/R C I

Source data checking and correction

Compiling migration file examples C I A/R

Picking tools for production of migration files

A/R C

The production of migration files based on specifications

A/R

Uploading of migration files to the Titta service

A/R

The checking of uploaded data R A

Production of a company-specific check and correction report

I R A

Overall data migration project reporting

R I A

Data correction and enrichment in the market parties' own systems

A/R C C

Necessary changes to market parties’ own data systems

A/R

datahub system upload tool A/R

Data migration from the Titta system to datahub

R A/R

3 Data migration project

The datahub data migration will be carried out as a sub-project of the datahub system

project. The aim of the data migration project is to harmonise the data supplied from

source systems, ensure data quality, and initial load of the datahub system. The datahub

data standard, in other words, the specified data model and external data sources, will be

utilised when harmonising data and ensuring quality.

The functionality of the data migration process and performance of the Titta service have

been verified in a separate data migration pilot project. The data migration processes


27.9.2018

Updated

10.5.2019 16 (34)

Fingrid Datahub Oy




were tested in the pilot project using proper source data from pilot companies and the

instructions were improved on the basis of the experiences gained. The pilot project was

completed in March 2018.

3.1 General description and schedule of the data migration project

In the data migration project, the market parties pick the data to be transferred to datahub

from their own systems and deliver it to Titta in a format that complies with the migration

file specification. The data quality must meet the quality objectives specified in the check

point for each data migration stage. The criteria for data quality and delivery are tightened

from one stage to another, in order to reach the final target. The quality requirements are

described in chapter 3.2 and the delivery requirements in 3.3. In order to facilitate

achievement of the targets in each stage, Fingrid Datahub Oy has compiled role-specific

task lists that are presented in appendices 7–10. Actions concerning preparation and

actual commissioning are specified in the commissioning plan.

The datahub data migration plan is divided into five stages, and the preliminary schedules

and targets specified for the first three stages are presented in figure 2. If desired, a

market party can also progress faster than the schedule presented in figure 2, taking into

account the fact that certain actions that apply to the entire industry must be performed

simultaneously in a centralised manner throughout the industry. These actions include

consistency checks, mass data checks against external registers, and introduction of

GS1 IDs. The schedules can be updated if necessary when the law takes effect.

FIGURE 2 DATA MIGRATION PROJECT PHASING AND SCHEDULE

3.2 Requirements for source data quality

The data supplied by different parties is checked in the Titta service against rules

specified in advance. Quality targets for the source data have been separately set for

each project checkpoint.

In the data migration project, the parties must ensure that data is corrected in the source

system and that it acceptably passes the check.


27.9.2018

Updated

10.5.2019 17 (34)

Fingrid Datahub Oy




The data in the source data is divided into four quality requirement levels according to

table 4.

TABLE 4 SOURCE DATA QUALITY REQUIREMENT LEVELS

Level Description Explanation

1 Essential for datahub

Data necessary in terms of the data standard’s data model is given in a format that complies with the data standard.

Incomplete data causes rejection of the entire record.

2 Essential for electricity market operations

Data necessary in terms of the datahub business rules, such as conditionally mandatory data that is not mandatory from the data model perspective.

Data in the wrong format is rejected.

3 Useful data, but not essential for market and datahub operations

Useful data from the perspective of electricity market functionality.

Data in the wrong format is rejected.

4 Possible error Data deviates from the information in the external data source but may not necessarily be wrong.

The quality criteria for the checkpoint in each data migration stage are presented in table

5. The checkpoints are consistent with the data migration stages, and progress to the

next stage only takes place when all market parties associated with datahub have

achieved the quality criteria.

The quality criteria presented in table 5 apply to large changing materials, which are:

Customer information

Agreement information.

Product information

Accounting point data

Accounting point supplier information

Metering data (starting from checkpoint 2)

Additional accounting point addresses

Authorisation information

The quality of the following migration files must be 100% at all levels starting from

checkpoint T1:


27.9.2018

Updated

10.5.2019 18 (34)

Fingrid Datahub Oy




Party information


Exchange point data

Production units, depending on the number of units


27.9.2018

Updated

10.5.2019 19 (34)

Fingrid Datahub Oy




TABLE 5 PARTY-SPECIFIC QUALITY CRITERIA FOR SOURCE DATA BY CHECKPOINT

Level Description T1

5/2019 T2

10/2019 T3

5/2020 T4

T5

Party, metering grid area and exchange point as well as production units, depending on the number of units

100% 100% 100% 100% 100%

1 Essential for datahub 95.00% 98.00% 99.50% 99.90% 99.98%

2 Essential for electricity market operations

80.00% 95.00% 98.00% 99.00% 99.50%

3 Useful data, but not essential for market and datahub operations

No requirement

No requirement

No requirement

No requirement

No requirement

4 Possible error No requirement

No requirement

No requirement

No requirement

No requirement

Applied validation checks for data quality

File Check Integrity Check

Consis-tency Check

Consis-tency Check

Consis-tency Check

3.3 Requirements for the duration of migration file production and delivery

Migration file production means all the necessary actions related to picking the source

data from the source system and writing in a format that complies with the migration file

instructions.

In terms of master data, migration file delivery means that all the migration files have

been transferred to Titta and the user has pressed the “Lataa kaikki” button. In terms of

metering data, migration file delivery means that all the migration files have been

transferred to Titta’s SFTP server.

The duration of source data production is measured from the beginning, but the related

requirements are only added to the acceptance criteria as the project progresses. The

requirements are market party-specific.


27.9.2018

Updated

10.5.2019 20 (34)

Fingrid Datahub Oy




TABLE 6 REQUIREMENTS FOR PRODUCTION OF SOURCE DATA AND DURATION OF

DELIVERY

Material T1

5/2019

T2

10/2019

T3

5/2020

T4

T5

Master data migration file production and delivery to Titta.

Measured 4h 4h 4h 4h

Metering data migration file production and delivery to Titta.

No requirement

Measured 120h 120h 120h

3.4 Targets of the project stages and acceptance criteria for checkpoints

Table 7 lists the key targets for each data migration stage and the acceptance criteria for

the related checkpoints.

TABLE 7 TARGETS AND ACCEPTANCE CRITERIA FOR STAGES 1–3 OF THE DATA

MIGRATION PROJECT

Targets Acceptance criteria for checkpoints

Checkpoint T0

The Titta service is ready to start the data migration project

Instructions concerning data migration are sufficient and up-to-date

Data security and data protection for the Titta service has been verified

The legislation concerning datahub has taken effect

Stage 1, 01/2019 - 05/2019

All market parties have signed the data migration service agreement.

The market parties have implemented the master data extraction tools as a system function.

The market parties deliver up-to-date master data in its entirety from their source systems.

The market parties have routines for maintaining data quality.

The market parties have a plan for collecting missing personal identification numbers.

Checkpoint T1

All the fields of the master data files uploaded by the market parties to the Titta service are in a format that complies with the data standard and the migration file specification.

The scope and quality of the source data meets the criteria set for the stage in chapters.

Each market party has measured the duration of all master data picking.

All distribution system operators have provided Fingrid with a complete list of exchange points and adjacent grids.


27.9.2018

Updated

10.5.2019 21 (34)

Fingrid Datahub Oy





Stage 2, 6/2019 - 10/2019

All distribution system operators have implemented the metering data extraction tools.

All market parties have delivered data for the coordinated mass check of customer information and addresses.

The mass check of customer information and addresses has been performed and the continuous checking service is in use.

The market parties have performed the party-specific integrity checks on all the material.

o The parties perform a sufficient number of integrity checks until the data quality is at an acceptable level in accordance with the description in chapter 3.2.

The market parties have ordered new party IDs (GLN), metering grid area IDs (EIC) and metering point IDs (GSRN).

o Only distribution system operators order EIC and GSRN IDs

Checkpoint T2

The data has been checked against external data sources maintained by third parties.

All the parties have performed integrity checks.

The scope and quality of the source data meets the criteria set for the stage in chapters.

Master data extraction performance meets the criteria set for the stage.

Stage 311/2019 - 5/2020

The market parties have introduced the new IDs in migration files.

The GSRN ID service has been taken into use. The service maintains the GSRN IDs until commissioning of datahub.

A consistency check covering the entire industry has been performed.

o A sufficient number of consistency checks are performed until data quality and performance are at an acceptable level in accordance with the description in chapters 3.2 and 3.3.

Data migration material covering the entire industry has been uploaded to the datahub system and the data migration process has been verified on the basis of reports produced by datahub.

The data migration process is ready to move to the datahub commissioning preparation stage.

Checkpoint T3

Metering point IDs have been changed to GSRN IDs.

Metering grid area IDs have been changed to EIC IDs.

The parties have begun using GLN party IDs.

Consistency checks covering the entire industry have been performed.

The quality of the source data meets the criteria set for the stage.

The master data and metering data extraction performance meets the criteria set for the stage in chapters 3.2 and 3.3.

Performance of datahub uploading and data migration verification meets the requirements for datahub commissioning.


27.9.2018

Updated

10.5.2019 22 (34)

Fingrid Datahub Oy




The preliminary targets and acceptance criteria for datahub commissioning preparations

and the actual commissioning are described in table 8. The targets and acceptance

criteria will be clarified as the commissioning plan progresses.

TABLE 8 TARGETS AND ACCEPTANCE CRITERIA FOR COMMISSIONING PREPARATIONS AND

FOR COMMISSIONING WITH REGARD TO DATA MIGRATION


Preparations for commissioning

The quality of data and data migration process performance is at a sufficient level for commissioning of datahub in accordance with the description in chapters 3.2 and Error! Reference source not found.3.3.

The commissioning exercises have been completed, and possible observations made during the exercises have been corrected and the corrections verified.

The data migration process is ready to move to the datahub commissioning stage.

Checkpoint T4

The material imported via data migration has been verified in the datahub business processes.

The source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Other criteria are specified in a separate commissioning plan.

Commissioning

The quality of data is at a sufficient level for commissioning of datahub in accordance with the description in chapters 3.2 and 3.3.

Checkpoint T5

The source data meets the criteria set for the stage in chapters 3.2 and 3.3.

Other criteria are specified in a separate commissioning plan.


27.9.2018

Updated

10.5.2019 23 (34)

Fingrid Datahub Oy




4 Instructions for data migration work

This chapter presents an upper-level description of the tasks to be performed in data

migration. The chapter presents the principles according to which the market parties

deliver data to the Titta service and describes how the service processes the data. More

detailed descriptions of the issues in the chapter are available in the appendices to this

document.

4.1 Titta service

The data migration service – “Titta” – is the most important tool in data migration work.

The Titta service is separate from datahub, so that the market parties can independently

check the data that they have transferred to datahub during the data migration project.

The parties can repeat the upload-check-upload process until the quality of the data is at

the required level.

Among others, Titta offers the following functions for market parties:

Delivery of master data via an interface

Delivery of metering data via an SFTP connection

Checking of master data against the data standard

Integrity check of metering data structural information

Checking of customer information and addresses against external data sources

Reports about the quality of the delivered source data

Observation reports for the checks

Market party users will be provided with user support directly from the Titta service. More

detailed instructions regarding use of the Titta service and the technical description are

available in the User instructions for Titta and the Data migration service description

appendix to the service agreement.

4.2 Description of data migration process

Figure 3 presents a description of the data migration process. Data migration work is by

nature a repetitive process, and the number of repetitions depends on matters such as

source data quality, source system compliance with the data standard, the quality of

source data extraction tools, and the market party’s ability to maintain quality during the

data migration.


27.9.2018

Updated

10.5.2019 24 (34)

Fingrid Datahub Oy




FIGURE 3 DATA MIGRATION PROCESS

4.3 Market party tasks during the data migration process

In each stage of the data migration project, the market parties repeat the process stages

until the data quality meets the acceptance criteria for the checkpoint.

Production of data migration files. Each market party produces the data

specified by the data standard from its own data systems for the migration files.

The structure and data content of the migration files are described in greater detail

in the document Siirtotiedosto-ohjeistus (Appendix 1).

Importing migration files to Titta. The market parties upload the master data to

the Titta service via an interface. Metering data is transferred to Titta via a

separate SFTP connection.

Data checking. Titta checks that the data delivered by the market parties

complies with the data standard. A detailed description of the checks is presented

in the document Data checking in Titta (Appendix 3).

Processing of observations. Titta produces party-specific check reports by

source data. The check reports are in xlsx format and they are saved in the file

archive. The parties can retrieve their own reports from the Titta service interface.


27.9.2018

Updated

10.5.2019 25 (34)

Fingrid Datahub Oy




Data correction and enrichment. Based on the check reports, the market parties

correct the data, enrich data in the source systems or correct the data extraction

tools.

Fingrid Datahub Oy’s work Fingrid Datahub Oy is responsible for transferring the data

to datahub and verifying the data migration.

Data transfer to datahub Data that has passed the check is transferred to the

Titta service publishing area, from where it is available to the datahub system.

Data uploading to datahub. The datahub system retrieves data from the Titta

service and uploads it to its own database.

Data migration verification. Verification of data migration is performed on the

basis of the data migration reports produced by datahub by comparing the reports

to the data transferred to datahub. The market parties can utilise the datahub

reports to ensure that the data uploaded to datahub corresponds to the data in the

source system.

The following chapters describe the work stages on a more detailed level.

4.3.1 Production of data migration files

In order to ensure reliability and performance, the production of migration files must

primarily be implemented as an semi-automated system function and the use of manual

phases is not recommended. The migration files must be produced from up-to-date

material that corresponds to the market situation.

The importance of reliability and performance is emphasised when running consistency

checks, in which case each market party must deliver up-to-date data to Titta in a very

short time so that it is possible to ensure that every party’s data is current. Minimisation of

run time is also of utmost importance in conjunction with datahub commissioning.

However, migration files that contain only a small amount of data, are nearly unchanged

by nature, and can presumably be made completely error-free with little work can be

created manually. These migration files are:

Party information


Exchange point data

Production units, depending on the number of units


27.9.2018

Updated

10.5.2019 26 (34)

Fingrid Datahub Oy




4.3.2 Import and checking of master data

The market party delivers the source data as migration files to the Titta service for

checking. The source data should be extracted from the production system or from a

system containing up-to-date data equivalent to production data. In each data upload, the

party delivers a migration file containing all the rows of master data, not only the changed

rows. However, this does not mean that all the migration files have to be delivered in

every upload. For example, the party can deliver only customer information several times

as part of the data quality improvement work.

Delivery of the material takes place as self-service, in which case each market party

handles their own schedules and ensures that all the material is delivered to Titta for

checking.

If necessary, source material is also delivered in a coordinated and simultaneous manner

by the entire industry. The market party’s obligation is to ensure that the source material

is delivered to Titta at the requested time.

The following checks are performed on the source data in Titta:

File check: The format of the migration file delivered by the party complies with

the data standard.

Integrity check: The source data delivered by the party is internally complete

from the perspective of the data standard and datahub business processes.

Consistency check: The source data delivered by the market party is consistent

with the material supplied by other parties.

The market parties monitor the quality of the uploaded data by means of reports

produced by the Titta service, which can be downloaded from the user interface. The

check report lists all the errors found in the data, including error type and category, data

content of the entire incorrect row, and a possible correction recommendation.

The following sections provide an upper-level description of the data migration checks. A

detailed description is provided in Appendix 3.

4.3.2.1 File check

The file check involves checking an individual migration file to ensure that its format

complies with the specification in Appendix 1. In addition, every data field is checked

against the datahub data standard.

Quality requirement levels are specified for the material imported in the data migration,

and the manner in which errors in different quality requirement data affect importing of the

data is specified in the Titta service. For example, errors in quality requirement level 1


27.9.2018

Updated

10.5.2019 27 (34)

Fingrid Datahub Oy




data cause rejection of the entire row. More detailed specifications related to the check

processes are provided in Appendix 3.

4.3.2.2 Integrity check

After a market party has delivered all the mandatory migration files for its role, an integrity

check can be performed on the data. This involves checking the reference integrities

between the migration files. For example, a customer specified in the agreement must be

found with a given ID from customer information imported to Titta by the market party.

Integrity check also takes into account that a value of a field in one migration file may

depend on a value of another field in another migration file. The comparison only targets

data uploaded by the party itself.

The market party launches the integrity check when the migration files containing the

reference integrities have been delivered to the Titta service.

4.3.2.3 Source data consistency check

Fingrid Datahub Oy initiates a datahub-level consistency check after the parties have run

integrity checks and the quality measured in the file and integrity checks is at a level that

makes it appropriate to run consistency checks.

The following checks are done as part of a consistency check:

Reference integrity checks between parties (for example, the accounting point ID

in an electricity supplier’s agreement must be found in the distribution system

operator’s material).

Unambiguous periods of validity (for example, the same accounting point may not

have agreements simultaneously valid between two different suppliers).

Functional integrity (for example, an accounting point has a sales and a grid

agreement, metering time series are available for the validity of the agreements).

Duplicate checks (for example, the same accounting point may not have several

records).

Only material that complies with the data standard data model is published in datahub.

Rows in which quality requirement level 1 errors have been observed are not transferred

to datahub. Final approval of the source data is done in this stage.

A consistency check is not performed on a party-specific basis, so the progress of its

process is not visible in the party’s user interface.


27.9.2018

Updated

10.5.2019 28 (34)

Fingrid Datahub Oy




4.3.3 Processing of observations produced in the Titta service

The task of the market parties is to check the error reports produced by the Titta service

and, if necessary, correct the material in their own source system as required. The errors

in the reports may also be due to an error in the extraction tool, in other words, the error

may not necessarily be in the source system data.

Performance of file and integrity checks and correction of the material based on the error

reports produced by these checks takes place as self-service. Each market party ensures

that all of its material meets the quality requirements described in chapter 3.2 in each

stage of data migration.

Each market party is responsible for the correctness of its own material.

4.3.4 Import and checking of metering data

The distribution system operators, transmission system operator and regional grid

operators deliver metering data via a separate SFTP server. The Titta service user

launches the metering data check after the metering data and master data have been

imported to Titta.

Among others, Titta carries out the following checks:

The format of the migration file structural data complies with the specification.

The metering point ID in the metering data can be found in the master data.

The active metering points reported in the master data can be found in the

metering data.

The time series for energy leaving the metering grid area and energy entering the

metering grid area have been imported for the exchange points (from the metering

responsible party’s perspective).

4.3.5 The transfer of data to the datahub system

4.4 Once a market party’s data has passed the consistency check and meets the quality requirements specified for data acceptance (see chapter 3.3Requirements for the duration of migration file production and delivery

Migration file production means all the necessary actions related to picking the source

data from the source system and writing in a format that complies with the migration file

instructions.


27.9.2018

Updated

10.5.2019 29 (34)

Fingrid Datahub Oy




In terms of master data, migration file delivery means that all the migration files have

been transferred to Titta and the user has pressed the “Lataa kaikki” button. In terms of

metering data, migration file delivery means that all the migration files have been

transferred to Titta’s SFTP server.

The duration of source data production is measured from the beginning, but the related

requirements are only added to the acceptance criteria as the project progresses. The

requirements are market party-specific.


27.9.2018

Updated

10.5.2019 30 (34)

Fingrid Datahub Oy




TABLE 6 REQUIREMENTS FOR PRODUCTION OF SOURCE DATA AND DURATION OF

DELIVERY

Material T1

5/2019

T2

10/2019

T3

5/2020

T4

T5

Master data migration file production and delivery to Titta.

Measured 4h 4h 4h 4h

Metering data migration file production and delivery to Titta.

No requirement

Measured 120h 120h 120h

), the data is transferred to a separate publishing area. The datahub system uploads the

data from the publishing area to the datahub system database.

4.4.1 Data migration check

After the datahub data upload, the datahub system creates so-called data migration

reports, which are returned to Titta. The data migration reports contain errors in data

uploaded to datahub on a market party-specific basis.

The Titta service publishes the data migration reports for the market parties so that they

can compare the data in the data migration reports with the data in the source systems,

and thus verify that the data in datahub and the source systems is consistent.

Titta also checks that the data uploaded to datahub corresponds to the data transferred to

the datahub system. Based on the check results, the Titta service produces reports about

possible deviations for the administartor users.

With regard to metering data, datahub delivers the following reports by distribution

system operator:

A list of accounting points, production units and exchange points that are missing

all or some metering data during the period of validity of the grid agreement.

Metering time series that datahub was not able to link to a master data structure.

Metering grid area-specific loss series.

Supplier-specific total series by metering grid area.

That is, datahub does not return the uploaded metering data to Titta.

4.5 Data check against external data sources

In the mass data check against an external register carried out in stage 2 of the data

migration project, each market party must ensure that all the material has been delivered


27.9.2018

Updated

10.5.2019 31 (34)

Fingrid Datahub Oy




to the Titta service prior to the check so that it is included in the check. After the mass

check, the system will begin using real-time on-line checking in conjunction with file

checks.

The checks performed and the registers utilised in the checks are described in more

detail in the document Tietojen tarkastus Titassa (Appendix 3).

4.6 The introduction of new IDs

GSRN accounting point IDs and GLN party IDs will be introduced in stage 3 of the data

migration project, and every market party must obtain these IDs itself. The instructions

are available in the EDIEL.fi portal in the document "Use of GS1 IDs in the electricity

retail market" (https://www.ediel.fi/datahub/business-processes/datahub-ohjeet).

In addition, several coordinated checks and other actions that apply to all market parties

will be carried out in the data migration project. Participation will be obligatory.

5 Data security for the data migration service

5.1 General information about data security for server rooms

Data security controls are observed in the server room service provider’s server rooms.

The server rooms meet the requirements of Vahti classification and all the service

provider’s services are Katakri-audited.

The confidentiality, integrity and availability of data have been taken into account when

implementing data security. The server room service provider has a data security policy

approved by the management and a valid ISO2001 certificate that sets requirements for

data security policy, including a deviation management process and risk management.

The data security practices of the server room service provider are based on the

accompanying certification requirements and on ITIL practices:

KATAKRI II auditing was acceptably completed in 2016.

The ISO20000 IT service management system was certified in 2017.

The ISO27000 information security management system was certified in 2017.

5.2 Data ownership

Ownership of the data delivered to the data migration service by the market parties

remains with the market parties. The obligation to secrecy is specified in the service

agreement between Fingrid Datahub Oy and the market parties and in the agreements

between Fingrid Datahub Oy and the subcontractors.

https://www.ediel.fi/sites/default/files/GS1-tunnuksien%20käyttö%20sähkön%20vähittäismarkkinoilla_0.pdf

https://www.ediel.fi/sites/default/files/GS1-tunnuksien%20käyttö%20sähkön%20vähittäismarkkinoilla_0.pdf


27.9.2018

Updated

10.5.2019 32 (34)

Fingrid Datahub Oy




Data will not be transferred outside the borders of Finland during any stage of Fingrid

Datahub Oy’s processing.

5.3 Maintenance of data security

Datahub’s data migration service applications were data security-audited prior to the start

of the data migration pilot stage and again after the commissioning project.

The following general principles apply to data security auditing:

All medium-serious and serious observations must be corrected in order to allow approval of the service’s data security and so that the service can be taken into production use.

All new external and inter-application interfaces must be data security-audited before production use.

If necessary, the service will be audited again if significant changes or a large number of

functional changes are made to the service.

6 Data protection for the data migration service

In terms of data protection, Fingrid Datahub Oy acts as a personal data processor within

the scope of the data migration service. In other words, Fingrid Datahub Oy processes

personal data on behalf of the market parties. When the data is transferred to the

datahub system, a personal data register is formed from the data, and Fingrid Datahub

Oy is the registrar.

Data protection for the data migration service is described in Appendix 4.

7 Definitions

Terms and abbreviations Description

Datahub A centralised information exchange solution for the electricity retail market.

Data standard A specification that describes the datahub business data and dependencies between the data.

Integrity check

A check carried out on the source data of one market party of the Titta service. The system ensures that the source data supplied by the market party is complete with regard to the data model in the data standard. The integrity check is a different concept than the consistency check, which examines the consistency of the source data at the market level.

Fingrid Datahub Oy A subsidiary established for the operational activities of datahub and wholly owned by Fingrid.


27.9.2018

Updated

10.5.2019 33 (34)

Fingrid Datahub Oy




Business process A group of tasks linked to each other, which are done in order to achieve a set target, for example, the switching of a customer's supplier.

Source data Data delivered to the Titta service by the market parties.

Master data Master data are all the data delivered to Titta in migration files.

Migration file A file used in data migration that is used to export data from the source system to datahub in the specified format.

File check Checking performed on an individual file of the data migration service. This includes a syntax check, logical checks of data content and duplicate tests.

Market party

A company that has signed a data migration service agreement with Fingrid Datahub Oy. In conjunction with datahub data migration, Market party is also used to refer to companies that use the Titta service but do not have a retail electricity market party ID.

Data migration service

A system service in which a check of data coming from market party source systems is carried out, and about which possible deviations are reported back to the market parties. The data migration service gives summary data to Fingrid about the progress of the migration process and what stage each market party is at. The name of the data migration service is “Titta”.

Data migration work group

A group of 11 pilot companies (pilot group) selected by Finnish Energy that, together with Fingrid and the data migration plan partner, participates in specification of the documentation and data migration process associated with data migration.

Data model A data model is an abstract model, which organises data fields and specifies how they are connected to each other.

Consistency check

A check performed on the entire source data by the Titta service. The system checks that the source data delivered by different market parties is unambiguous and non-contradictory. The consistency check is a different concept that the "integrity check", which is performed on source data delivered by a single market party.

8 Appendices

Appendix 1: Migration file instructions.

https://www.ediel.fi/Datahub/tietokonversio/siirtotiedosto-ohjeistus

Appendix 2: Preparation by the industry.

https://www.ediel.fi/sites/default/files/Ohje_tietokonversiotyohon_valmistautumiseksi.pdf

Appendix 3: Data check in Titta

Appendix 4: Data protection for the data migration service

https://www.ediel.fi/datahub/tietokonversio/tietokonversiopalvelun-tietosuoja

Appendix 5: User instructions for Titta

https://konversio.datahub.fi/Content/Tietokonversiopalvelun_kayttoohje.pdf

https://www.ediel.fi/Datahub/tietokonversio/siirtotiedosto-ohjeistus

https://www.ediel.fi/sites/default/files/Ohje_tietokonversiotyohon_valmistautumiseksi.pdf

https://www.ediel.fi/datahub/tietokonversio/tietokonversiopalvelun-tietosuoja

https://konversio.datahub.fi/Content/Tietokonversiopalvelun_kayttoohje.pdf


27.9.2018

Updated

10.5.2019 34 (34)

Fingrid Datahub Oy




Appendix 6: Description of the data migration service

https://www.ediel.fi/sites/default/files/Liite%201%20Tietokonversiopalvelun%20kuvaus.pdf

Appendix 7: The tasks of the distribution system operators in different stages of the data

migration project

https://www.ediel.fi/sites/default/files/Liite%207%20Jakeluverkonhaltijan%20teht%C3%A4

v%C3%A4listat.pdf

Appendix 8: The tasks of the electricity supply company in different stages of the data

migration project

https://www.ediel.fi/sites/default/files/Liite%208%20Myyntiyhti%C3%B6n%20teht%C3%A

4v%C3%A4listat.pdf

Appendix 9: The tasks of the service providers in different stages of the data migration

project

https://www.ediel.fi/sites/default/files/Liite%209%20Palvelutarjoajan%20teht%C3%A4v%

C3%A4listat.pdf

Appendix 10: The tasks of the transmission system operator and regional grid operators

in different stages of the data migration project

https://www.ediel.fi/sites/default/files/Liite%2010%20Kanta-

%20ja%20Alueverkon%20haltijan%20teht%C3%A4v%C3%A4listat.pdf

https://www.ediel.fi/sites/default/files/Liite%201%20Tietokonversiopalvelun%20kuvaus.pdf

https://www.ediel.fi/sites/default/files/Liite%207%20Jakeluverkonhaltijan%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%207%20Jakeluverkonhaltijan%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%208%20Myyntiyhtiön%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%208%20Myyntiyhtiön%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%209%20Palvelutarjoajan%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%209%20Palvelutarjoajan%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%2010%20Kanta-%20ja%20Alueverkon%20haltijan%20tehtävälistat.pdf

https://www.ediel.fi/sites/default/files/Liite%2010%20Kanta-%20ja%20Alueverkon%20haltijan%20tehtävälistat.pdf

Documents

Data migration plan - UNOFFICIAL - EDIEL Migration... · 2019-05-10 · Data migration plan - UNOFFICIAL 27.9.2018 Updated 10.5.2019 4 (34) Fingrid Datahub Oy Katuosoite Postiosoite