Interactive session: MetadataMaia EnnokHead of Data Warehouse ServiceStatistics Estonia24.05.2012

Schema

Maia Ennok ESSNet Data Warehouse05/24/12

1 Specify Needs

2 Design 3 Build 4 Collect 5 Process 6 Analyse 7 Disseminate

1.3. Establish output objectives

1.4. Identify concepts

1.5. Check data availability 

1.6. Prepare business case

2.1. Design outputs

2.2. Design variable descriptions

2.3. Design data collection methodology

2.4. Design frame and sample methodology

2.5. Design statistical processing methodology

2.6. Design production systems and workflow

3.1. Build data collection instrument

3.2. Build or enhance process components

3.3 Configure workflows

3.4. Test production systems

3.5. Test statistical business process

3.6 Finalize production systems

4.1. Select sample

4.2. Set up collection

4.3. Run collection

4.4. Finalize collection

5.1. Integrate data

5.2. Classify and code

5.3. Review, validate and edit

5.4. Impute

5.5. Derive new variables and statistical units

5.6. Calculate weights

5.7. Calculate aggregates

5.8. Finalize data files

6.1. Prepare draft outputs

6.2. Validate outputs

6.3. Scrutinize and explain

6.4. Apply disclosure control

6.5. Finalize outputs

7.1. Update output systems

7.2. Produce dissemination products

7.3. Manage release of dissemination products

7.4. Promote dissemination products

7.5. Manage user support

                                                                            Access Layer

                                                                           Interpretation and Analysis Layer

                                                                           Integration Layer

                                                                           Source Layer

off SDWH

Extra Layer

Task

Maia Ennok ESSNet Data Warehouse05/24/12

Put metadata subsets to schema (write examples)

• Same groups as previous ineractive session: SBS, STS, SBR, ET

• GSBPM phases 1.-7.

• SDWH layers, Extra Layer with description if we missed a layer, off SDWH

• Metadata subsets in different colors (Statistical, Process, Technical, Quality, Authorisation), Extra metadata subset with description if we miss a subset

• Presentations with metadata subsets, examples and answered fallowing questions

• Questions:

• What is in your opinion the key element of the S-DWH ? VARIABLE vs. DATASET

• What is the absolute minimum set of metadata that must be defined for that element?

• What should be the main function of the S-DWH (process support/driver,  output/dissemination)?

• What is the function of the metadata layer?

Generic Statistical Business Process Model (GSBPM)

Maia Ennok ESSNet Data Warehouse05/24/12

SDWH Layers

Maia Ennok ESSNet Data Warehouse05/24/12

I. source layer, is the level in which we locate all the activities related to storing and managing internal (surveys) or external (archives) raw data sources.

II. integration layer, on this layer performs the typical Extraction, Transformation and Loading functions; which must be realized in automatic or semi-automatic ways

III. interpretation and data analysis layer is specialized to interactive and not structural activities.

IV. access layer is addressed to a wide typology of users or informatics instruments for the final presentation of the information sought

Metadata subsets

Maia Ennok ESSNet Data Warehouse05/24/12

Statistical metadata are data about statistical dataThis definition will obviously cover all kinds of documentation with some reference to any type of statistical data and is applicable to metadata that refer to data stored in a S-DWH as well as any other type of data storeExamples: Variable definition; register description; code list.

Process metadata are metadata that describe the expected or actual outcome of one or more processes using evaluable and operational metricsExamples: Operator’s manual (active, structured, reference); parameter list (active, structured, reference); log file (passive, structured, reference/structural)

Technical metadata are metadata that describe or define the physical storage or location of data. Examples: Server, database, table and column names and/or identifiers; server, directory and file names and/or identifiers

Quality metadata are any kind of metadata that contribute to the description or interpretation of the quality of data. Examples: Quality declarations for a survey or register (passive, free-form, reference); documentation of methods that were used during a survey (passive, free-form, reference); most log lists (passive, structured, reference/structural)

Authorisation metadata are administrative data that are used by programmes, systems or subsystems to manage users’ access to data. Examples: User lists with privileges; cross references between resources and users

Mapping the BPM-Notation on a SDWH layerd architecture

Maia Ennok ESSNet Data Warehouse05/24/12

Schema with metadata subsets

Maia Ennok ESSNet Data Warehouse05/24/12

StatisticalProcessTechnicalQualityAuthorisation

3/28/12 Esitluse või esitleja nimi