FINAL TECHNICAL IMPLEMENTATION REPORT

Title: Horizontal and vertical integration: implementing technical and

statistical standards in the European Statistical system

Agreement Number: 11152.2015.001-2015.252

Applicant Address: National Statistics Office

Lascaris

Valletta, VLT2000 Malta

Contact Persons: Silvan Zammit

Director, Data Capability

Anne Marie Pace

Head of Unit, Methodology and Research

Ivan Salomone

Head of Unit, IT Systems

Date: 10 November 2017

TABLE OF CONTENTS

INTRODUCTION...................................................................................................................................5

DESCRIPTION OF ACTIVITIES CONDUCTED..................................................................................................5

DEVELOPMENT OF A TAILORED TOOL THAT FACILITATES THE DISSEMINATION OF METADATA IN VARIOUS FORMATS

INCLUDING SDMX..............................................................................................................................5

Analysis of needs, design of solution, and development of tool.......................................................6

Testing of application........................................................................................................................8

Updating of metadata information....................................................................................................9

IMPLEMENTATION OF PROCESS-RELATED METADATA STANDARDS.................................................................10

Identification of domains to be represented in GSBPM and compilation of process metadata......11

Verification of process metadata in GSBPM form...........................................................................12

Creation of an internal central repository for the storage of process metadata.............................13

Enrichment of key information on the ‘Sources and Methods’ web-pages.....................................13

INFORMATION ON THE ELIGIBLE COSTS DECLARED.....................................................................................15

APPENDIX A....................................................................................................................................18

APPENDIX B....................................................................................................................................20

APPENDIX C.....................................................................................................................................23

2

LIST OF FIGURES

Figure 1. Welcome page of the metadata backend application............................................................6

Figure 2. Grid table of existing concepts which can be edited or deleted.............................................8

Figure 3. Web-page for the addition of new statistical concepts..........................................................8

Figure 4. Snapshot of the application (MS Access-based form)...........................................................10

Figure 5. Intranet page featuring stock of GSBPM reports..................................................................13

Figure 6. Grid table of existing Units which can be edited or deleted.................................................23

Figure 7. Grid table of existing statistical concepts which can be edited or deleted...........................24

Figure 8. Grid table of existing statistical domains which can be edited or deleted............................24

Figure 9. Grid table of existing metadata/quality reports which can be edited or deleted.................25

Figure 10. Grid table of existing classification groups which can be edited or deleted.......................25

Figure 11. Grid table of existing statistical classifications which can be edited or deleted..................26

Figure 12. Web-page for the addition of new Units............................................................................26

Figure 13. Web-page for the addition of new statistical concepts......................................................27

Figure 14. Web-page for the addition of new statistical domains.......................................................27

Figure 15. Web-page for the addition of new metadata/quality reports............................................28

Figure 16. Web-page for the addition of new classification groups.....................................................28

Figure 17. Web-page for the addition of new statistical classifications...............................................29

3

ACRONYMS USED

CMS Content Management System

ESMS Euro SDMX Metadata Structure

ESQRS ESS Standard for Quality Reports Structure

GSBPM Generic Statistical Business Process Model

NSO National Statistics Office

RAMON Reference And Management Of Nomenclatures

SDMX Statistical Data and Metadata eXchange

4

1. INTRODUCTION

This report provides a brief description of activities and relative tasks which have been executed

under this grant agreement, together with a list of deliverables. The activities carried out for each of

the two main grant objectives will be described along with a summary of their implementation and

an analysis of results.

The ultimate goal of this project was to create specific metadata for various user types in order to

facilitate the understanding of our statistical products and encourage users to re-use the information

available on the NSO website.

2. DESCRIPTION OF ACTIVITIES CONDUCTED

The action under this grant is divided in two objectives, namely:

Objective 1: Development of a tailored tool that facilitates the dissemination of metadata in

various formats including SDMX; and

Objective 2: Implementation of process-related metadata standards.

The scope of the first objective was the creation of a tool which makes possible the updating of

metadata made available on the NSO’s metadata portal (www.nso.gov.mt /metadata ). The tool is

meant to serve as a unified system through which such metadata can be smoothly updated by staff

within the Methodology and Research Unit; without any dependency on IT people.

The second objective was aimed at harmonising and standardising the structure of process

documents in line with the GSBPM model. As a result, a central repository of process-related

metadata for a number of domains from the Economic, Business, and Social directorates was

developed and made available internally to NSO staff.

3. DEVELOPMENT OF A TAILORED TOOL THAT FACILITATES THE DISSEMINATION OF METADATA IN VARIOUS FORMATS INCLUDING SDMX

This objective was further divided into a number of tasks, which included:

Task 1: Analysis of needs and design of solution

5

Task 2: Development of the tool

Task 4: Testing of application

Task 5: Updating of metadata information

Each of these tasks was carried out successfully, and relevant details on their implementation are

outlined below.

Analysis of needs, design of solution, and development of tool

Following a series of meetings between the IT Systems Unit and the Methodology and Research

Unit, the main requirements of the CMS were listed for its design. These requirements included the

management of domains, metadata or quality reports, statistical concepts, classifications, and code

lists.

The designed solution consisted of a web application, based on ASPX.Net technologies interacting

with a MS SQL Server 2014 database to store the contents of the metadata portal. The CMS is not

web facing but is accessed only from the intranet. The tool can be accessed internally by staff within

the Methodology and Research Unit.

Figure 1. Welcome page of the metadata backend application

The tool allows for the addition, editing, and deletion of statistical domains. Statistical domains refer

to those statistical areas for which an ESMS or ESQRS report is available, and other data sources to

which the statistical concepts on the metadata portal relate. All statistical domains are linked to the

corresponding production unit at the NSO, and to the relevant News Release or Publication as

published on the NSO’s website (www.nso.gov.mt). At this level, domains are managed by applying

the necessary modifications to these links.

The reports’ management functionality is meant to cater for the addition, editing, and deletion of

metadata or quality reports. New ESMS or ESQRS reports can be uploaded in PDF, XML, and XLS

6

formats. For any new report, the type of reporting structure (ESMS vs. ESQRS), name, reference

year, date of upload, and the related statistical domain need to be determined. The link to a

particular domain is essential to show the report name appropriately in the reports’ sub-menu and

linked to the suitable production unit at NSO. The editing aspect involves the necessary

modifications to these attributes.

A number of standard statistical classifications are available on the metadata portal, and are

classified into five categories: business, economic, social, regional, and other. New classification

groups can be added, and any unnecessary groups can be removed accordingly. Furthermore, new

classifications may be included, and the association between a classification and the corresponding

group can be modified appropriately. Classifications which are no longer used or referred to in the

national statistical system can be deleted as necessary.

New statistical concepts and complementary definitions can be added through the backend system.

The listed concepts are related to statistical classifications, code lists, domains, and other concepts.

The tool permits the execution of the necessary corrections in these elements, and the amendment

of any textual information.

Finally, the tool permits the management of code lists which are associated with particular concepts.

Codes can be added to, or removed from the predefined list. Moreover, the text of codes can be

edited and enhanced as deemed appropriate.

Existing elements in the metadata portal are presented in the backend application through the use

of grid tables for any subsequent editing or deletion.

7

Figure 2. Grid table of existing concepts which can be edited or deleted

Separate pages are loaded whenever a new concept, classification, code list, or ESMS/ESQRS report

has to be added (or uploaded).

Figure 3. Web-page for the addition of new statistical concepts

Testing of application

After developing the tool, the system was implemented on a test environment. The product was

first tested by the system developers and then by the staff of the Methodology and Research Unit,

(the main users). No major issues were found, and modifications were limited to issues concerning

8

the better usability of the system. Consequently, the system has been switched to the live

environment for the necessary dynamic updating of reference metadata.

Updating of metadata information

Following the development of the tool, the necessary updates could take place. The list of statistical

concepts and their definitions, standard classifications, and code lists available on the NSO’s

metadata portal was reviewed to outline any necessary updating in reference metadata.

The reasoning behind such updates is explained in terms of new concepts that needed to be added,

better links between different concepts, and revised versions of standard classifications and code

lists.

New statistical concepts were added to the portal after reviewing the exhaustive list of News

Releases and Publications; and newly identified statistical classifications. Wherever applicable, each

statistical concept on the metadata portal is presented along with a list of other related concepts.

These links were further enhanced in the updating procedure, together with the list of related

sources. Revisions in sources are meant to reflect changes in the stock of News Releases and

Publications issued by the NSO.

A number of sources direct users to the ‘Sources and Methods’ web-page on the NSO’s website.

These dedicated web-pages have been likewise standardised and enhanced under a separate

objective of the same grant agreement. Upon comparison of multiple sources, and discussions with

technical units, a number of concept definitions were improved.

Checks were also carried out on the coverage of the list of statistical classifications listed on the

portal. Any classifications which are referred to or used by production units, as per the GSBPM

documentation, and which were not available on the metadata portal were suitably added.

Moreover, any revised versions of statistical classifications were taken into consideration. The

consistency of the presented code lists was also assessed relative to the standardised code lists

presented in RAMON, the SDMX glossary of statistical code lists, and methodological manuals and

handbooks. Subsequently, the necessary modifications were done.

The application used for identifying the necessary updates in reference metadata consists of a MS

Access-based form and a number of queries. The application allows the simultaneous viewing of all

information on each statistical concept, facilitating the confrontation of information from multiple

sources.

9

Figure 4. Snapshot of the application (MS Access-based form)

4. IMPLEMENTATION OF PROCESS-RELATED METADATA STANDARDS

This objective was further divided into a number of tasks, which included:

Task 6: Identification of domains to be represented in GSBPM

Task 8: Compilation and verification of process metadata in GSBPM form

Task 10: Creation of an internal central repository for the storage of process metadata

(revised specifications)

Task 12: Enrichment of key information on the ‘Sources and Methods’ web-pages (revised

specifications)

Tasks 8 and 9 were successfully accomplished, and pertinent details on their implementation are

outlined in the following sections. Tasks 10 and 12, as listed in this report, reflect the modified

proposal which has been suitably approved by Eurostat.

Modifications to the original proposal were necessary for a number of reasons. The original grant

agreement stipulated that process-related documentation had to be made available to external

users of statistics on the NSO’s metadata portal. In essence, this objective comprised the creation of

a repository in which GSBPM reports are stored; and the development of a software tool that allows

production units within NSO to interact with this repository and to export reports in different

formats (including SDMX). The development of this tool would have followed actions for exploring

ways how process metadata could have been represented in SDMX.

10

The necessary detail for compliance with the GSBPM model includes, but is not limited to,

information about the integration of administrative data in the statistical production system.

Currently, the NSO is amending national legislation to facilitate data sharing and the provision of all

administrative data that can be used for statistical purposes. In view of the sensitive nature of

certain administrative data sources accessed by the NSO, and the complexity involved in some

domains for their use, a more sensible approach was adopted whereby process metadata are shared

for internal purposes only. This solution was preferred over a concise version of the process

documentation considering that the latter would have only entailed a reproduction of the

information already contained in the various ESMS and ESQRS reports.

It must be stressed however, that although GSBPM reports have been retained for internal purposes

only, external users are provided with regular updates on metadata, methodological and quality

reports. The information to external users has been further supplemented by enhancing the key

information accessible on the ‘Sources and Methods’ web-pages that are available on the NSO’s

website.

Representing process-related documentation in SDMX would have been beneficial if the NSO had

to disseminate such reports to external users on the metadata portal. As a consequence,

considering that such documentation has been retained internally, prospective tasks on the

representation of process metadata in SDMX have been aborted. Standardised GSBPM reports are

disseminated on the intranet in PDF, adequately suiting the needs of internal users within the NSO.

Identification of domains to be represented in GSBPM and compilation of process metadata

The domains for which process metadata have been prepared were identified after taking a stock of

areas which are reported along the ESMS or the ESQRS structure on the national metadata portal. A

selected number of domains were chosen, based on their importance and relevance to other areas.

This was done through an internal survey which was carried out among all production units and

which was further complemented by a series of meetings.

A total of 33 statistical domains were selected and consequently documented in GSBPM. Following

the submission of the grant proposal, additional domains were added to enrich the portfolio,

including the Census of Population and Housing; and the process for updating the Register of

Population and Dwellings. In addition, other domains were merged or omitted due to major changes

in the respective business processes.

11

Prior to the drafting of process-metadata documentation, the Methodology and Research Unit

compiled a comprehensive set of guidelines and recommendations following a systematic literature

review process. These guidelines served as a guidebook for all technical units involved in the

documentation process, and outlined the level of detail required under each sub-process, and

various examples to ensure full compliance with GSBPM methodology.

The guidelines aimed at identifying where a particular process should be documented. Indeed, a

reporting template delineating GSBPM sub-processes was prepared with the necessary instructions

accompanying each sub-process of the model. These guidelines sought to ensure that no particular

process is reported in different phases/sub-processes, and most importantly that no processes are

left out. The reporting template was laid down as a MS Word document, which was afterwards

filled out by respective domain units.

Subsequently, a series of technical workshops were organised by the Methodology and Research

Unit to familiarise NSO staff with the GSBPM framework. These workshops aimed at explaining the

processes and sub-processes of the GSBPM thoroughly, as well as to clarify any known issues

beforehand. At least one member from each technical unit attended these workshops together with

their respective domain managers. In these sessions, the need for an adequate description of all the

relevant steps in a detailed and technical manner was stressed, particularly for sub-processes

describing operational issues. To facilitate the process, a number of examples of typical scenarios

were discussed; including considerations to be made for specific statistical processes (for example,

those based entirely on administrative data).

Following these workshops, each technical unit was then required to draft the corresponding work

processes subject to the provided guidelines. The documentation process progressed smoothly with

constant feedback provided by the Methodology and Research Unit which coordinated this exercise.

Verification of process metadata in GSBPM form

The verification of GSBPM documentation was split in two stages. The first stage necessitated

continuous interaction between the Methodology and Research Unit and the respective domain unit

responsible for the documentation. This process was quite resource intensive and required a

significant effort by all domains in view of the technicalities involved. Following a rigorous vetting

process, the reports were passed on to the respective Units for the necessary amendments.

Throughout the process, reports were cross-verified to ensure perfect synergy between domains in

terms of data sources, methodology and classifications.

12

Common issues that emerged during the vetting stage included insufficient information provided by

domain units and mis-classification relative to the GSBPM sub-processes.

The verification process was repeated until a comprehensive set of reports was achieved. At each

stage, the Methodology and Reseach Unit provided feedback to the respective domain units on how

to improve the reports.

Creation of an internal central repository for the storage of process metadata

Following the completion of GSBPM documentation, an internal central repository for their storage

was created. In fact, the reports for all 33 domains, including their annexes, can be retrieved

internally by NSO staff through the NSO’s intranet platform. The reports are organised by

production unit and can be downloaded in PDF. The launch of this page has been communicated

with all domain units, and NSO employees are continuously encouraged to refer to such reports as

necessary to facilitate day-to-day operations so that it can serve as a knowledge management tool.

Figure 5. Intranet page featuring stock of GSBPM reports

Enrichment of key information on the ‘Sources and Methods’ web-pages

The domains for which metadata or quality reports are disseminated on the metadata portal, and/or

domains which are referred to in the repository of statistical concepts, have been documented on

the respective ‘Sources and Methods’ page on the NSO’s website (www.nso.gov.mt) accessible

under the ‘Home’ menu.

13

These web-pages were updated with salient information and presented in a standardised way. The

necessary information was presented along six main indicators, namely: Relevance, Methodological

description, Accuracy and reliability, Timeliness and punctuality of data, Accessibility and clarity, and

Coherence and comparability/consistency of data.

Such information provides a concise but detailed description of a particular domain, adequately

satisfying the needs of external users who are mainly interested in getting an idea of what the

domain is about. The ‘Sources and Methods’ are meant to provide more detail than that made

available in the methodological section of News Releases and Publications.

External users who are interested in getting a deeper insight of a particular domain, can then access

the relevant metadata and quality reports made available on the NSO’s metadata portal. Indeed,

direct links to the relevant questionnaires and meta-information are available on the ‘Sources and

Methods’ web-pages.

Standardised information has been prepared for a total of 60 domains. For the main part, the

source of such information was the relevant metadata and quality reports, and/or process-metadata

as documented in the GSBPM reports. Reference to these is also made in all News Releases and

publications issued by the NSO.

In particular, the ‘Sources and Methods’ web-pages are aimed at providing information on users and

their particular needs with respect to the statistical data; and the extent to which all required

statistics that are available.

Most importantly, these dedicated pages outline the main characteristics of a particular data set,

together with the relevant classification systems, length of time for which data are available, and the

unit for which information is sought and for which statistics are ultimately compiled.

An assessment of the accuracy linked to the data, and the length of time between data availability

and the phenomenon they describe are additionally included. Furthermore these web-pages

comprise information on the modalities by which users of statistics can obtain, use, and interpret

data. The extent to which statistics are comparable or reconcilable over time or with those obtained

from other data sources or statistical domains has been additionally referred to.

14

5. INFORMATION ON THE ELIGIBLE COSTS DECLARED

The information below indicates the number of persons and days involved in each objective, by

grade.

Objective 1: Development of a tailored tool that facilitates the dissemination of metadata in

various formats including SDMX

Analysis of needs, design of solution, and development of tool:

Grade Number of persons Number of days

Programmer IV 1 2.5

Programmer III 1 27.75

Statistician 1 3.375

Testing of application:

Grade Number of persons Number of days

Programmer IV 1 1

Programmer III 1 0.4375

Statistician 1 1.375

Updating of metadata information:

Grade Number of persons Number of days

Manager 14 4.75

Statistician 1 16

Total (Objective 1):

Grade Number of persons Number of days

Manager 14 4.75

Programmer IV 1 3.5

Statistician 2 20.75

Programmer III 2 28.1875

15

Objective 2: Implementation of process-related metadata standards

Identification of domains to be represented in GSBPM and compilation of process

metadata:

Grade Number of persons Number of days

Manager 16 99.3125

Principal statistician 6 62.09375

Senior statistician 5 36.96875

Statistician 14 130.75

Verification of process metadata in GSBPM form:

Grade Number of persons Number of days

Statistician 3 233.09375

Creation of an internal central repository for the storage of process metadata:

Grade Number of persons Number of days

Statistician 1 2.5

Enrichment of key information on the ‘Sources and Methods’ web-pages:

Grade Number of persons Number of days

Manager 1 1.875

Statistician 1 12.5625

Total (Objective 2):

Grade Number of persons Number of days

Manager 16 101.1875

Principal statistician 6 62.09375

Senior statistician 5 36.96875

Statistician 14 378.90625

A total of 636.34375 days were involved for the whole project, of which 399.65625 days were

worked by statisticians, followed by 105.9375 days worked by managers. Principal statisticians were

involved over 62.09375 days, whereas senior statisticians worked a total of 36.96875 days. A total of

31.6875 days were worked by IT people.

16

The IT team was not involved in the second objective following modifications to the original grant

proposal, as explained above. The originally proposed tasks were replaced by other equally resource

intensive tasks which shifted implementation requirements from IT people to technical staff.

17

APPENDIX A

List of process-metadata in GSBPM format by sector:

Social:

- Census of Population and Housing 2011

- Population and Dwellings Registers

- Statistics on Income and Living Conditions

- Labour Force Survey

- Job Vacancies Statistics

- Collective Accommodation Establishments

- Inbound and Outbound Tourism

- ICT Usage by Households

- Education Statistics

- Migration

- Demography

Economics:

- Gross Domestic Product

- Government Deficit and Debt

- Research and Development in Malta (Government)

- Balance of Payments

- Foreign Direct Investment

- International Trade Statistics

- Harmonised Index of Consumer Prices

- Retail Price Index

Business:

- Structural Business Statistics

- Inward Foreign Affiliates Statistics

- Outward Foreign Affiliates Statistics

- Short-term Statistics

18

- Producer/Input Agricultural Price Indices

- Agricultural Production

- Aquaculture

- Economic Accounts for Agriculture

- Farm Structural Survey

- Waste Statistics

- Energy Statistics

- Transport Statistics

- Business Registers

- Research and Development (Business Sector)

19

APPENDIX B

List of domains included in the ‘Sources and Methods’ web-pages by sector:

Social:

- Census in Malta

- Lifestyle Survey

- Household Budgetary survey

- Statistics on Income and Living Conditions

- Culture Statistics

- Gender Pay Gap

- Job Vacancy Statistics

- Labour Cost Index

- Labour Cost Survey

- Labour Force Survey

- Structure of Earnings Survey

- Collective Accommodation Establishments

- National Tourism Survey

- TOURSTAT Survey

- Absenteeism in schools

- Adult Education

- Continuing Vocational Training Survey

- Education Statistics

- ICT Usage by Enterprises

- ICT Usage by Individuals and Households

- Demography

- Migration Statistics

Economics:

- Gross Domestic Product

- Regional Gross Domestic Product

- Non-Financial Accounts by Institutional Sector

- Council Directive 85/2011

20

- ESSPROS

- Expenditure of General Government Sector by Function

- General Government Balance and Debt under the Maastricht Treaty

- Government Expenditure on Social Security Benefits

- Government Finance Data

- Quarterly Accounts for General Government

- Research and Development in Malta (Government Sector)

- Structure of General Government Debt

- Tax Revenues

- Balance of Payments

- The International Investment Position of Malta

- External Trade Statistics

- Harmonised Index of Consumer Prices

- Retail Price Index

Business:

- Inward Foreign Affiliates Statistics

- Outward Foreign Affiliates Statistics

- Structural Business Statistics

- Short-term Statistics

- Agri-Environmental Statistics

- Animal Production

- Aquaculture

- Crop Production

- Economic Accounts for Agriculture

- Energy Statistics

- Farm Structure Survey

- Fish Landings

- Producer/Input Agricultural Price Indices

- Motor Vehicles

- Road Traffic Accidents

- Sea Transport between Malta and Gozo

- Waste Statistics

- Business Demography

21

- Business Innovation

- Research and Development in the Business Sector

Other:

- User Satisfaction Survey

22

APPENDIX C

Description of the various features of the metadata backend application:

Existing elements in the metadata portal are presented in the backend application through the use

of grid tables for any subsequent editing or deletion. The figures below show a snapshot of each

grid table related to every attribute.

Figure 6. Grid table of existing Units which can be edited or deleted

23

Figure 7. Grid table of existing statistical concepts which can be edited or deleted

Figure 8. Grid table of existing statistical domains which can be edited or deleted

24

Figure 9. Grid table of existing metadata/quality reports which can be edited or deleted

Figure 10. Grid table of existing classification groups which can be edited or deleted

25

Figure 11. Grid table of existing statistical classifications which can be edited or deleted

Separate pages are loaded whenever a new concept, classification, code list, or ESMS/ESQRS report

has to be added (or uploaded). The figures below show a snapshot of each web-page.

Figure 12. Web-page for the addition of new Units

26

Figure 13. Web-page for the addition of new statistical concepts

Figure 14. Web-page for the addition of new statistical domains

27

Figure 15. Web-page for the addition of new metadata/quality reports

Figure 16. Web-page for the addition of new classification groups

28

Figure 17. Web-page for the addition of new statistical classifications

29