ESSnet on Standardisation WP3 evaluation report structure ......ESS standards Corresponding questions from the WP3 assessment template Final clarifying questions for the last telephone

Proposal for the structure of WP3 report on evaluation / justification of case studies

based on the final recommendations of the Sponsorship on Standardisation (before their adoption by the ESSC) What should be kept in mind for all the extra dimensions (last column) during the interviews:

1. always ask the „how could have it been better” question

2. ask the expert to be as specific as he can, all the time, we want real practical experience and feedback and not general ones

3. do not forget to ask him about a general expression on how this whole process could be better (maybe drawing up the concept of the life cycle model and ask for opinion, based on the practical experience he gathered)

Standard setting process and implementation for ESS standards

Corresponding questions from the WP3 assessment template

Final clarifying questions for the last telephone or face-to-face

validation interviews

PHASE ACTIVITY DESCRIPTION

1. Establish the need

Collect initiatives

Identification of the aim of the standard and its ability to satisfy user needs. The motivation behind the development should be clearly indicated: user driven (wanted to find new solution to stakeholder needs) / process driven (involved business process steps requiring significant resources, producing large delays or constituting quality risk) / technology driven (using new technology in the production of statistics)

I. BASIC EVALUATION CRITERIA FOR THE NORMATIVE DOCUMENT

A. Standard methodology/document with linked IT tool?

B. Standard methodology/document with no linked IT tool?

C. Standalone IT tool

II. CRITERIA OF BEING AN ESS STANDARD

A. Is it a medium with information recorded on or in it?

E. Is it designed for common and repeated use by actors in the ESS?

F. Does it support the production of EU statistics?

G. How does it aim at the achievement of the optimum degree of order in the context of the implementation of the mission of the ESS? B. PRODUCT LEVEL

Where the costs and benefits analysed before the actual development of the standard started? If so, how? What aspects were considered? If there were initial cost/benefit evaluations, how did you do it (methodology, template)?

Were these initial costs/benefits hard to measure? What do you think, what is necessary to do reliable cost/benefit analysis at the beginning? What is needed for it?

If stakeholder analysis was carried out before development, what approach did you use to analyse the needs (written consultation, direct e-mail contact, user forum, etc.)?

Was each stakeholder asked directly to provide the needs or was it voluntary?

Do you think the approach you chose was the best to collect information for the analysis?

What is your experience so far: what works best for collecting information from stakeholder to identify their

Relevance:

Does the normative document have a clear connection to the production of EU statistics?

Is it known in which process of the GSBPM and/or subject-matter domains – according to DISA – the normative document is meant to be used?

Comparability and coherence

What is the impact of the normative document on the ESS as a whole?

E.g. ESS production system, position of the partner of the ESS, stakeholders of the ESS, etc.

C. Process level

Relevance

What was the motivation behind development?

Was it user driven (wanted to find new solution to stakeholder needs) / process driven (select business process steps requiring significant resources, producing large delays or constituting quality risk) /

needs?

When the development of the standard started (with establishment of needs), was there a plan available, how it is gonna be consulted, tested, finalized, adopted, maintaned? (3.1)

How did you compile such a roadmap? (3.1)

What can you recommend as good examples and practical proposals for a good management of such a roadmap? (3.1)

technology driven (using new technology in the production of statistics)?

Was there any analysis undertaken about the need to develop the normative document concerned before any decision on its development was made?

Cost-benefit analysis

Ex-ante costs and benefits associated with the standard setting and implementation processes (connection to SWOT)

IV. PROCESS LEVEL

Relevance

Was there any cost and/or benefit analysis undertaken before or during development?

There is also an approach that cost/benefit dimensions can continuously be added as the standard goes through its life cycle: elements can be added before development / during development /during adoption / during dissemination / during maintenance. Do you think that this is a feasible approach?

What is your experience? (5.1)

Stakeholder analysis

Identify and involve key stakeholders


Did all ESS members have a chance to express their opinion?

IV. Process level

Relevance

Were all possible stakeholders involved in the development process of the normative document?

Were there any consultations with the stakeholders?

Timeliness and punctuality

Were the stakeholders involved at appropriate time during development?

Detailed proposal

Professional (somehow detailed) justification of the need for standard-setting (including the key professional elements of the normative document). Assign responsibilities, estimate resources

III. PRODUCT LEVEL


Is the normative document free of any controversies?

What was the process of adoption of the normative document?


Who adopted the final normative document?

Was it explicitly mentioned in the

mandate of the organisation adopting the normative document, to adopt standards?

IV. PROCESS LEVEL

Accessibility and clarity

Did the development process follow a pre-defined schedule and approach?

Was it known at the beginning what phases and steps were to follow?

Was the process schedule available before the actual process started?

Were the milestones of the process clear enough at the start? Were corresponding responsibilities and decision making points clear enough for all stakeholders? Were they clearly regulated?

Decision and approval

Approval of the need for development / Appoint a project leader and a working party of experts with inclusion of opinion of all stakeholders


B. Is it established by consensus among ESS members? Who discussed it? Did all ESS members have a chance to express their opinion?

How was the consensus assured?

2. Develop

Draft the new standard

The appoined working party prepares a working draft of the standard

IV. PROCESS LEVEL

Relevance

What is the medium of development?

When you tested the draft of the new standard with your stakeholders, how did you carry out the tests? Were they directly asked to test it or was the test voluntary?

What do you think is the best way to test a draft standard still being developed by the stakeholders (ask direct questions or let them „play with it”)?

Were the feedback on tests enough for your to carry on with the development of the standard (or issues arised later that could have been omitted if test thorough tests during the development are involved)?

How did you consult the results of the draft standard with the stakeholders? Based on your experience, do think it should have

Test and consult

Test the clarity of the candidate standard and report results and feedback / Check whether the draft version meets all the original expected goals


B. Is it established by consensus among ESS members? Who discussed it? Did all ESS members have a chance to express their opinion? How was the consensus assured

III. PRODUCT LEVEL

Was the clarity of the normative document tested before official release? (pilot tests, focus groups, etc.)

been done better?

What do you think could work to be as effective as possible with these consultations during the development period?

When you drew up your dissemination plan, what aspects you considered to do it the best you can?

What do you think is needed to do a profound dissemination plan (time, contact, information)?

When you prepared the formal proposal, based on the draft standards and the tests and feedback, did you feel that you missed information or that something else should have been done compared to what have been done (involvement of more stakeholders, more explicit questions/tests)?

Quality assessment

Asses internally or externally the completeness and consistency with already existing standards or normative documents currently being developed / Evaluate the draft version against the quality criteria layed down earlier during development

III. PRODUCT LEVEL

Accuracy

Does the normative document use and/or refer to other normative documents in a clear context?

E.g. explicitly naming the classification, technical initiative, IT system to use, etc.

Does the normative document use widely known and used concepts, classifications, methods, etc. or does it define its own? If not only widely known and used concepts, classifications, methods, etc. are used; e.g. does it contain a glossary of terms?

In connection to the goal of common and repeated use, does it provide clear and unambiguous content/instructions/rules, etc. for practical purposes? Is it clear enough what to do in practice?


Does the normative document use up-to-date classifications, terms, methods?

Is it updated within reasonable timeframe whenever a connecting normative

Do you think it is feasible to separate the development and adoption phases of standard-making from the process of actual implementation? How do you think they are different? (3.4)

When the standard was still under development, did you use a piloting group to test the proposed standard? If so, was it a useful approach? If not, do you think it should have been done? (3.5)

During the whole life cycle of a standard, did you do some mapping on connecting initiatives, standards that could be relevant for your own? Was it burdensome? How do you think it can be assured that these connecting initiatives know about each other? (3.6 / 6.1)

document (classification, etc.) is updated?


Does the normative document use concepts, classifications, etc. used in the framework of other normative documents?

IV. Process level


Were other needs and possible connection points to other normative documents considered during the development of the normative document?

Were the normative documents under development also considered during the development of the given normative document?

Design support measures

Identify the organisation and the training necessary to support the implementation of the standard with special attention given to the expected needs the standards is meant to fulfill and all needs of stakeholders

III. PRODUCT LEVEL

Accuracy

In connection to the goal of common and repeated use, does it provide clear and unambiguous content/instructions/rules, etc. for practical purposes? Is it clear

IV.

enough what to do in practice?

Accessibility and clarity What supporting instruments are available for stakeholders? (training, workshop, mentoring, helpdesk material, etc.)

Are these supporting instruments available at the same place the normative document itself?

Is the normative document available in English?

Is the normative document available in various formats according to stakeholder needs?

V. IMPLEMENTATION

Are there any supporting instruments available to help implementation of the normative document?

Dissemination plan

Description of dissemination channels and forms / Communication plan to stakeholders and users


B. Is it established by consensus among ESS members? Did all ESS members have a chance to express their

opinion? V. PROCESS LEVEL

Relevance





Prepare formal proposal

Proposal for the the adoption and implementation of the standard with regard to the outcome of testing, consultations and quality assessment

3. Adopt

Formal presentation for stakeholders

Present the normative document to all stakeholders at different fora (especially the body professionally being responsible for the content of the normative document)


B. Is it established by consensus among ESS members? Who discussed it? Did all ESS members have a chance to express their opinion? How was the consensus

How did you choose the way of formal presentation of the final standard to the stakeholders? Do you think that written form is enough?

What do you think, when is oral form a real necessity for the formal presentation to stakeholders?

assured

C. What was the process of adoption of the normative document? Who adopted the final normative document? Was it explicitly mentioned in the mandate of the organisation adopting the normative document, to adopt standards?

IV. PROCESS LEVEL


Was the development process effective enough that the normative document was still relevant at the time of its availability?


How did the stakeholders accept the final version of the standard?

What do you think is the best way to manage the acceptance phase of a standard by the stakeholders? What works and what not, based on your experience?

During the ratification phase (when the responsibilities were assigned and the decision on resources was made), were there any problems with the process?

Acceptance by stakeholders

Acceptance by all stakeholders according to a formal procedure that makes sure all stakeholders have their chance to express their opinion. The working party appointed for the normative document gives feedback to all opinion and comment. The normative document is accepted only if no more major concerns (principle, strategic level) remain


B. Is it established by consensus among ESS members? Who discussed it? Did all ESS members have a chance to express their opinion? How was the consensus assured

C. What was the process of adoption of the normative document? Who adopted the final normative document? Was it explicitly mentioned in the mandate of the organisation adopting the normative document, to adopt standards?

V. PROCESS LEVEL

Relevance



How were the special national and/or subject-matter aspects considered?

How were their expectations/needs treated during the process?

Formal presentation for ratification

Present the normative document for the recognised body (explicit mandate for accepatance of standards)


C.What was the process of adoption of the normative document?


Was it explicitly mentioned in the mandate of the organisation adopting the normative document, to adopt standards?

Ratification, assignment of responsibilities, decision on resources

Formal ratification according to the standard-making process


C.What was the process of adoption of the normative document?


Was it explicitly mentioned in the mandate of the organisation adopting the normative document, to adopt standards?

4. Disseminate

, apply Disseminate

Dissemination of standard and supporting measures via publications, internet, websites, etc.

III. PRODUCT LEVEL

Relevance

Is there an appropriate documentation linked to the normative document to provide additional assistance to stakeholders to understand and clarify the content (if needed)?


Is the normative document publicly available without any restrictions (such as: registration requirements, authentication, etc.)?

What supporting instruments are available for stakeholders? (training, workshop, mentoring, helpdesk material, etc.)

Are these supporting instruments available at the same place the normative document itself?

Is the normative document available in

When you chose the dissemination channels, what did you consider (stakeholder needs identified at earlier stages / short mapping of available dissemination channels, etc.)?

When support was provided to stakeholders, how did you chose the best ways for that support?

Was the support dinamically modified according to exact needs for support?

English?

Is the normative document available in various formats according to stakeholder needs?

IV. PROCESS LEVEL


Did the development process follow a pre-defined schedule and approach?

Was it known at the beginning what phases and steps were to follow?

Was the process schedule available before the actual process started?

Were the milestones of the process clear enough at the start? Were corresponding responsibilities and decision making points clear enough for all stakeholders? Were they clearly regulated?

Apply and feedback

Report experiences and problems

III. PRODUCT LEVEL

Relevance

To what extent does the normative document satisfy the needs of stakeholders?

Monitor Use of standard is monitored by the maintenance agency according to rules set

V. IMPLEMENTATION

Relevance

Is the normative document currently implemented or under implementation by any ESS member?

What was the motivation behind the decision on implementation?

Who decided about the implementation?

Was there any cost and/or benefit analysis undertaken before implementation?

What were the actual costs of implementation in terms human, financial, educational and organisational perspectives (if any)? What were the one-off and recurring costs?

What were the main obstacles during implementation? How were these problems solved in practice?

At what level was the implementation undertaken?

Accuracy

If there was any preliminary cost and/or benefit analysis undertaken, how accurate were the estimation of costs?


When was the return on investment realised?

Were there any immediate benefits realised after implementation?


Was the implementation of the normative document planned before or during its development?



Was there any explicit and realised effect on the quality of the statistical output of the ESS (national and EU levels) as defined in the European Statistics Code of Practice in the following aspects?

What was the explicit and realised effect of the implementation on the quality of

the statistical processes used to produce EU statistics?

Were there any immediate gains realised at product and/or process level after the implementation of the normative document?

If so, at what level and in what context?

Do we have any information on use outside of the ESS?

If so, where is it used?

Are there any known risks of failure of adoption of the normative document? Are there any practical examples of such cases?

If yes, please describe.

Conformity assessment

Assessment of standard and formulation of correction and improvement actions. In case, decide the standard is out-of-date and archive

IV, PROCESS LEVEL

Accuracy Did the development process go according to the original schedule?

Were there any delays during development?

Was there any divergence from the

development plan during the actual development?

V,. IMPLEMENTATION

Relevance




Was there any cost and/or benefit analysis undertaken before implementation?



At what level was the implementation

undertaken?

Accuracy






Was the implementation of the normative document planned before or during its development?



Was there any explicit and realised effect on the quality of the statistical output of the ESS (national and EU levels) as defined in the European Statistics Code of Practice

in the following aspects?

What was the explicit and realised effect of the implementation on the quality of the statistical processes used to produce EU statistics?







5. Maintain, review

Collect and follow-up feedbacks

Systematically review the standard (especially if a related normative document is updated) with the active involvement of all stakeholders, preferably the body professionally being responsible for the content of the normative document

II. CRITERIA OF BEING A STANDARD

Is there an update process of the normative document? Please provide a brief description of the update process (major steps, responsible bodies, etc.).

III. PRODUCT LEVEL


Is there any option to receive feedback from stakeholders?

If feedback is received, what is the process to include the relevant feedback in the updates?

V.IMPLEMENTATION

Relevance




Was there any cost and/or benefit analysis undertaken before

When you collect follow-up feedback, what channels you used for collecting this information (ask stakeholders/users directly, collect already available information from other platforms, monitor the usage, etc.?)

When the necessary modifications of the standard were identified, how did you know it reflected the needs of all stakeholders?

Did you consult with the stakeholders? What channels and

implementation?



At what level was the implementation undertaken?

Accuracy






Was the implementation of the normative

forms of consultation did you use?

document planned before or during its development?



Was there any explicit and realised effect on the quality of the statistical output of the ESS (national and EU levels) as defined in the European Statistics Code of Practice in the following aspects?

What was the explicit and realised effect of the implementation on the quality of the statistical processes used to produce EU statistics?







Review

Review the standard(s) in light of the feedbacks and other circumstances and future needs, preferably with the responsibility of the body professionally being responsible for the content of the normative document

III. PRODUCT LEVEL

Relevance

Is the normative document complete?

Does it fulfil the needs of stakeholders?

Is there an appropriate documentation linked to the normative document to provide additional assistance to stakeholders to understand and clarify the content (if needed)?


E,Is it designed for common and repeated use by actors in the ESS

Is there an update process of the normative document?

Proposal for confirmation, revision or withdrawal

Based on the results of the review, there is a professional summary where the future of the standard (justified need for revision or withdrawal) is stated



Is there an update process of the normative document

Approval of the proposal

Formal acceptance of refusal of the reviewed normative document by the recognised body responsible for the adoption of the standard



Is there an update process of the normative document

Documents

ESSnet on Standardisation WP3 evaluation report structure ......ESS standards Corresponding questions from the WP3 assessment template Final clarifying questions for the last telephone