LUX, Ivan

IAEA

7/5/2012

2011Prompt Evaluation of the IRRS Mission to Switzerland

Prompt Evaluation of the IRRS Mission to the Switzerland Page | 1

Table of Contents1. Introduction...............................................................................................................................2

2. Basic Mission Data.....................................................................................................................2

3. Feedback from the Team Members on the Effectiveness of the Mission..................................2

Characteristic opinions.............................................................................................................3

Further general remarks...........................................................................................................4

General conclusion...................................................................................................................4

4. Feedback from the Host Country on the Effectiveness of the Mission......................................4

5. Evaluation of the Effectiveness and Efficiency of the Mission...................................................4

Discussion and conclusions.......................................................................................................7

6. Summary....................................................................................................................................7

Prompt Evaluation of the IRRS Mission to the Switzerland Page | 2

1. Introduction

The Contribution Agreement ENER/11/NUCL/SI2.588650 between the European Atomic Energy Community (represented by the European Commission) and the IAEA among others foresees as an expected result a performance monitoring based on the evaluation of some key performance indicators including:

annual reports of European IRRS mission findings, results actions; and identification and communication of emerging issues, trends and challenges.

The Nuclear Safety Action Plan of the IAEA has as one of its main objectives the requirement to “Strengthen IAEA peer reviews in order to maximize the benefits to Member States” and in specific, it calls the IAEA Secretariat

to review the effectiveness of the IRRS peer reviews.

In reply to these requirements a system of performance indicators have been elaborated in order to measure the efficiency and effectiveness of the IRRS missions. Some of the performance indicators are based on direct feedbacks from the IRRS team members and from the representatives of the country hosting the IRRS mission.

Results of and conclusions from the feedbacks and from the efficiency and effectiveness evaluations are given in the present Prompt Evaluation Report.

2. Basic Mission Data

Mission type: Full Scope First MissionHost country: SwitzerlandMission date: 20 November – 2 December 2011Team Leader: Jean-Christophe NIEL (France)Deputy Team Leader: Eric Leeds (USA)Team Coordinator: Adriana NICIC (IAEA NSNI)Deputy Team Coordinator: James STEWART (IAEA NSRW)Number of external experts: 18Number of IAEA staff: 5Number of observers: 1Scope of the mission:

Core modules (No. 1 through 10) Facilities and activities: NPP; research reactors; waste facilities and waste management;

decommissioning Additional areas (Module 11): transport of radioactive material; occupational radiation

protection; control of radioactive discharges, materials for clearance; environmental monitoring

Fukushima module

3. Feedback from the Team Members on the Effectiveness of the Mission

The team members are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) five questions and also expanding in free text their comments on the questions. The questions and the average values of the team members’ ratings are given below:

No. Question Average markQt1 How effective do you consider the activity of the expert team during the

mission?4.35

Qt2 How effective do you consider the activity of the IAEA staff in the team 4.35

Prompt Evaluation of the IRRS Mission to the Switzerland Page | 3

during the missionQt3 How effective do you consider the activity of the Team Leader? 4.00

Qt4 How effective do you consider the activity of the Deputy Team Leader? 4.45

Qt5 How satisfied are you with the preparations of the mission (enough time for preparing yourself, information provided by the IAEA, etc.)?

4.03

Total average 4.24

Characteristic opinions

Qt1:

The team was rather inexperienced in terms of participation in previous IRRS missions that has led to lengthy discussions and in that the expertise of inexperienced reviewers could not be fully used

The team was well configured with experts from all areas Discussion within the team could be optimized

Qt2:

The IAEA staff could have played more the role of “support” for the team (explaining the expectations)

It seems that some of the team members had not read the guidance, they would have needed more assistance

The role of the IAEA staff might be clarified The support of the IAEA was fairly effective. There were some difficulties to keep track of the

draft report versions The experience from former missions was helpful

Qt3:

The team leader could have played more the role of a “coach” and should have put more energy in integrating everybody in the team

Team leaders should have at least 3 prior IRRS mission participations The TL was very good and friendly

Qt4:

Considering that this was the DTL’s first mission he also did very well The DTL also should have prior IRRS experience He did an excellent job, motivated the team and worked well together with the TL He is one of the most enthusiastic leaders I have ever met

Qt5:

The formal announcement on the team membership cane rather late (little more than on a month prior to the mission)

The ARM was provided too late The “pressure” of requesting questions and positions prior to the mission was very good Reviewers (especially who entered as substitutes for others) had not enough time for the

preparations The overall information and coordination provided by IAEA and the host country were very

good IAEA should ask for prerequisites of the participation in a mission (CV, medical certificate,

etc.) in good time IAEA may consider to have a contingency plan and reserve experts for the case if someone

drops out of the team

Prompt Evaluation of the IRRS Mission to the Switzerland Page | 4

Further general remarks

IAEA should give guidance to the team leaders on how to ensure that the executive summary reflects the most important findings. The executive summary should be compiled in a systematic way, not in a rush.

An overall two-week schedule of the mission should be developed IAEA should have a check-list to cover all important information well before the mission

General conclusion

1) The team was rather inexperienced (only 3 experts out of 18 have had previous IRRS experience) yet it could work efficiently.

2) The activity of the IAEA team was appreciated yet more assistance was expected3) The team members considered the activity of both the TL and of the DTL good, although

their experience in IRRS missions would have been beneficial4) Preparation of the mission was in delay and hindered by substitutions5) Guidance for the team members and general planning of the preparations and of the

missions would be beneficial

4. Feedback from the Host Country on the Effectiveness of the Mission

The representatives of the host country are requested to offer their opinions on the effectiveness of the mission by rating from 1 to 5 (5 reflecting the highest satisfaction) seven questions and also expanding in free text their comments on the questions. The questions, the host country’s ratings and the associated comments are given below:

No. Question MarkQh1 How effective do you consider the mission in assisting the continuous

improvement of nuclear safety in your country?4

Qh2 How objective was the peer review? 5

Qh3 How has the mission helped the exchange of information, experience and good practice with other countries?

5

Qh4 How consistent was the use of the IAEA safety requirements and guides in the mission?

4

Qh5 How justified are the findings of the peer review? 4

Qh6 How relevant are the findings of the peer review for the future development of your regulatory body?

4

Qh7 How competent were the reviewers in their review and findings? 4

Average 4.29

Comment: It was sometimes difficult to answer since the team was somehow heterogeneous with respect to experience in IRRS missions. Generally we are happy with the result.

5. Evaluation of the Effectiveness and Efficiency of the Mission

The Performance Indicators developed for the measurement of the effectiveness and efficiency of a mission were evaluated in the extent as it was made possible by the available data. Accordingly the questionnaire on the ARM feedback was not available at the time of the mission.

In the next figure the results of the evaluation of the performance indicators as well as the overall effectiveness of the mission are presented. The rightmost columns (EFF. INDICATION) present the ranges where the particular PIs fall (green – optimum, yellow – acceptable, red – needing attention),

Prompt Evaluation of the IRRS Mission to the Switzerland Page | 5

whereas the frame in the right lower part summarizes the overall effectiveness of the mission (green – optimum, white – effective, yellow – acceptable, red – to analyse).

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 6

Efficiency and Effectiveness of an IRRS Mission first 2012-06-26

INPUT QUANTITIES valueA priory mission data No(id) Unit Value min.(>) max.(≤) min.(>) max.(≤) col PI

Type of mission (1 - first or 2 - follow-up) 1 optimum 0.0 C1(T) - 23 21.8 26.6 19.4 29.0 0.00 Team sizeHost country SWITZERLAND effective 0.1 C2(P) - 137 120 146 106 159 0.00 Report lengthStating time of the mission (yyyy-mm-dd) 2011-11-20 acceptable 0.2 C3(tARM) day 37 45 - 30 45 0.18 ARM review timeEnding time of the mission (yyyy-mm-dd) 2011-12-02 needs attention C4(NAWC) - 12.00 13.60 - 10.20 13.60 0.12 Advance commentsNumber of modules (nT) 12 C5(fbARM) - - 4.00 5.00 3.00 4.00 - - Feedback on ARMNumber of supplementary modules (in follow-up) (nM+) T 0-first -0.96 C6(Exp) - 0.17 0.50 0.66 0.33 1.00 0.30 Team experienceNumber of facility and activity types (nx) 4 λ T-first 1.48 C7(fbHost) - 4.29 4.00 5.00 3.00 4.00 0.00 Host feedbackNumber of additional technical areas (nT) 3 T 0-fu 3.09 C8(fbTeam) - 4.24 4.00 5.00 3.00 4.00 0.00 Team feedbackNumber of policy discussion issues (nP) 4 λ T-fu 0.71 C9(β1) - 1.09 1.00 - 0.80 1.00 0.00 Action Plan coverageNumber of NPP units in the host country (ν) 5 C10(β2) - 1.20 1.00 - 0.80 1.00 0.00 Beyond AP coverage

Pre-mission data P 0-first 17.60 C11(ρ) - 1.20 0.75 1.25 0.50 1.50 0.00 Balance of findingsNumber of Team Members (including IAEA) (T) 23 λM-first 5.83 C12(RCont) - 0.80 0.90 1.00 0.80 0.90 0.11 Report concisenessNumber of Experts in the Team (without IAEA) (Te) 18 λ F-first 0.63 C13(RTime) day 148 0 90 90 120 0.30 Report completionNumber of Experts with IRRS experience (Tx) 3 P 0-fu 54.20 C14(OpenI) - - 0.00 0.20 0.20 0.40 - - Open issues leftARM distribution date (yyy-mm-dd) 2011-10-13 λM-fu 1.55Number of issues in the Action Plan (NAP) 11 λ F-fu 0.64Number of advanced ARM comments from TM's (NAWC) 12 μ 0.90 Number of findings (R+S) in the first mission (for follow-up)

Mission data (R+S) av 39Number of Recommendations (R) 12 R av 15.1Number of Suggestions (S) 18 G av 12.3Number of Good Practices (G) 19 (R*G) av 157.4

Number of findings also in the Action Plan (N=) 6Number of issues remained open (for follow-up)Number of Report Pages (P) 137 EFF.& EFF. OF THE MISSION TOConciseness of the Mission Report (ρ) 0.80 Size of nucl.pr. 1 Mission type:

Questionnaire data Mission size 17 Missions duration:Average mark on ARM quality 0.00 Opt.team size 24.2 Average Measure of Deviation (Δ):Average mark by the host country 4.29 Opt.rep.length 133 Overall Effectiveness:Average mark by the Team Members 4.24

Post-mission dataDate of isuance of mission report (yyy-mm-dd) 2012-04-30

fields to input!

effective

EFF. INDICATIONRel.Dist from Opt.

Optimum rangePERFORMANCE INDICATORS Acceptable range

2011-11-20

For report length

CALCULATED DATA

For findings

SWITZERLAND

SWITZERLAND

2011-12-02first mission

0.08

LIMITS & PARAMETERS

For average deviation

For team size

Prompt Evaluation of the IRRS mission to the Slovak Republic P a g e | 7

Discussion and conclusions

1) Parameters related to the preparation of the mission show divers values. The Team Size was optimum, the ARM Review Time, although short (37 days), yet acceptable, however, the Team Experience indicator, i.e. the number of experienced reviewers was rather low thus resulting in the red range (only 3 experienced members out of 18). The number of Advance Comments was low, probably also due to the inexperienced team. Comments from the team members on the preparations need to be considered in the future.

2) The majority of the products of the mission were in the optimum range, thus the Action Plan Coverage, i.e. the number of Action Plan issues also covered by mission findings, the Beyond Action Plan Coverage, i.e. the number of findings above the issues in the AP and the Balance of Findings, i.e. the proportion of good practices and recommendations were in the green.

3) Feedbacks from both the hosts and the reviewers were in the optimum with certain fluctuations among the various questions. Propositions in the answers need to be taken into account in future missions and in the revision of the guide.

4) Most likely the insufficient Report Conciseness (i.e. the coverage of the relevant topics in the report) is also partly due to the inexperienced team.

5) Issuance of the report was rather slow (148 days).6) ENSI was satisfied with the results of the mission.7) The overall effectiveness of the mission is in the white (effective range) and its value (0.08)

is in harmony with other recent missions.

6. Summary

The data of and feedback from the full scope first IRRS mission to Switzerland have been analysed. The following conclusions are drawn:

The team was rather inexperienced (only 3 experts out of 18 have had previous IRRS experience) yet it could work efficiently.

The activity of the IAEA team was appreciated yet more assistance was expected The team members considered the activity of both the TL and of the DTL good, although

their experience in IRRS missions would have been beneficial Preparation of the mission was in delay and hindered by substitutions Guidance for the team members and general planning of the preparations and of the

missions would be beneficial Preparations of the mission suffered from some delays The Performance Indicators related to the activity of the team during the mission were in

the optimum range The mission was rated effective by the evaluated Performance Indicators