View
2
Download
0
Category
Preview:
Citation preview
2015 IAEA
[FURTHER IMPROVING THE
IRRS PROCESS] Discussions points for the consultancy by experienced IRRS reviewers, Vienna, 20-22.04.2015
Discussion Paper on IRRS Development 2015_v2.docx
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 2
Contents
FURTHER IMPROVING THE IRRS PROCESS .......................................................................................... 3
Introduction .................................................................................................................................. 3
I. Preparatory Phase of a Mission .............................................................................................. 4
I.1 Information meeting ...................................................................................................... 4
I.2 Preparatory meeting ...................................................................................................... 4
I.3 Advance Reference Material (ARM) ................................................................................ 5
I.4 Team size, assignment, recruitment, experience and training ......................................... 6
I.5 Expected preparatory activity of the team ...................................................................... 9
II. Conduct of a Mission ........................................................................................................... 10
II.1 Schedule of a mission ................................................................................................... 10
II.2 Observations ................................................................................................................ 10
II.3 Review actions ............................................................................................................. 14
II.4 Report writing .............................................................................................................. 16
II.5 Activity of team lead..................................................................................................... 17
II.6 Policy issues discussions ............................................................................................... 18
II.7 Feedback from the missions.......................................................................................... 18
II.8 Evaluation of team members’ performance .................................................................. 20
III. Post-mission Phase .......................................................................................................... 21
III.1 Finalization of the mission report .................................................................................. 21
III.2 Evaluation of missions .................................................................................................. 21
IV. Follow-up missions ........................................................................................................... 23
IV.1 Follow-up as a compulsory part .................................................................................... 23
IV.2 Timing of follow-up ...................................................................................................... 23
IV.3 Follow-up team composition ........................................................................................ 24
IV.4 Extended follow-up missions......................................................................................... 25
V. Development of the IRRS Process......................................................................................... 26
V.1 Scope of an IRRS mission .............................................................................................. 26
V.2 Possible changes in the IRRS Module structure ............................................................. 26
V.3 Possible changes in the IRRS Module review scope........................................................ 27
References .................................................................................................................................. 28
Appendix I: A typical IRRS preparatory meeting Agenda ............................................................... 29
Appendix II: Team assignment and grouping ................................................................................ 31
Appendix III: Standard Schedule of IRRS Missions ........................................................................ 33
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 3
FURTHER IMPROVING THE IRRS PROCESS
Discussions points for the Consultancy of Experienced IRRS Reviewers 20-22 April 2015
Introduction In order to further improve the IRRS process a series of Targeted Consultancies have been
conducted in 2012. For similar reasons of IRRS missions conducted between 2006 and 2013 have
been analysed. The IAEA Nuclear Safety Action Plan requires the secretariat to develop effectiveness
criteria of all peer reviews, including the IRRS. Partly to answer this requirement a set of
performance indicators have been introduced to measure the effectiveness and efficiency of a
mission. To summarize the lessons learned and set the development goals a Technical Meeting (TM)
was organized for team leaders and deputy team leaders of past IRRS missions in January 2013. The
documents summarizing the results of the above consultancies and the IRRS mission reports analysis
contain conclusions and proposals for improving further the IRRS process. Many other documents
related to the topics of the consultancy are available from the Share Point area1.
All these discussions, consultations and analyses have led to rapid recent developments of the IRRS
process on one hand and raised questions that need more thoughts for other developments on the
other hand. The Secretariat decided on inviting the most experienced IRRS reviewers for a
consultancy meant to discuss those questions, assist the Secretariat and to seek for further advice in
the development of the IRRS process and its continuous improvement. The present discussion paper
summarizes the topics suggested for discussions during the consultancy scheduled for 20-22 April
2015.
The issues to discuss are divided into five chapters (Preparatory phase of a mission, Conduct of a
mission, Post-mission phase, Follow-up mission and General development issues). Each chapter
contains as many sections as the number of issues belonging to the subject of the chapter.
For easier orientation among the various references, the Targeted Consultancy (TC)
recommendations as given in the consultancy reports are quoted; similarly the conclusions/
proposals of the IRRS analysis reports ‘Lessons Learned from IRRS Missions in 2006-2013 to
Countries with Operating NPPs’ [1} and ‘IRRS Missions 2006-2013: Analysis from a Radiation Safety
Perspective’ [2] are listed (if applicable). The reports above are available from the IAEA IRRS
website2. Whenever relevant, observations by the Technical Meeting of IRRS team leaders held in
2013 [5] are also quoted.
Each section first describes the actual status, then briefly quotes the related issues, recent
developments, and conclusions, proposals for further developments (if any), finally formulates the
questions to be discussed and answered by the consultancy.
The reader is encouraged to consult the underlying documents referred to footnotes 1 and 2 and
bring up further issues for discussion that are not covered by this report but are considered needing
discussion.
1 The Terms of Reference and the reports of the Targeted Consultancies, the Chairmen Summary of the TM of
IRRS team leaders, the report on the analysis, the report on the effectiveness and efficiency performance
indicators, the reports prepared to the IRRS Lessons Learned Ws in Moscow all are available from the Share
Point Area website:
http://gnssn.iaea.org/NSNI/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2FNSNI%2FShared%20
Documents%2FOPEN%20Shared%20Files%2FDocuments%20for%20the%20Consultancy%20of%20Experienced
%20IRRS%20Reviewers%2C%2020%2D22%20April%202015&InitialTabId=Ribbon%2EDocument&VisibilityCont
ext=WSSTabPersistence
2 http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 4
I. Preparatory Phase of a Mission
I.1 Information meeting
Status
The IRRS Guidelines does not mention information meetings as part of the preparatory phase. In
practice information meetings are held at the request of the host country in order to obtain
information on the IRRS process in general and the actual mission in specific. Often the information
meeting is organized in conjunction with a national self-assessment seminar.
Issues, developments and proposals
The TCs have dealt with the information meeting and offered the following suggestions (the number
ahead of dot is the sequence number of the consultancy, the one after dot is the sequence number
of the suggestion in the given consultancy):
10.23: After the decision on an IRRS Mission the first step should be a one-day information meeting
between IAEA and the host country about 18-24 months prior to the mission.
10.25: IAEA should consider to make information meeting compulsory and to modify the agenda of that
meeting and of the Preparatory Meeting accordingly
Question(s) to the consultancy
Q-I.1: Should the information meeting be a standard part of the IRRS process? If so, what would be
its standard agenda/duration?
Q-I.2: Do you think that the information meeting may be performed via video-conference?
I.2 Preparatory meeting
Status
The preparatory meeting is defined in Section 7.2 of the IRRS Guidelines and further detailed in
Appendix VI of the Guidelines. The main characters are: it is to be conducted six to nine months prior
to the mission, lasts two or three days, is to be attended by the IRRS Team Leader, the IAEA
Coordinators and, as necessary, the Deputy Team Leader and other representatives of appropriate
IAEA Divisions. The main purpose is to inform the RB about all important details related to an IRRS
mission. A typical agenda of a preparatory meeting is given in Appendix I.
Issues, developments and proposals
The TCs formulated the following suggestions related to the preparatory meetings:
1.08 and 10.28: In order to shorten the preparatory meetings the generic information about IRRS missions
to present by IAEA to the host should be provided in a separate event if needed and the meeting should
focus on high level issues.
1.09: The participants of the preparatory meetings from IAEA side should be limited to the persons given in
the IRRS Guidelines (Team Leader, Deputy Team Leader and Team Coordinator).
4.06: The preparatory meeting presentations be recorded and made available to the reviewers in advance
to the mission
10.29: At the preparatory meeting a thorough discussion should be held about the background of the
suggested Policy Issues.
Regardless of the guidelines and the TC proposals, in practice the preparatory meeting is also used as a review
of progress of the host state self-assessment (c.f. in Appendix I).
Question(s) to the consultancy
Q-1.3: What would be the optimum contents of the preparatory meeting agenda?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 5
Q-1.4: What would be the optimum composition of the team participating in preparatory meetings?
Q-1.5: How would you suggest coordinating the information and preparatory meetings?
I.3 Advance Reference Material (ARM)
Status
The ARM is described in Section 7.5 of the IRRS Guidelines while Appendix V of the Guidelines lists
the typical documents to be included into the ARM. The ARM is supposed to include the results of
the self-assessment, a summary report on the self-assessment results and all documents that may
be relevant for the review process. The documents in the ARM are typically related to the national
legal framework, the regulatory framework, and reports describing details of the activity of the
government and the regulatory body related to the scope of the mission.
Issues, developments and proposals
Previous consultancies and conducted IRRS missions raised the following issues:
a) Timely submission and quality of the ARM by the hosts
b) Preparation of a concise yet informative summary report of the ARM
c) Translation of parts of ARM to English
a) Time for ARM submission
The IRRS Guidelines requires the submission of the
ARM at least two months prior to the missions.
Recent practice is illustrated by Figure 1. It shows
the (time available for ARM review minus 45 days)
in the past missions. It is seen that in many missions
even 45 days were not available for the reviewers
and only in a few missions (values above the blue
line) was 60 days granted.
Figure 1: Time available for ARM review – 45 days in recent missions.
The TCs suggested:
2.05: Staff selected to participate in a mission should be allowed sufficient time for reading the ARM and
other necessary preparation for the mission
In report [1] the following related conclusion is reached:
C22: The ARM should be provided with sufficient time preceding the IRRS mission
b) ARM Summary Template
Since the result of the self-assessment is a very extensive document it is reasonable to request a
concise and informative summary of the ARM. The TCs offered a number of related suggestions:
1.03: The ARM should include summaries for every module highlighting the specific aspects of the
regulatory framework of the host country and give an overall perspective on the regulator. The host
country should prepare the ARM in the same format and template used by the IRRS team to prepare the
mission report
4.01: Summaries in the ARM for each module should be structured so that it helps the reviewer in
evaluating the compliance with the relevant GSR part 1 requirements specific to that module. IAEA should
develop a template for mapping to help the host country in the demonstration
7.06: The host country should provide the reviewers with a short summary for each module in the ARM (list
is given in the consultancy report)
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 6
10.16: IAEA should elaborate a template for the ARM, the host country should prepare the ARM using this
IRRS ARM template
The Technical Meeting of IRRS team leaders [5] also suggested the elaboration of such a template.
In report [1] it is concluded:
C24: The quality of ARM needs substantial improvement; for that purpose the hosts may need further
guidance
In report [2] it is proposed:
P22: Having read the Host Country’s ARM, including the self-assessment, IRRS Team members should pre-
draft their sections of the IRRS report, using an improved IRRS report template to facilitate the process and
ensure it is consistent…
P23: The IAEA should develop a template for the consistent and straightforward compilation of ARM,
possibly based on the SARIS self-assessment report and consistent with the IRRS mission report template
In reply to these suggestions the Agency has elaborated a Standard IRRS ARM Summary Template
(available from the Share Point area referred to in footnote 1).
c) Translation
Full translation of the documents included in the ARM is very seldom possible. An agreement needs
to be reached for the reasonable amount of translation and the completeness of information made
available to the reviewers. The related suggestions 5.01 and 6.12 by the TCs list those documents
that need to be translated. Furthermore, it is suggested that
6.02: There is a need for a quality check of English translations of documents submitted to the IRRS team
Question(s) to the consultancy
Q-1.6: What is the optimum timing of submission of ARM?
Q-1.7: Can a mission be postponed if the ARM is too late or of poor quality?
Q-1.8: Do you have any suggestion on modification of the Standard IRRS ARM Summary Template?
Q-1.9: What would you suggest as guiding principles for translating ARM documents to English?
Q-1.10: Is there any document presently not part of ARM that you feel necessary to include?
Q-1.11: How would you stimulate/invite Member States to ensure sufficient time of preparation for
their mission participants?
I.4 Team size, assignment, recruitment, experience and training
Status
The IRRS Guidelines describes the team size (5-20 international experts), composition (TL, DTL, IAEA
C. IAEA DC, IAEA Review Area Facilitator, review team members, Administrator) and the recruitment
in Section 7.6. No guidance is given as for the determination of the optimum size of a mission team
or on the workforce necessary for reviewing actual Modules. Neither are requirements on IRRS
specific training of reviewers. The Guidelines requires that “wherever possible the majority of the
team should have previous IRRS mission experience”. For follow-up missions “it is preferable to
include the reviewers who participated in the initial mission”.
Recruitment conditions of team members are described in full details in Section 7.6.2. of the
Guidelines.
Issues, developments and proposals
a) Team size
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 7
It is observed that the size of the IRRS teams tend to increase as time passes. This is demonstrated in
Figure 2, where the team-sizes are shown for initial missions to countries with large, medium and
small nuclear programs, respectively. The most recent missions are on the left hand side and earlier
missions follow to the right. The linear trend-lines are fitted by Excel.
It should be noted, that the optimum size of a team depends not only on the nuclear programme of
the host country, but also on the scope of the mission. This dependence is correctly taken into
account by a method reported in Ref [3], the increasing tendency remains also after that.
Figure 2: Team-sizes in past initial IRRS missions
This increase should not necessarily be judged negative; it may as well be due to a more concise
conduct of the mission or due to the experience collected from past missions.
In this subject the TCs formulated the following suggestions:
1.07: The usual number of experts in Module 11 exceeds the associated risk
1.14: There should be more flexibility regarding the size of the team in order to allow the team leader to
add additional resources where needed
b) Team assignment
The TCs suggested a standard team assignment scheme, displayed in Appendix II, Table 1, which
serves as a guidance in recruitment of recent missions
A further advantage of a Standard Assignment Scheme is that the reviewers are expected to work in
groups in order to be as effective as possible. Footnotes below the table clarify details of the
assignments. The numbers of experts in the groups in Table 1 were elaborated based on past
experience and common sense and have been tested in recent missions. These numbers are
indicative and averages, they may be adjusted/regrouped according to the specific needs of the
missions.
Further suggestions by the TCs on composition of teams are as below:
3.03: Two reviewers should work together when interviewing and assessing results of the Module 4 review
5.11: Whenever possible a jointly co-ordinated approach should be adopted for assessment of IRRS
modules with potential overlap. In particular consideration should be given to having shared membership
for modules 5 & 6
6.09: It may be an advantage that experts familiar with more than one type of facility/activity are members
of the IRRS team
An alternative assignment table assuming four types of host countries is presented in Table 2 of
Appendix II.
c) Team recruitment
As the number of IRRS missions to be conducted increases from year to year, recruitment of suitable
team members becomes more and more difficult. To alleviate this difficulty the TCs and the analysis
reports proposed the establishment of expert pools. The respective TC suggestions are:
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 8
1.17: There should be a pool of designated experts that are competent and available for IRRS missions
1.18: The qualification and performance of pool experts should be tracked. Every country delegating an
expert to the pool should formally assure that the expert is capable and has the needed skills on report
drafting as well as the technical expertise. Another source of information on each expert’s performance
should be the survey feedback provided by the Team Leaders
Report [2] proposes:
P14: Countries are encouraged to form a pool of experts in the IRRS process and regularly update the IAEA
with their details. Countries should ensure their named experts periodically participate in IAEA reviewer
training and maintain a good understanding of the IAEA safety standards
Note that the Agency, based on an Agreement with the European Commission maintains a pool of
European IRRS experts and uses it for recruiting European reviewers.
d) Experience and training of reviewers
The left hand side of Figure 3 shows the ratio of experienced reviewers in past initial missions to
nuclear countries. It is seen that the 50% ratio required by the Guidelines is seldom exceeded. On
the right hand side of the Figure the same ratio is shown for the follow-up missions. Also in some of
these missions it was not possible to ensure participation of optimum number of experienced
reviewers, although they always were in majority.
Figure 3: Ratio of experienced reviewers in IRRS missions
In this respect the TCs and the reports suggested:
1.15: The number of experts in an IRRS mission that have no previous IRRS experience should be limited
1.16: Experts in an IRRS mission that have no previous IRRS experience should not work alone on a topic
C26: When recruiting the IRRS team, more attention should be paid to the inclusion of sufficient number of
experienced reviewers
In the evaluation of effectiveness and efficiency of IRRS missions, there is a Performance Indicator
related to the ratio of experienced reviewers. The optimum range of this Indicator is between 0.5
and 0.66 for initial missions and between 0.66 and 1.0 for follow-up missions.
Organization of a Basic IRRS Training Course was proposed by the TCs and was developed and held
first in 2013, then in 2014 with the participation of more than 120 experts. At the request of the
nuclear regulatory bodies on-site training courses were held in the UK and in the USA. Several
experts having obtained training on the courses have since been invited to missions.
Refresher training material has been developed and refresher training sessions were organized prior
to the initial team meetings in recent missions. In other missions, this material was posted on the
mission’s web site and reviewers were asked to make use of it to save time during the initial team
meetings.
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 9
Question(s) to the consultancy
Q-1.12: Do you have any suggestion in general on the size and composition of an IRRS team?
Q-1.13: Which of the Tables in Appendix II (if any) do you consider more suitable? Do you have any
different suggestion on the team assignment (including grouping) for various IRRS mission types?
Q-1.14: What is your suggestion concerning the optimum ratio of experienced reviewers in a team?
Q-1.15: Do you have any specific suggestion on the recruitment technique to be followed by the
Agency?
Q-1.16: Do you have any suggestion concerning training of IRRS reviewers?
I.5 Expected preparatory activity of the team
Status
The IRRS Guidelines requires that the team members provide a short written summary of their
observation based on the ARM (also called “initial impressions”) and present it at the initial team
meeting (Sections 7.7., 8.2., 8.4.). Separately from this, it is required (Sec. 7.5.) that the IRRS
reviewer provides written feedback on any significant issues identified at least two week prior to the
mission.
The activity of the team lead prior to the mission is also discussed by the Guidelines in Sections 7.2.
and 7.8., mentioning mainly the Team Leader and the IRRS Coordinator.
Issues, developments and proposals
In recent missions it is a standard practice that the Team Leader and the IRRS Coordinator request
the reviewers to submit their initial impressions and list of issues at least two weeks prior to the
missions. It is also suggested that the reviewers record their questions stemming from the study of
ARM prior to the mission and hand them over to their counterparts at their first encounter during
the mission, thus making the discussions and interviews more effective. Note on the other hand,
that direct contact among reviewers and counterparts prior to the mission is to be avoided.
In report [2] the following is proposed:
P19: The IRRS Team’s initial impressions of the Host Country self-assessment and action plan should be
circulated to reviewers and the host immediately pre-mission and should form an annex to the IRRS report.
P22: Having read the Host Country’s ARM, including the self-assessment, IRRS Team members should pre-
draft their sections of the IRRS report, using an improved IRRS report template to facilitate the process and
ensure it is consistent. These ‘initial impressions’ should be shared with other Team members (and the
host) at least one week before the mission, accompanied by a list of any issues of concern, where specific
interviews and observations appear to be necessary. At the initial team meeting, immediately before the
IRRS mission begins, a collated first impressions report would effectively inform the team’s discussions
Question(s) to the consultancy
Q-1.17 What would you suggest to require from the reviewers as preparation for a mission?
Q-1.18: Do you have any suggestion on the activity of the team lead prior to a mission?
Q-1.19: What would be the most effective way of applying the initial impressions by the reviewers?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 10
II. Conduct of a Mission
II.1 Schedule of a mission
Status
The IRRS Guidelines does not contain specific guidance on how to schedule the various activities in a
mission.
Issues, developments and proposals
The TCs resulted in a Standard IRRS Mission Schedule as presented in Appendix III, which has been
tested and proven to be working well, with some flexibility, in recent missions. A striking feature of
this schedule is that the reviewers need to submit their preliminary findings by the end of the third
day. Some reviewers argue that the time available for interviews and/or external or internal
discussions is not sufficient; others are convinced that with effective preparations and use of time
two and a half days must be sufficient for the interviews and discussions. In addition, it must be
taken into account that some reviewers participate in site visits during this period of time.
Note also that if the deadline of submission of first findings is postponed, so must be the deadline
for the draft report text, which may also affect the programme of the week-end.
Question(s) to the consultancy
Q-2.1 Do you have any suggestion on modifying the Standard IRRS Mission Schedule? Please explain!
II.2 Observations
Status
The definitions of the observations (i.e. recommendation, suggestion and good practice), applied in
the present IRRS practice, are given in Section 9.1 of the Guidelines. The format of the observations
is not fixed by the Guidelines. The host country Action Plan is discussed at several places in the
Guidelines, yet its relation to the mission findings is not mentioned.
Issues, developments and proposals
a) Formulation
Practical application of the definitions raises concerns from time to time. The TCs offered the
following suggestions:
1.31: R,S & GP should be adequately justified in the text and a general conclusion set out even when there
are no R,S or GPs. The text should be based on fact and evidence (not assumptions) and the conclusions
justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive
explanation but should include some examples.
1.32: The IAEA Guidance should pick out and present the text supporting R, S & GPS from some good
examples (from various modules) of previous reports (I.e. to illustrate style, layout and justification).
3.08: In the formulation of the mission findings the team leaders should consider the application of the
principle of graded approach
In order to partly respond to the suggestions above the tabular structure of observations (originally
containing the basis (or bases) and the recommendation, suggestion or good practice) was
supplemented by an entry entitled “Observation” meant to briefly summarize the issue that leads to
the given observation.
Report [2] recently raised a further possible modification:
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 11
P28: The context and reasons for each recommendation or suggestion should be clearly set out in the
findings (body) text and summarised in the ‘observations summary’ field. To facilitate this is it proposed
that the fields of the IRRS Report observations box be changed.
Currently, this box includes three fields: one for the ‘observation’, one for the basis (or bases) and one for
the recommendations or suggestions. It is proposed that the observations field be replaced with two fields;
one briefly describing the current practice and another describing the shortcoming relative to the relevant
IAEA requirement
The current format of the observation box is seen in Figure 4:
Figure 4: Format of the observations in the report
b) Good practice
Good practice is the most controversial observation of the IRRS. On one hand, the definition of a
good practice is very strict and should allow for only a few of them in several missions, on the other
hand, experience shows that the reviewers tend to use it for acknowledging good performance of
the host otherwise not at all unique or outstanding.
The TCs have already observed that
3.09: It is important to highlight in the mission report also good achievements which do not formally meet
the definition of a Good Practice.
5.02: Some criteria be introduced for appreciating strong features in regulatory performance (not GP, but
good)
Report [2] also emphasizes the controversy of the present use of good practices and suggests:
P29: IAEA should consider the following:
− Require that all four elements of the definition of a good practice be comprehensively addressed in
the IRRS report when describing a good practice. Unless all four criteria are clearly met, the concept of
good practice should not be applicable;
− Define and encourage the reporting of a second level of practice (e.g. ‘Notable Practice’) which
acknowledges actions and processes notable in the circumstances of the Host Country but which do
not meet IRRS Guidelines criteria for good practice.
P30: IAEA should consider that a good practice should be documented only where it has not been seen
before. This suggests the development of an IAEA searchable database of good practice, easily accessible to
IRRS reviewers and the global radiation safety community. MS should be encouraged to report to the
database if (and how) they adopt the good practices they find there
The TM of team leaders suggested [5]:
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 12
To ensure that category of good practice is appropriately used, another category of good performance
should be created
In reply to these suggestions the Agency developed the definition of a Notable Performance:
Where a commendable activity is observed which clearly strengthens regulatory effectiveness and/or
organisational efficiency (but, unlike a good practice, is not definably ‘superior to those generally
observed elsewhere’) it may be referred to as ‘notable performance (NP)’. NP will be worthy of the
attention of other regulatory bodies as a model in the general drive for excellence.
c) Observations and the Action Plan
According to the present practice the fact that an issue is recognised by the host country and is
inserted into the Action Plan (AP), may not preclude the reviewers from offering respective
recommendations or suggestions. In fact on the contrary, whenever a noncompliance is recognised
by the reviewer (notwithstanding if the AP called it to her/his attention) it should call forth the
formulation of a recommendation.
Certain reviewers do not agree with the present practice and argue that there is no added benefit in
recommending something to the hosts when it has already decided to take action. Although this
reasoning seems to be logical, the following arguments stand against it: if the presence of an issue in
the Action Plan precluded its appearance as a finding then
• the basic definition of recommendations (“they are proposed where aspects relative to the
IAEA Safety Requirements are missing, incomplete or inadequately implemented”) is not
followed;
• the follow-up mission will not be in the position to review the progress reached in eliminating
issues, as the relevant recommendations were not formulated in the initial mission;
• it would be sufficient for the host to have a general AP item like e.g. “The Inspection system
needs a thorough and complete revision” to preclude any IRRS mission finding on inspection.
It is important also to note here that even for a same issue, the Action Plan items and the
corresponding recommendation or suggestion are usually different in their details.
The related proposals in report [2] are as follows:
P20: To the extent possible, the IRRS report should clearly document the validation (or otherwise) of the
host’s self-assessment as a preface to the reviewer’s findings. Where the country has properly identified
the issues through self-assessment, this should be acknowledged. Where the IRRS team discovers issues
not addressed in the self-assessment, this should also be stated in the IRRS report, since such issues will
clearly be new to the host country
P25: To the extent possible, recommendations and suggestions should be correlated with the Host
Country’s existing action plan and refined, as appropriate to be realistically in accordance with the action
plan
d) Correlations and balance of observations
Each IRRS mission report is headed by the disclaimer: “The number of recommendations,
suggestions and good practices is in no way a measure of the status of the regulatory body.
Comparisons of such numbers between IRRS reports from different countries should not be
attempted”. This, however, does not mean that the ratios and correlations of these numbers may
have no meaningful information on the particular missions. This issue is discussed here.
In reports [1], [3] and [4] correlations among the number of various observations were examined
and the following rules were established:
• The number of Recommendations and the number of Suggestions are expected to exhibit
similar statistical behaviour, i.e. whenever the number of Recommendations is high (in
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 13
missions as well as for specific topics), so is the number of Suggestions and vice versa.
Statistical analysis has shown that indeed, they are strongly positively correlated both mission-
by-mission and over the various subjects. Figure 5 illustrates this for various subjects (so called
Subject Groups are formed within each Module and the number of findings belonging to those
Subject Groups appear in the Figure). Nevertheless, in certain missions the expected positive
correlation is less pronounced.
• The number of Recommendations and the number of Good Practices are expected to change
in opposite way by missions, i.e. in missions where the number of Recommendations is high
only few Good Practices are expected and the other way around. According to the results of
the analyses this negative correlation is present in most missions, yet there are a few
particular missions where such correlation does not exist.
• Similar correlation is expected between the number of Suggestions and the number of Good
Practices as in the case of Recommendations and Good Practices, yet in many missions this
negative correlation is missing (c.f. Figure 6).
The reports [1], [3], and [4] offer reasonable explanations for the lack of the expected correlations:
C17: The lack of distinct positive mission-wise correlations between the numbers of Suggestions and of
Recommendations may be attributed to the observations that in some missions Recommendations are
converted to Suggestions as results of compromises between the team and the hosts; whereas in other
cases (especially when there are a number of serious findings) the team tends to offer Recommendations
although a Suggestion might be more adequate
C18: - The numbers of Suggestions and Good Practices are positively correlated over IRRS Modules as well
as over Subject Groups since there possibly are topics that are particularly important in the regulatory
practice (or are consistently pointed out by the missions);
- another reason (more psychological than technical) may be that reviewers formulating a number
of findings unintentionally feel obliged to compensate by also offering Good Practices
Figure 5: Number of Recommendations and Suggestion on various subjects
Figure 6: Number of Suggestions and Good Practices in various missions
0
5
10
15
20
25
30
35
40
45
a b c d e f g x y a b x y a b c d e f g h x y a b c d e f x y a b c d e f x y a b c d e f g h i x y a b c d e f g h x y a b c d x y a b c d e x y a b c d e x y
Findings/Subject Groups in all missions to NPP countries in 2006-2013
Recommendations Suggestions
Mod.3 Mod.4Mod.1 Mod.7 Mod.8 Mod.9 Mod.102 Mod.5 Mod.6
0
5
10
15
20
25
30
35
O E S R J G T F H U P I N Q L V A M C D K B
Nu
mb
er
of
fin
din
gs
Mission
Correlation of Suggestions and Good Practices
S GP
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 14
Based on the expected and partly experienced correlations discussed above two Performance
Indicators characterizing the efficiency and/or effectiveness of a mission were proposed [3]:
(1) Balance of Recommendations and Suggestions (essentially expressed through the ratio of the
respective numbers)
(2) Balance of Good Practices and Recommendations (essentially expressed through the product of
the respective numbers)
Figure 7: Balance of findings in missions Figure 8: Balance of Recommendations and Good
Practices in missions
Figures 7 and 8 show the relative deviations of these balance values from their optimum value (1.0)
in the past missions. The ranges between the green lines show the cases where the deviation from
the optimum value is less than 25%. It is seen from the figures that in most missions the deviation
from optimum is reasonable, yet there are missions where very likely an in-depth analysis might
reveal the reasons for the differences.
Question(s) to the consultancy
Q-2.2: Would you recommend changes in definitions of the presently existing observations, or you
are satisfied with it as of now?
Q-2.3: Do you agree with the introduction of a new type of observation: Notable Performance? If so,
do you agree with the definition given here or you suggest changes in it?
Q-2.4: What is your view on handling issues present in the AP when formulating potential findings?
Q-2.5: Would you consider making use of the statistical (correlational) properties of the
observations? If so, how would you suggest applying them?
II.3 Review actions
Status
The IRRS Guidelines describe the mission activity of reviewers in Chapter 8. The Chapter includes the
use of various review methods (reading documents, interviews, observations, visits). There is no
specific guidance on how to distribute the tasks among reviewers working in groups, neither on how
to co-operate among groups.
Issues, developments and proposals
a) Interviews
Interviews represent the most important way of collecting information, performing peer review. The
practical question arising relates to the number of persons sitting at the two sides of the table. Very
often at least two members of the team group listen to the counterpart(s). If the host
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 15
representatives are considerably less or considerably more than the team members the
effectiveness of the interview may be jeopardized.
b) Site visits
Site visit are usually very time consuming and reviewers participating in visits have considerably less
time for interviews and formulation of findings than other reviewers have. Furthermore, many of the
inspections held during visits are formal, are set-ups or are difficult to follow (usually for language
reasons).
c) Co-operation of reviewers
Table 3 shows the usual way of cooperation among the review groups. This Cross-Contribution Table
is part of the standard mission report template.
Gr. Topics to review, write and provide to Gr. Topics to obtain from Gr.
A
• Governmental responsibilities and
functions (mod 1)
• Global Nuclear Safety Regime (mod 2)
• Interface with security (mod 12)
-
-
-
• Module-wise legal
background
C,D,E,F
B
• Responsibilities of the RB (mod 3)
• Management system (mod 4)
• Development of regulations and guides –
in general (mod 9)
-
-
-
• Contents of regulations and
guides
C,D,E,F
C
• Authorization – in general and for NPPs
(mod 5)
• Review & assessment (R&A) – in general
and for NPPs (mod 6)
• Legal background of authorization and
R&A
• Contents of regulations and guides
related to authorization and R&A
-
-
A
B
• Authorization of facilities and
activities other than NPP
• R&A of facilities and activities
other than NPP
E
E
D
• Inspection – in general and for NPPs
(mod 7)
• Enforcement – in general and for NPPs
(mod 8)
• Legal background of inspection and
enforcement
• Contents of regulations and guides
related to inspection & enforcement
-
-
A
B
• Inspection of facilities and
activities other than NPP
• Enforcement of facilities and
activities other than NPP
E
E
E
For facilities and activities (other than NPP)
• Legal background
• Authorization and review & assessment
• Inspection and enforcement
• Contents of regulations and guides
A
C
D
B
F
• Emergency preparedness and response
(EPR) (mod 10)
• Legal background of EPR
• Contents of regulations and guides
related to EPR
-
A
B
G • Other optional thematic area -
Table 3: Standard IRRS Cross-Contribution Table
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 16
The first column shows the Group identifier (A through G). In the second column the activities
(review areas) of the groups are given, those areas which shall be contributions (inputs) to other
groups are shown in red. In the third column the addressee groups of the contributions (red items)
are given. The fourth column gives in blue those parts that are received by the current group from
other groups as contributions (inputs), while the last column contains the groups from where the
fourth column contributions are expected.
Question(s) to the consultancy
Q-2.6: Do you have any suggestion on how to make the interviews even more effective?
Q-2.7: What would you suggest for making visits even more efficient and effective?
Q-2.8: What would you suggest for further enhancement of an effective cooperation within the
review team?
Q-2.9: are you happy with using “groups’ or is it easier to just use names of reviewers
II.4 Report writing
Status
The IRRS Guidelines contains no specific requirements on the content of a mission report, yet
Appendix I of the Guidelines summarizes those topics that belong to the various Modules and thus
that need to appear in the final mission report. Nothing is said in the Guidelines on the timing, style
or method of writing the mission report neither on the expected size of a report.
Issues, developments and proposals
a) IRRS Mission Report Template
The TCs realized the need for an extended table of contents, template and guidance for the
preparation of a mission report and elaborated such a template. It has been developed further,
based on the experience from its application. The actual form of the IRRS Report Template (version
6) is available from the Share Point area referred to in footnote 1.
Further suggestions on the template are given in report [2]:
P31: The IRRS report template should explicitly ensure that every element of each requirement is
addressed in full, even indicating clearly if elements are not applicable. For each element of each IAEA
requirement clear statements of compliance should be included for all regulated facilities and activities
encompassed by the national framework for safety. All parts of any requirement validated by the Team
should be listed or as a minimum, the number of each requirement and its sub-paragraphs should be listed,
clearly indicating compliance or otherwise.
In addition to the above, it is proposed that at the beginning of any review discussion, the applicable
elements of each requirement should be agreed and noted by reviewer and counterpart.
b) Details and length of a report
The needs for completeness and for readability of a report always present antagonism. Thus details
still to be included always compete with the wish to keep it as short as possible. The ‘Observation’
line in the observation box (see in Fig. 4) was introduced for providing a brief description of the issue
and possibly avoiding lengthy and repeated explanations in the main body of the report.
Report [2] proposes some kind of repetition, as already quoted in Section II.2:
P28: The context and reasons for each recommendation or suggestion should be clearly set out in the
findings (body) text and summarised in the ‘observations summary’ field. …
In report [4] analysis of the length of reports of past missions is presented. Based on past values and
on the assumption that past reports were overwritten by about 10%, optimum report lengths were
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 17
determined for each mission. This length depends on the scope of the mission, size of the host
country’s programme and the number of observations in the mission. Optimum lengths of the
chapters in the report are also defined and included into the mission report template. The lengths of
the reports of past missions relative to their optimum values are shown in Figure 9.
Figure 9: Mission report lengths relative to their optimum values
Question(s) to the consultancy
Q-2.10: What do you think necessary for developing further the IRRS Report Template?
Q-2.11: What would be the right way of describing the context and reasons for a finding?
Q-2.12: Do you agree that the lengths of mission reports should be kept at minimum?
Q-2.13: What would you suggest as guiding principles for preparation of mission reports?
II.5 Activity of team lead
Status
Responsibilities of the IRRS team participants are described in Appendix IV of the IRRS Guidelines.
Roles and responsibilities of the Team Leader and of the IAEA Coordinator are given in sufficient
details; those of their deputies are much less elaborated.
Issues, developments and proposals
The TCs concluded in the following suggestions:
1.26: The team leader should not be assigned a function in reviewing a module
1.27: On missions to countries with large nuclear programs the DTL should not have a formal function in
reviewing a module. In smaller missions he can support sub team A
These suggestions are being considered in recent missions.
Report [2] proposes:
P15: The IAEA Deputy Coordinator’s role in explaining and ensuring correct use of IAEA requirements
should be reinforced, with training as necessary. The IRRS Guidelines should clearly assign to the Deputy
Coordinator, responsibility for alerting IRRS reviewers and counterparts where misunderstanding /
misinterpretation of a requirement is evident
P17: The IAEA Deputy Coordinator (DC) should be assigned responsibility for gathering feedback on the
application of IAEA safety standards and formally reporting it to IAEA. Special attention should be paid to
requirements that appear to be interpreted variably.
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
UK
FR
A
JPN
ME
X
SP
A
GF
R
UK
R
CA
N
RU
S
FR
A-f
u
UK
-fu
CP
R
US
UK
R-f
u
SP
A-f
u
GFR
-fu
CA
N-f
u
RO
M
RO
K
SLO
SW
I
SW
E
SLR FIN
BU
L
UK
-fu
2
RU
S-fu
CZ
R
BE
L
US
-fu
Nu
mb
er o
f p
age
s re
lati
ve
to
op
tim
um
Mission
Change of relative report length
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 18
P34: The Deputy Team Leader with the Deputy Coordinator should ensure that all observations relevant to
non-compliance with IAEA Safety Requirements lead to the appropriate recommendation and basis
Question(s) to the consultancy
Q-2.14: Do you have any suggestion on clarification/modification/extension of roles and
responsibilities of an IRRS Mission team lead?
II.6 Policy issues discussions
Status
Policy issues discussions are introduced and treated in Section 8.6 of the IRRS Guidelines. In
Appendix III a detailed list of the issues suggested for policy discussions and the key elements
therein are given.
Issues, developments and proposals
Policy issues discussed in the IRRS missions in 2006-2013 were collected and analysed. Figure 10
shows the frequency of the various issues in these discussions. The subjects highlighted in the table
were novel suggestions by the hosts, the other subjects are from among those listed in Appendix III
of the Guidelines.
Figure 10: Frequency of policy issues discussed in missions in 2006-2013
Question(s) to the consultancy
Q-2.15: Do you deem it necessary to modify the concept and/or the practice of policy issues
discussions so far followed? If so, what would you suggest?
II.7 Feedback from the missions
Status
The IRRS Guidelines do not mention feedback on the missions. It mentions, on the other hand, at
several places that the mission may provide effective feedback for the improvement of IAEA
Standards and Guides.
Issues, developments and proposals
The TCs suggested that
1.44: Continue the initiative to use surveys of IRRS Team members and the host countries to provide
feedback on the effectiveness of the IRRS mission
2.02: IAEA should develop a consistent approach to collect and evaluate feedback from IRRS host countries
and participants on Missions to learn lessons
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 19
2.34: The Consultants recommend IRRS participants give structured feedback on the conduct and
effectiveness of the mission. The Consultants endorse IAEA proposals for developing a process for
collecting and evaluating this information. This is to promote the continuing improvement to the IRRS
process
In report [2] P17 (as quoted in Section II.5) proposes that the DC should gather feedback on the
application of IAEA safety standards.
In report [3] four types of feedback questionnaires were suggested. These will be discussed below.
The questionnaires contain questions that are to be rated by marks between 1 and 5 (5 is the most
favourable), and at every question the answerers have the possibility of offering further comments
or suggestions.
Results of the evaluation of the questionnaires for recent missions are given in report [3]
a) Feedback on the ARM by the team
The team members are requested to evaluate the quality of the ARM by answering the questions:
Qr1: How complete do you consider the ARM?
Qr2: How realistic is the picture which you were able to obtain on the area you
will be reviewing from the ARM?
Qr3: What is your overall evaluation on the quality of the ARM?
b) Feedback on the mission by the team
At the end of a mission every member of the team is requested to answer a number of questions
characterising the effectiveness of the participants and of the mission.
c) Feedback on the mission by the host
Representatives of the host are requested to answer the questions below at the end of the mission:
Qh1: How effective do you consider the mission in assisting the continuous
improvement of nuclear safety in the host country?
Qh2: How objective was the peer review?
Qh3: How has the mission helped the exchange of information, experience and
good practice with other countries?
Qh4: How consistent was the use of the IAEA safety requirements and guides in the
mission?
Qh5: How justified are the findings of the peer review?
Qh6: How relevant are the findings of the peer review for the future development
of your regulatory body?
Qh7: How competent were the reviewers in their reviews and findings?
d) Feedback on the initial mission by the host before the follow-up
Representatives of the host are requested to answer two questions for every findings of the initial
mission prior to the beginning of the follow-up mission:
Qf1: How accurate was the particular recommendation/suggestion?
Qf2: How helpful was the given recommendation/suggestion for the improvement
of the regulatory body?
e) Feedback on the IAEA safety standards
No systematic collection of such feedback has so far been conducted.
Question(s) to the consultancy
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 20
Q-2.16: Do you suggest any modification in the question sets in use?
Q-2.17: What questions would you ask from the team to feedback on the mission?
Q-2.18: How would you suggest introducing practical feedback on IAEA safety standards?
II.8 Evaluation of team members’ performance
Status
The IRRS Guidelines does not address performance evaluation of reviewers.
Issues, developments and proposals
Evaluation of the performance of reviewers is primarily important for ensuring the highest possible
quality and expertise of future IRRS teams. The sensitive nature of this type of evaluations is well
understood, therefore introduction of such procedure shall be thoroughly thought over and, if so
decided, the rules need to be meticulously set. It is assumed that the results of evaluation are kept
strictly confidential by the Agency and are not disclosed in any event.
The TM of team leaders [5] suggested:
Team Leader should give the IAEA an evaluation on the performance of individual Team Members at the
end of the mission
Question(s) to the consultancy
Q-2.19: Do you agree or disagree in principle with the evaluation of reviewers’ performance? Please
explain!
Q-2.20: If you agree in principle, what would be the necessary and sufficient conditions of such a
procedure to introduce?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 21
III. Post-mission Phase
III.1 Finalization of the mission report
Status
The IRRS Guidelines sets the scene for the preliminary and final reports in Sections 9.5. and 9.6.
Accordingly the preliminary report (the version left back by the IRRS team) is commented by the
host, but the comments should be limited to issues relating to factual correctness of information
contained in the report. The comments are assessed by the IAEA C., the TL and, if necessary, by
other team members. The final report is to be released within two months following receipt of host
comments.
Issues, developments and proposals
Report completion times of past missions are shown in Figure 11
Figure 11: Time (in days) necessary for releasing mission reports
It is seen that the 60 days deadlines was only a few times achieved and in several cases the release
of the reports took more than 120 days.
Report [1] concludes:
C31: The IAEA should investigate possible ways of speeding up report completion
Question(s) to the consultancy
Q-3.1: Do you suggest any modification in the process of finalizing mission reports?
Q-3.2: What would you suggest for decreasing the time needed for releasing the report?
III.2 Evaluation of missions
Status
The IRRS Guidelines does not refer to evaluation of IRRS missions.
Issues, developments and proposals
Based on the Performance Indicators introduced in report [3] and on the feedback comments and
suggestions discussed in Section II.7 above, Prompt Evaluation Reports were prepared on missions
conducted to countries with operating NPPs. In terms of Performance Indicators the efficiency and
effectiveness of the missions are evaluated, discussed and conclusions are drawn. Examples of such
reports are available from the Share Point area referred to in footnote 1. Cover pages of the reports
prepared during the so called pilot-phase are shown in Figure 12.
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 22
Figure 12: Prompt Evaluation Reports of recent IRRS missions
Question(s) to the consultancy
Q-3.3: Do you consider important evaluating the IRRS missions?
Q-3.4: What would you suggest for improving further the evaluation methodology?
Q-3.5: Can you suggest additional items to be included in this evaluation?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 23
IV. Follow-up missions
IV.1 Follow-up as a compulsory part
Status
The IRRS Guidelines does not explicitly state that a follow-up mission is a necessary consequence of
an initial IRRS mission. The implicit statement in the Introduction says: “An IRRS mission addresses
all relevant areas, facilities and activities regulated in the state, with a follow-up IRRS mission no
more than four years later to review progress…”.
Issues, developments and proposals
Some countries that invited an initial IRRS missions failed to invite a follow-up within a reasonable
time-frame.
The TCs suggest:
4.22: Follow up mission should be made a compulsory part of the mission. The IAEA should not conduct a
mission unless there is a clear statement from the host country to host a follow up mission
A similar suggestion was formulated by the TM of team leaders [5]
Follow-up mission in a given time frame should be compulsory part of the IRRS process
Question(s) to the consultancy
Q-4.1: How should IRRS Guidelines be modified (if at all) to reflect that the follow-up mission be part
of the package when the initial request for an IRRS mission is made?
Q-4.2: What other means do you consider effective in stimulating host countries to invite follow-up
missions?
Q-4.3: Would you agree requesting the countries hosting initial and follow-up missions to submit a
final report on how all recommendations and suggestion of the missions have been implemented
(“closing the IRRS loop”)?
IV.2 Timing of follow-up
Status
The IRRS Guidelines mentions timing of a follow-up mission in the Introduction as quoted above (“no
more than four years later”), and in Section 10.2: “Typically a follow-up mission will take place two
to four years following the initial IRRS”
Issues, developments and proposals
Some countries that invited an initial IRRS
missions failed to invite a follow-up within a
reasonable time, on the other hand, those
countries which have invited follow-up
missions mostly did it within the time-frame
foreseen by the Guidelines. This is show in
Figure 13.
Figure 13: Time elapsed beweet the initial
an follow-up missions
There is, however, another aspect that may influence the optimum timing of the follow-up missions;
this is the success in closing the issues raised by the initial mission. Figure 14 shows the ratios of
issues remained open in the follow-up as per the number of issues in the initial mission. The average
of this ratio is 0.23, i.e. on average 23% of the issues remain open. Accordingly, report [1] concludes:
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 24
C14: The relatively high number of issues that remained open suggests that either the time between the
initial and follow-up missions were sometimes not sufficient to reach compliance with the findings of the
initial missions, or the host country did not or could not place equal emphasis on all improvements
Figure 14: Ratio of open issues in the follow-up missions
There are views that express concern on the functioning of the follow-up system based on the
relatively low number of follow-up missions so far. Report [2] expresses this as
P2: In order to identify why IRRS Guidelines on follow-up missions are not being followed, the IAEA should
consider an anonymous survey of those Member States having hosted IRRS, but not yet requested a follow
up within the four years suggested in the Guidelines
A fast overview of the missions since 2006 shows, that there are 16 countries that had initial
missions but have not yet invited a follow-up. Two of them are nuclear countries two others are
having minor nuclear programmes and 11 are from the Africa region. The initial missions to these
latter countries had very limited scopes and therefore the potential follow-up missions to these
countries need specific considerations.
Question(s) to the consultancy
Q-4.4: Do you see the necessity of specific steps related to the timing of follow-up missions? If so,
what would you suggest?
Q-4.5: Do you consider it necessary to urge MSs to invite follow-up missions?
IV.3 Follow-up team composition
Status
This question is treated in Section 10.3.1. of the IRRS Guidelines as “it is preferable that the follow-
up mission includes the TL TC and reviewers who participated in the initial mission”.
Issues, developments and proposals
In Section I.4.d) above the experience of the reviewers in past missions was discussed. Figure 3 there
shows that it was not always possible to recruit the same reviewers for a follow-up mission who
participated in the initial one. (In fact it shows that sometimes even it was not possible to recruit
experienced reviewers for certain positions of a follow up.)
The TCs have also recommended that
4.23: The reviewers of the follow-up mission be the same as in the initial mission as much as possible
Question(s) to the consultancy
Q-4.6: Do you see the necessity of specific rules or measures on recruiting a follow-up team? If so,
what would you suggest?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 25
IV.4 Extended follow-up missions
Status
The IRRS Guidelines defines a follow-up extended if it also includes the review of specific topical
areas not covered in the initial mission. The Guidelines does not contain any preference or dis-
preference concerning extended follow-up missions.
Issues, developments and proposals
The TCs considered this issue and suggested:
7.34: It was the consensus of the consultancy that full scope IRRS missions should be promoted and
practice of adding modules to the follow-up missions (modules which were not covered during the main
mission) should be avoided to the extent possible
Question(s) to the consultancy
Q-4.7: What is your suggestion concerning possible extension of future follow-up missions?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 26
V. Development of the IRRS Process
V.1 Scope of an IRRS mission
Status
The IRRS Guidelines discusses the scope of a mission in Chapter 5 as:
“The minimum scope of an IRRS mission includes the core modules, i.e., Modules 1 to 10, together
with policy issues discussions. In order to be called an IRRS mission, the mission scope should always
cover the minimum scope.
When requesting an IRRS the state is strongly encouraged to include all regulated facilities and
activities in the mission. Without prejudice to the previous paragraph, if the state wishes to exclude
specific facilities and activities, a detailed explanation for the exclusion of these facilities or activities
should be provided. The mission report will reflect this in both the title and content of the relevant
report sections, including the Executive Summary”
“Additional technical issues may be included in the scope of the IRRS. These are incorporated in
Modules 11 (Additional Areas) and 12 (Interfaces with Nuclear Security).”
No definitions of a full-scope or a reduced-scope mission are given in the Guidelines.
Issues, developments and proposals
According to some views the integrated character of IRRS must be reflected by covering the full
extent of the national legal framework and regulatory infrastructure for nuclear and radiation safety.
In this sense report [2] proposes:
P7: To be defined as an integrated mission, IRRS should review the full extent of the State’s national, legal
and governmental framework and regulatory infrastructure for safety, directly involving all organisations
that comprise the national regulatory body for nuclear and radiation safety and have oversight of all
radiation facilities and activities in the country. As per the IRRS Guidelines, the scope of the IRRS should be
clearly explained to the requesting Government and fully agreed at the IRRS preparatory meeting.
The IAEA should consider revising the modular structure of IRRS, by making the currently optional Module
11 a ‘core’ module or explicitly incorporating all elements of Module 11 into the core modules to ensure
the full extent of the Host’s national framework for safety is addressed
P8: Where the Host Country requests an integrated review as proposed above, then all agencies
collectively comprising the national regulatory body and thereby assigned responsibility and providing the
full scope of regulatory responsibilities and functions (as defined in GSR Part-1) should be subject to the
review, with their various identities and roles confirmed at the preparatory meeting
P9: When requesting a peer review which would not be integrated to the extent proposed above, the Host
Country should formally notify the IAEA of its specific requirements so that a more appropriate alternative
mission can be offered
Question(s) to the consultancy
Q-5.1: Do you consider that a peer-review mission could be called an IRRS mission only if it reviews
the full extent of legal, governmental, and regulatory framework related to nuclear and radiation
safety, or do you favour the present practice (full-scope and limited-scope IRRS missions)?
Q-5.2: Would you recommend defining specific IRRS mission types differing in their scope? If so, what
would you suggest?
V.2 Possible changes in the IRRS Module structure
Status
The IRRS module structure is described in Chapter 5, Section 5.1 and Appendix I of the Guidelines.
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 27
Issues, developments and proposals
Most of the core Modules in the IRRS process cover
approximately equal areas of the national or
regulatory framework, some of them, however, are
substantially smaller. Figure 15 illustrates this
through the example of the number of observations
in the various Modules from all missions to nuclear
countries in 2006-2013.
It is apparent that Module 2 (Global Nuclear Safety
Regime) and Module 8 (Enforcement) represent
considerably less weight in the review than the other
core Modules
Figure 15: Total number of observations from missions to nuclear countries
This feature was also realized in report [1] that concluded:
C8: Recommendations and Suggestions, relating to most of the core regulatory functions from initial
missions demonstrate equal emphasis by the peer review, while Modules 2 and 8 have a considerable
lower rate of findings. The definition of the IRRS Modules may therefore need revision to be better
balanced out
Question(s) to the consultancy
Q-5.3: Do you consider it worthwhile changing the Module structure of the IRRS process? If so, what
would you suggest?
V.3 Possible changes in the IRRS Module review scope
Status
The scope of the IRRS review is defined by the IAEA safety standard Requirements and is described
in broad terms in Appendix I of the Guidelines.
Issues, developments and proposals
Post-mission round-table discussions often point to certain shortcomings of the review scope. No
systematic investigation of this question has so far been conducted, yet a number of examples are
already worth mentioning:
• the review is often dominated by legal and regulation details while the regulatory practice
does not receive sufficient attention; • a specific review approach may be needed in countries where the regulatory body includes
several independent organizations; • special attention should be given to the top level of those regulatory bodies which are
headed by special commissions, committees or boards with a right of veto; • generally little attention is paid to the inspection techniques applied by the host regulator.
Question(s) to the consultancy
Q-5.4: What are the topics that you would suggest for discussion as potential modifications or new
elements in the review scope?
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 28
References
[1] Lessons Learned from IRRS Missions in 2006-2013 to Countries with Operating NPPs,
http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
[2] IRRS Missions 2006-2013: Analysis from a Radiation Safety Perspective,
http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
[3] Efficiency and Effectiveness of IRRS Missions,
http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
[4] Analysis of the IRRS Missions in 2006-2013 to Countries with Operating NPPs
http://gnssn.iaea.org/regnet/irrs/Pages/IRRS_pub_docs.aspx
[5] Chairmen’s Summary Report on the Technical Meeting of the Integrated Regulatory Review
Service Team Leaders and Deputy Team Leaders for Sharing Experience and Improving the
Implementation of the Service, 28-31 January 2013, Vienna
Appendix I: A typical IRRS preparatory meeting Agenda
Tuesday, 27 May 2014
9:00 – 9:15 Opening Remarks and Introductions RB*, TL*, IAEA
9.15 – 9:30 Preparatory Meeting Objectives IAEA
9:30 – 10:30
General Presentation on legal and regulatory framework in the
host country
- Governmental and legal authority
- Regulatory body structure (organization, staffing,
resources, responsibilities, etc.)
- Overview of facilities and activities regulated
RB
10:30 – 11:00 Coffee Break
11:00 – 12:00 IRRS Process IAEA, TL, DTL*
12:00 – 13:15 Lunch
13:15 – 16:00
IRRS preparation: self-assessment, ARM, choice of interviews
and of activities to be observed
Presentation of RB’s self-assessment process
For each IRRS module:
- Presentation of self-assessment preliminary findings,
- Discussion regarding:
o ARM,
o choice of interviews or visits to be planned
during the IRRS mission.
RB
16:00 – 16:30 Break
16:30 – 18:15
IRRS mission implementation and logistics
a. finalize dates
b1. finalize scope (including discussion of policy issues)
b2. participant roles and responsibilities
c. schedule
1. team selection including access clearances
2. submittal of Self-assessment
3. submittal of reference material (discuss content)
4. submittal of policy issue write – ups
5. initial team meeting
d. communication prior to mission (identify liaison officer)
e. entrance meeting
f. communication during mission (identify counterparts)
g. mission team activities
1. document review
2. RB presentations
3. interviews
4. direct observation/site visits (facilities, organizations,
meetings, emergency exercise, other government officials, etc.)
5. daily team meetings
6. policy issue discussions
h. development of observations, conclusions,
recommendations, suggestions and good practices
i. draft mission report
j. exit meeting
k. communication following mission
RB, TL, DTL,
IAEA
18:15 Adjourn for the day
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 30
Wednesday, 28 May 2014
9:00 – 10:30
Logistical Preparations
a) security – clearances and badging
b) conference space and equipment
c) administrative services
d) lodging and transportation
e) lodging and travel to/from site visit(s)
f) emergency exercise/drills (date(s), duration, venue(s),
participants)
g) initial team meeting (time, location)
h) entrance meeting (time, location, participants)
i) exit meeting (time, location, participants)
j) media relations
k) cultural activities
10:30 – 10:45 Coffee break
10:45 – 12:30 Continue discussions
12:30 – 13:45 Lunch
13:45 – 14:30
Summary of Meeting
a. agreements
b. follow – up items
14:30 Closing remarks
15:00 Meeting adjourned
∗ RB – Regulatory Body
TL – IRRS Team Leader
DTL – IRRS Deputy Team Leader
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 31
Appendix II: Team assignment and grouping
Note
s Team activities
a) Small
b) Medium Large Follow-up
c)
1) 2) Team Lead (TL, DTL, TC, DTC) 3 3 3 3
2)
Group A
Governmental R&F (mod 1)
Global nuclear safety regime (mod 2)
Interface with security (mod 12)
3
2 3
2
3)
Group B
Responsibilities of the regulatory body (mod 3)
Management system (mod 4)
Development of regulations and guides (mod 9)
2 3
3)
4)
Group C
Authorization (mod 5)
Review and assessment (module 6) 2
2 2
2
3)
4)
Group D
Inspection (mod 7)
Enforcement (mod 8)
2 2
3)
5)
6)
Group E – Facilities/Activities
Nuclear power plants
Radiation sources applications
Research reactors
Fuel cycle facilities
Waste management facilities
Decommissioning
Transport of radioactive material
- 2 5
3
3) Group F
Emergency preparedness (mod 10)
2
2 2
6)
Group G – Other additional thematic areas (mod 11)
- control of medical exposures
- control of chronic exposures (radon, NORM
and past practices) and remediation
1 1
Administration 1 1 1 1
Total 11 17 22 11 a)
Numbers of experts include IAEA staff, b)
Definitions:
Small: No NPPs, few Facilities/Activities
Medium: Less than 5 NPP units (in construction, operating, to decommission), few Facilities/Activities
Large: 5 or more NPP units, and/or many Facilities/Activities c)
Numbers of experts of follow-up missions are defined here for large nuclear programmes, in other missions
should be adjusted according to the needs.
1) DTC is to be assigned to a Group
2) DTL supports Group A
3) Groups C through F review the respective contents of the regulations and guides (if in the scope) and
report to Group B
4) Groups C and D review the respective regulatory activities in general and the NPP-related activities in
specific
5) Group F reviews all core regulatory activities (authorization, review and assessment, inspection, enforce-
ment) for the specific facilities and activities in the scope of the mission and report to Groups C and D
6) For small nuclear programs the Facilities and Activities are to be reviewed by Groups C , D, F and G. Experts
allocated to Facilities/Activities (Group E) in medium and large missions may need to work together with
the experts in Group G. For large programs the number of reviewers may be less than 5 according to the
needs of the mission
Table 1: Assignment of the modules to groups and guidance on the number of experts in an IRRS
mission as suggested by the Targeted Consultancies
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 32
Gr Team activities Cntry
Cat I
Cntry
Cat II
Cntry
Cat III
Cntry
Cat IV
Follow-up
of Cat IV
Team Lead (TL, DTL, TC, DTC)
Team Leader 1 1 1 1 1
Deputy Team Leader 0 0 1 1 1
Team Coordinator 1 1 1 1 1
Deputy Team Coordinator 0 0 1 1 1
A Governmental R&F (mod 1) Global nuclear safety regime (mod 2) Interface with security (mod 12)
1 + TL 1 2 2 1
B Responsibilities of the regulatory body (mod 3) Management system (mod 4) Development of regulations and guides (mod 9a)
TL 1 + TL 2 2 1
C Authorization (mod 5) Review and assessment (mod 6) Review of content of relevant regulations (mod 9 b)
C1: Nuclear power plants 0 0 0 2 2 or 3
(profile of
reviewers
selected
based on
R/S)
C2: Radiation source applications 2 2 2 2
C3: Research reactors 0 0 1 1
C4: Fuel cycle facilities 0 0 1 1
C5: Waste management facilities 0 0 1 1
C6: Decommissioning 0 0 1 1
C7: Transport of radioactive Material = C2 1 1 1
D Inspection (mod 7) Enforcement (mod 8) Review of content of relevant regulations (mod 9 b)
D1: Nuclear power plants 0 0 2
= C1 to
C7
D2: Radiation source applications = C2 = C2 = C2 = C2
D3: Research reactors 0 = C3 = C3
D4: Fuel cycle facilities 0 = C4 = C4
D5: Waste management facilities 0 = C5 = C5
D6: Decommissioning 0 = C6 = C6
D7: Transport of radioactive material = C7 = C7 = C7 = C7
F Emergency preparedness (mod 10) 1 1 2 2 1
G Additional thematic areas (mod 11)
G1: Control of medical exposure 1 1 1 1 1
G2: Occupational radiation protection 1 1 1 1 = G1
G3: Control of chronic exposures (Radon, NORM and past practices) and remediation
= G2 = G2 1 1 = G1
G4: Control of rad. discharges and material for clearance 1 1 1 1 = G1
G5: Environmental monitoring associated with authorized practices for public rad. protection purposes
= G4 = G4 = G4 = G4 = G1
I Administration 1 1 1 1 1
Total 10 12 22 26 12
Categorization of countries
Country Cat I: Countries with very limited types and number of regulated radiation sources
facilities/activities
Country Cat II: Countries with no nuclear facilities/activities but with radiation sources facilities/activities
and with well-established regulatory oversight
Country Cat III: Countries with no NPPs, but with other nuclear facilities/activities in addition to radiation
sources facilities/activities with well-established regulatory oversight
Country Cat IV: Countries with NPPs, other nuclear facilities/activities in addition to radiation sources
facilities/activities well-established regulatory oversight,
Table 2: Alternative assignment of the modules to groups and guidance on the number of experts in
an IRRS mission
Appendix III: Standard Schedule of IRRS Missions Initial Mission First Week
Time SAT SUN MON TUE WED THU FRI SAT SUN
9:00-10:00 A
rriv
al o
f T
ea
m M
em
be
rs
Team building and
Initial Team
meeting:
• 5 minutes/TM
self-intro
• Refresher
training
Entrance
Meeting
Inte
rvie
ws
Vis
its
Inte
rvie
ws
Vis
its
Inte
rvie
ws
Vis
its/
EP
R e
xerc
.
DT
C w
rite
s
intr
od
uct
ory
pa
rts
TM write Report
TL and DTL review
introductory part
• Discussing and
improving Draft
Report
• Cross-Reading
• TL, DTL, TC and
DTC read
everything
Fre
e d
ay
, S
oci
al T
ou
r
Re
ad
ing,
Cro
ss-r
ead
ing
of
the
Re
po
rt
10:00-11:00
11:00-12:00 Draft text to TL
12:00-13:00
Lunch Lunch with
Host
Standing lunch Standing
lunch Standing lunch Standing lunch Standing lunch
13:00-14:00
Inte
rvie
ws
Vis
its
Inte
rvie
ws
+ in
-gro
up
dis
cuss
ion
s
Vis
its
Inte
rvie
ws
+ in
-gro
up
dis
cuss
ion
s
Vis
its/
EP
R E
xerc
ise
DT
C w
rite
s in
tro
du
cto
ry p
art
s Policy Discussions
Finalisation of the
Draft Report
14:00-15:00
Team building and
Initial Team
meeting
continued
• First
observations
• In-Group
discussions
Inte
rvie
ws 15:00-16:00
Secr
eta
riat
ed
its
the
rep
ort
Pre
lim
ina
ry D
raft
Re
po
rt
Re
ad
y
Cro
ss-r
ea
din
g b
y T
M
16:00-17:00
Written
preliminary
findings
delivered
17:00-18:00 Daily Team
Meeting
Daily Team
Meeting
Daily Team
Meeting:
Discussion of
findings
Daily Team
Meeting
Daily Team
Meeting
18:00-20:00 Informal
dinner Team Dinner Dinner Dinner Dinner Dinner Dinner Dinner
20:00-24:00 Writing of
the report
Writing of the
report
Secretariat
edits Report
TM write
Report
Writing of the
report TM Read Draft
Secretariat edits the
report
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 34
Initial Mission Second Week
MON TUE WED THU FRI
9:00-10:00
Individual discussions
of Rs, Ss and GPs with
counterparts
Cross-Reading
TL, DTL, TC and DTC read
everything
Finalisation
Common read through
and finalisation by the
Team Discussion with Host
Submission of the Final Draft 9:00-10:00
10:00-12:00 Exit Meeting
Press Conference 10:00-12:00
Submission of the Draft
to the Host
12:00-13:00 Standing lunch Standing lunch Lunch Standing Lunch Lunch 12:00-13:00
13:00-15:00 Policy Discussions
Discussion of the
report by the
team
TC
, D
TC
pre
pa
re E
xecu
tive
Sum
ma
ry a
nd
exi
t p
rese
nta
tio
n
Ho
st r
ead
s D
raft
TL
fin
alis
es
Exe
cuti
ve
Sum
ma
ry a
nd
exi
t
pre
sen
tati
on
TC
Dra
fts
the
Pre
ss
Re
lea
se
Written comments by the
Host
Team meeting for
finalisation of the Report
Departure Home
13:00-15:00
15:00-17:00
Individual discussions
of Rs, Ss and GPs with
counterparts
15:00-17:00
17:00-18:00 Daily Team Meeting Discussion of Executive
Summary
Briefing of the DDG
Finalisation of the press
release
17:00-18:00
18:00-20:00 Dinner Dinner Dinner
Farewell Dinner
18:00-20:00
20:00-21:00 Secretariat includes
changes Secretariat finalises text Free
20:00-21:00
21:00-24:00 Free 21:00-24:00
F u r t h e r I m p r o v i n g t h e I R R S P r o c e s s P a g e | 35
Follow-up Mission
Time SAT SUN MON TUE WED THU FRI SAT
9:00-10:00
Arr
iva
l of
Te
am
Me
mb
ers
Team building
meeting:
• 5 minutes/TM
self-intro
• Refresher
training
Entrance
Meeting
Interviews
TM write Report
TL and DTL review
introductory part
Finalisation
Discussion by the
Team
10:00-11:00
Interviews Discussion by the
Host
Exit Meeting
&
Press Conference
(alternative) 11:00-12:00 Draft text to TL Submission of the
Draft to the Host
12:00-13:00
Lunch
Standing
Lunch
Standing
lunch Standing lunch Standing lunch Standing Lunch
De
pa
rtu
res
of
Te
am
Me
mb
ers
13:00-14:00
Interviews
Interviews
Policy Discussions
Ho
st r
ead
s D
raft
TL
fin
alis
es
the
pre
sen
tati
on
TC
dra
fts
the
Pre
ss R
ele
ase
Written comments
by the Host
14:00-15:00 Initial Team
Meeting:
• IRRS process
• Main
objectives
• Report writing
• Schedule
• First
observations
Finalisation of the
Report 15:00-16:00
Secr
eta
riat
ed
its
the
rep
ort
Cro
ss-r
ea
din
g
16:00-17:00
Written
preliminary
findings
delivered
Preliminary Draft
Report Ready
Presenting the final
Draft of the Report
to the Host
17:00-18:00 Daily Team
Meeting
Daily Team
Meeting:
Discussion of
findings
Daily Team
Meeting
Discussion of
Executive
Summary
Exit Meeting
&
Press Conference
(optional) 18:00-19:00 Informal
dinner Team Dinner Dinner Dinner Dinner Dinner
19:00-20:00
Farewell Dinner 20:00-24:00
Writing of
the report
Secretariat
edits Report
TM write
Report
TM Read Draft Free
Recommended