2 DRAFT WORKING DOCUMENT FOR COMMENTS: 3

Working document WLA OpG Rev. 1 July 2021

1

DRAFT WORKING DOCUMENT FOR COMMENTS: 2

3

OPERATIONAL GUIDANCE 4

EVALUATING AND PUBLICLY DESIGNATING REGULATORY 5

AUTHORITIES AS WHO-LISTED AUTHORITIES 6

7

Please send your comments to Hiiti Sillo, Team lead, Regulatory Systems Strengthening, Regulation

and Safety Unit ( [email protected] ), with a copy to Anna Laura Salvati ( [email protected] ), before 19

September 2021. Please use the “Table of Comments” document for this purpose.

This working document is sent out electronically and will also be placed on the WHO Regulation and Safety website ( https://www.who.int/initiatives/who-listed-authority-reg-authorities ) for comments.

© World Health Organization 2021

All rights reserved.

This is a draft. The content of this document is not final, and the text may be subject to revisions before publication. The document may not be reviewed, abstracted, quoted, reproduced, transmitted, distributed, translated or adapted, in part or in whole, in any form or by any means without the permission of the World Health Organization.

Please send any request for permission to:

Hiiti Sillo, Regulatory System Strengthening, Regulation and Safety Unit, Department of Regulation and Prequalification, World Health Organization, CH-1211 Geneva 27, 21 Switzerland, email: [email protected]. The designations employed and the presentation of the material in this draft do not imply the expression of any opinion whatsoever on the part of the World Health Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by the World Health Organization in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by the World Health Organization to verify the information contained in this draft.

However, the printed material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall the World Health Organization be liable for damages arising from its use.

This draft does not necessarily represent the decisions or the stated policy of the World Health Organization. 8

mailto:[email protected]


https://www.who.int/initiatives/who-listed-authority-reg-authorities


Working document WLA OpG Rev. 1

Page 2

SCHEDULE FOR DRAFT WORKING DOCUMENT WLA OpG: 9


EVALUATING AND PUBLICLY DESIGNATING REGULATORY 11

AUTHORITIES AS WHO-LISTED AUTHORITIES 12

13

Description of Activity Date

Preparation of first draft working document October 2020

Working Group discussions among experts of

National Regulatory Authorities to set the

Performance Evaluation Process of each

function

October 2020 - March 2021

Consolidation of comments received.

Preparation of working document for public

consultation

January - July 2021

Public consultation 26 July - 19 September 2021

Consolidation of comments received and review

of feedback. September - October 2021

Any other follow up action as required

14

15

16

17

18

19


Page 3

20


EVALUATING AND PUBLICLY DESIGNATING REGULATORY AUTHORITIES AS WHO-LISTED 22

AUTHORITIES 23

24

Abbreviations.................................................................................................................................... 5 25

1. Introduction.............................................................................................................................. 7 26

2. Purpose .................................................................................................................................... 7 27

3. Scope ........................................................................................................................................ 8 28

4. Evaluating regulatory performance: terms and general considerations .................................. 9 29

5. Transparency and information-sharing ..................................................................................... 11 30

6. Process for applying for, evaluating and listing a WLA ........................................................... 12 31

6.1 Submission of Expression of Interest 17 32

6.1.1 Eligibility criteria ...................................................................................................... 20 33

6.1.2 Regional Regulatory Systems ......................................................................................... 21 34

6.2 Evaluation of EOI 22 35

6.3 Agreement on scope of evaluation and listing 23 36

6.4 Performance evaluation 24 37

6.5 Decision and listing 27 38

6.5.1 Two-level decision-making process ............................................................................... 27 39

6.5.2 WAG recommendation .................................................................................................. 28 40

6.5.3 Decision on listing and Notice of Listing Decision ......................................................... 29 41

6.5.4 Listing as WLA on WHO website .................................................................................... 31 42

7. Renewal of listing and ongoing monitoring ............................................................................ 33 43

7.1 Renewal 33 44

7.2 Ongoing reporting and monitoring 36 45

7.2.1 Interim reporting by WLA ........................................................................................ 36 46

7.2.3 Ongoing monitoring by WHO .................................................................................. 36 47

8. Re-evaluation and delisting .................................................................................................... 37 48

8.1 Re-evaluation 37 49

8.2 Delisting 39 50

9. Information management system .......................................................................................... 41 51

10. Monitoring and evaluation.................................................................................................... 41 52

11. Glossary ................................................................................................................................. 42 53


Page 4

12. References ............................................................................................................................ 45 54

13. Annexes and Appendices ...................................................................................................... 45 55

Annex 1 – Template for Expression of Interest to be designated as WHO Listed Authority 47 56

Annex 2 – Agreement on the conditions to be designated as WHO Listed Authority 51 57

Annex 3 – Sample Listing 55 58

Annex 4 – WLA Technical Advisory Group 57 59

Annex 5 – Renewal reporting form for WHO Listed Authorities 62 60

Annex 6 – Performance Evaluation Process 66 61

Appendix 1 to Annex 6 (6.1) – Vigilance Filed Visit Manual 191 62

Appendix 2 to Annex 6 (6.2) – GxP Observed Audit Manual 276 63

64


Page 5

Abbreviations 65

ADG WHO Assistant Director-General 66

API Active Pharmaceutical Ingredient 67

CT Clinical Trial, Clinical Trials Oversight 68

EC Ethics Committee 69

EOI Expression of Interest 70

FP Finished Product 71

GBT Global Benchmarking Tool 72

GCP Good Clinical Practices 73

GDP Good Distribution Practices 74

GMP Good manufacturing Practices 75

GRP Good Regulatory Practices 76

GRelP Good Reliance Practices 77

GVP Good Vigilance Practices 78

IT Information Technology 79

LI Licensing Establishments 80

LR Lot Release, National Lot Release 81

LT Laboratory Testing 82

MA Registration and Marketing Authorization 83

MC Market Control, Market Surveillance and Control 84

ML Maturity Level 85

MS Member State 86

NCL National Control Laboratory 87

NOC Notice of Concern 88

NOD Notice of Delisting 89

NOLD Notice of Listing Decision 90

NOLD-R Notice of Listing Decision – Renewal 91

NOLD-Rev Notice of Listing Decision -Re-evaluation 92

NRA National Regulatory Authority 93

OA Observed Audit 94

PE Performance Evaluation 95

PEP Performance Evaluation Process 96


Page 6

PMS Post Market Surveillance 97

PQ Prequalification 98

PV Pharmacovigilance 99

QMS Quality Management System 100

REG Regulation and Safety Unit 101

RI Regulatory Inspections 102

RPQ Regulation and Prequalification Department 103

RRS Regional Regulatory System 104

RS Regulatory System 105

RSS Regulatory System Strengthening 106

SOP Standard Operating Procedure 107

TOR Terms of Reference 108

VL Vigilance 109

WAG WLA Advisory Group 110

WHO World Health Organization 111

WLA WHO Listed Authority 112


Page 7

1. Introduction 113

The introduction of a framework for designating and publicly listing a regulatory authority as a 114

WHO-listed authority (WLA) provides a transparent and evidence-based pathway for 115

regulatory authorities to be globally recognized as meeting WHO and other international 116

recognized standards and practices. 117

While the ultimate responsibility and decision for use of the WLA list resides with the users and 118

depends on the specific context of its intended use, the benefits of a robust, transparent, 119

evidence-based, global system for recognizing regulatory excellence serve the interests of a 120

variety of stakeholders that are committed to promoting access to safe, effective, and quality 121

medical products. 122

This guidance, together with supporting documents, including the Resolution WHA 67.20: 123

Regulatory system strengthening for medical products (1); the Roadmap for access to 124

medicines, vaccines and health product 2019-2023: comprehensive support for access to 125

medicines, vaccines and other health products (2); the Policy: Evaluating and publicly 126

designating regulatory authorities as WHO listed authorities (3); the Global Benchmarking Tool 127

(GBT) and GBT Manual (4), constitute the framework for designating and publicly listing 128

regulatory authorities and regional regulatory systems (RRSs) as WHO Listed Authorities (WLA 129

Framework). 130

The guidance takes account of the work undertaken by WHO and other organizations in 131

benchmarking and strengthening regulatory systems with the aim of ensuring sound and 132

effective regulatory oversight of medical products. 133

This guidance and its annexes were developed following international consultative meetings 134

with Member States (MSs) and interested stakeholders, broad public consultations, review of 135

comments received on draft WLA-related documents, and numerous technical working group 136

discussions focused on each regulatory function. 137

138

2. Purpose 139

This guidance describes the process for evaluating and publicly designating regulatory 140

authorities and regional regulatory systems as WLAs. The guidance provides the details 141


Page 8

required to operationalize the corresponding Policy document, a document that sets out the 142

rationale, definitions and high-level operating principles underpinning the WLA framework. 143

Details are provided on the steps, timelines and processes involved in evaluating and 144

designating a WLA. These steps include submission of an application or Expression of Interest 145

(EOI), assessment of the application against eligibility criteria, evaluation and documentation 146

of regulatory performance, and public listing of those authorities or RRSs that comply with 147

established requirements as outlined in this document. 148

The guidance also describes the process and criteria for renewal, re-evaluation and possible 149

delisting, the roles and responsibilities of the WLA Advisory Group (WAG) and the undertakings 150

of the WHO and the candidate WLA. Definitions are also provided for terms used in this 151

document (see Glossary). 152

153

3. Scope 154

As further explained in the following sections, a regulatory authority or RRS (including 155

individual member authorities) can be listed for one or more product categories and/or for one 156

or more regulatory functions. The overarching Regulatory System, being the backbone and 157

foundation of any regulatory authority, is always included in the evaluation process, as is any 158

function or product category covered by the application for listing. 159

The initial scope of the WLA designation will be limited to multisource (generics), new 160

medicines (new chemical entities), biotherapeutics, similar biotherapeutic products and 161

vaccines. There is an option to expand the scope to other categories of products in the future, 162

in line with the expansion of the scope of the GBT. Consistent with the GBT, the regulatory 163

oversight of export-only products is not included in the WLA evaluation framework. 164

165

This guidance also defines the Performance Evaluation (PE) process used to evaluate 166

regulatory performance. Two pathways can be followed, either the full or the abridged 167

pathway, which reflect the expected extent and depth of the evaluation process (see section 168

6 - Process for application, evaluation and listing a WLA). In either case, a regulatory authority 169

or RRS must demonstrate overall maturity level (ML) 3, as established by the GBT, to be eligible 170

for consideration as a WLA. 171

172


Page 9

This guidance should be read in conjunction with the Policy: Evaluating and publicly designating 173

regulatory authorities as WHO-listed authorities (3) and the GBT and GBT Manual (4). Readers 174

are also encouraged to consult relevant WHO guidelines, including Good regulatory practices 175

for regulatory oversight of medical products (5), Good reliance practices in regulatory decision-176

making: high-level principles and recommendations (6), and The implementation of quality 177

management systems for national regulatory authorities (7) to gain a thorough understanding 178

of the regulatory attributes a WLA should exhibit. 179

180

4. Evaluating regulatory performance: terms and general 181

considerations 182

While the benchmarking process and tool provide a stable and consolidated framework to 183

evaluate how well a regulatory system is configured to provide effective regulatory oversight 184

of medical products, the Performance Evaluation Process (PEP) enables the measurement of 185

the regulatory system operations, its level of performance, and ultimately, its effectiveness. 186

For the purpose of this guidance, performance represents the degree to which regulatory 187

inputs and processes consistently and efficiently result in the desired regulatory outputs1. 188

Effectiveness is the extent to which a specific intervention, procedure, regimen or service, 189

when deployed in the field in routine circumstances, achieves the intended result for a 190

specified population. While more difficult to measure, performance may be extended to 191

regulatory outcomes, effectiveness, or impact (Fig.1). 192

1 According to ISO 9000 (2015), performance can relate to the management of activities, processes, products, services,

systems or organizations and be measured in terms of quantitative or qualitative findings.


Page 10

193

Figure 1: Elements to be considered for evaluating performance of a regulatory system (NOTE: Please 194 refer to abbreviation list for definitions.) 195

196

197

The WHO GBT represents the foundation for assessing and classifying regulatory systems 198

according to ML, by providing a robust and structured approach to analysing the required 199

inputs, regulatory processes and intended outputs. The PE elements of WLA complement the 200

GBT, by rendering a more detailed picture of the regulatory system, through an expanded PEP 201

that examines key regulatory outputs and consistency in adherence to good regulatory 202

practices (GRP) and international standards and guidelines. 203

In this context, PE refers to the process of evaluating the performance of a regulatory authority 204

or RRS in terms of specific regulatory functions and/or product categories, by gathering and 205

analysing output data and evidence related to key regulatory parameters and activities. The 206

specific regulatory functions and/or product categories that are evaluated represent those 207

selected by the regulatory authority or RRS for which WLA listing is sought. Regulatory 208

functions and product categories eligible for listing are described in Table 1 and Table 2, section 209

6.1 – Submission of Expression of Interest. 210

The PEP builds upon the GBT and specifically on the performance indicators in the GBT and on 211

the well-established PE mechanisms already used by WHO, for example, the WHO Observed 212

Manufacturers

& Developerse.g.: MA/CT

applications,

Scientific Advice,Licensing

applications,PV reports

RegulatoryInputs

Healthcare

professionalse.g.: safety signals

Other

stakeholderse.g.: patient/trade

associations

inputs,requests from

other NRAs

RegulatoryFunctions &

Processes(e.g.: CT, MA, LI,

RI, VL, MC, LT, LR)

Regulatory

Framework(legal provisions, laws, regulations,

guidelines, others)

RegulatorySystem

Resources

(human resources, financial resources, infrastructures, IT

systems, others)

e.g.: inspection/

assessmentreports,

regulatory

decisions,

acts,

approved product labelling

information

RegulatoryOutputs

RegulatoryOutcomes

e.g.:

increasedcompliance

with regulatory

requirements

e.g.:

access to safe, effective and

assured quality

medical products,

less SF medical

products,

increasedpharmaceutical

contribution to country’s

economicreveneues

RegulatoryImpact

Regulatory

Institutions(NRA, NCL, PV

centers, EC,

others)

QualityManagement

System(manuals, SOPs,

instructions)


Page 11

Audit (OA) for Good Manufacturing Practices (GMP) inspections and the WHO vigilance field 213

visit. To avoid duplication of efforts, waste of resources and unnecessary burdens, the PEP 214

also takes into account any established and internationally or regionally recognized regulatory 215

benchmarking or audit mechanisms. 216

Tools and methodologies for PE were developed through a collaborative international effort 217

that made a judgment regarding how meaningful, measurable and reasonable specific 218

regulatory activities and parameters were in documenting and promoting a high degree of 219

confidence in a regulatory authority or RRS. The PE framework distinguishes between 220

technical performance (e.g., scientifically sound decision-making and quality and consistent 221

implementation of regulatory outputs) and operational performance (e.g., the time it takes to 222

render a decision on a marketing authorization (MA) application). 223

Annex 6 provides a full description of the tools and methodologies for measuring regulatory 224

functions and of the product categories eligible for listing. 225

226

5. Transparency and information-sharing 227

Transparency is a hallmark of a WLA and a critical pre-requisite to building trust and 228

confidence in the regulatory system, as noted in the WHO guideline on Good regulatory 229

practices for regulatory oversight of medical products (5): “Transparency is a key enabler to 230

adopting new, more efficient ways of conducting regulatory operations. It is incumbent upon 231

regulators to practice transparency in regulatory operations and decisions as a fundamental 232

principle of Good Regulatory Practices (GRP), but also towards building trust and maximizing 233

opportunities for cooperation and reliance as part of a shared regulatory community 234

responsibility.” 235

In other words, regulatory authorities are an increasingly important audience and beneficiary 236

of measures that promote transparency in regulation through the publishing and sharing of 237

regulatory information such as product assessments and site inspections. 238

As a general principle, WHO supports the implementation of reliance on other regulators’ 239

work, in order to make the best use of available resources and expertise, as noted in the WHO 240

guideline on Good reliance practices in regulatory decision-making: high-level principles and 241

recommendations (6). Given its importance in building trust and promoting reliance, 242


Page 12

transparency in regulatory operations and decisions is a key focus area in evaluating 243

candidate WLAs. Regulatory authorities and RRSs seeking designation as WLAs are therefore 244

expected to make regulatory information publicly available (while protecting personal or 245

commercially confidential information) and to engage in information-sharing, thereby 246

enabling greater regulatory efficiencies and more informed decision-making at the regional 247

and global level. 248

Towards the same end, the introduction of a framework for publicly designating WLAs serves 249

to promote transparency in the requirements, processes, outcomes, and bases for listing a 250

regulatory authority or RRS that operates at an advanced level of performance and 251

demonstrates continuous improvement and transparency. The same considerations and 252

principles will be applied with respect to the issuance of renewals, issues of concern, or 253

delisting. In so doing, WHO has taken into consideration practices adopted by the WHO 254

Prequalification (PQ) Team and the regulatory community. 255

In addition to the information made public, further information on regulatory evaluations will 256

be available, via a secure platform, to other regulatory authorities and RRSs that have 257

undergone the WLA process. Sharing of information, as described in this guidance (Section 9 258

– Information management system), provides a mechanism for facilitating further 259

transparency and information-sharing. 260

261

6. Process for applying for, evaluating and listing a WLA 262

The steps and general processes for the application for and the evaluation and listing of a 263

regulatory authority seeking WLA designation are described below. The overall process for 264

initial listing, illustrated in Figure 2, is divided into the following steps: 265

1. Submission of Expression of Interest (EOI). 266

2. Evaluation of EOI against eligibility criteria. 267

3. Agreement on the scope of the evaluation and roadmap. 268

4. PE exercise. 269

5. Decision and listing. 270


Page 13

271

Figure 2: Overview of the application, evaluation, and listing steps (NOTE: Please refer to abbreviation 272

list for definitions.) 273

274

275

General principles and considerations 276

o The decision to apply for evaluation and listing as a WLA is voluntary and is initiated by or 277

on behalf of the regulatory authority or RRS. 278

o The operational framework for evaluating regulatory performance provides flexibility in 279

the approach to suit the circumstances and takes into consideration the availability and 280

applicability of existing information from benchmarking or audit exercises or from other 281

sources relevant to the designation being sought. While established requirements must 282

be met, the adoption of a risk-based approach means the evaluation plan should be 283

customized for each regulatory authority and RRS to ensure optimal use of existing 284

information and resources. 285

o As a consequence of the above, WHO may: 286

• select either a full or an abridged PE pathway, reflecting the extent and depth of 287

evaluation; 288

• conduct benchmarking and PE activities sequentially or, whenever possible, in 289

parallel, thus reducing the time needed for completing the whole evaluation 290

process. 291

o The evaluation plan will be developed in discussion with the candidate WLA and will be 292

documented in the Agreement on the conditions to be designated a WHO Listed Authority 293

(Annex 2) and in the associated evaluation plan, which may be revised from time to time 294

based on mutual agreement. 295

Expression of interest (EOI)

Evaluation of EOI

Agreement of scope and listing, roadmap

Performance evaluation

Decision and publication

Scope:

1. Products• Generic

medicines• New medicines• Vaccines2. Regulatory

functions• MA• RI• CT...

Voluntary request by NRA or RRS

Risk-based approach

• Full vs. Abridgedbenchmarking

• Extent and duration of PE

• Sequential vs. parallel benchmarking

• PE

• Fullimplementationof all mandatorysub-indicators

• PE activities specific for RS + the relevant function(s)

Eligibility criteria• Two-layer

decision-making process

• Publication


Page 14

o All information pertaining to the submission of an EOI and subsequent steps will be held 296

in confidence through the use of secure platforms, as stipulated in the Agreement, until 297

the regulatory authority or RRS is listed as a WLA. The information to be released once 298

listing is achieved is managed through the Agreement on the conditions to be designated 299

a WHO Listed Authority (Annex 2). 300

o Should listing not be achieved, the overall ML of the National Regulatory Authority (NRA) 301

or RRS, reached through the GBT assessment, will not be modified, unless major changes 302

potentially impacting the ML are identified. 303

o Prioritization of requests for designation as a WLA may be necessary depending on 304

number of eligible requests and availability of resources, as further described in section 305

6.1 – Submission of Expression of Interest. 306

o A WLA is expected to maintain comprehensive oversight of regulatory functions and 307

related activities for the product categories relevant to the scope of WLA listing, even when 308

reliance is applied. In other words, although the NRA may rely on other recognized 309

authorities for some regulatory activities, the decision-making process and regulatory 310

decisions should remain under the full responsibility and control of the NRA, which in turn 311

should demonstrate that it has the regulatory and technical capacity to perform the same 312

type of activities adequately and independently. As a consequence of the above, functions 313

in which full reliance is applied cannot be considered for listing. 314

o Outsourcing (i.e., via agreements with third parties to perform tasks assigned to the NRA) 315

of some technical regulatory activities (e.g., lab testing) is considered acceptable for a WLA, 316

provided that the regulatory oversight of these activities and consequent regulatory 317

decisions remain under the full control and responsibility of the NRA. Written contracts 318

with organizations approved for the type of activity outsourced, are expected to be in place 319

and to define clearly the duties and responsibilities of each party. Through continuous 320

monitoring of any outsourced activities, the NRA is responsible for periodically assessing the 321

competence of contracted organization(s). The contracted organization should not pass 322

along to a third party any work entrusted to it under the contract, without prior evaluation 323

and approval of the arrangements by the NRA. WHO reserves the right to conduct 324

independent assessment of contracted organizations, both at national and international level, 325

based on the criticality of outsourced activities. 326


Page 15

o The overall listing process is time bound, as illustrated below (Fig.3). The candidate 327

regulatory authority or RRS should abide by the relevant timelines that were agreed to as 328

part of the evaluation plan. If a regulatory authority or RRS is not in a position to complete 329

the PE phase, it may resubmit a new EOI without prejudice at a later date. 330

331

332

333

Figure 3: Overview of the process and timelines for listing (NOTE: Please refer to abbreviation list for 334

definitions.) 335

336

Full and abridged evaluation pathways 337

Regardless of the pathway chosen, WHO will use all available information and the most 338

efficient methodology to reach a decision on listing. 339

340

Abridged evaluation pathway 341

The abridged pathway is an abbreviated, risk-based evaluation process that will be 342

preferentially adopted, whenever possible, if an established and documented track record of 343

regulatory performance already exists and can be taken into account when determining 344

compliance with the requirements for designation as a WLA. Evidence already existing should 345

contribute substantially to the WLA designation being sought. Depending on available 346

documented information, WHO will customize the evaluation process with the aim of 347

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation

WLA team of assessors report

WAG report and recommendations

NOLD

Listing

Decision on EOI

Agreementon scope &

roadmap

Start evaluation

30 days

REG Unit Head review

60 daysWAG review of the report and meeting

30 daysListing

decision by the ADG

Publication procedure

45 days

Deferral

90

day

s

Appealrequest

30

day

s

60 days 30 days

AppealPanel

meeting

AppealPanel report

12-14 months


Page 16

addressing any gaps between WLA requirements (i.e., GBT indicators as well as PEP) and 348

existing evidence. 349

Abridged evaluations may be undertaken through a desk-based review of a self-350

benchmarking exercise and through a remote assessment of all available evidence to 351

demonstrate consistent adherence to international standards, guidelines and regulatory best 352

practices, ongoing interactions and leadership in international fora, and compliance with 353

criteria and requirements for benchmarking audits by WHO or other internationally 354

recognized organizations or initiatives. To this end, WHO reserves the right to request 355

unredacted reports of non-WHO assessments and undertake targeted verifications. 356

357

358

Fig. 4: Schematic representation of evaluation pathways to achieve listing. (NOTE: Please refer to 359 abbreviation list for definitions.) 360

361

Full evaluation pathway 362

The full pathway applies when evidence to support an abridged process, as described above, 363

either is not available or insufficient, and an assessment of compliance against all criteria of 364

the designation being sought (i.e., GBT indicators and PEP) is deemed necessary (Fig. 4). With 365

the aim of avoiding overlaps with recognized mechanisms of evaluation and optimizing the 366

use of resources, WHO, even in the full evaluation pathway, will take into consideration 367

existing evidence and will apply recognition or reliance for specific and well-defined 368

evaluation steps. Given the variability of available resources worldwide, details of the 369

recognized pathways and evaluation activities, as well as the rationale and basis for the 370

adopted flexibility, are defined in Annex 6 for each regulatory function. 371


Page 17

Full evaluation pathways can be run either sequentially or through parallel assessments (Fig. 372

4). 373

• Formal GBT benchmarking and the PEP can be run in parallel (parallel assessment) for 374

NRAs that are expected to meet both eligibility criteria (ML3) and requirements for 375

WLA in the function(s) for which listing was requested. This expectation can be based 376

on other evidence, including, for example, verification of self-benchmarking using 377

GBT, previous benchmarking and audit exercises, memberships to recognized 378

organizations relevant to the listing being sought, or other mechanisms of PE. In the 379

end, the same requirements of performance must be met and documented. 380

• In the sequential assessment, candidate WLAs have access to the PEP once the 381

eligibility criteria (i.e.: overall ML3 as detailed in section 6.1.1 - Eligibility criteria) have 382

been assessed and confirmed through a formal GBT benchmarking. 383

384

6.1 Submission of Expression of Interest 385

386

Figure 5a: Timelines for listing – EOI submission (NOTE: Please refer to abbreviation list for definitions.) 387

388

A regulatory authority or RRS interested in seeking recognition as a WLA is encouraged to meet 389

with WHO to discuss the WLA evaluation and listing process prior to the formal application 390

process. 391

The formal process is initiated by a request from a MS (Fig. 5a), in response to a WHO call for 392

EOI for WLAs, which will be regularly published on the WHO website. WHO will schedule a plan 393

for PE, taking into consideration the following considerations: 394

• Chronology of eligible requests. 395

• Readiness of candidate WLA. 396

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation


WAG report and recommendation

NOLD

ListingDecision on EOI

Agreementon scope &

roadmap

Start evaluation

30 days

60 daysWAG review of the report and

meeting


30 daysListing decision

by ADG

45 daysPublication procedure


Page 18

• Anticipated level of effort required to conduct PE exercise. 397

• RRS versus a single regulatory authority. 398

• Wide recognition of the candidate WLA as a regional or international reference authority. 399

• Potential to enhance and expand the PQ of medical products. 400

• Country’s current or potential capacity for the production and export of medical 401

products. 402

• Degree to which a regulatory authority or RRS actively participates in mechanisms for 403

cooperation with and strengthening of other regulatory authorities or regional 404

systems. 405

• A record of, or a commitment to, making regulatory assessments of products and 406

facilities available to other regulatory authorities in the form of publicly available 407

information or other information-sharing mechanisms. 408

409

The request must include a completed EOI (Annex 1) signed by a person duly authorized to 410

represent the regulatory authority (i.e., head of the regulatory authority or designate). The 411

EOI should also identify the person responsible for filing the request and for serving as the 412

official contact for correspondence (hereafter ‘the applicant’). 413

414

A regulatory authority or RRS can be listed for one or more product categories and/or for one 415

or more regulatory functions. Functions and product categories may be combined to achieve 416

targeted listing. 417

The EOI must specify the WLA listing category(ies) being sought (see Table 1 and 2), provide 418

evidence that eligibility criteria are met, and confirm acceptance with the general provisions 419

described in the Agreement on the conditions to be designated a WHO Listed Authority (Annex 420

2). 421

The EOI should also provide the necessary evidence of performance in support of a request for 422

an abridged evaluation. 423

The completed EOI and all accompanying information should be submitted in the format 424

prescribed under Annex 1 together with a covering letter to WHO.INTwho.int. 425

426

427

428



Page 19

Table 1: Functions eligible for listing 429

Function(s) to be assessed

Eligible for listing Regulatory Functions

RSi MA VL MC LIii RIii CT LT LRiii

Registration and Marketing Authorization (MA)

✓ ✓

Vigilance (VL) ✓ ✓

Market Surveillance and Control (MC)

✓ ✓

Licensing Establishments (LI) ✓ ✓ ✓

Regulatory Inspection (RI) ✓ ✓ ✓

Clinical Trial Oversight (CT) ✓ ✓

Laboratory Access and Testing (LT)

✓ ✓

Lot Release (LR) ✓ ✓ ✓

Notes: 430

i. Evaluation of the Regulatory System (RS) is always included in the process, in addition to the 431

function(s) applying for listing. 432

ii. The activities related to establishment licensing and regulatory inspections are both considered 433

necessary to ensure compliance to GxP. Thus, independent listing of LI or RI is not foreseen, 434

unless LI does not apply to the candidate WLA. 435

iii. LR listing is foreseen only in conjunction with the correlated function LT. 436

437

438

Table 2: Product categories eligible for listing 439

Function(s) to be assessed

Eligible for listingiv Product categories

RS MA VL MC LI RI CT LT LR

Multisource (generics) ✓ ✓ ✓ ✓ ✓ ✓ ✓

New medicines (new chemical entities) and/or biotherapeutics and/or similar biotherapeutic products

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Vaccines ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

iv. In the case of product categories, all applicable regulatory functions performed in the 440

regulation of a product category will be evaluated. 441


Page 20

6.1.1 Eligibility criteria 442

A regulatory authority or RRS must have attained overall ML 3 to be eligible for listing and have 443

access to the PEP of the relevant function(s) (see Table 1 and 2). This means that the 444

overarching RS and all regulatory functions defined by the GBT for the regulation of medicines 445

or vaccines must comply with respective sub-indicators. For further details, consult the 446

Manual for benchmarking of the national regulatory system of medical products and 447

formulation of institutional development plans, section 5.3 - Scoring and algorithm for 448

determining maturity level (4). 449

As each regulator and situation is unique, WHO will work with the authority or RRS to establish 450

a roadmap to reach a listing decision. The following considerations apply: 451

• Eligible NRAs (i.e., overall ML 3 as assessed by formal benchmarking by WHO with the 452

GBT) for which the benchmarking validity period 2 has not expired, should provide 453

updated information on the RS, by submitting a self-benchmarking assessment, along 454

with any other evidence required to confirm that no major change has occurred that 455

could represent a potential risk to regulatory oversight or that could impact the overall 456

maturity level (full pathway, sequential assessment). 457

• If a regulatory body has not been officially benchmarked (or if the official benchmarking 458

has expired), but is expected to meet requirements defined for ML3 and WLA based on 459

other evidence, including verification of self-benchmarking using GBT, WHO will adopt 460

a pragmatic approach and exercise good judgement when assessing compliance with 461

these requirements and could consider a parallel assessment (full pathway, parallel 462

assessment). 463

• Depending on evidence available in support of a requested WLA designation and on 464

subsequent discussions with the regulatory authority or RRS, WHO will consider 465

whether an abridged pathway can be applied. 466

• Functions in which full reliance is applied cannot be considered for listing. 467

• Candidate NRAs or RRSs must concur with the general provisions and obligations 468

described in the Agreement on the conditions to be designated a WHO Listed Authority 469

(Annex 2), including the assignment of sufficient resources, the full and continued 470

2 The validity period, typically from three to seven years, is established after formal GBT benchmarking using a risk-based approach (4).


Page 21

commitment of senior management and staff, and the agreement that the outcome 471

will be made publicly available in the form of a public listing as described in section 6.5 472

– Decision and listing. 473

• Regulatory authorities or RRSs applying for listing should be able to comply with the 474

requirements for listing within the agreed upon target timeframes. Should it become 475

evident during the benchmarking-PE exercise that the regulatory authority or RRS is 476

unable to meet requirements for listing within the agreed upon timeframe, the 477

evaluation process will be suspended and the authority or RRS will be invited to re-478

apply at a later time once identified areas for improvement have been addressed. 479

480

6.1.2 Regional Regulatory Systems (RRSs) 481

RRSs are composed of individual regulatory authorities operating under a common regulatory 482

framework. Regional regulatory bodies may also be established to represent the RRS. The 483

following additional considerations apply for RRSs. 484

• The EOI should be submitted by the regional body (where it exists) or by another 485

institution representing the RRS, following coordination with the individual authorities 486

that are part of the system. 487

• A RRS must provide evidence that the common regulatory framework ensures 488

equivalence among the members in terms of regulatory requirements, regulatory 489

practices and quality assurance policies. This should include a description of the 490

organization and governance of the RRS including enforcement powers. 491

• A RRS must have attained overall ML 3 to be eligible for listing and have access to the 492

PEP of the relevant function(s) (see Tables 1 and 2). 493

• When the RRS is further underpinned by a common legal framework, it can be 494

considered as a single entity and, as such, is eligible for listing as a WLA, together with 495

each of the individual authorities that form the system (see section 6.4 Performance 496

evaluation). All member NRAs are expected to meet ML 3 requirements. The EOI 497

should state whether WLA listing is being sought for the individual members of the RRS 498

in addition to the regional entity. 499

500

501


Page 22

6.2 Evaluation of EOI 502

503 Figure 5b: Timelines for listing – EOI evaluation (NOTE: Please refer to abbreviation list for definitions.) 504

505

Upon receipt, WHO evaluates the EOI together with supporting data for completeness and 506

compliance with eligibility criteria. WHO will endeavour to review the EOI within 30 calendar 507

days of receipt (Fig. 5b). The applicant may be contacted to obtain further clarification or 508

additional information needed to complete the review. The review will resume upon receipt 509

of requested information. 510

511

Based on information provided in the EOI and any subsequent responses to requests for 512

information, the review will result in one of the following outcomes, which is communicated 513

to the applicant: 514

a. The regulatory authority or RRS complies with eligibility criteria for the requested WLA 515

listing. 516

A preliminary decision on the PE pathway (full or abridged) will also be communicated, 517

together with an estimate of the earliest start date for the PEP based on the number 518

of similar requests received and the availability of WHO resources. The start date is 519

also dependent on the readiness of the regulatory authority or RRS, as established in 520

subsequent interactions with the candidate WLA. The regulatory authority or RRS will 521

be invited to discuss the outcome of the review and subsequent planning and 522

coordination matters. 523

b. The regulatory authority or RRS does not comply with eligibility criteria. 524

A rationale for the decision will be provided. The regulatory authority or RRS may 525

request a meeting to seek further clarification on the rationale for the decision and on 526

how to address deficiencies. 527

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Agreementon scope &

roadmap

Start evaluation

30 days


meeting



by ADG



Page 23

c. A decision on eligibility is deferred, pending resolution of issues which, in the view of 528

WHO, could in principle be addressed within 90 calendar days. 529

The regulatory authority or RRS will be invited to discuss the resolution of outstanding 530

issues. Should the regulatory authority or RRS not be able to comply within the stated 531

time, the application will be closed, without prejudice to any future EOI filings. 532

533

Note: The outcome of the review may conclude that the regulatory authority or RRS complies with 534

eligibility criteria for one or more, but not all, of the requested listing categories. The rationale for the 535

decision should accompany the response. 536

537

6.3 Agreement on scope of evaluation and listing 538

539 Figure 5c: Timelines for listing – Agreement on scope and roadmap (NOTE: Please refer to abbreviation 540 list for definitions.) 541

542

A regulatory authority or RRS that complies with eligibility criteria will be requested to meet 543

with WHO to discuss the scope of the benchmarking-performance evaluation, leading to 544

signing of Agreement on the conditions to be designated a WHO Listed Authority (hereafter 545

‘the Agreement’) and to approving a roadmap and project plan for reaching a listing decision 546

(Fig. 5c). The Agreement specifies the provisions on the confidentiality of non-public 547

information, and documents the roles and responsibilities of the WHO and the candidate WLA, 548

the intended scope of listing, and the overall PE methodology and pathway (including reliance 549

in principle on existing evidence of performance). Furthermore, the Agreement requests the 550

consent of the regulatory authority or RRS to disclose a positive outcome of the PE exercise, in 551

the form of a public listing as described in section 6.5 – Decision and listing and Annex 3. 552

553

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Start evaluation

30 days


meeting



by ADG


Agreementon scope &

roadmap


Page 24

The Agreement also identifies the senior officials from the regulatory authority or the RRS and 554

from the WHO that are responsible for reviewing progress, addressing issues, and making 555

necessary adjustments to the plan. The roadmap and project plan form part of the overall 556

Agreement and provide further detail on the evaluation process and methodology, timelines, 557

milestones, and decisions points, as well as specifying the assigned focal point for each 558

regulatory function that is subject to the PE exercise. The roadmap and project plan may be 559

updated from time to time with the mutual agreement of the WHO and candidate WLA. 560

561

The scheduled start date for the benchmarking-PEP will be established based on the readiness 562

of the candidate WLA and the WHO planning cycle. 563

564

6.4 Performance evaluation (PE) 565

566 Figure 5d: Timelines for listing – PE (NOTE: Please refer to abbreviation list for definitions.) 567

568

As previously explained, WHO will design, in discussion with the candidate WLA, an integrated 569

benchmarking – PE plan that is customized to the regulatory system and scope of listing under 570

review. Whenever possible, existing evidence of performance will be used in lieu of 571

conducting an evaluation of regulatory activities and outputs. The plan, which forms part of 572

the overall agreement with the candidate WLA, may be adjusted from time to time to reflect 573

progress and findings (Fig. 5d). The evaluation process is designed tentatively to be 574

completed in 6 months, but clock-stops may be considered, subject to mutual agreement. 575

While each situation is unique, the PEP will normally include the following steps: 576

1. A comprehensive review of available information to determine areas and regulatory 577

activities that require further assessment by WHO. 578

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Start evaluation

30 days


meeting



by ADG


Agreementon scope &

roadmap


Page 25

2. For those RRSs considered as single entity, which seek listing for all member NRAs in the 579

same evaluation pathway, a risk-based selection of the individual NRAs that are to be 580

assessed in addition to the RRS. 581

Selected individual NRAs can be assessed for all or a subset of functions and product 582

categories relevant for the RRS listing application, in order for WHO to build a comprehensive 583

picture of the system and its performance. The following criteria are considered, among 584

others: 585

• Number of applications processed (e.g., MA applications, CT applications, and 586

Regulatory Inspections) 587

• Number of local facilities (e.g., clinical centres and manufacturers) 588

• Type of local manufacturers (e.g., active pharmaceutical ingredient (API), finished 589

product (FP), biologicals, or generics) 590

• Number and volume of exported products 591

3. A baseline ‘top-up’ benchmarking exercise, to address any gaps between any previous 592

benchmarking exercise and the current revision of the GBT. 593

4. The design of the PE plan and specific methodology to be used. The goal is to demonstrate 594

the following: 595

a. Full implementation of ML 3 GBT sub-indicators in the function(s) applying for 596

listing, including the RS module. (Please note that, according to the GBT flexible 597

algorithm (4), NRAs can achieve the eligibility criteria of overall ML 3 with up to 598

10% of ML 3 sub-indicators ongoing or partially implemented; however, for WLA, 599

full implementation of ML 3 sub-indicators is required in the relevant functions. 600

Refer to Tables 1 and 2 for details). 601

b. Full implementation of mandatory ML 4 GBT sub-indicators in the function(s) 602

applying for listing and in the RS module (see Annex 6 for details). 603

c. Full implementation of WLA PE indicators specific for the function(s) applying for 604

listing, including the RS module. 605

d. Compliance with additional measurements to demonstrate consistent 606

performance (i.e., regulatory outputs) over a defined period of time. The type and 607

number of these additional measurements are defined and agreed upon 608

according to the listing being sought and the information already available. The 609

PE exercise will be representative of the range of critical variables (e.g., staff and 610

management involved in the regulatory activity, and the variety of product 611


Page 26

applications or production sites regulated by the candidate WLA). This PEP will 612

also include a mix of desk-based and on-site evaluations (see Annex 6 for details). 613

Related to the above, the PE exercise is meant to assess both management and 614

administrative processes – including the decision-making process - that ensure the 615

robustness, timeliness and consistency of regulatory operations and outputs, and 616

that enable a well-functioning quality management system (QMS). 617

5. Selection of the WLA PE team, including WHO team leader. To the extent possible, two 618

assessors will be assigned to each function to increase robustness of the process. 619

Additional technical experts may be required to perform specific activities. 620

Refer to the GBT Manual for further information on the competencies, selection, training, 621

and role of team members (4). 622

6. Execution of PE plan, including: 623

a. Development and endorsement of the Terms of Reference (TOR) for on-site visits. 624

b. Conduct of activities described in the PE plan and the specific TORs. 625

c. Review of progress at critical milestones defined in the plan, with adjustments as 626

necessary. This would normally include the preparation of an interim report. 627

d. Resolution of issues identified during the PE critical to reaching a listing decision. 628

7. Preparation of preliminary draft WLA Team report, which describes the methodology, 629

timelines and milestones, findings, scoring, suggested areas for improvement, and 630

conclusions, including a preliminary recommendation on listing. 631

8. Closeout meeting with the candidate WLA to present and illustrate the outcomes. 632

9. Preparation of the final WLA Team report following review of feedback from the 633

candidate WLA on the draft report. The final WLA Team report is then forwarded to the 634

WHO Regulatory System Strengthening (RSS) Team Lead for review and endorsement 635

(see next section). 636

637

Note: Refer to Annex 6 for more detailed information on the tools and methodologies to be used for 638

measuring the performance of regulatory functions. 639

640

641

642


Page 27

Additional considerations: 643

• WLA-related activities are preferably conducted in English. Alternatively, the local 644

language may be used if it is understood by WHO assessors. Translation into English 645

should be arranged, if needed. 646

• In case of opposite or non-consistent results in one or more of the activities conducted 647

under the PEP (e.g., two OAs or two MA reviews with opposite outcomes), the WLA 648

Team should reach a consensus analysing all the evidence and issue a recommendation 649

on listing or non-listing. 650

• All documents, correspondence, analyses, presentations, and reports related to the 651

WLA PE exercise will be uploaded to the relevant secure platform on the WHO NRA 652

SharePoint (see section 9 – Information management system). 653

654

6.5 Decision and listing 655

6.5.1 Two-level decision-making process 656

657 Figure 5e: Timelines for listing – Two-layer decision-making (NOTE: Please refer to abbreviation list for 658 definitions.) 659

660

The decision-making process is initiated with the submission of the WLA Team report to the 661

Team Lead, RSS for review and endorsement. The RSS Team Lead then forwards the report 662

to the Head, Regulation and Safety Unit (REG) for a second level review and endorsement. 663

664

If the Team Lead or Unit Head have additional comments or wish to recommend an 665

amendment to conclusions and recommendations prior to sign-off, their comments and 666

recommendations should be appended to the WLA Team report. The two-level review and 667

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Agreementon scope &

roadmap

Start evaluation

30 days


meeting


by ADG




Page 28

sign-off should be completed within 30 calendar days of receipt by the RSS Team Lead (Fig. 668

5e). 669

670

6.5.2 WAG recommendation 671

672

673 Figure 5f: Timelines for listing –WAG evaluation (NOTE: Please refer to abbreviation list for definitions.) 674 675

To ensure impartiality of the WLA process, a recommendation to list or delist a regulatory 676

authority or RRS is made following a review of the WLA Team report by the WAG. The WAG 677

review process provides an additional level of assurance that due process was followed and 678

that decisions are supported by findings. 679

The advisory group is set up by WHO based on established and transparent criteria that 680

ensure equitable geographical representation, gender balance and professional 681

competencies in order to provide a representation of different approaches and practical 682

experience from all parts of the world. The remit and membership of the WAG is described 683

in Annex 4. 684

WHO will organize a WAG meeting and the WAG recommendation will be issued no later than 685

60 (sixty) calendar days after receiving the recommendation from the REG Head (Fig.5f). 686

The WAG report will confirm whether or not, on the basis of the evidence, the conclusions 687

and listing recommendations provided in the WLA Team report and in the subsequent two-688

level review are confirmed and supported by the Committee. If not supported, the rationale 689

for the WAG conclusion will be provided. WAG recommendations should be reached by 690

consensus. In the event consensus is not possible, dissenting views should be documented 691

and deferred to WHO Assistant Director General (ADG), Access to Medicines and Health 692

Products Division (see 6.5.3 - Decision on listing). 693

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Agreementon scope &

roadmap

Start evaluation

30 days


meeting


by ADG




Page 29

694

6.5.3 Decision on listing and Notice of Listing Decision (NOLD) 695

696

Figure 5g: Timelines for listing – Issuing of NOLD (NOTE: Please refer to abbreviation list for definitions.) 697

698

The final decision on listing will be made ADG, through the Director responsible for RSS 699

activities. 700

The ADG will request the preparation of a Notice of Listing Decision (NOLD) by the RSS Team. 701

The NOLD serves as the official decision on listing. 702

The target for issuing a NOLD is 30 calendar days from the receipt of the WAG Report (Fig. 703

5g). 704

The NOLD includes the following information: 705

• Decision on listing 706

• Rationale for the decision 707

• Scope of listing 708

In addition, the WLA Team report, the two-level review, and the WAG report will accompany 709

the NOLD. 710

711

The following outcomes are possible: 712

a. A positive decision to list is issued. 713

This decision signifies that all requirements for designating a WLA have been met 714

within the scope of the listing and confirm that the regulatory authority or RRS 715

consistently operates at an advanced level of performance as verified and 716

documented through a robust and transparent PE exercise. 717

718

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Agreementon scope &

roadmap

Start evaluation

30 days


meeting


by ADG




Page 30

b. A positive decision is issued but the NOLD restricts the scope of listing. This would 719

occur if the regulatory authority or RRS requested designation as a WLA in more than 720

one category; however, the requirements for at least one category are not met. 721

722

c. A negative decision is issued. 723

In the event the candidate WLA does not satisfy all requirements for designation as a 724

WLA, a negative decision will be issued together with the reasons for the decision. 725

The decision will not be disclosed by WHO. 726

A negative decision for listing has no impact on the ML reached through GBT and does 727

not prejudice the filing of a new EOI upon the resolution of those areas for 728

improvement identified during the PE exercise. 729

730

The NOLD is signed by the ADG and forwarded in electronic and hard copy format to the 731

appropriate official in the regulatory authority, ministry of health or RRS, as previously 732

identified by the MS or regional body. 733

734

Right to appeal 735

The regulatory authority or RRS has the right to appeal a negative listing decision. The 736

rationale for contesting the decision must be based on procedural grounds and/or evidence 737

that was available at the time of the PE exercise. Subsequent actions undertaken by the 738

regulatory authority or RRS to resolve identified issues should be included in a new EOI. 739

740

The request for an appeal should be filed no later than 30 calendar days (Fig.3) from receipt 741

of the negative NOLD and addressed to the ADG, Access to Medicines and Health Products 742

Division with copy to the Director, Regulation and Prequalification Department (RPQ). If the 743

request is found to comply with grounds for appeal, the ADG will authorize the convening of 744

an appeal panel to consider the request. 745

746

The appeal panel will be comprised of the ADG, Access to Medicines and Health Products 747

Division, a Director from a WHO Regional Office external to the MS in question, one member 748

from a listed authority and a member from a non-listed authority. 749


Page 31

A meeting will be organized within a target of 60 calendar days to review the request. The 750

regulatory authority or RRS will be invited to present its case and respond to questions in a 751

closed meeting. The panel will issue a report with its decision no later than 30 calendar days 752

after the meeting (Fig. 3). 753

754

6.5.4 Listing as WLA on WHO website 755

756 Figure 5h: Timelines for listing – Listing (NOTE: Please refer to abbreviation list for definitions.) 757

758

Following a positive decision to list, the regulatory authority or RRS will be listed on the WHO 759

website (https://www.who.int/initiatives/who-listed-authority-reg-authorities) within a 760

target of 45 calendar days from the issuance of the NOLD (Fig. 4h). The listing consists of the 761

following: 762

• Scope of the designation (e.g., vaccines or generic and multisource medicines, in the 763

case of product categories; or regulatory inspections or vigilance, in the case of 764

regulatory functions). The listing may include one or more categories depending on 765

the scope and outcome of the PE exercise. 766

• Period of validity, normally issued once for an initial period of five (5) years, provided 767

that interim reporting obligations are met and that no cause for concern occurs which 768

would affect the basis for the original listing decision (see section 8 – Re-evaluation 769

and delisting). 770

771

In the interest of promoting transparency and confidence in the process and outcome, the 772

listing includes the following additional information: 773

EOI submission

30 daysEvaluation

of EOI

6 monthsPerformance

evaluation



NOLD


Agreementon scope &

roadmap

Start evaluation

30 days


meeting


by ADG





Page 32

• an outline of the organizational structure and responsibilities of the regulatory 774

authority or RRS; 775

• a summary of evidence reviewed, and the process undertaken to support the listing; 776

• steps and timelines associated with the PE and listing process; 777

• achieved results and ML for each regulatory function; 778

• a link to relevant websites of the NRA or RRS. 779

Prior to publication on the WHO website, a draft of the listing and accompanying summary 780

information is provided to the regulatory authority or RRS, using agreed upon communication 781

channels, to confirm the accuracy of the information. The authority or RRS will have 14 782

calendar days to confirm the accuracy of listing information. 783

If a newly designated WLA appears on one of the interim lists introduced prior to the 784

introduction of the WLA framework 785

(https://www.who.int/initiatives/who-listed-authority-reg-authorities) the interim list will be 786

updated to reflect the WLA designation. 787

See Annex 3 for further detail on listing. 788

In addition to the information made available through the listing process, a more detailed 789

picture of scoring against selected sub-indicators is uploaded to a secure WHO regulatory 790

platform (Section 9 – Information management system). 791

As noted, full details of the PE are available to the regulatory authority or RRS in question, 792

including the final WLA Team and WAG reports. WLAs are free to share any information 793

related to the evaluation and listing with other parties. 794

795

796

797

798

799



Page 33

7. Renewal of listing and ongoing monitoring 800

801

802 Fig. 6: Outline of the process for WLA reporting and renewal. (NOTE: Please refer to abbreviation list 803

for definitions.) 804

805

806

7.1 Renewal 807

A listing will initially be valid for a period of 5 years; at that time, listing is subject to the 808

renewal process described in this section. If the listing is renewed, it will no longer be subject 809

to a validity period (Fig. 6); however, the listing will be subject to continuous monitoring based 810

on risk management principles to ensure that requirements for listing continue to be met 811

(see section 7.2 – Ongoing reporting and monitoring). 812

The WLA listing will be renewed provided that evidence reviewed by WHO continues to 813

support the listing and that the ongoing reporting requirements described below are met. 814

Renewal of listings should be finalized by the end of the initial validity period and is specific 815

to the scope of the initial listing. 816

The renewal process will normally involve a desk-based review by WHO of the following: 817

• The Renewal Report issued by the WLA at least 2 months before the end of the validity 818

period. In this report, the WLA is required to document changes to the regulatory 819

framework, processes, resources, or organizational structure or to the governance 820

mechanisms that were assessed as part of the WLA benchmarking-PE, but do not, in 821

the view of the WLA, warrant interim reporting (see below) or require an update to 822

the listing information. 823

The WLA is requested to complete the electronic WLA Renewal Reporting Form (see 824

Annex 5) and to upload it to the WHO NRA SharePoint; completion of this step serves 825


Page 34

as notification of the WHO. A description of changes and their potential impact on 826

the listing decision should be recorded on the form and accompanied by relevant 827

documents (in electronic format). If relevant information is available on the WLA’s 828

website, a link to the relevant webpages is acceptable in lieu of submitting documents, 829

provided that information on the website is in English or is summarized in English. 830

WHO reserves the right to seek further clarification of changes, as required. 831

If no changes of note have taken place, this would be indicated on the WLA Renewal 832

Reporting Form. 833

• Documents describing any major changes that have occurred to the regulatory 834

framework relevant for the listing decision, as submitted by the WLA during the 835

validity period (see section 7.2.1 - Interim reporting by WLA). 836

• Any other evidence relevant to the re-assessment of performance or with the 837

potential to impact the listing decision, including any information derived from 838

ongoing monitoring by WHO (see section 7.2 – Ongoing monitoring by WHO). 839

840

If required, WHO may draw from the list of qualified assessors, and request that they review 841

evidence submitted by the WLA in support of renewal. 842

An on-site evaluation may be considered to supplement the remote evaluation and properly 843

assess the nature and impact of reported changes or trends. 844

The following options are foreseen: 845

a. Renewal is granted. 846

The renewal of listing decision is communicated to the WLA through a Notice of Listing 847

Decision - Renewal (NOLD-R) based on a Renewal of Listing report prepared by the 848

RSS Team. The Renewal of Listing report describes the methodology, evidence, 849

findings, and conclusions, and provides a recommendation on renewal of listing. The 850

report must be reviewed and endorsed by the RSS Team Lead and the REG Unit Head. 851

The decision on renewal is made by the ADG. The NOLD-R is signed by the ADG. The 852

endorsed Renewal of Listing report accompanies the NOLD-R. 853

The public listing will be then updated to reflect the renewal, the nature of evidence 854

reviewed, and process followed in reaching a renewal decision. Prior to publication 855


Page 35

on the WHO website, a draft of the listing renewal and accompanying summary 856

information is provided to the regulatory authority or RRS to confirm the accuracy of 857

information. The authority or RRS will have 14 calendar days to confirm the accuracy 858

of listing information. 859

Renewed listing remains valid indefinitely, subject to compliance with ongoing 860

reporting requirements and provided that no change or incident takes place which 861

calls into question the basis for the initial listing. 862

b. The listing is renewed on condition. 863

Should an issue of concern arise during the evaluation process, the listing may be 864

renewed on condition that corrective measures addressing the identified issue of 865

concern are fully implemented and evaluated by WHO. The issue of concern, 866

measures being implemented to address the concern, and timelines for completion 867

should be described in the renewal notice. 868

c. Renewal is deferred. 869

WHO may decide not to renew the listing until information requested to confirm the 870

WLA status has been provided and that ongoing reporting requirements are met. A 871

statement to this effect would be communicated in the NOLD-R and published on the 872

WHO website. 873

d. Renewal is not granted and WLA is de-listed. 874

A listing may not be renewed in the event that: 875

• an issue of concern triggers the issuance of a Notice of Concern (NOC; see 876

below); 877

• information requested pertinent to a renewal of listing decision has not 878

been provided; or 879

• ongoing reporting requirements are not met. 880

881

See section 8.2 - Delisting for details. 882

Note: The above conditions are relevant to continued listing at any time, not only at the (one 883

time) renewal of the listing. 884

885


Page 36

7.2 Ongoing reporting and monitoring 886

A condition of initial and continued listing of a regulatory authority or RRS as a WLA is a 887

commitment to and compliance with ongoing reporting requirements. 888

889

7.2.1 Interim reporting by WLA 890

The WLA is required to advise the WHO of significant changes to the regulatory framework, 891

processes, resources, organizational structure, governance, or other parameters that could 892

have an impact on the listed function(s) or on the product category(ies) assessed as part of 893

the WLA benchmarking-PEP. For example, this would include continuous improvement 894

measures (e.g., the creation of a dedicated QMS unit within the organizational structure) or 895

changes that could negatively impact the listing decision (e.g., a major downsizing of the 896

regulatory authority). 897

A description of changes and accompanying information in electronic format should be 898

uploaded on the WHO NRA SharePoint and the RSS Team Lead notified. If information is 899

available on the WLA’s website, a link to the relevant webpages is acceptable in lieu of 900

submitting documents, provided that information on the website is in English or summarized 901

in English. 902

WHO may, upon review of the changes, request further clarification and/or a meeting with 903

the WLA in order to fully evaluate the impact of the change on the listing decision (see also 904

next section on Re-evaluation and delisting). 905

As required, listing information is updated to reflect changes. 906

907

7.2.3 Ongoing monitoring by WHO 908

In addition to the evaluation interim reports issued by WLAs, WHO will perform independent, 909

ongoing monitoring of the WLA’s regulatory performance based on facts and findings coming 910

from different sources (e.g., concerns raised in the context of PQ exercises, issues reported 911

by stakeholders, including industries or patient or trade associations, or any other evidence 912

of scandals or lack of regulatory oversight). 913

WHO reserves the right to request information, clarifications and justifications on events 914

potentially impacting the WLA status and to undertake a formal re-evaluation should this be 915


Page 37

considered necessary to confirm the validity of listing (refer to section 8 – Re-evaluation and 916

delisting). 917

918

8. Re-evaluation and delisting 919

920 Fig. 7: Outline of the process for re-evaluation and listing/delisting. (NOTE: Please refer to 921

abbreviation list for definitions.) 922

923

924

8.1 Re-evaluation 925

WHO reserves the right to conduct a formal re-evaluation of the WLA to assess the impact on 926

the continued validity of the listing of proposed or implemented changes or of events of a 927

significant and serious nature. The ‘for cause’ re-evaluation would follow prior notification of 928

the WLA through a Notice of Concern (NOC) signed by the ADG, Access to Medicines and 929

Heath Products Division. The NOC should describe the nature of the concern and the 930

proposed re-evaluation plan. The re-evaluation plan would include TORs for an on-site 931

assessment. An abbreviated version of the NOC describing the nature of concern would be 932

published on the WHO website (Fig. 7). 933

The first step in the re-evaluation process would normally involve a virtual or on-site meeting 934

with the senior management of the WLA to discuss the concern and the process for the re-935

evaluation, including timelines and responsibilities. 936

WA

G m

eeti

ng

req

ues

t3

0 d

ays

Active ongoing monitoring by WHO

NOC &re-evaluation

plan

Re-evaluationreport

Two-levelreview

NOLD-Rev

Report preparation

14 days

Listing


45 days

Listingdecision

Re-evaluation

Special WAG meeting

WAGreport

Recommendation to list/delist

NOLD-Rev

NOD/Delisting

Listing


45 days


Page 38

A re-evaluation team will be formed from WHO staff, supplemented, as required, by external 937

experts from the list of qualified assessors. 938

The WLA is expected to provide timely access to the necessary individuals, information, and 939

facilities as outlined in the re-evaluation plan and TOR for on-site assessment. 940

The re-evaluation team will prepare a report and a recommendation on the continued listing 941

of the WLA upon completing the re-evaluation. The report would include evidence reviewed, 942

findings and conclusions, and, as appropriate, recommended actions to resolve the concern. 943

The report is then forwarded to the ADG. Based on findings, conclusions, and 944

recommendations in the Re-evaluation report, the ADG may decide that: 945

a. The WLA should continue to be listed. 946

The decision is communicated to the WLA through a Notice of Listing Decision – Re-947

evaluation (NOLD-REV) signed by the ADG. The endorsed Re-evaluation report will 948

accompany the NOLD-REV together with a summary of the concern, evidence 949

reviewed, conclusion and listing recommendation, and re-evaluation timelines. The 950

summary is published as an update to the listing on WHO website. The WLA has 14 951

working days to review and provide comment on the summary prior to publication. 952

b. The WLA should continue to be listed provided recommended corrective actions are 953

undertaken according to agreed plan and timelines. 954

Ongoing monitoring of progress would also constitute a condition for continued 955

listing. The decision is communicated to the WLA through a NOLD-REV signed by the 956

ADG. The endorsed Re-evaluation report will accompany the NOLD-REV. 957

A meeting is scheduled with the WLA to discuss and seek agreement with conditions 958

for continued listing, which may reflect subsequent adjustments as endorsed by the 959

WLA and WHO. A summary of the concern, evidence reviewed, conclusions, and 960

listing recommendation and process is forwarded to the WLA for review and comment 961

at least 14 working days prior to publication as a listing update on the WHO website. 962

Should the WLA not agree with or refuse to undertake recommended actions, the 963

ADG may decide to convene a special meeting of the WAG to make a recommendation 964

on delisting. 965

c. Grounds for delisting the WLA exist. 966


Page 39

The conclusion is communicated to the WLA through a NOLD-REV signed by the ADG. The 967

endorsed Re-evaluation report will accompany the NOLD-REV together with a 968

description of the grounds for the possible delisting. The NOLD-REV will describe the 969

process for possible delisting, including the convening of the WAG and the right of 970

appeal. The NOLD-REV is not published until the final decision on listing or delisting is 971

taken (see section 8.2 - Delisting). 972

973

974

8.2 Delisting 975

WHO reserves the right to delist a regulatory authority or RRS if, upon evaluation and 976

subsequent discussion with the regulatory authority or RRS, WHO concludes that the basis 977

for supporting the listing is no longer valid. This could be due to evidence of serious and 978

persistent lapses in regulatory oversight, ineffective enforcement activities, a major 979

downsizing or re-organization of the authority, concerns regarding the independence and 980

integrity of the decision-making process, lack of response to requests for re-evaluation, or 981

refusal to undertake recommended corrective actions following the re-evaluation. 982

Upon receipt of the Re-evaluation report and a recommendation to delist, the ADG requests 983

that the RSS Team convene a special meeting of the WAG to make an independent 984

recommendation on delisting. In exceptional circumstances, the ADG may choose to proceed 985

with delisting the WLA without convening the WAG if concerns are considered to be of an 986

urgent public health nature. 987

The WAG should be convened no later than 30 calendar days following the ADG’s request 988

(Fig. 7). The WAG is provided the Re-evaluation report, the two-level review (i.e., by the RSS 989

Team Lead and the REG Unit Head), and any official response from the WLA. The WLA will be 990

invited to present its case before the WAG meeting. The WAG report shall be addressed to 991

the ADG no later than 14 (fourteen) calendar days following the meeting. 992

The ADG will issue a recommendation on listing or delisting based on the Re-evaluation 993

report, interventions by the WLA, and recommendations from the WAG. The decision is 994

communicated to the WLA by ADG in the form of either: 995

a. Notice of Delisting (NOD) in the case of decision to delist, or 996


Page 40

b. NOLD-REV, if a decision to maintain the listing is made based on the recommendation 997

of the WAG and/or the WLA’s consent to undertake actions to address the identified 998

concern. 999

A virtual or in-person meeting will be convened with the regulatory authority or RRS prior to 1000

the issuance of a NOD in order to explain the rationale for and implications of the decision, 1001

to clarify any misunderstandings, to outline the process of delisting, and to clarify the 1002

procedure for appealing the decision. A draft of the NOD will be provided to the authority or 1003

RRS in advance of the meeting, together with the re-evaluation report and WAG report. 1004

The NOD will include the nature of the concern, evidence reviewed, and process followed 1005

(including the convening of the WAG). If the reason(s) for delisting has any impact also on 1006

the overall ML of the NRA, the NOD will indicate potential consequences on prequalified 1007

medical products. Actions taken in relation to Prequalified products will be specified on the 1008

WHO PQ website. 1009

The responsibility for determining the implications of delisting on external users resides with 1010

the users and may depend on the nature of the concern and the specific context of the WLA 1011

listing in question. 1012

The NOD is made public on the WHO website and the WLA is delisted immediately following 1013

issuance of the NOD to the regulatory authority or RRS. 1014

The regulatory authority or RRS has the right to appeal a negative listing decision, as outlined 1015

in section 6.5 – Decision on listing. The rationale for contesting the decision must be based 1016

on procedural grounds and/or evidence available at the time of the decision to issue a NOD. 1017

The regulatory authority or RRS may reapply for listing by submitting a new EOI that includes 1018

a proposed corrective action plan. 1019

Note: Delisting of a WLA could also be a result of a voluntary decision by the listed regulatory authority 1020

or RRS. A statement to this effect would accompany delisting on WHO website, together with any 1021

consequences in terms of PQ, if the reason of delisting has an impact on the overall ML of the NRA. 1022

1023


Page 41

9. Information management system 1024

Similar to benchmarking of regulatory systems, the PEP is also an information intensive 1025

exercise that involves the collecting, sharing, processing, analysing and storing of large 1026

amounts of information, much of which is of a confidential nature. 1027

The aforementioned information, including all electronic copies of documents collected prior 1028

to, during or following the evaluation process, as well as all correspondence, analyses, 1029

presentations, and interim and final reports, will be stored, with restricted access, in the 1030

secured WHO NRA information sharing platform known as RSS SharePoint. This is the same 1031

platform that is utilised for benchmarking of NRAs. 1032

All archives are kept as electronic documents and distributed only on a need-to-know basis. 1033

Requests for official copies of master file documents should be submitted to WHO 1034

Headquarters. Distribution of copies, either inside or outside WHO, is possible only if WHO 1035

has obtained clearance from the NRA. The RSS SharePoint is access restricted. Access to users 1036

will be granted by the administrator only after the confidentiality agreement and the 1037

Declaration of Interests are signed. 1038

The final evaluation report is posted on the NRA information sharing platform. Distribution of 1039

the report is restricted to NRA and WHO staff, including WHO regional and country staff who 1040

either have been involved in the benchmarking or have responsibility for its oversight. Prior 1041

to its issuance, the Regional Office and Country Office will be requested to specify those to 1042

whom the report shall be distributed and should ensure confidentiality of all reports 1043

distributed. A list of all WHO staff receiving the report should be communicated to WHO 1044

Headquarters. 1045

1046

10. Monitoring and evaluation 1047

The introduction of the WLA framework is expected to have a positive impact on the global 1048

access to medical products with assured safety, efficacy and quality. The Framework will 1049

establish a transparent, consistent and reliable mechanism to identify robust regulatory 1050

systems operating at high level of performance, whose regulatory decisions will be used to 1051

streamline regulatory processes and decisions by authorities relying on them, avoid 1052

duplication of effort, and increase the efficiencies in overall regulatory oversight of medical 1053


Page 42

products. The impact is expected to evolve over time, depending on the number of regulatory 1054

bodies that will gradually complete the process and achieve listing. 1055

In this respect, specific performance indicators and methods aimed at assessing the level of 1056

implementation of the WLA framework and its impact in the different WHO regions need to 1057

be established. Possible measurement indicators include the number of NRAs and RRSs being 1058

listed compared to the number of applications received, the compliance to agreed upon 1059

timelines, the number of NRAs relying on listed WLAs, and the number of WLAs being relied 1060

upon by the WHO PQ team. 1061

1062

1063

11. Glossary 1064

Common regulatory framework 1065

A common regulatory framework is a unified set of requirements, processes and controls 1066

used by a Regional Regulatory System (RRS) and applied to the oversight of medical products. 1067

The common regulatory framework ensures equivalence among the members in terms of 1068

regulatory requirements, regulatory practices, and quality assurance policies. 1069

1070

Common legal framework 1071

Unified legislation which underpins the common regulatory framework in a Regional 1072

Regulatory System (RRS). The common legal framework provides the RRS and individual 1073

members enforcement powers to ensure compliance with the common regulatory 1074

framework. 1075

1076

Delisting 1077

Removal of the regulatory body from the publicly available list of WHO Listed Authorities 1078

(WLAs), due to evidence of serious and persistent lapses in regulatory oversight that have an 1079

impact on the function(s) and/or product category(ies) listed. 1080

1081

Effectiveness 1082

The extent to which a specific intervention, procedure, regimen or service, when deployed in 1083

the field in routine circumstances, does what it is intended to do for a specified population. 1084

1085

Efficacy 1086

The extent to which a specific intervention, procedure, regimen or service produces the 1087

intended result under ideal conditions. 1088

1089

Efficiency 1090

The capacity to produce the maximum output for a given input. 1091


Page 43

1092

Export-only products 1093

Products not marketed nor registered in the producing country. Such products are often 1094

excluded from the remit of regulatory bodies. 1095

1096

International standards and guidelines 1097

For the purpose of this document, the term includes relevant WHO standards and guidelines 1098

and any other relevant internationally recognized standards (e.g., ISO or pharmacopoeial 1099

standards) and guidelines (e.g., International Council for Harmonisation of Technical 1100

Requirements for Pharmaceuticals for Human Use (ICH) or Pharmaceutical Inspection 1101

Convention and Pharmaceutical Inspection Co-operation Scheme (PIC/S) guidelines). 1102

1103

Listing 1104

Public designation, on the WHO website, of reference regulatory authorities that have 1105

successfully undergone the performance evaluation process in the context of the WHO Listed 1106

Authority (WLA) framework. Listing as a WLA includes the scope of the designation (product 1107

categories and/or regulatory functions); the evidence reviewed, and the process undertaken 1108

to support the listing. The listing also specifies the original date and the period of validity of 1109

the initial listing. 1110

1111

Maturity level (ML) 1112

Maturity of regulatory systems is divided into four levels: (ML 1) some elements of regulatory 1113

systems exist; (ML 2) evolving national regulatory system that partially performs essential 1114

regulatory functions; (ML 3) stable well-functioning and integrated regulatory system; and 1115

(ML 4) regulatory system operating at advanced level of performance and continuous 1116

improvement. 1117

1118

Performance 1119

The degree to which regulatory inputs and processes consistently and efficiently result in 1120

desired regulatory outputs. 1121

1122

Performance Evaluation Process (PEP) 1123

Set of activities, specific for each regulatory function, aimed at assessing how a regulatory 1124

system operates and evaluating its level of performance and effectiveness. The process 1125

considers the nature and extent of evaluation and is designed to address the function(s) 1126

and/or the product category(ies) included in the Expression of Interest. 1127

1128

Product category(ies) 1129

For the purpose of this document, this refers to “medical products” which may include the 1130

following product categories: multisource (generics), new medicines (new chemical entities), 1131

biotherapeutics, similar biotherapeutic products and vaccines. 1132


Page 44

1133

Re-evaluation 1134

Assessment of the impact of proposed or implemented changes, or of events of a significant 1135

and serious nature that may impact the continued validity of listing. 1136

1137

Regional regulatory system (RRS) 1138

A system composed of individual regulatory authorities, or a regional body composed of 1139

individual regulatory authorities, operating under a common regulatory framework that may 1140

include or exclude a common legal framework. The common framework must at least ensure 1141

equivalence between the members in terms of regulatory requirements, practices and quality 1142

assurance policies. The system or regional body, where it exists, may have enforcement 1143

powers to ensure compliance with the common regulatory framework. A regional regulatory 1144

system (RRS) so described may be considered a single entity and therefore eligible for listing 1145

as a WLA. Each of the individual authorities that are part of the system are also eligible for 1146

listing consideration. In cases where an RRS is further underpinned by a common legal 1147

framework, it should be considered as a single entity and, as such, eligible for listing as a WLA, 1148

as well as each of the individual authorities that are part of the system. 1149

1150

Regulatory function 1151

WHO defines a national regulatory system in terms of the enabling legal system and 1152

infrastructure related to eight common regulatory functions and one non-common regulatory 1153

function. Common regulatory functions that apply to all medical products are regulatory 1154

system (RS), registration and marketing authorization (MA), vigilance (VL), market surveillance 1155

and control (MC), licensing establishments (LI), regulatory inspection (RI), laboratory testing 1156

(LT), and clinical trials oversight (CT). National lot release (LR) for vaccines represents the non-1157

common regulatory function. 1158

1159

Renewal 1160

Process of re-assessment of a WLA, aimed at extending the validity of listing. Initial listing is 1161

valid for five years. 1162

1163

WHO Listed Authority (WLA) 1164

A WHO Listed Authority (WLA) is a regulatory authority or a regional regulatory system which 1165

has been documented to comply with all the indicators and requirements specified by WHO 1166

for the requested scope of listing based on an established benchmarking and performance 1167

evaluation process. 1168

1169

WLA Framework 1170

Set of documents, principles, guidelines, tools and processes for designating regulators 1171

responsible for regulation of medical products as WHO Listed Authorities (WLAs), meant to 1172

provide a transparent and evidence-based pathway globally recognized. 1173


Page 45

12. References 1174

1. WHA Resolution 67.20 - Regulatory system strengthening for medical products (2014) 1175

https://apps.who.int/gb/ebwha/pdf_files/WHA67/A67_R20-en.pdf 1176

2. Roadmap for access to medicines, vaccines and health product 2019-2023: 1177

comprehensive support for access to medicines, vaccines and other health products 1178

https://apps.who.int/iris/handle/10665/330145 1179

3. Policy document: Evaluating and publicly designating regulatory authorities as WHO-1180

listed authorities 1181

(https://www.who.int/publications/i/item/9789240023444) 1182

4. Global Benchmarking Tool (GBT) and Manual 1183

(https://www.who.int/publications/i/item/9789240020245) 1184

5. Good regulatory practices for regulatory oversight of medical products 1185

WHO TRS 1033, Annex 11 1186

https://apps.who.int/iris/bitstream/handle/10665/340323/9789240020900-eng.pdf 1187

6. Good reliance practices in regulatory decision-making: high-level principles and 1188

recommendations 1189

WHO TRS 1033, Annex 10 1190


7. The implementation of quality management systems for national regulatory authorities 1192

https://www.who.int/publications/m/item/trs-1025-annex-13-qms-nra 1193

8. Implementing quality management systems in national regulatory authorities: 1194

examples and practices 1195


1197

13. Annexes and Appendices 1198

Annex 1: Expression of Interest application form 1199

Annex 2: Agreement on the conditions to be designated a WHO Listed Authority 1200

Annex 3: Sample listing 1201

Annex 4: Technical Advisory Group on WLA (WAG) - Terms of Reference 1202

Annex 5: WLA Renewal Reporting Form 1203

Annex 6: Performance evaluation tools and methodologies by regulatory function 1204

Appendix 6.1: Field Visit Manual for Assessing the Performance of Vigilance Function 1205

https://apps.who.int/gb/ebwha/pdf_files/WHA67/A67_R20-en.pdf

https://apps.who.int/iris/handle/10665/330145

https://www.who.int/publications/i/item/9789240023444


https://apps.who.int/iris/bitstream/handle/10665/340323/9789240020900-eng.pdf


https://www.who.int/publications/m/item/trs-1025-annex-13-qms-nra



Page 46

Appendix 6.2: GxP Observed Audit Manual for Assessing the Performance of Regulatory 1206

Inspection Function 1207


Page 47

Annex 1 – Template for Expression of Interest to be designated as WHO Listed

Authority

REGULATORY SYSTEMS STRENGTHENING Evaluating and publicly designating NRA/RRS as WHO Listed

Authority

Working document WLA OpG Rev. 1 July 2021

Expression of Interest (EOI) of National Regulatory Authority or Regional Regulatory 1208

System to be listed as WHO Listed Authority (WLA) 1209

1210 1211

Information contained in this FORM is strictly confidential 1212

1213 1214

1. NRA or RRS general information

Name of the regulatory body applying for listing:

Country or Region3:

Address of the organization:

Contact person name:

Contact person role in the organization:

E-mail address: Tel. no:

1215

2. Eligibility criteria

NRA or RRS

WHO Benchmarked: Yes No

If yes, please indicate

last date of benchmarking: overall ML reached:

Additional information related to RRS only

Member NRAs WHO benchmarked: Yes No

If yes, please list all NRAs WHO benchmarked (add fields as necessary):

NRA: Country:

last date of benchmarking: overall ML reached:

1216

3. Listing application

NRA or RRS

Please indicate the regulatory function(s) applying for listing:

3 For Regional Regulatory Systems indicate the region represented by the regulatory body, including the list of member NRAs.


Page 49

RS MA VL MC LI RI LT CT LR

Please indicate the product category(ies) applying for listing:

Multisource (generics)


Vaccines

Other


Common legal framework present: Yes No

RRS applying as single entity4: Yes No

1217 1218 Further information 1219 For further information on WHO listing process please visit WLA website at: 1220 https://www.who.int/initiatives/who-listed-authority-reg-authorities 1221 1222 For further information on WHO benchmarking process please visit WHO website at: 1223 https://www.who.int/tools/global-benchmarking-tools 1224 1225 If you have any questions relating to the procedure for applying to WLA, please write to WHO RSS 1226 Team at the following email address [email protected] 1227 Your question(s) will be directed to the Regulatory System Strengthening team member who can 1228 best advise you. 1229 1230 1231 References 1232 WHA Resolution 67.20 - Regulatory system strengthening for medical products (2014) 1233

https://apps.who.int/gb/ebwha/pdf_files/WHA67/A67_R20-en.pdf 1234 1235

Roadmap for access to medicines, vaccines and health product 2019-2023: comprehensive support 1236 for access to medicines, vaccines and other health products 1237

https://apps.who.int/iris/handle/10665/330145 1238 1239

WLA Policy document: Evaluating and publicly designating regulatory authorities as WHO-listed 1240 authorities 1241

https://www.who.int/publications/i/item/9789240023444 1242 1243

Global Benchmarking Tool (GBT) and Manual 1244 https://www.who.int/publications/i/item/9789240020245 1245

1246 Good regulatory practices for regulatory oversight of medical products 1247 WHO TRS 1033, Annex 11 1248

4 RRS with a common legal framework are entitled to apply as single entity, and as such can be listed as WLA together with each of the individual authorities that form the system through the same evaluation process.


https://www.who.int/tools/global-benchmarking-tools






50

Working document WLA OpG Rev. 1 Page

https://apps.who.int/iris/bitstream/handle/10665/340323/9789240020900-eng.pdf 1249 1250

Good reliance practices in regulatory decision-making: high-level principles and recommendations 1251 WHO TRS 1033, Annex 10 1252


The implementation of quality management systems for national regulatory authorities 1255 https://www.who.int/publications/m/item/trs-1025-annex-13-qms-nra 1256

1257

Implementing quality management systems in national regulatory authorities: examples and 1258 practices 1259







Page 51

Annex 2 – Agreement on the conditions to be designated as WHO Listed

Authority

This Annex is subject to review and approval by WHO LEG Office


Authority

52


BACKGROUND 1261

The principle of reliance is central to WHO’s approach to regulatory system strengthening 1262

and effective regulation. Reliance is an increasingly important strategy in confronting the 1263

challenges posed by globalization, technology, workload pressures and growing public 1264

expectations. The introduction of a framework for designating and publicly listing a 1265

regulatory authority as a WLA is expected to promote a sound and transparent approach 1266

to reliance by providing a transparent and evidence-based pathway for regulatory 1267

authorities to be globally recognized as meeting WHO and other international recognized 1268

standards and practices. 1269

The construction of a WHO list of Authorities which have reached high level regulatory 1270

capacity will lead to an improved recognition of regulatory decisions taken by other 1271

Authorities, and consequently to avoidance of overlapping activities, better use of human 1272

and economic resources, increased oversight of the pharmaceutical products along the 1273

whole supply chain, enhanced global access to high quality, efficacious and safe medicines 1274

and vaccines. 1275

With the aim of strengthening the collaboration with WHO and with other National 1276

Regulatory Authorities worldwide, Country ________, NRA/RRS _________ expressed its 1277

interest to achieve the WHO Listed Authority status for the following product stream: 1278

Multisource (generics) / New medicines (new chemical entities) and/or biotherapeutics 1279

and/or similar biotherapeutic products / Vaccines 1280

and/or the following regulatory functions: 1281

Marketing Authorization / Vigilance / Market Control / Licensing establishments / 1282

Regulatory Inspections / Clinical Trial Oversight / Laboratory Testing / Lot Release. 1283

In addition, in line with the principles laid down in the WLA Operational Guidance, the NRA 1284

of Country ____________ is aware that the overarching Regulatory System function, being 1285

the backbone and foundation of any regulatory authority, is always included in the 1286

evaluation process, in conjunction with any function or product category which applies for 1287

listing. 1288

1289

1290


Page 53

GOALS 1291

NRA/RRS __________ is recognized as a regulatory system operating continually at 1292

advanced level of performance that achieves enhanced international level of regulatory 1293

capacity and leadership in the area of (Multisource (generics) / New medicines (new 1294

chemical entities) and/or biotherapeutics and/or similar biotherapeutic products / 1295

Vaccines) regulation, and such it is included in the publicly available WHO list of advanced 1296

regulatory authorities (WHO Listed Authority). 1297

1298

COMMITMENTS 1299

WHO commits to: 1300

• Hold all information pertaining to the submission of an EOI and subsequent steps 1301

in confidence and through the use of secure platforms, until and as restricted to 1302

the listing of the regulatory authority or RRS as a WLA. 1303

A separate confidentiality agreement between WHO and NRA/RRS will be signed, 1304

if requested. 1305

• Respect the agreed timeframe and concur to arrange any drift from the original 1306

plan. 1307

1308

NRA/RRS _________ commits to: 1309

• Assign sufficient resources to the WLA process and provide access to requested 1310

documents and relevant (responsible) individuals. 1311

• Respect the agreed timeframe and concur to arrange any drift from the original 1312

plan. Should it become evident that the NRA/RRS is unable to meet requirements 1313

for listing within the agreed upon timeframe, the evaluation process will be 1314

interrupted and the authority or RRS will be invited to re-apply at a later time once 1315

identified areas for improvement have been addressed. 1316

• Accept public disclosure of positive outcome (listing) and further disclosure on 1317

secure regulatory platform. 1318

• Make regulatory decisions available to other regulatory authorities in the form of 1319

publicly available information and/or through information-sharing mechanisms. 1320

54


• Should listing be achieved, comply with the provisions of interim reporting in case 1321

of major changes, and renewal reporting. 1322

KEY ACTIVITIES 1323

1. Agreement on scope and roadmap: 1324

NRA/RRS _________ and WHO will jointly develop a roadmap and project plan for 1325

reaching a listing decision. The roadmap will consider all available evidence already 1326

hold by the NRA/RRS that may contribute to an abbreviated evaluation pathway 1327

(parallel assessment or abridged pathway). The roadmap will delineate the 1328

Performance Evaluation methodology and activities to be conducted per each 1329

function as well as tentative timelines with milestones and decisions points. The 1330

roadmap could be adjusted, as required, based on periodic reevaluation of progress 1331

being made. 1332

1333

2. Identification of contact persons: 1334

The following focal points are identified for the Performance Evaluation Process: 1335

WHO NRA/RRS__________

Contact

person

Role Contact

details

Contact

person

Role Contact

details

1336

Any modification to the above-mentioned list of contact persons is promptly 1337

communicated. 1338

1339

3. Risk-based approach (for RRSs only): 1340

Regional Regulatory Systems provided with a common legal framework that 1341

ensures the existence and enforcement in all member NRAs of relevant provisions 1342

to control medicinal products over their entire lifecycle, can achieve listing together 1343

with all member NRAs (single entity). 1344

RRSs that expressed their interest to be considered as single entity will be assessed 1345

together with individual member NRAs. The number of individual member NRAs to 1346

be assessed, as well as the functions for which the Performance Evaluation Process 1347

will be conducted, are selected according to a risk-based approach. 1348


Page 55

Annex 3 – Sample Listing


Authority

56


1349

1350 1351

1352

The list will be updated regularly as new information becomes available. 1353 1354

1355 1356

1357

1358

1359

1360

1361

1362

1363

1364

1365 1366 1367 1368 1369 1370 1371 1372 1373 1374

5 WLA have reached overall Maturity Level 3 or 4 6 Product stream can be multisource (generics), new medicines (new chemical entities) and/or biotherapeutics and/or biosimilar products, vaccines 7 RS Regulatory System, MA Marketing Authorization, VL Vigilance, MC Market Control, LI Licensing establishments, RI Regulatory Inspections, LT Laboratory Testing, CT Clinical Trials oversight, LR Lot Release

Country

Regulatory Authority /Regional

Regulatory System

Maturity Level5

Product stream(s)6

Listed function(s)7

Date of first listing

Date of renewal

Link to the

summary report

A

B

C

D

E

F


Page 57

Annex 4 – WLA Technical Advisory Group (WAG)

This Annex is subject to review and approval by WHO LEG Office


Authority

58


I. Background 1375

1376

WHO has developed the WLA framework to promote trust, confidence and reliance between 1377

regulatory authorities; encourage continuous improvement of regulatory systems and 1378

efficient use of regulatory resources; expand the pool of regulatory authorities contributing 1379

to the efficiency of the WHO Prequalification (PQ) programme through the increased use of 1380

abridged/streamlined procedures to PQ listing; promote the supply of quality assured 1381

medical products for use by UN procurement agencies and countries; and create an enabling 1382

environment for innovation and local production of medical products. 1383

1384

A regulatory authority or RRS that has attained overall maturity level (ML) 3 through the GBT 1385

is eligible for listing and has access to the Performance Evaluation Process of the relevant 1386

function(s). This implies that the overarching regulatory system and all involved regulatory 1387

functions must at least comply with respective sub-indicators of ML1, ML2 and ML3. The 1388

Performance Evaluation Process builds upon the GBT and provides a mechanism to establish 1389

the regulatory performance of the NRA or RRS, through a set of additional activities 1390

specifically designed for each regulatory function. 1391

In this context, the WHO RSS Secretariat will require advice from an independent advisory 1392

group, known as the WLA technical Advisory Group (WAG), on whether or not these 1393

assessed NRAs/RRSs can be included among the WHO Listed Authorities. 1394

The aforementioned Advisory Group (WAG) will act as an advisor body to WHO in this field. 1395

1396

II. Functions 1397

1398

In its capacity as an advisory body to WHO, the WAG shall have the following functions: 1399

1400

1. To review the WLA team reports prepared by the WHO team of assessors as part of 1401

the WLA Performance Evaluation Process, including the eligibility criteria; 1402

2. To provide a recommendation to WHO if the assessed NRA or RRS should be listed 1403

as WLA, and which designation(s) should be included in the listing. 1404

3. To provide a recommendation to WHO in case of re-evaluations of NRAs/RRSs 1405

following a Notice of Concern. 1406

1407

III. Composition 1408

1409

1. The WAG will be composed of twelve (12) members8, six prime and six alternates, 1410

two each from the six (6) WHO regions, who shall serve in their personal capacities 1411

to represent the broad range of disciplines relevant to assessment of NRAs/RRSs: 1412

knowledge of regulatory systems, quality management systems, marketing 1413

authorization activities, pharmacovigilance, market surveillance and control, 1414

licensing of premises, GMP/GDP/GCP inspections, clinical data evaluations, 1415

8 Members serve as full participants and partake in the decision -making process of the procedure/meeting in

which they are involved


Page 59

analytical testing and lot release. In the selection of the WAG members, 1416

consideration shall be given to attaining an adequate distribution of technical 1417

expertise, geographical representation and gender balance. Membership is 1418

restricted to senior officials from regulatory authorities and regional regulatory 1419

systems. 1420

1421

2. Members of the WAG shall be appointed by WHO from candidates proposed by 1422

Member States following an EOI issued by the respective WHO Regional Offices. A 1423

Chair will be selected by members. The Chair will preside over meetings, facilitate 1424

discussions, help formulate WAG deliberations and recommendations, ensure 1425

adherence to timelines and serve as the primary liaison with the WHO. 1426

1427

3. Members are expected to serve for a period of 3 years, with staggered rotation from 1428

alternate members and shall be eligible for reappointment. 1429

A Chairperson is eligible for reappointment as a member of the WAG, but is only 1430

permitted to serve as Chairperson for one term. Their appointment and/or 1431

designation as Chairperson may be terminated at any time by WHO if WHO's interest 1432

so requires or, as otherwise specified in these terms of reference or letters of 1433

appointment. Where a member’s appointment is terminated, WHO may decide to 1434

appoint a replacement member. 1435

1436

4. WAG members must respect the impartiality and independence required by WHO. 1437

In performing their work, members may not seek or accept instructions from any 1438

Government or from any authority external to the Organization. They must be free 1439

of any real, potential or apparent conflicts of interest. To this end, proposed 1440

members/members shall be required to complete a declaration of interests form and 1441

their appointment, or continuation of their appointment, shall be subject to the 1442

evaluation of completed forms by the WHO Secretariat, determining that their 1443

participation would not give rise to a real, potential or apparent conflict of interest. 1444

A member who is from the regulatory authority or RRS under discussion must be 1445

recused from WAC proceedings. Members and observers are bound to provisions 1446

on confidentiality during and following their participation in WAC proceedings. 1447

1448

5. Following a determination that a proposed member’s participation in the WAG 1449

would not give rise to a real, potential or apparent conflict of interest, the proposed 1450

member will be sent a letter inviting them to be a member of the WAG. Their 1451

appointment to the WAG is subject to WHO receiving the countersigned invitation 1452

letter and letter of agreement. Notwithstanding the requirement to complete the 1453

WHO declaration of interest’s form, WAG members have an ongoing obligation to 1454

inform the WHO of any interests real or perceived that may give raise to a real, 1455

potential or apparent conflict of interests. 1456

1457

6. As contemplated in paragraph II.4 above, WHO may, from time to time, request WAG 1458

members to complete a new declaration of interest form. This may be before a WAG 1459

meeting or any other WAG-related activity or engagement, as decided by WHO. 1460

Where WHO has made such a request, the WAG member’s participation in the 1461

60


advisory group activity or engagement is subject to a determination that their 1462

participation would not give rise to a real, potential or apparent conflict of interest. 1463

1464

7. Where a WAG member is invited by WHO to travel to an in-person advisory group 1465

meeting, WHO shall, subject to any conflict-of-interest determination as set out in 1466

paragraph II.7 above, issue a letter of appointment as a temporary adviser and 1467

accompanying memorandum of agreement (together Temporary Adviser Letter). 1468

WHO shall not authorize travel by an WAG member, until it receives a countersigned 1469

Temporary Adviser Letter. 1470

1471

8. WAG members do not receive any remuneration from the Organization for any work 1472

related to the WAG. However, when attending in-person meetings at the invitation 1473

of WHO, their travel cost and per diem shall be covered by WHO in accordance with 1474

the applicable WHO rules and policies. 1475

1476

IV. Operations 1477

1478

1. The WAG shall meet at least once at the end of the Performance Evaluation Process, 1479

when the WLA team of assessors has finalized the WLA team report. WAG meetings 1480

will be convened by WHO and will predominantly be held virtually and in closed 1481

sessions, via video or teleconference. 1482

1483

2. The quorum for WAG meetings shall be two thirds of the members. 1484

1485

3. External observers may be invited to attend meetings, at the discretion of the Chair 1486

and the consent of the WAG. Observers may be invited either in their personal 1487

capacity, or as representatives from donor organizations in Official Relations with 1488

WHO, other interested UN agencies involved in the procurement and distribution of 1489

medical products, regional regulatory bodies, organizations collaborating with WHO 1490

under the Coalition of Interested Parties (CIP) Regulatory Systems Strengthening 1491

Network, and representatives from regulatory authorities or RRSs interested in 1492

serving as future members. The WLA Team Lead may be requested to speak at the 1493

meeting to respond to questions the WAG may have in reaching a recommendation. 1494

WHO will request observers to complete a confidentiality undertaking and a 1495

declaration of interests form prior to attending a session of the advisory group. 1496

Observers shall normally attend meetings of the WAG at their own expenses and be 1497

responsible for making all arrangements in that regard. At the invitation of the 1498

Chairperson, observers may be asked to present their personal views and/or the 1499

policies of their organization. Observers will not participate in the process of adopting 1500

decisions and recommendations of the WAG. 1501

1502

4. WAG members are expected to attend meetings. If a member misses two consecutive 1503

meetings, WHO may end his/her appointment as a member of the AG. 1504

1505

5. Reports of each meeting, together with the recommendation on NRA/RRS listing 1506

under discussion, shall be submitted by the WAG to WHO not later than 60 days after 1507

receipt of the WLA team report prepared by the WHO team of assessors as part of the 1508

WLA Performance Evaluation Process. All recommendations from the WAG are 1509


Page 61

advisory to WHO, who retains full control over any subsequent decisions or actions 1510

regarding any proposals, policy issues or other matters considered by the WAG. 1511

1512

6. The WAG shall normally make recommendations by consensus. If, in exceptional 1513

circumstances, a consensus on a particular issue cannot be reached, minority opinions 1514

will be reflected in the meeting report. 1515

1516

7. Active participation is expected from all WAG members, including in working groups, 1517

teleconferences, and interaction over email. WAG members may, in advance of WAG 1518

meetings, be requested to review meeting documentation and to provide their views 1519

for consideration by the WAG. 1520

1521

8. WHO shall determine the modes of communication by the WAG, including between 1522

WHO and the WAG members, and the WAG members among themselves. 1523

1524

9. WAG members shall not speak on behalf of, or represent, the WAG or WHO to any 1525

third party. 1526

1527

V. Secretariat 1528

1529

WHO RSS shall provide the secretariat for the WAG, including necessary scientific, 1530

technical, administrative and other support. In this regard, the WHO Secretariat shall 1531

provide the members in advance of each meeting with the agenda, working 1532

documents and discussion papers. Distribution of the aforesaid documents to 1533

Observers will be determined by the WHO Secretariat. The meeting agenda shall 1534

include details such as: whether a meeting, or part thereof, is closed or open; and 1535

whether Observers are permitted to attend. 1536

1537

VI. Information and documentation 1538

1539

1. Information and documentation to which members may gain access in performing 1540

WAG related activities shall be considered as confidential and proprietary to WHO 1541

and/or parties collaborating with WHO. In addition, by counter signing the letter of 1542

appointment and the accompanying terms and conditions referred to in section II.5 1543

above, WAG members undertake to abide by the confidentiality obligations contained 1544

therein and also confirm that any and all rights in the work performed by them in 1545

connection with, or as a result of their WAG-related activities shall be exclusively 1546

vested in WHO. 1547

1548

2. WAG members and Observers shall not quote from, circulate or use WAG documents 1549

for any purpose other than in a manner consistent with their responsibilities under 1550

these Terms of Reference. 1551

1552

3. WHO retains full control over the publication of the reports of the WAG, including 1553

deciding whether or not to publish them. 1554

1555

62


Annex 5 – Renewal reporting form for WHO Listed Authorities


Authority


Page 63

Renewal Reporting Form 1556

for WHO Listed Authorities (WLA) 1557

1558 Information contained in this FORM is strictly confidential 1559

1560

1561

1. NRA or RRS general information

Name of the regulatory body applying for listing:

Country or Region9:

Address of the organization:

Contact person name:

Contact person role in the organization:

E-mail address: Tel. no:

1562

1563

2. WLA status

WHO Benchmarking

Date of benchmarking: overall ML reached:

Listing

Date of first listing:

Please indicate the listed regulatory function(s):


Please indicate the listed product category(ies):



Vaccines

9 For Regional Regulatory Systems indicate the region represented by the regulatory body, including the list of

member NRAs.

Page 64



Member NRAs WHO benchmarked: Yes No

If yes, please list all NRAs WHO benchmarked (add fields as necessary):

NRA: Country:

date of benchmarking: overall ML reached:

Member NRAs assessed through PEP (for single entity RRS): Yes No

If yes, please list all NRAs WHO assessed through PEP (add fields as necessary):

NRA: Country:

Please indicate the listed regulatory function(s):


Please indicate the listed product category(ies):



Vaccines

1564 1565

3. Major changes10

Provide a brief description of major changes occurred since listing was achieved, including those already communicated in the interim reports to WHO, if any:

Description Date of the change

Documents provided11 Current status (ongoing, completed)

1566 1567 1568 1569 1570 1571

10 Changes to the regulatory framework, processes, resources, organizational structure and governance or other parameters which may have an impact on the listed function(s) and/or product category(ies) assessed as part of the WLA benchmarking-performance evaluation. 11 Accompanying information in electronic format should be uploaded on the WHO NRA SharePoint and the RSS Team Lead notified. If information is available on the WLA’s website, a link to the relevant webpages is acceptable in lieu of submitting documents, provided that information on the website is in English or summarized in English.


Page 65

Further information 1572 For further information on WHO listing process please visit WLA website at: 1573 https://www.who.int/initiatives/who-listed-authority-reg-authorities 1574 1575 For further information on WHO benchmarking process please visit WHO website at: 1576 https://www.who.int/tools/global-benchmarking-tools 1577 1578 If you have any questions relating to the procedure for applying to WLA, please write to WHO RSS 1579 Team at the following email address [email protected] 1580 Your question(s) will be directed to the Regulatory System Strengthening team member who can 1581 best advise you. 1582

1583 1584 References 1585 WHA Resolution 67.20 - Regulatory system strengthening for medical products (2014) 1586

https://apps.who.int/gb/ebwha/pdf_files/WHA67/A67_R20-en.pdf 1587 1588

Roadmap for access to medicines, vaccines and health product 2019-2023: comprehensive support 1589 for access to medicines, vaccines and other health products 1590

https://apps.who.int/iris/handle/10665/330145 1591 1592

WLA Policy document: Evaluating and publicly designating regulatory authorities as WHO-listed 1593 authorities 1594

https://www.who.int/publications/i/item/9789240023444 1595 1596

Global Benchmarking Tool (GBT) and Manual 1597 https://www.who.int/publications/i/item/9789240020245 1598

1599

Good regulatory practices for regulatory oversight of medical products 1600 WHO TRS 1033, Annex 11 1601


Good reliance practices in regulatory decision-making: high-level principles and recommendations 1604 WHO TRS 1033, Annex 10 1605


The implementation of quality management systems for national regulatory authorities 1608 https://www.who.int/publications/m/item/trs-1025-annex-13-qms-nra 1609

1610

Implementing quality management systems in national regulatory authorities: examples and 1611 practices 1612


1614 1615

1616


https://www.who.int/tools/global-benchmarking-tools










Page 66


Annex 6 – Performance Evaluation Process


Authority


Page 67

WHO-Listed Authorities Performance Evaluation 1617

1618

Contents 1619

Abbreviations 69 1620

Introduction 70 1621

1. Regulatory system (RS) 73 1622

1.1. RS PEP methodology 73 1623

1.2. Mandatory ML4 GBT sub-indicators for RS 73 1624

1.3. New RS performance evaluation indicators 74 1625

2. Registration and marketing authorization (MA) 80 1626

2.1. MA PEP methodology 80 1627

2.2. Mandatory ML4 GBT sub-indicators for MA 82 1628

2.3. New MA performance evaluation indicators 83 1629

2.4. MA new performance evaluation tools 95 1630

3. Vigilance (VL) 109 1631

3.1. VL PEP methodology 109 1632

3.2. Mandatory ML4 GBT sub-indicators for VL 110 1633

3.3. New VL performance evaluation indicators 110 1634

3.4. New VL performance evaluation tools 137 1635

4. Market surveillance and control (MC) 138 1636

4.1. MC PEP methodology 138 1637

4.2. Mandatory ML4 GBT sub-indicators for MC 138 1638

4.3. New MC performance evaluation indicators 139 1639

5. Licensing establishments (LI) 143 1640

5.1. LI PEP methodology143 1641

5.2. Mandatory ML4 GBT sub-indicators for LI 143 1642

6. Regulatory inspection (RI) 144 1643

6.1. RI PEP methodology 144 1644

6.2. Mandatory ML4 GBT sub-indicators for RI 145 1645

6.3. New RI performance evaluation tools 146 1646

7. Laboratory testing (LT) 147 1647

7.1. LT PEP methodology 147 1648

7.2. Mandatory ML4 GBT sub-indicators for LT 148 1649

7.3. New LT performance evaluation tools 152 1650

8. Clinical trials oversight (CT) 169 1651

8.1. CT PEP methodology 169 1652

8.2. Mandatory ML4 GBT sub-indicators for CT 170 1653

Page 68


8.3. New CT performance evaluation indicators 170 1654

8.4. New CT performance evaluation tools178 1655

9. NRA lot release (LR) 189 1656

9.1. LR PEP methodology 189 1657

9.2. Mandatory ML4 GBT sub-indicators for LR 190 1658


Page 69

Abbreviations 1659

1660

ADR Adverse Drug Reaction 1661

AEFI Adverse Event Following Immunization 1662

BEMA Benchmarking of European Medicines Agencies 1663

CPP Certificate of Pharmaceutical Product 1664

CT Clinical Trial/Clinical Trials oversight 1665

EDQM European Directorate for the Quality of Medicines & Healthcare 1666

GBT Global Benchmarking Tool 1667

GCP Good Clinical Practices 1668

GDP Good Distribution Practices 1669

GMP Good Manufacturing Practices 1670

GVP Good Vigilance Practices 1671

GXP Good Practices 1672

ICH International Council for Harmonisation of Technical Requirements for 1673

Pharmaceuticals for Human Use 1674

ICSR Individual Case Safety Reports 1675

IMP Investigational Medical Product 1676

ISO International Organization for Standardization 1677

KPI Key Performance Indicator 1678

LI Licensing establishments 1679

LR NRA Lot Release 1680

LT Laboratory Testing 1681

MA Marketing Authorization/registration and Marketing Authorization 1682

MAA Marketing Authorization Application 1683

MAH Marketing Authorization Holder 1684

MC Market surveillance and Control 1685

M&E Monitoring and Evaluation 1686

ML Maturity Level 1687

NCL National Control Laboratory 1688

NIP National Immunization Programme 1689

NRA National Regulatory Authority 1690

OMCL Official Medicines Control Laboratory 1691

PE Performance Evaluation 1692

PEP Performance Evaluation Process 1693

PIC/S Pharmaceutical Inspection Co-operation Scheme 1694

PMS Post-Market Surveillance 1695

PQ WHO Prequalification 1696

PSUR Periodic Safety Update Report 1697

PTS Proficiency Testing Scheme 1698

QMS Quality Management System 1699

RBA Risk-Based Approach 1700

RI Regulatory Inspections 1701

RRS Regional Regulatory System 1702

RS Regulatory System 1703

SBP Similar Biotherapeutic Products 1704

SOP Standard Operating Procedure 1705

Page 70


SRA Stringent Regulatory Authority 1706

TOR Terms of reference 1707

VL Vigilance 1708

WHO World Health Organization 1709

1710

1711

Introduction 1712

1713

The Global Benchmarking Tool (GBT) is the WHO’s primary tool for objectively evaluating 1714

regulatory systems to identify strengths and areas for improvement in the context of 1715

regulatory system strengthening. It is used to evaluate the overarching regulatory system 1716

(RS) as well as each component regulatory function, comprising: registration and marketing 1717

authorization, vigilance, market surveillance and control, licensing establishments, regulatory 1718

inspection, laboratory testing, clinical trials oversight, and NRA lot release. 1719

1720

The GBT also provides the foundation for designating WHO-Listed Authorities (WLAs). But it 1721

is complemented by a series of performance evaluation processes (PEPs) that are designed to 1722

establish a more detailed picture of how the regulatory system performs on relevant 1723

regulatory processes, including how consistently it adheres to quality procedures and how 1724

well it delivers the desired regulatory outputs and outcomes. 1725

1726

To be listed as a WLA for any given regulatory function, the applicant must, as a pre-requisite, 1727

fully implement all related GBT sub-indicators of maturity level (ML) 1–3. In addition, the 1728

NRA or RRS applying for WLA status must also acceptably meet the requirements of the 1729

relevant function PEP, which may include one or more of the following elements: 1730

• a set of mandatory ML4 GBT sub-indicators (that may or may not include additional 1731

requirements through a PEP expansion); 1732

• a set of newly developed performance evaluation indicators; and/or 1733

• a set of newly developed performance evaluation tools (see Figure A and Table A). 1734

1735

Note that the three areas of PEP assessment are sequential so that it is only possible to 1736

evaluate any newly developed indicators once all mandatory ML4 sub-indicators have been 1737

shown to be acceptably met. specific requirements for each regulatory function are set out in 1738

the sections below. 1739

1740


Page 71

Figure A. The PEP for each regulatory function includes two main areas of assessment. 1741

1742

1743

Table A. Different components of assessment included in the PEP for each regulatory function. 1744

Regulatory function

Number of:

Mandatory ML4 GBT sub-

indicators

New PE indicators

New PE tools

RS a Regulatory system 16 3 -

MA Registration and marketing authorization

2 11 1

VL Vigilance 4 16 1

MC Market surveillance and control 2 2 -

LI b Licensing establishments 4 - -

RI c Regulatory inspection 5 - 1

LT Laboratory testing 5 - 3

CT Clinical trials oversight 2 10 1

LR d

NRA lot release 1 - -

a NRA or RRS cannot be listed in RS function only 1745 b WLAs for LI must also acceptably meet all requirements for RI 1746 c Where applicable, WLAs for RI must also acceptably meet all requirements for LI 1747 d WLAs for LR must also acceptably meet all requirements for LT 1748

1749

While the GBT indicators are considered to adequately evaluate whether or not the WLA 1750

applicant has suitable regulations, guidelines and written procedures in place, new 1751

performance evaluation indicators and tools focus on assessing specific elements of technical 1752

or operational performance, including output quality and effectiveness. In each case, new 1753

PEP indicators and tools are designed to be meaningful, measurable and reasonable. They 1754

generally follow a similar structure to the GBT factsheets to give assessors all the information 1755

Acceptably meet requirements for all related GBT sub-indicators for ML1, ML2 and ML3 Agree scope and roadmap for listing with WHO

Acceptably meet mandatory GBT ML4 sub-indicators

Acceptably meet newly developed performance evaluation indicators

WLA status for relevant

function

ELIGIBILITY PEP

NEW INDICATORS

GBT ML4

WLA

Acceptably meet newly developed performance evaluation tools

NEW TOOLS

Page 72


they need to do an evaluation, including a description and objective, a list of evidence to 1756

review and a rubric for assessment with rating scale. 1757

In each case where new indicators have been developed, the NRA or RRS should complete a 1758

self-assessment against the indicators and submit it to WHO for review and evaluation. 1759

1760

All PEPs were developed in consultation with regulatory experts from around the world 1761

through a series of working group meetings that were convened between October 2020 and 1762

March 2021. Each PEP builds on existing GBT sub-indicators as well as the standards and 1763

practices for performance evaluation used by: international initiatives such as Pharmaceutical 1764

Inspection Co-operation Scheme (PIC/S); specialized organizations such as the European 1765

Directorate for the Quality of Medicines & Healthcare (EDQM) and the Benchmarking of 1766

European Medicines Agency (BEMA) initiative; and WHO programmes such as the WHO 1767

Prequalification for Laboratories. 1768

1769

Completing the PEP for any given regulatory function is expected to take no longer than six 1770

months, depending on the requested scope of the WLA listing and on the size of the 1771

evaluation team, as detailed in the sections that follow. 1772

1773

In all cases, performance evaluations will be done by a group of experts appointed by WHO. 1774

The number of assigned evaluators will vary depending on the requested scope of the WLA 1775

listing, and on evaluator availability. In general, a minimum of two evaluators is preferred to 1776

ensure peer review. 1777

1778

Review and evaluation may be conducted on-site or remotely (information shared through 1779

secured platforms with virtual meetings, as necessary). 1780

1781

References 1782

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory systems of 1783

medical products: revision VI. Geneva: World Health Organization; 2021 1784

(https://apps.who.int/iris/handle/10665/341243, accessed 25 May 2021). 1785

2. Policy: Evaluating and publicly designating regulatory authorities as WHO listed authorities. 1786

Working document QAS/19.828/Rev.1, July 2020. Geneva: World Health Organization; 2020 1787

(https://www.who.int/docs/default-source/medicines/norms-and-standards/current-1788

projects/qas19-828-rev1-policy-on-who-listed-authorities.pdf?sfvrsn=49504b99_2, accessed 1789

03 May 2021). 1790

1791


https://www.who.int/docs/default-source/medicines/norms-and-standards/current-projects/qas19-828-rev1-policy-on-who-listed-authorities.pdf?sfvrsn=49504b99_2



Page 73

1. Regulatory system (RS) 1792

1793

1.1. RS PEP methodology 1794

The PEP for RS is designed to assess the national regulatory authority (NRA) or regional 1795

regulatory system (RRS) as the overall framework for ensuring a sustainable, well-functioning 1796

regulatory system that can ensure independent and competent oversight of medical 1797

products. Listing of RS function alone is excluded, since it doesn’t entail any specific 1798

regulatory activity, but requirements described below must be acceptably met by NRA/RRS 1799

applying for listing in any other function and/or product category. 1800

1801

In addition to meeting the eligibility criteria, PEP for RS is considered fulfilled if the NRA or 1802

RRS demonstrates to: 1803

a. fully implement 16 mandatory ML4 GBT sub-indicators for RS; and then 1804

b. acceptably meet a series of newly developed RS performance evaluation indicators (see 1805

Figure 1.1). 1806

1807

The full RS PEP is estimated to take around three to six months to complete. 1808

1809

Figure 1.1. Flowchart of the RS PEP. 1810

1811

1812

1813

1.2. Mandatory ML4 GBT sub-indicators for RS 1814

WLAs for regulatory system must fully implement the following ML4 indicators, as defined in 1815

the GBT: 1816

1. RS03.05: The NRA is promoting good regulatory practices. 1817

Are eligibility criteria met?

Are ML4 mandatory sub-indicators met?

START

PEP for other function(s)

YES

NO

Report negative PEP conclusion & outcome

Assess against new RS indicators

RS PEP fulfilled

NO

Are all RS indicators acceptably met?

NO

YES

YES

NO

Page 74


2. RS04.01: Leadership ensures that the strategic priorities and objectives are well known and 1818

communicated throughout the NRA. 1819

3. RS05.05: The NRA establishes mechanisms to continually improve the QMS. 1820

4. RS05.06: The NRA has identified its regulatory processes, determined their interactions and 1821

defined the methods needed to control these processes. 1822

5. RS05.08: External and internal issues including relevant potential risks are defined and 1823

assessed periodically for proper risk mitigation. 1824

6. RS05.12: Corrections, corrective actions, and other actions for risk mitigation and overall 1825

improvement, are implemented and documented and their effectiveness is verified. 1826

7. RS05.13: Top management reviews and documents the organization’s QMS at planned 1827

intervals (i.e., management review). 1828

8. RS05.14: A mechanism is established to evaluate and demonstrate the effectiveness of 1829

training activities. 1830

9. RS06.01: The NRA has the power to select and recruit its own staff following documented 1831

procedures based on its own written criteria (i.e., education, training, skills and experience). 1832

10. RS06.02: A periodic staff appraisal system is established to review performance and 1833

competencies, to identify training needs, and to agree on performance targets 1834

11. RS07.04: The NRA has authority to manage the funds allocated and/or generated internally. 1835

12. RS07.05: The NRA periodically publicizes its budget. 1836

13. RS09.03: Information on decisions related to regulatory activities is available to the public. 1837

14. RS09.05: All publicly available information is periodically reviewed and maintained. 1838

15. RS10.01: Requirements established to monitor, supervise and review the performance of the 1839

NRA and affiliated institutions using key performance indicators (KPIs). 1840

16. RS10.02: Reports on the regulatory activities and on the progression and status of resources 1841

are available at regular intervals. 1842

1843

1.3. WLA RS performance evaluation indicators 1844

WLAs for RS must be assessed against three new performance evaluation indicators and 1845

meet the acceptability criteria as set out in the rating scale (see PE.RS.01–PE.RS.03 below). 1846

1847

PE.RS.01. The NRA or RRS participates in the WHO certification scheme on the quality of pharmaceutical products moving in international commerce and issue Certificate of Pharmaceutical Product (CPP)

Description: The assessor should verify that the NRA or RRS is an active member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce; and is listed as a participant on the WHO website.

Note that this indicator should be evaluated according to the terms and conditions explained and covered by the WHO CPP guideline. Any bilateral agreement or additional request by importing NRAs or commercially interested parties is out of scope.

Objective: This indicator aims to ensure that the NRA or RRS controls medical products exported from its jurisdiction; and that it follows the latest WHO guidelines and requirements for issuing CPPs, including having a reliable system for


Page 75

authorizing medical products, and licensing and inspecting manufacturing facilities.

Although this indicator does not cover the export only products, similar regulatory oversight on these group of products is encouraged.

Evidence to review:

The assessor should review:

• the list of contacts for competent authorities of countries participating in the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce, which is available at: https://www.who.int/teams/regulation-prequalification/regulation-and-safety/rss/certification-scheme/contacts.

References: 1. Guidelines on the implementation of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: Fifty-fifth report. Geneva: World Health Organization; 2021: Annex 9 (WHO Technical Report Series, No. 1033, https://www.who.int/publications/i/item/55th-report-of-the-who-expert-committee-on-specifications-for-pharmaceutical-preparations, accessed 29 June 2021).

2. Resolution WHA50.3. WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce. In: Fiftieth World Health Assembly, Geneva, 5–14 May 1997. Resolutions and decisions, annexes. Geneva: World Health Organization; 1997 (https://apps.who.int/iris/bitstream/handle/10665/179726/WHA50_R3_eng.pdf?sequence=1&isAllowed=y, accessed 15 February 2021).

3. WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce: questions and answers (Q & A). Adopted guidance. WHO Drug Information. 2016; 30(3) (https://apps.who.int/iris/bitstream/handle/10665/331015/DI303-376-388-eng.pdf?sequence=1&isAllowed=y, accessed 15 February 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS is not a member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce.

IMPLEMENTED (I): The NRA or RRS is a member of the WHO Certification Scheme on the quality of pharmaceutical products moving in international commerce and is included in the WHO list of contacts published online.

Limitations and remarks:

This sub-indicator cannot be scored as "not applicable " (i.e. this sub-indicator always applies when assessing RS WLAs).

https://www.who.int/teams/regulation-prequalification/regulation-and-safety/rss/certification-scheme/contacts

https://www.who.int/teams/regulation-prequalification/regulation-and-safety/rss/certification-scheme/contacts

https://www.who.int/publications/i/item/55th-report-of-the-who-expert-committee-on-specifications-for-pharmaceutical-preparations

https://www.who.int/publications/i/item/55th-report-of-the-who-expert-committee-on-specifications-for-pharmaceutical-preparations

https://apps.who.int/iris/bitstream/handle/10665/179726/WHA50_R3_eng.pdf?sequence=1&isAllowed=y

https://apps.who.int/iris/bitstream/handle/10665/179726/WHA50_R3_eng.pdf?sequence=1&isAllowed=y

https://apps.who.int/iris/bitstream/handle/10665/331015/DI303-376-388-eng.pdf?sequence=1&isAllowed=y

https://apps.who.int/iris/bitstream/handle/10665/331015/DI303-376-388-eng.pdf?sequence=1&isAllowed=y

Page 76


PEI.RS.02. The NRA or RRS has established an effective competency framework

Description: A competency framework defines the knowledge, skills, attitudes, and behaviours needed for people within an organization that are developed through education, training, and experience (Hoge et al., The fundamentals of workforce competency: implications for behavioral health. APMH 32, 509–531 (2005)). https://doi.org/10.1007/s10488-005-3263-1

At minimum, it comprises six outputs:

1) A fit-for-purpose organizational competency manual or similar document.

2) Documentation of competency requirements for each position.

3) Competency assessment methods and tools.

4) Records of competency assessments and feedback to staff.

5) Training plans.

6) Mechanism for monitoring and evaluation (M&E) of training programmes.

The assessor should verify the existence, implementation, effectiveness and performance of the NRA’s competency framework, including the components detailed above.

This includes verifying the quality (including consistency) and effectiveness of internal mechanisms to monitor and evaluate the performance and impact of the competency framework. Such M&E should be done regularly (for example, annually); and should align with the NRA’s strategic plan and commitment to continual improvement. The assessor should also verify the existence, and adequate resourcing, of a system that ensures a coordinated and integrated approach across all regulatory activities.

Note that the competency framework may go by a different name; but it should still fulfil the principles and components mentioned herein.

In all cases, this indicator should be evaluated alongside related GBT human resources indicators under each function; as well as GBT sub-indicators RS06.01, RS06.02, RS06.03, RS06.04, RS05.14.

Objective: This indicator aims to ensure that the NRA has established and implemented a competency framework that can be used to:

• identifying required qualifications for staff performing various regulatory and supportive activities;

• select staff with the qualifications and level of competency needed;

• assign regulatory work to scientific staff members according to their qualification;

• define training needs;

• plan and provide training designed to meet competency objectives;

• measuring the impact of training and use this information to support continual improvement.

https://doi.org/10.1007/s10488-005-3263-1


Page 77

Evidence to review:

The assessor should ask for and review:

• The competency framework manual (or equivalent document) for all

applicable regulatory and operational functions.

• Evidence that all components of the competency framework have been

implemented, including documents, records, reports or assessments to show

that:

o descriptions of the required competencies are clear and fit for purpose;

o competency assessment tools used and respective outputs are appropriate;

o interventions to build and maintain competencies are relevant; o training has been provided and impact assessments carried out;

o there are mechanisms to evaluate and demonstrate the effectiveness of the competency framework in conjunction with organization’s continual improvement plans.

• Periodic (trend) reports and/or reviews of the competency framework’s performance, including subsequent actions taken for improvement.

• Records of communications to staff about the competency framework, including the results of any competency assessments carried out.

To evaluate the quality and impact of the competency framework’s outputs, the assessor should liaise with other assessors involved in GBT and WLA performance evaluation.

References: Towards a global competency framework for regulators of medical products. WHO Drug Information Vol 33, No. 1, 2019 https://www.who.int/medicines/publications/druginformation/issues/WHO-DI_33-1_Global-Framework.pdf

Rating scale:

NOT IMPLEMENTED (NI): The NRA or RRS does not have a comprehensive competency framework to cover both regulatory and operational functions. Various components of a competency framework may exist separately, but they are not connected or integrated properly to have an effective and forward-looking competency management plan.

PARTIALLY IMPLEMENTED (PI): The NRA or RRS has established most of the

components of the competency framework. But there is inadequate evidence

of a mechanism to evaluate the framework’s effectiveness in line with the

organization’s continual improvement plan, and/or there is inadequate

evidence of actions taken for improvement.

IMPLEMENTED (I): The NRA or RRS has established an effective and forward-looking competency framework for continual improvement, including all the required components as explained under description, objective and evidence to review above.

For an authority to be given WLA status, this indicator should at least be scored as partially implemented.

https://www.who.int/medicines/publications/druginformation/issues/WHO-DI_33-1_Global-Framework.pdf

https://www.who.int/medicines/publications/druginformation/issues/WHO-DI_33-1_Global-Framework.pdf

Page 78



This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing of RS WLAs).

PE.RS.03. The NRA has implemented measures to monitor, evaluate and sustain the performance of the quality management system (QMS)

Description: A fit-for-purpose QMS facilitates the consistent, effective, transparent and efficient performance of technical and administrative activities within the capacity, resources, needs and context of the NRA. In all cases, the NRA should be able to monitor and evaluate the performance of key operational aspects and areas of the QMS over time, by integrating results achieved by all different activities of the organization.

The assessor should verify that there is a system and procedures in place to manage the performance of the QMS for all applicable regulatory and operational functions; and that it works effectively. The system and related procedures should cover the all essential components of a QMS, including qualified staff and management, processes, resources (financial, infrastructure and tools), documentation, trend reports for key QMS outputs, management reviews and actions taken for continual improvement.

Objective: This indicator aims to ensure that the NRA has established, implemented and adequately resourced a system and procedures for measuring the performance of the QMS across all applicable regulatory and operational functions.

Evidence to review:


a. written procedures for measuring QMS performance; b. records of resources assigned for measuring QMS performance (including

staff, IT infrastructure, financial etc.) for measuring QMS performance.

In addition, the assessor should ask for and review evidence of M&E towards continual improvement, including:

• Trend reviews and reports on non-conformances and subsequent corrective actions.

• Reviews and revisions of processes and documents (such as manuals, SOPs, templates etc.) as per QMS requirements.

• Reports and follow-ups on areas of improvement and (service) complaints.

• Risk and opportunity management procedures; and related actions taken.

• Records to show top management's commitment to QMS performance (for example, meeting minutes detailing frequency and type of meetings held, and management’s participation).

• List of KPIs (or equivalent) for QMS in line with organization’s plan for continual improvement.

• Periodic (trend) reports or reviews of QMS KPIs, and subsequent actions taken.


Page 79

To evaluate the QMS’ contribution to the NRA’s overall performance, the assessor should liaise with other assessors; or review the benchmarking report and other WLA PEP reports.

Note that procedures and resources for measuring QMS performance may be described in the same document that details other key performance indicators (KPIs) for the NRA.

References: 1. WHO guideline on the implementation of quality management systems for

national regulatory authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: Fifty-fourth report. Geneva: World Health Organization; 2020: Annex 13 (WHO Technical Report Series, No. 1025: https://www.who.int/docs/default-source/medicines/norms-and-standards/guidelines/trs1025/trs1025-annex13.pdf?sfvrsn=8e6a17ee_2, accessed 29 June 2021).

2. Implementing quality management systems in national regulatory

authorities: examples and practices

https://apps.who.int/iris/bitstream/handle/10665/341942/9789240022379-

eng.pdf)

Rating Scale: NOT IMPLEMENTED (NI): The NRA has no mechanisms, system and/or procedures to measure QMS performance.

PARTIALLY IMPLEMENTED (PI): The NRA has identified and implemented a system or mechanism for monitoring and evaluating QMS performance in critical areas of its functions and operations (criticality to be based on the needs of the NRA with respect to the scope of WLA listing).

IMPLEMENTED (I): The NRA has developed and implemented a robust mechanism or system with relevant procedures for monitoring, evaluating QMS performance; and for implementing actions to sustain its performance.

For an authority to be given WLA status, this indicator should at least be scored as partially implemented.


This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing of RS WLAs).

1848

1849

https://www.who.int/docs/default-source/medicines/norms-and-standards/guidelines/trs1025/trs1025-annex13.pdf?sfvrsn=8e6a17ee_2

https://www.who.int/docs/default-source/medicines/norms-and-standards/guidelines/trs1025/trs1025-annex13.pdf?sfvrsn=8e6a17ee_2



Page 80


2. Registration and marketing authorization (MA) 1850

1851

2.1. MA PEP methodology 1852

The PEP for MA is designed to assess MA for specific products and activities only, including: 1853

• Products: new chemicals entities (new medicines), multisource/generic medicines, vaccines, 1854

similar biotherapeutic products (SBPs). 1855

• Processes: pre-submission procedure; submission, screening and validation of the 1856

application; selection of regulatory pathway; administrative and scientific evaluation; 1857

advisory committee procedure and adoption of final decision; transparency and structure; 1858

post-approval actions (including renewals, variations, extensions of MA, withdrawals and 1859

transfers of MA); and effectiveness of the MA process. 1860

1861

In addition to meeting the eligibility criteria, PEP for MA is considered fulfilled if the NRA or 1862


a. fully implement two mandatory ML4 GBT sub-indicators for MA (see Section 2.2); and then 1864

b. acceptably meet 11 newly developed MA performance evaluation indicators (see Section 1865

2.3); and then 1866

c. successfully undergo an Expert Review of Marketing Authorization Application (MAA) 1867

Assessments (see Section 2.4). 1868

1869

Figure 2.1. Flowchart of the MA PEP. 1870


Page 81

1871

1872

1873

The full MA PEP is estimated to take 4–8 months to complete, depending on the number of 1874

product categories the NRA or RRS applies to be listed for, and on the number of assessors in 1875

the evaluation team. 1876

1877

The evaluation of mandatory ML4 sub-indicators and the newly developed performance 1878

evaluation indicators should take 1–2 months to complete; a minimum of two assessors is 1879

preferred to ensure peer review. 1880

1881

The expert review is expected to take 3–6 months to complete and will require 2–6 assessors 1882

who may or may not be the same assessors completing other elements of the MA PEP. 1883

1884

All NRA or RRS files, records and reports used for this performance evaluation must be less 1885

than three years old. Selection of this documentation, including number and type, is to be 1886

done by the assessor. 1887

1888



START

END

YES

YES

NO

Are all parts of the expert review acceptably met?

Assess against new MA indicators

MA PEP fulfilled

NO

Are all new MA indicators acceptably met?

Conduct Expert Review of MAA Assessments

NO Report negative PEP conclusion & outcome

YES

YES

NO

Page 82


2.2. Mandatory ML4 GBT sub-indicators for MA 1889

WLAs for registration and marketing authorization must fully implement the following ML4 1890

indicators, as defined in the GBT: 1891

1. MA05.03 A summary of technical evaluation report for approved registration MA applications 1892

is published and available to the public (plus expansion, see 2.2.1 below). 1893

2. MA06.02 Performance indicators for registration and MA activities are established and 1894

implemented. 1895

1896

2.2.1. Expansion for GBT ML4 indicator MA05.03 1897

In addition to fully implementing sub-indicator MA05.03 as defined in the GBT, NRAs should 1898

ensure that the published summary technical evaluation report for product approvals and 1899

registrations is sufficiently detailed. 1900

1901

The assessor should select three files and check that the respective public assessment 1902

reports (and NRA or RRS guidelines for developing them) include the elements in Table 2.2.1 1903

as minimum required information. 1904

1905

Table. Minimum required information for published summary technical evaluation reports. 1906

Section of report Minimum required information

Section 1 Type of application

• Marketing authorization, licensure, registration, opinion, temporary authorization for use, emergency use

• Licensing number

• 3Granted at national level, regional/community level, other if applicable

• Granting regulatory authority

• Standalone application, product line extension, abbreviated application (bibliographic, generics, biosimilar)

• Category such as orphan, paediatric, and others.

• Intended for domestic market, for export, both

• 8. Conditional approval, exceptional circumstances, accelerated approval/fast track/reliance, etc

Section 2 Information about the product

• Brand name, as applicable

• Dosage form/pharmaceutical form

• Anatomical Therapeutic Chemical (ATC) code

• Composition o Active substance(s) o Excipients

• Strength

• Container, closure and administration device(s),

• All available presentations

• Route of administration

• Legal entity (sometimes referred to as the Marketing Authorization Holder or MAH), which is liable for the quality, security and efficacy of the medical product.

Section 3 Labelling system

• Product information for the user (package leaflet, patient package inserts)

• Samples of labels on containers, secondary packaging

Section 4 Scientific discussion during the initial procedure

• Introduction (type of marketing authorization, main features of disease/condition etc)

• Quality/CMC aspects o Introduction (pharmaceutical form, formulation, container system,

etc)


Page 83

o Drug substance (INN; chemical features like chemical class, chirality, manufacturing, substrate used for production of biologicals, stability)

o Drug product (pharmaceutical development, manufacture of the product, stability of the product)

o Discussion on chemical, pharmaceutical and biological aspects

• Non-clinical aspects o Introduction o Pharmacology o Pharmacokinetics o Toxicology o Ecotoxicity/environmental risk assessment/GMO o Discussion on the non-clinical aspects o For generics: brief explanation that abridged applications avoid the

need for repetitive tests on animals and humans. Reference to the reference medicinal product

• Clinical aspects o Introduction o Pharmacokinetics o Pharmacodynamics o Clinical efficacy

• Clinical safety o Discussion on the clinical aspects o For generics: brief explanation that abridged applications avoid the

need for repetitive tests on animals and humans. Reference to the reference medicinal product. For these applications the bioequivalence studies are pivotal and should be described.

• Overall conclusion, benefit/risk assessment and recommendation (specific obligations, follow-up measures, if applicable)

Section 5 Procedural steps

• Procedural steps taken before authorization/licensure.

• Link or reference to the webpage of procedural steps taken after authorization/licensure of this application (major variations, PSURs, commitments)

1907

1908

2.3. WLA MA performance evaluation indicators 1909

To be listed as WLA for MA, an NRA or RRS must be assessed against 11 newly developed MA 1910

indicators, as detailed below. In each case, the NRA or RRS must meet the criteria for an 1911

“implemented” rating. 1912

1913

2.3.1. Pre-submission procedure 1914

1915

PE.MA.01. The NRA or RRS has a well-established pre-submission procedure, supported by adequate guidelines and SOPs, including pre-submission meetings and regulatory/scientific advice, as applicable.

Description: The assessor should verify that a mechanism is in place (guidelines, procedures, instructions, interpretation guides) to provide scientific and regulatory guidance, including pre-submission guidance, to manufacturers in advance of MA applications (MAAs). Those activities are supported by adequate guidelines and/or SOPs.

Page 84


There is data available on the effectiveness of advice in relation to the quality of subsequent MAAs.

Objective: This indicator aims to ensure that a pre-submission framework, supported by adequate guidelines and SOPs, exists and is communicated to manufacturers. The NRA or RRS provides scientific and regulatory advice upon request.

Evidence to review:


• guidelines, SOPs, instructions or interpretation guide on providing scientific and regulatory guidance.

In addition, the assessor should select at least three files and review:

• records of scientific and regulatory advice provided;

• pre-submission meeting minutes;

• any data on the effectiveness of advice in relation to the quality of subsequent MAAs.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national

regulatory system of medical products. Revision VI version 1. Geneva: World Health Organization; 2018: MA sub-indicators (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): The NRA or RRS has not developed or implemented

a well-established pre-submission mechanism, system and/or procedures.

IMPLEMENTED (I): The NRA or RRS has defined, implemented and published

a mechanism, system and procedures for the pre-submission step, including

pre-submission meetings and regulatory/scientific advice to manufacturers.


This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing WLAs for MA).

PE. MA.02. The NRA or RRS consistently complies with the procedures and timelines established in its guidelines and SOPs for pre-submission activities.

Description: The assessor should verify that the NRA or RSS has established procedures and timelines related to pre-submission activities (including scheduling of pre-meetings, product eligibility evaluation, regulatory/scientific advice, as applicable); and that these are consistently respected.

The assessor should further verify that, in cases where pre-submission activities do not comply with established procedures, there is an acceptable justification or rationale given (to show that non-compliance is not common practice). This indicator should be assessed alongside WHO GBT sub-indicator MA04.06.



Page 85

Objective: This indicator aims to ensure that the NRA or RRS consistently applies the requirements of its pre-submission system, guidelines and formal procedures, including timeline requirements. The ultimate objective is to ensure that pre-submission is an enabler of the MA process, not an obstacle to access to medicines.

Evidence to review:


• guidelines, SOPs, instructions or equivalent document establishing procedures and timelines for pre-submission activities;

• at least three files selected by the assessor (to check procedures followed);

• at least three files, selected by the assessor, that do not comply with established procedures (to check for presence of justification or rationale)

In addition, the assessor should select at least three files and review:

• procedures followed, including timelines, for pre-submission. The assessor should also select at least three files that do not comply with timeline requirements and review:

• justification or rationale for the delay, showing that non-compliance with timelines is not a common practice for pre-submission activities.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI version 1. Geneva: World Health Organization; 2018: Sub-indicators MA 04.06 and MA 06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): The NRA or RRS does not comply with the established procedures and requirements, including timelines, in the applications reviewed.

IMPLEMENTED (I): The NRA or RRS complies with the established procedural requirements, including timelines, in the applications reviewed.



1916

1917

1918

2.3.2. Submission, screening and validation of the application 1919

1920

PE.MA.03. The NRA or RRS consistently complies with the procedures and timelines established in its guidelines and SOPs for the receipt, screening and validation of applications; and for the period between application validation and scientific evaluation.


Page 86


Description: The assessor should verify that that the NRA or RRS has established procedures and timelines related to submission, screening and validation; and that these are consistently respected.

The assessor should further verify that, in cases where procedures for pre-submission activities do not comply with established procedures, there is an acceptable justification or rationale given (to show that non-compliance is not common practice). This indicator should be assessed alongside WHO GBT sub-indicator MA04.06.

Objective: This indicator aims to ensure that the NRA or RRS consistently applies the requirements of its system, guidelines and formal procedures on submission, screening and validation activities, including timeline requirements. The ultimate objective is to ensure that this step is an enabler of the MA process, not an obstacle to access to medicines.

Evidence to review:


• guidelines, SOPs, instructions or equivalent document establishing procedures and timelines for submission, screening, validation and pre-scientific evaluation activities

• at least three files selected by the assessor (to check procedures followed);

• at least three files, selected by the assessor, that do not comply with timeline requirements (to check for justification or rationale).


Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with procedures and established requirements, including timelines, in the reviewed applications.

IMPLEMENTED (I): The NRA or RRS complies with the established procedural requirements, including timelines, in the applications reviewed.



1921

1922

2.3.3. Selection of regulatory pathway 1923

1924

PE.MA.04. The NRA or RRS consistently complies with the procedures and timelines established in its guidelines and SOPs to approve donation of medical products, as applicable.

Description: The assessor should verify that the NRA or RRS has established procedures and timelines for accepting or providing donations of medical products; and that these are consistently are respected.



Page 87

The assessor should further verify that, in cases where procedures and timelines for approving donations of medical products do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance is not common practice).

This indicator should be assessed alongside WHO GBT sub-indicator MA04.06.

Objective: This indicator aims to ensure that donations are included as a source of medical products with acceptable and demonstrated quality, safety and efficacy, and that relevant procedures and timelines are established and consistently respected.

Evidence to review:


• Guidelines, SOPs, instructions or equivalent document establishing procedures and timelines for accepting or providing donations of medical products.

• At least three files selected by the assessor (to check procedures followed).

• At least three files, selected by the assessor, that do not comply with timeline requirements (to check for justification or rationale).


Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with the donation procedures and timelines in the reviewed applications.

IMPLEMENTED (I): The NRA or RRS complies with the established procedural requirements for donations, including timelines, in the applications reviewed.


This sub-indicator can be scored as "not applicable" if the NRA or RRS has not received or donated products in the last three years.

PE.MA.05. The NRA or RRS has a well-defined and established risk-based approach or mechanism for consistently selecting the most adequate regulatory pathway for each application (supported by appropriate guidelines or SOPs).

Description: The assessor should verify that a risk-based mechanism is in place to select the most appropriate regulatory pathway for assessing MAAs, including expedited pathways such as accelerated, abridged and reliance pathways, and emergency assessments.

Objective: This indicator aims to ensure that the NRA always chooses the most appropriate regulatory pathway, considering all relevant factors impacting public health conditions.

Evidence to review:


• Guidelines, SOPs, decision trees or other evidence of a risk-based mechanism;

• At least three files, representing three different regulatory pathways.


Page 88



regulatory system of medical products. Revision VI version 1. Geneva: World Health Organization; 2018: Sub-indicators MA 01.12 and MA 04.07 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS has not developed and established a mechanism (procedures, guidelines, decision trees) to select regulatory pathways for MAAs.

IMPLEMENTED (I): The NRA or RRS has defined and implemented a risk-based approach and mechanism (guidelines, procedures, decision trees) to appropriately select regulatory pathways.



1925

1926

2.3.4. Administrative and scientific evaluation 1927

1928

PE.MA.06. The NRA or RRS consistently complies with the timelines established in its guidelines and SOPs for the administrative and scientific evaluation of medical products.

Description: The assessor should verify that the NRA or RRS has established timelines for the administrative and scientific evaluation of medical products; and that these are consistently respected.

The assessor should further verify that, in cases where timelines for the administrative and scientific evaluation of medical products do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance is not common practice). This indicator should be assessed alongside WHO GBT sub-indicator MA04.06.

Objective: This indicator aims to ensure that the timelines for administrative and scientific evaluation are efficiently and consistently adhered to by the NRA or RRS. The ultimate objective is to ensure that this step is an enabler of the MA process, and does an obstacle to access to medicines.

Evidence to review:


• Guidelines, SOPs, instructions or equivalent document establishing timelines

for the administrative and scientific evaluation of medical products. • At least three files selected by the assessor (to check timelines).

• At least three files, selected by the assessor, that do not comply with

timeline requirements (to check for justification or rationale).

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI version 1. Geneva: World Health Organization; 2018: Sub-indicators MA 04.06 and MA



Page 89

06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with established timelines for administrative and scientific evaluation of medical products in the reviewed applications.

IMPLEMENTED (I): The NRA or RRS complies with the established timelines for administrative and scientific evaluation of medical products in the reviewed applications.



1929

1930

2.3.5. Advisory committee procedure and adoption of final decision 1931

1932

PE.MA.07. The NRA or RRS has a procedure for calling on and consulting an advisory/technical committee or external experts whenever this is deemed necessary for MA-related activities.

Description: A qualified committee, or group of external experts, can provide advice and technical assistance to support MA-related activities. Such committees can meet according to a regular schedule; or be called upon ad hoc as a need emerges. The assessor should verify that:

• The terms of reference for the committee cover the major needs for

specialist advice on MA-related activities. The frequency of meetings

should be noted.

• The committee expertise enables specialist advice to be provided on

most issues relevant to MA.

• Committee members have relevant experience or training for their role.

• Appointment letters for committee members, including statements of

confidentiality and conflict of interest, are available.

• Minutes from past meetings are available. The frequency, attendance

and recommendations from meetings should be noted; and reasons

identified for any discrepancy between actual and planned meeting

frequency.

Objective: This indicator aims to ensure that there is a mechanism, supported by adequate guidelines and SOPs, for consulting internal and external partners in the MA evaluation process, if necessary.


Page 90


Evidence to review:


• the TOR for the committee; • SOPs defining the expert committee’s review activities and programme of

work.

• letters of appointment of committee members;

• minutes from committee meetings; and

• proofs of regulatory decisions taken based on inputs provided by the

committee.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MA 04.05 and RS 09.06 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating Scale: NOT IMPLEMENTED (NI): The NRA or RRS has not developed a mechanism, system and/or procedures to involve an expert committee or external experts in the evaluation process of MAAs.

IMPLEMENTED (I): The NRA or RRS has implemented a mechanism, system and/or procedures to involve an expert committee or external experts, when deemed necessary for the evaluation of MAAs.



1933

1934

2.3.6. Transparency and post-approval actions 1935

1936

PE.MA.08. The NRA or RRS has a mechanism, supported by adequate guidelines or SOPs, for sharing a summarized MA technical evaluation report or a justification of rejected applications with other authorities.

Description: The assessor should verify that there is a mechanism for facilitating the exchange of information on regulatory decisions about MA with other NRAs or RRSs, at least upon their request. The exact form of mechanism doesn’t matter as long as it ensures that relevant data is promptly made available to requesting NRAs or RRSs.

In all cases, the sharing mechanism should include details of any information exchange arrangement, confidentiality agreement, or legal requirements from product sponsors required to allow information to be shared with other regulatory authorities. Sharing mechanisms should also include the sharing of summarized technical evaluation reports or justifications of regulatory decisions.

The assessor should verify that information is shared according to NRA or RSS guidelines; and that it is shared in a timely manner.



Page 91

Objective: This indicator aims to ensure the existence of guidelines or regulations that allow the NRA or RRS to share information about their decisions to refuse or reject MA. The ultimate goal is to ensure the NRA or RRS embraces and follows adequate transparency policies and principles.

Evidence to review:


• Documented guidelines or procedures to support information sharing;

• Templates for MoUs, confidentiality agreements or other relevant legal requirements;

• Any other evidence that a mechanism exists for sharing information with other regulatory authorities, for example publication of summarized reports on a public website; records of sharing information with a network of authorities through a private digital platform; emails of information shared with relevant contacts upon request.

• A number of files, selected by the assessor, for refused or rejected MA applications that have been shared.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MA 05.04 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating Scale: NOT IMPLEMENTED (NI): A mechanism for sharing information on refused and rejected applications with other regulators does not exist, or is not promptly applied.

IMPLEMENTED (I): The NRA or RRS has defined and established a mechanism for sharing information on refused and rejected MA applications with other regulators, at least upon their request. Exchange of information is promptly carried out.



PE.MA.09. The NRA or RRS consistently complies with the requirements, including timeline requirements, established in its guidelines/regulations for post-approval actions.

Description: The assessor should verify that the NRA or RRS has established procedure requirements, including timeline requirements, for post-approval activities (including MA renewals, variations, extensions, withdrawals and transfers); and that these are consistently respected.

The assessor should further verify that, in cases where procedures for post-approval activities do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance is not common practice). This indicator should be assessed alongside WHO GBT sub-indicator MA04.06.


Page 92


Objective: This indicator aims to ensure that post-approval activities are efficiently and consistently performed by the NRA or RRS. The ultimate objective is to ensure that post-approval activities enable timely access to quality assured medicines.

Evidence to review:


• Guidelines, SOPs, instructions or equivalent document establishing procedures, including timelines, for post-approval activities.

• Records, selected by the assessor, for a number of different post-approval actions (to check procedures and timelines).

• At least three files of applications, selected by the assessor, that do not comply with timeline requirements for post-approval activities (to check for justification or rationale).

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MA01.02, MA01.04, MA01.05, MA04.02, MA04.03, MA04.06, and MA06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating Scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with established procedures, including timelines, in the reviewed applications.

IMPLEMENTED (I): The NRA or RRS complies with the established procedures, including timelines, in the reviewed applications.



PE.MA.10. The NRA or RRS consistently publishes its regulatory actions on a registered product.

Description: The assessor should verify that regulatory actions taken on registered products (including warning letters, renewals, variations, extensions of MA, withdraws and transfer of MA) are regularly published and made available to the public; and that this is supported by a regulation or a guidance.

Objective: This indicator aims to ensure that guidelines or regulations exist to allow the NRA or RRS to publish its post-approval regulatory actions on registered products and make these publicly available.

Evidence to review:

The assessor should review:

• The same files selected for PE.MA.09. • The website of the NRA or RRS (to check for publication of actions)


regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MA01.05, MA04.02, MA04.03, MA04.06, and MA06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).




Page 93

Rating Scale: NOT IMPLEMENTED (NI): There is no evidence of publication and public availability of regulatory actions taken on registered products after MA is granted by the NRA or RRS.

IMPLEMENTED (I): The NRA or RRS publishes and makes available to the public post-approval regulatory actions on registered products.



1937

1938

2.3.7. Effectiveness of the MA process 1939

1940

PE.MA.11. The authorization of medical products contributes, with other factors, to the availability of a range of appropriate medicines for patients.

Description: The assessor should verify that the NRA or RRS takes an active role in a range of national and international initiatives to ensure that patients can be treated with medicines that have a favourable benefit to risk balance. In addition, the assessor should verify that the NRA or RRS regularly reviews the effectiveness and impact of its authorization system on the availability of medicines; and take actions for improvement where possible.

Objective: This indicator aims to ensure that mechanisms are in place to review the authorization system’s effectiveness in supporting the availability of safe, effective and quality-assured medical products, with improvements implemented accordingly.

Page 94


Evidence to review:

The assessor should review documents, ask for and review documentation that can help answer the following questions, as applicable:

• Does the NRA or RRS facilitate early access to medicines, including before authorization? Are existing flexibilities within the regulatory framework explored?

• Have areas of unmet need within the country been identified? What actions have been taken as a result?

• What efforts have been undertaken to implement the concept of adaptive pathways to support early approval of a medicine for a restricted patient population? What kind of reviews are performed to ensure timely access to new medicines? How does the NRA or RRS balance out the need for more information on the quality, safety and efficacy against the need for access, particularly in areas of unmet need?

• Does the NRA or RRS carry out regulatory /scientific activities to ensure that the needs of special populations including children and elderly are met?

• Does the NRA or RRS facilitate access to medicines through appropriate reclassification?

• Is the NRA or RRS responsible for publishing information on the marketing status of medicines?

• Is the NRA or RRS aware of restricted availability of medicines on the marketplace? If it is in the organization’s mandate, are medicine shortages monitored and is information on shortages available on the agency’s website?

• Has there been analysis of the particular features of the authorization system that may be contributory factors, including loss of availability due to removal of indications or withdrawal of older products from the market, or the effect of cross-border internet sales in smaller markets?

• What types of action can be taken to mitigate the impact of shortages? Are actions taken with government departments and other relevant national agencies? Has the NRA or RRS sought to address broader causes of supply problems with other state agencies, stakeholders or within a network of NRAs? Does the NRA or RRS use or strengthen and support cross-border collaboration in case of supply disruption that affect multiple Member States? Has the effect of these actions been measured by the NRA OR RRS?

• Has the NRA or RRS assessed whether the established timelines or non-compliance with timelines leads to shortages or low availability?

• Has the NRA or RRS assessed whether inappropriate selection of regulatory pathways or approaches (for example emergency approval or reliance pathways) led to delays in making products available to patients?

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not take an active role in improving the availability of medicines on the market.

IMPLEMENTED (I): The NRA or RRS is involved in national and international initiatives to improve the availability of medicines with a favourable benefit to risk balance. Where necessary and possible, the NRA or RRS has changed its authorization system to encourage marketing of authorized medicines. The NRA or RRS regularly reviews the effectiveness and impact of its authorization system on the availability of medicines; and takes actions towards improvement where possible.


Page 95

Note that if the NRA has been assessed throughout the BEMA IV cycle and can show it was rated as 4 or 5 in KPI 12.3, the NRA is considered to acceptably meet the requirements of this indicator.



The PEP for MA recognizes that many different activities can influence the effectiveness of MA processes; and that not all of these necessarily fall within the mandate of all NRAs or RRSs (including, for example, the monitoring of medical product shortages). If that is the case, the relevant questions under “evidence to review” may be considered not applicable; although the overall indicator must still be rated according to the rating scale above.

1941

1942

2.4. WLA MA performance evaluation tools 1943

To be listed as WLA for MA, once an NRA or RRS has acceptably met all mandatory ML4 GBT 1944

sub-indicators and new MA performance indicators, it must complete an Expert Review of 1945

MAA Assessments. 1946

1947

PE.MA.12. Expert Review of MAA Assessments 1948

1949

Description and objective 1950

The Expert Review of MAA Assessments should be done by a group of 2–3 experts with 1951

specific expertise on the type of product (chemical or biological) and module (quality, clinical 1952

or non-clinical) in the WLA scope. Two groups of experts may be required, depending on how 1953

many assessment reports are selected for review. The experts doing the review may or may 1954

not be the same assessors as those that evaluated the other elements of the MA PEP. 1955

1956

The experts should select a representative number (minimum 2–3) of NRA or RRS MAA 1957

assessment reports for review. Selection should only include assessment reports that are less 1958

than three years old. As a guiding principle, selection should also include at least one 1959

assessment report for each of the product categories included in the WLA scope, unless the 1960

NRA or RRS applies to be listed for all product types, in which case the experts should review 1961

three assessment reports covering: a new chemical entity, a vaccine and a biosimilar 1962

pharmaceutical product. The results of the assessment for a new chemical entity will then be 1963

extrapolated for a multisource product. 1964

1965

In preparation for its assessment, the NRA or RRS should make the MAA assessment reports 1966

available, along with the respective product dossiers. 1967

1968

The Expert Review of MAA Assessments is done using the rubric below (see Table PE.MA.12). 1969

The experts should ensure that all guidelines used as reference materials are relevant and 1970

up-to-date (but during each review, the guidelines that were in place at the time of the MA 1971

application and assessment should be used as the reference point). 1972

1973

Page 96


Evidence to review 1974

For each selected assessment report, the expert assessors should complete each of the items 1975

set out in Table PE.MA.12 (comprising 4 indicators and 30 sub-indicators). 1976


Page 97

Table PE.MA.12. Rubric for the Expert Review of MAA Assessments. 1977

1978 # Evaluation Criteria

Performance goal(s) to be met by the NRA OR RRS Performance

adequately met by the NRA/RRS?

(Y/N)

Comments from the Expert on how the NRA/RRS complies or not with area measured

(required to be filled in)

1 Application Process (including pre-submission procedures, assessment, compliance with regulatory requirements and policies, communication, interactions with stakeholders)

The focus of the evaluation of the Application Process should be directed to all activities that can have an impact in the Assessment.

1.1

Was there Scientific/Regulatory advice or other similar activities provided by the NRA prior to the submission to support the success of a complete application with quality? Were there pre-submission meetings with the company for this application arranged by the NRA prior to the submission to support the success of a complete application with quality? Are the applicants made aware of the NRA’s expectations, including the target timeframes, guidelines, requirements, templates and checklists?

Appropriate support and information have been provided to the sponsor by the NRA for the success of an application submission. All actions were taken by the NRA to allow a more predictable and clear process for applicants. The NRA benefited from a complete application submission at the outset by the applicant.

1.2 Were the relevant documented procedures to support the full review process adequately followed? Was a review projection developed beforehand by the responsible team for the given application (including timelines for the different steps)? Was an assessment team leader assigned (based on a well-defined criteria)? Was the assessment team well formulated including involvement of all other relevant teams (e.g. staff with specific expertise for the given therapeutic area is involved, other teams are involved as necessary, such as inspections, NCL, etc.)?

- How was communication and interdisciplinary work between scientific staff ensured?

Were the roles and tasks well distributed among team members?

The NRA adequately followed the relevant internal Guidelines and SOPs for the review process. The Review process was well organized and projected by the NRA beforehand, including relevant timelines established. Roles and responsibilities were clearly defined and followed as per the indications in the guidelines and SOPs. The communication and interdisciplinary work between scientific staff were properly handled for the given application.

1.3 Are there available documented procedures, templates and checklists relevant to support the screening, validation and assessment processes? Were they appropriately followed?

- Was there a previous screening/validation of the information in the submission package to ensure that it is well-organized and that all the required forms and relevant documents have been submitted?

- Was the submitted dossier compliant with relevant International Standards such as ICH CTD format?

- ICH Guidelines, as applicable. - WHO Guidelines, as applicable.

The NRA has available and adequately followed relevant documented procedures, templates and checklists for the screening, validation and assessment processes. If those were not strictly followed in some cases, this was well justified for each of those cases and the NRA ensured it didn’t impact the outcomes of the assessment. The NRA identified any lack or missed information in the application prior to scientific review, avoiding spending time and review resources on an application that does not

Page 98


# Evaluation Criteria

Performance goal(s) to be met by the NRA OR RRS Performance adequately met

by the NRA/RRS? (Y/N)



allow critical analysis, signal identification or regulatory decision-making. The screening, validation and assessment processes are perceived to be compliant and aligned with the established procedures and international standards and practices.

1.4 Quality and consistency of the assessment, reports and decision-making Is the system for ensuring quality and consistency of scientific work, assessments, assessment reports and decision-making, adequate and subject to review and improvements? For which type of applications are assessment reports required, and not required? Is there guidance provided regarding the quality of the reports? Is training on report-writing provided? Overall, does the assessment report complies with local and International Standards, defined by the NRA to be applied upon? Are appropriate record-keeping requirements in place for assessment reports? How is past experience and regulatory and scientific memory for assessment and decision-making created and maintained? What system is in place for reviewing the appropriateness of the assessors’ opinion?

The NRA has robust procedures in place for evaluation of the quality and consistency of the assessment, assessment reports and decision-making. An adequate system is in place for maintenance of regulatory memory. Regulatory memory is effectively and actively disseminated. The agency ensures opinion making to standards and continuous improvement. The assessment, assessment reports and decision-making at the NRA is perceived to be consistently conducted and ensured.

Overall outcome evaluation of the section Overall, the NRA performance evaluation for this section is considered adequate.

2 Assessment Report The Experts are expected to have access to the whole dossier and consult it as needed to be able to evaluate the level of NRA’s performance on this section.

2.1 Quality of the report

2.1.1 Considers context Does the assessment report consider the data and the conclusions from the applicant in the context of the proposed conditions of use and storage? Does it include perspectives from patients/patient associations, health-care professionals and other NRAs’ analyses and decisions? Was there a mechanism/process activated to obtain opinion or advise from outside stakeholders, as necessary, in the adequate moments of the assessment and as per the NRA guidelines establish?

The Assessment report considers all relevant data and conclusions from the applicant on the proposed conditions of use and storage. The assessment report also considers any feedback provided by patients/patient associations and health-care professionals as well as other NRAs’ analyses and decisions. The NRA adequately followed its guidelines in terms of consulting and requesting advise from external experts, healthcare professionals and patients/patients association, as necessary and as per its guidelines.


Page 99






2.1.2 Balanced and Evidence-based Is the assessment report objective and unbiased? Is the assessment report evidence-based? Does it reflect both updated scientific and regulatory state of the art? Does it integrate legislative, regulatory and policy frameworks with emerging science? Are the type and number of objections raised and clarifications requested supported by evidences? Are concerns categorized in major and minor (or similar, based on national guidance)? Is the classification appropriate and supported by scientific discussion? Are the assessment of responses provided by the applicant considered into the final decision?

The Assessment report is evidence-based and factual. It considers and integrates emerging scientific and regulatory aspects and it is aligned with relevant legislative, regulatory and policy frameworks. It is based on updated and relevant technical guidelines. Specifically, the type and number of objections raised and clarifications requested are supported by appropriate evidences. The assessment of responses provided by the applicant is integrated into the final decision of the NRA.

2.1.3 Depth Does the assessment report comprehensively highlight potential areas of concern, providing a detailed analysis on those?

The Assessment report properly highlights and deeply analyses potential areas of concern supported by adequate justifications and observations.

2.1.4 Investigates problems Does the assessment report provide both the applicant’s and the assessors’ in-depth analyses and findings of key scientific data? Does the assessor demonstrate the use of risk-based tools, analyses and synthesis skills to ask relevant questions where needed?

The Assessment report provides comprehensive analysis and findings of key scientific data. The assessor demonstrated the use of risk-based tools, analyses and synthesis skills, to ask relevant questions and make appropriate judgments, where needed.

2.1.5 Makes linkages Does the assessment report provide integrated analysis across all aspects of the application: quality, non-clinical; clinical; chemistry/biocompatibility; manufacturing; and risk management plan? Does it include timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application?

The Assessment report provides good quality and integrated analysis of all relevant aspects of the application: non-clinical; quality; clinical; chemistry/ biocompatibility; manufacturing; and risk management plan. It includes timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application.

2.1.6 Thorough Does the assessment report reflect adequate follow-through of all the issues by the assessors?

The Assessment report reflects adequately follow-through of all issues raised by the assessors.

2.1.7 Utilizes critical analyses Does the assessment report assess the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application? Observations are well classified or categorized according to national agreed terms (e.g. major, minor)?

The Assessment report critically assesses the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application. Observations made throughout the report are categorized according to national agreed terms.

Page 100







2.1.8 Well-documented Does the review report provide a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions and rationale for reaching a decision? Does it contain clear, succinct recommendations that can stand up to scrutiny by all the parties involved and could be leveraged by others? Observations are well described and detailed? Observations are well grouped or categorized?

The Assessment report provides a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions as well as rationale for reaching a decision. It contains clear recommendations and well described, detailed and categorized observations.

2.1.9 Well-managed Does the review report apply project and quality management processes, including clearly defined steps with specific activities and targets? Were timelines well managed throughout the assessment, including for drafting the list of questions? Assessment Report is finalized within the agreed timeframe?

The Assessment report applied adequate project and quality management processes, including clearly defined steps, targets and timelines. The timelines were well managed throughout the assessment procedure and this is reflected in the report. The final report was complete within the established timelines, as per the NRA guidelines stipulate.

2.1.10 Peer Reviews Is there a system for peer review? Was the assessment report subject to peer reviews? How is it completed and recorded? For which type of applications? How are the comments of the peer reviewer handled? Are they documented and kept?

The agency has an effective system for peer-review of reports. The assessment report was subject to adequate and well documented peer reviews. The comments provided by the peer reviewer were appropriately handled and addressed. When it’s not applicable, a proper justification is provided.

2.1.11 Product Information to the public Is the product information to the public, such as SPC-like information, product information leaflets, product packaging and labelling, or other type of product information communication to the public, of good quality, easily readable and clearly communicated? Does the product information appropriately reflect the conclusions from the assessment regarding the approved indication, posology, method of administration, contra-indication, precautions for use, interactions, shelf-life, storage conditions, the available information on safety, efficacy, potential risk and how to manage them?

The product information, such as SmPC, leaflets or other type of product information communication to the public is of good quality, easily readable and clearly communicated to the target audience.

Overall outcome evaluation of the sub-section Overall, the NRA performance evaluation for this sub-section is considered adequate.

2.2 Completeness of the report to provide a comprehensive and complete picture of the situation or sample under consideration.

2.2.1 Were all relevant parts/modules of the dossier reviewed? All relevant parts/modules of the dossier were reviewed and they are reflected in the assessment report.


Page 101






2.2.2 Are all relevant regulations, standards and guidance referenced in the report, as necessary, linked to the respective observation?

All relevant regulations, standards and guidance are referenced in the report, as necessary, linked to the respective observation.

2.2.3 Is the Assessment Report complaint to the content and format described in the relevant SOP?

The Assessment report is compliant with the content and format described in the relevant SOPs or guidelines.

2.2.4 Were the risk management plans considered and included in the assessment? Are the risk management plans adequate to address the potential risks?

Risk management plans are part of the assessment report and are adequate from a qualitative point of view.


2.3 Scientific rigor to ensure the application of the scientific approach for unbiased analysis and interpretation of the evidence or data

High-quality scientific work provides a sound basis for appropriate consistent and harmonised opinions and decisions that affect public health. - Are well described the main critical features of the product, salient

findings and those deficiencies that justify any questions intended for the applicant?

- Is the assessor’s own critical assessment and observations to the applicant data included, particularly with respect to scientific elements and adherence to specific guidance documents?

- Are cross-references adequately used to clearly indicate the origin of any information used in the report, such as to the specific parts of the dossier (e.g. overview, summary, study reports), the references to the literature or other sources?

- Are those findings that need to be reflected in the SPC, Labels & Package Leaflet well emphasized?

- Are conclusions on the different scientific components well developed and described by the assessors?

The Experts are expected to look at the essential elements under each of those sections considering 1) the specific category of the product (chemical (new or multisource) or biological (vaccines or biosimilars)) and 2) the type of module (quality, clinical or non-clinical). He/she should use the list of items provided for guidance but mainly, his/her experience and judgement to analyse and evaluate the assessment conducted by the NRA on each of the 3 areas for the specific type of product. The Expert should aim to answer specific technical questions from a qualitative point of view. The Experts should write a summary of his/her findings for each of the sections (quality, clinical and non-clinical) on how the assessment was conducted by the NRA (in terms of evidence assessed by the assessor, quality of such assessment and observations, and decision-making done by the assessor). Note, if it’s a multisource FPP, based on WHO Guidance, it’s only required:

- demonstrating the bioequivalence of the FPP - demonstrating the quality of the API(s) - demonstrating the quality of the FPP - demonstrating adherence to WHO Good Manufacturing

Practices.

2.3.1 Clinical ICH CTD Module 2 and 5 for new chemical entities, vaccines and biosimilars (for the last one, reduced clinical data will depend on proof of its similarity to an appropriate RBP through the comparability exercise – based on WHO guidance). Note, if it’s a multisource FPP, based on WHO Guidance, it’s only required demonstration of bioequivalence of the FPP. It may necessitate that the manufacturer carries out a bioequivalence study and an assessment of the bioequivalence study (trial) information: the data generated should provide a bridge between the comparator product for which safety and efficacy data are

https://extranet.who.int/pqweb/medicines/full-assessment-multisource-generic-fpps-0

https://www.who.int/biologicals/publications/trs/areas/biological_therapeutics/TRS_977_Annex_2.pdf?ua=1

https://www.who.int/medicines/areas/quality_safety/quality_assurance/Annex9-TRS992.pdf

Page 102







available and the generic products for which such data are not available.

2.3.1.1 For the clinical component, the following aspects should be considered: - GCP aspects - Biopharmaceutics - Clinical pharmacology - Clinical efficacy - Clinical safety - Paediatric studies - Risk Management Plan - PV system/Post-marketing experience

For clinical efficacy, the following sections should be discussed in the report, tabular overview of clinical studies, pharmacokinetics, pharmacodynamics, discussion on clinical pharmacology, clinical efficacy (Main study(ies), Treatments, Objectives, Outcomes/endpoints, Randomisation and blinding (masking), Statistical methods, Results, Participant flow, Baseline data, Numbers analysed, Outcomes and estimation, Ancillary analyses, Summary of main efficacy results, Analysis performed across trials (pooled analyses and meta-analysis), Clinical studies in special populations, Supportive study(ies)). The discussion on clinical efficacy should be clear and concise. For each section, the discussion should identify the most important findings and deficiencies and how results agree. It should indicate if the data submitted fulfil the requirements (legal, guidelines, scientific advice) The major issues raised how they were addressed should be reflected. Uncertainties should be considered by mentioning what is the source of the uncertainty (e.g., missing data), what is the item that the assessment is uncertain about (e.g., efficacy in a subgroup) and what are the possible coping strategies if possible (e.g., need to collect further data to reduce uncertainty; acknowledge through labelling changes). Both study design and results should be subject to the critical discussion. Be explicit about the view on key elements like choice of comparators, endpoints as well as shortcoming of the data. The following is a compilation of potential aspects to be addressed in such discussion. Design and conduct of clinical studies ▪ Was the design of the studies adequate (randomised active and placebo-controlled trials)? If not, what are the justifications and are they acceptable? ▪ Was the patient population adequately selected (reflection on inclusion/exclusion criteria)? Was there any age limit exclusion? ▪ Is the comparator considered appropriate? In case of an active comparator, discuss the relevance in view of the national/local established clinical practice

guideline and treatment options. ▪ Critical discussion of the appropriateness of the choice of endpoints as well as the duration of the study considering regulatory guidance/scientific advice.

Validity of surrogate markers to replace hard endpoints? Acceptability of a composite endpoint and its domains? ▪ Adequacy of the methods, conduct, analysis and reporting of results from main studies, as appropriate. Discuss any particular issues raised regarding the

study design. ▪ Is the design in accordance with legal requirements, available guidelines, and scientific advice? ▪ What are the implications of any GCP inspection? Efficacy data and additional analyses ▪ Magnitude and clinical relevance of the effect. Clinical relevance of the observed effect should be described since it may be particularly important for the

benefit /risk assessment. ▪ What are the key findings (or uncertainties)? What key findings (or uncertainties) should be part of the benefit-risk assessment? ▪ Generalisability (external validity) of trial findings. Do the results support the claimed indication? ▪ Are any additional analyses required and what are the reasons for this request? ▪ If sub-group data is considered of particular relevance for the overall assessment of efficacy, this should be explained. ▪ What major issues were raised during the assessment (major objections and other important concerns) ▪ Discuss any justifications for waiving certain studies or replacing original studies by literature data.


Page 103






▪ Lack of information in certain groups of patients (children, elderly women with childbearing potential etc.) should be mentioned to qualify statement made in the product information.

▪ Which are specific considerations for the paediatric population? ▪ How are the findings (or lack of information) reflected in the product information? Ensure that all information in the product information is explicitly

assessed and supported by the scientific assessment. A brief statement about the conclusions in terms of establishing efficacy that can be drawn from the clinical efficacy documentation should be provided. For the clinical safety, the following aspects should be considered: patient exposure, adverse events, serious adverse events and deaths, laboratory findings, safety in special populations, immunological events, safety related to drug-drug interactions and other interactions, discontinuation due to adverse events, post marketing experience, proposed risk management plan (including identified and potential risks) and risk minimisation measures and adequacy of the plan an measures to address the risk. The discussion on clinical safety should be clear and concise. For each section, the discussion should address the following points: ▪ Identify the most import findings and deficiencies and describe how results agree. A summary of evidence for each conclusion should be given. ▪ Indicate how the data submitted fulfil the requirements. ▪ Describe the major issues raised during the assessment and how they have been addressed. ▪ Conclude and state what information should be reflected in the product information. ▪ What key findings (or uncertainties) should be part of the benefit- risk assessment.

Specific points for discussion ▪ Patient exposure: Discuss any limitations of the safety database in relation to the proposed target population. ▪ How are the findings (or lack of information) reflected in the product information? Ensure correspondence product information (e.g., contraindications,

special warnings, Effects on ability to drive and use machines Undesirable effects, Overdose, as appropriate) and that all information in product information is explicitly assessed and supported by the scientific assessment.

▪ Description of the safety profile of the medicinal product and degree of safety assessed ▪ Is the safety profile in accordance with that expected from non- clinical studies and known class effects? ▪ Describe relevant safety aspects specific for the paediatric population by age group where appropriate. Link this closely to the recommendations in the

product information. Are there any specific (serious) ADRs and/or monitoring requirements? ▪ Sufficient long-term data? Mention if there are any outstanding data which remain as follow-up after authorization and if this is reflected in the product

information. Additional post-marketing studies? A brief statement about the conclusions that can be drawn from the clinical safety documentation should be provided.

2.3.2 Quality ICH CTD Module 2 and 3 for new chemical entities, vaccines and biosimilars (for the last one, a comparison of the SBP and the RBP with respect to quality represents an additional requirement to the “traditional” full quality dossier – comparability exercise - based on WHO guidance). Note, if multisource, based on WHO Guidance, it’s only required: demonstrating the quality of the API(s), demonstrating the quality of the FPP, demonstrating adherence to WHO Good Manufacturing Practices

2.3.2.1 For the quality, the following aspects should be considered: - Drug substance (CTD module 3.2.S)

▪ General information,


https://extranet.who.int/pqweb/medicines/full-assessment-multisource-generic-fpps-0

Page 104







▪ Manufacture, ▪ Characterisation, ▪ GMP compliance, ▪ Control of drug substance, ▪ Reference standards of materials, ▪ Container closure system, ▪ Stability

- Drug product (CTD module 3.2.P) ▪ Description and composition of the drug product, ▪ Pharmaceutical development, ▪ Manufacture, ▪ Control of excipients, ▪ GMP compliance, ▪ Control of drug product, ▪ Reference standards or materials, ▪ Container closure system, Stability

- Elements from Appendices (CTD module 3.2.A) ▪ Facilities and equipment, ▪ Adventitious agents safety evaluation, ▪ Novel excipients

Key aspects and summaries of relevant studies (including comparability, if applicable) that are essential in providing reassurance with regard to the quality of drug substances and drug product should be provided in the assessment report. The assessment report should include a general background of the product to identify the main critical features: active substance (e.g. new chemical entity, known chemical active substance, biosimilar), if paediatric formulation has been/is to be developed, orphan status, indications, target population, posology, method of administration (e.g. use of device), use of delivery/administration systems and preparation/reconstitution of product. It should be mentioned whether a CEP or ASMF procedure or full information of the active substance in the dossier is used. When ASMF procedure is used, restricted part with information which is protected by intellectual property rights or is otherwise sensitive should not be disclosed to the applicant. Letters of Access in relation to specific drug product should be described. The report should be sufficiently detailed to allow for secondary assessment. Quality matters should relate to efficacy and safety consequences as much as possible. It should be indicated if there is any quality aspect either in the active substance or in the finished product which could lead to impact on the Benefit- Risk Balance. Scientific argumentation in the assessment report should support the proposed questions and the report should emphasise those findings that need to be reflected in the SPC, labelling and package leaflet. A very brief summary of the conclusions drawn from the quality documentation should be provided.

2.3.3 Non-clinical ICH CTD Module 2 and 4 structure for new chemical entities, vaccines and biosimilars (for the last one, reduced non-clinical data will depend on proof of its similarity to an appropriate RBP through the comparability exercise – based on WHO guidance). Note, if it’s a multisource FPP, this section is not required.



Page 105






2.3.3.1

For the non-clinical, the following aspects should be considered: - GLP Aspects, - pharmacology, - pharmacokinetics, - toxicology, - ecotoxicity/environmental risk assessment and - Implications of the assessment of non-clinical data for the Safety Specification of the Risk Management Plan (RMP).

For each section, the discussion should address the following points: ▪ Identify the most import findings and deficiencies. Describe how results agree. Summarise evidence for each conclusion. ▪ State if the data submitted fulfil the requirements, comment if the non-clinical study program was built up by the risk-based approach i.e. with possible

omission of in vivo studies. ▪ Describe the major issues raised and how they have been addressed. For example, for each indent of the non-clinical part, the following consideration should be included: ▪ data submitted in accordance with legal requirements, available guidelines and scientific advice, ▪ any justifications for waiving certain studies or replacing original studies by literature data, ▪ what major issues were raised (major objections and other important concerns) and how they were addressed? ▪ how are the findings (or lack of information) reflected in the product information? ▪ ensure correspondence with product information (particularly preclinical safety data but also e.g., contraindications, Interactions, Pregnancy and lactation,

non-clinical pharmacodynamic properties, non-clinical pharmacokinetic properties, if relevant), ▪ ensure that all information in the product information is explicitly assessed and supported by the scientific assessment ▪ what key findings (or uncertainties) should be part of the benefit- risk assessment, or biosimilarity assessment for biosimilars? A very brief summary of the conclusions drawn from the non-clinical documentation should be provided.


2.4 Scientific Opinions/outcomes & Final Decision-making

2.4.1 Scientific Opinion How is an overall benefit risk assessment generated for an application? Are the conclusions on risk-benefit analysis and overall assessment outcomes consistently and adequately reached and concluded, in line with the assessment report observations, concerns and evidence reviewed? Are the assessment outcomes adequately considering the Risk management plans? How did the assessors achieve an integrated opinion? Is there input or advice from scientific committees, or from external experts? How was this integrated into the scientific opinion? How were divergent views handled, if any?

Overall, the assessment outcomes/opinions are aligned with the observations made throughout the assessment process. It reflects all observations and concerns as per those identified in the assessment report. All input received during the assessment is adequately reflected in the report and in the scientific opinion. Benefit-risk based decisions are inclusive, comprehensive, documented and consistent. In the positive scientific opinions, the benefits clearly outweigh the risks, based on sound scientific evidence. The production of the integrated opinion of assessors and their senior managers for the final decision-making by the agency (e.g. to the development and agreement on a

Page 106







positive opinion to authorise a medicinal product), is consistently and adequately achieved.

2.4.2 Final Decision-Making Are the structures and levels of decision-making adequate and relevant? What was the basis on which decisions were taken? Is the information provided to the decision-makers adequate and relevant? Are some decisions taken at lower levels in the agency? On what basis is this done? Was the expertise used for decision making adequate? Were scientific committees and/or other stakeholders involved? How were divergent views handled, if any? How was consistency of opinion making ensured? Was the appropriateness of the process and of the decisions subject to review?

Overall, the final decision-making is aligned with the assessment report and observations made throughout the assessment process. It reflects all observations and concerns as per those identified in the assessment report and its outcomes. All input received during the assessment is adequately reflected in the final decision-making. The decision-making at the NRA is consistently ensured, supported by documented procedures which were adequately followed. The conclusion on the final decision by decision-makers at the agency was adequately achieved.



3 Assessment follow-up

3.1 Are the post-approval actions well reflected in the product file? Further post-approval actions taken (if any) are adequately reflected in the product file

3.2 In case of emergency approvals (or approvals provided under exceptional circumstances), are there follow-ups after introduction with respective reflection in the product file?

In case of emergency approvals (or approvals provided under exceptional circumstances), there are appropriate follow-ups after introduction of the product, with respective reflection in the product file.


4 Assessors’ Technical Competency

4.1 Assessment team members have the required background and education. Assessment team members have the adequate qualifications (in terms of training and experience) in the field of MA assessments.

The team formulated for this assessment is perceived to be adequate, in terms of background, experience and apparent theoretical and practical knowledge on the relevant fields (MA).


Page 107






4.2 Assessment team members are familiar with national and international regulations and guidance.

The team members made reference of relevant national and international regulations and guidance throughout the assessment report.

4.3 Assessment team members are coherent in scientific outcomes and decision and provide adequate rational and scientific justifications for their comments

The team members were coherent in scientific outcomes and decision and provided adequate rational and scientific justifications for their comments.

4.4 Were conflicts of interest of staff and external experts properly dealt with, by the NRA (if any)? Was there any conflict of interest of staff and external experts perceived by the expert during this review?

The NRA properly deals with conflict of interests of assessors. The team formulated is perceived to be adequate for the assessment of this product, in terms of apparent conflict of interests.


1979

1980

References 1981

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health 1982

Organization; 2021: Sub-indicators MA01.09; MA04.01; MA04.04; MA04.09; MA04.10 (https://apps.who.int/iris/handle/10665/341243, accessed 1983

29 June 2021). 1984

2. Good review practices: guidelines for national and regional regulatory authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 1985

Preparations: forty-ninth report. Geneva: World Health Organization; 2015: Annex 9 (WHO Technical Report Series, No. 992/2015; 1986

https://www.who.int/medicines/areas/quality_safety/quality_assurance/Annex9-TRS992.pdf, accessed 29 June 2021). 1987

1988

Rating scale 1989

The assessor uses the expert evaluation and comments to conclude whether or not the NRA or RRS can be considered to acceptably meet the 1990

requirements across the for main sections of the tool. 1991

1992

If the experts disagree on the outcomes of this evaluation, the decision will be referred to the WHO secretariat. 1993

1994

In all cases, the outcomes and findings of the expert review should be written up in a final report (using the tool and template presented in Table 1995

MA.12) and handed to the WHO secretariat together with the other MA PEP results. 1996

1997


https://www.who.int/medicines/areas/quality_safety/quality_assurance/Annex9-TRS992.pdf

Page 108


Limitations and remarks 1998

N/A1999

Page 109 of 330


Page 109

3. Vigilance (VL) 2000

2001

3.1. VL PEP methodology 2002

The PEP for VL is designed to assess the performance of an NRA or RRS in conducting medical 2003

products vigilance. 2004

2005

In addition to meeting the eligibility criteria, PEP for VL is considered fulfilled if the NRA or RRS 2006

demonstrates to: 2007

a. fully implement four mandatory ML4 GBT sub-indicators for VL (see Section 3.2); and then 2008

b. acceptably meet 16 newly developed VL performance evaluation indicators (see Section 3.3); and 2009

c. successfully undergo a vigilance field visit (see Section 3.4). 2010

2011

Performance against the new VL indicators is evaluated using a combination of self-assessment, 2012

remote review and onsite assessment during the field visit (see Figure 3.1). 2013

2014

Figure 3.1. Flowchart of the VL PEP. 2015

2016

2017

2018



START

END

YES

YES

Does the conclusion of field visit & evaluation support WLA listing for VL?

Are new VL indicators met & VL scorecard completed by NRA/RRS (self-assessment)?

VL PEP fulfilled

NO

WHO remote review of new VL indicators self-assessment

Conduct VL field visit, including onsite assessment of new VL indicators

NO


YES

NO

YES

NO

Page 110


The GBT indicators benchmark systemic aspects of the function (for example, established 2019

legislations, organization and governance, available resources, QMS, transparency). The new PE 2020

indicators are seen as a proxy of performance through qualitative and quantitative indicators of 2021

VL. The VL field visit helps to evaluate and assess the VL and related activities in practice and 2022

establish whether these are properly in place. Decisions on the overall performance of the VL 2023

function are made based on the collective evidence of these tools and methodologies. 2024

2025

The full VL PEP is estimated to take 3–6 months to complete. 2026

2027

3.2. Mandatory ML4 GBT sub-indicators for VL 2028

WLAs for vigilance must fully implement the following ML4 indicators, as defined in the GBT: 2029

1. VL04.03: Standard procedures exist and are implemented for enforcement of the national 2030

vigilance system. 2031

2. VL04.07: With respect to vigilance data, assessment of the risk-benefit balance of medical 2032

products is regularly conducted. 2033

3. VL04.08: Active vigilance activities, as well as proactive monitoring programmes (when needed) 2034

have been developed and implemented. 2035

4. VL05.02: Performance indicators for vigilance activities are established and implemented. 2036

2037

3.3. WLA VL performance evaluation indicators 2038

To be listed as WLA for VL, once an NRA or RRS has acceptably met all mandatory ML4 GBT sub-2039

indicators, it must be assessed against 16 newly developed VL indicators (see PE.VL.01—PE.VL.16 2040

below). These indicators have been designed to enable an evaluation of the baseline situation 2041

and progress in the performance of key vigilance structures and processes. As far as possible, 2042

each indicator aims to be specific, measurable, achievable, realistic and timely. 2043

2044

Data for assessing performance against the indicators can be obtained from multiple sources, 2045

including: 2046

• databases, including national database (census figures, registers) and pharmaceutical databases 2047

(sales, prescription, consumption); 2048

• national pharmacovigilance centres; 2049

• immunization programmes; 2050

• hospital or clinic records; 2051

• surveys; and 2052

• peer-reviewed publications. 2053

2054

These data may be qualitative or quantitative. In some cases, data may be required over several 2055

years to identify and track trends. In these cases, the NRA or RRS can use any administrative 2056

calendar with a cycle of 365 days that is routinely used in the country; it does not necessarily 2057

have to run from January to December. 2058

2059

Some impact or outcome indicators will require data that can only be gathered through new 2060

surveys or studies, which may require specific expertise, may be time- and resource-intensive, 2061

Page 111 of 330


Page 111

and will need to be closely coordinated with other relevant organizations (such as the ministry of 2062

health, national office of health statistics, universities, and research agencies). 2063

2064

Each new indicator has a rating scale that comprises three distinct categories: 2065

1. IMPLEMENTED (I) 2066

2. PARTIALLY IMPLEMENTED (PI) 2067

3. NOT IMPLEMENTED (NI) 2068

2069

If the NRA or RRS being assessed cannot fully meet the criteria for either the first or second 2070

category, the indicator must be scored as “not implemented”. Once each indicator has been 2071

assessed, its score should be recorded in the VL PE indicators scorecard (see Section 3.4 below). 2072

2073

PE.VL.01. Dissemination of vigilance information through various channels such as newsletters, information bulletins or websites.

Description: One of the expected functions of a national vigilance system is to provide effective communication on aspects related to safety of medical products, including both routine communication and communication during crises. The assessor should verify:

• The existence of a communication plan for routine medical product

safety information identifying: target audience, communication channels

and strategies differentiated by audience type (including multi-language

strategies if appropriate); frequency of routine messaging; feedback

mechanisms and records; and systems for responding and acting on

feedback received.

• The consistency of information dissemination with the dissemination

plan over the past three years.

• The sufficiency of printed material that is distributed, in terms of reach

across all stakeholders (including across different regions, languages and

sub-populations).

• If printed material is distributed, the availability of a mechanism to

ensure the edition is sufficient for distribution to stakeholders across the

whole country and language areas, if any, and the availability of a

mechanism to ascertain the functionality of the distribution system.

• The presence of a responsible communication officer or team.

• The accessibility of language used and quality of design in distributed

information, both electronic and printed.

• The existence of a separate communication plan for times of crisis,

including information on when and how this should be applied (for

example by identifying targets, methods, content and time scale for crisis

Page 112


communication), and who is authorized to put the crisis communication

plan into action.

This indicator should be evaluated alongside GBT sub-indicators VL02.02,

VL06.01, and VL06.02.

Objective: This indicator aims to ensure that the NRA or RRS has established mechanisms for raising awareness, and disseminating information, about VL-related activities, processes and outputs/outcomes. Such mechanisms encourage all stakeholders to be involved and contribute to the vigilance system.

Evidence to review:

The assessor should ask for and review: • the plan for standard communications of safety messages;

• the plan for crisis communication; and

• records of any feed-back from recipients of messages including possible telephone hot-lines.

The assessor should also examine samples of past communications, including:

• materials produced by the NRA, such as newsletters, bulletins, web sites,

social media posts, messaging apps targeted towards different

stakeholders;

• contributions in external media (printed or broadcast). References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory

system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators VL02.02, VL06.01 and VL06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicator CST9 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Communication tools or mechanisms for dissemination of vigilance related activities, processes and outputs/outcomes do not exist PARTIALLY IMPLEMENTED (PI): Communication tools or mechanisms for dissemination of vigilance related activities, processes and outputs/outcomes exist however quality of their content is not satisfactory or vigilance information is not appropriately disseminated IMPLEMENTED (I): Communication tools or mechanisms for dissemination of vigilance related activities, processes and outputs/outcomes exist, vigilance information is appropriately disseminated and quality of their content is satisfactory



Page 113 of 330


Page 113

For an authority to be given WLA status for VL, this indicator must be scored

as “implemented”.


This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing WLAs for VL). The information is qualitative, and the presence or absence of the tool(s) is noted.

PE.VL.02 The VL centre provides learning activities for healthcare professionals as well as for the public.

Description: The assessor should verify that:

• The VL centre is actively involved in promoting the reporting of individual

case safety reports. This can be a responsibility of the centre itself or may

be done in collaboration with partners (for example, academic institutions,

healthcare providers, immunization programmes, disease programmes,

MAHs, media or civil society organizations).

• The VL centre documents face-to-face training activities, including

audience targeted, training materials used and, where relevant, any

partners involved. Training materials should be checked for quality.

Estimates of the number of individuals reached by the training should be

given. Continuous training material, for example, materials provided on-

line, should be checked and verified for quality and feedback.

• The VL centre regularly reviews and, if necessary updates, training

materials and methods for continual improvement.

This indicator should be evaluated alongside GBT sub-indicators VL02.02, VL06.01, and VL06.02.

Objective: This indicator aims to ensure the presence of a learning programme within the NRA or VL centre for healthcare professionals and the public. Such programmes help spread knowledge and awareness about VL among key audiences, including how to reach to the NRA/VL centre.

Evidence to review:


• sample training materials for different audiences;

• training statistics (including number, type, and geographic spread of

participants);

• feedback from trainees and partners; and

• sample evaluations of training effectiveness.

In addition, the assessor should request any evidence of a positive impact from learning activities, for example, an increase in the rate of reporting or positive opinions about the vigilance system.

Page 114


The information is qualitative and the presence or absence of the information is noted.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators VL02.02, VL06.01 and VL06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).


Rating scale:

NOT IMPLEMENTED (NI): Learning programme (which would normally include objectives, deliverables, target audience, resources, plan and schedule) for healthcare professionals and the public do not exist. PARTIALLY IMPLEMENTED (PI): Learning programme for healthcare professionals and the public exist but their coverage, quality, or quantity of the implementation is not satisfactory. IMPLEMENTED (I): Learning programme for healthcare professionals and the public exist and their coverage, quality, as well as quantity of the implementation is satisfactory.




This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing WLAs for VL). Learning activities can be organized by organizations other than the NRA or VL centre. But the NRA or VL centre should significantly contribute to them.

PE.VL.03 A national ADR or vigilance advisory committee or AEFI committee or an expert committee in the setting is functional and capable of providing advice on medical product safety and produces quality outputs

Description: A qualified committee can provide advice and technical assistance to support all the main functions of a VL system. Such committees should be made up of at least three people with different professional backgrounds in health care. The committee can meet according to a regular schedule; or be called upon ad hoc as a need emerges. Some of the qualitative aspects of this indicator overlap with GBT sub-indicator VL04.06 but more consideration is given to quantitative aspects. The assessor should verify that:



Page 115 of 330


Page 115

• The terms of reference for the committee cover the major needs for

specialist advice facing the VL centre. The frequency of meetings should be

noted.

• The committee expertise enables specialist advice to be provided on most

safety issues relevant to the VL centre.

• Committee members have relevant experience or training for their role.

• Appointment letters for committee members, including statements of

confidentiality and conflict of interest, are available.

• Minutes from past meetings are available. The frequency, attendance and

recommendations from meetings should be noted; and reasons identified

for any discrepancy between actual and planned meeting frequency.

This indicator should be evaluated alongside GBT sub-indicator VL04.06. Objective: This indicator aims to ensure that the VL centre has access to a qualified

committee for advice and technical assistance on causality assessment, risk assessment, risk management, case investigation and, where necessary, crisis management and communication.

Evidence to review:


• the TOR for the committee;

• letters of appointment of committee members;

• minutes from committee meetings; and

• proofs of regulatory actions taken based on inputs provided by the

committee.

The information is qualitative and the presence or absence of the information is noted.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator VL04.06 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).


Rating scale:

NOT IMPLEMENTED (NI): Qualified committee does not exist; or qualified committee exists but there is a clear conflict of interest for one or more of its members. PARTIALLY IMPLEMENTED (PI): Qualified committee exists but it is not fully functional or the quality of its outputs is not satisfactory. IMPLEMENTED (I): Qualified committee exists and is functional with quality outputs over at least two years.



Page 116





This indicator can be scored as “not applicable”; but if it is, the NRA/VL centre should provide evidence that there is internal capacity to fulfil all the usual roles and responsibilities of an external qualified committee.

PE.VL.04 Total number of ADR reports received in the last three years (also expressed as number of ADRs per 100 000 persons in the population).

Description: This indicator serves to measure VL activity in the setting, including the awareness and willingness of health professionals and the public to report ADRs. Valid case reports should contain four core data elements, as per ICH E2A: 1. reporter

2. identifiable patient

3. suspected medical product

4. adverse reaction.

Trends in this indicator enable authorities to appreciate the effectiveness of measures taken to improve quantitative reporting. Measures expressed in relation to population size allow for comparisons across and within countries.

The assessor should verify:

• The reporting trend over the last three years and ascertain that the

methodology producing the reporting statistics has remained consistent

over time and that the underlying population base has remained the

same.

• Possible reasons for major deviations between annual reporting rates e.g.,

technical development, expansion of products to be monitored or

reporting base (direct patient reporting or mandatory reporting

requirements for MAH), promotional activities or media attention on

specific safety issues.

This indicator should be evaluated alongside GBT sub-indicator VL04.01. Objective: This indicator aims to provide a crude measure of the attention paid to

medical product related harm in society and in the healthcare system. The extent of reporting also demonstrates a general understanding that the prevention of future harm from medical products must be based on observations of current harm; and that reporting is a prerequisite for learning.

Evidence to review:

The assessor should ask for and review data from at least the last three years, including:

• absolute number of reports; and

Page 117 of 330


Page 117

• number of reports per 100 000 people in the population.

These data should be available through the VL centre where reports are received and collated.


2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicator CP1 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale: NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e. decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator show a negative (i.e.: decreasing), but justified, trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show a stable or positive trend (i.e.: increasing).


as “partially implemented” or “implemented”. Limitations and remarks:

This sub-indicator cannot be scored as "not applicable" (i.e.: this sub-indicator always applies when assessing WLAs for VL).

PE.VL.05 Percentage of individual case safety reports (ICSRs) acknowledged and/or given feedback in the last three years.

Description: Staff at a VL centre are expected to acknowledge and give feedback to all ICSRs received (even if this through electronic feedback or automatic acknowledgements). The number of responses given, expressed as a fraction of the total number of reports received during a year is a measure of the responsiveness of the centre to submitted reports. A high number suggests a commendable level of response; a low one may discourage reporters.

The assessor should verify that:

• The VL centre has routines for acknowledging or responding to all reports

irrespective of how they are received (paper, electronic, telephone, or

other). The acknowledgement confirms receipt and preferably gives an

internal record number of the case to enable follow up by the reporter.

Additional feedback, for example on prior experience of the suspected

problem, can be valuable in stimulating further reports.



Page 118


• Processes exist for verifying the receipt of ICSRs submitted in compliance

with regulatory reporting requirements for MAHs, as specified in ICH E2B

guidelines.

• The VL centre provides expedient responses. An immediate response is

most gratifying for the reporter but may not be administratively feasible.

This indicator should be evaluated alongside GBT sub-indicators VL04.01,

VL06.01, and VL06.02.

Objective: This indicator aims to ensure that reporters receive some individual acknowledgement and information from the VL centre.

Evidence to review:

The assessor should ask for and review records for the past three years on: • the number of reports received; and

• the number of responses sent out.

This indicator can then be calculated as: (Number of reports given a response during one year / Total number of reports received in the same year) × 100 In addition, the assessor should identify the reasons for any deviation from

100% response rate; and should identify the number of, and reasons for,

responses that were sent later than one month after receiving the original

report.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator VL04.01, VL06.01, VL06.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).


3. ICH Harmonised tripartite guideline: maintenance of the ICH guideline on clinical safety data management: data elements for transmission of individual case safety reports E2B(R2). Geneva: International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use; 2001 (https://admin.ich.org/sites/default/files/inline-files/E2B_R2_Guideline.pdf, accessed 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e. decreasing) trend over the last three years; or the absolute value of this indicator for any one year over the last three years is less than 60%. PARTIALLY IMPLEMENTED (PI): Data for this indicator show a negative (i.e., decreasing) but justified trend over the last three years; or the absolute value of this indicator for any one year over the last three years is less than 90%.



https://admin.ich.org/sites/default/files/inline-files/E2B_R2_Guideline.pdf

https://admin.ich.org/sites/default/files/inline-files/E2B_R2_Guideline.pdf

Page 119 of 330


Page 119

IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e. increasing) trend; and the absolute value of the indicator over the last three years is more than 90%. For an authority to be given WLA status for VL, this indicator must be scored

as “partially implemented” or “implemented”.


This sub-indicator cannot be scored as "not applicable" (i.e. this sub-indicator always applies when assessing WLAs for VL).

PE.VL.06 Percentage of annual reports satisfactorily completed and submitted to the national VL centre in the last three years.

Sub-indicator: Of the reports satisfactorily completed and submitted to the national vigilance centre, percentage of reports committed to the WHO global database of ICSRs (VigiBase).

Description: This indicator reflects the completeness of reports received by the VL centre and points to reporters’ understanding of, and willingness to complete, the critical elements in ADR forms (where low values of the indicator suggest high numbers of poor-quality reports). The sub-indicator reflects the centre’s contribution to global learning about the harm caused by medical products. Submitting reports to WHO is a requirement for full members of the WHO Programme for International Drug Monitoring. According to ICH E2B, valid ICSRs include at least:

• one identifiable patient,

• one identifiable reporter,

• one reaction/event, and

• one suspect drug.

Information about the patient and reporter (name and contact details) is confidential and should not be shared with VigiBase. The assessor should verify:

• the proportion of reports received that fail to meet international criteria

for a valid ICSR (identified reporter, identifiable patient, suspected medical

product, suspected reaction);

• the efforts made to contact original reporters of incomplete reports to get

missing information and turn incomplete reports into valid ICSRs;

• the proportion of valid ICSRs in the national VL database that have been

shared with VigiBase over the last three years; and

• any reports that have not been shared with VigiBase, including reasons for

non-submission.

Page 120


This indicator should be evaluated alongside GBT sub-indicators VL04.01, VL04.02 and VL06.03.

Objective: This indicator aims to ensure the completeness of reports received by the VL centre; and the centre’s commitment to sharing that information through global platforms.

Evidence to review:

The assessor should ask for and review annual data from the last three years on:

• the total number of ICSRs received;

• the number of incomplete reports received;

• the number of reporters submitting incomplete reports that could be

contacted, allowing incomplete reports to be turned into valid reports.

The indicator can then be calculated as: (Number of complete reports received during one year / Total number of reports received during the same year) × 100 In addition, the assessor should check how many completed reports were submitted to VigiBase; and then calculate the value of the sub-indicator as: (Number of complete reports submitted to VigiBase during one year / Total number of complete reports added to the national database during the same year) × 100

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator VL04.01, VL04.02 and VL06.03 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicators CP5 and CP5a (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show unjustified negative (i.e.: decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator shows negative (i.e.: decreasing) but justified trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend over the last three years.







Page 121 of 330


Page 121

PE.VL.07 Number of active surveillance activities started, ongoing or completed during the past five calendar years.

Description: Active surveillance efforts follow a defined population of exposed individuals and actively look for adverse reactions. They include phase 4 clinical trials, cohort event monitoring, targeted spontaneous reporting, pregnancy exposure registries and a range of other epidemiological studies. Active surveillance is particularly useful to characterize specific adverse reactions and to investigate specific populations or problems. The assessor should verify:

• the number of active surveillance projects started, ongoing or completed

in the last five years (not including studies commissioned as part of Post

Authorization Safety Studies);

• whether the VL centre was directly involved in planning or implementing

any of these projects (not including projects in which individual staff

members may have participated if these were not commissioned by the

centre);

• the purpose, methodology, size, duration and main outcome of each

project; and

• the adherence of project protocols and analyses with scientific good

practice.

This indicator should be evaluated alongside GBT sub-indicator VL04.08. Objective: This indicator aims to ensure an active surveillance system. It is designed to

reflect the dynamism of VL activities, and the VL centre’s awareness of efforts in the setting.

Evidence to review:

The assessor should look for: • and verify the number of active surveillance activities carried out over the last

five years; and

• and identify any trends of the data.

The assessor should also select: • a sample of projects and studies (to verify quality).





Page 122


Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e. decreasing) trend over the last five years; or the quality of sample studies checked is not satisfactory. PARTIALLY IMPLEMENTED (PI): Data for this indicator show a negative (i.e. decreasing) but justified trend over the last five years; and the quality of the sample studies checked is satisfactory. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend; and the quality of the sample studies checked is satisfactory.




This sub-indicator cannot be scored as "not applicable" (i.e. this sub-indicator always applies when assessing WLAs for VL). The VL centre may not be aware of all studies conducted in the setting (in which case it should be able to justify its lack of knowledge).

PE.VL.08 Number of signals detected in the last three years by the vigilance centre.

Description: The ability of the vigilance system to detect signals underscores its relevance in ensuring the safe use of medical products. A signal is defined as reported information on a possible causal relationship between an adverse event and a medical product, the relationship being previously unknown or incompletely documented. It usually takes more than a single report to generate a signal, depending upon the seriousness of the event and the quality of the information. In this document, a signal includes a previously unreported ADR, problems of use, and poor-quality medical products.

The assessor should verify the number and character of safety signals produced by the signal analysis process during the past three years. The assessor should then review these values to identify, analyse and seek justification for any trends. This indicator should be evaluated alongside GBT sub-indicator VL04.02.

Objective: This indicator aims to assess the functionality of the signal detection system to ensure the safety of medical products.

Evidence to review:

The assessor should ask for and review documentation about signals from the VL centre, preferably classified as: • pharmacological/biological action related

• medical product use related

• medical product quality related

Page 123 of 330


Page 123

The indicator should be stated as the absolute values of the number of signals detected in the preceding three years. The assessor should then check the trend of these values along with any trend analysis/justification.


2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicator CO1 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show unjustified negative (i.e. decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): data for this indicator show negative (i.e.: decreasing) but justified trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e. increasing) trend over the last three years. For an authority to be given WLA status for VL, this indicator must be scored as “partially implemented” or “implemented”.



PE.VL.09 Number of regulatory actions taken over the last three years as a consequence of national VL activities including:

• number of product label changes (variation);

• number of safety warnings on medical products to: (i) health professionals, (ii) general public.

• number of withdrawals of medical products;

• number of other restrictions on use of medical products

Description: An effective VL system enables regulatory authorities to issue advice and take action that ensures the safe use of medical products. The assessor should verify the regulatory actions taken as a result of safety signals from the national VL system over the last three years, including the number of:

• product labelling changes (variations);

• safety warnings on medical products to: (i) health professionals, (ii) general public;

• withdrawals of medical products; and

• other restrictions on use of medical products. For each regulatory action taken, the assessor should verify:



Page 124


• the time taken between signal identification and action taken;

• the contribution of domestic safety data to the action taken, relative to foreign data and information in the literature (regulatory measures taken solely on the basis of information or data from other countries should not be counted); and

• any impact of the action taken that was followed up by the NRA or RSS, for example by measuring a decrease in reporting of the problem or in the consumption of the implicated product.

In addition, the assessor should verify the number of signals not leading to any regulatory actions. This indicator should be evaluated alongside GBT sub-indicator VL04.03.

Objective: This indicator aims to provide a measure of regulatory decisions, based on vigilance activities, that are taken to ensure safety in the use of medical products. It also measures the functionality of the VL centre and the interface between the centre’s activities and those of the regulatory agency.

Evidence to review:

The assessor should ask for and review: • documentation on the number and characterization of regulatory actions.

Note that regulatory measures taken solely on the basis of information or data from other countries should not be counted.



Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show unjustified negative (i.e. decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator shows negative (i.e.: decreasing) but justified over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend over the last three years.







Page 125 of 330


Page 125

PE.VL.10 The NRA performs or initiates studies on VL regulatory impact (for example, medical product-related hospitalizations or deaths).

Description: This indicator evaluates the outcome and impact of the VL function. It is a measure of injury to health resulting from medical products, including ADRs, medication errors, misuse or abuse of medical products, counterfeit or substandard medical products, and poisonings. It may also include injury to health from the under-use of medical products (for example, poorly managed diabetes). The assessor should verify that:

• the VL centre has taken initiatives to measure the burden of medical product-related harm in society, and particularly in the healthcare system (for example, through scientific studies); and

• study protocols and practices adhere to normal scientific good practice. Scientific studies are usually managed by the VL centre, or conducted in collaboration with partners (in clinical or academic settings). They usually cover a limited period of time and look at the incidence of medical product-related admissions to hospitals, the number of ADRs or deaths happening during hospital stay, or the extension of hospital stays due to medical product related problems. If studies cover an extended period of time, the burden of harm should show a decreasing trend.

Objective: This indicator aims to ensure the effectiveness of provisions for safeguarding health through VL.

Evidence to review:

The assessor should look for and review information on this indicator from: • regulatory agency records, including publications of scientific studies on the

burden of medical product-related harm.

References: 1. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicators CO3, CO4 and CO5 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show unjustified negative (i.e. decreasing) trend over the last three years; or quality of the sample studies checked is not satisfactory. PARTIALLY IMPLEMENTED (PI): Data for this indicator show negative (i.e.: decreasing) but justified trend over the last three years; and the quality of the sample studies checked is satisfactory. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend over the last three years; and the quality of the sample studies checked is satisfactory.




Page 126



It is difficult to establish with certainty a causal link between the use of a medical product and an ADR. Furthermore, except in a case of poisoning, a medical product-related event is seldom considered to be the underlying cause of a hospitalization and is not recorded at the time of hospital admission. It is also difficult to follow up the deaths of outpatients, which may lead to an underestimate of this indicator. In many cases, medical products (or the lack of medical products) may contribute to rather than cause fatalities. Past studies of the burden of medical product-related harm in different settings often show high levels of preventability, indicating considerable potential for vigilance to contribute to safer patient care.

PE.VL.11 Percentage of registered medical products with a VL plan and/or a risk management strategy from the MAHs in the country.

Description: This indicator contributes to measuring the enactment of the regulatory requirements for vigilance plan and/or a risk management strategy of some registered medical products. The assessor should verify:

• the country’s regulatory requirements for MAHs to submit a VL plan or risk management strategy (including when these first entered into force); and

• how many products registered during the last three years submitted a VL plan or risk management strategy.

This indicator should be evaluated alongside GBT sub-indicator VL04.08.

Objective: This indicator aims to provide a measure of how regulatory requirements for VL plans and risk management strategies are enacted for specific registered medical products.

Evidence to review:

The assessor should ask for and review records from the VL centre; and use them to calculate this indicator as: (Number of registered products with a vigilance plan and/or a risk management strategy/ total number of registered products in the same period) × 100


2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicator P10 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e. decreasing) trend over the last three years.



Page 127 of 330


Page 127

PARTIALLY IMPLEMENTED (PI): Data for this indicator show a negative (i.e.: decreasing) but justified trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e. increasing) trend over the last three years.





PE.VL.12 Percentage of registered medical products for which periodic safety update reports (PSURs) were submitted and evaluated by the NRA or RRS as stipulated in the country over the last three years.

Description: This indicator assesses how many registered medical products PSURs submitted by MAHs; and to what extent these have been reviewed by the NRA or RRS. The assessor should verify:

• the country’s regulatory requirements for MAHs to submit PSURs (and when these requirements first entered into force);

• how many products registered during the last three years had submitted and evaluated PSURs; and

• how many PSURs submitted during the last three years have still not been evaluated.


Objective: This indicator aims to assess the enactment of regulatory requirements for MAHs to submit PSURs.

Evidence to review:

The assessor should ask for and review records from the vigilance centre; and use them to calculate this indicator as: (Number of registered medical products for which PSURs were submitted and evaluated by the NRA or RRS / total number of registered medical products in the same time period) × 100


2. WHO pharmacovigilance indicators: a practical manual for the assessment of pharmacovigilance systems. Geneva: World Health Organization; 2015: Indicator P11 (https://apps.who.int/iris/handle/10665/186642, 30 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e.: decreasing) trend over the last three years; or the



Page 128


absolute value of this indicator for any one year over the last three years is less than 60%. PARTIALLY IMPLEMENTED (PI): Data for this indicator shows a negative (i.e.: decreasing) but justified trend over the last three years; or the absolute value of this indicator for any one year over the last three years is less than 90%. IMPLEMENTED (I): Data for this indicator show stable or positive (i.e.: increasing) trend over the last three years; and the absolute value of this indicator for every year over the last three years is more than 90%.





PE.VL.13 Percentage of healthcare institutions or public health programmes in the country that have submitted ICSRs during the past three years.

Description: This indicator evaluates how many healthcare institutions and public health programmes are actively reporting ICSRs. The assessor should verify:

• where and how many health care facilities operate in the country (including facilities of any size that prescribe and distribute any kind of registered medical products);

• the inclusion and exclusion criteria used to calculate the number of healthcare facilities;

• how many public health programmes there are in the country; and

• how many health care facilities and public health programmes have contributed at least one ICSR during the last three years.


Objective: This indicator aims to ensure the involvement of a wide range of health care institutions and public health programmes in the VL system. refers to the percentage of healthcare institutions and public health programmes that are actively reporting ICSRs to the NRA or VL center.

Evidence to review:

The assessor should ask for and review statistics and records from the VL centre for the last three years; and use these to calculate this indicator as:

• (Number of healthcare institutions that submit ICSRs/ total number of

healthcare institutions in the same period) × 100

• (Number of public health programmes that submit ICSRs/ total number of

public health programmes in the same period) × 100

Page 129 of 330


Page 129

In addition, the assessor should identify whether any specific public health

programmes, or geographic regions, appear not to be involved in VL activities. References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory

system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator VL02.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show unjustified negative (i.e. decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator shows a justified negative (i.e.: decreasing) trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e. increasing) trend over the last three years.





PE.VL.14 Number of registered medical products for which post-marketing safety or effectiveness studies were required and evaluated over the last three years.

Description: The assessor should verify:

• the country’s regulations on granting a conditional product registration,

subject to post-marketing safety or effectiveness studies (and when these

regulations entered into force.

• how many of the products registered over the last three years required

additional safety or effectiveness studies (including what the timeframe for

submission was for each one and how many were submitted on time); and

• how many post-marketing safety studies were evaluated and approved by

the NRA or RRS.


Objective: This indicator aims to evaluate regulatory oversight of the VL function for post-marketing safety and effectiveness studies.

Evidence to review:

The assessor should request records and documentation from the VL centre; and use them to calculate:

• Number of registered products with post-marketing safety or effectiveness

studies evaluated and approved over the last three years.




Page 130


Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available or show an unjustified negative (i.e.: decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator show a justified negative (i.e.: decreasing) trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend over the last three years.





PE.VL.15 Number of good vigilance practices (GVP) regulatory inspections over the last three years.

Description: This indicator helps determine the extent to which MAHs comply with regulatory requirements for VL, including the timely submission of required reports. The assessor should verify that the regulatory system includes requirements for MAHs to maintain VL systems and to regularly report safety information to the VL centre. Regulations should also empower the national regulatory authority to carry out vigilance inspections. The enforcement date of this legislation should be noted. In addition, the assessor should verify the existence of:

• an up-to-date routine vigilance inspection plan following risk-based principles;

• a separate emergency inspection plan for crises (for example, when the safety of patients or workers in manufacturing sites are under immediate threat); and

• routine and emergency site inspections (including the dates and locations visited over the last three years).

This indicator should be evaluated alongside GBT sub-indicators VL01.02, VL01.04 and VL04.03.

Objective: This indicator aims to ensure regulatory oversight of the legally required GVPs of MAHs.

Evidence to review:

The assessor should randomly select a number of inspection reports from the last three years and review them, along with other documents from the VL centre, including:

• plan for routine vigilance inspections;

Page 131 of 330


Page 131

• generic plan for crisis vigilance inspections, which should ideally be

adapted whenever needed; and

• records of vigilance inspections carried out the last three years. References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory

system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators VL01.04 and VL04.03. (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).


Rating scale:

NOT IMPLEMENTED (NI): Data for this indicator are not available (for example, a GVP inspection plan is not available or has not been implemented); or show an unjustified negative (i.e. decreasing) trend over the last three years. PARTIALLY IMPLEMENTED (PI): Data for this indicator shows a negative (i.e.. decreasing) but justified trend over the last three years. IMPLEMENTED (I): Data for this indicator are available and show stable or positive (i.e.: increasing) trend over the last three years.




This sub-indicator cannot be scored as "not applicable" (i.e. this sub-indicator always applies when assessing WLAs for VL).

PE.VL.16 All AEFI reports at the National Immunization Programme (NIP) from the last three years are also represented in the national VL database.

Description: The assessor should verify the country’s regulatory requirements for safety surveillance of medical products; and check whether the NRA or RRS responsible for vaccine registration is also responsible for safety surveillance. (Note that in some countries the functions are split between medicines and vaccines.) In addition, the assessor should verify:

• that routine processes are in place to ensure that all AEFI reports received

by the NIP are transferred to the VL centre responsible for vaccine safety

(and consequently to VigiBase); and

• whether AEFI reports about vaccines only used in the private sector (i.e.:

that are not part of a NIP) are expected to be shared with the NIP.


Objective: This indicator aims to ensure good coordination in AEFI reporting between the NIP and the NRA or VL centre.



Page 132


Evidence to review:

The assessor may ask for and review:

• Documentation and demonstration of functionality of processes by which

AEFI data is transferred from the NIP to the vigilance centre and in the

reverse direction if applicable.

The data for calculating this indicator should be obtained from the VL centre records.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators VL02.02. (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating Scale:

NOT IMPLEMENTED (NI): Data for this indicator is not available. PARTIALLY IMPLEMENTED (PI): Some, but not all, AEFI reports at the NIP from the last three years are included in national vigilance databases. IMPLEMENTED (I): All AEFI reports at the NIP from the last three years are included in the national vigilance database.





2074

2075

3.3.1 VL PE indicators scorecard 2076

The VL PE indicators scorecard is designed to enable the assessment against newly developed VL 2077

PE indicators. An NRA or RRS must complete and submit the VL PE indicators scorecard in advance 2078

of the vigilance field visit, as detailed below. 2079

As the NRA or RRS completes its self-assessment against each performance indicator PE.VL.01—2080

PE.VL.16 above, it should record its score in the VL PE indicators scorecard, using the template 2081

below. In each case, the NRA or RRS should justify its score through comments made under the 2082

respective column. 2083

The completed scorecard should be submitted to the WHO Secretariat as part of the assessment 2084

of the performance of medical products vigilance system. The WHO assessment team will then 2085

review the self-assessment and may amend the scoring, adding a justification under the respective 2086

column while keeping NRA self-assessment justification unchanged. The WHO team will also 2087

provide an overall conclusion and recommendation for WLA listing. Results from the amended 2088

scorecard will be combined with results from the VL field visit to inform the final decision on WLA 2089

listing. 2090



Page 133

Table PE.VL.01 VL PE indicators scorecard. 2091

2092 County: <please add input> Institution: <please add input> Dates: <please add input> Assessors: <please add input> 2093

2094

ID Indicator Scoring

(please tick one box) Justification by NRA (for self-

assessment) Justification by WHO team (for

formal assessment)

PE.VL.01 Dissemination of vigilance information through various channels such as newsletters, information bulletins or websites

☐ Not implemented

☐ Partially implemented

☐ Implemented

PE.VL.02 The VL centre contributes in learning activities for healthcare professionals as well as for the public

☐ Not implemented


☐ Implemented

PE.VL.03 A national ADR or vigilance advisory committee or AEFI committee or an expert committee in the setting is functional and capable of providing advice on medical product safety and produces quality outputs

☐ Not implemented


☐ Implemented

PE.VL.04 Total number of ADR reports received in the last three years (also expressed as number of ADRs per 100 000 persons in the population)

☐ Not implemented


☐ Implemented

PE.VL.05 Percentage of total number of ICSRs acknowledged and/or issued feedback in the last three years

☐ Not implemented


☐ Implemented

Page 134





formal assessment)

PE.VL.06 Percentage of total annual reports satisfactorily completed and submitted to the national vigilance centre in the last three years

Sub-indicator: Of the reports satisfactorily completed and submitted to the national vigilance centre, percentage of reports committed to VigiBase

☐ Not implemented


☐ Implemented

PE.VL.07 Number of active surveillance activities initiated, ongoing or completed during the past five calendar years

☐ Not implemented


☐ Implemented

PE.VL.08 Number of signals detected in the last 3 years by the VL centre

☐ Not implemented


☐ Implemented

PE.VL.09 Number of regulatory actions taken over the last 3 years as a consequence of national VL activities including:

➢ number of product label changes (variation); ➢ number of safety warnings on medical products to:

(i) health professionals, (ii) general public. ➢ number of withdrawals of medical products; ➢ number of other restrictions on use of medical

products

☐ Not implemented


☐ Implemented


Page 135




formal assessment)

PE.VL.10 The NRA performs or initiates studies on VL regulatory impact (e.g., medical product-related hospitalization or deaths)

☐ Not implemented


☐ Implemented

PE.VL.11 Percentage of registered medical products with a VL plan and/or a risk management strategy from the MAHs in the country

☐ Not implemented


☐ Implemented

PE.VL.12 Percentage of registered medical products for which periodic safety update reports were submitted and evaluated by the NRA or RRS as stipulated in the country over the last three years

☐ Not implemented


☐ Implemented

PE.VL.13 Percentage of healthcare institutions or Public Health Programmes in the country that have submitted ICSRs during the past three years

☐ Not implemented


☐ Implemented

PE.VL.14 Number of registered medical products for which post marketing safety or effectiveness studies were required and evaluated over the last three years

☐ Not implemented


☐ Implemented

PE.VL.15 Number of GVP regulatory inspections over the last three years

☐ Not implemented


☐ Implemented

Page 136





formal assessment)

PE.VL.16 All AEFI reports at the NIP are also represented in the national VL databases over the last three years

☐ Not implemented


☐ Implemented

Overall conclusion and recommendation by the WHO team:

<Please provide an overall conclusion of the PE indicators considering the guidance provided under each of the indicators (e.g., in terms of acceptance for the purpose of WLA designation) and any consequent recommendation>

2095

2096

2097

2098

2099

2100

2101

2102

2103

2104

2105

2106

2107

2108

2109

2110

2111


Page 137

3.4. WLA VL performance evaluation tools 2112

To be listed as WLA for VL, in addition to completing a self-assessment against the new VL 2113

performance indicators, an NRA or RRS must be subject to a vigilance field visit, as detailed 2114

below. 2115

2116

PE.VL.17. VL field visit 2117

2118


The VL field visit follows an established WHO practice used to help to document and evaluate 2120

the VL performance of a national medical products regulatory system. The activity consists of 2121

a field visit made by a WHO team of assessors to several layers of the vigilance system (for 2122

example, national, sub-national and health facility levels) to assess how VL operates and 2123

performs throughout the target country. 2124

2125

The field visit may also include onsite assessment of the VL PE indicators detailed above. 2126

2127


The VL field visit should be planned, prepared, conducted and reported according to the 2129

manual for vigilance field visits. In particular, assessors should use the “vigilance field visit 2130

assessment questionnaire”, which covers several areas of VL operations including VL systems, 2131

structures and stakeholder coordination; detection, reporting and data management; case 2132

investigation and analysis; risk assessment and management; information sharing, education 2133

and communication with concerned groups; and human and financial resources. 2134

2135

References 2136

1. Please refer to the Field Visit Manual for Assessing the Performance of Vigilance Function 2137

(Appendix 6.1) for guidance on planning, preparation, conduct and report of VL field visit, 2138

including the “vigilance field visit assessment questionnaire”. 2139

2140

Rating scale 2141

Results of the field visit should be written up according to the vigilance field visit manual. 2142

These will be combined with results from the amended scorecard to inform the final decision 2143

on WLA listing. 2144

2145


N/A 2147

Page 138


4. Market surveillance and control (MC) 2148

2149

4.1. MC PEP methodology 2150

The PEP for MC is designed to build on the GBT to further evaluate performance within 2151

specific areas of the MC function. 2152

2153

In addition to meeting the eligibility criteria, PEP for MC is considered fulfilled if the NRA or 2154


d. fully implement two mandatory ML4 GBT sub-indicators for MC (see Section 4.2); and then 2156

e. acceptably meet two newly developed MC performance evaluation indicators (see Section 2157

4.3). 2158

2159

The assessment against new MC indicators should be done as a self-assessment in the first 2160

instance, which is then submitted to WHO for review before a final recommendation on WLA 2161

listing is made. The full MC PEP is estimated to take between 3–6 months to complete. 2162

2163

Figure 4.1. Flowchart of the MC PEP. 2164

2165

2166

2167

4.2. Mandatory ML4 GBT sub-indicators for MC 2168

WLAs for market surveillance and control must fully implement the following ML4 indicators, 2169

as defined in the GBT: 2170

1. MC05.02: Database for product batches that have undergone surveillance along with their 2171

relevant testing results and regulatory actions is established and periodically reviewed. 2172



START

END

YES

NO


Assess against new MC indicators

MC PEP fulfilled

NO

Are all new MC indicators acceptably met?

NO

YES

YES

NO


Page 139

2. MC05.03: Performance indicators for market surveillance and control activities are 2173

established and implemented. 2174

2175

4.3. WLA MC performance evaluation indicators 2176

In addition to meeting the mandatory ML4 GBT sub-indicators above, to be listed as WLA for 2177

MC, an NRA or RRS must be assessed against two newly developed MA indicators and meet 2178

the acceptability criteria for each, as detailed below. 2179

2180

PE.MC.01 The NRA or RRS has developed and implemented a risked-based post-market surveillance (PMS) plan

Description: Having a risk-based approach to PMS is critical for assuring medical product quality, safety and effectiveness. The assessor should verify that a PMS plan exists and is being implemented, and that it clearly considers: • all the stakeholders involved in implementing the plan;

• each entity’s roles and responsibilities;

• how to communicate and coordinate internal and external stakeholders;

• the resources required to implement and maintain the plan;

• what data analysis and management system to use; and

• how to ensure evidence-based follow up, including regulatory actions, enforcement and communication.

The assessor should further evaluate the risk factors and rationale used to inform the plan’s approach to sampling and testing to ensure it uses a risk-based approach. The assessor should verify the quality of the PMS plan and consistency in implementation for the last 3–5 years. The assessor should also evaluate findings that require follow-up regulatory actions and assess whether these have been enforced by the NRA or RRS when warranted. Note that this indicator covers all products circulated in the market, including products distributed through online pharmacies (if applicable). This indicator should be evaluated alongside all GBT sub-indicators for substandard and falsified products, as well as all MC sub-indicators, especially MC02.02, MC04.04, MC05.02, MC06.02, and MC06.03. In reviewing the evidence on follow-up actions, the assessor should also look at GBT sub-indicators RS04.02–04 on rapid alert and recall.

Objective: This indicator aims to ensure that the NRA or RRS has established and implemented an efficient risked-based plan for PMS related activities at the national level.

Evidence to review:

The assessor should ask for and review the PMS plan and outputs from it, including documented evidence of: • roles and responsibilities for the plan for internal and external

stakeholders;

Page 140


• procedures and records showing efficient management and coordination of timely communication among the relevant stakeholders;

• clearly assigned resources to implement the PMS plan;

• the risk factors and rationale used to establish the sampling and testing plan for medical products (including selection of medical products for monitoring, the type and number of samples, the location and frequency of sampling, the type of controls, scope and number of laboratory tests performed etc);

• statistical analysis and data management of vigilance data generated through PMS activities; and

• appropriate follow-up regulatory actions (including rapid alerts and product recalls) and enforcement actions where relevant.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MC02.02, MC04.04, MC05.02, MC06.02, and MC06.03. (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

2. Guidelines on the conduct of surveys of the quality of medicines. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fiftieth report. Geneva: World Health Organization; 2016: Annex 7 (WHO Technical Report Series, No. 996, https://apps.who.int/iris/handle/10665/255338, accessed 1 July 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS has not developed and implemented a comprehensive risk-based PMS plan; or has established only some components of the PMS plan. IMPLEMENTED (I): The NRA or RRS has developed and fully implemented an effective national and risk-based PMS plan including all required components as described under description, objective and evidences. For an authority to be given WLA status for MC, this indicator must be scored as “implemented”.


This sub-indicator cannot be scored as "not applicable " (i.e.: this sub-indicator always applies when assessing MC WLAs).

PE.MC.02 The NRA or RRS has implemented measures to monitor, evaluate and sustain the performance of PMS-related activities.

Description: To sustain the expected performance of the national PMS plan, an NRA or RRS should be able to monitor and evaluate the performance of key operational aspects and areas of the plan. The assessor should verify the existence, implementation and effectiveness of an M&E system for the PMS plan and make sure that it tracks the plan’s implementation and enables appropriate action towards continual improvement. In particular, this system should enable the monitoring and evaluation of: a. Effectiveness of PMS activities, including:

• Implementation of planned PMS activities.




Page 141

• Timeliness of implementation (i.e.: timing between when a PMS activity is planned versus when it is implemented).

• Use of the rapid alert and recall system (including timelines and role in regulatory decisions).

• Periodic review of the capability and performance of the PMS system to detect and receive PMS events and reports.

b. Use of PMS feedback and findings and other related outcomes to adopt:

• Strategies, regulatory activities and regulatory decisions

• Timely decisions, evidence-based enforcement and resource allocations

• Actions for continual improvement c. Transparency and timeliness of reporting to:

• public and other stakeholders.

M&E activities should be carried out at least once a year. They should align with the organization’s regulatory performance evaluation framework, strategic plan and commitment to continual improvement. M&E outcomes should be used when considering: resource and training needs, output quality, the impact on regulatory decisions and plans for continual improvement. This indicator should be evaluated alongside performance evaluation indicator PE.MC.01, as well as GBT sub-indicators MC02.02, MC04.04, MC04.05, MC04.07, MC05.02, MC05.03, MC06.02, and MC06.03.

Objective: This indicator aims to ensure that the NRA or RRS has a robust and resourced system and procedures for measuring the effectiveness and impact of PMS activities in assuring medical products consumer safety. Its outcome is particularly useful in informing resource requirements for PMS and in ensuring that additional resources (staff with the required expertise, competencies, tools, finances etc.) are being appropriately used to maintain PMS performance and support continual improvement.

Evidence to review:

The assessor should review the system, processes and procedures implemented by the NRA to regularly monitor and evaluate the performance and impact of the PMS plan. This includes requesting and reviewing: • Documented and implemented procedures for measuring the

performance of PMS-related activities, including key aspects of regulation, infrastructure, NRA functions and structure, resources required (including human resources, materials, and financial resources), planned versus implemented activities and non-compliances.

• Documented evidence of performance indicators or outputs for the PMS-activities.

• Periodical (trend) reports or reviews of the performance of the PMS activities including identified areas for improvement.

• Follow-up actions taken to address areas for improvement.

• Risk and opportunity management controls on PMS activities and actions taken to manage them.

Page 142


• Evidence of appropriate communication and coordination activities and their respective timelines related to the regulatory actions and outcomes of the PMS related activities with all relevant internal and external stakeholders.

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicators MC02.02, MC04.04, MC04.05, MC04.07, MC05.02, MC05.03, MC06.02, and MC06.03. (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

2. Guidelines on the conduct of surveys of the quality of medicines. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: fiftieth report. Geneva: World Health Organization; 2016: Annex 7 (WHO Technical Report Series, No. 996, https://apps.who.int/iris/handle/10665/255338, accessed 1 July 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA has no mechanisms, system and/or procedures to measure the performance of PMS-related activities. PARTIALLY IMPLEMENTED (PI): The NRA has identified and implemented some aspects of a system or mechanism for measuring the performance of PMS-related activities in critical areas of its functions and operations (criticality to be based on the needs of the NRA with respect to the scope of WLA listing). IMPLEMENTED (I): The NRA has developed and implemented a robust mechanism or system with relevant procedures for measuring the performance of the MC activities and for implementing actions to sustain its performance. For an authority to be given WLA status for MC, this indicator should be scored as “partially implemented” or “implemented”.


This sub-indicator cannot be scored as "not applicable " (i.e.: this sub-indicator always applies when assessing MC WLAs).

2181

2182

2183

2184

2185




Page 143

5. Licensing establishments (LI) 2186

2187

5.1. LI PEP methodology 2188

The PEP for LI is designed to complement the GBT and ensure that the process of issuing 2189

licenses is based on, and complies with, quality standards for good practices (GXP). 2190

2191

Note that WLA status cannot be granted for LI function alone; it is only granted in 2192

conjunction with WLA status for regulatory inspection (RI). This means that in addition to 2193

meeting the eligibility criteria, any NRA applying for WLA status in LI must also meet all the 2194

PEP requirements for RI (see Section 6 below). In practice, the two functions will be assessed 2195

together. 2196

2197

In addition to meeting the eligibility criteria, PEP for LI is considered fulfilled if the NRA or RRS 2198

demonstrates to: 2199

a. fully implement two mandatory ML4 GBT sub-indicators for LI (see Section 5.2). 2200

2201

Figure 5.1. Flowchart of the LI PEP. 2202

2203

2204

The full PEP for LI and RI combined is estimated to take 3–6 months to complete. 2205

2206

5.2. Mandatory ML4 GBT sub-indicators for LI 2207

WLAs for licensing establishments must fully implement the following ML4 indicators, as 2208

defined in the GBT: 2209

1. LI05.01: A database is established and regularly updated that includes all licensing 2210

applications received, approved, refused, suspended or withdrawn, along with the essential 2211

documentation for each application. 2212

2. LI05.02: Performance indicators for licensing activities are established and implemented. 2213

2214

Are usual eligibility criteria met?

Are all PE requirements for RI acceptably met?

START

END

YES

NO


LI PEP fulfilled

Are all mandatory LI ML4 sub-indicators acceptably met?

NO

YES

YES

NO

Page 144


6. Regulatory inspection (RI) 2215

2216

6.1. RI PEP methodology 2217

The PEP for RI is designed to complement the GBT and ensure that the process of issuing 2218

licenses is based on, and complies with, quality standards for GXP, including good 2219

manufacturing practices (GMP), good clinical practices (GCP) and good distribution practices 2220

(GDP). 2221

2222

In addition to meeting the eligibility criteria, PEP for RI is considered fulfilled if the NRA or 2223


a. fully implement five mandatory ML4 GBT sub-indicators for RI (see Section 6.2); and then 2225

b. successfully undergo an observed audit for GXP, including GMP, GCP and GDP (see Section 2226

6.3). 2227

2228

Note that WLA status is granted for RI function alone only for NRA or RRS where the LI 2229

function is not applicable (see GBT LI fact sheets for additional information); where 2230

applicable, it will be granted in conjunction with WLA status for licensing establishments (LI). 2231

This means that in addition to meeting the eligibility criteria, any NRA applying for WLA status 2232

in RI must also meet all the PEP requirements for LI (see Section 5 above). In practice, the 2233

two functions will usually be assessed together. 2234

2235

The full PEP for LI and RI combined is estimated to take 3–6 months to complete. 2236

2237

Recognition mechanisms 2238

To make the best use of available resources and expertise and avoid duplication of activities 2239

already confirmed by relevant organizations or processes, the RI PEP recognizes other 2240

evaluation mechanisms, in particular: 2241

• Observed inspections by Pharmaceutical Inspection Co-operation Scheme (PIC/S). 2242

2243

This means that, once the NRA or RRS successfully demonstrates full implementation of all 2244

mandatory ML4 GBT sub-indicators, it will be checked for PIC/S membership and, if it is a 2245

PIC/S member and has been subject to a PIC/S observed inspection for GMP in the last five 2246

years, it will only need to complete an observed audit for GCP and GDP (see Figure 6.1 2247

below). 2248

2249

2250


Page 145

Figure 6.1. Flowchart of the RI PEP. 2251

2252

2253

6.2. Mandatory ML4 GBT sub-indicators for RI 2254

WLAs for regulatory inspection must fully implement the following ML4 indicators, as defined 2255

in the GBT: 2256

1. RI05.01: A database is established and regularly updated of all establishments which may be 2257

subject to inspection, along with their relevant regulatory decisions (certification and/or 2258

enforcement activities). 2259

2. RI05.03: Inspection reports are subjected to a regular and robust review by experts other 2260

than the designated inspection team. 2261

3. RI05.04: Inspection data and outcomes are systematically evaluated or interpreted. 2262

4. RI05.05: Performance indicators for regulatory inspection activities are established. 2263

5. RI06.02: The updated list or database of all inspected facilities along their regulatory 2264

decisions, actions and enforcement activities, is regularly published and publicly available. 2265

2266

2267


START

END

YES

YES

NO

Do the conclusions of observed audits support WLA listing for RI?

Is the NRA or RRS a PIC/S member?

RI PEP fulfilled

NO

Prepare and conduct observed audit for GMP

Prepare and conduct observed audit for GCP


YES

NO

Are PE requirements for LI also under assessment?

YES

NO

NO

YES

Get PIC/S assessment data for GMP

Has the NRA or RRS had a PIC/S inspection in the past five years?

YES

NO


Page 146


6.3. WLA RI performance evaluation tools 2268

As part of the RI PEP, an NRA or RRS must complete an observed audit for GXP. 2269

2270

2271

PE.RI.01. Observed audit for GxP 2272

2273


The observed audit follows a standard process in which a WHO team observes routine 2275

inspection at an authorized site. It is used by WHO to document and evaluate the 2276

performance of the national RI function for a range of GXP, including GMP, GCP and GDP. 2277

2278

The RI under observation should ideally be a routine inspection, carried out according to 2279

national references including regulations and guidelines. National references are expected to 2280

be at least equivalent to WHO GXP guidelines or other recognized guidelines. 2281

2282

As a general rule, the RI PEP should include three audits per GXP type (GMP, GCP and GDP). 2283

Fewer audits may be justified using a risk-based approach (for example, because a limited 2284

number of sites are subject to GXP regulatory inspection, the team of observers has former 2285

experience with the NRA or RRS, or the NRA or RRS is at an advanced stage of becoming a 2286

PIC/S member). 2287

2288


The observed audit for GXP should be planned, prepared, conducted and reported according 2290

to the GxP Observed Audit Manual for assessing the performance of Regulatory Inspection 2291

Function (Appendix 6.2). During the audit, the WHO observers should use the “Observed 2292

Audit Inspectors’ Evaluation Form” included in Appendix 6.2 to assess and evaluate five 2293

indicators through a set of sub-indicators, as summarized in Table RI.01. 2294

2295

Table RI.01. Main areas covered by the Observed Audit Inspectors’ Evaluation Form. 2296

Indicators of the Observed Audit Inspectors’ Evaluation Form

No. of sub-indicators

1. Inspection preparation process 12

2. Inspection conduct process 17

3. Inspection reporting process 9

4. Inspectors’ technical competency 7

5. Inspectors’ skills and attitude 11

2297

References 2298

1. PIC/S audit checklist: interpretation guide. Geneva: Pharmaceutical Inspection 2299

Convention/Cooperation Scheme; 2020 (PS/W 31/2019, 2300

https://picscheme.org/docview/3549, accessed 1 July 2021). 2301

2302

Rating scale 2303

The assessor uses the findings of all sub-indicators to conclude whether or not the NRA or 2304

RRS can be considered to acceptably meet the requirements of the observed audit. 2305

https://picscheme.org/docview/3549


Page 147

2306

In all cases, the outcomes and findings of the observed audit should be written up in a final 2307

report and handed to the WHO secretariat together with the other RI PEP results. 2308

2309


N/A 2311

2312

7. Laboratory testing (LT) 2313

2314

7.1. LT PEP methodology 2315

The PEP for LT is designed to assess laboratories that carry out medical product testing for 2316

the NRA or RRS, including chemical, biological and any other relevant laboratories. This 2317

includes both national control laboratories (NCLs) and external laboratories that perform 2318

tests on behalf of the NRA or RRS. The NRA or RRS can apply to be a WLA for LT for one or 2319

more product categories. 2320

2321

In addition to meeting the eligibility criteria, PEP for LT is considered fulfilled if the NCL or 2322

external laboratory applying for WLA status demonstrates to: 2323

a. fully implement five mandatory ML4 GBT sub-indicators for LT (see Section 7.2); and then 2324

b. successfully fulfils three newly developed LT performance evaluation tools (see Section 7.3). 2325

2326

The full PEP for LT is estimated to take no longer than six months to complete. 2327

2328

Recognition mechanisms 2329

To make the best use of available resources and expertise and avoid duplication of activities 2330

already confirmed by relevant organizations or processes, the LT PEP recognizes other 2331

evaluation mechanisms. In particular, the following laboratories are recognized by the LT 2332

PEP: 2333

• WHO Prequalified laboratories (for medicines) 2334

• WHO contracted laboratories (for vaccines) 2335

• Official Medicines Control Laboratories (OMCLs) attested by EDQM 2336

2337

These laboratories can be considered to acceptably meet the performance evaluation 2338

requirements for a WLA in LT, and are not subject to further assessment through the LT PEP 2339

(see Figure 7.1). 2340

2341

Note that accreditation against ISO/IEC 17025: 2017 General requirements for the 2342

competence of testing and calibration laboratories is acknowledged to be valuable; but it is 2343

not formally recognized under the LT PEP. This means that even if a laboratory is ISO-2344

accredited it will have to fully meet all eligibility criteria, as well as fully implementing all 2345

mandatory ML4 sub-indicators for LT and completing all newly developed LT performance 2346

evaluation tools. 2347

2348

2349

2350

Page 148


Figure 7.1. Flowchart of the LT PEP 2351

2352

2353

2354

7.2. Mandatory ML4 GBT sub-indicators for LT 2355

WLAs for laboratory testing must fully implement the following ML4 indicators, as defined in 2356

the GBT: 2357

1. LT08.01: Updated database of all medical products batches that have undergone quality 2358

testing. 2359

2. LT08.03: Regular participation in proficiency schemes, collaborative studies and inter-2360

laboratory comparisons (plus expansion, see 7.2.1 below). 2361

3. LT08.04: Performance indicators for laboratory testing activities are established (plus 2362

expansion, see 7.2.2 below). 2363

4. LT09.02: A laboratory safety programme exists and a designated person is responsible for its 2364

management. 2365

5. LT09.03: Staff immunization requirements are defined, implemented and monitored (plus 2366

expansion, see 7.2.3 below). 2367

2368

Is the laboratory: - a WHO PQ laboratory (for medicines, - a WHO-contracted laboratory (for vaccines), or - an OMCL attested by EDQM?

START

END

NO

NO

Are the requirements of all new LT performance evaluation tools acceptably met?

Are ML4 mandatory sub-indicators (plus expansions) met?

LT PEP fulfilled

NO

Complete QMS on-site evaluation


YES


YES

NO

NO

YES

Complete competencies on-site evaluation

Complete review of work samples


Page 149

2369

2370

7.2.1. Expansion for GBT ML4 indicator LT08.03 2371

The laboratory should define an appropriate mechanism of external control to provide 2372

objective evidence of overall laboratory performance and competence in an ongoing 2373

manner. There should be an analysis of the tests and analysts available in the laboratory with 2374

a scheme in place to monitor performance and competence at a reasonable frequency (for 2375

example once a year), with justification of the strategy and frequency implemented. 2376

2377

The assessor should evaluate the appropriateness and consistency of external control by 2378

considering: 2379

• PTS provider. Tools to monitor performance should include participation in proficiency 2380

testing schemes (PTS) organized by recognized providers (e.g. WHO, EDQM, National 2381

Institute for Biological Standards and Control, LGC, National Institutes for Food and Drug 2382

Control) where possible. 2383

2384

• Acceptable alternatives. It is not always possible for laboratories to find a suitable PTS 2385

scheme. In this case other approaches may be used, including external mechanisms such 2386

as: collaborative studies organized by reference laboratories; collaborative studies 2387

between the NCL and academia’s accredited laboratories or between the NCL and 2388

manufacturer; inter-laboratory comparisons according to pre-determined conditions with 2389

exchange of samples to be tested; or independent comparison with another NCL or 2390

manufacturer for the same batch of the product. Acceptable alternatives also include 2391

enhanced internal quality control procedures. For these, evaluation should use internally 2392

generated data. Note that for some non-complex testing methods individual PTSs are of 2393

no added value as the main operational steps are covered by other individual PTSs (e.g. 2394

weighing operations during high performance liquid chromatography) or are related to 2395

the qualification of equipment (e.g. friability). 2396

2397

• Animal use. No additional animals should be used in PTS to evaluate test performance. 2398

Where tests involve animals, means should be found to evaluate test performance that does 2399

not include use of animals only for the performance evaluation, for example trend analysis. 2400

The plan and its implementation should be reviewed at least yearly. 2401

2402

• Product stream. The laboratory should consider the availability of PTS based on product 2403

stream. If none are available, as a minimum the laboratory should implement blind testing 2404

and/or comparison of results with the manufacturer and must provide evidence that no PTS 2405

are available and justifies the alternative means used for assessing proficiency. 2406

2407

• Frequency. For specific tests, the frequency of external control should be determined 2408

according to: the number of tests performed each year, the number of analysts involved and 2409

their general level of experience, analyst turnover, the availability of reference standards, the 2410

criticality of the testing results, complexity of the technique, etc. External control should 2411

preferably be done at least once a year for all the laboratory’s relevant testing (“regular” 2412

participation). 2413

2414

• Follow up. After receiving the results of external control mechanisms, the laboratory should 2415

check the consistency of measurements’ accuracy, precision and uncertainty, and follow this 2416

Page 150


up according to the QMS. If the result obtained is unsatisfactory, or on the borderline of 2417

unsatisfactory, the laboratory should investigate root causes, identify possible consequences, 2418

and define appropriate corrective actions. This investigation should cover various factors that 2419

can influence results, including: 2420

o sample management (receiving, transport and storage conditions); 2421

o methods for testing performed (compendial or validated methods, confirmation of 2422

system suitability, validity and acceptance criteria); 2423

o testing conditions (calibration certificates validity, fulfilment of environmental 2424

conditions requirement, procedures followed in the preparatory activities and 2425

testing conduction, compliance of duration of the sub-processes with prescribed 2426

one, reference material and internal controls validity, etc.); 2427

o analyst competencies and specific knowledge needed for the testing performed; 2428

o equipment for testing performed (including intermediate equipment checks); and 2429

o documentation that supports decisions for planning participation in external control 2430

tests. 2431

2432

Proficiency indicators that the laboratory can use to evaluate its use of PTS include: 2433

• Continuity of participation (percentage or ratio) in PTS for specific type of testing in a 2434

defined period of time (e.g. past five years). 2435

• Percentage or ratio of satisfactory results from PTS (satisfactory against total number of 2436

the same type of studies performed). 2437

• Percentage or ratio of effective root-cause analysis performed (satisfactory against total 2438

number) following PTS. 2439

• Percentage of effectiveness of corrective measures implemented following PTS in defined 2440

period of time. 2441

• Number of external control mechanisms engaged in for specific type of testing per year. 2442

• Ratio of requalification/education of the analysts after deviations detected during PTS to 2443

total pool of analysts qualifications for laboratory testing. 2444

• Number of reviews of resources required to perform the PTS testing properly per 2445

resource type, or the number of initiatives taken (e.g.: method/technique compliance 2446

with up-to-date guidelines, norms and standards, new equipment/lab utensils 2447

introduction, software upgrades, environmental conditions interventions). 2448

• Number of (significant) improvements related to specific testing implemented during PTS 2449

in a defined period of time (e.g.: past three years). 2450

2451

References 2452

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical 2453

products. Sub-indicator LT08.03. Geneva: World Health Organization; 2018 2454

(https://apps.who.int/iris/handle/10665/341243, accessed 20 January 2021). 2455

2. Laboratory quality management system (LQMS). Chapter 10: Assessment—external quality 2456

assessment (EQA). Geneva: World Health Organization: 2011 2457


3. Alternatives to proficiency testing Schemes (PTS). PA/PH/OMCL (15) 50 R8. Strasbourg: 2459

European Directorate for the Quality of Medicines; 2020 (internal GEON document). 2460

4. Proficiency testing: guidelines on the level of participation and evaluation of performances in 2461

proficiency testing activities in the context of accreditation assessments. Brussels: Belgian 2462

Organization For Accreditation; 2011: BELAC 2-106 Rev 1 2463




Page 151

(https://economie.fgov.be/sites/default/files/Files/Publications/files/Belac-EN/2-106-EN.pdf, 2464

accessed 12 January, 2021). 2465

2466


The assessor should consider whether the following areas are covered by performance 2468

indicators: 2469

• Comparison of internal/standard methods vs other compendial methods or validated 2470

methods (use of z-scores). 2471

• Comparison of subject equipment results vs qualified equipment that has been calibrated to 2472

provide traceable results. 2473

• Functional check(s) of measuring and testing equipment. 2474

• Intermediate checks on measuring equipment. 2475

• Assessment of uncertainty of measurement. 2476

• Review of documented analyst competency through testing against in place e.g.: retesting of 2477

retained items, or testing “blind” samples (prepared for this purpose by the Head of the 2478

Laboratory or Quality Manager etc.) 2479

• Comparison of replicate test using the same or different methods. 2480

• Compliance with pre-defined acceptance criteria of the method (e.g.: deviation of replicates, 2481

system suitability criteria). 2482

• Data monitoring, trending and assessment of assay controls and test samples. 2483

2484

Quality control procedures should be embedded within the routine working of the 2485

laboratory. Data should be collected on an ongoing basis and reviewed as part of the formal 2486

documented plan. 2487

2488

References 2489




2. ISO 13528:2015. Statistical methods for use in proficiency testing by interlaboratory 2493

comparisons. Geneva: International organization for standardization; 2015 2494

(https://www.iso.org/standard/56125.html, accessed 12 January 2021). 2495

3. Alternatives to proficiency testing Schemes (PTS). PA/PH/OMCL (15) 50 R8. Strasbourg: 2496

European Directorate for the Quality of Medicines; 2020 (internal GEON document). 2497

4. Selection, use and interpretation of proficiency testing (PT) schemes by laboratories. 2498

Teddington: Eurachem; 2011 2499

(https://www.eurachem.org/index.php/publications/guides/usingpt, accessed 12 January 2500

2021). 2501

2502


This indicator only applies to laboratories that are involved in biological activities, use live 2504

animals or engage in any other relevant activities that justify the need for immunization. The 2505

NRA should provide clear guidance with adequate justification in this regard. 2506

2507

In all cases, data protection and privacy legislation should be considered when requesting 2508

access to health records of laboratories staff. 2509

2510

https://economie.fgov.be/sites/default/files/Files/Publications/files/Belac-EN/2-106-EN.pdf


https://www.iso.org/standard/56125.html

https://www.eurachem.org/index.php/publications/guides/usingpt

Page 152


References 2511




2515

7.3. WLA LT performance evaluation tools 2516

As part of the LT PEP, the NCL or external laboratory must complete: 2517

• An expanded on-site QMS evaluation 2518

• An extended on-site competencies framework evaluation 2519

• Assessment of analytical reports 2520

2521

PE.LT.01 Expanded on-site QMS evaluation 2522

2523


This tool is a checklist designed to build on the GBT to ensure the NCL or external laboratory 2525

has a robust QMS system. It is adapted from the WHO Good practices for pharmaceutical 2526

quality control laboratories (GPPQCL), which is used to support self-inspections and peer 2527

audits carried out as part of the WHO Prequalification programme. 2528



Page 153


The assessor should evaluate and rate each of the requirements set out in the checklist below (Table PE.LT.01). 2530

2531

The assessor should put the checklist to the laboratory first for self-assessment; but it is essential that both the assessors and the laboratory 2532

understand that all answers to the checklist must be confirmed through the provision, review and assessment of appropriate quality documents. 2533

These documents will include the quality manual, if it exists, SOPs, test records, monitoring sheets and records of instrument qualification etc. 2534

Any non-applicable requirements should be clearly indicated with the term "N/A" and must be fully justified. 2535

2536

Table PE.LT.01. Expanded on-site QMS evaluation. 2537

Requirement description Rating scale Passing score

The scope of the laboratory’s activities is well described and established in a QMS

Scale 0–3 0 = no scope in QMS; 1 = scope unclearly defined in QMS; 2 = scope clearly defined but not established in accordance with international standards; 3 = scope clearly defined, written and established in accordance with international standards.

3

A written and clear statement of the laboratory management’s intentions with respect to the standard of customer service it will provide

Scale 0–2 0 = no statement; 1 = written statement but not clear; 2 = written and clear statement.

2

A written and clear statement on the laboratory management’s commitment to comply with specific technical guidance e.g. ISO, WHO, OMCL etc.

Scale 0–2 0 = no statement; 1 = written statement but not clear; 2 = written and clear statement.

2

An organizational structure that clearly defines the extent and limits of responsibilities for operational and functional activities pertaining to quality.

Scale 0–3 0 = a structure available but does not describe operations; 1 = reporting structure defined but does not relate to quality and responsibilities are not well defined; 2 = structure for quality defined but responsibilities are not clear; 3 = structure for quality is clear and responsibilities are well defined.

3

Page 154



A clear structural outline for documents used in the laboratory QMS

Scale 0–3 0 = no outline; 1 = outline available but does not follow reference guidance or requirement; 2 = outline available, in line with reference guidance but not fully complied with and no other templates described; 3 = outline available, in line with reference guidance, and no evidence of other templates found.

2

A written policy for internal and external audits Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with; 3 = written policy available, in line with reference guidance or standard, and evidence of compliance.

2

A written policy for implementing and verifying corrective actions and risk management system

Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with; 3 = written policy available, in line with reference guidance or standard, and evidence of compliance.

2

A written policy for dealing with complaints Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with; 3 = written policy available, in line with reference guidance or standard, and evidence of compliance.

2

A written policy for performing management reviews of the QMS

Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with; 3 = written policy available, in line with reference guidance or standard, and evidence of compliance.

2


Page 155


A written policy for participating in appropriate proficiency testing schemes and evaluation of the results

Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with(; 3 = written policy available, in line with reference guidance or standard, and evidence of evaluation of compliance.

3

A written policy to select service providers and suppliers

Scale 0–3 0 = no policy available; 1 = practiced but no written policy available; 2 = written policy available, but not fully complied with; 3 = written policy available, in line with reference guidance or standard and legislation, and evidence of compliance.

2

2538

References 2539

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Sub-indicators LT03.01 and LT03.04. 2540

Geneva: World Health Organization; 2018 (https://apps.who.int/iris/handle/10665/341243, accessed 20 January 2021). 2541

2. WHO good practices for pharmaceutical quality control laboratories. Geneva: World Health Organization; 2010: Annex 1 (WHO Technical Report 2542

Series, No. 957; https://apps.who.int/iris/handle/10665/44291, accessed 1 July 2021). 2543

Rating scale 2544

For the NCL or external laboratory to gain WLA status in LT, each component of the checklist above must achieve a passing score. 2545

2546


N/A 2548



Page 156


PE.LT.02. Extended on-site competencies framework evaluation 2549

2550


The laboratory should have well defined competencies for each laboratory role. It should use 2552

these to inform employment, career progression and professional development; and to 2553

assign responsibilities for each position. Defined competencies should also be used as a 2554

framework for assessing training needs and curricula to support the competencies required 2555

for performing each role-specific function. 2556

2557

The assessor should review records and reports, or observe human resources performing 2558

analyses, evaluating results or doing other technical laboratory tasks. Alternatively, the 2559

assessor can review internal documents of past internal competency assessments. 2560

2561


The assessor should request and review: 2563

• Records of written or oral tests, including following training. 2564

• Work samples using established rubric. 2565

• Results of internal competency assessments, where possible (done through intra-laboratory 2566

proficiency testing or comparisons using internally generated data). 2567

• Trainers’ training records. 2568

• Internal human resources qualification records. 2569

• Problem logs and incident investigations. 2570

2571

In addition, the assessor should complete directly observed competency assessments of 2572

analysts, using the matrix below (Table PE.LT.02). For each analyst under observation, the 2573

assessor should first define responsibility and job function to identify non-applicable 2574

components. Analysts should then be observed as they perform routine work processes and 2575

procedures, and the matrix should be used to determine if all steps were properly 2576

completed. 2577

2578

Before performing a test, analysts are expected to have read all SOPs, manuals, logs, work 2579

instructions and workstation tasks, as well as any other procedure-relevant documents. All 2580

procedures should be performed according to SOPs; and competency should be based on 2581

how well the analyst under observation adheres to these. 2582

2583


Page 157

Table PE.LT.02. Matrix for directly observed competency assessments. 2584

2585 1 Sample receipt and record initiation YES NO N/A/remarks

1.1 Did the responsible staff member complete the sample chain of custody control form

1.2 Did the responsible staff member check if sample information matches the analysis request form and check

the general applicability of the test requested?

1.3 If sample and analyses request form details do not match, was the responsible staff member able to apply

sample acceptance and rejection criteria?

1.4 If the samples do not match the general applicability of the request, did the responsible staff member note

this under the remarks section?

1.5 Did the responsible staff member assign laboratory reference numbers to the forms and samples correctly?

1.6 Did the responsible staff member log in the samples, forms and documents in the appropriate database or

laboratory information system?

1.7 Did the responsible staff member capture and save records correctly?

OVERALL SCORE FOR SECTION 1 (select one, as appropriate): Satisfactory / Not satisfactory / Not applicable

2. Reagents/sample and standard preparation YES NO N/A/remarks

2.1. Did the responsible staff member select appropriate test method (pharmacopeial/inhouse

validate/manufacturer)?

2.2. Did the responsible staff member check availability of necessary equipment to do the test?

2.3. Did the responsible staff member perform the appropriate daily or pre-use calibration or checks on

equipment (e.g.: pH meter) and document it?

2.4. Did the responsible staff member select appropriate reference material?

2.5. Did the responsible staff member select appropriate volumetric flasks?

Page 158


2.6. Did the responsible staff member carry out appropriate sample preparations and dilutions?

2.7. Did the responsible staff member prepare media, mobile phases, reagents correctly?

2.8. Did the responsible staff member label samples as per procedure?


3. Use of equipment YES NO N/A/remarks

3.1. Did the responsible staff member perform bench clearing, cleaning and environmental monitoring (including

safety precautions) and complete relevant logs?

3.2. If equipment initializing failed, did the responsible staff member inform the supervisor?

3.3. Did the responsible staff member arrange samples according to the worksheet or work instruction?

3.4. Did the responsible staff member check label samples as per procedure and equipment?

3.5. Did the responsible staff member apply the correct sample on the correct slot of the worksheet/sample

tray/map?

3.6. Did the responsible staff member decontaminate devices used in sample transfer or dilution etc.?

3.7. Did the responsible staff member use the correct pipetting technique?

3.8. Did the responsible staff member measure the right quantities of sample, reference standards, controls and

reagents?

3.9. Did the responsible staff member use the correct method of sample extraction?


4. Running of sample analysis YES NO N/A/remarks

4.1. Did the responsible staff member create a new method / select an appropriate existing method?

4.2. Did the responsible staff member place control/blank samples on the correct slot of the worksheet or sample

tray?


Page 159

4.3. Did the responsible staff member select the correct protocol for running the test?

4.4. Did the responsible staff member follow the correct sequence testing steps during processing?

4.5. Did the responsible staff member address any “error” notification on the equipment?

4.6. Did the responsible staff member record error logs on equipment use log and appropriate action taken?

4.7. Did the responsible staff member follow correct gowning procedure before running the sample analysis?

4.8. Did the responsible staff member respect aseptic technique and cross contamination risk minimization

measures, where applicable?

4.9. Did the responsible staff member apply safe procedure for disposal of contaminated waste?


5. Reporting of results YES NO N/A/remarks

5.1. Did the responsible staff member follow quality control rules to review the test data and records?

5.2. Did the responsible staff member review results of sample against the controls/standards?

5.3. Did the responsible staff member interpret results according to the SOP?

5.4. If correction factors were included to interpret results, did the responsible staff member use the correction

factor correctly?

5.5. Did the responsible staff member review results against acceptance criteria or specifications on the request

form?

5.6. In case of OOS results was the procedure for OOS investigations followed?

5.7. Did the responsible staff member submit results to the supervisor for review and approval?

5.8. Did the responsible staff member enter results in the laboratory information system/ data base correctly?

5.9. Did the responsible staff member dispatch results correctly in their respective boxes/envelopes?

Page 160


5.10. Did the responsible staff member formally note or advise on the significance or give judgment with reference

to the results including recommending corrective action when controls are unacceptable?

5.11. Did the responsible staff member follow the correct sequence in filing the records?

5.12. Did the responsible staff member follow the correct sequence in compiling reports?


6. Storage of records YES NO N/A/remarks

6.1. Did the responsible staff member store records in the appropriate area?


7. Stock management YES NO N/A/remarks

7.1. If there were any reagent or consumables replaced, did the responsible staff member verify these and

complete the appropriate records?


8. Reviewing quality and technical records (observe the analyst reviewing quality & technical records) YES NO N/A/remarks

8.1. Did the responsible staff member record or demonstrate knowledge of the following records review correctly? a. Correct formulae and spreadsheets (where applicable used) – calculations and results derivation b. Review of Internal Quality Control data c. Detecting data trends d. Recognition and interpretation of inconsistent results and test system problems and troubleshooting e. Corrected reports f. Equipment error logs g. Acceptance criteria/Specifications and critical values


2586

2587


Page 161

References 2588


products. Sub-indicators LT04.01 and LT04.04. Geneva: World Health Organization; 2018 2590


2. Pilot implementation of the global competency framework as part of institutional 2592

development of National Regulatory Authorities (NRAs): V30. Geneva: World Health 2593

Organization; 2020 (draft working document). 2594

Rating scale 2595

For the NCL or external laboratory to gain WLA status in LT, each of the eight areas of the 2596

matrix must achieve a satisfactory score; and overall, not less than 85% (e.g. excluding non-2597

applicable components) should be scored “YES”. 2598

2599


N/A 2601

2602

2603

PE.LT.03. Assessment of analytical reports 2604

2605


This tool aims to complement the GBT in assessing the performance of the NCL or external 2607

laboratory when it comes to documenting its activities’ outcomes. The tool uses a rubric that 2608

is adapted from the WHO Global Framework for Competency of Regulators, and is designed 2609

to ensure that the outcomes of a laboratory’s activities are documented according to a QMS, 2610

including quality and completeness of reports, scientific rigour of approaches, compliance 2611

with regulatory requirements and data integrity. 2612

2613

The assessor should randomly select a representative quantity (minimum 3–5) of work 2614

samples (e.g. analytical reports) for assessment; each one must have already undergone an 2615

internal quality assessment review and been filed as closed. 2616

2617

In preparation for their assessment, laboratories should: 2618

• Make available the full reports of any recognized international standard (including WHO 2619

prequalification, ISO 17025 or similar) inspections/audits (including corrective action plan). 2620

• Conduct internal audits, review non-compliances including corrective actions and risk 2621

management and monitor their implementation. 2622

2623


For each randomly selected work sample, the assessor should evaluate and rate each of the 2625

items set out in the table below. 2626

2627


Page 162


Topics covered by the rubric for assessing analytical reports. 2628 Quality of the report (QR) to communicate and facilitate comprehension by the reader.

Completeness of the report (CR) to provide a comprehensive and complete picture of the situation or sample under consideration.

Scientific rigour (SR) to ensure a scientific approach is applied for unbiased analysis and interpretation of the evidence or data.

Compliance (C) with regulatory requirements and policies.

Data integrity (DI) to ensure data are attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring and available (ALCOA Plus)

2629

2630

Table. PE.LT.03. Rubric for assessing analytical reports. 2631 RATING

Item Not acceptable Basic Intermediate Advanced

1 Critical test equipment

(DI)

None of the critical test equipment (including balances, pipettes, glassware and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified.

Some of the critical test equipment (including balances, pipettes, glassware and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. There was some cross-reference to instrument ID, last calibration/ qualification date, and usage record (instrument record).

Most of the critical test equipment (including balances, pipettes, glassware and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. Some critical equipment was missed. But there was cross-reference to instrument ID, last calibration/ qualification date, and usage record (instrument record).

All of the critical test equipment (including balances, pipettes, glassware and sample preparation devices) were appropriately confirmed as functioning correctly and appropriately calibrated/ qualified. This includes cross-reference to instrument ID, last calibration/ qualification date, and usage record (instrument record).

2 Test methods

(DI)

Some test methods and calculations followed were ambiguous with an unclear source. There was use of unapproved spreadsheets and formulae. Inappropriate rounding-off was used.

Some ambiguous test methods and calculations were followed. It was not clear if the analyst used approved/ validated spreadsheets or approved formulae. There was inconsistent application of policies such as rounding off.

All test methods and calculations were unambiguous and followed. Approved spreadsheets, formulae and policies were not always applied with no valid reason.

All test methods and calculations were unambiguous and followed. Approved spreadsheets, formulae and policies were consistently applied. All calculations were reviewed and approved.

3 Reagents & reference materials

(DI)

There was no evidence or records showing that reagents and reference materials used in the test were appropriate for use (i.e. prepared according to procedure and within expiry date). There was

There was some evidence or records showing that reagents and reference materials used in the test were appropriate for use (i.e. prepared according to procedure and within expiry date). There was no reference

There was evidence or records showing that reagents and reference materials used in the test were appropriate for use (i.e. prepared according to procedure and within expiry date). There was some

All reagents and reference materials used in the test were appropriate for use (i.e. prepared according to procedure and within expiry date). There was always reference to standards (e.g. solution preparations working standard, primary


Page 163

RATING Item Not acceptable Basic Intermediate Advanced

no reference to standards (e.g. solution preparations working standard, primary standard, chemical reference standard, SST solution preparation and usage details).

to standards (e.g. solution preparations working standard, primary standard, chemical reference standard, SST solution preparation and usage details).

reference to standards (e.g. solution preparations working standard, primary standard, chemical reference standard, SST solution preparation and usage details).

standard, chemical reference standard, SST solution preparation and usage details).

4 Validation

(DI)

There was no information on whether the test methods used were validated/ verified and authorized for use within the laboratory.

There was some information on whether the test methods were authorized for use within the laboratory in the test. There was no information on whether the test methods were validated/ verified.

There was adequate information that all test methods used were authorized for use within the laboratory. Although method validation/ verification was claimed, no reference or evidence was included.

There was adequate information that all test methods used were validated/ verified and authorized for use within the laboratory. There was reference to a validation report as evidence.

5 Sample &

materials use (DI)

There was no information recorded to confirm: - Use of correct samples/

reagents/reference materials (including microbiological media).

- Appropriate storage conditions for samples.

- Storage of samples or media/ reagents for the correct time before being used.

There was information recorded to confirm: - Use of correct samples/

reagents/reference materials (including microbiological media).

- Appropriate storage conditions for samples.

This information was not always found in the appropriate sections of records.

Information was clearly recorded to confirm: - Use of correct samples/

reagents/reference materials (including microbiological media and written details).

- Appropriate storage conditions for samples and media/ reagents.

- Storage of samples for the correct time before being tested.

This information was found in the appropriate sections of records for analytical preparation at various stages; although some relevant documents/ reports were not easily recognizable and may affect traceability.

Information was clearly recorded using a continuous monitoring device to confirm: - Use of correct samples/

reagents/reference materials (including microbiological media and written details).

- Appropriate storage conditions for samples and media/ reagents.

- Storage of samples for the correct time before being tested.

This information was found in the appropriate sections of records for analytical preparation at various stages, with clear written evidence with document and record IDs to facilitate traceability and complete reconstruction of all tests.

6 Conditions for

testing (DI)

There was no record of whether the conditions for testing (e.g. system suitability and assay validity criteria)

There were some records of statements that conditions for testing (e.g. system suitability and assay validity criteria) were met before

There were records and confirmation that conditions for testing (e.g. system suitability and assay validity criteria) were met before and during

There were records and confirmation that conditions for testing (e.g. system suitability and assay validity criteria) were met before and during analyses. Missing or

Page 164



were met before and during analyses.

analyses. There was no record of the actual conditions.

analyses. Missing or failed conditions were noted.

failed conditions were noted and assessed through risk analysis.

7 Sample &

media integrity (SR or DI)

For microbiological analysis. Some of the following was observed during analyses: - No evidence of use of negative

and positive controls. - Records of satisfactory incubation

conditions were not maintained.

For microbiological analysis. All of the following was observed during analyses: - Inadequate evidence of use of

negative and positive controls. - Some records of satisfactory

incubation conditions were maintained.

For microbiological analysis. Some of the following was observed during analyses - Some adequate evidence of use of

negative and positive controls. - Full records of satisfactory

incubation conditions were maintained.

For microbiological analysis. All of the following was observed during analyses: - Adequate evidence of use of negative

and positive controls. - Full records of satisfactory incubation

conditions were maintained.

8 Templates

(CR)

The correct and approved templates and forms were not used or available (including OOT, OOS, deviation as applicable).

The correct and approved templates and forms were almost never used, but were used sometimes (including OOT, OOS, deviation as applicable), with no justification when not used.

The correct and approved templates and forms were almost always used, but not always. (including OOT, OOS, deviation as applicable), with justification when not used.

The correct and approved templates and forms were always used (including OOT, OOS, deviation as applicable).

9 Inclusion of invalid data

(CR)

Only compliant (valid) data included in the report. No inclusion of “invalid” data or data from “rejection report” in case of re-test and re-sampling in the report.

Compliant (valid) data included in the report. Inclusion of some, but not all, “invalid” data from re-sampling and re-testing in the report.

Inclusion of all invalid data in the report, including from “rejection report” in case of re-test and re-sampling. But these data were not appropriately labelled as “invalid”.

Inclusion and appropriate labelling of all invalid data, including from “rejection report” in case of re-test and re-sampling.

10 Blank spaces

(DI)

No blank pages/spaces in worksheets were crossed out.

Blank pages/spaces in worksheets were almost never crossed out, but sometimes crossed out.

Blank pages/spaces in worksheets were almost always crossed out, but not always crossed out; and details such as signature, date and “n/a” were included.

Blank pages/spaces in worksheets were always crossed out; and details such as signature, date and “n/a” were included.

11 Links to raw

data (DI)

There was no reference or link to raw data as evidence of contemporaneous recording.

There was an occasional reference or link to raw data as evidence of contemporaneous recording. There were no printed and signed outputs from electronic equipment that does not store data (e.g. some balances, pH meter etc).

There was mostly adequate reference or link to raw data as evidence of contemporaneous recording, including printed and signed outputs from electronic equipment that does not store data (e.g. some balances,

There was always adequate reference or link to raw data as evidence of contemporaneous recording, including printed and signed outputs from electronic equipment that does not store data (e.g. some balances, pH meter etc) and adequate reference to audit trails.


Page 165


pH meter etc) and a link to audit trails.

12 Presentation

(QR)

The flow of test data or discourse is not organized and difficult to follow.

The test data or discourse are organized and easy to follow in some sections of the report.

The test data or discourse are organized and easy to follow in most sections of the report.

The test data or discourse are organized and easy to follow, enhancing readability and comprehension in all sections the report.

13 Re-testing & re-sampling

(DI)

No justification given for a re-test or re-sampling.

Justification given for re-testing and re-sampling (e.g. calculation error, power outage, equipment failure; testing errors). But relevant SOP was not used appropriately and there was no cross reference or link to other records as evidence.

Justification given for re-testing and re-sampling (e.g. calculation error, power outage, equipment failure; and testing errors). Appropriate SOPs used and some cross reference or link to other records as evidence (e.g. maintenance record not always provided).

Justification given for re-testing and re-sampling (e.g. calculation error, power outage, equipment failure; and testing errors). Appropriate SOPs used and cross reference or link to other records as evidence, including maintenance records for power outages and equipment failure.

14 Environment

(CR)

No records to show that environmental conditions such as temperature and humidity were checked or monitored while sample was handled and testing conducted. For microbiological analysis: tests are not performed under pharmacopeial conditions (LAF, BSC, isolator, clean room).

Environmental conditions such as temperature and humidity were checked at some unspecified points, but there was no demonstration of monitoring at critical times while sample was handled and testing conducted. The relevant SOP was not used. For microbiological analysis: tests are partially performed under pharmacopeial conditions (LAF, BSC, but no clean room when applicable).

Environmental conditions such as temperature and humidity were checked and monitored at all critical points while sample was handled and testing conducted. But records were not complete for some critical conditions. Inadequate records of any other potentially interfering/testing activities at the time of the testing, if applicable. The relevant SOP was only partly followed. For microbiological analysis: tests are performed under pharmacopeial conditions (LAF, BSC, isolator, in a clean room when applicable) but

Complete records to show that critical environmental conditions such as temperature and humidity were checked and monitored while sample was handled and testing conducted. Records were made of any other potentially interfering/ testing activities at the time of the testing, if applicable. The relevant SOP was appropriately followed. For microbiological analysis: tests are performed under pharmacopeial conditions (LAF, BSC, isolator, and clean room when applicable) and there is regular monitoring and annual requalification.

Page 166



there is no regular monitoring or requalification.

15 Facilitating information

(CR)

Information needed to facilitate testing and interpretation of results is incorrect or rarely included in the report.

Some of the information needed to facilitate testing and interpretation of results is included in the report for simple issues.

Most of the information needed to facilitate testing and interpretation of results where few factors or non-complex issues are involved is included in the report.

All of the correct information needed to facilitate testing and interpretation of results, including all limitations, is included in the report.

16 Report

summary (CR)

The report summary is incomplete and lacks adequate detail. It does not outline essential test data, requiring the reader to make substantial reference to the source data and other references.

The report is largely incomplete and lacks detail. It does not include significant or essential information, requiring the reader to corroborate information from source or other reference sources.

Most of the essential information is provided in the report, requiring the reader to refer to the source test data only occasionally. There were some omissions of essential information, which could be referred from easily accessible records. - Special techniques (e.g. dilution)

were well reported. - Deviations from specified

(authorized) methods were recorded but not justified.

The report is complete and detailed, and consistently includes essential information necessary to understand the analyst’s conclusions. - Special techniques (e.g. dilution) were

well reported. - Deviations from specified (authorized)

methods were recorded, justified and approved.

17 Critical

features (QR)

The report does not include any analyst comments/ remarks on the critical features of the sample that may impact test results.

The report includes the analyst’s comments/remarks on some of the critical features of the sample that may impact test results.

The report includes the analyst’s comments/remarks on most of the critical features of the sample that may impact test results.

The report includes the analyst’s comments/remarks on all of the critical features of the sample that may impact test results.

18 Presence of conclusions

(SR)

The analyst’s opinions and conclusions are largely absent from the report, if relevant (or placed in incorrect secions). - It is difficult to understand the

significance/relevance of the

The analyst’s opinions and conclusions are present and accurately placed in some sections of the report, if relevant. - It is often not clear what the basis

for the analyst’s discussion and conclusions are.

The analyst’s opinions and conclusions are present and accurately placed in most sections of the report, if relevant. - The basis for the analyst’s

discussion and conclusions are clear in most cases.

The analyst’s opinions and conclusions are present and accurately placed throughout the report, if relevant. - The analyst noted anomalous or suspect

data (e.g. chromatographic peaks )or observations (e.g. abnormal appearances of samples, reagents,


Page 167


analyst’s discussion and conclusions.

solvents or solutions; reagents not dissolved completely; or out of trend results).

- Throughout the report it is clear to the reader what the analyst thinks of the information, approach and conclusions.

19 Validity of

conclusions (SR)

The analyst’s conclusions are inconsistent with the test data generated; or are scientifically invalid.

The analyst’s conclusions are sometimes inconsistent with the test data generated; or are sometimes scientifically invalid.

The analyst’s conclusions are almost always consistent with the test data; and are scientifically valid.

The analyst’s conclusions are always consistent with the test data; and are scientifically valid.

20 Alignment of conclusions

(SR)

Recommendations and conclusions were not in line with applicable regulatory requirements and internal policies.

Some recommendations and conclusions were not in line with applicable regulatory requirements and internal policies

Most recommendations and conclusions were in line with applicable regulatory requirements and internal policies

All recommendations and conclusions were in line with applicable regulatory requirements and internal policies

21 ALCOA

principles (DI)

The following was noted: - Evidence of unintentional or

unauthorized data changes. - Records were not attributable,

legible, original and accurate; or copies were not verifiable as true.

- Evidence of selective reporting. - Overwriting or deletion of

original data.

The following was noted: - No evidence of unintentional or

unauthorized data changes. - Records were sometimes not

attributable, legible, original and accurate; or, where applicable, they were evidently true copies.

- No evidence of selective reporting. - Clear evidence of documentation

of sequence of processes (e.g. sequence of injections).

- Some evidence of overwriting or deletion of original data.


unauthorized data changes. - Records were attributable, legible,

original and accurate; or, where applicable, they were evidently true copies.

- No evidence of selective reporting. - Modified or adjusted parameters

were traceable or retrievable with some difficulty (e.g. use of manual integration).

- Clear evidence of documentation of sequence of processes (e.g. sequence of injections).


unauthorized data changes. - Records were attributable, legible,

original and accurate; or, where applicable, they were evidently true copies.

- No evidence of selective reporting. - Modified or adjusted parameters were

easily traceable or retrievable (e.g. use of manual integration).

- Clear evidence of documentation of sequence of processes (e.g. sequence of injections).

- All printed records were supported with unique identifiers.

Page 168



- Not all printed records supported with unique identifiers.

- No overwriting or deletion of original data. Any cancellations were signed, dated and justified.

- No overwriting or deletion of original data. Any cancellations were signed, dated and justified.

22 Requirement interpretation

(SR)

The analyst made significant misinterpretations of applicable regulatory requirements.

The analyst made some misinterpretation of applicable regulatory requirements.

The analyst shows an adequate level of interpretation of requirements for non-complex issues.

The analyst shows an expert-level interpretation of regulatory requirements, including complex issues.

23 Context

(C)

There is no evidence of consideration of the context-associated risk of the issue in any cases when applying policy, guidance and procedures.

There is some evidence of consideration of the context-associated risk of the issue in some cases when applying policy, guidance and procedures.

There is evidence of consideration of the context-associated risk of the issue in most cases when applying policy, guidance and procedures.

There is sufficient evidence of consideration of the context-associated risk of the issues in all cases when applying policy, guidance and procedures.

2632

References 2633

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Indicator LT06. Geneva: World Health 2634

Organization; 2018 (https://apps.who.int/iris/handle/10665/341243, accessed 20 January 2021). 2635

2. Pilot implementation of the global competency framework as part of institutional development of National Regulatory Authorities (NRAs): V30. 2636

Geneva: World Health Organization; 2020 (draft working document). 2637

Rating scale 2638

For the NCL or external laboratory to gain WLA status in LT: 2639

- At least 88% of applicable items (e.g. 20 out of 23) in each randomly selected report should be scored as advanced level; 2640

- No item can score below intermediate level. 2641

2642


N/A 2644



Page 169

8. Clinical trials oversight (CT) 2645

2646

8.1. CT PEP methodology 2647

The PEP for CT is designed to assess CT for specific products and activities only, including: 2648

• Products: new chemicals entities (new medicines), multisource/generic medicines (BE 2649

studies), vaccines and SBPs. 2650

• Processes: CT listing in a registry; submission and assessment of CT application and CT 2651

protocol approval, registration or licensing; transparency and structure; post-approval safety 2652

and compliance and reporting (pharmacovigilance, i.e. safety monitoring system, including AE 2653

assessment and safety reporting during trials); and effectiveness of the CT process. 2654

2655

In addition to meeting the eligibility criteria, PEP for CT is considered fulfilled if the NRA or 2656


c. fully implement two mandatory ML4 GBT sub-indicators for CT (see Section 8.2); and then 2658

d. acceptably meet 10 newly developed CT performance evaluation indicators (see Section 8.3); 2659

and then 2660

e. successfully undergo an Expert Review of CT Application Assessments (see Section 8.4). 2661

2662

Figure 2.1. Flowchart of the CT PEP. 2663

2664



START

END

YES

YES

NO

Are all sections of the expert review acceptably met?

Assess NRA against new CT indicators

CT PEP fulfilled

NO

Are all new CT indicators acceptably met?

Conduct Expert Review of CT Application Assessments


YES

YES

NO

Page 170


2665

2666

The full CT PEP is estimated to take 4–8 months to complete, depending on the number of 2667

product categories the NRA or RRS applies to be listed for, and on the number of assessors in 2668

the evaluation team. 2669

2670

The evaluation of mandatory ML4 sub-indicators and the newly developed performance 2671

evaluation indicators should take 1–2 months to complete; a minimum of two assessors is 2672

preferred to ensure peer review. 2673

2674

The expert review is expected to take 3–6 months to complete and will require 2–6 assessors 2675

who may or may not be the same assessors completing other elements of the CT PEP. 2676

2677

All NRA or RRS files, records and reports used for this performance evaluation must be less 2678

than three years old. Selection of this documentation, including number and type, is to be 2679

done by the assessor. 2680

2681

8.2. Mandatory ML4 GBT sub-indicators for CT 2682

WLAs for clinical trials oversight must fully implement the following ML4 indicators, as 2683

defined in the GBT: 2684

1. CT04.01 NRA has access to an advisory committee for review of CT applications and post-2685

approval safety and compliance issues. 2686

2. CT06.02 Performance indicators for CT oversight activities are established and implemented. 2687

2688

8.3. WLA CT performance evaluation indicators 2689

To be listed as WLA for CT, an NRA or RRS must be assessed against 10 newly developed MA 2690

indicators, as detailed below. In each case, the NRA or RRS must meet the criteria for an 2691

“implemented” rating. 2692

2693

8.3.1. Listing of CT in a registry 2694

2695

PE.CT.01 The NRA or RRS ensures that CTs are listed in a publicly-accessible register as part of the procedure for accepting CT submissions.

Description: The assessor should verify that a mechanism is in place (guidelines and procedures) at the NRA or RRS to assess whether, and demonstrate that, a CT is listed in a publicly accessible website or register, as part of the acceptance criteria of CT submissions.

Objective: This indicator aims to ensure that all CTs evaluated by the NRA or RRS are listed in a publicly accessible website or register.

Evidence to review:


• Written guidelines and SOPs.

• Publicly accessible registers identified in the NRA’s or RRS’ guidelines as the repository for CTs.

• At least three CT submissions accepted by the NRA or RRS, as selected by the assessor (to check listing).

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health


Page 171

Organization; 2021: CT sub-indicators (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale:

NOT IMPLEMENTED (NI): The NRA or RRS has not developed or implemented a well-established procedure to ensure that CT submissions accepted by the NRA or RRS relate to CTs listed in a publicly-accessible register; or it does not comply with the procedure in the applications reviewed. IMPLEMENTED (I): The NRA or RRS has defined and implemented a procedure to ensure that CT submissions accepted by the NRA or RRS relate to CTs listed in a publicly-accessible register; and it consistently complies with the procedure in the applications reviewed.


This sub-indicator cannot be scored as "not applicable " (i.e. this sub-indicator always applies when assessing CT WLAs).

2696

8.3.2. Submission and assessment of application for CT license and CT protocol approval, 2697

registration or licensing 2698

2699

PE.CT.02 The NRA or RRS follows the procedures established in its guidelines related to the requirements of investigational medical products (IMPs) to comply with GMP.

Description: The assessor should verify that the NRA or RRS has established guidelines for IMPs, in compliance with GMP; and that these are consistently respected. In particular, the assessor should verify that IMPs used in CTs accepted by the NRA or RRS were were adequately certified by the NRA or RRS of the country of origin.

Objective: This indicator aims to confirm that the NRA or RRS ensures IMPs used in CTs comply with GMP requirements.

Evidence to review:

The assessor should ask for and review: • Guidelines, SOPs, instructions or equivalent document establishing

requirements for IMPs.

• At least three CT files selected by the assessor (to check compliance).

References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator CT01.04 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with the established requirements on IMPs’ compliance with GMP in the reviewed applications. IMPLEMENTED (I): The NRA or RRS complies with the established requirements on IMPs’ compliance with GMP in the applications reviewed.



PE.CT.03 The NRA or RRS has a well-defined and established risk-based approach or mechanism for consistently selecting the most adequate pathway for each CT application evaluation (supported by appropriate guidelines or SOPs).

Description: The assessor should verify that a risk-based mechanism is in place to select the most appropriate pathway for assessing CT applications, including non-



Page 172


routine procedures such as expedited pathways; and that this mechanism is consistently and adequately applied.

Objective: This indicator aims to ensure that the NRA or RRS selects the most appropriate regulatory pathway, considering all relevant factors impacting public health conditions.

Evidence to review:

The assessor should ask for and review: • Guidelines, SOPs, decision trees or other evidence of a risk-based

mechanism.

• At least three files, selected by the assessor, for CTs with non-routine procedures.


Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS has not developed and established a mechanism (procedures/guidelines/decision trees) to select the most suitable pathway for each CT application. IMPLEMENTED (I): The NRA or RRS has defined and implemented a risk-based approach and mechanism (guidelines, procedures, decision trees) for appropriately selecting the most suitable pathway for each CT application.



PE.CT.04 The NRA or RRS complies with procedural requirements established in its guidelines and SOPs regarding the content and format of CT applications for non-routine procedures such as expedited pathways.

Description: The assessor should verify the procedural requirements for the content and format of CT applications for non-routine procedures; and then verify that the NRA or RRS consistently complies and applies with these requirements.

Objective: This indicator aims to ensure that non-routine procedures for CT applications (such as expedited pathways) adhere to content and format requirements. The ultimate objective is to ensure that this step enables timely and high-quality CT evaluation and authorization under certain circumstances (e.g. public health emergencies).

Evidence to review:

The assessor should request and review:

• Guidelines, SOPs or equivalent document establishing requirements for the content and format of CT applications for non-routine procedures.

• At least three CT files, selected by the assessor, that were subject to non-routine procedures.


Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with the established procedural requirements regarding the content and format of CT applications for non-routine CT procedures in the reviewed applications.




Page 173

IMPLEMENTED (I): The NRA or RRS complies with the established procedure requirements regarding the content and format of CT applications for non-routine CT procedures in the reviewed applications.



PE.CT.05 The NRA or RRS consistently complies with the timelines established in its guidelines and SOPs for assessing CT applications.

Description: The assessor should verify that the NRA or RRS has established expected timelines for assessing CT applications; and that these are consistently respected. The assessor should further verify that, in cases where timelines do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance with timelines is not common practice). The assessment of this indicator should be conducted alongside GBT sub-indicator CT06.04. In particular, assessors should ensure that: • Timelines for routine and non-routine CT applications, and for different stages

of the same application, are established following well-defined criteria; and are publicly available.

• Regulation and guidelines exist to ensure that CT applications are consistently processed and assessed according to prescribed timelines.

• An internal tracking system for active monitoring of compliance with published timelines is used and outcomes are documented.

Objective: This indicator aims to ensure that the NRA or RRS efficiently and consistently evaluates CT applications in a timely way. The ultimate objective is to ensure that this step is an enabler, rather than an obstacle, to the whole CT process.

Evidence to review:

The assessor should ask for and review: • Guidelines, SOPs, instructions or equivalent document establishing timelines

for assessing CT applications. • At least three files selected by the assessor (to check timelines).

At least three files, selected by the assessor, that do not comply with timeline requirements (to check for justification or rationale).

In addition, the assessor should ask for and review: evidence of an internal monitoring system for tracking compliance with timelines.


Rating Scale: NOT ACCEPTABLE: The NRA or RRS does not comply with established timelines for the evaluation of CT applications in the reviewed applications. ACCEPTABLE: The NRA or RRS complies with the established timelines for the evaluation of CT applications in the reviewed applications.


This sub-indicator cannot be scored as "not applicable" (i.e. this sub-indicator always applies when assessing CT WLAs).

2700


Page 174


8.3.3. Transparency and structure 2701

2702

PE.CT.06 The list of CT applications, including their current status, is publicly available or recorded in a domestic or international database.

Description: The assessor should verify that all CT applications, and their current status, are listed in the public domain; and that this is supported by a regulation or guideline. The assessor should also verify that existing regulations, guidelines or similar documents require all CT applications (including approved, terminated, suspended, withdrawn and other applications) to be listed in an easy-to-access domestic or international database; and that those lists be regularly updated. Established guidelines should include guidance on the information to be listed and the publication mechanism to be used.

Objective: This indicator aims to ensure that guidelines or regulations exist to mandate the NRA or RRS to publish, and make publicly available, information about the CT applications it receives, including their status.

Evidence to review:


• Regulations, guidelines or equivalent documents on listing of CT applications.

• Written guidance or SOPs on format and information of listing.

• Publicly available domestic and international databases with CT listings.

• At least three CT files, selected by the assessor (to check compliance) References: 1. WHO Global Benchmarking Tool (GBT) for evaluation of national

regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator CT05.02 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS has not developed or implemented a well-established procedure to regularly publish information about the CT applications and their status in a domestic or international database; or it does not comply with the procedure in the applications reviewed. There is no evidence of regular and consistent publication of information about CT applications and their status. IMPLEMENTED (I): The NRA or RRS has defined and implemented a well-established procedure to regularly publish the CT applications and their status in a domestic or international database, and it complies with the procedure in the applications reviewed. The NRA or RRS regularly and consistently publishes and makes available to the public the CT applications and their status.



PE.CT.07 The NRA or RRS has a mechanism, supported by adequate guidelines or SOPs, for sharing a summarized evaluation report of CT applications (or any other information about its decisions on CT applications) with other authorities.

Description: The assessor should verify that there is a mechanism for facilitating the exchange of information on regulatory decisions about CT applications (either approved, rejected or suspended) with other regulatory authorities, at least upon their request. The exact form of mechanism doesn’t matter



Page 175

as long as it ensures that relevant data is promptly made available to requesting NRAs or RRSs. In all cases, the sharing mechanism should include details of any information exchange arrangement, confidentiality agreement, or legal requirements from product sponsors required to allow information to be shared with other regulatory authorities. Sharing mechanisms should include the sharing of summarized technical evaluation reports or justifications of regulatory decisions. The assessor should verify that information is shared according to NRA or RSS guidelines; and that it is shared in a timely manner.

Objective: This indicator aims to ensure the existence of guidelines or regulations that allow the NRA or RRS to share information about their decisions on CT applications. The ultimate goal is to ensure the NRA or RRS embraces and follows adequate transparency policies and principles.

Evidence to review:

The assessor should ask for and review: • Documented guidelines or procedures to support information sharing;

• Templates for MoUs, confidentiality agreements or other relevant legal requirements;

• Any other evidence that a mechanism exists for sharing information with other regulatory authorities, for example publication of summarized reports on a public website; records of sharing information with a network of authorities through a private digital platform; emails of information shared with relevant contacts upon request.

• A number of CT application files, selected by the assessor, that have been shared.


Rating Scale: NOT IMPLEMENTED (NI): A mechanism for sharing information on CT applications with other regulators does not exist, or it is not promptly applied. IMPLEMENTED (I): The NRA or RRS has defined and established a mechanism for sharing information on CT applications with other regulators, at least upon their request. Exchange of information is promptly carried out.



2703

2704

8.3.4. Post-approval safety and compliance and reporting 2705

2706


Page 176


PE.CT.08 The NRA or RRS consistently complies with the requirements, including timeline requirements, established in its guidelines and SOPs for changing a CT’s original protocol or other relevant document.

Description: The assessor should verify that the NRA or RRS has established post-approval procedural requirements, including timeline requirements, for making changes and dealing with variations to the original protocol or other relevant document of the CT; cand that it consistently complies with these. The assessor should further verify that, in cases where CT post-approval procedures do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance with timelines is not common practice).

This indicator should be assessed alongside GBT sub-indicator CT06.04. In particular, the assessor should ensure that:

• Timelines for routine and non-routine CT applications, and for different stages of the same application, are established following well defined criteria and are publicly available.

• Regulation and guidelines exist to ensure that CT applications are consistently processed and assessed according to prescribed timelines.

• An internal tracking system for active monitoring of compliance with published timelines is used and outcomes are documented.

Objective: This indicator aims to ensure that post-approval activities for CT are efficiently and consistently performed by the NRA or RRS in relation to procedure requirements such as timelines.

Evidence to review:


• Guidelines, SOPs or equivalent document establishing requirements for post-approval procedures.

• At least three CT files, selected by the assessor, with changes to the initial submission (to check post-approval procedures, including timelines).

• At least three files, selected by the assessor, that do not comply with timeline requirements (to check for justification or rationale).

In addition, the assessor should ask for and review:

• evidence of an internal monitoring system for tracking compliance with timelines.


regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator CT01.02 and CT06.04 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS does not comply with established procedures, including timelines, in the reviewed applications.

IMPLEMENTED (I): The NRA or RRS complies with the established procedures, including timelines, in the reviewed applications.



Page 177



PE.CT.09 The NRA or RRS has documented procedures defining how long to store progress reports and when to destroy them; and applies them consistently.

Description: The assessor should verify that the NRA or RRS has established guidelines or procedures defining how long progress reports should be stored and when

they should be destroyed; and that these procedures are consistently respected. The assessor should further verify that, in cases where procedures for storing and destroying progress reports do not comply with guidelines, there is an acceptable justification or rationale given (to show that non-compliance is not common practice).

Objective: This indicator aims to ensure that the NRA or RSS stores and destroys progress reports consistently and effectively.

Evidence to review:

The assessor should ask for and review: • Guidelines, SOPs or other documented evidence of procedure requirements

for storing and destroying progress reports. • At least three CT files, selected by the assessor (to check procedures). • At least three CT files, selected by the assessor, that did not comply with

procedures for storing and destroying progress reports (to check justification and rationale).


regulatory system of medical products. Revision VI. Geneva: World Health Organization; 2021: Sub-indicator CT06.03 (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021).

Rating scale: NOT IMPLEMENTED (NI): The NRA or RRS has not implemented a well-established procedure for storing and destroying progress reports; or it does not comply with the procedure in the applications reviewed. IMPLEMENTED (I): The NRA or RRS has defined and implemented a procedure for storing and destroying progress reports; and it consistently complies with the procedure in the applications reviewed.



2707

8.3.5. Effectiveness of the CT process 2708

2709

PE.CT.10 The NRA or RRS provides regulatory support to clinical researchers and sponsors to assist in the development of new therapies for patients.

Description: The assessor should verify that the NRA or RRS gives clinical researchers and sponsors specific support. The NRA or RRS should have strong links and co-operation with ethics committees, including joint initiatives to improve the quality and extent of clinical trials. The NRA or RRS should also be actively involved in initiatives to create a positive regulatory environment for the development of new or improved therapies. It should regularly


Page 178


review the effectiveness and impact of its CT registration or license system on the development of new therapies; and take action towards improvement where possible.

Objective: This indicator aims to ensure that there are mechanisms in place to review how effective the CT registration or license system is in supporting the development of safe, effective and quality-assured new therapies.

Evidence to review:

The assessor should review documents, ask for and review documentation that can help answer the following questions, as applicable:

1. Is regulatory support to the development of clinical research in the country an objective of the agency? What actions are being taken in that regard?

2. Is regulatory support given to the development of new medicines/combination therapies/advanced therapies? In what way? Is there particular regulatory support for first-in-man trials?

3. Has support been given to researchers in terms of risk adaptation in clinical trials?

4. Is the agency involved in regional or international harmonised procedures or initiatives for assessing multinational clinical trials?

5. Is there provision of direct regulatory support and advice to industry and academic sponsors? How is this done?

6. What is the extent of interaction and co-operation with national and local ethics committees? What particular achievements have resulted from this?

7. Are there reviews of the agency’s contribution to an efficient regulatory environment for national clinical research? What improvements have been made?

Rating Scale:

NOT IMPLEMENTED (NI): The NRA or RRS does not take an active role in the development of new therapies in the country. IMPLEMENTED (I): The NRA or RRS is involved in national and international initiatives to create a positive regulatory environment for the development of new or improved therapies. There are strong links and co-operation with ethics committees, including joint initiatives to improve the quality and extent of clinical trials. Advice and assistance is provided, especially to academic sponsors. Where necessary and possible, changes have been made to the CT registration or license system, and its effectiveness and impact on the development of new therapies for the country is regularly reviewed. Note that if the NRA has been assessed throughout the BEMA IV cycle and showed to be rated as 4 or 5 in KPI 12.1, the NRA is considered to acceptably fulfil this indicator.



2710

8.4. WLA CT performance evaluation tools 2711

To be listed as WLA for CT, once an NRA or RRS has acceptably met all mandatory ML4 GBT 2712

sub-indicators and performance indicators PE.CT.01—PE.CT.10, it must complete an Expert 2713

Review of CT Application Assessments. 2714

2715


Page 179

PT.CT.11. Expert Review of CT Application Assessments 2716

2717


The Expert Review of CT Application Assessments should be done by a group of 2–3 experts 2719

with specific expertise on the type of product (chemical or biological) in the WLA scope. Two 2720

groups of experts may be required, depending on how many assessment reports are selected 2721

for review. The experts doing the review may or may not be the same assessors as those that 2722

evaluated the other elements of the CT PEP. 2723

2724

The experts should select a representative number (minimum 2–3) of NRA or RRS CT 2725

application assessment reports for review. Selection should only include assessment reports 2726

that are less than three years old. As a guiding principle, selection should include at least one 2727

assessment report for each of the product categories included in the WLA scope, unless the 2728

NRA or RRS applies to be listed for all four product types, in which case the experts should 2729

review three assessment reports covering: a new chemical entity, a vaccine and a biosimilar 2730

pharmaceutical product. The results of the assessment for a new chemical entity will then be 2731

extrapolated for a multisource product. 2732

2733

In preparation for their assessment, the NRA or RRS should make the CT application 2734

assessment reports available, along with the respective product dossiers. 2735

2736

The Expert Review of CT Application Assessments is done using the rubric below (see Table 2737

PE.CT.11), which comprises 4 indicators and 33 sub-indicators. The experts should ensure 2738

that all guidelines used as reference materials are relevant and up-to-date (but during each 2739

review, the guidelines that were in place at the time of the CT application and assessment 2740

should be used as the reference point). 2741

Page 180



For each selected assessment report, the expert assessors should complete each of the items set out in the table below. 2743

2744

Table PE.CT.11. Rubric for the Expert Review of CT Application Assessments. 2745 # Evaluation Criteria

Performance goal(s) to be met by the NRA Performance

adequately met by the NRA?

(Y/N)

Comments from the Expert on how the NRA complies or not with area measured


1 Application Process (including pre-submission procedures, assessment, compliance with regulatory requirements and policies, communication, interactions with stakeholders)

The focus of the evaluation of the Application Process should be directed to all activities that can have an impact in the Assessment.

1.1

Was there Regulatory advice or other similar activities provided by the NRA prior to the submission of CT application to support the success of a complete application with quality? Were there pre-submission meetings with the company for this application arranged by the NRA prior to the CT application submission to support the success of a complete application with quality? Are the applicants made aware of the NRA’s expectations, including the target timeframes, guidelines, requirements, templates and checklists?

Appropriate support and information have been provided to the sponsor by the NRA for the success of a CT application submission. All actions were taken by the NRA to allow a more predictable and clear process for applicants. The NRA benefited from a complete application submission at the outset by the applicant.

1.2 Were the relevant documented procedures to support the full CT application review process adequately followed? Was a review projection developed beforehand by the responsible team for the given application (including timelines for the different steps)? Was an assessment team leader assigned (based on a well-defined criteria)? Was the assessment team well formulated including involvement of all other relevant teams (e.g., staff with specific expertise for the given therapeutic area is involved, other teams are involved as necessary, such as inspections, MA, etc.)?

- How was communication and interdisciplinary work between scientific staff ensured?

Were the roles and tasks well distributed among team members?

The NRA adequately followed the relevant internal Guidelines and SOPs for the review process. The Review process was well organized and projected by the NRA beforehand, including relevant timelines established. Roles and responsibilities were clearly defined and followed as per the indications in the guidelines and SOPs. The communication and interdisciplinary work between scientific staff were properly handled for the given application.

1.3 Are there available documented procedures, templates and checklists relevant to support the NRA assessment process? Were they appropriately followed?

The NRA has available and adequately followed relevant documented procedures, templates and checklists for the CT application assessment process. If those were not strictly followed in some cases, this was well justified for each of


Page 181


Performance goal(s) to be met by the NRA Performance adequately met

by the NRA? (Y/N)



- Was the national guidance appropriately followed in terms of Ethics, Medical Care and records, conducting CT in the country and confidentiality?

- Was the submitted application based on relevant National and International standards and practices?

- Were International standards appropriately followed during the assessment of CT application?

o Declaration of Helsinki o WHO Guidelines, as applicable (e.g. WHO

Guidance for organizations performing in vivo bioequivalence studies)

o ICH Guidelines, as applicable (e.g. ICH Guidelines for good clinical practice E6 (R1))

those cases and the NRA ensured it didn’t impact the outcomes of the assessment. The NRA identified any lack or missed information in the application prior to scientific review, avoiding spending time and review resources on an application that does not allow critical analysis, signal identification or regulatory decision-making. The processes followed for this application assessment are perceived to be compliant and aligned with the established procedures and international standards and practices.

1.4 Quality and consistency of the assessment, reports and decision-making Is the system for ensuring quality and consistency of scientific work, assessments, assessment reports and decision-making, adequate and subject to review and improvements? For which type of applications are CT application assessment reports required, and not required? Is there guidance provided regarding the quality of the reports? Is training on report-writing provided? Overall, does the CT application assessment report complies with local and International Standards, defined by the NRA to be applied upon? Are appropriate record-keeping requirements in place for assessment reports? How is past experience and, regulatory and scientific memory for CT application assessment and decision-making created and maintained? What system is in place for reviewing the appropriateness of the assessors’ opinion/decision?

The NRA has robust procedures in place for evaluation of the quality and consistency of the assessment, assessment reports and decision-making. An adequate system is in place for maintenance of regulatory and scientific memory. Regulatory and scientific memory is effectively and actively disseminated. The agency ensures opinion making to standards and continuous improvement. The assessment, assessment reports and decision-making at the NRA is perceived to be consistently conducted and ensured.


2 Assessment Report The Experts are expected to have access to the whole dossier and consult it as needed to be able to evaluate the level of NRA’s performance on this section.

2.1 Quality of the report

Page 182




by the NRA? (Y/N)



2.1.1 Considers context Does the assessment report consider the data and the conclusions from the applicant? Does it include perspectives from patients/patient associations, health-care professionals and other NRAs’ analyses and decisions? Was there a mechanism activated to obtain opinion and advise from relevant stakeholders, as necessary, in the adequate moments of the assessment and as per the NRA guidelines establish?

The Assessment report considers all relevant data and conclusions from the applicant. The assessment report also considers any feedback provided by patients/patient associations and health-care professionals as well as other NRAs’ analyses and decisions. The NRA adequately followed its guidelines in terms of consulting and requesting advise from external experts, healthcare professionals and patients/patients association, as necessary and as per its guidelines.

2.1.2 Balanced and Evidence-based Is the assessment report objective and unbiased? Is the assessment report evidence-based? Does it reflect both the updated scientific and regulatory state of the art? Does it integrate legislative, regulatory and policy frameworks with emerging science, such as the Declaration of Helsinki? Are the type and number of objections raised and clarifications requested supported by evidences? Are concerns categorized in major and minor (or similar, based on national guidance)? Is the classification appropriate and supported by scientific discussion? Are the assessment of responses provided by the applicant considered into the final decision?

The Assessment report is evidence-based and factual. It considers and integrates emerging scientific and regulatory aspects and it is aligned with relevant legislative, regulatory and policy frameworks. It is based on updated and relevant technical guidelines. Specifically, the type and number of objections raised and clarifications requested are supported by appropriate evidences. The assessment of responses provided by the applicant is integrated into the final decision of the NRA.

2.1.3 Depth Does the assessment report comprehensively highlight potential areas of concern, providing a detailed analysis on those?

The Assessment report properly highlights and deeply analyses potential areas of concern supported by adequate justifications and observations.

2.1.4 Investigates problems Does the assessment report provide both the applicant’s and the assessors’ in-depth analyses and findings of key scientific data? Does the assessor demonstrate the use of risk-based tools, analyses and synthesis skills to ask relevant questions where needed?

The Assessment report provides comprehensive analysis and findings of key scientific data. The assessor demonstrated the use of risk-based tools, analyses and synthesis skills, to ask relevant questions and make appropriate judgments, where needed.

2.1.5 Makes linkages Does the assessment report provide integrated analysis across all aspects of the application: quality, pre-clinical; clinical; GxP compliance; study protocol? Does it include timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application?

The Assessment report provides good quality and integrated analysis of all relevant aspects of the application: quality, pre-clinical; clinical; GxP compliance; study protocol. It includes timely communication and consultation with applicants, internal stakeholders and, as needed, with external stakeholders who have expertise relevant to the various aspects of the application.


Page 183



by the NRA? (Y/N)



2.1.6 Thorough Does the assessment report reflect adequate follow-through of all the issues by the assessors?

The Assessment report reflects adequately follow-through of all issues raised by the assessors.

2.1.7 Utilizes critical analyses Does the assessment report assess the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application? Observations are well classified or categorized according to national agreed terms (e.g. major, minor)?

The Assessment report critically assesses the scientific integrity, relevance and completeness of the data and proposed labelling, as well as the interpretation thereof, presented in the application. Observations made throughout the report are categorized according to national agreed terms.

2.1.8 Well-documented Does the review report provide a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions and rationale for reaching a decision? Does it contain clear, succinct recommendations that can stand up to scrutiny by all the parties involved and could be leveraged by others? Observations are well described and detailed? Observations are well grouped or categorized?

The Assessment report provides a well-written and thorough explanation of the evidence-based findings and conclusions provided by the applicant in the dossier, and the assessors’ conclusions as well as rationale for reaching a decision. It contains clear recommendations and well described, detailed and categorized observations.

2.1.9 Well-managed Does the review report apply project and quality management processes, including clearly defined steps with specific activities and targets? Were timelines well managed throughout the assessment? CT application assessment report is finalized within the agreed timeframe?

The Assessment report applied adequate project and quality management processes, including clearly defined steps, targets and timelines. The timelines were well managed throughout the assessment procedure and this is reflected in the report. The final report was complete within the established timelines, as per the NRA guidelines stipulate.

2.1.10

Peer Reviews Is there a system for peer review? Was the assessment report subject to peer reviews? How is it completed and recorded? For which type of applications? How are the comments of the peer reviewer handled? Are they documented and kept?

The agency has an effective system for peer-review of reports. The assessment report was subject to adequate and well documented peer reviews. The comments provided by the peer reviewer were appropriately handled and addressed. When it’s not applicable, a proper justification is provided.

2.1.11

Information to the public Is the information to the public on the CT application assessment outcomes, easily readable and clearly communicated? Is it aligned with the national guideline requirements?

The information to the public on the CT application assessment outcomes of good quality, easily readable and clearly communicated to the target audience.

Page 184




by the NRA? (Y/N)




2.2 Completeness of the report to provide a comprehensive and complete picture of the situation or sample under consideration.

2.2.1 Were all relevant parts of the application file reviewed? All relevant parts of the dossier were reviewed and they are reflected in the assessment report.

2.2.2 Are all relevant regulations, standards and guidance referenced in the report, as necessary, linked to the respective observation?

All relevant regulations, standards and guidance are referenced in the report, as necessary, linked to the respective observation.

2.2.3 Is the Assessment Report complaint to the content and format described in the relevant SOP? Is the Assessment Report aligned with the registries’ information?

The Assessment report is compliant with the content and format described in the relevant SOPs or guidelines. It is also aligned with the published registries information.

2.2.4 Did it include the analysis on the oversight of these trials already occurring in other parts and by which government/organisation?

The Assessment report includes analysis of other trials (if any) already occurring in other parts and by which government/organisation.


2.3 Scientific rigor to ensure the application of the scientific approach for unbiased analysis and interpretation of the evidence or data

High-quality scientific work provides a sound basis for appropriate consistent and harmonised opinions and decisions that affect public health. - Are well described the main critical features of the CT, salient findings and

those deficiencies that justify any questions intended for the applicant? - Is the assessor’s own critical assessment and observations to the applicant

data included, particularly with respect to scientific elements and adherence to specific guidance documents?

- Are cross-references adequately used to clearly indicate the origin of any information used in the report, such as to the specific parts of the dossier (e.g. overview, summary, study reports), the references to the literature or other sources?

- Are those findings that need to be reflected in the SPC, Labels & Package Leaflet well emphasized?

- Are conclusions on the different scientific components well developed and described by the assessors?

The Experts are expected to look at the essential elements under each of those sections considering 1) the product scope – new chemical entities, multisource (BE studies), vaccines or biosimilars, and 2) the type of scientific components. He/she should use the list of items provided for guidance but mainly, his/her experience and judgement to analyse and evaluate the assessment conducted by the NRA on each of the areas for assessment. The Expert should aim to answer specific technical questions from a qualitative point of view. The Expert should write a summary of his/her findings for each of the scientific areas on how the assessment was conducted by the NRA (in terms of evidence assessed by the assessor, quality of such assessment and observations, and decision-making done by the assessor).

2.3.1 Pre-clinical data

2.3.1.1

Aspects to be considered: ▪ Comments on adequacy in relation to the proposed protocol (study) ▪ Demonstration of relevance of the animal model ▪ Nature of the target ▪ Pharmacodynamics ▪ Pharmaco- and toxicokinetics ▪ Safety pharmacology ▪ Toxicology


Page 185



by the NRA? (Y/N)



2.3.2 Quality

2.3.2.1

Aspects to be considered: ▪ investigational medicinal product (IMP), including comparators, blinded comparators, blinded test products and placebos and

the auxiliary medicinal products (if applicable) quality data ▪ The assessment of Manufacturing and import information of IMP ▪ Comments on adequacy in relation to the proposed protocol (study)

The assessment should focus on the compliance with the requirements concerning the manufacturing and import of investigational medicinal products and auxiliary medicinal products as well as compliance with the labelling requirements. Information to be provided for investigational medicinal products (IMPs) should focus on the risk aspects and should consider the nature of the product, the state of development/clinical phase, patient population, nature and severity of the illness as well as type and duration of the clinical trial. IMPs based on innovative and/or complex technologies may need more detailed data to be submitted.

2.3.3 Clinical (if any)

2.3.3.1

Aspects to be considered: ▪ Data from previous clinical trials and human experience (if applicable) ▪ Comments on adequacy in relation to the proposed protocol (study)

2.3.4 Investigational Brochure

2.3.4.1

Aspects to be considered: ▪ Confidentiality statement, ▪ Investigational product, physical chemical and pharmaceutical properties and formulation, ▪ nonclinical studies, ▪ effects in humans, ▪ summary of data and guidance for the investigator

Consideration should be given to the completeness and adequateness of the investigator's brochure.

2.3.5 GCP, GLP and GMP compliance

2.3.5.1

Aspects to be considered: ▪ the assessment of validity of GCP, GLP and GMP certificates

2.3.6 Study Protocol – risk benefit analysis

2.3.6.1

Aspects to be considered: ▪ the trial design, ▪ selection and withdrawal of subjects, ▪ treatment of subjects, ▪ assessment of efficacy,

Page 186




by the NRA? (Y/N)



▪ assessment of safety, ▪ discontinuation criteria for participants and stopping criteria, ▪ statistics, ▪ data handling and record-keeping, ▪ ethics and local suitability and compliance, including protection of subjects and informed consent, ▪ financing and insurance, ▪ quality control and quality assurance, and ▪ publication policy.

The assessment should include the following: i) The anticipated therapeutic and public health benefits taking account:

▪ the characteristics of and knowledge about the investigational medicinal products; ▪ the relevance of the clinical trial, including whether the groups of subjects participating in the clinical trial represent the

population to be treated, or if not, the explanation and justification; the current state of scientific knowledge; whether the clinical trial has been recommended or imposed by regulatory authorities in charge of the assessment and authorisation of the placing on the market of medicinal products;

▪ the reliability and robustness of the data generated in the clinical trial, taking account of statistical approaches, design of the clinical trial and methodology, including sample size and randomisation, comparator and endpoints;

(ii) The risks and inconveniences for the subject, taking account: ▪ the characteristics of and knowledge about the investigational medicinal products and the auxiliary medicinal products; ▪ the characteristics of the intervention compared to normal clinical practice; ▪ the safety measures, including provisions for risk minimisation measures, monitoring, safety reporting, and the safety plan; ▪ the risk to subject health posed by the medical condition for which the investigational medicinal product is being investigated.


2.4 Assessment Outcomes & Decision-making

2.4.1 Assessment Outcomes How is an overall assessment generated for an application? Are the conclusions on analysis and overall assessment outcomes consistently and adequately reached and concluded, in line with the assessment report observations, concerns and evidence reviewed? How did the assessors achieve an integrated opinion/outcome? Is there input or advice from technical committees, or from external experts? How was this or other input from ethics committee integrated into the opinion/outcomes? How were divergent views handled, if any?

Overall the assessment outcomes/opinions are aligned with the observations made throughout the assessment process. It reflects all observations and concerns as per those identified in the CT application assessment report. All input received during the assessment is adequately reflected in the report and in the opinion/outcome. Those are inclusive, comprehensive, documented and consistent. The production of the integrated opinion/outcomes from the assessors and their senior managers for the final decision-making by the agency is consistently and adequately achieved.

2.4.2 Final Decision-Making Overall the final decision-making is aligned with the CT application assessment report and observations made


Page 187



by the NRA? (Y/N)



Are the structures and levels of decision-making adequate and relevant? What was the basis on which decisions were taken? Is the information provided to the decision-makers adequate and relevant? Are some decisions taken at lower levels in the agency? On what basis is this done? Was the expertise used for decision-making adequate? Were technical committees and/or other stakeholders involved? How were divergent views handled, if any? How was consistency of opinion making ensured? Was the appropriateness of the process and of the decisions subject to review?

throughout the assessment process. It reflects all observations and concerns as per those identified in the assessment report and its outcomes. All input received during the assessment is adequately reflected in the final decision-making. The decision-making at the NRA is consistently ensured, supported by documented procedures which were adequately followed. The conclusion on the final decision by decision-makers at the agency was adequately achieved.



3 Assessment follow-up

3.1 Are there any changes made to the initial submission well reflected in the CT file?

Further changes to the initial submission (if any) are adequately reflected in the CT file

3.2 In case of emergency approvals (or expedited approvals provided under exceptional circumstances), are there follow-ups after CT licensing with respective reflection of any update in the CT file?

In case of emergency approvals (or expedited approvals provided under exceptional circumstances), there are appropriate follow-ups after CT licensing with respective reflection of any update in the CT file.


4 Assessors’ Technical Competency

4.1 Assessment team members have the required background and education. Assessment team members have the adequate qualifications (in terms of training and experience) in the field of CT assessments.

The team formulated for this assessment is perceived to be adequate, in terms of background, experience and apparent theoretical and practical knowledge on the relevant fields (CT)

4.2 Assessment team members are familiar with national and international regulations and guidance.

The team members made reference of relevant national and international regulations and guidance throughout the assessment report.

Page 188




by the NRA? (Y/N)



4.3 Assessment team members are coherent in scientific outcomes and decision and provide adequate rational and scientific justifications for their comments.

The team members were coherent in scientific outcomes and decision and provided adequate rational and scientific justifications for their comments.

4.4 Were conflicts of interest of staff and external experts properly dealt with, by the NRA (if any)? Was there any conflict of interest of staff and external experts perceived by the reviewer during this expert review?

The NRA properly deals with conflict of interests of assessors. The team formulated is perceived to be adequate for the assessment of this product, in terms of apparent conflict of interests.


2746

2747

References 2748

1. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory system of medical products. Revision VI. Geneva: World Health 2749

Organization; 2021: CT sub-indicators (https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021). 2750

2. Good review practices: guidelines for national and regional regulatory authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 2751

Preparations: forty-ninth report. Geneva: World Health Organization; 2015: Annex 9 (WHO Technical Report Series, No. 992 2752

https://apps.who.int/iris/handle/10665/176954, accessed 1 July 2021). 2753

2754

Rating scale 2755

The assessor uses the expert evaluation and comments to conclude whether or not the NRA or RRS can be considered to acceptably meet the 2756

requirements across the for main sections of the tool. 2757

2758

If the experts disagree on the outcomes of this evaluation, the decision will be referred to the WHO secretariat. 2759

2760

In all cases, the outcomes and findings of the expert review should be written up in a final report (using the tool and template presented in Table 2761

CT.1) and handed to the WHO secretariat together with the other CT PEP results. 2762

2763


N/A 2765

2766




Page 189

9. NRA lot release (LR) 2767

2768

9.1. LR PEP methodology 2769

The PEP for LR is designed to assess laboratories that do LR for an NRA or RRS, in vaccine-2770

producing countries only. This includes both NCLs and external laboratories that perform 2771

tests on behalf of the NRA. 2772

2773

In all cases, in addition to meeting the eligibility criteria, any NCL or external laboratory 2774

applying for WLA for LR status must also ensure that all the laboratory testing activities it 2775

does for LR meet the LT PEP requirements (see Section 7 above). 2776

2777

In addition to meeting the eligibility criteria, PEP for LR is considered fulfilled if the NCL or 2778

external laboratory demonstrates to: 2779

a. fully implement one mandatory ML4 GBT sub-indicator for LR (see Section 9.2). 2780

2781

Figure 9.1. Flowchart of the LR PEP. 2782

2783

2784

The full PEP for LR and LT combined are estimated to take no longer than six months to 2785

complete. 2786

2787

References 2788


products. Revision VI. Geneva: World Health Organization; 2021: Indicators for LT and LR 2790

(https://apps.who.int/iris/handle/10665/341243, accessed 29 June 2021). 2791

2792

Are usual eligibility criteria met?

Are all PE requirements for LT acceptably met?

START

END

YES

NO


LR PEP fulfilled

Are all mandatory LR ML4 sub-indicators acceptably met?

NO

YES

YES

NO


Page 190


2793

9.2. Mandatory ML4 GBT sub-indicators for LR 2794

WLAs for lot release must fully implement the following ML4 indicators, as defined in the 2795

GBT: 2796

1. LR06.04: Performance indicators for national lot release activities are established and 2797

implemented. 2798

2799


Page 191

2800

2801

2802

2803

2804

2805

2806

2807

2808

2809

2810

2811

2812

2813

2814

2815

2816

Appendix 1 to Annex 6 (6.1) – Vigilance Filed Visit Manual 2817


Authority

Page 192


2818

2819

2820

2821

2822

2823

2824

2825

2826

2827

Field Visit Manual for Assessing 2828

the Performance of Vigilance Function 2829

2830

2831

2832

2833

2834

2835

2836

2837

2838

2839

2840

2841

2842

2843

2844

2845

2846

2847

2848 2849

2850


Page 193

Contents 2851

CODE OF CONDUCT 194 2852

Acronyms and Abbreviations 195 2853

Specific definitions and terminology 196 2854

1. Introduction 198 2855

2. Purpose of the Manual 199 2856

3. Scope 200 2857

4. Objectives 201 2858

5. Expected Outcomes 202 2859

6. Deliverables 203 2860

7. Code of Conduct, Declaration of Interest and Confidentiality Undertaking 204 2861

8. Overview of Vigilance Field Visit Process 205 2862

8.1 GENERAL PRINCIPLES 205 2863

8.2 VIGILANCE FIELD VISIT PLANNING 206 2864

8.3 VIGILANCE FIELD VISIT PREPARATION 209 2865

8.4 VIGILANCE FIELD VISIT CONDUCT 210 2866

8.5 VIGILANCE FIELD VISIT REPORT 212 2867

9. Roles and Responsibilities 213 2868

9.1 Relevant NRA 213 2869

9.2 WHO Secretariat (HQ, RO and CO) 213 2870

9.3 WHO Team Leader 214 2871

9.4 WHO Team Member 215 2872

9.5 NRA Participants 215 2873

9.6 Visited site(s) 216 2874

10. References 217 2875

11. Acknowledgement 218 2876

12. Document Change History 219 2877

13. Annexes 219 2878

Annex 1: VIGILANCE FIELD VISIT CHECKLIST/QUESTIONNAIRE 220 2879

Annex 2: VIGILANCE FIELD VISIT CLOSING MEETING PRESENTATION FORM 264 2880

Annex 3: VIGILANCE FIELD VISIT REPORT FORM 266 2881

Annex 4: VIGILANCE FIELD VISIT TERMS OF REFERENCE FORM 272 2882

2883

Page 194


2884

CODE OF CONDUCT

WHO values and relies upon the normative and technical advice that is provided by leading subject matter experts in the context of its advisory/technical committees, meetings and other similar processes. Such advice contributes to the formulation of public health policies and norms that are promulgated by WHO for the benefit of its Member States.

In order to ensure the integrity of such processes, thereby contributing to their credibility in the eyes of WHO’s stakeholders, it is critical that experts appointed by WHO to render technical or normative advice

a. fully and honestly disclose all relevant interests and biases on the DOI Form that

may give rise to real or perceived conflicts of interest. Such disclosure must also be made orally to all fellow expert committee, meeting or group members at the outset i.e. unless this is done by the Chairperson or Secretariat;

b. spontaneously report any material changes to their disclosed interest on an on-

going basis during the period in which the expert serves the Organization; c. respect the confidential nature of committee or meeting deliberations or of the

advisory function assigned by WHO and not make any public statements regarding the work of the committee or meeting or regarding the expert’s advice without prior consent from WHO;

d. undertake not to engage in activities that may bring reputational harm to the WHO

process that they are involved in; e. undertake to represent their views in a personal and individual capacity with the

best interest of WHO in mind as opposed to representing the views of their employers, other institutions or governments;

f. actively and fully participate in discussions and deliberations within the relevant

advisory group, committee or meeting.

2885


Page 195

2886

Acronyms and Abbreviations 2887

2888

DOI Declaration of Interests

GBT Global Benchmarking Tool

HQ WHO Headquarters

ISO International Organization for Standardization

IT Information Technology

MOH Ministry of Health

MS Member State

NRA National Regulatory Authority

OpG Operational Guidance

PE Performance Evaluation

PEP Performance Evaluation Process

QMS Quality Management System

RO WHO Regional Office

RS National Regulatory System

RSS Regulatory Systems Strengthening

SOP Standard Operating Procedures

TOR Term of Reference

VL Vigilance (regulatory function)

WCO WHO country office

WHA World Health Assembly

WHO World Health Organization

WLA WHO Listed Authorities

2889

Page 196


Specific definitions and terminology 2890

2891

The definitions given below apply to the terms as used in the current document. These 2892

terms may have different meanings in other contexts. 2893

2894

Assessment Questionnaire: The questionnaire/form/template used for the evaluation of 2895

the performance and practice of VL function at several administrative levels of the target 2896

country. 2897

2898

Field Visit agenda: A plan developed by the WHO team leader, in agreement with other 2899

WHO Team Members and WHO Secretariat, to detail different activities, timings, and 2900

assignments to be performed during the conduct phase of the field visit. 2901

2902

Field Visit Report: A report prepared in English language which is delivered by WHO team 2903

following the predefined field visit report template. Field visit report provide an overview 2904

of the field visit activities, findings and recommendations, if any. 2905

2906

NRA Participants: One or more expert, ideally familiar with national medical products 2907

vigilance system, who is/are nominated by the NRA to represent it and to participate in the 2908

VL field visit. 2909

2910

Performance evaluation (PE) Indicators: a set of 16 qualitative or quantitative indicators 2911

which may be used by the WHO team to assess and evaluate the performance of VL function 2912

at the target country. Guidance for PE indicators is available in the form of fact sheets. 2913

2914

Team Leader: A competent expert in the area of medical products vigilance with team 2915

management skills. Team Leader is designated by WHO Secretariat and may or may not be 2916

a WHO staff. 2917

2918

Vigilance Field Visit: A process, using a WHO developed practice, that helps to document 2919

and evaluate the level of performance of vigilance function of a national medical products 2920

regulatory system. The activity consists of a field visit made by WHO team to several layers 2921

of the vigilance system (e.g., national, sub-national and health facility levels) to assess the 2922

performance and functionality of VL throughout the target country. The field visit may 2923

comprise onsite assessment of performance evaluation indicators of VL function for the 2924

purpose of WHO listed authorities designation. 2925


Page 197

2926

WHO Secretariat: The WHO unit in charge of organization of the vigilance field visit. 2927

2928

WHO Team (also called WHO Assessors): The team established by the WHO secretariat as 2929

indicated in the respective terms of reference (TORs) to perform the VL field visit. WHO team 2930

is usually composed of three experts including a designated team leader. WHO team may 2931

be accompanied by observers when needed. 2932

2933

WHO team members (also called WHO Assessor): A competent expert, who is familiar with 2934

WHO published regulations and guidelines in the area of medical products vigilance as 2935

relevant to the scope of VL field visit. 2936

2937 2938

Page 198


2939

1. Introduction 2940

2941

The World Health Assembly Resolution WHA 67.20 recognizes the important role that 2942

regulatory authorities play in a well-functioning healthcare system. As the regulation of 2943

medicines and vaccines and their procurement become more globalized, the World Health 2944

Organization (WHO) has sought to harmonize and strengthen regulatory activities through 2945

capacity building and the promotion of regulatory cooperation in order to encourage the 2946

supply of safe, effective and high-quality products. Towards this end, few decades ago, WHO 2947

established a regulatory system strengthening (RSS) programme to support Member States 2948

in development, establishment, and maintenance of their regulatory authorities/system. 2949

WHO’s support covers several areas including, among others, technical assistance, 2950

assessment of regulatory systems (using WHO global benchmarking tool (GBT), training and 2951

capacity building activities and most recently designation of WHO listed authorities (WLA) 2952

upon which other MS may rely. 2953

2954

Medical products vigilance, defined as the science and activities relating to the detection, 2955

assessment, understanding and prevention of adverse effects or any other medical product‐2956

related problems, is one of the regulatory functions which should be undertaken by National 2957

Regulatory Authorities (NRAs) or systems to contribute in guaranteeing that safe and effective 2958

medical products of high quality are used within the country. 2959

2960

One of the common regulatory functions subject to assessment during WHO benchmarking 2961

process, in the context of capacity building or WLA designation is medical products VL. 2962

Consequently, the need for comprehensive assessment and evaluation of the performance 2963

and functionality of VL was raised. In response to this need, WHO, in consultation with MS, 2964

partners and regulatory experts, developed the process and methodology of VL field visit. 2965

2966

Following this manual will ensure the necessary consistency in organizing VL field visit 2967

including defined roles and responsibilities which in turn will contribute to quality output and 2968

proper interaction among the involved and interested parties. 2969

2970

2971

2972

2973


Page 199

2974

2. Purpose of the Manual 2975

2976

The purpose of this manual are to: 2977

2978

• provide guidance to WHO Secretariat and WHO team as well as the concerned NRA, 2979

and other interest or involved parties on all aspects of the WHO VL field visit process 2980

and methodology, including the relevant procedures and timelines for planning, 2981

preparing, conducting, reporting, and following up along with templates for related 2982

documentation. 2983

• defines the composition of the WHO Team assigned to perform VL field visit as well 2984

as the required competencies, roles and responsibilities of team members. 2985

• describes the roles and responsibilities of the three levels of WHO (Headquarters 2986

(HQ); Regional Offices (ROs); and Country Offices (COs)) as well as the concerned NRA 2987

in this process accordingly. 2988

2989

This manual also intends to familiarize WHO Secretariat and WHO team , national regulatory 2990

authorities and other parties with a systemic approach for VL field visit, therefore this manual 2991

should be read as applicable in conjunction with other relevant manuals, guidelines, standard 2992

operating procedures (SOPs), and work instructions. 2993

2994

Finally, this manual is primarily designed to establish a level of consistency and uniformity 2995

within the VL Field Visit process and consequently provides for reliance on the outcomes of 2996

such process for benchmarking and WLA related purposes. 2997

2998

This manual is subject to periodic review and revision as part of the quality system approach 2999

applied by WHO. So, it is always advised for the user to ensure the utilization of the latest 3000

version of the manual. 3001

3002

Page 200


3003

3. Scope 3004

3005

The manual scope concerns the VL Field Visit conducted by WHO team for assessment and 3006

evaluation of the performance and functionality of medical products VL system at the target 3007

country. 3008

3009

This manual describes the process to initiate, plan, prepare, conduct, report upon and follow-3010

up VL field visit. It identifies the critical and key steps involved during a field visit to confirm 3011

that the performance of VL function at all levels of the target country is aligned with applicable 3012

requirements which in turn should meet WHO or other internationally recognized 3013

requirements. 3014

3015

In terms of medical product streams, this manual equally applies to field visit pertinent to 3016

medicines and biological products including biotherapeutic products as well as vaccines. 3017

However, some particularities may be noted within the questionnaire for VL performance 3018

assessment and PE indicators. All product specific requirements are marked as so in the 3019

respective documentation. 3020

3021


Page 201

3022

4. Objectives 3023

3024

3025

The objectives of the VL field visit are: 3026

I. Assessment of the performance of VL activities and operations, conducted at the site(s) 3027

selected for the field visit, 3028

II. Assessment of knowledge, competence and experience of the officials and staff 3029

involved in VL related activities at the selected site(s), 3030

III. Identification of strengths of the VL activities performed at the selected site(s), 3031

IV. Identification of areas which need further improvement and for which a specific 3032

development plan might be needed, and 3033

V. Feedback the relevant GBT sub-indicators or WLA performance evaluation process 3034

(PEP) sections on performance of the VL function. 3035

3036

Page 202


3037

5. Expected Outcomes 3038

3039

The expected outcomes of the field visit include: 3040

I. Determination if the performance and functionality of VL at the target country 3041

complies with WHO or other internationally recognized requirements and its own 3042

national regulatory requirements, 3043

II. Identification of the strengths, areas to improve and provide comments on how to 3044

address the identified gaps when necessary (e.g., in case of field visit for capacity 3045

building purposes), and 3046

III. Contribution to the conclusion of the overall performance of VL regulatory function as 3047

part of WHO benchmarking or WLA designation activities. 3048

3049


Page 203

3050

6. Deliverables 3051

3052

After completion of the VL field visit, the following deliverables should be provided to WHO 3053

Secretariat: 3054

I. VL field visit report (in English) to be delivered by WHO team following the template 3055

attached as annex 3 to this manual. 3056

3057

In case the field visit is organized for the purpose of WLA designation, in addition to the above 3058

mentioned deliverable, WHO team should also provide: 3059

II. Updated onsite assessment an evaluation of PE indicators following the respective 3060

template (included in the WLA Operational Guidance “OpG” as part of VL performance 3061

evaluation process). 3062

3063

3064

3065

Page 204


3066

7. Code of Conduct, Declaration of Interest and Confidentiality Undertaking 3067

3068

a) Prior to the VL Field visit process, confidentiality undertaking and DOI forms should 3069

be signed by all members of the WHO team, other than WHO Staff12, if any. 3070

3071

b) Completed and signed confidentiality and DOI forms should be assessed and archived 3072

by WHO Secretariat prior to the Field Visit following the respective WHO procedures. 3073

3074

c) Nominated members of the WHO team who might be found to have a conflict of 3075

interest with the VL field visit should be excluded. 3076

3077

d) The signed forms will be available at WHO and may be shared with the relevant 3078

authorities, if required. 3079

3080

e) All members of the WHO team shall familiarize themselves, respect and follow the 3081

WHO’s code of conduct (a short version of the same is included at the beginning of 3082

this manual). 3083

3084

f) If necessary, WHO and relevant NRA may engage in discussion towards the signature 3085

of confidentiality disclosure agreement. If so, the respective guidance and procedure 3086

given in the manual for WHO benchmarking should apply. 3087

3088

3089

3090

12 For WHO staff, other confidentiality and declaration of interest arrangements, as applied by Human Resources

Department during staff recruitment process, are in place.


Page 205

3091

8. Overview of Vigilance Field Visit Process 3092

3093

The field visit aims to assess the performance of the VL function with an emphasis on VL 3094

systems, structure and stakeholders along with VL activities including detection, reporting and 3095

data management; case investigation and analysis; risk assessment and management; 3096

information, education and communication with concerned groups; and human and financial 3097

resources. 3098

3099

8.1 GENERAL PRINCIPLES 3100

3101

a) Areas and components, other than the aforementioned ones, are contributing to a well-3102

functioning VL system, such as the legislations and regulatory requirements, 3103

infrastructures and resources, alert and crisis systems, surveillance programmes and 3104

quality management systems (QMS). It is worth to mention that VL field visit focuses on 3105

some, but not all aspects, related to VL function. Other tools and methodologies are 3106

indeed complementing VL field visit in the assessment of the overall VL function (e.g., GBT, 3107

PE indicators). It is essential then to consider these tools and methodologies together and 3108

not in a standalone mode (i.e., consider how GBT assessment contributes to and interacts 3109

with VL field visit and PE indicators). At the end of the assessment process (using one or 3110

more of these tools and methodologies), careful consideration of totality of evidence 3111

should be in place. In practical terms, WHO team performing VL field visit should be well 3112

briefed and aware of the outcomes of any earlier assessment, if any. 3113

3114

b) VL field visit is concerned with actual activities and operations of the vigilance system in 3115

the field (across the target country). GBT, on the other side, is concerned with systemic 3116

aspects of the VL function while PE indicators are concerned with quantitative and 3117

qualitative performance evaluation of the VL function. 3118

3119

c) The NRA, WHO and if necessary the site(s) subject to VL field visit should discuss, in 3120

advance, and agree on all details and aspects related to the visit, including the 3121

participants, the observers and translation (if any). 3122

3123

Page 206


d) To facilitate the WHO VL field visit in evaluating the VL function, a copy of the VL related 3124

procedures or SOPs including reporting and communication forms should be shared by 3125

the NRA with WHO preferably two weeks before the visit. 3126

3127

e) WHO team should have unlimited access to information, people and assets relevant to the 3128

VL field visit while respecting all applicable confidentiality arrangements and code of 3129

conduct. With regard to unlimited access to people, WHO team should have the right to 3130

interview employees without formally respecting the hierarchical lines. However, WHO 3131

team should always demonstrate respect for the relevant organization’s culture and 3132

habits. 3133

3134

8.2 VIGILANCE FIELD VISIT PLANNING 3135

3136

8.2.1 Terms of Reference (TOR) 3137

3138

a) Prior to the field visit, WHO Secretariat should prepare the TOR in collaboration and 3139

agreement with the counterparts at the concerned NRA and following the applicable WHO 3140

procedures. 3141

3142

b) The TOR should specify, among others, objectives, proposed dates, a tentative agenda, 3143

expected outcome and deliverables, the composition of the WHO team, along with initial 3144

list of persons to be met and documents or information needed during the visit. It should 3145

be noted that during the visit, WHO team may request additional documents, information, 3146

or people to interview (please refer to 8.4 b). 3147

3148

c) The TOR should consequently be shared with the concerned NRA through official 3149

communication channels for agreement and concurrence. 3150

3151

d) The TOR to be used for the field visit should be available and distributed to all participants, 3152

including the sites which will be visited if necessary. The TOR should also be available at 3153

WHO secure information sharing platform for access and archival purposes. 3154

3155

e) The list of WHO team members along with the designated Team Leader should ideally be 3156

shared with the NRA at least 15 working days prior to the VL field visit. 3157

3158


Page 207

f) A template for “Terms of Reference Form for WHO Vigilance Field Visit” is attached as 3159

annex 4 of this manual. 3160

3161

8.2.2 Agenda 3162

3163

a) Additionally, as part of the TOR, a tentative agenda should be developed. After discussion 3164

and agreement with NRA officials, WHO team should finalize and issue “WHO Field Visit 3165

Agenda” as per the template included in annex 4 attached to the current manual. 3166

Finalized agenda should be shared well in advance of the visit among all participants 3167

including observers. 3168

3169

8.2.3 Secure information sharing 3170

3171

a) Once the field visit is agreed and confirmed between WHO and NRA as part of the 3172

benchmarking or WLA related activities, a specific page under the MS site at the WHO 3173

secure information sharing platform should be created. 3174

3175

b) All related documents should to be uploaded to such secure information sharing platform 3176

for access and archival purposes including, but not limited to, TOR, agenda, background 3177

documents, documented evidences submitted by the NRA (e.g., procedures), WHO team 3178

information, travel and accommodation information, presentations, and reports. 3179

3180

c) By default, access to the said WHO information sharing platform is restricted to 3181

authorized users. Access to the platform should be granted to WHO team as well as 3182

selected officials nominated by the relevant NRA, after the confidentiality agreement and 3183

the DOI are signed, in order to enable them to communicate with each other and 3184

upload/download relevant information. 3185

3186

8.2.4 Entities or Sites selection 3187

3188

a) Selection of the site(s) subject to the field visit should be agreed between the NRA 3189

and WHO Secretariat. 3190

3191

b) When needed, the NRA should provide WHO Secretariat with a comprehensive list 3192

of sites (including name and address of entities) at a specific administrative level or 3193

Page 208


geographical area, in order to help in selection of the site(s) which will be involved in 3194

the VL field visit. 3195

3196

c) In principle, the site(s) should be selected among those sites which are regularly 3197

involved in VL activities (e.g., reporting, investigation, response). 3198

3199

d) Factors to consider in selection of the site(s) include, among others, complexity of 3200

activities or processes, criticality of products or the geographic and multi ethnic 3201

outreach. The ultimate objective is to have a representative sample of the VL 3202

activities and operations. 3203

3204

e) Simulations or VL activities scheduled for the sole purpose of the field visit should 3205

not be considered by any means. 3206

3207

8.2.6 Translations 3208

3209

a) The VL field visit should be performed in a language well understood by the WHO team 3210

and the participants. 3211

3212

b) Preferably, the VL field visit should be performed in the official language of the target 3213

country, if well understood by WHO team. 3214

3215

c) If needed, simultaneous translation service may be provided for the WHO team. 3216

Translators should preferably have technical expertise with VL function. Translation 3217

can be provided by the site(s) subject to the field visit. 3218

3219

d) Apart from simultaneous translation, WHO Secretariat, in coordination and 3220

agreement with the NRA, may request for translation of documentation related to 3221

the VL field visit (e.g., procedures, guidelines). 3222

3223

e) If translation services are required, these should be confirmed well in advance of 3224

the field visit and arrangements should be agreed between the NRA and WHO, and, 3225

if applicable, the inspected site(s). 3226

3227

8.2.7 Team members attributes and competencies 3228

3229


Page 209

a) A roster of qualified experts should be accessible for WHO Secretariat to conduct VL 3230

field visit on behalf of the organization. 3231

3232

b) The WHO team members should be qualified and competent according to a well-3233

established criteria to conduct the requested field visit. 3234

3235

c) The following represent the criteria for designation of WHO team members: 3236

➢ Background and education: WHO team member should be a staff of a NRA or a 3237

WHO staff or expert (e.g., consultant) who is familiar with the relevant WHO 3238

published and other internationally recognized standards and guidelines. 3239

➢ Experience: WHO team member should be experienced in the field of medical 3240

products vigilance and should have at least seven (7) years of experience in that 3241

field. 3242

➢ Training: apart from technical training on the medical products vigilance which is 3243

covered under experience section, WHO team member should be well trained on 3244

the processes and methodologies related to field visit. WHO team Member 3245

should also be familiar with WHO benchmarking and WLA concepts and 3246

methodologies. 3247

➢ Skills: WHO team member should have advanced skills in benchmarking, 3248

assessment and investigation activities as well as questioning and listening, and 3249

team management skills. 3250

➢ Evaluation: WHO team member should be subject to formal evaluation by WHO 3251

staff (e.g., WHO team leader) against preset criteria and in accordance with the 3252

relevant procedures. 3253

3254

d) In terms of number, the VL field visit should be performed by a group of experts whose 3255

number is commensurate to the assigned roles and responsibilities. 3256

3257

8.3 VIGILANCE FIELD VISIT PREPARATION 3258

3259

8.3.1 Briefing session 3260

3261

a) Apart from initial qualification and enlisting within the roster of qualified WHO 3262

assessors, the WHO team member selected for each individual VL field visit should be 3263

Page 210


thoroughly briefed on the principles described in this manual prior to the start of the 3264

visit. 3265

3266

b) The WHO Secretariat or Team Leader should brief all Team Members remotely as part 3267

of preparation for the visit. The briefing should include details related to: 3268

• Context of the field visit including objectives and expected outcomes; 3269

• Methodology of field visit; 3270

• Availability of required documents; 3271

• Access and utilization of WHO secure information sharing platform; 3272

• Roles and responsibilities of different team members, including specific task(s); 3273

• Other related logistical arrangements (e.g., travel, accommodation); and 3274

• Answer question raised and provide clarifications sought by the Team Members. 3275

3276

c) When necessary, such briefing may be repeated between 2 to 3 times to cover all team 3277

members. 3278

3279

8.3.2 Documentation Review 3280

3281

a) As part of the preparation for the field visit, the WHO team should review the following 3282

documents, to the extent possible, well in advance prior to the conduct of the visit: 3283

• Quality Manual along with all standard operating procedures (SOPs) particularly 3284

those related to medical products vigilance function, 3285

• A copy of national VL code/regulations/guidelines, 3286

• Background documents about the institution/entity/site/facility subjected to the 3287

VL field visit. 3288

3289

b) Each team member, no matter how experienced he/she is, will need to spend the 3290

necessary time preparing for the field visit, reading background documents. 3291

3292

c) To facilitate the preparation process for the visit, 10 days before the start of the field 3293

visit at the latest, the relevant NRA coordinator(s) shall upload the above mentioned 3294

documents to the relevant secure WHO information sharing platform. 3295

3296

8.4 VIGILANCE FIELD VISIT CONDUCT 3297

3298


Page 211

a) The VL activities and operations subject to the field visit should take place in accordance 3299

to routine practice, as defined in the procedures of the NRA and in accordance with the 3300

relevant NRA Quality Management System (QMS). 3301

3302

b) WHO team may ask questions, request documents from the representatives of the visited 3303

site(s) or request interview of one or more of the staff working at the site(s). 3304

3305

c) Documents review alone cannot usually assure the degree to which documents accurately 3306

reflect work activities. So, documents review should always be combined with 3307

discussions, interviews, questions and most importantly observation. 3308

3309

d) WHO team should, to the extent possible, witness actual operations and activities. 3310

3311

e) Records and documents should be selected carefully for review to ensure that they are 3312

representative and adequately characterize the program, system, or process being 3313

assessed. 3314

3315

f) For the purpose of evaluation and assessment of the VL processes, operations and 3316

practice, WHO team should make use of the "checklist/questionnaire" attached as annex 3317

1 to the current manual. 3318

3319

g) The agenda of the VL field visit should be respected however it may amended/adjusted if 3320

needed. Amendment of the agenda should be discussed with participants from the NRA. 3321

3322

h) At agreed intervals (e.g. end of each working day), WHO team should review the process 3323

and plan of the VL field visit with the participants from the NRA. WHO Team should 3324

provide feedback on the identified strengths and gaps so far during such meetings. Other 3325

participants from the visited site(s) may join one or more of the meetings. 3326

3327

i) Throughout the field visit, WHO team should make clear, accurate and legible notes. Such 3328

notes should provide relevant yet detailed facts that serve as a record of what was 3329

assessed and evaluated. Such notes should ideally be used for the formulation of the field 3330

visit report. 3331

3332

j) Once the field visit is finished, the WHO team should hold a de-briefing meeting with the 3333

NRA, involving, as appropriate, other representatives from the NRA (e.g. top 3334

Page 212


management). The purpose is to brief the attendees about the field visit activities and 3335

present the findings including the identified strengths, gaps, areas to be improved and 3336

recommendations (if any). 3337

3338

k) For the purpose of debriefing meeting with the NRA representatives, WHO team should 3339

make use of the "VL Field Visit Closing Meeting Presentation Form" attached as annex 2 3340

of the current manual. 3341

3342

8.5 VIGILANCE FIELD VISIT REPORT 3343

3344

a) The WHO team should issue a VL field visit report (in English or bilingual) as per the format 3345

attached as annex 3 to the current manual. 3346

3347

b) The finalized VL field visit Report should be made available to WHO Secretariat within 14 3348

working days from the last day of the visit. 3349

3350


Page 213

3351

9. Roles and Responsibilities 3352

3353

VL field visit should be seen as a collaborative exercise for which several parties, including 3354

NRA, WHO Secretariat, WHO team, and visited site(s), are contributing. This section is meant 3355

to provide guidance on the roles and responsibilities among the aforementioned parties. 3356

However, each party should collaborate as much as possible with other entities towards 3357

meeting the objectives of the field visit. 3358

3359

9.1 Relevant NRA 3360

3361

a) The NRA concerned with the VL field visit is responsible to: 3362

• Discuss and agree with WHO Secretariat on selection of the site(s) which will be 3363

subject to the field visit. 3364

• Designate one or more focal person to coordinate the field visit related activities. 3365

• Nominate the NRA participant joining the field visit. 3366

• Share with WHO, through the secure information sharing platform or any other 3367

agreed means, all necessary information and documentations including, among 3368

others, national code/regulations/guidelines, relevant procedures, data specific to 3369

the site(s) selected for the visit. 3370

• Nominate officials for granting them access to the WHO secure information 3371

sharing platform. 3372

• Communicate and coordinate with the visited site(s) including all necessary 3373

management and logistical arrangements. 3374

• Grant WHO team’s access to all relevant date and information throughout the field 3375

visit. 3376

3377

9.2 WHO Secretariat (HQ, RO and CO) 3378

3379

a) WHO HQ (RSS team), in collaboration with six WHO ROs as well as relevant COs, is 3380

responsible for establishment and maintenance of the tools and databases related to 3381

field visit as well as establishment of a roster of qualified assessors along with their 3382

training in order to ensure consistency and quality of the process as well as robustness 3383

of the outcome of the field visit. 3384

3385

Page 214


b) WHO Secretariat is also responsible for: 3386

3387

• Discussion and agreement with NRA on selection of the site(s) which will be subject 3388

to the field visit. 3389

• Establishment of a dedicated page on the country’s site of the WHO information 3390

sharing platform for the field visit and upload all relevant documentation for access 3391

and archive purposes. 3392

• Designation of the WHO team leader. 3393

• Selection of the WHO team members from the roster of qualified assessors to 3394

perform the field visit on behalf of WHO. 3395

• Organization of any necessary contractual and logistical arrangements. 3396

3397

9.3 WHO Team Leader 3398

3399

a) The WHO team leader is responsible for: 3400

• Leading and coordinating the VL field visit from the beginning to the end of the 3401

process. He/she will also participate in the evaluation and assessment of the 3402

performance and functionality of VL during the field visit. 3403

• Briefing the WHO team members on different aspects related to the field visit 3404

including context, background, objectives, process and methodology, roles and 3405

responsibilities as well as safety issues, if any. 3406

• Coordination of work among all members of the WHO team in order to ensure 3407

smooth and harmonized execution of the field visit with avoidance of work 3408

duplication and/or conflicts. 3409

• Communication with the NRA: During the field visit, WHO team leader should 3410

communicate with officials of the relevant NRA as well the visited sites on behalf 3411

of WHO. 3412

• Delivering presentations: Presentations during the field visit opening and close 3413

meetings will be ideally made and handled by WHO team leader. Nevertheless, 3414

the preparation of these presentation as well as inputs from different WHO team 3415

members would be necessary. Similarly, WHO team leader may invite any of the 3416

WHO team members to present about the findings, provide clarifications, answer 3417

questions of the NRA or the visited site if needed. 3418


Page 215

• Delivering the field visit report: The overall report of the field visit should ideally 3419

be prepared by all WHO team however the responsibility of delivering the finally 3420

agreed report lies on the WHO team leader. 3421

3422

9.4 WHO Team Member 3423

3424

a) The WHO team members are responsible to: 3425

• Review and sign the relevant administrative documents including invitation letter, 3426

confidentiality agreement, and DOI. 3427

• Make necessary travel arrangements (e.g., book flights and obtain visa) as 3428

described in the invitation letter. 3429

• Comply with the immunization requirements and bring with them a copy of their 3430

immunization certificates, if necessary. 3431

• Respect all applicable protocols, ethics and codes of conduct. 3432

• Assess and evaluate the performance of VL operations and activities using the 3433

checklist/questionnaire attached as annex 1 to the current manual. 3434

• Identify the strengths as well as the gaps and areas for improvement, if any. The 3435

identified strengths and areas for improvement should be presented in the visit 3436

closing meeting using the "Vigilance Field Visit Closing Meeting Presentation Form" 3437

attached as annex 2 of the current manual. 3438

• Prepare a detailed report on the field visit conducted including general information 3439

of the field visit, activities, findings (strengths, gaps and areas for improvement) 3440

and recommendations, if applicable, to address the identified gaps as per the 3441

"Field Visit Report" attached as annex 3 to the current manual. The field visit 3442

report should be provided to the WHO within 14 working days, at the latest, from 3443

the last day of the field visit. If possible, a draft of the same shall be delivered by 3444

the WHO team on the last day of the visit. The report may quote the different 3445

components/sections in the checklist/questionnaire. 3446

3447

9.5 NRA Participants 3448

3449

a) The NRA participants are responsible to: 3450

• Establish and maintain a formal and professional communication between the 3451

WHO team, and the visited site, 3452

• Coordinate the field visit on-site, 3453

Page 216


• Discuss and consider any request for adjustment of the field visit agenda, 3454

• Ensure easy access of the WHO team to the requested documents, information 3455

and persons, and 3456

• Respond to questions and provide clarification sought by WHO team. 3457

3458

9.6 Visited site(s) 3459

3460

a) The inspected site(s) is responsible to: 3461

• Prepare all materials requested by WHO Team, if any, prior to the planned visit, 3462

• Provide clarifications and explanations, sought by WHO team, of systems and 3463

protocols used for daily activities, and 3464

• Respond to WHO team’s questions and calls for interview, if any. 3465

3466

3467

3468


Page 217

3469

10. References 3470

3471

For the purpose of developing the current manual, the below references have been 3472

consulted: 3473

3474

WHO Global Benchmarking Tool (GBT) for Evaluation of National Regulatory System of 3475

Medical Products: Glossary and Definitions. [online] available at: 3476

<https://www.who.int/medicines/regulation/10_gbt_glossary_revvi.pdf> [accessed 9 3477

February 2021]. 3478

3479

WHO Global Benchmarking Tool (GBT) for Evaluation of National Regulatory System of 3480

Medical Products. Vigilance: Indicators and Fact Sheets, Revision VI version 1. [online] 3481

available at: < 3482

https://www.who.int/medicines/regulation/03_gbt_vl_rev_vi_ver_1nov2018_final_adjuste3483

d.pdf> [accessed 11 February 2021]. 3484

3485

Policy: Evaluating and publicly designating regulatory authorities as WHO listed authorities, 3486

Working document QAS/19.828/Rev.1, July 2020. [online] available at: 3487

<https://www.who.int/docs/default-source/medicines/norms-and-standards/current-3488

projects/qas19-828-rev1-policy-on-who-listed-authorities.pdf?sfvrsn=49504b99_2> 3489

[accessed 11 February 2021]. 3490

3491

WHO Pharmacovigilance Indicators: A Practical Manual for the Assessment of 3492

Pharmacovigilance Systems. [online] available at: 3493

<https://www.who.int/medicines/areas/quality_safety/safety_efficacy/EMP_PV_Indicators_3494

web_ready_v2.pdf?ua> [accessed 11 February 2021]. 3495

3496

https://www.who.int/medicines/regulation/10_gbt_glossary_revvi.pdf

https://www.who.int/medicines/regulation/03_gbt_vl_rev_vi_ver_1nov2018_final_adjusted.pdf

https://www.who.int/medicines/regulation/03_gbt_vl_rev_vi_ver_1nov2018_final_adjusted.pdf



https://www.who.int/medicines/areas/quality_safety/safety_efficacy/EMP_PV_Indicators_web_ready_v2.pdf?ua

https://www.who.int/medicines/areas/quality_safety/safety_efficacy/EMP_PV_Indicators_web_ready_v2.pdf?ua

Page 218


3497

11. Acknowledgement 3498

3499

Authors: Jun Kitahara, PMDA, Japan; Mohamed Refaat, WHO/HQ, Geneva; and Sten Olsson, 3500

Sweden. 3501

3502

Contributors: Abena Asamoa-Amoakohene, FDA Ghana, Ghana; Amira Amin, Egyptian Drug 3503

Authority, Egypt; Adriana Padilla Mendoza, COFEPRIS, Mexico; Alexandre Lemgruber, 3504

WHO/AMRO (PAHO), Washington; Alireza Khadem, WHO/HQ, Geneva; Ayako Fukushima, 3505

WHO/HQ, Geneva; Brigitte Mauel-Walbrol, BfArM/BEMA, Germany; Claudia Alfonso, 3506

WHO/HQ, Geneva; Diego Alejandro Gutiérrez Triana, INVIMA, Colombia; Elham Kosary, 3507

WHO/HQ, Geneva; Fernanda Maciel Rebelo, ANVISA, Brazil; Fernanda Simioni Gasparotto, 3508

ANVISA, Brazil; Giset Jiménez López, CECMED, Cuba; Hiiti Sillo, WHO/HQ, Geneva; Houda 3509

Langar, WHO/EMRO, Cairo; Houda Sefiani, National PV Centre, Morocco; Ines Hassan, WHO 3510

Consultant, London; Iris Lorena Arreola Dominguez, COFEPRIS, Mexico; Jinho Shin, 3511

WHO/WPRO, Manila; Johan Ellenius, Uppsala Monitoring Centre, Sweden, Jose Luis Castro, 3512

WHO/AMRO (PAHO), Washington; Jun Kitahara, PMDA, Japan; Libert Chirinda, MCAZ, 3513

Zimbabwe; Madhava Ram Balakrishnan, WHO/HQ, Geneva; María Francisca Aldunate 3514

González, ISP, Chile; Mauricio Beltran, WHO/AMRO (PAHO),Washington; Mohamed Refaat, 3515

WHO/HQ, Geneva; Monica Plöen, Uppsala Monitoring Centre, Sweden; Mubarak Saeed 3516

Alshahrani, Saudi FDA, Saudi Arabia; Sten Olsson, Consultant, Sweden; Sunday Kisoma, TMDA, 3517

Tanzania; Tohlang Sehloho, SAHPRA, South Africa; Verónica Vergara Galván, ISP, Chile; and 3518

Vincent Chow Wai-yan, Department of Health, Hong Kong SAR. 3519


Page 219

3520

12. Document Change History 3521

3522

3523

Version no. Date of issue Main changes

1 April 2021 First version 3524

3525

3526

13. Annexes 3527

3528

Page 220


Annex 1: VIGILANCE FIELD VISIT ASSESSMENT QUESTIONNAIRE 3529

3530

Vigilance Field Visit Assessment Questionnaire 3531

Guidance on how to use this assessment questionnaire 3532

• The objective of this questionnaire is to assess the performance of the medical products vigilance function during the VL field visit. 3533

• The questionnaire is made of two parts; part I for assessment of vaccine vigilance systems and part II for assessment of medicine 3534

vigilance systems. 3535

• Each of the two parts of this questionnaire comprises three different sections targeting national, sub-national and health facility levels 3536

• This questionnaire comprises both “open-ended questions” and “closed-ended questions”. 3537

• WHO team should complete the respective fields in this questionnaire and attach a copy of the completed questionnaire to the VL field 3538

visit report. 3539

• Whenever possible, please attach “electronic” copy of the relevant documents reviewed during the field visit to this questionnaire. 3540

3541

3542


Page 221

Part I: Questionnaire for assessment of the performance of vaccine vigilance systems 3543

Section 1: assessment at the national level 3544

This section is targeting the assessment of the performance of vaccine vigilance system at the national levels, namely: 3545

1) The National Regulatory Authority (NRA) including central vigilance center, and 3546

2) The National Immunization Programme (NIP) – also called Expanded Programme on Immunization (EPI) 3547

3548

ID Question Guidance and value range Related GBT or

PE indicators Comment and justification

General information

➢ Country: ➢ Institution(s) assessed: ➢ Persons met and interviewed:

SYSTEMS, STRUCTURE AND STAKEHOLDER COORDINATION

I-1-01 Do you have a national vigilance center ? No need to address this question as it has

been addressed by the GBT or PE indicators.

VL02.01 Please refer to the related GBT or

PE indicator.

I-1-02 If YES to question I-1-01, is the national vigilance

center a full or associate member of the WHO

collaborating centre for international drug

monitoring ?

YES, NO. PE.VL.06

I-1-03 Do you have designated National focal point for

vaccine AEFI ?

YES, NO.

If YES, provide contact information

➢ At NRA ➢ At NIP/EPI ➢ At MoH

VL02.01

I-1-04 Do you have written National AEFI surveillance

guidelines?

No need to address this question as it has


VL01.06 Please refer to the related GBT or

PE indicator.

I-1-05 does the national AEFI guidelines fulfill WHO

recommended format ?

Guidelines includes:

• Objectives of the system • List of AEFI to be reported • Case definitions of AEFI to be reported • Clear definitions of terminology relevant

for analysis and response (e.g. adverse event versus adverse reaction;

VL01.06

Page 222




coincidental, program error, serious events, cluster events)

• Information on how to report (who, how, where, when)

• All vaccines to be included in the reporting system (not only EPI vaccines)

• Procedure for analyzing data • Feedback procedure back to key players,

parents, communities of findings and relevant actions

• Procedure for investigating and actions to be taken in case of serious AEFI or cluster events

• Definition of the people in charge

I-1-06 Have these guidelines been communicated to

staff at all levels?

Select all that apply

• National level • At sub-national level • At Health facility level

VL03.02

I-1-07 Do EPI and NRA collaborate regularly to review

vaccine safety issues?


(1) Notifying each other on AEFI,

(2) Sharing AEFI reports,

(3) Convening regular meeting

between the institutions,

(4) being involved or

coordinating analysis of data;

(5) Sharing report analysis or

summaries,

(6) Jointly participating in

national AEFI committee reviews,

(7) other- please specify

VL02.02

I-1-08 Do you have a national database or system for

collating, managing and retrieving AEFI reports?



VL04.01

VL04.02

Please refer to the related GBT or

PE indicator.

I-1-09 Do you have a quality management system for

vaccine pharmacovigilance activities ?



RS05 Please refer to the related GBT or

PE indicator.


Page 223



I-1-10 Do you have a management system to ensure

traceability of actions ?



RS05 Please refer to the related GBT or

PE indicator.

Overall evaluation of the SYSTEMS, STRUCTURE AND STAKEHOLDER COORDINATION

The WHO team conclude that the assessed areas are: (please tick one of the below checkboxes)

☐ Satisfactory

☐ Unsatisfactory

Justification: (please provide text)

DETECTION, REPORTING AND DATA MANAGEMENT

I-1-11 Do you have written procedures on actions to be

taken in case of serious AEFI or cluster of AEFIs,

e.g. SOP for reporting and case management?

YES, NO

If YES, please provide document.

VL04.01

VL04.02

I-1-12 Is it mandatory to report serious AEFI ? At national level

At sub-national level

At health facility level

VL04.01

VL04.02

I-1-13 Is it mandatory to report non-serious At national level

At sub-national level

At health facility level

VL04.01

VL04.02

I-1-14 At which level is the list of AEFIs eligible for

reporting disseminated?


• National level • Sub-national level • Health facility level • Do not have list of eligible AEFIs

VL04.01

VL04.02

I-1-15 At which level is the current case definitions for

AEFI reporting disseminated?


• National level

• Sub-national level

• Health facility level

• Do not have list of eligible AEFIs

VL04.01

VL04.02

I-1-16 Which type of reporting tool do you use? Select all that apply. If “YES” to any of the

below, please attach a sample.

a) Line-listing of AEFI cases b) Case-based reporting c) Aggregate reporting

VL04.01

Page 224




d) Other (specify type):

I-1-17 Are the reporting tools being used, standardized

for the country

YES, NO. VL04.01

I-1-18 If “YES” for I-1-16 (a) and/or (b), please indicate

whether the following minimum information are

collected.


(a) In Line-listing of AEFI cases

• Event • Place of the event • Patient • Vaccine • Reporter (b) In Case-based reporting

• Event • Place of the event • Patient • Vaccine • Reporter

VL04.01

I-1-19 Do you have a specified time frame for reporting

serious AEFIs?

YES, NO.

If YES, specify the timeframe:

• 24-48 hr • # of days

VL04.01

VL04.02

VL05.02

I-1-20 Do you have a specified time frame for reporting

non-serious AEFIs?

YES, NO.

If YES, please specify the timeframe: e.g.

• # of days • # of weeks • # of months

VL04.01

VL04.02

VL05.02

I-1-21 What is the proportion of AEFI reported within

the expected timelines in previous year?

% of AEFI reports within the timelines

• Serious AEFI • Non-serious AEFI

PE.VL.04

VL05.02

I-1-22 What is the proportion of AEFI reports fully

completed in previous year ?

% of AEFI reports fully completed (= no

missing data)

VL05.02

I-1-23 Do you receive AEFI reports from the private

sector?

YES, NO

I-1-24 At which level(s) is data coding/entry

performed?


• At National level • At sub-national level

VL03.02


Page 225



I-1-25 Are AEFI reports forwarded from EPI/AEFI system

to the NRA/pharmacovigilance center?

YES, NO PE.VL.06

I-1-26 Are AEFI reports forwarded from

NRA/pharmacovigilance center to the EPI/AEFI

system?

YES, NO PE.VL.06

I-1-27 Summary of AEFI data for last year:

Are AEFI rates (serious, non-serious) consistent

with expected rates ?

Provide summary statistics available on AEFI

data reported at national level during last

year

YES, NO

VL05.02

I-1-28 Among the AEFI reports submitted to the

national level, which one are shared with WHO

UMC ?

Please specify

• All AEFIs (A), • Only Serious AEFI (S) • Other (O)

PE.VL.06

VL05.02

Overall evaluation of the DETECTION, REPORTING AND DATA MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory


CASE INVESTIGATION AND ANALYSIS

I-1-29 Do you have written standard procedures for

case investigation?

YES, NO. If Yes, please provide document VL04.02

I-1-30 If Yes to question 29, at what level have they

been disseminated ?


• National level • Sub-national level • Health facility level

I-1-31 Do you have case investigation forms? YES, NO.

If available, please provide form

VL04.02

I-1-32 How many AEFI cases have been investigated in

last year?

Please provide number of AEFI cases

investigated in the last year

VL05.02

I-1-33 Is there a monitoring of peripheral (sub- national

and health facility) levels to determine whether

YES, NO.

If yes, please specify

VL02.01

Page 226




AEFI cases were reported and investigated

according to National policy?

I-1-34 What proportion of AEFI case investigations

started within 48 hours following reporting in the

last year?

% of cases investigated within 48hours VL05.02

I-1-35 What proportion of preliminary investigation

reports was available within 1 week from the

start of investigation in last year

% of preliminary investigation reports

available within 1 week

VL05.02

I-1-36 What are the expected timelines for AEFI

investigation reports?

Select the correct one

• <6 weeks • 6-12 weeks • >12 weeks

VL05.02

I-1-37 What is the proportion of AEFI investigation

reports available within the expected timelines ?

% of AEFI investigation report within the

timelines

VL05.02

I-1-38 Do you have access to appropriate resources to

conduct AEFI investigation ?


been addressed by the GBT or PE indicators. VL03.01

VL03.02

Please refer to the related GBT or

PE indicator.

I-1-39 Of the AEFI investigation conclusions available,

what proportion is supported by findings?

For each kind of "finding" indicate an

estimated proportion of <10%, 10 to <25%,

25 to <50%, 50 to <75% OR >=75%

• Lab findings (positive or negative) on clinical specimen(s)

• Postmortem findings (among AEFI deaths)

• Lab findings (positive or negative) for vaccine samples

VL05.02

I-1-40 Do you have any of the following summary

(analysis) reports of AEFIs?


• Monthly or quarterly summary reports • Annual summary reports • Other (specify type)

I-1-41 If YES for any type of summary reports at the

previous question, then specify at which level(s)

such summary reports are prepared.


Monthly or quarterly summary reports



Page 227



Annual summary reports National level

• Sub-national level • Health facility level Other (specify type)_______


Overall evaluation of the CASE INVESTIGATION AND ANALYSIS


☐ Satisfactory

☐ Unsatisfactory


RISK ASSESSMENT AND MANAGEMENT

I-1-42 Do you have a national vaccine safety

committee(s) for:


• AEFI case investigation • AEFI causality assessment • Both • Neither

VL04.06

PE.VL.03

I-1-43 Do you have written procedures and criteria for

the selection of members of the national vaccine

safety committee(s)?

YES, NO

if YES please attach document

PE.VL.03

I-1-44 Are confidentiality and conflicts of interest

appropriately regulated within the national

vaccine safety committee(s) ?

YES, NO

Please specify

PE.VL.03

I-1-45 Do you have documents that clearly define the

roles and responsibilities of the national vaccine

safety committee(s) members?

YES, NO

If YES please attach document (TOR of

national vaccine safety committee(s))

VL04.06

PE.VL.03

I-1-46 Do you have documented evidence (meeting

reports) of regular meetings of national

immunization safety committee(s)?

YES, NO VL04.06

PE.VL.03

I-1-47 Do you use WHO classification of AEFI type

(vaccine product related reaction, vaccine quality

YES, NO. State if done and at what level of

reporting

VL04.02

Page 228




defect, immunization error, immunization

anxiety reaction, coincidental event)?

system

• At National level • At sub-national level • At health facility level

I-1-48 If NO to 47, is there another system you use for

causality classification for AEFIs?

YES, NO.

If YES (another system used for causality

classification for AEFIs), give the name or

reference for the system used.

I-1-49 If initial causality categorization is done for at

least some of the AEFI case reports at sub-

national level (or below), do you have a routine

system for review and validation or final

categorization?


• National committee • At National level (e.g., AEFI focal point)

I-1-50 In case of serious vaccine related AEFI detected

in the past three years, were regulatory decision

taken according to NRA guideline (suspension,

recall, update of product leaflet…)

YES, NO.

If yes, please specify action taken

Overall evaluation of the RISK ASSESSMENT AND MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory


INFORMATION, EDUCATION AND COMMUNICATION (IEC) WITH CONCERNED GROUPS

I-1-51 Do you have any document(s) that provides

guidance on establishment of a communication

system or communication plan relevant to

vaccine safety/AEFIs?

YES, NO.

If YES, specify type of document and the

level(s) (e.g., national) to which it applies.

Please attach the document.

VL02.02

I-1-52 Do you have a communication unit at national

level responsible for communication with

concerned groups on vaccine safety/AEFIs?

YES, NO. Please specify VL02.01


Page 229



I-1-53 Do you have a designated spokesperson for

media enquiries relevant to vaccine safety or

AEFI?

YES, NO. If yes, name, affiliation. VL02.01

I-1-54 Do you have a written communication plan in

case of vaccine safety crisis?

YES, NO.

Please specify

VL02.01

I-1-55 Does your organization regularly check the local,

including social, media for reports of adverse

events?

YES, NO.

I-1-56 Do you have information material/leaflets

relevant to vaccine safety/AEFI issues developed

for community, vaccinees and parents?

YES, NO. Specify

• Community • Vaccinees and parents

Please make sure to request for and check

the materials, if any!

VL06.01

I-1-57 Do you provide/share information relevant to

vaccine safety/AEFI to the private sector?

YES, NO. VL06.01

I-1-58 How often do you share AEFI investigation

outcomes with concerned groups

Please indicate corresponding figure: Almost

never=1; Occasionally=2; Often=3; Almost

always=4

• AEFI reporters • Immunization staff/Other health care

providers • Parents/Vaccinees/Community • Media

VL06.01

I-1-59 Please describe any vaccine safety crisis that

recently occurred; use the checklist in next

column as a guide to elements to include in your

brief description.

• what specific AEFI or vaccine safety issue it involved

• date when it occurred • how promptly the situation was

handled (timing of initial response) • whether you had a focal point or unit

for communication • how promptly you responded to the

community and AEFI reporters,

Page 230




• if an investigation was conducted and how long it took to complete the investigation

• what was the impact of this incident on your immunization program (vaccine acceptance and/or coverage, resources and staff, other)

Overall evaluation of the INFORMATION, EDUCATION AND COMMUNICATION WITH CONCERNED GROUPS


☐ Satisfactory

☐ Unsatisfactory


HUMAN AND FINANCIAL RESOURCES

I-1-60 Is there a budget component specific for the AEFI

surveillance system available?


• At national level • At sub-national level • At health facility level

I-1-61 Is there a specific budget line for AEFI case

management (treatment of the person with

suspected AEFI)?

For routine immunization

For immunization campaign

If Yes, Specify:

• Name of document: • Service where document can be found

I-1-62 Do you have pre-assigned investigation team(s)

responsible for AEFI investigation when needed?


• At national level • At sub-national level If YES briefly describe the team composition

(e.g., pediatrician, epidemiologist,

Immunization

supervisor etc.) of the persons in the pre-

assigned team (s)

VL03.01

I-1-63 What percent (%) of staff involved in AEFI

surveillance (reporting, investigating or

For each level, indicate an estimated

proportion of <10%, 10 to <25%, 25 to

<50%, 50 to <75% OR >=75%

VL03.03

PE.VL.02


Page 231



managing cases) have attended training relevant

to AEFI/vaccine safety last year?


I-1-64 On an average, what proportion of

trainees/participants in all training activities

relevant to AEFI/Vaccine safety conducted last

year have been staff from the private

sector (physicians and other health care

workers)?

Numerator = number of staff from private

sector attended the training, denominator =

total number of

participants attended the training)

VL03.03

VL03.04

PE.VL.02

I-1-65 Is there a document where information on

vaccine safety trainings is reported (including

number of participants, course

description/agenda)?

YES, NO

Specify:

• Name of document: • Training plan • Training report • Other, specify Service where document can be found

PE.VL.02

I-1-66 Which type of training relevant to AEFI has been

provided in the last year?

Please describe VL03.03

I-1-67 Is updated information (including training

materials) on AEFI detection and reporting

procedure provided to health staff at all levels?



PE.VL.02

Overall evaluation of the HUMAN AND FINANCIAL RESOURCES


☐ Satisfactory

☐ Unsatisfactory


VACCINE UTILIZATION

I-1-68 Current routine childhood immunization

schedule.

(please provide list, table or PPT slide)

I-1-69 List of vaccines used in EPI program in your

country.

Provide list of vaccines currently used in EPI

programme

Page 232




I-1-70 Do you receive information on total # of doses

distributed?



I-1-71 Do you receive information on lot/batch # of

doses distributed?



I-1-72 Do you receive information on total # of doses

administered?



I-1-73 Do you receive information on lot/batch # of

doses administered?



Overall evaluation of the VACCINE UTILIZATION

The WHO eeam conclude that the assessed areas are: (please tick one of the below checkboxes)

☐ Satisfactory

☐ Unsatisfactory


Overall evaluation of the vigilance system at the national/central level


☐ Satisfactory

☐ Unsatisfactory


3549


Page 233


Section 2: assessment at the sub-national level 3551

This section is targeting the assessment of the performance of vaccine vigilance system at the sub-national levels, namely: 3552

1) Regional regulatory authorities (e.g., at state or provincial levels), and 3553

2) Regional Immunization Programme. 3554

3555

Guidance: 3556

• Identify critical issues to be assessed during the data collection process (from the information collected at the national level, 3557

background documents provided and the “informal” information gathered). 3558

• Don’t use the Discussion Guide as a questionnaire, or attempt to ask all the questions listed as discussion points. 3559

• Instead, try to focus on the critical issues the team agreed after the data was collected at national and health facilities levels. 3560

• You’ll have more success obtaining information if you try to establish an open dialogue with health staff and stakeholders and 3561

observe them while they are working. 3562

• If necessary, this section can be repeated in case of visiting several institutions at this level. If so, please clearly indicate the visited 3563

site/facility and its pertinent information. 3564

3565

ID Question Guidance and value range Comment and justification

General information ➢ Institution(s) assessed: ➢ Persons met and interviewed:


I-2-01 Do you have contact information of designated national

focal point for vaccine AEFI ?

YES, NO.


➢

I-2-02 Are you aware of the written National AEFI surveillance guidelines?

YES, NO.

I-2-03 Have these guidelines been communicated to your staff? YES, NO.

I-2-04 Interview some staff using the respective guidance and

ask if they have read the guidelines and assess their

knowledge of the contents:

Questions to guide the discussion • What is an AEFIs ?

Page 234



• Do you have a list of AEFIs eligible for reporting?

• Do you have AEFI case definitions for expected vaccine reactions?

• How is the reporting done to this level, what forms are used ?

• How is the reporting from this level done? To whom? Routinely – nil reports? What frequency?

• Communication mechanism (phone, fax, email? )

• What is the timeframe for reporting cases? • Do they know the local drug inspector (s)/

NRA officials? • Were they involved in AEFI investigation for

previous AEFIs • In the last year, was there any joint NRA EPI

meetings /trainings? • Is there an AEFI committee at this level? Is

this functional? Who are the members? How frequently does this committee meet? How do you support the AEFI committee ?


I-2-05 • Review AEFI reports received from health facilities and investigations in the last year. Review numbers of AEFI reports received in the last year and compare to the number of reports received in the year before the last one.

• Estimate the rate of AEFIs reported by comparing AEFIs with the number of doses of vaccine administered.

• Look at the AEFI reports and data management process.

• Look at the AEFI reports submitted to national level.

Questions to guide the discussion • Which AEFI (serious/non serious) are

reported from operational level and how (forms, communication mechanism, frequency, timelines)?

• How do you decide which cases should be reported as AEFI cases ?

• Do you have a list of AEFIs eligible for reporting?

• Do you manage AEFI cases not reported to supervisor? If YES, do you refer to those when you find similar case?

• Do you have AEFI case definitions for expected vaccine reactions? Ask what are


Page 235


the expected vaccine reactions for specific vaccines

• Do you compile, analyze and interpret AEFI data you receive on a regular basis? How often?

• Do you have procedure for analyzing the data?

• How do you decide which AEFI cases should be investigated?

• Which AEFI are communicated to the national level ? to whom? (EPI? PV centre?)

• Where do you register the AEFIs? Do you have a database or repository? Do you have designated personnel/data manager for data entry?

• How do you proceed with reporting to national level?

• Which forms/mechanism do you used? can you please show me those forms?

• To who do you report, and when? • How do you send AEFI reports:

electronically, hardcopy? • Is AEFIs reporting included into routine

immunization reports to national level?

• Have you ever received feedback from your supervisor/national level? How often do you receive feedback ?

• If you received request to fill missing information to AEFI case, which you report to your supervisor, do you respond? If YES, how often?

CASE INVESTIGATION AND CAUSALITY ASSESSMENT

I-2-06 Review the availability of SOP for case investigation Questions to guide the discussion • Please describe what do you do when you

receive a serious AEFI report from health facility?

Page 236



• Do you have written procedure for investigating and actions to be taken in case of serious AEFI or cluster events?

• Who conducts the investigation? • Do you have standard form for

investigation? Do you have SOP for specimen collection? Forms associated?

• Assess whether the different type of AEFIs are known (vaccine reaction, vaccine quality defect, immunization error…)?

• In the last year, how many AEFI have you investigated personally? What were the outcomes? Was there any impact on the program?

• Are you familiar with the Brighton collaboration definition?

• Who joins you for AEFI investigations? • How frequently do you conduct discussion

of the results of investigated cases among your staffs?

• Have you ever conduct cross checking of investigation results among investigators for consistency of investigation?

• If you find missing information in the reported AEFI, do you request reporter to provide such information? If YES, how often? Do they respond to your request?

INFORMATION, EDUCATION AND COMMUNICATION (IEC) WITH CONCERNED GROUPS (AEFI reporter [person who reported AEFI, not only health care provider], parents, vaccinees, public, community, immunization staff, other health care providers, AEFI case, investigators, media etc..)

I-2-07 Review IEC materials, including training materials (slides,

booklets, SOPs), posters, leaflets.

Questions to guide the discussion • Have you conducted training for AEFI

investigation in the last year? How many? For whom? When? Can you please show me some of training materials?

• Has AEFI reporting improved after training? • Do health workers feel comfortable

reporting program errors? Are they


Page 237


confident that they will not be blamed by the department?

• Do parents / public report minor AEFI (e.g. fever/pain) first to the staff who vaccinated or to the medical officer?

• Do you have information material/leaflets relevant to vaccines and AEFI to communicate to health care workers?

• In case of previous serious AEFI, were the results of the investigation shared with the vaccinee/parents/community? How ? by who? How long after the event?

• Do you receive regular information on vaccine safety and AEFI (newsletter, epidemiological bulletin…)? Do you share those information with health care workers ?

• Do you actively collect vigilance information? If yes, please specify what kind of information and how do you collect.

• If you think correction of vigilance information distributed by MoH, EPI deemed necessary, do you provide your feedback to the source of the information? What frequency?

•


I-2-08 Staffing:

Review staffing list for the facility and qualification

Questions to guide the discussion • Ask staff if there are enough of the right kind

of staff in the facility. If not, ask them to give you details.

• How many posts are now vacant in the health facility?

• In the last year, have you solicited the assistance of your supervisors/next level for AEFI investigations?

I-2-09 Training: Questions to guide the discussion

Page 238



WHO team should review …. • Information material on AEFI detection, reporting

and management • Training material and certificate

• Have you ever attended a training on AEFI ? if yes, which type of training, when was it ?

• Is updated information (including training materials) on AEFI detection and reporting procedure provided?

• Do you maintain training record of your staff and if they were not attended regularly (at least once a year), do you encourage them to attend?

• Do you organize regular training on AEFI for health care workers ? If YES, could we look some of training materials?

•

I-2-10 Supervision : health workers’ performance is regularly

evaluated and feedback provided

WHO team should review ….

• health worker performance ’s reports. • supervisor’s reports.

Questions to guide the discussion

• Ask the district officer/medical officer to tell you who has visited them from the national level. How often do they visit? What do the visitors do while they are in the facility?

• Ask whether she/he conducts regular review / observation of health workers performance, how? how often ?

Overall evaluation of the vigilance system at the sub-national level


☐ Satisfactory

☐ Unsatisfactory


3566


Page 239


Section 3: assessment at the health facility level 3568

This section is targeting the assessment of the performance of vaccine vigilance system at the sub-national levels, namely: 3569

1) Immunization centre 3570

2) Points of care (POC) 3571

3572

Guidance: 3573

• Identify critical issues to be assessed during the data collection process (from the information collected at the national and sub-3574

national levels, background documents provided and the “informal” information gathered). 3575

• Don’t use the Discussion Guide as a questionnaire, or attempt to ask all the questions listed as discussion points. 3576

• Instead, try to focus on the critical issues the team agreed after the data was collected at national and health facilities levels. 3577

• You’ll have more success obtaining information if you try to establish an open dialogue with health staff and stakeholders and 3578

observe them while they are working. 3579

• If necessary, this section can be repeated in case of visiting several institutions at this level. If so, please clearly indicate the visited 3580

site/facility and its pertinent information. 3581

3582




I-3-01 Do you have contact information of designated National

focal point for vaccine AEFI ?

YES, NO.


➢

I-3-02 Are you aware of the written national AEFI surveillance guidelines?

YES, NO.

I-3-03 Have these guidelines been communicated to your staff? YES, No.

I-3-04 Interview some staff using the respective guidance and

ask if they have read the guidelines and assess their

knowledge of the contents:

Questions to guide the discussion • What is an AEFIs?

Page 240



• Do you have a list of AEFIs eligible for

reporting?

• Do you have AEFI case definitions for expected

vaccine reactions?

• How is the reporting done to this level, what

forms are used ?

• How is the reporting from this level done? To

whom? Routinely – nil reports? What

frequency?

• Communication mechanism (phone, fax,

email?)

• What is the timeframe for reporting cases?

• Do they know the local drug inspector(s)/NRA

officials?

• Were they involved in AEFI investigation for

previous AEFIs?

• In the last year, was there any joint NRA EPI

meetings/trainings?

• Is there an AEFI committee at this level? Is this functional? Who are the members? How frequently does this committee meet? How do you support the AEFI committee ?

• Do you receive regular information on vaccine safety and AEFI (newsletter, epidemiological bulletin…)? Do you share that information with health care workers?

• Do you actively collect vigilance information issued by MoH, EPI? If YES, please specify what kind of information and how do you collect.

DETECTION AND MANAGEMENT

I-3-05 • Review AEFI reports received from health facilities

and investigations in the last year. Review numbers

of AEFI reports received in the last year and compare


• Have you ever had AEFI at your health facility?


Page 241


to the number of reports received in the year before

last one.

• Estimate the rate of AEFIs reported by comparing

AEFIs with the number of doses of vaccine

administered.

• If yes, do you know what to do to help the patient with AEFI at the first minutes, when to call for emergency?

• Do you have emergency kit? Can you please show me the kit? Have you been trained on how to use this?

• Before each session, do you inform vaccinees/parents about possible adverse reaction after immunization?

• How do you decide which cases should be reported as AEFI cases ?

• Do you have a list of AEFIs that should be reported?

• Do you have AEFI case definitions for

expected vaccine reactions? Ask health

workers what are the expected vaccine

reactions for specific vaccines

AEFI REPORTING

I-3-06 WHO team should look at and review: • the periodic reports (routine reports) sent from the

institution • the AEFI reports sent and check the time lines and

completeness, compare consistency with the onsite logbook/registry.


• Have you ever reported an AEFI? • Where do you register the AEFIs? Do you

have log book, can I see it? • How do you proceed with reporting? • Which forms do you used? Can you show

me those forms? • To whom do you report, and when? • How do you send AEFI reports:

electronically, hardcopy? • Ask and check if AEFI reports are submitted

on time. If not, why? • Do you include AEFIs reports into routine

immunization reports to higher supervisory level?

• If you ever reported AEFI case, have you

received feedback from your

Page 242



supervisor(positive/negative)? How often do

you receive feedback?

• If you received request to fill missing

information to AEFI case, which you report to

your supervisor, do you respond? If YES, how

often?

INFORMATION, EDUCATION AND COMMUNICATION (IEC) WITH CONCERNED GROUPS (AEFI reporter [person who reported AEFI, not only health care provider], parents, vaccinees, public, community, immunization staff, other health care providers, AEFI case, investigators, media etc..)

I-3-07 Review IEC materials, including training materials (slides,

booklets, SOPs), posters, leaflets.

Questions to guide the discussion • Have you conducted training for AEFI

investigation in the last year? How many? For

whom? When? Can I see some of training

materials?

• Has AEFI reporting improved after training?

• Do health workers feel comfortable reporting

program errors? Are they confident that they

will not be blamed by the department?

• Do parents / public report minor AEFI (e.g.

fever/pain) first to the staff who vaccinated or

to the medical officer?

• Do you have information material/leaflets

relevant to vaccines and AEFI to communicate

to health care workers?

• In case of previous serious AEFI, were the

results of the investigation shared with the

vaccinee/parents/community? How? By who?

How long after the event?

• Do you receive regular information on vaccine

safety and AEFI (newsletter, epidemiological

bulletin…)? Do you share those information

with health care workers?

• Are there any anti vaccination groups

communicating concerns of AEFI?


I-3-08 Staffing: Questions to guide the discussion


Page 243


Review staffing list for the facility and qualification • Ask staff if there are enough of the right kind of staff in the facility. If not, ask them to give you details.

• How many posts are now vacant in the health

facility?

I-3-09 Training:


• Information material on AEFI detection, reporting

and management

• Training material and certificate

Questions to guide the discussion • Have you ever attended to a training on AEFI ?

if yes, which type of training, when was it? • Is updated information (including training

materials) on AEFI detection and reporting procedure provided?

• Do you maintain training record of your staff and if they were not attended regularly (at least once a year), do you encourage them to attend?

• Have you been a resource person in trainings?

I-3-10 Supervision:


• supervisor’s reports.


• Ask health workers to tell you who has visited

them from the district/regional office. How

often do they visit? What do the visitors do

while they are in the facility?

Overall evaluation of the vigilance system at the health facility level


☐ Satisfactory

☐ Unsatisfactory


3583


Page 244

Part II: Questionnaire for assessment of the performance of medicine vigilance systems 3584

Section 1: assessment at the national level 3585

This section is targeting the assessment of the performance of medical product vigilance 3586

system at the national levels, namely: 3587

1. National Regulatory Authority/National Vigilance Center (please note that 3588

majority of the assessment of the National Regulatory Authority/National 3589

Vigilance Center is covered by the global benchmarking tool as well as the 3590

performance evaluation indicators). 3591

2. Central Health Programme (e.g., HIV, NCDs, Malaria, TB, Tropical diseases, etc…). 3592

3593

ID Question Value range Guidance

Related

GBT or PE

indicators

Comment and

justification

General information

➢ Institution(s) assessed:

➢ Persons met and interviewed:


II-

1-

01

Do you have a

designated national

focal point for medical

product vigilance?

YES, NO.

If YES, provide TOR

and contact

information

• At NRA

• At PHP

• At MoH

Identifying the

post with the

ultimate

responsibility for

the national

medical product

vigilance is

essential

VL02.01

II-

1-

02

Are guidelines for

medical product

vigilance included

within the strategic

and/or annual

operational plans of

your public health

program?

No need to

address this

question as it has

been addressed by

the GBT or PE

indicators.

No need to

address this

question as it has

been addressed

by the GBT or PE

indicators.

VL01.06

VL02.02

II-

1-

03

Have these guidelines

for medical product

vigilance been

communicated to staff

at all levels?

Select all that

apply

• National level

• At sub-

national level

• At health

facility level

Availability of

vigilance

guidelines

throughout the

organization to

be assured

VL03.02

II-

1-

04

Do your PHP and NRA

collaborate regularly

to review medical

product safety issues?

Select all that

apply

• Notifying each

other on

medical

Regular and close

collaboration

between PHP and

NRA/VL function

is essential

VL02.02


Page 245


Related

GBT or PE

indicators

Comment and

justification

product safety

issues,

• Sharing

medical

product ICSRs,

• Convening

regular

meeting

between the

institutions,

being involved

or

coordinating

analysis of

data,

• Sharing report

analysis or

summaries,

• Jointly

participating

in national

medical

product

vigilance

advisory

committee

reviews,

• Other - please

specify.

II-

1-

05

Do you have a national

system for collating,

managing and

retrieving reports of

suspected adverse

reactions to medical

products?

No need to

address this

question as it has

been addressed by

the GBT or PE

indicators.

No need to

address this

question as it has

been addressed

by the GBT or PE

indicators.

VL04.01

VL04.02

Overall evaluation of SYSTEMS, STRUCTURE AND STAKEHOLDER COORDINATION


☐ Satisfactory

☐ Unsatisfactory



II-1-

06

Do you have written

procedures on actions to

be taken in case of

YES, NO An action plan for

crisis management

should be in place

VL04.01

VL04.02

Page 246


serious medical product

related safety concerns

e.g., SOP for reporting

and case management?

If YES, please

provide

document

II-1-

07

Which reporting tool do

you use for ICSRs of

medical products?

Specify if

different from

tool used by

national vigilance

centre

VL04.01

II-1-

08

Is the reporting tool being

used, standardized for

the country?

YES, NO

If NO record

justification

VL04.01

II-1-

09

At which level(s) is data

coding/entry performed?

Select all that

apply

• At national

level

• At sub-

national

level

VL04.02

II-

1_10

Are all reported ICSRs

forwarded from the PHP

system to the

NRA/vigilance center?

YES, NO If NO, provide justification YES, NO If NO, provide

justification

Important to

establish that all

reports meeting the

minimum criteria

for completeness

are shared with the

NRA/VL centre. No

potentially

embarrassing cases

should be hidden.

PE.VL.06

II-1-

11

Are summary rates of

ICSRs last year consistent

with expected rates?

YES, NO Reasons for large

annual variations

should be

investigated

VL05.02

Overall evaluation of DETECTION, REPORTING AND DATA MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory


CASE INVESTIGATION AND ANALYSIS

II-1-

12

What is the total number

of patients receiving

medicines in the PHP

who were reported to

experience medical

product-related adverse

events the last year?

Record reporting

statistics if

available

Absolute number

of ICSRs divided by

number of patients

treated provides an

estimate of

attention paid to

safety surveillance

and the

PE.VL.04


Page 247

functionality of the

vigilance system.

II-1-13

How many active medical product safety surveillance studies have been conducted in the last three years (36 months) in your PHP?

Indicate type of study (e.g. cohort event monitoring, targeted spontaneous reporting, etc.) and stage of completion (e.g. initiated, on-going or completed) for each study

Engagement in active safety surveillance indicates ambitions to learn about mechanisms and risk factors, enabling future prevention

VL04.08 PE.VL.07

Overall evaluation of CASE INVESTIGATION AND ANALYSIS


☐ Satisfactory

☐ Unsatisfactory



II-1-14

Does your PHP have representation in the national vigilance advisory committee?

Select all that apply • For ICSR

causality assessment

• ICSR signal investigation

• other

If YES, strengthens II-1-04 and documents a coherent vigilance system

VL04.06 PE.VL.03

II-1-15

Have any medical product related problem, detected in the past three years in your PHP resulted in a regulatory decision by the NRA (suspension, recall, update of product leaflet…)?

YES, NO. If yes, please specify action taken

If YES, supports impression of a functional and coherent national vigilance system. If NO, can be due to lack of actual safety concerns but also due to lack of communication.

VL04.03 PE VL.09

II-1-16

How many medicine safety issues identified from outside sources were acted on at national level in the previous year?

Outside sources refer to literature data or information from other countries

Important for patient safety to be alert to new and relevant international data. Lack of identified such issues does not prove failure.

PE VL09Record

II-1-17

What is the number of suspected product quality problems detected through the PHP in the previous year?

Record statistics if available

If the vigilance system is considered to be an important component in the national combat against sub-

PE.VL.08

Page 248


standard and falsified medicines this question should be documented carefully, otherwise it is not critical

Overall evaluation of RISK ASSESSMENT AND MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory



II-1-18

Do you have any document(s) that provides guidance on establishment of a communication system or communication plan relevant to safety of medical products used in your program?

YES, NO. If YES, specify type of document and the level(s) (e.g., national) to which it applies. Please attach the document.

The availability of a communication system and plan for medical product safety is essential

VL02.02 VL06.02: PE.VL.01

II-1-19

Do you have a communication unit at National level responsible for communication with concerned groups on safety of medical products used in your program?

YES, NO. Please specify

Identification of the responsible office or manager for communication of medical product safety issues is required

VL02.01 VL06.02 PE VL 01

II-1-20

Do you have a designated spokesperson for media enquiries relevant to the safety of medical products used in your program?

YES, NO. If yes, name, affiliation.

A spokesperson for media questions should be identified

VL02.01 VL06.02 PE VL01

II-1-21

Do you have a written communication plan in case of a safety crisis related to medical products used in your program?

YES,NO If YES, specify the level(s) (e.g., national) to which it applies. Please attach the document.

A crisis communication plan should be developed jointly between the PHP and the NRA

VL02.01 PE VL01

II-1-22

Do you have information material/website, free telephone line etc. by which relevant safety information of medical products used in your program is made available to the community?

YES, NO. Specify • Community • Children and

parents

An information service should be available for the community, preferably developed in collaboration with the NRA/VL function

VL06.01 PE VL01

II-1-23

How many public or community education

Specify method of training and

Follow-on question to II-1-22

VL 02.02 PE.VL.02


Page 249

activities relating to medical product safety were carried out by the PHP in the previous year?

number of activities

II-1-24

How many requests for information about medical product safety were received in the previous year? How many were addressed?

Provide communication channels and numbers if available

Not critical if statistics is not available

VL 02.02 VL 06.01

II-1-25

How long does it take from when a medical product safety signal or significant safety issue is identified to when it is communicated to health workers and the public?

Provide time estimate in number of days

The efficiency of the regulatory system in terms of giving priority to actions to protect patients at risk is an important indicator to record.

VL04.03 PE VL.09

II-1-26

Are pharmacovigilance data being considered when updating standard treatment guidelines for your PHP?

Explain frequency and process of guideline update

The main justification for vigilance activities is to improve future practices. The use of vigilance data to achieve this needs to be documented.

Vl 05.01 VL.06.02

Overall evaluation of INFORMATION, EDUCATION AND COMMUNICATION WITH CONCERNED GROUPS


☐ Satisfactory

☐ Unsatisfactory



II-1-27

Is there an annual budget component specific for vigilance of medical products used in your program?

Specify public and donor funding

II-1-28

Is there a specific budget line for case management of patients affected by adverse effects of medical products used in your program?

YES or NO

II-1-29

Do you have pre-assigned investigation team(s) responsible for investigation of suspected medical product related adverse reactions when needed?

YES or NO Select all that apply • At national

level • At sub-

national level

VL03.01

II-1-30

What percentage (%) of staff involved in patient management component of your program have attended

For each level, indicate an estimated proportion of

Maintenance of system for continuous competence

VL03.03 PE.VL.02

Page 250


training relevant to safety surveillance of medical products last year?

<10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75% • At National

level • At sub-

national level • At Health

facility level

development in safety surveillance is critical for the long-term operation

II-1-31

Is there a document where information on medical product safety surveillance training is reported (including number of participants, course description/agenda)?

YES, NO Specify: • Name of

document: • Training plan • Training

report • Other, specify

Documentation of safety surveillance training on an individual level should be required

PE.VL.02

II-1-32

Which type of training relevant to medical product vigilance has been provided in the last year?

Please describe Evidence of recent performance in competence development to be provided

VL03.03

II-1-33

Is updated information (including training materials) on medical product safety surveillance, including detection and reporting procedures, provided to health staff at all levels?

Select all that apply • At national

level • At sub-

national level • At health

facility level

Implementation and follow-on from II-1-03

PE.VL.02

Overall evaluation of HUMAN AND FINANCIAL RESOURCES


☐ Satisfactory

☐ Unsatisfactory


MEDICAL PRODUCT UTILIZATION

II-1-34

List of medical products currently used in your PHP in your country

Provide list of medical products currently used in the Public Health Program

Questions II-1-34 to II-1-39 do not strictly refer to collection of patient safety information. However, statistics on utilization of medical products in the program and availability of batch/lot numbers may be invaluable in the analysis of

II-1-35

Do you receive information on total number of doses distributed annually?


level • At sub-


facility level


Page 251

II-1-36

Do you receive information on lot/batch number of doses distributed?


level • At sub-


facility level

the magnitude of risk to patients and the influence of sub-standard medical products. Collection of such information should therefore be considered as essential for the proper analysis of medical product safety information

II-1-37

Do you receive information on total number of doses administered?


level • At sub-


facility level

II-1-38

Do you receive information on lot/batch numbers of doses administered?


level • At sub-


facility level

II-1-39

What is the total number of patients receiving medicines under the PHP annually?

Record number if available

Overall evaluation of MEDICAL PRODUCT UTILIZATION


☐ Satisfactory

☐ Unsatisfactory


Overall evaluation of the vigilance system at the national level


☐ Satisfactory

☐ Unsatisfactory


3594

3595

Page 252



Section 2: assessment at the sub-national level 3597

This section is targeting the assessment of the performance of medicine vigilance system at 3598

the sub-national levels, namely: 3599

1) Regional Regulatory bodies (e.g., at state or provincial levels), and 3600

2) Regional Health Programme (e.g., HIV, NCDs, Malaria, TB, Tropical diseases, etc…) 3601

3602


Related

GBT or PE

indicators

Comment and

justification

General information

➢ Institution(s) assessed:

➢ Persons met and interviewed:


II-

2-

01

Do you have contact

information of

designated National

focal point for medical

product vigilance?

YES, NO.

If YES, provide TOR

and contact

information

• At NRA

• At PHP

• At MoH

Identifying the

post with the

ultimate

responsibility for

the national

medical product

vigilance in the

PHP is essential

VL02.01

II-

1-

02

Are guidelines for

medical product

vigilance included

within the strategic

and/or annual

operational plans of

your public health

program?

No need to

address this

question as it has

been addressed by

the GBT or PE

indicators

No need to

address this

question as it has

been addressed

by the GBT or PE

indicators

VL01.06

VL02.02

II-

1-

03

Have these guidelines

for medical product

vigilance been

communicated to staff

at all levels?

Select all that

apply

• National level

• At sub-

national level

• At health

facility level

Availability of

vigilance

guidelines

throughout the

organization to

be assured

VL03.02

II-

1-

04

Do your center and

NPVC collaborate

regularly to review

medical product safety

issues?

Select all that

apply

• Notifying each

other on

medical

product safety

issues

Regular and close

collaboration

between NPVC

and NRA/VL

function is

essential

VL02.02


Page 253


Related

GBT or PE

indicators

Comment and

justification

• Sharing

medical

product ICSRs,

• Convening

regular

meeting

• between the

institutions,

• being involved

or

• coordinating

analysis of

data;

• Sharing report

analysis or

• summaries,

• Jointly

participating

in

• national

medical

product

vigilance

advisory

committee

reviews,

• other- please

specify.

Overall evaluation of SYSTEMS, STRUCTURE AND STAKEHOLDER COORDINATION


☐ Satisfactory

☐ Unsatisfactory



II-1-

05

Do you have written

procedures on actions to

be taken in case of serious

medical product related

safety concerns e.g., SOP

for reporting and case

management?

YES, NO

If YES, please

provide

document

An action plan for

crisis management

should be in place

VL04.01

VL04.02

II-1-

06

Which reporting tool do

you use for ICSRs of

medical products?

Specify if

different from

tool used by

VL04.01

Page 254


national vigilance

centre

II-1-

07

At which level(s) is data

coding/entry performed?

Select all that

apply

• At National

level

• At sub-

national

level

VL04.02

II-1-

08

Are all reported ICSRs to

your center forwarded to

the NPVC?

YES, NO If NO, provide justification YES, NO If NO, provide

justification

Important to

establish that all

reports meeting the

minimum criteria

for completeness

are shared with

NPVC. No

potentially

embarrassing cases

should be hidden.

PE.VL.06

Overall evaluation of DETECTION, REPORTING AND DATA MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory



II-1-09

Does your center have representation in the national vigilance advisory committee?

Select all that apply • For ICSR

causality assessment

• ICSR signal investigation

• other

If YES, strengthens II-1-04 and documents a coherent vigilance system

VL04.06 PE.VL.03

II-1-10

Have any medical product related problem, detected in the past three years in your center resulted in a regulatory decision by the NRA (suspension, recall, update of product leaflet…)?

YES, NO. If yes, please specify action taken

If YES, supports impression of a functional and coherent national vigilance system. If NO, can be due to lack of actual safety concerns but also due to lack of communication.

VL04.03 PE VL.09

II-1-11

How many medicine safety issues identified from outside sources were acted on locally in the previous year?

Outside sources refer to literature data or information from other countries

Important for patient safety to be alert to new and relevant international data. Lack of identified

PE VL09Record


Page 255

such issues does not prove failure.

II-1-12

What is the number of suspected product quality problems detected through the PHP in the previous year?

Record statistics if available

If the vigilance system is considered to be an important component in the national combat against sub-standard and falsified medicines this question should be documented carefully, otherwise it is not critical

PE.VL.08

Overall evaluation of RISK ASSESSMENT AND MANAGEMENT


☐ Satisfactory

☐ Unsatisfactory



II-1-13

Do you have any document(s) that provides guidance on establishment of a communication system or communication plan relevant to safety of medical products used in your program?

YES, NO. If YES, specify type of document and the level(s) (e.g., national) to which it applies. Please attach the document.

The availability of a communication system and plan for medical product safety is essential

VL02.02 VL06.02: PE.VL.01

II-1-14

Do you have a communication unit responsible for communication with concerned groups on safety of medical products used in your program?

YES, NO. Please specify Identification of the responsible office or manager for communication of medical product safety issues is required

VL02.01 VL06.02 PE VL 01

II-1-15

Do you have a designated spokesperson for media enquiries relevant to the safety of medical products used in your program?

YES, NO. If yes, name, affiliation. A spokesperson for media questions should be identified

VL02.01 VL06.02 PE VL01

II-1-16

Do you have a written communication plan in case of a safety crisis related to

YES,NO If YES, specify the level(s) (e.g., national) to which it applies. Please attach the

A crisis communication plan should be developed jointly

VL02.01 PE VL01

Page 256


medical products used in your program?

document. between the PHP and the NRA

II-1-17

Do you have information material/website, free telephone line etc. by which relevant safety information of medical products used in your program is made available to the community?

YES, NO. Specify • Community • Children and parents

An information service should be available for the community, preferably developed in collaboration with the NRA/VL function.

VL06.01 PE VL01

II-1-18

How many public or community education activities relating to medical product safety were carried out by your center in the previous year?

Specify method of training and number of activities

Follow-on question to II-1-22

VL 02.02 PE.VL.02

II-1-19

How many requests for information about medical product safety were received in the previous year? How many were addressed?

Provide communication channels and numbers if available

Not critical if statistics is not available

VL 02.02 VL 06.01

II-3-20

How are you providing feed-back to internal individual reporters of medical product related case safety reports?

Specify all that apply e.g.,

• acknowledgement (electronic/paper/verbal, automatic or not)

• feedback with case assessment

• advise re. possible prevention

no feed-back

Identify mechanisms available to stimulate, acknowledge and give feedback to reporters including the result of the local causality assessment made.

VL04.02

Overall evaluation of INFORMATION, EDUCATION AND COMMUNICATION WITH CONCERNED GROUPS

The WHO t

eam conclude that the assessed areas are: (please tick one of the below checkboxes)

☐ Satisfactory

☐ Unsatisfactory



II-1-21

Do you have pre-assigned investigation team(s) responsible for investigation of suspected medical

YES or NO VL03.01


Page 257

product related adverse reactions when needed?

II-1-22

What percentage (%) of staff involved in patient management component of your center have attended training relevant to safety surveillance of medical products last year?

Indicate an estimated proportion of <10%, 10 to <25%, 25 to <50%, 50 to <75% OR >=75%

Maintenance of system for continuous competence development in safety surveillance is critical for the long-term operation

VL03.03 PE.VL.02

II-1-23

Is there a document where information on medical product safety surveillance training is reported (including number of participants, course description/agenda)?

YES, NO Specify: • Name of

document: • Training plan • Training

report • Other, specify

Documentation of safety surveillance training on an individual level should be required

PE.VL.02

II-1-24

Which type of training relevant to medical product vigilance has been provided in the last year?

Please describe Evidence of recent performance in competence development to be provided

VL03.03

Overall evaluation of HUMAN AND FINANCIAL RESOURCES


☐ Satisfactory

☐ Unsatisfactory


Overall evaluation of the vigilance system at the sub-national level


☐ Satisfactory

☐ Unsatisfactory


3603

3604

3605

3606

3607

Page 258



Section 3: assessment at the health facility level 3609

This section is targeting the assessment of the performance of medicine vigilance system at 3610

the health facility levels, namely: 3611

1) Hospitals, polyclinics or other points of care (POC) 3612

3613


Related

GBT or PE

indicator

s

Comment

and

justificatio

n



II-

3-

0

1

Does the health

facility have a

functional Drug

and

Therapeutics

Committee

(DTC) or

equivalent,

responsible for

vigilance

activities?

Provide TOR for DTC and minutes

from latest meetings

The governance and management structure for medical product vigilance in the facility needs to be established

VL02.01

II-

3-

0

2

Within the

previous year,

has the DTC

carried out any

vigilance

activities or

addressed

medicine safety

issues?

Specify

• kind of activity

• purpose of activity

• number of activities

The level of recent activity and engagement in vigilance activities to be established

VL02.01

II-

3-

0

3

Do you have

designated focal

point for

medical product

vigilance in the

health facility ?

YES, NO. If YES, provide TOR and contact information

The responsible person for medical product vigilance to be identified

VL02.01

II-

3-

0

4

Which are the

reporting lines

between the VL

focal point, the

DTC and the

Should be clear from TOR Reporting lines in the management structure for medical product vigilance in the

VL02.01 VL03.02


Page 259


Related

GBT or PE

indicator

s

Comment

and

justificatio

n

management of

the health

facility?

facility to be identified


II-

3-

0

5

By which

mechanisms are

suspected

medical product

related adverse

events or

problems

identified in

your health

facility?


• patient counseling and diagnosis

• laboratory and other test results,

• systematic chart reviews etc.

This is to identify all sources of reports of medical product adverse events within the facility

VL04.01

II-

3-

0

6

How are

suspected

medical product

related adverse

events reported

within your

health facility?


• paper forms

• web-based form

• SMS

• communication app

• patient record system etc.

Understanding the communication channels for medical product adverse events in the facility is essential and allows identification of possible gaps

VL04.01

II-

3-

0

7

Which kind of

medical product

related

problems are

reportable?

Specify all that apply: • suspected adverse effects • lack of effect • medical product quality

problem • medication errors

The coverage and scope of the internal vigilance system is investigated, allowing detection of omissions

VL04.01

II-

3-

0

8

Who are

entitled

(authorized) to

reports medical

product related

adverse events

in your health

facility?


• anybody

• assistant nurses

• dentists

• doctors

• nurses

• patients

• pharmacists

Allows identification of possible hierarchical hurdles in the sensitivity of the vigilance system if certain categories are kept from reporting directly

VL04.01

II-

3-

How many

medical product

Provide statistics, if available, preferably specified by category:

The absolute numbers of

VL04.01 PE VL04

Page 260



Related

GBT or PE

indicator

s

Comment

and

justificatio

n

0

9

adverse event

reports were

recorded from

the health

center in the

previous year?

1. adverse effect (pharmacological/biological)

2. use-related events 3. quality related effects

reported adverse events provide a certain indication of the level of attention paid to vigilance activities.. The numbers should be put in relation to the number of patients treated during the same period. If no use-related reports have been recorded questions should be asked about identification of medication errors (they do occur everywhere)

II-

3-

1

0

How many

medical

products

adverse event

reports did the

health facility

submit to the

national

vigilance center

in the previous

year?

Provide statistics All reports of suspected adverse events recorded in the facility, that fulfill the completeness criteria, should also be submitted to the national vigilance Centre. Important to establish that no reports are left behind because of possible embarrassment (e.g.,

VL04.01 PE.VL.06


Page 261


Related

GBT or PE

indicator

s

Comment

and

justificatio

n

medication errors)

RISK ASSESSMENT AND EVALUATION

II-

3-

1

1

Which is the

process for

assessing

validity and

causality of

individual

medical product

case safety

reports received

from within

your health

facility?


• verifying completeness of case details

• consulting national or international literature or databases

• use of decision support algorithm

• expert committee consensus

• no local verification or assessment

Describe classification system used

Ascertain that efforts are made to verify the validity and completeness of case reports originating from the health facility. Routines should be in place for regular causality assessment and route-cause analysis if warranted.

VL04.02


II-

3-

1

2

How are you

providing feed-

back to internal

individual

reporters of

medical product

related case

safety reports?


• acknowledgement (electronic/paper/verbal, automatic or not)

• feed.back with case assessment

• advise re. possible prevention

no feed-back

Identify mechanisms available to stimulate, acknowledge and give feed-back to reporters including the result of the local causality assessment made.

VL04.02

II-3-13

How are you using the internally collected reports on medical product related safety problems in teaching staff how to contribute to safer patient care?

Describe mechanism of collective learning from reports of medical related problems in terms of improved standard operating procedures and routines for safer patient therapy and care

Identify mechanisms by which the health facility management or DTC use the information received through vigilance activities to continuously improve SOPs and routines for medical product handling to

VL04.03

Page 262



Related

GBT or PE

indicator

s

Comment

and

justificatio

n

improve patient safety

II-3-14

Are there mechanisms in place to disseminate vigilance or medical product safety information to members of staff of your health facility?


• newsletter

• information bulletin

• bulletin board,

• website,

• message app

• phone line,

• other.

Document the different channels by which health facility management is providing up-to-date information related to safe use and vigilance of medical products to its staff members. Identify possible gaps.

VL04.07: PE VL 01

II-3-15

How many requests for information about medical product safety were received by the health facility in the previous year?

Provide statistics if available on:

• Inquiries received from inside facility

From external parties and community

This question refers to the medical product safety information service provided internally to health facility staff and to the external community. This is not function considered as critical for a vigilance Centre, but has implications for the wider understanding of safe use of medicines

VL06.01

II-3-16

How many healthcare workers has the facility trained on vigilance and safe use of medical

Provide statistics and records of staff members trained

Providing training in safe use of medical products and vigilance practices is considered an

VL03.03: PE.VL.02


Page 263


Related

GBT or PE

indicator

s

Comment

and

justificatio

n

products in the previous year (through in-service training)?

integral part of the function of a vigilance Centre.

II-3-17

How many public or community education activities relating to medical product safety were carried out by the health facility in the previous year

Specify method of education and number of events

Assessors should verify if education of the public or community in medical product safety is part of the TOR of the health facility. This is not a given for all vigilance Centers. If, YES, the educational activities should be recorded.

VL03.03 PE.VL.02

II-3-18

How many training events/sessions related to medical product safety were conducted in the previous year?

Specify

• kind of activity

• purpose of activity number of activites

See II-3-16 above

VL03.03:

II-3-19

How many and which regular communications on medical product safety issues did the health facility receive from the national or regional vigilance centre in the previous year?

Specify all that apply e.g.

• Newsletters (printed/electronic)

• Dear Health Professional letters

Other

This question intends to verify the health facility perspective of the outreach activities of the national vigilance Center. Identify how often communications are received from the Center and how they are distributed in the health facility. Document any feed-back on

VL 06.02 PE.VL.01

Page 264



Related

GBT or PE

indicator

s

Comment

and

justificatio

n

the communications provided to the national Center.

II-3-20

How many vigilance training sessions organized by the national or regional vigilance centre did staff of the health facility attend in the previous year?

Specify number of training events and number of staff members attending?

This is to document the engagement of the health facility and its staff in training sessions organized by the national or regional vigilance centres. Document local judgement on the quality of the training being offered. The question intends to verify the coherence of the national vigilance system.

VL03.03: PE.VL.02

Overall evaluation of the vigilance system at the health facility level


☐ Satisfactory

☐ Unsatisfactory


3614

Annex 2: VIGILANCE FIELD VISIT CLOSING MEETING PRESENTATION FORM 3615

3616

The presentation form is embedded into this document and can be accessed by clicking on 3617

the below picture. 3618

3619


Page 265

3620 3621

Field Visit for assessment of the performance of medical products

vigilance in Country

Page 266


Annex 3: VIGILANCE FIELD VISIT REPORT FORM 3622

3623

This form should be used as a template (for guidance purpose) to formulate the VL field 3624

visit report however WHO team may amend the headlines or the content of the report, if 3625

needed. 3626

3627

3628

3629

3630

3631

3632

3633

3634

3635

3636

3637

3638

3639

3640

3641

3642

3643

3644

3645

3646

3647


Page 267

3648

3649

3650

Regulatory Systems Strengthening 3651

3652

VIGILANCE FIELD VISIT REPORT 3653

3654

3655

3656

Name of Country 3657

Field Visit Dates: <DD-DD/MONTH/YYYY> 3658

3659

3660

3661

3662

3663

3664

3665

3666

3667 3668

Page 268


3669

TABLE OF CONTENTS 3670

3671

3672

1. EXECUTIVE SUMMARY 269 3673

1.1. CONCLUSION AND RECOMMENDATIONS 269 3674

2. INTRODUCTION AND CONTEXT 269 3675

2.1. BACKGROUND 269 3676

2.2. OBJECTIVES 269 3677

2.3. METHODOLOGY OF THE FIELD VISIT 269 3678

2.4. PROGRAMME OF THE FIELD VISIT 269 3679

2.5. WHO TEAM 270 3680

2.6. INSTITUTIONS VISITED AND OFFICIALS MET 270 3681

3. CONDUCTED ACTIVITIES 270 3682

3.1. First day of the visit, Date <DD/MM/YYYY> 270 3683

3.2. Secon day of the visit, Date <DD/MM/YYYY> 270 3684

3.3. Third day of the visit, Date <DD/MM/YYYY> 270 3685

3.4. Fourth day of the visit, Date <DD/MM/YYYY> 270 3686

3.5. Fifth day of the visit, Date <DD/MM/YYYY> 270 3687

4. FINDINGS OF THE FIELD VISIT 271 3688

4.1 Strengths 271 3689

4.2 Gaps – areas for improvement 271 3690

5. CONCLUSION 271 3691

6. RECOMMENDATIONS 271 3692

7. ANNEXES 271 3693

ANNEX 1: List of persons met including NRA participants .......................................... 271 3694

ANNEX 2: Completed vigilance field visit assessment questionnaire .......................... 271 3695

ANNEX 3: Completed and scored PE indicators <OPTIONAL in case of VL field visit for 3696

the purpose of WLA designation. Please delete if not applicable> ............................. 271 3697

3698

3699


Page 269

3700

1. EXECUTIVE SUMMARY 3701

3702

<Please add an exectuve summary of the VL field visit, preferably in less than one page> 3703

3704

1.1. CONCLUSION AND RECOMMENDATIONS 3705

3706

✓ <Please list your conclusion and recommendations here using these bullets> 3707



3710

2. INTRODUCTION AND CONTEXT 3711

3712

2.1. BACKGROUND 3713

<Please provide a background about the VL field visit including context and country 3714

related information> 3715

3716

2.2. OBJECTIVES 3717

3718

The objectives of the VL field visit are to: 3719

• <Please list the VL field visit objectives (you may use the information included 3720

in the relevant field visit terms of reference for this purpose)> 3721





3726

2.3. METHODOLOGY OF THE FIELD VISIT 3727

3728

The procedure of VL field visit has been explained in detail in the “Field Visit Manual 3729

for Assessing the Performance of Vigilance Function”. Please refer to the mentioned 3730

manual for further details on the standard process and methodology of VL field visit. 3731

3732

2.4. PROGRAMME OF THE FIELD VISIT 3733

3734

The VL field visit was conducted at the below listed sites (please refer to 3735

INSTITUTIONS VISITED AND OFFICIALS MET section), from <Date> to <Date> 3736

according to the below agenda which was discussed and agreed with the <NRA> 3737

before the conduct of the visit. 3738

3739

Time Activity Team Location

Day, Date <DD/MM/YYYY>

Please insert data Please insert data Please insert data Please insert data



Page 270






3740

2.5. WHO TEAM 3741

3742

The following expert(s) performed the VL field visit on behalf of WHO: 3743

1. <Name, Institution, country (Team Leader, if applicable)> 3744

2. <Name, Institution, country> 3745

3746

3747

2.6. INSTITUTIONS VISITED AND OFFICIALS MET 3748

3749

The WHO team visited the following institutions where they met persons whom 3750

details are listed in Annex 2. 3751

3752

• <Please list the institutions visited> 3753

• <Please list the institutions visited> 3754

3755

Please see Annex 1 for the list of persons met during the VL field visit including NRA 3756

participants. 3757

3758

3. CONDUCTED ACTIVITIES 3759

3760

3.1. First day of the visit, Date <DD/MM/YYYY> 3761

3762

<Please describe activities performed on the first day of the VL field visit> 3763

3764

3.2. Secon day of the visit, Date <DD/MM/YYYY> 3765

3766

<Please describe activities performed on the second day of the VL field visit> 3767

3768

3.3. Third day of the visit, Date <DD/MM/YYYY> 3769

3770

<Please describe activities performed on the third day of the VL field visit> 3771

3772

3.4. Fourth day of the visit, Date <DD/MM/YYYY> 3773

3774

<Please describe activities performed on the fourth day of the VL field visit> 3775

3776

3.5. Fifth day of the visit, Date <DD/MM/YYYY> 3777

3778

<Please describe activities performed on the fifth day of the VL field visit> 3779


Page 271

3780

4. FINDINGS OF THE FIELD VISIT 3781

3782

<Please provide detailed findings of the visit including identified strengths and 3783

gaps/areas for improvement. The VL field visit assessment questionnaire may be used 3784

for this purpose> 3785

3786

Please see Annex 2 of the completed VL field visit assessment questionnaire. 3787

3788

<OPTIONAL in case of VL field visit for the purpose of WLA designation. Please delete if 3789

not applicable> In addition, please see Annex 3 of the scored and completed VL 3790

performance evaluation (PE) indicators. 3791

3792

4.1 Strengths 3793

3794

➢ <Please list strengths here using these bullets> 3795



3798

4.2 Gaps – areas for improvement 3799

3800

➢ <Please list gaps here using these bullets> 3801



3804

5. CONCLUSION 3805

3806

<Please provide a summarized conclusion of the VL field visit here> 3807

3808

6. RECOMMENDATIONS 3809

3810

<Please list here your recommendations based on the findings and conclusion> 3811

3812

7. ANNEXES 3813

3814

ANNEX 1: List of persons met including NRA participants 3815

ANNEX 2: Completed vigilance field visit assessment questionnaire 3816

ANNEX 3: Completed and scored PE indicators <OPTIONAL in case of VL field visit for 3817

the purpose of WLA designation. Please delete if not applicable> 3818

3819

3820

Page 272


Annex 4: VIGILANCE FIELD VISIT TERMS OF REFERENCE FORM 3821

3822

INTRODUCTION/BACKGROUND 3823

<Please provide an introduction to the visit including country specific information> 3824

RATIONALE 3825

<Please provide the rationale to the visit including the context> 3826

OBJECTIVES 3827

The objectives of the vigilance field visit include <please amend as necessary>: 3828

1. Assessment of the performance of VL activities and operations, conducted at the 3829

site(s) selected for the field visit, 3830

2. Assessment of knowledge, competence and experience of the officials and staff 3831

involved in VL related activities at the selected site(s), 3832

3. Identification of strengths of the VL activities performed at the selected site(s), 3833

4. Identification of areas which need further improvement and for which a specific 3834

development plan might be needed, and 3835

5. Feedback the relevant GBT sub-indicators or WLA performance evaluation process 3836

(PEP) sections on performance of the VL function. 3837

3838

EXPECTED OUTCOMES 3839

At the end of the vigilance field visit, the following outcomes are expected <please amend 3840

as necessary>: 3841

1. Determination if the performance and functionality of VL at the target country 3842

complies with WHO or other internationally recognized requirements and its own 3843

national regulatory requirements, 3844

2. Identification of the strengths, areas to improve and provide comments on how to 3845

address the identified gaps when necessary (e.g., in case of field visit for capacity 3846

building purposes), and 3847

3. Contribution to the conclusion of the overall performance of VL regulatory function 3848

as part of WHO benchmarking or WLA designation activities. 3849

3850

DELIVERABLES 3851


Page 273

At the end of the vigilance field visit, the following will be delivered<please amend as 3852

necessary>: 3853

1. VL Field Visit Report to be delivered by WHO team inclusive of completed vigilance 3854

field visit assessment questionnaire <optional, please delete if not applicable> and 3855

completed and scored performance evaluation indicators. 3856

METHODOLOGY <please amend as necessary>: 3857

This vigilance field visit is prepared, organized and conducted as per the Quality 3858

Management System (QMS) principles, following the relevant guidelines and manuals 3859

which are available on the WHO secure information sharing platform (please click HERE to 3860

access the platform using your access credentials). These manuals are: 3861

1. Field visit manual for assessing the performance of vigilance function, version 1, 3862

February 2021. 3863

2. Manual for benchmarking of the national regulatory system of medical products and 3864

formulation of institutional development plans, version 1, February 2021. 3865

3. Manual for self-benchmarking of the national regulatory system of medical products 3866

using WHO global benchmarking tool (GBT), draft version 2, February 2021. 3867

The visit is being coordinated between the WHO headquarters and WHO regional office as 3868

well as the WHO country office. 3869

DATES OF THE MISSION 3870

The visit will take place from <DD to DD MM YYYY>. 3871

TENTATIVE PROGRAMME OF THE VISIT 3872

The vigilance field visit will be conducted according to the below provided tentative 3873

programme. Prior to the visit, a teleconference will be organized among all interested 3874

parties (NRA, WHO/HQ, RO and CO) to confirm and/or adjust the programme including 3875

institutions to be visited. 3876

Begin End Session and location of the visit Institutions, and/or persons involved or

speakers

Day One: <date>

09:00 09:30 • Opening Remarks

• WHO/HQ (Presentation of objectives,

expected outcomes, programme and

team of the WHO)

• NRA

• WR or his

delegate

• WHO team

http://workspace.who.int/sites/att/default.aspx

Page 274


WHO TEAM 3877

The WHO team will be composed of following team members 3878

Country Name Function and or title Organisation

and/or dept. & division

<please add text> <please add text> <please add text> <please add text>

3879

It is important that the relevant institutions in charge can designate a focal point to 3880

coordinate the programme with the various institutions and /or departments as well as to 3881

assist the team members. 3882

WHO will fund the travel of the WHO team and will organize travel, hotel booking and visa 3883

requests for team members. This WHO mission is coordinated between HQ, RO and the 3884

CO. 3885

INSTITUTIONS TO BE VISISTED 3886

The team will be visiting the following institutions: 3887

1. <Name and address of the institution> 3888



LIST OF DOCUMENTS OR INFORMATION NEEDED BEFORE AND DURING THE VISIT 3891

The following is a non-exhaustive list of documents which may be needed during the visit. 3892

Other documents may be requested prior to or during the visit <please amend as needed>. 3893

1. Names, titles, functions and organizations of the participants including NRA 3894

participants. 3895

2. Organization chart of different institutions which will be visited. 3896

3. Reference regulatory legal basis documents such as law, regulations or 3897

circulars/orders that document the regulatory system enforcement. 3898

4. Key standards operating procedures (SOPs) relevant to VL function. 3899

5. <OPTIONAL in case of VL field visit for the purpose of WLA designation. Please 3900

delete if not applicable> completed self-assessment against vigilance PE 3901

indicators. 3902

3903

ACCESS TO RELEVANT INFORMATION 3904


Page 275

All information related to this WHO mission, as well as earlier missions, are archived and 3905

stored at the secure WHO NRA SharePoint at the following address: 3906

http://workspace.who.int/sites/ATT/default.aspx, particularly under <country> site. 3907

3908

3909

3910

3911

3912

3913

3914

3915

3916

3917

3918

3919

3920

http://workspace.who.int/sites/ATT/default.aspx


Page 276

3921

3922

3923

3924

3925

3926

3927

3928

3929

3930

3931

3932

3933

3934

3935

Appendix 2 to Annex 6 (6.2) – GxP Observed Audit Manual 3936

3937

3938

3939

3940

3941

3942

3943

3944

3945

3946

3947

3948


Authority


Page 277

3949

3950

GxP Observed Audit Manual 3951

for Assessing the 3952

Performance of Regulatory 3953

Inspection Function 3954

3955

3956

3957

3958

3959

3960

3961

3962

3963

3964 3965

3966

Page 278


3967

Contents 3968

Acronyms and abbreviations 279 3969

Specific definitions and terminology 280 3970

1. Introduction 282 3971

2. Purposes of the manual 283 3972

3. Scope 283 3973

4. Objectives 284 3974

5. Expected outcomes 284 3975

6. Deliverables 284 3976

7. Overview of observed audit process 285 3977

7.1 General principles 285 3978

7.2. Code of conduct, conflicts of interest and confidentiality undertaking 286 3979

7.3 Planning of the observed audit 287 3980

7.4 Preparation for observed audit 291 3981

7.5 Conducting the observed audit 293 3982

7.6 Reporting of the observed audit 295 3983

8. Roles and responsibilities 296 3984

8.1 Relevant NRA/Inspectorate 296 3985

8.2 WHO Secretariat (headquarters, regional and country office) 296 3986

8.3 WHO team leader 297 3987

8.4 WHO team members (observers) 298 3988

8.5 Inspection team leader 299 3989

8.6 Inspection team members 299 3990

8.7 Inspection site(s) 299 3991

Bibliography 302 3992

Acknowledgements 303 3993

Document change history 304 3994

Annexes 305 3995

Annex 1: OBSERVED AUDIT INSPECTORS' EVALUATION FORM 305 3996

Annex 2: OBSERVED AUDIT FINDINGS PRESENTATION FORM 319 3997

Annex 3: OBSERVED AUDIT REPORT FORM 320 3998

Annex 4: OBSERVED AUDIT TERMS OF REFERENCE FORM 327 3999


Page 279

4000

Acronyms and abbreviations 4001

4002

CAPA

CSP

corrective actions and preventive actions

clinical study protocol

DoI

GBT

declaration of interest

Global Benchmarking Tool

GCP Good Clinical Practices

GDP Good Distribution Practices (interchangeably used with GSDP)

GMP Good Manufacturing Practices

GSDP Good Storage and Distribution Practices (interchangeably used with GDP)

GxP Good Practices

MoH Ministry of Health

NRA National Regulatory Authority

PEP performance evaluation process

RI regulatory inspection

RSS regulatory systems strengthening

SMF site master file

SOP standard operating procedure

ToR term of reference

TRS technical report series

WHO World Health Organization

WLA WHO-listed authority

4003

4004

Page 280


Specific definitions and terminology 4005

4006

The definitions given below apply to the terms as used in the current document. These 4007

terms may have different meanings in other contexts. 4008

4009

Inspectors' Evaluation Form: the form/template used for the evaluation of the inspection 4010

process and practice, including inspection preparation, conduct and reporting, as well as the 4011

competency, skills and attitude of the inspection team. 4012

4013

Inspection findings: results of the inspection undertaken documented in a written report. In 4014

principle, the findings are compared against established guidelines, including regulations and 4015

guidelines. Based on the inspection findings, a conclusion can be made to indicate whether 4016

the inspected site conforms to the country legislation, regulations and code of practice or 4017

does not conform to these. The findings may be positive or negative. Negative inspection 4018

findings are usually referred to as deficiencies. 4019

4020

Inspection report: a report prepared in English or the local language with an English 4021

translation by the NRA inspection team, which documents the different inspection activities 4022

performed along with the observations, deficiencies and findings of the inspection. The 4023

inspection report is usually prepared according to the predefined template/format at the 4024

relevant inspectorate. 4025

4026

Inspection team: the team established by the NRA to perform the regulatory inspection as 4027

part of the provision of the national legislation and/or regulations enforced in the country 4028

relevant to different medical products and health technologies. In principle, an inspection 4029

should generally be performed by a team of inspectors; however, it may be conducted by a 4030

single inspector as well. For the inspection team, an inspector should ideally be appointed 4031

as a team leader. In addition, if the inspectorate procedures provide for it, the inspection 4032

team may include inspectors in training, observers or external consultants. 4033

4034

Inspection team leader: a trained, qualified inspector (according to well-established criteria) 4035

appointed or designated as such by the NRA/Inspectorate. 4036

4037

Inspection workplan: a plan usually developed by the inspection team to detail different 4038

activities, timings and assignments to be performed during the conduct phase of the 4039


Page 281

inspection process. The inspection plan should be prepared and cleared, if necessary, 4040

according to the procedures at the relevant inspectorate. 4041

4042

Observed audit: a process used by WHO to document and evaluate the level of performance 4043

of a national GxP13 regulatory inspection function. Observed audit may complement WHO 4044

benchmarking using the Global Benchmarking Tool for capacity-building purposes or the 4045

performance evaluation process (PEP) for the purpose of designation as a WHO-listed 4046

authority (WLA). The activity consists of an observation made by WHO observers of a routine 4047

inspection at an authorized site. The regulatory inspection under observation should ideally 4048

be a routine inspection and executed according to national references, including regulations 4049

and guidelines. National references are expected to be at least equivalent to WHO good 4050

practice guidelines (e.g. Good Manufacturing Practices [GMP], Good Distribution Practices 4051

[GDP], and Good Clinical Practices [GCP]) and/or any other internationally accepted 4052

guidelines. 4053

4054

Observed audit report: a report prepared in English, which is delivered by WHO observers 4055

according to the predefined observed audit report template. An observed audit report 4056

provides an overview of the observed regulatory inspection along with details of findings and 4057

recommendations of the WHO observers on the inspection process and inspectors’ 4058

performance. 4059

4060

WHO observer: a competent expert who is familiar with WHO published regulations and 4061

guidelines on the specific field of regulatory inspection as relevant to the inspection activities 4062

under observation (i.e. GMP, GDP or GCP). WHO observers should have extensive (more than 4063

7 years) experience and advanced skills in conducting national and international regulatory 4064

inspections as regulatory inspectors or WHO auditors. WHO observers are also referred to as 4065

the WHO team. 4066

4067

WHO-listed authority (WLA): a national regulatory authority or a regional regulatory system 4068

that has been documented to comply with all the indicators and requirements specified by 4069

WHO for listing based on an established benchmarking and performance evaluation process. 4070

A regulatory authority can be listed for one or more product categories or for one or more 4071

regulatory functions. 4072

4073

4074

13 For the purpose of this document, GxP should mean GMP, GSDP and GCP.

Page 282


14. Introduction 4075

4076

4077

The World Health Assembly Resolution WHA 67.20 recognizes the important role that 4078

regulatory authorities play in a well-functioning health-care system. As the regulation of 4079

medicines and vaccines and their procurement become more globalized, the World Health 4080

Organization (WHO) has sought to harmonize and strengthen regulatory activities through 4081

capacity-building and the promotion of regulatory cooperation in order to encourage the 4082

supply of safe, effective and high-quality products. Towards this end, decades ago, WHO 4083

established a regulatory system strengthening (RSS) programme to support Member States 4084

in developing, establishing and maintaining their regulatory authorities/systems. WHO’s 4085

support covers several areas, including, among others, technical assistance, assessment of 4086

regulatory systems (using the WHO Global Benchmarking Tool), training and capacity-4087

building activities and, most recently, designation of WHO-listed authorities (WLAs) upon 4088

which other Member States may rely. 4089

4090

One of the common regulatory functions subject to assessment in the context of 4091

benchmarking for capacity-building or WLA designation is GxP regulatory inspection. 4092

Consequently, the need for comprehensive assessment and evaluation of the performance 4093

and functionality of GxP regulatory inspection was raised. In response to this need, WHO, in 4094

consultation with Member States, partners and regulatory experts, developed the process 4095

and methodology of a WHO observed audit. 4096

4097

Observed audit forms an essential part of and crucial activity for benchmarking of regulatory 4098

systems for medical products (in the area of GMP inspections) and WLA designation (in the 4099

area of GxP inspections). 4100

4101

This manual summarizes the various activities for planning, preparing, conducting and 4102

reporting WHO observed audits. It also provides guidance on the competencies, duties and 4103

responsibilities of the various parties involved in the observed audit. 4104

4105

Following this manual will ensure the necessary consistency in organizing an observed audit, 4106

including defined roles and responsibilities, which in turn, will contribute to quality output 4107

and proper interaction among involved and interested parties. 4108

4109


Page 283

15. Purposes of the manual 4110

4111

The purposes of this manual are as follows: 4112

4113

• to provide guidance to WHO staff and observers as well as national regulatory 4114

authorities and other interested or involved parties on all aspects of the WHO 4115

observed audit process, including procedures and timelines for planning, preparing, 4116

conducting, reporting and following up as well as related documentation; 4117

• to define the composition of the observed audit team(s) as well as the required 4118

competencies, roles and responsibilities of team members; 4119

• to describe the roles and responsibilities of the three levels of WHO (headquarters, 4120

regional offices and country offices) in this process. 4121

4122

This manual also intends to familiarize the WHO Secretariat and WHO team, NRAs and other 4123

parties with a systemic approach for observed audit. Therefore, this manual should be read 4124

as applicable in conjunction with other relevant manuals, guidelines, standard operating 4125

procedures (SOPs) and work instructions. 4126

4127

Finally, this manual is primarily designed to establish a level of consistency and uniformity 4128

within the observed audit process and consequently enhance the reliability of benchmarking 4129

and WLA-related outcomes. 4130

4131

This manual is subject to periodic review and revision as part of the quality system approach 4132

applied by WHO. So, the user is advised to always ensure utilization of the latest version of 4133

the Manual. 4134

4135

16. Scope 4136

4137

The scope of the manual concerns the GxP regulatory inspections; namely, Good 4138

Manufacturing Practices (GMP), Good Distribution Practices (GDP), and Good Clinical 4139

Practices (GCP) inspections conducted by a national regulatory or competent authority for 4140

medical products and health technologies. 4141

4142

This manual describes the process of initiating, planning, preparing, conducting and reporting 4143

on an observed audit. It identifies the critical and key steps involved during an observed audit 4144

in order to determine if the regulatory inspection performed by the participating 4145

Page 284


authority/auditee complies with its own inspection procedures and with applicable national 4146

and international requirements, including WHO guidelines on GxP. 4147

4148

In terms of medical products streams, this Manual applies equally to observed audits 4149

pertinent to medicines (new chemical entities, multisource/generic medicines) and biological 4150

products (biotherapeutic products and vaccines). 4151

4152

17. Objectives 4153

4154

The objectives of the observed audit are: 4155

VI. assessment of the performance of regulatory inspectors to prepare, conduct and 4156

report GxP regulatory inspections; 4157

VII. assessment of knowledge, competence, skills and attitudes of NRA inspectors; 4158

VIII. identification of the strengths of the inspection activities performed; 4159

IX. identification of areas that need further improvement and for which a specific training 4160

plan might be needed; and 4161

X. feedback of the relevant GBT sub-indicators or WLA performance evaluation process 4162

(PEP) on performance of the GxP regulatory inspection. 4163

4164

18. Expected outcomes 4165

4166

The expected outcomes of the observed audit include: 4167

IV. determining whether the GxP regulatory inspection process and practice at the target 4168

inspectorate complies with WHO or other internationally recognized requirements 4169

and its own national regulatory requirements; 4170

V. identifying strengths, areas for improvement and provision of comments on how to 4171

address the identified gaps when necessary (e.g. in case of observed audit for 4172

capacity-building purposes); and 4173

VI. contributing to the conclusion of the overall performance of the GxP regulatory 4174

inspection function as part of WHO benchmarking or WLA activities. 4175

4176

19. Deliverables 4177

4178

After completion of the observed audit, the following deliverables shall be provided to the 4179

WHO Secretariat: 4180


Page 285

4181

III. Inspection report (prepared in English or in the local language with English translation) 4182

to be delivered by the NRA inspection team following the predefined 4183

template/format of inspection reports. 4184

IV. Observed audit report (in English) to be delivered by the WHO observers’ team 4185

following the predefined observed audit report attached as Annex 3 to this Manual. 4186

4187

20. Overview of observed audit process 4188

4189

The observed audit aims to assess the performance of the GxP regulatory inspection function 4190

with an emphasis on inspection activities and inspectors’ competency, skills and attitude. 4191

4192

7.1 General principles 4193

4194

f) There are two concurrently related activities taking place: the inspection process by the 4195

inspection team on behalf of the NRA and the observed audit by the observer(s) on behalf 4196

of WHO. It should be ensured that neither of the two processes negatively affects the 4197

other. This can be achieved through close collaboration between the inspection team 4198

and the WHO observer(s). 4199

4200

g) A GxP regulatory compliance programme is not limited to the GxP inspection process. It 4201

also includes components such as the supporting infrastructure of legislative and 4202

regulatory requirements, GxP standards, inspection/enforcement resources and 4203

procedures, performance standards, alert and crisis system, analytical capability, 4204

surveillance programme and quality management systems. While the observed audit 4205

process focuses on activities conducted during GxP regulatory inspection, along with 4206

inspectors’ competency, skills and attitude, other components are covered by systemic 4207

assessment as part of benchmarking activities (e.g. WHO GBT). 4208

4209

h) During the on-site inspection, it is expected that the inspection team and the observer(s) 4210

collaborate to ensure that the above-stated objectives are met. For this purpose, a 4211

briefing meeting between the inspection team and the observer(s) should be planned in 4212

advance and prior to the conduct of the inspection. 4213

4214

Page 286


i) The NRA and site/firm should discuss and agree in advance on the process of the 4215

observed audit, roles of the observer(s) and translator (if any), and the number of 4216

observer(s) to be included in the inspection. 4217

4218

j) To facilitate the WHO observed audit in evaluating the inspection process, a copy of the 4219

inspection process manual or regulating procedures or SOPs, including the NRA 4220

procedure for the format and content of inspection reports, should be sent to WHO, 4221

preferably two weeks before the observed audit. Similarly, a copy of the inspected 4222

institution/entity/site/facility Information master file, quality manual or similar 4223

file/document should be shared with WHO observers as soon as available, preferably 4224

before the commencement of the inspection. 4225

4226

k) It is not the objective of the observed audit to inspect the entities/firms or evaluate the 4227

level of implementation and consequently compliance with GxP. Observed audit does 4228

not constitute, by any means, a WHO inspection/audit of the site/firm. The site inspected 4229

by the NRA should refrain from misuse of the WHO observed audit (e.g. for promotional 4230

purposes). 4231

4232

l) Unrestricted access: WHO observers should have unlimited access to information, 4233

documents, people and assets of the inspected site/firm during the observed audit while 4234

respecting all applicable confidentiality arrangements and code of conduct. With regard 4235

to unlimited access to people, WHO observers may directly interview the firm’s 4236

employees at any hierarchical level while respecting the organization’s culture and habits. 4237

4238

m) Discussion among NRA inspectors and WHO observers related to the observed audit, 4239

including any major disagreement, should be made or resolved away from the inspected 4240

site/firm. 4241

4242

7.2. Code of conduct, conflicts of interest and confidentiality undertaking 4243

4244

WHO values and relies upon the technical advice provided by leading subject matter 4245

experts regarding its advisory and technical committees, meetings and other activities. 4246

4247

To ensure the integrity of processes such as WHO benchmarking and WLA-related activities, 4248

it is critical that WHO staff and observers appointed by WHO to perform observed audit 4249

activities: 4250


Page 287

a) fully and honestly discloses, in the Declaration of Interest (DoI) form, all relevant 4251

interests and biases that may give rise to real or perceived conflicts of interest; 4252

b) spontaneously reports, on an ongoing basis, any material changes to their disclosed 4253

interests during the period in which the assessor serves WHO; 4254

c) respects the confidential nature of the advisory function assigned by WHO and 4255

makes no public statements regarding the assessor’s advice without prior consent 4256

from WHO; 4257

d) refrains from engaging in any activity that may bring reputational harm to the WHO 4258

benchmarking process; 4259

e) represents their views in a personal and individual capacity, keeping in mind the 4260

best interests of WHO as opposed to the views of their employers or other 4261

institutions or governments; 4262

f) respects cultural differences and dress codes of the host country. Team members 4263

are advised to wear official attire for opening and closing meetings. 4264

4265

Prior to the observed audit process, confidentiality and DoI forms must be signed by all 4266

WHO observers. Completed and signed confidentiality and DoI forms should be checked 4267

and archived by the WHO Secretariat (RSS team) prior to the observed audit. The signed 4268

forms will be available with WHO and may be shared with the relevant authorities, if 4269

required. 4270

4271

7.3 Planning of the observed audit 4272

4273

7.3.1 Terms of reference 4274

4275

g) Prior to the observed audit, the WHO/RSS team should prepare the terms of reference 4276

(ToR) in collaboration and agreement with counterparts at the concerned 4277

NRA/Inspectorate following the applicable WHO procedures. 4278

4279

h) The ToR should specify, among others, objectives, proposed dates, a tentative agenda, 4280

expected outcomes and deliverables, the composition of the WHO team (observers), 4281

persons to be met and documents or information needed during the process. 4282

4283

i) The ToR should consequently be shared with the concerned NRA through official 4284

communication channels for agreement and concurrence. 4285

4286

Page 288


j) The ToR to be used for the observed audit should be available and distributed to all 4287

participants, including the inspected firm, if necessary. The ToR should also be available 4288

at a WHO secure information-sharing platform for access and archival purposes. 4289

4290

k) The list of observers should be shared with the NRA as well as the inspected firm, if 4291

applicable, ideally at least 30 working days prior to the observed audit mission. 4292

4293

l) A template for the “WHO NRA observed audit terms of reference form” is attached as 4294

Annex 4 of this Manual. 4295

4296

7.3.2 Agenda 4297

4298

b) As part of the ToR, a tentative agenda is agreed with NRA officials. The WHO team should 4299

finalize and issue the “WHO observed audit agenda” as per the template given in Annex 4300

4 of this Manual. The finalized agenda is shared in advance with NRA inspectors, as well 4301

as all entities (e.g. inspected site) involved in the observed audit, if applicable. 4302

4303

7.3.3 Secure information-sharing 4304

4305

d) Once the observed audit is agreed and confirmed between WHO and the Member State 4306

as part of the benchmarking or WLA-related activities, a specific page in the WHO secure 4307

information-sharing platform is created under the Member State’s NRA site. 4308

4309

e) All related documents should be uploaded to a secure information-sharing platform for 4310

access and archival purposes. These documents should include, but are not limited to, 4311

ToR, agenda, background documents, documented evidence submitted by the NRA (e.g. 4312

procedures), inspectors and observers’ information, travel and accommodation 4313

information, presentations, inspection and observed audit reports. 4314

4315

f) By default, access to the said WHO information-sharing platform is restricted to 4316

authorized users. Access to the platform should be granted to WHO observers as well as 4317

selected officials nominated by the relevant NRA, after the confidentiality agreement and 4318

the DoI are signed, in order to enable them to communicate with each other and 4319

upload/download relevant information. 4320

4321

4322


Page 289

7.3.4 Inspection workplan 4323

4324

a) Prior to the observed audit and the inspection process, the inspection team leader, 4325

together with the other inspection team members, should prepare an inspection 4326

workplan (also referred to as inspection agenda, inspection schedule or inspection 4327

programme). 4328

4329

b) The observer(s) should evaluate and comment on the developed inspection workplan 4330

within their observed audit report, as part of their overall evaluation and assessment 4331

of the GxP regulatory inspection. 4332

4333

c) In exceptional situations, WHO, in agreement with the NRA, should make a case-by-4334

case decision on whether a remote inspection, in full or in part, is considered 4335

appropriate and feasible for observed audit. Remote inspections should follow the 4336

applicable procedures that should be developed for coordinating, preparing and 4337

conducting GxP inspections, but should also take into consideration the limitations 4338

imposed due to the use of a remote process and recognize that such a remote process 4339

cannot completely replace on-site GxP inspections. If necessary, a face-to-face 4340

(physical) mission may be organized by WHO to the NRA/Inspectorate to follow up 4341

remote inspections and remote observed audits once the reasons that called for the 4342

remote approach are resolved. The activities of such face-to-face observed audits will 4343

be decided on a case-by-case basis. In general, remote observed audit is discouraged. 4344

However, it may be sought if there are strong reasons and the NRA, WHO and the site 4345

undergoing inspection agree that it is justified (e.g. public health emergencies 4346

involving travel restrictions). 4347

Site-specific issues (e.g. access restrictions due to safety/biosafety reasons) should 4348

not be considered as the sole justification for remote observed audits. 4349

4350

7.3.5 Selection of sites or entities 4351

4352

f) Selection of the site(s) for the observed GxP regulatory inspection should be agreed 4353

between the NRA and WHO. 4354

g) The NRA should provide WHO with the inspectorate routine inspection schedule 4355

(including names and addresses of entities, designated inspector(s) and tentative 4356

dates, if possible) in order to help in selection of the site(s). 4357

Page 290


h) In principle, the site(s) should be selected from among those sites scheduled for 4358

inspection as per the annual or regular inspection plan. 4359

4360

i) The number of the site(s) selected for observed audit should be done through a risk-4361

based approach. Factors to consider include the criticality of the products or the 4362

complexity of activities or processes, number of licensed/authorized firms/sites, 4363

capacity, geographical distribution, national/international exposure and earlier 4364

performance assessment experience of the relevant NRA/Inspectorate. The 4365

ultimate objective is to have a representative sample of the inspection process at 4366

the concerned NRA/Inspectorate. 4367

4368

j) Mock inspection, simulation or inspections scheduled for the sole purpose of 4369

observed audit should not be considered. 4370

4371

7.3.6 Translations 4372

4373

f) The observed audit should be performed in the language that is normally followed 4374

during routine inspection. 4375

4376

g) If possible, and in case this does not compromise the quality of the inspection process, 4377

English (or any other language that is understood by the observers) can be used. 4378

4379

h) If needed, translation services may be provided for the WHO observers. 4380

Translators should preferably have technical expertise in the field of inspection. 4381

Translation can be provided by the site(s) subject to the regulatory inspection. 4382

4383

i) If translation services are required, these should be clarified well in advance of the 4384

observed audit and arrangements should be agreed between the NRA and WHO, 4385

and, if applicable, the site(s) to be inspected. 4386

4387

j) Apart from simultaneous translation, the WHO Secretariat, in coordination and 4388

agreement with the NRA, may request for translation of relevant documentation 4389

(e.g. procedures, guidelines) well in advance of the observed audit. 4390

4391

4392

4393


Page 291

7.3.7 Team members’ attributes and competencies 4394

4395

e) A roster of qualified observers should be accessible for WHO to conduct observed audits 4396

on behalf of the organization. 4397

4398

f) The WHO observers should be qualified and competent according to well-established 4399

criteria to conduct the requested observed audit. 4400

4401

g) In terms of numbers, for each observed GxP regulatory inspection, one or two WHO 4402

observers should be designated regardless of the number of NRA inspectors or the 4403

inspection style (e.g. splitting of the inspection team). 4404

4405

h) The following represent the criteria for designation of WHO observers: 4406

➢ Background and education: WHO observers should be staff members of an NRA or 4407

WHO or experts (e.g. consultants) who are familiar with the relevant WHO published 4408

and other internationally recognized standards and guidelines. Observers should 4409

also be familiar with WHO benchmarking and WLA concepts and methodologies. 4410

➢ Experience: WHO observers should be experienced in the field of the inspection (e.g. 4411

GMP, GDP or GCP). Observers should have at least seven years of experience in 4412

conducting national or international regulatory inspections or audits. 4413

➢ Skills: WHO observers should demonstrate advanced skills in investigation, desk-4414

based inspections as well as questioning and listening skills. 4415

➢ Training: WHO observers should be well trained on the processes and 4416

methodologies related to observed audit. 4417

➢ Evaluation: WHO observers should be subject to formal evaluation by WHO staff 4418

(e.g. WHO team leader) against preset criteria and in accordance with the relevant 4419

procedures. 4420

4421

7.4 Preparation for observed audit 4422

4423

7.4.1 Briefing session 4424

4425

d) Apart from the initial qualification and enlisting within the roster of qualified WHO 4426

observers, the WHO observers selected for each individual observed audit should be 4427

thoroughly briefed on the principles described in this manual prior to the start of the 4428

mission. 4429

4430

Page 292


e) The WHO team leader should brief all team members (i.e. observers) remotely as part 4431

of preparation for the mission. The briefing should include details of the following: 4432

• context of the observed audit, including the objectives and expected outcomes; 4433

• methodology of the observed audit process; 4434

• availability of the required documents; 4435

• access to and utilization of a WHO secure information-sharing platform; 4436

• roles and responsibilities of different observers, including in specific area(s), if any; 4437

• other related logistical arrangements (e.g. travel, accommodation). 4438

4439

In addition, the WHO team leader should be able to clarify any doubts that arise 4440

during the briefing session. 4441

4442

f) When necessary, such briefing may be repeated two to three times to cover all 4443

observers. 4444

4445

7.4.2 Documentation review 4446

4447

d) Each team member, no matter how experienced, will need to spend the necessary 4448

time preparing for the observed audit, and reading the background documents. 4449

4450

e) As part of the preparation for the observed audit, WHO observers should review the 4451

following documents, to the extent possible, well in advance of the observed audit: 4452

• NRA or Inspectorate quality manual along with all relevant SOPs, particularly 4453

those related to inspection planning, preparation, conduct, reporting, 4454

enforcement, and follow up of corrective actions and preventive actions (CAPA); 4455

• a copy of national GxP (GMP, GCP or GDP) code/regulations/guidelines; 4456

• previous inspection reports of the same firm selected for the observed audit along 4457

with the CAPAs, if any; 4458

• a background document about the institution/entity/site/facility subjected to 4459

inspection (e.g. inspection site(s) information, site master file (SMF), 4460

investigator’s brochure, clinical study protocol (CSP), others as applicable); 4461

• major changes at the inspection site since the last inspection; 4462

• list of inspectors designated by the NRA to perform the GxP regulatory inspection, 4463

including their curricula vitae (CVs), job description, qualification and training 4464

overview; 4465


Page 293

• inspection workplan (also called inspection agenda, inspection schedule or 4466

inspection programme of work); 4467

• compliance history of the inspection site; 4468

• list of recalls, complaints, safety issues, among others, related to the site or 4469

products to be inspected; 4470

• recent regulatory or enforcement actions related to the site or products to be 4471

inspected, if any. 4472

4473

Translation of the afore-mentioned documents, if needed, should be coordinated between 4474

the NRA/Inspectorate and WHO. 4475

4476

f) To facilitate the preparation process for the observed audit, 10 days before the start 4477

of the observed audit at the latest, the relevant NRA focal person(s) should upload 4478

the above-mentioned documents to the relevant secure WHO information-sharing 4479

platform. 4480

7.5 Conducting the observed audit 4481

4482

l) The GxP regulatory inspection subjected to observed audit should take place in 4483

accordance with normal practice, as defined in the procedures of the NRA/Inspectorate 4484

and in accordance with the relevant NRA quality system. 4485

4486

m) The observer(s) should not take any active part in conducting or performing the 4487

inspection. However, if necessary, observers may ask further questions, request 4488

additional documents from the representatives of the inspected site or interview one or 4489

more of the staff working at the inspected site. The objective of such requests is not to 4490

evaluate the level of compliance at the inspected site. Rather, the objective of such 4491

requests by the observers is to help in comprehensively understanding the context of the 4492

regulatory inspection and evaluating the performance of the NRA inspectors. 4493

4494

n) The observer(s) may question the inspection team about findings made or not made by 4495

them during the inspection. The purpose of such questions is to evaluate the inspection 4496

process or the inspectors’ competency. 4497

4498

o) For the purpose of evaluation of the process and practice of the inspection as well as the 4499

competency of the inspection team, the observer(s) should make use of the "Inspectors' 4500

evaluation form" attached as Annex 1 of this manual. 4501

Page 294


4502

p) Good knowledge and proper understanding are crucial for effective use of the Inspectors' 4503

evaluation form and, consequently, for the quality of the observed audit report, including 4504

the respective conclusions and recommendations. 4505

4506

q) The GxP regulatory inspection should always be led and managed by the NRA inspectors. 4507

4508

r) At agreed intervals (e.g. end of each working day), the observer(s) should review the 4509

inspection process with the inspectors and give a feedback on the strengths and gaps in 4510

their progress. 4511

4512

s) Throughout the observed audit, observers should make clear, accurate and legible notes. 4513

Such notes should provide relevant yet detailed facts that serve as a record of what is 4514

directly observed. The notes should be used for the formulation of the observed audit 4515

report. 4516

4517

t) In the unfortunate situation that one or more critical findings, which are or may 4518

potentially have a negative public health impact, are overlooked by the inspection team 4519

but identified by the WHO team, the WHO team leader should be informed and act by 4520

reporting the issue to the proper managerial level at the NRA, in close coordination with 4521

the WHO Secretariat. 4522

4523

u) Once the inspection is finished, the WHO observer(s) should hold a debriefing meeting 4524

with the NRA inspection team, involving, as appropriate, other representatives from the 4525

NRA or the Inspectorate (e.g. top management). The purpose is to brief the attendees 4526

about the observed audit activities and present the observed audit findings, including the 4527

identified strengths, gaps, areas for improvement and recommendations (if any). This 4528

debriefing meeting should not include representatives of the inspected site and 4529

preferably not be held at the inspected site. 4530

4531

v) For the purpose of the debriefing meeting with the inspection team and 4532

NRA/Inspectorate representatives, the observer(s) should make use of the "Observed 4533

audit findings presentation form”, attached as Annex 2 of this Manual. 4534

4535

4536


Page 295

7.6 Reporting of the observed audit 4537

4538

c) In conjunction with the observed audit, two sets of reports should be issued: an 4539

inspection report by the inspection team and an observed audit report by the 4540

observer(s). 4541

4542

d) The inspection team should provide an inspection report (prepared in English or in 4543

the local language with English translation) following the predefined template/format 4544

at the NRA/Inspectorate. 4545

4546

e) The content of the inspection report is expected to correspond to the latest WHO 4547

technical report series that is applicable or other internationally recognized guidelines 4548

or recommendations. 4549

4550

f) The final inspection report should be made available to WHO within 14 working days 4551

from the last day of the inspection. 4552

4553

g) The observer(s) should prepare an observed audit report in English as per the format 4554

given in Annex 3 of this manual. 4555

4556

h) The finalized observed audit report should be made available to WHO within 21 4557

working days from the last day of the observed audit. A draft of the same should 4558

ideally be delivered by the observers on the last day of activity of the observed audit. 4559

4560

i) The final observed audit report should be subjected to a thorough review by the WHO 4561

Secretariat according to the relevant procedures to ensure consistency in and 4562

robustness of the output. 4563

4564

j) The final observed audit report should be shared with the respective 4565

NRA/Inspectorate and uploaded to the relevant site of the WHO secure information-4566

sharing platform for archiving purposes. 4567

4568

4569

4570

Page 296


21. Roles and responsibilities 4571

4572

Observed audit should be seen as a collaborative exercise to which several parties contribute, 4573

including the NRA/Inspectorate, WHO, inspection team, observers and inspection site(s). 4574

This section is meant to provide guidance on the roles and responsibilities of these parties. 4575

However, each party should collaborate as much as possible with the other entities to meet 4576

the objectives of the observed audit. 4577

4578

8.1 Relevant NRA/Inspectorate 4579

4580

b) The NRA/Inspectorate subject to observed audit is responsible for: 4581

• commenting and contributing to the ToRs of the observed audit; 4582

• discussing and agreeing with WHO on selection of the site(s) that will be subjected 4583

to the observed regulatory inspection; 4584

• designating the inspection team, including the inspection team leader, for the 4585

observed GxP regulatory inspection; 4586

• sharing with WHO, through the secure information-sharing platform or any other 4587

agreed means, all necessary information and documentation including, among 4588

others, national GxP code/regulations/guidelines, annual inspection 4589

schedule/plan, data specific to the selected regulatory inspection site (e.g. 4590

inspection site data, inspection team data), which will be subjected to the 4591

observed audit; 4592

• nominating officials and granting them access to the WHO secure information-4593

sharing platform; 4594

• communicating and coordinating with the inspection site(s), including on all 4595

necessary management and logistical arrangements; 4596

• confirming the regulatory inspection in writing at least 15 working days before the 4597

inspection date, along with the latest details of the inspection information. 4598

4599

8.2 WHO Secretariat (headquarters, regional and country office) 4600

4601

c) The WHO headquarters (RSS team), in collaboration with the six WHO regional offices 4602

as well as the relevant country offices, is responsible for the establishment and 4603

maintenance of the tools and databases related to observed audit. They are also 4604

responsible for the establishment of a roster of qualified observes along with their 4605


Page 297

training in order to ensure consistency in the quality as part of a quality management 4606

approach. 4607

4608

d) The WHO Secretariat is also responsible for: 4609

• drafting and finalizing the ToR of the observed audit; 4610

• discussing and agreeing with NRA on selection of the site(s) that will be subjected 4611

to the observed regulatory inspection; 4612

• establishing a dedicated page on the country’s site of the WHO information-4613

sharing platform for the observed audit and uploading all relevant documentation 4614

for access and archival purposes; 4615

• designating the WHO team leader; 4616

• selecting the observers from the roster of qualified observers to perform the 4617

observed audit on behalf of WHO; 4618

• organizing any necessary contractual and logistical arrangements for the 4619

observers. 4620

4621

4622

8.3 WHO team leader 4623

4624

b) The WHO team leader is responsible for: 4625

• leading and coordinating the WHO observed audit from the beginning to the end 4626

of the process. The team leader will lead the team of observers, and also 4627

participate in the observation and evaluation of the inspection process. The team 4628

leader will lead, advise and guide the WHO team. 4629

• briefing WHO observers on different aspects related to the observed audit, 4630

including context, background, objectives, process and methodology, as well as 4631

any safety issues relevant to the observed audit such as vaccination, if applicable; 4632

• coordinating work among all members of the WHO team in order to ensure 4633

smooth and harmonized execution of the WHO observed audit with avoidance of 4634

work duplication and/or conflicts; 4635

• communicating with the NRA: during the observed audit, the WHO team leader 4636

should communicate with officials of the relevant NRA as well as the inspection 4637

team on behalf of WHO; 4638

• delivering presentations: presentations made during the WHO observed audit 4639

opening and closing meetings will ideally be done by the WHO team leader. 4640

Nevertheless, different WHO team members will also help in preparing for these 4641

Page 298


presentations and providing inputs as necessary. Similarly, the WHO team leader 4642

may invite any of the WHO observers to present the findings, provide 4643

clarifications, answer questions of the NRA if needed; 4644

• delivering the WHO observed audit report: the overall report of the WHO 4645

observed audit shall ideally be prepared by all WHO team members; however, the 4646

responsibility of delivering the finally agreed report lies with the WHO team 4647

leader. 4648

4649

8.4 WHO team members (observers) 4650

4651

b) The observers are responsible for: 4652

• reviewing and signing the relevant administrative documents, including the 4653

invitation letter, confidentiality agreement and DoIs; 4654

• making necessary travel arrangements (e.g. booking flights and obtaining visas), 4655

as described in the invitation letter; 4656

• complying with the immunization requirements, if any, and bringing with them a 4657

copy of their immunization certificates; 4658

• respecting all applicable protocols and codes of ethics and conduct; 4659

• observing and evaluating the inspection process and inspectors’ performance, 4660

including planning, meetings, interviews, reviewed documents, inspection 4661

methodology, inspector’s competence and skill, as well as the leadership skills and 4662

duties of a team leader. For the purpose of assessing the performance of the 4663

inspection process, observers should use the "Inspectors' evaluation form" 4664

attached as Annex 1 of this Manual; 4665

• identifying the strengths as well as gaps and areas for improvement, if any. The 4666

identified strengths and areas for improvement should be presented by the 4667

observers following the "Observed audit findings presentation form" given in 4668

Annex 2 of this Manual; 4669

• preparing a detailed report on the observed audit conducted, including general 4670

information of the observed audit, activities, findings (strengths, gaps and areas 4671

for improvement) and recommendations, if applicable, to address the identified 4672

gaps as per the "Observed audit report" attached as Annex 3 of this Manual. The 4673

observed audit report should be handed by the observer to WHO within a 4674

maximum of 21 working days from the last day of the observed audit. If possible, 4675

a draft of the same should be delivered by the observers on the last day of the 4676


Page 299

observed audit activity. The report may quote the different components/sections 4677

in the evaluation form. The observers should highlight in the report the indicators 4678

that were not or were partially met. 4679

4680

8.5 Inspection team leader 4681

4682

b) The NRA inspection team leader is responsible for: 4683

• establishing and maintaining a formal and professional communication between 4684

the inspection team, the firm to be inspected and the WHO team or its 4685

representative in charge of the WHO observed audit; 4686

• planning and preparing the inspection workplan/agenda/programme/schedule; 4687

• planning, coordinating and preparing the on-site visit and informing the 4688

inspection and WHO team accordingly; 4689

• briefing the inspection and WHO teams and arranging the distribution of duties 4690

of the inspection team as per the agreed programme; 4691

• leading the inspection team and coordinating with the WHO observers; 4692

• coordinating and finalizing the inspection report to be provided to the NRA as well 4693

as WHO. 4694

4695

8.6 Inspection team members 4696

4697

a) The inspection team members are responsible for: 4698

• assisting and collaborating with the WHO team; 4699

• ensuring the necessary flow and sharing of information between parties; 4700

• providing support on translation process if necessary and applicable; 4701

• respecting all applicable protocols and codes of ethics and conduct. 4702

4703

8.7 Inspection site(s) 4704

4705

b) The inspection site(s) is responsible for: 4706

• communicating and coordinating all the necessary management and logistical 4707

arrangements with the NRA; 4708

• providing documents or information requested by inspection team as well as 4709

WHO observers in a timely manner; 4710

Page 300


• making available any information and/or documentation related to the site or 4711

product(s) (e.g. SMF, investigator’s brochure, others as applicable) required for 4712

the preparation and conduct of the inspection; 4713

• ensuring that relevant staff involved in the main activities or related activities are 4714

present and available during the inspection for interviews or clarification of issues 4715

identified. 4716

The following table provides an overview of the roles and responsibilities of the different 4717

parties involved in the observed audit, including shared responsibilities. 4718

4719

Activity

NRA / Inspectorate

WHO Secretariat

14

WHO team leader

WHO observers

Inspection team leader

Inspection team

members

Inspection site

R = responsible; C = contributor; I = informed; NA = not applicable

Draft the terms of reference of the observed audit, including objectives, scope, activities and timelines

C R C I I I I

Select sites for the observed audit C R C I I I I

Designate the WHO team leader I R NA I I I I

Designate WHO observers (team members) I R C NA I I I

Establish a dedicated site under the relevant WHO information-sharing platform

I R I I I I I

Share information related to the inspection site (including uploading to the WHO information-sharing platform)

R I I I I NA NA

Designate the inspection team, including the team leader R I I I NA NA NA

Nominate officials to access the WHO information-sharing platform R I I NA I I NA

Inform the inspection site and coordinate the observed audit R I I NA NA NA NA

Organize any necessary contractual arrangements for the WHO team - R I NA NA NA NA

Organize any necessary overseas logistical arrangements for the WHO team

I R NA NA NA NA NA

14 Also called WHO responsible officer


Page 301

Activity

NRA / Inspectorate

WHO Secretariat

14

WHO team leader

WHO observers

Inspection team leader

Inspection team

members

Inspection site

R = responsible; C = contributor; I = informed; NA = not applicable

Organize any necessary domestic logistical arrangements for the WHO team as well as the inspection team

R C I I I I I

Organize any necessary on-site or off-site translation services R R I I I I I

Brief WHO observers on different aspects related to the observed audit NA C R NA NA NA NA

Lead the observed audit and coordinate with the NRA/Inspectorate on-site C NA R C I I I

Deliver presentations during the opening and closing meetings of the observed audit, apart from representatives of the inspected site

I NA R C I I NA

Conduct the GxP observd audit I NA R R I I I

Lead the inspection process, including inspection opening and closing meetings NA I I I R C I

Conduct the GxP inspection NA I I I R R I

Prepare the inspection report NA I I I R R I

Deliver the inspection report, including uploading it to the WHO information-sharing platform

R I I I R C I

Prepare the observed audit report NA NA R R I I NA

Deliver the observed audit report, including uploading it to the WHO information-sharing platform

I NA R C I I NA

Respect all applicable protocols, codes of conduct and ethics R R R R R R R

4720

4721

Page 302


Bibliography 4722

4723

For the purpose of developing the current manual, the following sources were consulted: 4724

4725

WHO Global benchmarking tool (GBT) for evaluation of national regulatory system of 4726

medical products: glossary and definitions. Geneva: WHO; 2018 4727

(https://www.who.int/publications/m/item/10-gbt-glossary-rev-vi-ver-1nov2018-final-4728

adjusted, accessed 24 February 2021). 4729

4730

WHO Global benchmarking tool (GBT) for evaluation of national regulatory system of 4731

medical products. Regulatory inspection (RI): indicators and fact sheets, revision VI, version 4732

1. Geneva: WHO; 2018 (https://www.who.int/publications/m/item/06-gbt-ri-rev-vi-ver-4733

1nov2018-final-adjusted, accessed 24 February 2021). 4734

4735

Policy: evaluating and publicly designating regulatory authorities as WHO listed authorities, 4736

working document QAS/19.828/Rev.1. Geneva: WHO; July 2020 4737

(https://www.who.int/docs/default-source/medicines/norms-and-standards/current-4738

projects/qas19-828-rev1-policy-on-who-listed-authorities.pdf?sfvrsn=49504b99_2, 4739

accessed 24 February 2021). 4740

4741

Manual for planning, conducting and reporting of observed audit for assessing regulatory 4742

inspection (as part of WHO NRA assessment), Rev 6.0, 6/10/2015. (WHO internal 4743

document). 4744

4745




Page 303

Acknowledgements 4746

4747

Authors: Alireza Khadem Broojerdi, WHO headquarters, Geneva; Anna Laura Salvati, AIFA, 4748

Italy; Hamidreza Hozouri, Pasteur Institute, Iran; Helena Baiao, INFARMED, Portugal; and 4749

Mohamed Refaat, WHO headquarters, Geneva. 4750

4751

Contributors: Abdul-Samad Issah, FDA, Ghana; Achiraya Praisuwan, Thai FDA, Thailand; 4752

Alexandra Mata-Espinoza, WHO Region of the Americas, Washington, DC; Anna Laura Salvati, 4753

AIFA, Italy; Biorkys Yáñez Chamizo, CECMED, Cuba; Blouie Tse, Department of Health, Hong 4754

Kong SAR; Claudia Alfonso, WHO headquarters, Geneva; Caroline Getrude Samatanga, MCAZ, 4755

Zimbabwe; Cristina Baccarelli, EDQM, Strassbourg; Deon Poovan, SAHPRA, South Africa; 4756

Elvira Espinosa Gutiérrez, COFEPRIS, Mexico; Emmanuel Alphonce, TMDA, Tanzania; Engy El 4757

Hosary, Egyptian Drug Authority, Egypt; Eun-Hye (Grace) Park, MFDS, Republic of Korea; 4758

Graciela Aguilar Gil Samaniego, COFEPRIS, Mexico; Helena Baiao, INFARMED, Portugal; 4759

Ijeoma Ukachi Nwankwo, NAFDAC, Nigeria; Ines Hassan, WHO Consultant, London; Irene 4760

Gamal Shehata, Egyptian Drug Authority, Egypt; Liliana Silvia Patricia Álvarez Espejo, INVIMA, 4761

Colombia; Luz Andrea Gordillo Alarcón, INVIMA, Colombia; María de Lourdes Rodriguez 4762

Aveleyra, COFEPRIS, Mexico; Mohamed Refaat, WHO headquarters, Geneva; Mónica Bobbi, 4763

ANMAT, Argentina; Murilo Freitas Dias, WHO Region of the Americas, Washington, DC; 4764

Naoyuki Yasuda, MHLW, Japan; Sia Chong Hock, Health Sciences Authority, Singapore; Turki 4765

Abdulaziz AlRafie, Saudi FDA, Saudi Arabia; Varley Dias Sousa, ANVISA, Brazil; Vimal 4766

Sachdeva, WHO headquarters, Geneva; and Ximena Andrea Barbosa Aguillon, INVIMA, 4767

Colombia. 4768

4769

4770

4771

Page 304


Document change history 4772

4773

Version no. Date of issue Main changes

6th 6 October 2015 NA

7th 7 April 2021 Revision of the document outline and expansion of the content in light of discussions related to performance evaluation process (process) of WLA

4774

4775

4776


Page 305

Annexes 4777

4778

Annex 1: OBSERVED AUDIT INSPECTORS' EVALUATION FORM 4779

4780

OBSERVED AUDIT EVALUATION CRITERIA Level of assessment

WHO observers’ comments Team Leader Member

1. Inspection preparation

- Relevant procedures should be applied across the inspection preparation process. Observers should assess and evaluate adherence of the inspection team to the procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using WHO GBT.

- A risk-based approach (RBA) should be considered while preparing for the inspection.

1.1. Inspection team is well formulated according to the relevant procedure(s)

Guidance: National Control Laboratory (NCL) staff, product reviewers/assessors, product specialists, among others, may join the inspection team, if necessary.

X

1.2. Inspection team leader is assigned according to the criteria set in the relevant NRA procedure

Guidance: the assignment process should be based on a well-defined criterion (e.g. number of years of experience, level of administration/hierarchy, nomination by the NRA/inspectorate, etc.)

X

1.3. Roles and tasks are well distributed among inspection team members

Guidance: assigning specific roles and responsibilities should ideally consider qualifications, experience and skills important for the scope of the inspection. Also, as part of inspection preparation, the team leader can consider the possibility of the splitting the team or working together, as part of risk-based decision and time management.

X

Page 306




1.4. Inspected site(s) info (e.g. site master file, investigator’s brochure, protocol source documents, others as applicable) is reviewed

Guidance: the format and content of these documents should match relevant WHO guidelines or relevant international guidelines.

X

1.5. Inspected product(s) information (e.g. monograph, marketing authorization [MA] file, investigational medicinal product dossier [IMPD], clinical study protocol, recent related national control laboratory [NCL] and quality control [QC] data, others as applicable) is reviewed

Note: particularly for GMP inspection, recent related NCL and QC data should be reviewed.

X

1.6. Related regulations, standards and guidance are refreshed, if necessary

X X

1.7. Last relevant inspection report along with the CAPAs undertaken is reviewed

Guidance: corrective and preventive actions (CAPAs) here may refer to the detailed CAPA plan, CAPA evidence or CAPA evaluation/outcome according to national procedures and practices.

X

1.8. Recent related regulatory actions are reviewed

Guidance: recalls, marketing authorization (MA) suspension, clinical trial study suspension, revocation, etc.

X


Page 307



1.9. Recent related complaints, adverse drug reactions (ADRs), and other vigilance outcomes are reviewed

Guidance: it is expected that, as part of inspection preparation, other departments and units within the NRA (e.g. pharmacovigilance centre, market control department, NCL) should provide some input to properly meet this item.

X

1.10. An inspection workplan is developed by the inspection team

Guidance: the term workplan refers to the inspection programme, agenda or schedule of each single inspection activity and is not meant to refer to the overall inspection programme/schedule, which usually extends to several months.

X

1.11. Inspection workplan considered a risk-based approach X

1.12. Inspection workplan is available to each inspection team member

Guidance: the workplan should be shared among team members well in advance of the inspection either in paper or electronic format as part of inspection preparation.

X X

Page 308




WHO observers’ conclusion of the overall inspection preparation

Guidance: WHO observers should use the above-listed items to qualitatively evaluate the overall inspection preparation process. As preparation for the inspection, (i) the inspection team should be properly organized with clear roles and responsibilities of the team leader and members, (ii) the inspection team should have access to essential background documents and information, with input from other units within the NRA. A risk-based inspection workplan should consequently be developed and shared among the inspection team.

➢ The WHO observers conclude that the overall inspection conduct is: (please tick

one of the below checkboxes)

☐ Satisfactory

☐ Unsatisfactory

Justification: (please provide

text)

2. Inspection conduct

- Relevant procedures should be applied across the inspection conduct process. Observers should assess and evaluate adherence of the inspection team to these procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using the WHO GBT.

- A risk-based approach should be considered while conducting the inspection.

2.1. Inspection opening meeting was conducted X

2.2. The inspection team introduced themselves to the inspectee

Guidance: introductions can be made by each of inspection team members or the team leader on behalf of the team.

X X


Page 309



2.3. Inspection team leader presented the scope, objectives and expected outcomes to the inspectee

Guidance: this presentation can be verbal or using any sort of formal communication (e.g. inspection notification, paper or PPT) or, in particular situations, by a mixed methodology.

X

2.4. Inspection workplan was made available to the inspectee X

2.5. The inspection plan/agenda was adjusted, where warranted, based on the findings of the inspection (a risk-based approach is applied)

X X X

2.6. Last inspection findings and related CAPAs were checked considering a risk-based approach

Guidance: in terms of CAPA review, the on-site check should ensure that appropriate corrective actions include both short-term actions to address the immediate problem and long-term actions to prevent the recurrence of the problem, particularly for critical and major findings. Any CAPA pending from the last inspection should also be prioritized for on-site check.

X

2.7. Inspection facilitation aids (e.g. checklists, aide memoires) were used, if necessary

Guidance: use of inspection facilitation aides is optional and may not always be justified or required. In all cases, it is essential to review the NRA/Inspectorate policy on this aspect and check if the reviewed policy is followed in a consistent manner.

X X

2.8. Inspection team focused on primary objectives and risk areas identified

X X

Page 310




2.9. Updated guidelines, currently applicable to the inspection scope, were followed during the inspection

Guidance: the guidelines followed, as applicable to the inspection scope, are the current ones in use, and not those that are obsolete or in transition/phasing in stages.

X X

2.10. Actual operations were witnessed by the inspection team

Guidance: as part of the inspection methodology, normally it is expected to observe activities or operations to confirm compliance with MA, procedures and guidelines. Strong justification should be provided in case actual operations are not witnessed.

X X

2.11. Essential documentation was reviewed by the inspection team:

Guidance: examples of essential documentation include product quality review (PQR), SOPs, trend analysis, raw data, deviation reports, out-of-specification (OOS) reports, and others.

X

2.12. Critical stages and parameters of the inspected processes were covered during the inspection

Guidance: during the inspection preparation phase, the critical stages and parameters should be identified and addressed accordingly during the inspection process.

X

2.13. Essential calibrations, qualifications and validations were assessed X

2.14. Critical changes since the last inspection were reviewed, where applicable

X


Page 311



2.15. Identification of systemic findings, if any, out of isolated ones, was made

X

2.16. Daily feedback/debrief to the inspectee was done

Guidance: the feedback/debrief may be done during or by the end of the working day or the morning of the next day. The debrief should ideally cover the main findings and pending issues.

X

2.17. Inspection closing/exit meeting is conducted X

WHO observers’ conclusion of the overall conduct of the inspection

Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall inspection conduct process. A properly conducted inspection process should be considered as far as administrative procedure (e.g. opening, daily briefing and closing meeting) are respected and properly followed, the inspection team properly utilized a risk-based approach throughout the inspection conduct phase, including documentation review, observation of actual operations and interview of staff.



☐ Satisfactory

☐ Unsatisfactory


text)

3. Inspection reporting

- Relevant procedures should be applied across the inspection reporting process. Observers should assess and evaluate adherence of the inspection team to the procedures. Please note that evaluation of the procedures themselves should be subject to review and assessment during the benchmarking process using the WHO GBT.

- Deficiencies should be factual, evidence- and risk-based, and supported by GxP requirements.

Page 312




3.1. Deficiencies are well described and detailed

Guidance: deficiencies should be written such that they provide a comprehensive understanding of the context and the exact deviation from GxP requirement. Formulation of the deficiencies and how they are written and grouped can also be a factor affecting the classification of the deficiency.

X X X

3.2. Deficiencies are well grouped and categorized

Guidance: grouping and categorization of the deficiencies (may be also called findings, observations or non-conformities) should be well justified and consider the context of the deficiencies based on GxP chapters. Formulation of the deficiencies and how they are written and grouped can also be a factor affecting the classification of the deficiency. Categorization should not be confused with classification. The latter is addressed under item 3.3.

X

3.3. Deficiencies are well classified/ranked according to agreed definitions (e.g. critical, major and other/minor), according to internal procedures

Guidance: ideally, deficiencies should be classified in order to help prioritization of the actions to address them. It is expected that the NRA/Inspectorate will have a policy or a system of classification of the deficiencies and the same is respected by the inspection team.

X

3.4. Deficiencies and observations are supported with evidence X X X

3.5. Observations are referenced to regulations and guidelines X

3.6. The inspection report is finalized within the agreed time-frame X


Page 313



3.7. The inspection report adheres to the content and format described in the relevant SOP

X

3.8. Conclusion and overall compliance rating is in line with the inspection observations (in terms of the number and classification of deficiencies)

X

3.9. Inspectors’ recommendations to the NRA in response to the inspection objective, if any, is consistent with the level of detected risks

Guidance: recommendation is meant for the advice by inspectors with respect to the objective of the inspection (e.g. pre-approval inspection, licensing inspection, etc.).

X

WHO observers’ conclusion of the overall reporting of the inspection

Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall inspection reporting process. An inspection reporting process should be considered proper if the administrative procedures (e.g. format, timelines) are respected and properly followed, inspection deficiencies are well-described, classified and grouped, as applicable, and reference made to the relevant GxP guidelines.



☐ Satisfactory

☐ Unsatisfactory


text)

4. Inspectors’ technical competency

Page 314




4.1. Inspection team members have the required background and education to carry out the assigned inspection, in accordance with the inspectorate’s procedures

X X

4.2. Inspectors’ designation (lead, senior, junior) and qualification (initial training and observation) corresponds to the scope of the inspection

Guidance: the concept of qualification entails all the minimum prerequisite competencies considered necessary by the inspectorate to qualify inspectors for a specified activity. It includes: (i) initial training (theoretical and practical) and (ii) required experience (i.e. junior vs senior; or qualification for specific product categories).

X X

4.3. Qualification of inspectors is maintained through continuous training in accordance with the inspectorate’s training programme

X X

4.4. Inspection team members are aware of, knowledgeable in and actually utilizing relevant regulations and guidelines (GMP, GSDP and GCP) throughout the inspection process

X X

4.5. The depth of the inspection is appropriate and applies a risk-based approach

Guidance: inspection depth is considered an expression of the competency of inspectors

X X X

4.6. Inspection team members can engage in scientific discussions and provide rational reasons

X X


Page 315



4.7. Inspection team members’ performance in the field of inspection is satisfactory

Guidance: performance is meant to assess the overall inspection process, including critical thinking, root cause analysis, capability of taking decisions and immediate regulatory actions, when necessary, appropriateness (justification) of such decisions. Inspectors should be skilled in making professional judgements based on facts and science.

X X X

WHO observers’ conclusion of the overall inspectors’ technical competency

Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall technical competency of the inspection team. A technically competent team can be described as one having a team leader and team members with proper qualifications (background, education, training and experience), and knowledge (GxP guidelines and requirements), as evidenced by their proper performance throughout the inspection process.



☐ Satisfactory

☐ Unsatisfactory


text)

5. Inspectors’ attitude and skills

5.1. Inspection team members follow the code of ethics and conduct

Guidance: it should be noted that ethics, values and code of conduct may not be limited to the inspectors. Rather, a code may be applicable to all NRA staff or even all civil servants.

X X

Page 316




5.2. Inspection team members communicate effectively among themselves during the whole inspection process

X X

5.3. Inspection team members communicate effectively with the inspectee during the whole inspection process

X X

5.4. Inspection team members have good and proper questioning skills

Guidance: in this context, “questioning skills” are intended more to evaluate investigational skills such as critical thinking, root cause analysis, for example, other than communication skills in general.

X X

5.5. Inspection team members have good and proper listening skills

Guidance: in this context, “listening skills” should be measured with respect to the capability of identifying issues, understanding the context and collecting evidence, and arguing or questioning to obtain clarification and arrive at conclusions.

X X

5.6. Inspection team members take notes during the inspection process

Guidance: it is important to take good notes during all communication, interaction or observation; these notes may or may not be part of the inspection documentation process. However, such notes will contribute to the final inspection report.

X X

5.7. Inspection team members manage their time well

Guidance: time management is an important part of the inspection process, as the process of organizing and planning how to divide time between specific activities allows the fulfilment of the objectives and scope.

X X

5.8. A good environment was observed among the inspection team and the inspectee during the inspection process

X


Page 317



5.9. The team leader manages different aspects of the team well (e.g. conflicting opinions, redistribution of workload, coordination of the team assignments)

X

5.10. The team leader manages time effectively and respects the inspection workplan

Guidance: please note that the workplan may be modified in order to address situations seen/found during the course of the inspection process, as applicable or necessary.

X

5.11. The team leader has sufficient ability to make final decisions

Guidance: this is meant to assess the team leader’s skills and capabilities to resolve issues among the inspection team in case of any dissenting opinions.

X

WHO observers’ conclusion of the overall inspectors’ attitude and skills

Guidance: the WHO observers should use the above-listed items to qualitatively evaluate the overall skills and attitude of the inspection team. The team skills and attitude can be concluded as satisfactory if the team leader as well as team members are found to respect procedural arrangements, professionally conduct the inspection process in a positive environment among the team and with the inspectee, and the team shows advanced communication and time management skills.



☐ Satisfactory

☐ Unsatisfactory


text)

6. Overall conclusion

Page 318




6.1. WHO observers’ overall conclusion of the inspection performance

Guidance: the overall conclusion should be based on the evaluation of each of the individual five afore-mentioned indicators. If one of these indicators is found to be unsatisfactory, the overall conclusion should be consequently scored as unsatisfactory.

➢ Based on the collective evidence and findings of this observed audit, the WHO observers conclude that the performance of the GxP regulatory inspection, including inspection preparation, conduct and reporting as well as inspectors’ technical competence, skills and attitude is:

☐ Satisfactory

☐ Unsatisfactory


4781


Page 319

Annex 2: OBSERVED AUDIT FINDINGS PRESENTATION FORM 4782

4783

The presentation form is embedded in this document and can be accessed by clicking on the 4784

picture below. 4785

4786

4787

WHO Observed Audit of GxP Regulatory Inspection in Country

Page 320


Annex 3: OBSERVED AUDIT REPORT FORM 4788

4789

This form should be used as a template (for guidance purposes) to formulate the observed 4790

audit report. However, WHO observers may amend the headlines or the content of the report, 4791

if needed. 4792

4793

4794

4795

4796

Regulatory systems strengthening 4797

4798

OBSERVED AUDIT REPORT 4799

4800

4801

4802

National Regulatory Authority 4803

Name of Country 4804

Observed Audit Dates: <DD-DD/MONTH/YYYY> 4805

4806

4807

4808

4809

4810

4811

4812

4813

4814 4815


Page 321

4816

TABLE OF CONTENTS 4817

4818

4819

1. EXECUTIVE SUMMARY .......................................................................................................... 322 4820

1.1. CONCLUSION AND RECOMMENDATIONS 322 4821

2. INTRODUCTION AND CONTEXT ............................................................................................ 322 4822

2.1. BACKGROUND 322 4823

2.2. OBJECTIVES 322 4824

2.3. METHODOLOGY 322 4825

2.4. PROGRAMME OF OBSERVED AUDIT 322 4826

2.5. WHO TEAM 323 4827

2.6. INSTITUTIONS VISITED AND OFFICIALS MET 323 4828

2.6.1 Name of the inspected site(S): .................................................................. 323 4829

2.6.2 Background of the inspected site(s) .......................................................... 323 4830

2.6.3 Products or investigational product(s) relevant to the inspection ........... 323 4831

3. CONDUCTED ACTIVITIES ....................................................................................................... 324 4832

3.1. First day of the visit, Date <DD/MM/YYYY> 324 4833

3.2. Secon day of the visit, Date <DD/MM/YYYY> 324 4834

3.3. Third day of the visit, Date <DD/MM/YYYY> 324 4835

3.4. Fourth day of the visit, Date <DD/MM/YYYY> 324 4836

3.5. Fifth day of the visit, Date <DD/MM/YYYY> 324 4837

4. FINDINGS OF THE OBSERVED AUDIT .................................................................................... 324 4838

4.1 Strengths ....................................................................................................................... 325 4839

4.2 Gaps – areas for improvement ..................................................................................... 325 4840

5. CONCLUSION ......................................................................................................................... 325 4841

6. RECOMMENDATIONS ........................................................................................................... 325 4842

7. ANNEXES ............................................................................................................................... 325 4843

ANNEX 1: List of persons met during the observed audit 325 4844

ANNEX 2: Details of the NRA inspection team 325 4845

ANNEX 3: Inspection plan developed by the NRA inspection team 325 4846

ANNEX 4: Inspection report prepared by the NRA inspection team 325 4847

ANNEX 6: Completed observed audit inspectors’ evaluation form 325 4848

Page 322


4849

8. EXECUTIVE SUMMARY 4850

4851

<Please add an exective summary of the observed audit, preferably in less than one page> 4852

4853

1.2. CONCLUSION AND RECOMMENDATIONS 4854

4855




4859

9. INTRODUCTION AND CONTEXT 4860

4861

2.7. BACKGROUND 4862

<Please provide a background of the observed audit, including the context and 4863

country-related information, and brief information about the regulatory 4864

authority/inspectorate e.g. organization, applicable legislations and standards> 4865

4866

4867

2.8. OBJECTIVES 4868

4869

The objectives of the observed audit are as follows: 4870

• <Please list the observed audit objectives (you may use the information included 4871

in the relevant observed audit terms of reference for this purpose)> 4872





4877

2.9. METHODOLOGY 4878

4879

The procedure of observed audit has been explained in detail in the “GxP observed audit 4880

manual for assessing the performance of regulatory inspection function”. Please refer 4881

to this manual for further details on the standard process and methodology of the 4882

observed audit. 4883

4884

2.10. PROGRAMME OF OBSERVED AUDIT 4885

4886

The observed audit was conducted at the sites listed below (please refer to the 4887

INSTITUTIONS VISITED AND OFFICIALS MET section), from <Date> to <Date> according 4888

to the agenda below, which was discussed and agreed with the <NRA> before the 4889

conduct of the observed audit. 4890

4891

Time Activity Team Location


Page 323









4892

2.11. WHO TEAM 4893

4894

The following expert(s) performed the observed audit on behalf of WHO: 4895

3. <Name, institution, country (team leader, if applicable)> 4896

4. <Name, institution, country> 4897

4898

2.12. INSTITUTIONS VISITED AND OFFICIALS MET 4899

4900

The WHO team visited the following institutions where they met persons whose details 4901

are listed in Annex 1. 4902

4903

• <Please list the institutions visited, if any, apart from the inspected site> 4904

• <Please list the institutions visited, if any, apart from the inspected site> 4905

4906

In addition, please see Annex 2 for the details of the NRA inspection team. 4907

4908

i. Name of the inspected site: 4909

4910

Please include the name and full address of the inspected site. 4911

4912

ii. Background of the inspected site 4913

4914

Please provide an introduction to and a background of the inspected site here. 4915

For this purpose, you may use the information included in the received relevant 4916

manufacturer facility site master file. 4917

4918 iii. Authorized product(s) or investigational product(s) relevant to the inspection 4919

4920

The following table represents the list of authorized products or investigational 4921

product(s) relevant to the inspected site. 4922

4923 Authorized products or

investigational product(s) name

Dosage form Package Specifications

Clinical trial approval

no./National Drug

Page 324


Regulatory Authority

approval no.

Please insert data Please insert data Please insert data

Please insert data

Please insert data

Please insert data Please insert data Please insert data

Please insert data

Please insert data

4924

4925

4926

10. CONDUCTED ACTIVITIES 4927

4928

3.6. First day of the visit, Date <DD/MM/YYYY> 4929

4930

<Please describe activities performed on the first day of the observed audit > 4931

4932

3.7. Second day of the visit, Date <DD/MM/YYYY> 4933

4934

<Please describe activities performed on the second day of the observed audit > 4935

4936

3.8. Third day of the visit, Date <DD/MM/YYYY> 4937

4938


4940

3.9. Fourth day of the visit, Date <DD/MM/YYYY> 4941

4942


4944

3.10. Fifth day of the visit, Date <DD/MM/YYYY> 4945

4946

<Please describe activities performed on the second day of the OA> 4947

4948

Please see Annex 3 for the inspection plan developed by the NRA inspection team. 4949

4950

11. FINDINGS OF THE OBSERVED AUDIT 4951

4952

<Please provide a clear and detailed description of the findings of the observed audit, 4953

including identified strengths and gaps/areas for improvement. As the observed audit 4954

inspectors’ evaluation form is a mandatory annex to this report, the narrative should be 4955

used to summarize the topics of the different components and to justify the observations 4956

raised and opportunities for improvement identified. For each indicator, clearly mention if it 4957

is fulfilled or if it is not or partially fulfilled. Finally, it is also advised to start this section with 4958

a summary of the deficiencies, conclusion and recommendations of the NRA inspection 4959

team> 4960

4961


Page 325

Please see Annex 4 for the inspection report prepared by the NRA inspection team, 4962

including the inspection team findings and deficiencies. 4963

4964

11.1 Strengths 4965

4966

➢ <Please list the strengths here using these bullets> 4967



4970

11.2 Gaps – areas for improvement 4971

4972

➢ <Please list the gaps here using these bullets> 4973



4976

Please see Annex 5 of the inspectors’ evaluation form completed by the WHO team 4977

(observers). 4978

4979

12. CONCLUSION 4980

4981

<Please provide a clear and summarized conclusion of the observed audit here> 4982

4983

13. RECOMMENDATIONS 4984

4985

<Please list here your recommendations based on the findings and conclusion> 4986

4987

14. ANNEXES 4988

4989

<Add additional annexes if required, ensuring correct numbering in this section and in the 4990

entire report.> 4991

4992

ANNEX 1: List of persons met during the observed audit 4993

ANNEX 2: Details of the NRA inspection team 4994

ANNEX 3: Inspection plan developed by the NRA inspection team 4995

ANNEX 4: Inspection report prepared by the NRA inspection team 4996

ANNEX 5: Completed inspectors’ evaluation form 4997

4998

4999

Signature of all observers of WHO team leader 5000

Name

Date

Page 326


Name

Signature

5001

Page 327


Annex 4: OBSERVED AUDIT TERMS OF REFERENCE FORM 5002

5003

INTRODUCTION/BACKGROUND 5004

<Please provide an introduction to the observed audit, including country-specific information> 5005

RATIONALE 5006

<Please provide the rationale to the observed audit, including the context> 5007

OBJECTIVES 5008

The objectives of the observed audit include <please amend as necessary>: 5009

6. Assessment of the performance of regulatory inspectors to prepare, conduct and 5010

report GxP regulatory inspections; 5011

7. Assessment of the knowledge, competence, skills and attitude of the NRA inspectors; 5012

8. Identification of the strengths of the inspection activities performed; 5013

9. Identification of areas that need further improvement and for which a specific training 5014

plan might be needed; and 5015

10. Feedback on the relevant GBT sub-indicators or WLA performance evaluation process 5016

(PEP) on the performance of the GxP regulatory inspection function. 5017

5018

EXPECTED OUTCOMES 5019

At the end of the observed audit, the following outcomes are expected <please amend as 5020

necessary>: 5021

4. Determination if the GxP regulatory inspection process and practice at the target 5022

inspectorate complies with WHO or other internationally recognized requirements and 5023

its own national regulatory requirements; 5024

5. Identification of strengths, areas for improvement and provision of comments on how 5025

to address the identified gaps when necessary (e.g. in case of observed audit for 5026

capacity-building purposes); and 5027

6. Contribution to the conclusion of the overall performance of the GxP regulatory 5028

inspection function as part of WHO benchmarking or WLA activities. 5029

5030

5031

DELIVERABLES 5032

At the end of the observed audit, the following will be delivered<please amend as 5033

necessary>: 5034

Page 328


2. Inspection report (prepared in English or in the local language with English translation) 5035

to be delivered by the NRA inspection team following the predefined template/format 5036

of inspection reports. 5037

3. Observed audit report (in English) to be delivered by the WHO observers’ team 5038

following the predefined observed audit report attached as Annex 3 to this Manual. 5039

METHODOLOGY <please amend as necessary>: 5040

This observed audit is prepared, organized and conducted as per quality management 5041

system (QMS) principles, following the relevant guidelines and manuals available on the 5042

WHO secure information-sharing platform (please click HERE to access the platform using 5043

your access credentials). These manuals are as follows: 5044

4. GxP observed audit manual for assessing the performance of regulatory inspection 5045

function, version 7, 2021. 5046

5. Manual for benchmarking of the national regulatory system of medical products and 5047

formulation of institutional development plans, version 1, February 2021. 5048

6. Manual for self-benchmarking of the national regulatory system of medical products 5049

using WHO global benchmarking tool (GBT), draft version 2, February 2021. 5050

The visit is being coordinated by WHO headquarters and the WHO regional Office as well as 5051

the WHO country Office. 5052

DATES 5053

The observed audit will take place from <DD to DD MM YYYY>. 5054

TENTATIVE PROGRAMME 5055

The observed audit will be conducted according to the tentative programme provided 5056

below. Prior to the mission, one or two teleconferences/remote meetings will be organized 5057

among all interested parties (NRA, WHO headquarters, Regional Office and Country Office) 5058

to confirm and/or adjust the programme, including the institutions to be visited. 5059

Begin End Session and location of the visit Institutions, and/or

persons involved

Day One: <date>

09:00 09:30 • Opening remarks

• WHO headquarters (presentation of

objectives, expected outcomes,

programme and team of WHO)

• NRA

• WHO

Representative or

delegate

• WHO team

WHO TEAM 5060

The WHO team will be composed of the following team members: 5061

http://workspace.who.int/sites/att/default.aspx


Page 329

Country Name Function and or title Organization and/or

department & division

<please add text> <please add text> <please add text> <please add text>

5062

It is important that the relevant institutions in charge designate a focal point to coordinate 5063

the programme with the various institutions and/or departments as well as assist the team 5064

members. 5065

WHO will fund the travel of the WHO team and will organize travel, hotel bookings and visa 5066

requests of team members. This WHO mission is coordinated between headquarters, the 5067

Regional Office and Country Office. 5068

INSTITUTIONS TO BE VISISTED 5069

The team will be visiting the following institutions: 5070




LIST OF DOCUMENTS OR INFORMATION NEEDED BEFORE AND DURING THE VISIT 5074

The following is a non-exhaustive list of documents that may be needed during the visit. 5075

Other documents may be requested prior to or during the visit <please amend as needed>. 5076

6. NRA or Inspectorate quality manual along with all relevant standard operating 5077

procedures (SOPs), particularly those related to inspection planning, preparation, 5078

conduct, reporting, enforcement, and follow up of corrective actions and 5079

preventive actions (CAPA) 5080

7. A copy of national GxP (GMP, GCP or GDP) code/regulations/guidelines 5081

8. Previous inspection reports of the same firm selected for the observed audit along 5082

with the CAPAs, if any 5083

9. A background document about the institution/entity/site/facility subjected to 5084

inspection (e.g. inspected site(s) information, site master file, investigator’s 5085

brochure, protocol source documents, others as applicable) 5086

10. Major changes at the inspected sites since the last inspection 5087

11. List of inspectors designated by the NRA to perform the GxP regulatory inspection, 5088

including their CVs, job descriptions, qualifications and training overview 5089

12. Inspection workplan (also called inspection agenda, inspection schedule or 5090

inspection programme of work) 5091

13. Compliance history of the inspected site 5092

14. List of recalls, complaints, safety issues, among others, related to the site or 5093

products subjected to the observed regulatory inspection 5094

15. Recent related regulatory or enforcement actions related to the site or products 5095

subject to the observed regulatory inspection, if any 5096

16. Inspection plan. 5097

5098

Page 330


ACCESS TO RELEVANT INFORMATION 5099

All information related to this WHO mission, as well as earlier missions, are archived and 5100

stored at the secure WHO NRA information-sharing site at the following address: 5101

http://workspace.who.int/sites/ATT/default.aspx, particularly under <country> site. 5102

5103

5104

http://workspace.who.int/sites/ATT/default.aspx