FDA and Pharmaceutical Manufacturing Research Projects

Jeffrey T. Macher Jackson A. Nickerson

Co-Principal Investigators

Presentation Overview

Executive summary Project goals Data collection and synthesis Analysis methodology Findings Development opportunities and constraints

Executive Summary

We develop statistical models that predict the: Probability of a facility being chosen for inspection. Effect of investigator training, experience, and individual

effects on the probability of investigational outcomes. Characteristics and identities of facilities that correlate

with the probability of non-compliance.

We present initial results for each of these analyses.

We identify additional opportunities and next steps to create value along with some constraints.

FDA Research Project Goals

Risk-based assessment of FDA cGMP outcomes. Identify underlying ability of investigators and their

training. Identify underlying compliance of each facility.

Identify attributes (currently recorded by the FDA) that impact inspection outcomes.

Transfer “learning” to FDA.

Progress to Date

Just as new drugs go through Discovery Development and Commercialization….

Our model and this presentation concludes the discovery phase of our project.

Please think of our model as a “platform” that can be developed to assess a variety of compliance issues.

FDA Project Approach

Compile and link FDA databases. Estimate the likelihood of various outcomes:

NAI, VAI, OAI; Warning Letters; Field Alerts; Product Recalls.

based on… compound/product, facility, firm, FDA district,

investigator and training derived factors.

in order to … evaluate the allocation of investigational resources. inform effectiveness of investigator training and

management.

FDA Databases

DQRS (Field alerts) EES FACTS (Inspections) – CDER only Product Listing Product Recalls Product Shortages Facility Registration (DRLS) ORA Training database Warning letter database

Data Preparation

Started with FACTS (1990-2003). Manufacturing facilities only.

Assembled investigator training database: Identified corporate ownership by plant by

year and firms operating at a specific facility each year.

Constructed facility-year data Added observations for years NOT inspected.

Corrected FEI/CFN mismatches. Constructed numerous other variables.

Some basic “facts” about the FDA data

Years covered: FY 1990-2003

Total number of facilities inspected: 3753 Total number of “Pac codes”:

38,341 Total number of “Inspections”:

14,162 Total number of investigators: 783

Empirical Methodology

Inspection Probability of choosing a facility to inspect.

Detection Probability of a non-compliance inspection outcome.

Noncompliance Probability of noncompliance, inspection, and

detection. Detection control estimation.

Inspection

Groups of variables: Technology variables

• Rx Prompt Release Ext or Delayed Rel

• Gel Cap Soft Gel Cap Ointment• Liquid Powder Gas• Parenteral Lg. Vol. Parent. Aerosol• Bulk Sterile Suppositories

Industry variables• Vitamins (IC 54) Necessities (IC 55)• Antibiotics (IC 56) Biologics (IC 57)

Inspection decision variables• Ln(Days between inspections)• Surveillance = reason for inspection (0 = Compliance)• Last inspection outcome (1 = OAI, 0 = NAI, VAI)

Years 1992-2003 (binary variables for each year)

Inspection: Explained Variance

R2 Cumulative R2

Technology variables 12% 12%

Industry variables 9 21Inspection Decision variables 20 51Year dummy variables ~0 51

Omitted categories: Human Drugs (IC 60-66), select technologies, Year dummies 1990-91.

Foreign inspection included in analysis but uniquely identifies many inspections and is dropped from the analysis.

Probit analysis of decision to inspect.

Rx 0.13 **

Promp Rel. -0.19 **

Ext/del Rel. -0.19 **

Gel Cap -0.25 **

Soft Gel Cap -0.36 **

Ointment -0.32 **

Liquid -0.30 **

Powder -0.37 **

Technology Variables:Change in Probability of Inspection

Gas -0.68 **

Parenteral -0.32 **

Lg Vol Parent. -0.08 +

Aerosol -0.26 **

Bulk -0.37 **

Sterile -0.07 **

Suppositories -0.23 **

** 99% confidence interval* 95% confidence interval+ 90% confidence interval

Omitted categories: Not Classified, Bacterial antigens, Bacterial vaccines, Modified bacterial vaccines, Blood serum, Immune serum.

Industry and Inspection Variables:Change in Probability of Inspection

Antibiotics (IC 56) 0.19 **

Vitamins (IC 54) 0.11 **

Necessities (IC 55) -0.06 **

Biologics (IC 57) -0.07 **

Industry Variables Inspection Variables

Ln(Days btwn Insp) -0.28 **

Surveillance -0.84 **

Last outcome 0.13 **

Omitted category: Human drugs

** 99% confidence interval* 95% confidence interval+ 90% confidence interval

Days Between Inspections

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1 2 3 4 5 6

Years Since Last Inspection

Probability of Inspection

Probability of

Inspection

Years Since Last Inspection

Detection

Groups of variables Technology Industry Training

• Total training days prior to inspection (other than 5 main drug courses)

• Drug course 1: Basic drug school• Drug course 2: Advanced drug school • Drug course 3: Pre-approval inspections• Drug course 4: Active Pharmaceutical Ingrediant Mfg.• Drug course 5: Industrial sterilization

Investigator Experience• Number of inspections in the prior 12 months• Number of inspections in the prior 12-24 months

ORA District Office Investigator Classification

• A consolidation of position classifications

Detection: Explained Variance

R2 Cumulative R2

Technology variables 0.9 % 0.9 %Industry variables 0.3

1.2 Training and Experience vars. 0.3 1.5Office and Position variables 1.4 2.9Investigator effect 4.2 7.1

Probit analysis of decision to inspect.

Training and Experience Variables: Change in Probability of Detection1

1Without investigator fixed effects.

Total training days prior to inspection (less 1-5) -2.2E-03

Drug course 1: Basic drug school 0.07 *

Drug course 2: Advanced drug school -0.05

Drug course 3: Pre-approval inspections -0.23 **

Drug course 4: Activ. Ingred. Mfg. -0.15 *

Drug course 5: Industrial sterilization 0.08 *

No. of inspections in the prior 12 months 4.8E-03 +

No. of inspections in the prior 12-24 months -1.4E-03

ORA Office and Classification Variables:

Change in Probability of Detection2

ORA LOS 0.07 +

ORA KAN -0.06 +

ORA NYK -0.07 *

ORA SJN -0.09 **

ORA SRL -0.10 *

ORA ATL -0.10 **

ORA DAL -0.10 **

ORA SAN -0.11 **

ORA DET -0.13 **

ORA NWE -0.15 **

All other ORA off. insignificant.

Compliance 0.04

Microbiologist -0.02

Investigator -0.04

Chemist -0.05

Eng/Sci -0.07

Dist/Reg. Admin. -0.10 +

FDA Bureau -0.15 *

Technician -0.18

ORA Office Variables

Position Variables

2With investigator fixed effects.

01

23

De

nsi

ty

-.2 0 .2 .4 .6Probability of Plant Noncompliance by Investigator (with >50 inspections)

Distribution of Investigator Abilities

425 Investigators

Non-compliance

Detection Control Estimation Relatively new procedure used in academic literature. Used for assessing tax evasion, EPA compliance, and

other applications. FDA application more complicated than other

applications.

Assume three actors: Facility decides level of compliance. Inspection decision-maker chooses when to inspect. Investigator chooses detection or not.

Estimate all three processes simultaneously.

Non-compliance model

Assume inspection decisions are non-random. Assumption is different from other applications.

Construct a likelihood function that models the probabilities of: a plant being selected for inspection and the outcome of the inspection.

Constructing a Likelihood Function

L1i = 1

L1i = 0

L2i = 1

L3i = 1

L2i = 0

The likelihood that facility i is non-

compliant

The likelihood that facility i is compliant

The likelihood that facility i is inspected

The likelihood that facility i is not

inspected

The likelihood that facility i is found non-compliant

The likelihood that facility i is found

compliant

L3i = 0

Likelihood Function

Three probabilities are combined to form the function: Probability that a non-compliant facility is inspected and

detected: L1i=1, L2i=1, L3i=1

Probability of inspecting and not detecting noncompliance:• probability that the facility is compliant:

L1i=0, L2i=1

• probability that noncompliance goes undetected:

L1i=1, L2i=1, L3i=0 Probability that a facility is not inspected in a given year:

L2i=0

Estimating the Likelihood Function

Select covariates associated with non-compliance, selection, and detection. Non-compliance: facility-related characteristics. Selection: factors currently used in selecting facilities. Detection: investigator-related factors.

Use a maximum likelihood estimation to find coefficient estimates that maximize the function. Initialize parameter estimates with results from

inspection and detection analyses.

Change in Probability of Non-compliance

Rx -0.10 * -0.09 -0.05 -0.04

Prompt rel. 0.07 0.08 -0.13 -0.13

Ext/Del rel. 0.17 + 0.21 + 0.13 0.14

Gel cap 0.20 + 0.19 + 0.05 0.06

Soft gel cap -7.E-05 0.02 -0.04 -0.04

Ointment 0.11 0.08 -0.18 -0.15

Liquid 0.21 * 0.22 + -0.04 -0.03

Powder 4.E-03 -0.01 -0.26 -0.22

Gas -0.24 0.15 0.41 0.36

Parenteral 0.14 0.14 -0.04 -0.01

Lg. vol Parent. -0.24 + -0.25 -0.26 -0.27

Aerosol 0.08 0.08 0.11 -0.07

Bulk -0.18 ** -0.15 + -0.24 + -0.27 +

Sterile 0.09 0.09 0.03 0.01

Suppositories 0.12 0.12 -0.26 -0.27

Number of obs. 81570 55371 22456 17499

Vitamins 0.07 0.17

Necessary 0.13 0.12

Antibiotics 0.23 ** 0.22 *

Biologics -0.05 0.06

No. Thera. Classes/Plant 2.E-03 -3.E-03

No. Products/Plant -2.E-03 -1.E-03

No. Dose forms/Plant -4.E-03 -0.01

No. D.F. Routes/Plant -3.E-04 0.00

No. Sponsor Appl./Plant 0.02 * 0.02 **

Ownership change (t=0) 0.16

Ownership change (t=1) -0.13

Ownership change (t=2) -0.09

Ownership change (t=3) 0.34 +

Firms per plant -0.07

Inspection Technology Yes Yes Yes Yes

Plant Select No Yes Yes No

Detection Training Yes Yes Yes Yes

No. of obs 81570 55371 22456 17499

Predicted Level of Facility Non-compliance For 50 Most Inspected

Facilities

28

41

45

25 35 4 33 38 13 42 14 43 30 37 10 39 50 49 46 2217 31 12 40

1 34 26 47 8 36 21 32 18 23 2 44 3 5 19 169 29

20 15 7

27

28 41 45 25 35 4 33 38 13 42 14 43 30 37 10 39 50 49 46 22 17 31 12 40

1 34 26 47 8 36 21 32 18 23 2 44 3 5 19 16 9 29 20 15 7 27

Statistically more noncompliant than the mean facility.Statistically not different from the mean facility.

Statistically more compliant than the mean facility.

Immediate Implications

Inspection and Non-compliance New suggestions for inspection choices.

• Use non-compliance analysis to assess risk of any given facility, firm, or technology.

– Increase focus on particular facilities and attributes.– Ownership changes.

Mixed strategy inspection plan. Detection

Use detection analysis to assess quality of investigators and their training.

Focus investigator activities to build and maintain short-run experience.

Broader Implications

Our statistical methods provide a test-bed for asking and answering management and oversight questions. Further development is needed.

DCE has potentially broad applicability to CDER and other centers at the FDA including CBER, food, etc.. What facilities are most at risk of non-compliance?

• Base-line non-compliance• Technology• Ownership changes, etc.

What manufacturers are more/less prone to non-compliance. DCE has implications for the type, format, and processing

of data to be collected and analyzed.

Development Opportunities

Additional variables can and are being constructed to examine additional issues. Recall, shortages, supplement filings. More fine-grain information on technology,

manufacturing knowledge, organizational capabilities.

Evaluate manufacturer data collected in our study.

More heavily weight more recent investigations. Expand to full set of investigators and facilities

(requires additional computational resources). Evaluate endogeneity concerns.

Development Constraints

Software/computer limitation. Data preparation/man-power. Funding resources are nearly exhausted. Teaching.

Current Plan

Document current progress in a white paper. Further develop data in hand (EES, Shortages,

etc.). We received cooperation from the gold sheets.

Work with you to develop plan for transferring results to FDA.

Look for additional funding sources.