Embed Size (px)
Citation preview
Produced by: FOI Services, Inc. 704 Quince Orchard Road • Suite 275 Gaithersburg MD 20878-1770 USA
Phone: 1-800-654-1147 or +1-301-975-9400Fax: 301-975-0702Email: [email protected]
TC141020
Determining a Statistically Valid Sample Size: What Does FDA Expect to See?
by Steven Walfish
Date: Monday, October 20, 2014Time: 1:00pm – 2:30pm Eastern Daylight Time (GMT/UT 1700)
12:00pm – 1:30pm Central Time 11:00am – 12:30pm Mountain Time 10:00am – 11:30am Pacific Time
Teleconference Course MaterialsYou may duplicate this for each person attending the conference.
Call-In: Dial 1-888-848-0354 approximately 10-15 minutes prior to the start time. When prompted, enter the PIN code followed by the # key: 9374489# Outside the U.S. & Canada, you will receive special instructions by email. If you become disconnected and cannot reconnect through the number above, you can also call in on +1-801-853-0800.
At the conclusion of the conference, an audio file will be made available for order. Attendees receive a special reduced price of $225. To order go to www.foiservices.com/valid4489.htm or call 301-975-9400.
Important NoticeThe information provided in this course by the instructor is his/her personal opinion and does not necessarily
represent the opinions of FOI, Inc. or its staff. Companies relying on the information do so at their own risk and assume the risk and any subsequent liability that results from relying on the information.
The information provided does not constitute legal advice.
Determining a Statistically Valid Sample Size:
What Does FDA Expect to See?
Presented by: Steven Walfish, Staff Statistician, BD
October 2014 An FOI Services Teleconference
Agenda
2
Building the Case Principals of Risk Management Relationship of Risk to Sample Size The OC Curve Tools for Continuous Data One-Sample One Proportion Individual values Capability Index
Tools for Attribute Data Z1.4 C=0
Case Study and Examples
Sampling Plans
3
Decisions are often based on our analysis of a sample.
How we conduct a sample is very important.
Want:
Minimize bias
Representative sample
An economical sample size
What is Unbiased & Representative?
4
The word bias is thrown around in the statistical literature.
The concept of unbiased means the sample is representative of the population.
Why is this so difficult?
Sampling Plans are Poorly Written
5
Most documents that detail a sampling plan state the sample size, but not the sampling method. “Measure the pH of ten samples.” “Inspect thirty labels.”
Should be written: “Measure the pH of ten samples throughout the process.” “Inspect thirty labels, 10 from the beginning, middle and end.”
Some Suggested Methods
6
Simple random sampling (SRS) ensures that all samples of size n are equally likely to be selected – units are selected independently – can use standard statistics
Stratified random sampling ensures that each of the strata are represented in the sample and we can construct the sample to either minimize variability of the estimator or to minimize cost
Composite sampling can save costs making sampling more efficient but you lose information about the individual sampling units.
Systematic sampling is a convenient sampling method for items coming off a line – ensures that items from the beginning, middle and end of production are sampled
Principles of Risk Management
7
Two primary principles of quality risk management are: The evaluation of the risk to quality should be based on
scientific knowledge and ultimately link to the protection of the patient; and
The level of effort, formality and documentation of the quality risk management process should be commensurate with the level of risk.
Sample size is a function of risk
Risk Management
8
Design (Verification and Validation)
Process Development (Process Validation)
In-Process (Process Control)
Final Product (Product Specifications)
Low Risk
High Risk
CAPA
Complaints
NCMR
Bad Outcomes
Risk & Sample Size
9
Confidence and reliability in a statistical assertion are expressed as Type I and Type II error.
Decision
Reality
Accept Reject
Accept
Correct Decision (green)
Type II Error (β) Consumer Risk
(red)
Reject
Type I Error (α) Producer Risk
(red)
Correct Decision (green)
Relationship of CTQ to Risk
10
CTQ Type Suggested α Suggested β Safety (Special Situation)
1% 0.1%
High 1% 1%
Medium 5% 5%
Low 5% 10%
Example
Why do Sampling Plans Protect Type I Error Only?
11
Acceptance criteria mean the product specifications and acceptance/rejection criteria, such as acceptable quality level and unacceptable quality level, with an associated sampling plan, that are necessary for making a decision to accept or reject a lot or batch (or any other convenient subgroups of manufactured units). [21CFR210.3 B20]
Type I error is also called the producer’s risk. Typically companies power sampling plans to minimize this risk!
Type II error is also called the consumer’s risk. The FDA would like to see sampling plans determined to control this risk.
Problem: Usually we are concerned with Type I error only when we do hypothesis
testing.
OC Curve
12
0%10%20%30%40%50%60%70%80%90%
100%
0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% 8.00% 9.00% 10.00%
Pro
ba
bil
ity o
f Acc
ep
tanc
e
Percent Defective
Type I error Probability of Acceptance = 95% Percent Defective = 0.09%
Type II error Probability of Acceptance = 10% Percent Defective = 3.80%
N=59, C=0
Definitions
13
AQL definition from ANSI/ISO/ASQC A3534 : “When a continuing series of lots is considered, a quality level which for the purpose of sampling inspection is the limit of satisfactory process average.”
LTPD is usually defined as the quality level at which the plan is 90% likely to fail.
Sampling Size
14
Decisions are often based on our analysis of a sample.
How we conduct a sample is very important. Want:
Minimize bias Representative sample An economical sample size
Sample Size
15
What is more important? Risk to the company of rejecting good product
Risk to consumer of getting bad/defective product
One Sample
16
( )2 2
2
Z Z Sn α β+
=∆
Zα and Zβ are the Type I and Type II errors
S is the estimated standard deviation
∆ is the difference to detect
You can substitute the t-value for Z when the sample size is small
Example
17
Type I error rate is 5% (95% Confidence) Zα=1.96
Type II error rate is 1% (99% Power) Zβ = 2.326
Unknown Standard Deviation and Delta, but want to detect a 0.5 standard deviation shift.
2-Sided Specification
( )
745.0
)1(326.296.12
22
=
+=
n
n
Values of Alpha & Beta
18
Confidence Level β α
1-Sided Specification 2-Sided Specification
80% 0.842 0.842 1.282
90% 1.282 1.282 1.645
95% 1.645 1.645 1.960
98% 1.960 1.960 2.241
99% 2.326 2.326 2.576
100% 2.576 2.576 2.807
19
Sample Size for Proportions
Where: α = alpha level of the test (two-sided) 1 – β = power of the test p0 = proportion under the null hypothesis p1 = proportion under the alternative hypothesis
2
01
111001 2
−
+=
−−
pp
qpzqpzn
βα
Example
20
How large a sample is needed to test H0: p = .21 against Ha: p = .31 at α = 0.05 (two-sided) with 90% power?
1943.193
21.031.0)69.0)(31.0(28.1)79.0)(21.0(96.1
2
⇒=
−+
=n
⇒ means round up to ensure stated power
Power Calculation
21
Where: α = alpha level of the test (two-sided) n = sample size p0 = proportion under the null hypothesis p1 = proportion under the alternative hypothesis
−−Φ=−
−
11
001012 1
qp
qpznpp α
β
Power - Example
22
What is the power of testing H0: p = .21 against Ha: p = .31 at α = 0.05 (two-sided) when n = 57?
( )46%)about (or 4641.0
09.0
)69.0)(31.0(
)79.0)(21.0(96.15721.031.0 1
=−Φ=
−−Φ=− β
K-Value Sample Size
23
Using tolerance interval Determine the Confidence Level Determine the percent of population coverage (Reliability) Use an ISO 16269-6 (ISO standard for tolerance intervals)
to find the value of K
LSk
USk
>−
<+
*
*
X
X
K-Value Example
24
How large a sample do I need to have 95% confidence with 95% reliability for a process with a mean of 10.0 and standard deviation of 0.55?
The specification are 8.5 to 12.0
Since the mean is closest to the lower specification
From ISO 16269-6, a sample size is approximately 21 for k=2.727
727.25.855.0*0
=>−
kk1
Sample Size for Capability
25
Some companies are calculating sample size based on capability indices.
This can only be done using a confidence interval approach.
Have to set a lower limit on the capability index and an observed capability index.
Solve for N
26
1ˆ
1ˆ
21;2
21;21
−≤≤
−−−−
npCCp
npC
nn αα χχ
−++≤≤
−+−
)1(21
)ˆ(911ˆ
)1(21
)ˆ(911ˆ
2222 npkCnZpkCCpk
npkCnZpkC αα
Acceptance Sampling
27
ANSI sampling plans for attributes and relationship to statistical hypothesis testing
AQL: Acceptable Quality Level “is the maximum percent nonconforming (or the maximum number of nonconformities per hundred units) that, for purposes of sampling inspection, can be considered satisfactory as a process average.” §4.2 Note: AQL is not lot or batch specific but rather a process average.
AQL is stated in the standard as a percent:
An AQL = 0.15 is a rate of 0.15 nonconforming units per 100 units or 0.15%.
Different Plans
28
Normal
Reduced
Tightened
Switching Rules
29
Sampling starts with normal inspection (general inspection level II).
When 2 out of 5 consecutive lots are not accepted, a switch is made to tightened inspection. Tightened inspection requires a larger sample size and uses a smaller acceptance number.
Normal inspection may resume when 5 consecutive lots are accepted under tightened inspection. Should 10 consecutive lots remain on tightened inspection (i.e., no 5 consecutive lots are accepted in the first 10 lots inspected under tightened inspection), sampling under Z1.4 should be discontinued.
A switch is often made from normal to reduced inspection when 10 consecutive lots are accepted under normal inspection, production is steady, and the switch is approved by the responsible authority. This enables the lot sentence to be determined using a smaller sample size.
Switching Rules
30
Binomial Confidence Intervals
31
( ) x)(nx p-1 p xn
−⋅⋅
=α
Binomial Distribution
Solve the equation for p given a, x and n. When c=0 then
)ln(
)1ln(p
n α−=
Confidence/Reliability
32
Confidence Reliability Sample Size
90% 90% 22
90% 95% 45
90% 99% 229
95% 90% 28
95% 95% 58
95% 99% 298
99% 90% 44
99% 95% 90
99% 99% 458
Binomial Confidence Intervals
33
( )
( ) 2)(272
x)(nx
p-1 p 2
27 .01
p-1 p xn
−
−
⋅⋅
=
⋅⋅
=
0
α
x=2, n=27 and a=0.01 (99% confidence) p=0.298
Would You Want to be the Patient?
34
If the manufacturer had an AQL of 0.5%?
A sampling plan with n=10 and c=0 is… AQL=0.5%
Patient Risk = 20.5%
This means that there is a 10% chance that the lot has a defective rate as high as 20.5%!
So Now What?
35
Any sampling plan can be justified depending on the risks.
The question is who should take the risk.
The intention of any sampling plan should be to minimize PATIENT risk.
Case Study 1
36
Have a new design for a medical device that requires torque to remove a cap to be between 2 and 6N. During development, the average torque was 4.3N with a SD of 0.65N.
What sample size do we need to have 95% confidence with 99% reliability that the average torque will be the requirement?
What sample size do we need to have 95% confidence with 99% reliability that the individual torque values will be the requirement?
Calculations (Mean)
37
𝑁 =𝑧𝛼 + 𝑧𝛽
2𝑠2
∆2
𝑁 =1.96 + 2.326 2 ∗ 0.652
6 − 4.3 2
𝑁 = 3
Calculations (Individual)
38
1000615.2
665.0*3.4*X
>=
<+<+
Nk
kUSk
Why the Difference?
39
A mean difference of 1.7N is different than having all the individual values below 6N.
The capability analysis would show a Ppk of 0.87.
The lower limit on the capability is 0.78. A sample size of 210 and a Ppk of 0.87 would give 95% confidence and 99% reliability.
What is the best answer? In this case, use the capability analysis approach.
Thank You! Questions?
40
All questions are welcome. To protect your privacy, anyone asking a question will be announced only by the first name of the person reported to the operator at initial dial-in. However, anyone in your group is welcome to ask a question. Please try to avoid using a speakerphone while asking your question.
If you think of a question later, or prefer to research an issue privately, you are invited to contact Steven Walfish via his website at www.statisticaloutsourcingservices.com. We value your feedback. Please complete and return the evaluation at the back of this handout to be entered into a drawing for a $100 amazon.com gift card and/or arrange to receive a certificate verifying your participation in the 1.5 hours of this educational session. Handouts, links & audio files. For an online copy of these slides go to www.foiservices.com/valid4489.htm. The site also describes how to order an audio file of this presentation at a 50% attendee discount.
Training Evaluation Form Determining a Statistically Valid Sample Size: What Does FDA Expect to See?Steven Walfish, Presenter – October 20, 2014
Win a $100 Amazon.com Gift Certificate from FOI!Just complete this form and fax it to +1-301-569-7506 by October 27, 2014
and you will be entered in a drawing for an Amazon.com Gift Certificate
Please indicate your primary job responsibility (circle one):
Regulatory Affairs QA/QC Statistics Medical Writing Electronic Submissions General Management
Other (please specify):____________________________________
Please rate various aspects of this course (1 = Poor; 2=Fair; 3=Good; 4=Very Good; 5=Excellent – please circle your answers)
What is your overall rating of the course? 1 2 3 4 5
How do you rate the content of the course? 1 2 3 4 5
Were course materials clear and understandable? 1 2 3 4 5
Was the length of the course adequate to cover the content? 1 2 3 4 5
Was the instructor knowledgeable about the subject matter? 1 2 3 4 5
How would you rate the instructor overall? 1 2 3 4 5
Other comments: Please feel free to comment on any aspect of this course, including the instructor, content, and technical arrangements:
TC141020
By providing the information below and faxing this page to +1-301-569-7506 by October 27, 2014 you will be entered in a drawing for a $100 Amazon.com Gift Certificate from FOI.
Your Name: _______________________________________________________________________
Company Name: ___________________________________________________________________
Email Address: ____________________________________________________________
Check here to receive a Certification of Attendance verifying your 1.5 training contact hours for this educational session. Please provide your legibly printed name with your signature next to it, email address, and fax this form to the number below. Your certificate will be emailed to you.
