Adjustment of Calibration Interval

Mass Calibration Laboratory Recommended Practice Page 0RP-04: Monitoring and Adjustment of Calibration Intervals for Mass Standards docRP4.002N. Dupuis-Désormeaux, Senior Engineer Gravimetry August 2002

RP-04: Monitoring and Adjustment of Calibration Intervals for Mass Standards

Table of Contents

1. ABSTRACT........................................................................................................................................................................................................... 1

2. RATIONALE......................................................................................................................................................................................................... 2

3. DETAILED STEPS................................................................................................................................................................................................ 3

3.1 Establishment of the Minimum Sample Size..................................................................................................................................................... 33.1.1 Maximum Variance of the Data................................................................................................................................................................... 33.1.2 Maximum Tolerable Error ......................................................................................................................................................................... 43.1.3 Z-statistic Corresponding to the Confidence Level.................................................................................................................................... 43.1.4 Minimum Sample Size Computation......................................................................................................................................................... 5

3.2 Computation of the As-Found Values.............................................................................................................................................................. 6

3.3 Determination of Uncertainties......................................................................................................................................................................... 6

3.4 Establishment of Reliability Targets................................................................................................................................................................. 6

3.5 Formulation of Statistical Hypotheses.............................................................................................................................................................. 7

3.6 Analysis of Data............................................................................................................................................................................................... 73.6.1 Maximum Allowable Number of Defects.................................................................................................................................................. 93.6.2 Minimum Number of Defects.................................................................................................................................................................... 93.6.3 Examples.................................................................................................................................................................................................... 10

3.7 Adjustment of the Calibration Interval............................................................................................................................................................. 11

4. SUMMARY - AT A GLANCE............................................................................................................................................................................ 12

5. REFERENCES..................................................................................................................................................................................................... 13


RP-04: Monitoring and Adjustment of Calibration Intervals for Mass Standards

1. ABSTRACT

Measurement Canada recognizes that it is important to ensure that the standards calibration activities arewell documented and monitored. This is especially pertinent given the private sector’s increasing use ofquality assurance and quality control programs.

This document addresses exclusively one quality control aspect of Measurement Canada’s masscalibration activities: the monitoring and adjustment of calibration intervals.

Monitoring:An evaluation mechanism is suggested to determine if the number of masses found to be “out-of-tolerance” at the end of their calibration cycle is statistically significant. This verification is done throughthe analysis of “as-found” values that have been accumulated by the Calibration Standards Laboratory(CSL).

Adjustment of the Calibration Intervals:A method for shortening or lengthening the calibration intervals is proposed when the above analysisreveals significant differences from expected results.


2. RATIONALE

Various acceptance sampling methods are used to estimate the maximum (and minimum) number ofdefects to be (statistically) expected.

For the monitoring of mass calibration activities, two evaluation tools are most appropriate: a maximumlikelihood estimate method and the method proposed in ISO 2859-1.

A) Given a known sample size, confidence level and reliability target, a maximum likelihood estimatemethod with the probability density function of the normal distribution is used. This is similar to themethod of ISO 7966: 1993 (E) when used with proportions.

The seven steps involved in the method areas follows:

1) Establishment of the Sample Size2) Computation of the “As-found” Values3) Determination of Uncertainties4) Establishment of Reliability Targets5) Formulation of Statistical Hypotheses6) Analysis of Data7) Adjustment of the Calibration Interval

These steps will be explained in detail in thefollowing section.

B) The ISO 2859-1 standard with multiple sampling plans (Table IV-4) can also be used.

Although the ISO 2859-1method offers a more advanced technique for computing acceptance qualitylevels, the benefits gained from using a more elaborate sampling plan seem to be limited at this time.


3. DETAILED STEPS

3.1 Establishment of the Minimum Sample Size (N)

The determination of the sample size is crucial to any statistical analysis since it defines theconfidence that we will have in the results. The analysis is then conclusive only if all pointsrecommended are sampled.

The maximum variance of the process, the maximum tolerable error between sample mean and truepopulation average, and the confidence level must be known before the minimum sample size canbe calculated.

The sample size is a function of:

- a) the maximum variance of the data under study (F2Max) ;

- b) the maximum tolerable error (E) between the observed sample mean and the truepopulation average; and

- c) the confidence level (expressed as a Z"/2 value) with which we can say that the populationaverage is contained within the sample mean ± E.

This relationship can be expressed as:

3.1.1 Maximum Variance of the Data (F2 Max)

The maximum variance F2 Max of the values observed for the correction from nominal can beestimated based on experience. In the past, the values for the correction have fluctuatedbetween +11/3 tolerance and -11/3 tolerance. Therefore, the range of values observed can beexpressed as *Max - min* = * 11/3 tolerance - -11/3 tolerance * = *4/3 + 4/3* tolerance = 8/3tolerance. If we treat this fluctuation as a rectangular distribution, we have


3.1.2 Maximum Tolerable Error (E )

The maximum tolerable error E is the maximum difference that we are willing to acceptbetween the average value obtained for the sample and the “true” average of the population.In our case, we are gathering data on the correction from nominal value and thus will obtainan average value for this correction.

For mass calibrations, we have that the maximum expanded combined uncertainty 2ucombined_maxmust be smaller than 1/3 the applicable tolerance on that mass; therefore, the maximumucombined_max (at 1F) on the measurement is no greater than 1/6 tolerance.

It is reasonable to assume that we want our experimental average to be no farther than 1/6tolerance from the true average of the population. Thus we set E = ucombined_max = 1/6 tolerance.However, when the calculated value of the combined uncertainty ucombined (see 3.3) is known,this value can be used instead of the maximum combined uncertainty ucombined_max , discussedabove.

This means that we require a sample of “N” points (see 3.1.4) in order to ensure that the“true” average correction from nominal value is no greater than 1/6 tolerance away from theobserved average correction from nominal.

Technical Note:

Since the average correction is c = (3xi)/n, where xi are the individual corrections, we have that theuncertainty of the average correction observed “c”can be expressed as:(u(c))2 = ((Mc/Mx1)u(x1))2 + ((Mc/Mx2)u(x2))2 + (Mc/Mx3)u(x3))2 + (...) + (Mc/Mxi)u(xi))2

Because we know that MAX (u(x1)) = u(x1) = u(x2) = (...) = u(xi) = 1/6 tolerance, we have that (u(c))2 = (1/n)2 3 (u(xi))2 = (1/n)2 (n) (u(xi))2 = (1/n) (u(xi))2. Therefore, at 1F, u(c)= 1F(c) = F(xi)/%nrepresents the chance variations of the sample mean from the “true” mean. This respects the CentralLimit Theorem that states that, the standard deviation of the means (of samples of ‘n’ points) is equalto the standard deviation of the entire population divided by the square root of the number of points ineach sample.

3.1.3 Z-statistic (Z "/2) Corresponding to the Confidence Level

Finally, we set the confidence level at 95%. This implies that if we want to be 95% certainthat the true population average will be located within ± E from the observed sample average,then we must collect at least “N” points. Because we are considering both possibilities + E and-E, we use a two-sided Z-statistic Z "/2.

. For a 95% confidence level, " = 0.05, or "/2 = 0.025; to this "/2 corresponds a Z "/2 of 1.96.


3.1.4 Minimum Sample Size (N) Computation

The minimum sample size is therefore computed as follows:

A number of N $ 82 is computed from the above. With n = 100, we have sufficient points toensure that the confidence level is respected. It should be borne in mind that if fewer than 82points are sampled, the confidence level will be directly affected. Further, if the sample sizeis smaller than 30 points the analytic techniques presented in this paper are inappropriate andshould not be used.

Please note that ISO 7966:1993(E) could also have been used to determine the sample size. In this case, wewould have set the following: ISO 7966:1993(E)

Alpha risk set at 0.05 and beta risk set at 0.05 because we want to be 95% certain of the results of ourfindings. These risks are one-sided; hence, Z" = Z $ = 1.645

Zp0 = Z0.05 = one-sided = 1.645 = accept lot if fewer than 5% of standards are outside the control limits.Zp1 = Z0.10 = one-sided = 1.282 = reject lot if more than 10% of standards are outside the control limits.

USL =0.5 tolerance, LSL = -0.5 tolerance, because outside these values the mass will need to be adjusted

S = F within = random variability of each data point, which in our case corresponds to a maximum valueof 1/6 of the tolerance = 0.17 tolerance. Please note that this parameter is different from the standarddeviation expected in the population as described in section 3.1.1.

APL ----- USL - Zp0S =USL - 1.645 S = 0.5 tolerance - 1.645 (0.17 tolerance) = 0.2203 tolerance ----- LSL + Zp0S = LSL + 1.645 S = -0.5 tolerance + 1.645 (0.17 tolerance) = -0.2203 tolerance

Masses with corrections smaller than ± 0.2203 tolerance away from nominal have a 95% chance of beingaccepted (and fewer than 5% will be over the ± 0.5 tolerance limit)RPL ----- USL - Zp1S = USL - 1.282 S = 0.5 tolerance - 1.282 (0.17 tolerance) = 0.2821 tolerance

----- LSL + Zp1S = LSL + 1.282 S = -0.5 tolerance + 1.282 (0.17 tolerance) = -0.2821 tolerance(RPL - APL)2 = (0.2821 tolerance - 0.2203 tolerance )2 = (0.0618 tolerance)2 Masses with corrections greater than ± 0.2821 tolerance away from nominal have a 95% chance of beingrejected (and more than 10% will exceed the ± 0.5 tolerance limit)

N = (Z" + Z $ )2 Fwithin 2 ÷ (RPL - APL )2 = (3.29)2 (0.17 tolerancel)2 ÷ (0.0618 tolerance)2 = 82


3.2 Computation of the “As-found” Values

Full calibration procedures are provided in RP-01 Laboratory Calibration Procedures for Standards of Mass

Using the usual calibration procedures outlined in RP-01 Laboratory Calibration Procedures forStandards of Mass, determine the “as-found” value of at least the same number of standards as theminimum sample size determined in 3.1.4 above (i.e., n $ N = 82).

The “as-found” values are obtained prior to cleaning or adjusting the standards.

3.3 Determination of Uncertainties

Full computation of the uncertainties for mass calibration activities can be found in RP-02 Determination of MassCalibration Values and Related Uncertainties

Compute or note the associated uncertainty for the nominal value and class of the mass standardunder evaluation. If this information is not available, then a maximum uncertainty (at 1F) of 1/6tolerance can be used; this is discussed in section 3.1.2.

3.4 Establishment of Reliability Targets

Parameters for analysis:- Reliability target: 90% of population must fall within control limits- Control Limits: ± ½ tolerance from nominal- Confidence Level: 95%

The reliability target is set such that: at the end of their calibration cycle, 90% of the standards mustfall within the control limits.

The control limits are set according to the assumption that we want the values for the correction fromnominal to be within ±½ tolerance from nominal when the masses return for calibration. This alsoaccounts for an expanded uncertainty in the determination of the correction equal to ± a tolerance.

In other words, a mass is called a defect if, when it returns for calibration, its mass value is beyondthe control limits of ± ½ tolerance from nominal. The number of total defects observed is calledXdefects.

The confidence level represents that Measurement Canada can be 95% certain that the analysisperformed will adequately detect when more than 10% of the standards are outside the control limitsat the end of their calibration period.


3.5 Formulation of Statistical Hypotheses

After the reliability target, control limits and confidence level have been established, a comparisoncriteria is defined via statistical hypotheses. The “as-found” data are then compared to thesestatistical hypotheses.

In hypothesis testing, a null hypothesis is formulated and compared against its alternative hypothesis.

In our case, - the null hypothesis Ho is: no more than 10% of the population will be outside the control limits; - its alternative hypothesis H1 is: more than 10% of the population will be outside the control limits.

Note: the population is the total number of active standards of the same nominal value, class andusage.

This can be written as:Ho: X # (10%) M out of control limits

H1: X > (10% ) M out of control limitswhere X is the total number of defects within a population of M standards.

3.6 Analysis of Data

The “as-found” data are now compared against the statistical hypotheses. This step is crucial as itdetermines if the number of observed “good” points (within the control limits) is sufficient to not-reject the Ho hypothesis. Likewise, the number of observed points that are “out of control limits” canbe used to test our hypotheses.

To determine if the null hypothesis is to be rejected in favor of the alternative hypothesis, we mustcalculate what is called a test statistic. Hence, we are now ready to compare our sample data to ourexpected results by means of the test statistic.

Technical note:

Using a maximum likelihood estimate and the probability density function of the binomial distribution we canestimate what fraction of items will still be in-tolerance at the end of the calibration period. Note that aBernoulli trial is used since there are only two possible outcomes: either the points are within control limitsor they are outside these limits. Note that when using the (cumulative) normal distribution instead of the(discrete) binomial distribution, a correction for continuity is necessary when the sample is small (n smallerthan 30). Further, if the following conditions are met, the normal distribution can be used instead of thebinomial distribution:


If n $ 30 ORIf np $ 5 AND n(1-p)$ 5

where n is the number of points sampled and p is the probability of success(or failure) expected

The Z test statistic will be used in our analysis. It is based on the normal distribution. When using a normalapproximation, it is necessary to know the sample mean and standard deviation. Regarding proportions, themean is expressed as : = np and the standard deviation is expressed as F = %npq, where n is the number ofpoints sampled, p is the proportion of defects expected and q = 1- p is the proportion of non-defective itemsexpected.

The Z statistic is expressed as:

Z X npnp p

=−

−( )1where

X = the number of defects found in the samplen = the total number of points sampledp = the expected (target) probability of defects

The Z statistic is compared to the Z" value for expected results corresponding to the confidence level of ouranalysis. In fact we are computing a critical value for testing the null hypothesis by means of Z" against whichthe results are compared. If the calculated Z value falls within the acceptance region, the null hypothesis is notto be rejected.

As explained in the technical note above, the Z test statistic will be used in our analysis. It is basedon the normal distribution. For a 95% confidence level, the corresponding one-sided Z" value is1.645.


Z X npnp p

Z np p np X

n n X

α

α

=−

−

− + =

+ =

max( )

( ) max

. * . * . ( * . ) max

1

1

1645 01 0 9 01

− =−

−

− − + =

− + =

Z X npnp p

Z np p np X

n n X

α

α

min( )

( ) min

. * . * . ( * . ) min

1

1

1645 01 0 9 01

3.6.1 Maximum Allowable Number of Defects

The following equation is used to calculate the maximum allowable number of defects to beobserved in the sample of “n” points:

Note: It is given that p = 10% = 0.1. This is because we want no more than 10% defects in the entire population.

The total number of defects observed Xdefects should be no greater than the valueof X max above; otherwise, the calibration interval should be shortened accordingto section 3.7.

3.6.2 Minimum Number of Defects

The same process can be used to decide if the number of defects is lower and statisticallysignificant from the 10% mark. In this case, we have:

The total number of defects observed Xdefects should be no smaller than the valueof X min above; otherwise, perhaps the calibration interval should be lengthened.


1645 01 0 9 01

1645 100 01 0 9 100 01 14 935 15

1645 01 0 9 01

1645 100 01 0 9 100 01 5065 5

. * . * . ( * . ) max

. * . * . ( * . ) .

. * . * . ( * . ) min

. * . * . ( * . ) .

n n X

n n X

+ =

+ = =

− + =

− + = =

1645 0 2 08 0 2

1645 100 0 2 08 100 0 2 2658 26

1645 0 2 08 0 2

1645 100 0 2 08 100 0 2 1342 14

. * . * . ( * . ) max

. * . * . ( * . ) .

. * . * . ( * . ) min

. * . * . ( * . ) .

n n X

n n X

+ =

+ = =

− + =

− + = =

1645 0 3 0 7 0 3

1645 100 0 3 0 7 100 0 3 37 53 37

1645 0 3 0 7 0 3

1645 100 0 3 0 7 100 0 3 22 46 23

. * . * . ( * . ) max

. * . * . ( * . ) .

. * . * . ( * . ) min

. * . * . ( * . ) .

n n X

n n X

+ =

+ = =

− + =

− + = =

3.6.3 Examples

Here are a few examples of the calculations involved:

Example 1: Reliability Target q = 90% , or p = 10%. In other words, 90% of thestandards are within control limits at the end of their calibration cycle.Points sampled n=100.

T h eabove implies that the total number of defects observed Xdefects out of the n=100standards sampled must be larger than 5 and smaller than 15.

Example 2: Reliability Target q = 80% , or p = 20%. In other words, 80% of thestandards are within control limits at the end of their calibration cycle.

Points sampled n=100.

The above implies that the total number of defects observed Xdefects outof the n=100 standards sampled must be larger than 14 and smaller than 26.

Example 3: Reliability Target q = 70% , or p = 30%. In other words, 70% of thestandards are within control limits at the end of their calibration cycle.

Points sampled n=100.

The above implies that the total number of defects observed Xdefects outof the n=100 standards sampled must be larger than 23 and smaller than 37.


3.7 Adjustment of the Calibration Interval

The method used to determine the new calibration interval is based on an exponential reliability model and issimilar to Method A-3 described in the document Establishment and Adjustment of Calibration Intervals,Recommended Practice RP-1, January 1996, produced by the National Conference of Standards Laboratories.

The calibration interval may need to be shortened (or lengthened) if the above analysis detects more(or fewer) defects than is statistically expected. The interval is then adjusted using an exponentialreliability model as follows:

IRR

I10

0=lnln

where I1 = the revised calibration interval I0 = the present calibration interval

R = the target (expected) reliability in terms of numbers of defects per population= (maximum number of defects allowed) ÷ (total population of standards)

R0 = the observed reliability for the interval I0 = (number defects found at I0) ÷ (number of points sampled at I0)= Xdefects / n

Please note that lengthening calibration intervals requires modifications to the Weights andMeasures Act and Regulations and should not take place unless approved by SeniorManagement.

Calibration intervals can be shortened as required using the above exponential reliabilitymodel.


1645. * * ( * ) maxn p q n p X+ =

1645 01 0 9 01. * . * . ( * . ) maxn n X+ =

− + =1645. * * ( * ) minn p q n p X

− + =1645 01 0 9 01. * . * . ( * . ) minn n X

4. SUMMARY - AT A GLANCE

At a Glance

To re-establish calibration intervals, the following calculations must be made:

a) Select at least n$N standards out of the total population of standards and compute theiras-found value prior to cleaning and adjustment.

b) Use the following equation to compute the maximum number of defects to be observed:

If the reliability target is 90%, then the equation to use is:

c) Use the following equation to compute the minimum number of defects to be observed:

If the reliability target is 90%, then the equation to use is:

d) Compute the number of standards found to have a correction from nominal that is beyondthe control limits of ± ½ tolerance. This number is the total number of defects observedXdefects.

e) If Xdefects < Xmin or if Xdefects > Xmax, discuss the issue with the Director of the CSL andcalculate the new calibration interval using the following expression:

IRR

I10

0=lnln


5. REFERENCES

1. Chao, Lincoln L., Introduction to Statistics, Brooks/Cole Publishing Company, Monterey, California, 1980.

2. ISO Standards Handbook, Statistical methods for quality control, vol. 1: Terminology and symbols,Acceptance sampling, fourth edition, 1995.

3. ISO Standards Handbook, Statistical methods for quality control, vol. 2: Measurement methods and results,Interpretation of statistical data, Process control, fourth edition, 1995.

4. Joint International Committee ISO/IEC/OIML/BIPM- TAG-4, Guide to the Expression of Uncertainty inMeasurement, first edition, 1993.

5. Johnson, Robert, Elementary Statistics, third edition, North Scituate, Massachusetts: Duxbury Press, 1980.

6. Taylor, John K., and Oppermann, Henry V., Handbook for the Quality Assurance of MetrologicalMeasurements, NBS Handbook 145, 1986.

7. Wonnacott, Thomas H., and Wonnacott, Ronald J., Introductory Statistics, New York: John Wiley & Sons,Inc., 1969.

Documents

Adjustment of Calibration Interval