Upload
ngocong
View
219
Download
0
Embed Size (px)
Citation preview
Calibration of Quantitative PCR Assays
A M I Roberts C M Theobald and M McNeil
Quantitative real-time PCR (polymerase chain reaction) assays are increasinglyused to measure quantities of nucleic acids in samples They may be used to provide ahigh-throughput alternative to more traditional biological assays In this case a calibra-tion process may be required to convert the PCR measurements onto a more relevantscale This is most commonly undertaken using simple linear regression However suchcalibration models are usually unrealistic since they ignore the various sources of varia-tion associated with the PCR and conventional assays Taking account of these varioussources is necessary if the errors associated with predictions based on the calibrationmodel are to be well estimated In this article we demonstrate a more complete approachto calibration of quantitative PCR As an example we develop a Bayesian calibrationmodel for measuring the quantity of the fungus common bunt (Tilletia caries) on wheatseed based on our understanding of the properties of the assays As well as illustrat-ing the steps in developing such a model we show how the fit of the model might beassessed
Key Words Bayesian prediction Measurement error Posterior predictive p valueSeed testing Tilletia caries
1 INTRODUCTION
Modern molecular techniques are increasingly used to detect and quantify nucleic acidsin samples These can provide a high-throughput alternative to traditional biological assaysIn particular quantitative real-time polymerase chain reaction (PCR) assays (EdwardsLogan and Saunders 2004) have been produced for a wide range of applications Examplesinclude plant pathogens in plants (Schena Nigro Ippolito and Gallitella 2004) and soil(Lievens et al 2006) genetically modified organisms (GMO) in food (Kunert et al 2006)human medicine (Raynor et al 2002) and veterinary science (Gallina et al 2006)
The real-time PCR assay measures the quantity of specific nucleic acids in a sample Inmany contexts this may not be the quantity of direct interest for example an estimate ofthe concentration of spores in a soil sample may be required So it is necessary to develop acalibration model to relate the DNA quantity to the level observed by an existing assay based
A M I Roberts is Senior Statistician Biomathematics and Statistics Scotland James Clerk Maxwell BuildingKingrsquos Buildings Edinburgh EH9 3JZ UK (E-mail adrianbiossacuk) C M Theobald is Research LecturerBiomathematics and Statistics Scotland James Clerk Maxwell Building Kingrsquos Buildings Edinburgh EH9 3JZUK M McNeil is Seed Technologist Scottish Agricultural Science Agency 1 Roddinglaw Road EdinburghEH12 9FJ UK
ccopy 2007 American Statistical Association and the International Biometric SocietyJournal of Agricultural Biological and Environmental Statistics Volume 12 Number 3 Pages 364ndash378DOI 101198108571107X227379
364
Calibration of Quantitative PCR Assays 365
on data from a calibration experiment This model can then be used to make predictions ofthe quantity of interest for future cases based on PCR assay measurements
When relating DNA measurements and corresponding quantities of direct interest oftenlogarithmic transformations are used (eg Bowman et al 2001 Brinkman et al 2003 Varmaet al 2003 Winton Manter Stone and Hansen 2003) This is thought to be appropriatewhere measurements can span many orders of magnitude and because this transformationtends to stabilize the PCR measurement error Simple linear regressions are then commonlyused to relate the results of the two assays In our experience a naıve analysis based onsimple linear regression does not provide a sound basis for calibration and prediction Amodel based on biological understanding should provide better predictions than an ldquooff-the-shelfrdquo solution
In McNeil Roberts Cockerell and Mulholland (2004) calibration of a real-time PCRassay to estimate average numbers of fungal spores carried on seed was undertaken usingsimple linear regression It was noted that this was unsatisfactory for various reasons
(a) there was evidence that DNA measurements vary systematically between runs of thePCR assay
(b) the measurements of DNA and of disease level were made on random sub-samplesfrom each seed sample and thus both are subject to sampling variation and
(c) there was evidence that the relationship between the logarithms of the DNA quantityand disease level is not linear across the whole range of interest
In this article we illustrate a more considered approach to calibration We adopt a Bayesianframework this is intended to ensure a valid inferential basis for prediction We referthroughout to an example concerning common bunt (Tilletia caries) an important fungaldisease of wheat that is transmitted via spores attached to the surface of seeds (Cockerellet al 2004)
2 A STRATEGY FOR CALIBRATION
We believe that widely used statistical principles should be adopted for the PCR cali-bration problem rather than applying standard but inappropriate theory A sensible strategywould have the following steps
(i) Consider the objectives of the calibration
(ii) Consider the properties of the PCR and conventional assays and the relationshipbetween them
(iii) Construct a calibration dataset designed to meet the objectives
(iv) Carry out an exploratory analysis of the calibration data
(v) Develop and fit an appropriate model
366 A M I Roberts C M Theobald and M McNeil
(vi) Assess the fit of the model using the calibration set and if possible a test set
(vii) Modify the model if necessary and reassess the fit
3 ASSESSMENT OF BUNT IN SAMPLES OF WHEAT SEED
Common bunt (Tilletia caries) is an important fungal disease of wheat transmitted viaspores attached to the surface of seeds It affects grain yield and more importantly quality(Cockerell et al 2004) Control of the disease relies on treatment of seed with fungicideprior to sowing A strategy of treatment according to need requires an assessment of the levelof contamination Cereal seed growers send samples from their seed lots to a seed testingstation for fungal disease testing The current standard procedure for assessing the level ofbunt contamination of wheat seed is a microscopic assay In this assay the bunt spores arewashed from a subsample of seed the wash is filtered and the spores are then counted onthe filter paper using a microscope Although the microscopic assay can produce resultswithin 24 hours the test is labor-intensive it has a low throughput and the morphologicalidentification of spores relies on the skill of experienced analysts To increase throughputa real-time PCR assay was developed using TaqMan rcopy chemistry to quantify the TilletiaDNA in a sub-sample of seed (McNeil et al 2004)
We have a calibration set based on 108 seed samples chosen to represent a wide range ofcontamination levels rather than being selected completely at random from those receivedThe set has a greater proportion of lots with high levels of contamination than has typicallybeen seen in recent years Samples from these lots were tested in 2003 using the PCR methodin three batches repeated once So in total there were six runs of the PCR assay In each runduplicate extract sub-samples were tested for each of 36 lots these duplicates were placedside-by-side on the PCR block One lot did not have microscopic assay measurements andanother lot was not tested in one PCR run these were omitted from the dataset One lotgave microscopic and PCR assay results that were inconsistent in one run and these werealso omitted from the dataset
We also have a test set to assess the model based on 169 samples received during 2004Again these samples were assessed by the microscopic and PCR assays PCR assays werecarried out over 11 runs as the samples arrived at the seed testing station during the seasonUnlike the calibration set the duplicate extract sub-samples were allocated to the PCRwells according to a two-replicate randomized complete block design This improvementin practice was intended to reduce the clear spatial effects over the PCR block observed inuniformity experiments
4 PROPERTIES OF THE ASSAYS
In developing a calibration model it is valuable to identify the essential statistical
properties of the conventional and PCR assays The form of the relationship between the
two assays should be considered along with their sampling properties and measurement-
error structure Note that in some calibration experiments the PCR assays are run on samples
Calibration of Quantitative PCR Assays 367
from prepared materials with precisely known quantities This is particularly true for GMO
examples where certified reference materials are often available (eg Berdal and Holst-
Jensen 2001) In these cases the problem is simplified since only consideration of the
properties of the PCR assay is required
In our example understanding of the biology indicates that the amount of Tilletia DNA in
a seed sample should be proportional to the number of spores present As mentioned earlier
often logarithmic transformations are used when relating DNA and corresponding pathogen
quantities The proportionality property would then translate into a linear relationship with
slope one This property is applicable to many but not all examples An exception is provided
by Microdochium nivale another fungal disease of wheat which is usually assessed in terms
of the proportion of infected seeds (Mulholland and McEwan 2000) In this case the amount
of DNA per infected seed might be expected to vary between seed lots
In many cases the PCR assay is calibrated by reference to a conventional assay rather
than through use of materials with precisely known quantities It will normally not be
possible to use the same sample for both assays because of their destructive nature Thus
the sampling properties of both need to be considered separately The samples available for
testing are usually taken from a much larger population for example the soil in a field or the
flour in a cargo hold usually it is this population about which we wish to make inferences
not the samples
In the bunt example the seed sample sent by the grower is assumed to be representative
of the seed lot usually comprising several tonnes of seed The seed testing station takes
representative sub-samples from this seed for testing For our purposes we assume that
all sampling is at random In the microscopic assay it is usual to wash and filter three
sub-samples each of 300 seeds On each filter paper the number of spores is estimated by
counting spores in 10 randomly chosen fields of view using a microscope If it is assumed
that the spores are distributed randomly through the seed that the distribution of spores
on the filter paper is random and that all spores are identified correctly then the number
of spores found has a Poisson distribution with expectation Fλ where λ is the number of
spores per seed in the lot The effective sample size factor F depends on the field of view
of the microscope here F is 1207 or 1167 as two microscopes were used
The PCR assay for bunt is also based on a sub-sample of 900 seeds Each T caries spore
contains a single diploid genome Thus the expected number of genomes in the extract
solution is 900λ The extract solution is 600 microL but the extract subsamples taken for PCR
are 1 microL Using assumptions similar to those for the microscopic assay the number of
genomes in an extract subsample will have a Poisson distribution with expectation 15λ
The PCR assay can be expected to be subject to measurement error both within and
between runs in addition to sampling variability Within each run systematic effects across
the PCR block can exist due to temperature gradients In the bunt example DNA measure-
ments are also subject to left censoring since observation of the amplification process is
limited to 40 cycles The censoring point is known but varies slightly from run to run In
the calibration set the censoring points varied from 31 times 10ndash3 ng to 41 times 10ndash3 ng in the
test set they varied from 75 times 10ndash4 ng to 70 times 10ndash3 ng In other examples censoring may
368 A M I Roberts C M Theobald and M McNeil
be a less important issue because samples containing nonzero quantities will produce DNA
measurements well above the censoring point
5 EXPLORATORY ANALYSIS
It is informative to examine the data before fitting a model We illustrate how this may
be done through the bunt example
The bunt calibration and test datasets are illustrated in Figure 1 In these plots the
variability in log DNA measurements appears to be higher at lower contamination levels
Above about 10 sporesseed the variability is more constant This variability for samples
below one sporeseed is similar to that for samples with no spores detected microscopically
These effects are consistent with the greater influence that Poisson sampling variation is
expected to have at low contamination levels under log-transformation
It is worth noting that there are more zero microscopic assay observations than cen-
sored DNA measurements in the calibration set This is unexpected given our assumptions
about the sampling properties of the two assays The probability of a zero microscopic
assay observation is exp(ndashFλ) and the probability of having no spores in the PCR tube
is exp(ndash15λ) Given that F is about 12 there should be more cases where there are no
spores in the PCR tube This indicates that such cases may not always be censored In other
experiments that we have carried out with this PCR assay samples from lots thought to
have zero bunt contamination on the basis of microscopic assay results frequently gave
uncensored DNA measurements Erroneous amplification is known to occur after a large
number of cycles (Mullis Ferreacute and Gibbs 1994) and may be the cause of such anomalies
Discreteness is apparent in the bunt microscopic assay observations at very low con-
tamination levels although modified by the slight differences in microscope fields of view
noted earlier We might expect to find a similar aggregation in the DNA measurements but
there is little evidence of this The lack of aggregation provides evidence of the level of
measurement error in the DNA measurements
Figure 2 shows the differences between the log DNA quantities for the duplicate extract
sub-samples against their means for the two datasets For this plot censored observations
are given the value of their censoring point Again it is apparent that the variability in log
DNA quantities decreases with increasing DNA concentration The variability between the
duplicates is greater for the test set due to the randomized allocation to wells The mean-
variance relationship seen in the log DNA quantities might be explained by the greater
influence of sampling variability at lower contamination levels compared to higher levels
where PCR measurement error is dominant
Evidence for systematic run-to-run variation in DNA measurements was examined by
a linear regression of DNA measurements on contamination level (both log transformed)
allowing for run There were clear differences between runs in the intercepts but not in the
slopes These run-to-run differences are apparent in Figure 1(a) Ignoring this source of
variation would result in incorrect prediction intervals for future runs
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 365
on data from a calibration experiment This model can then be used to make predictions ofthe quantity of interest for future cases based on PCR assay measurements
When relating DNA measurements and corresponding quantities of direct interest oftenlogarithmic transformations are used (eg Bowman et al 2001 Brinkman et al 2003 Varmaet al 2003 Winton Manter Stone and Hansen 2003) This is thought to be appropriatewhere measurements can span many orders of magnitude and because this transformationtends to stabilize the PCR measurement error Simple linear regressions are then commonlyused to relate the results of the two assays In our experience a naıve analysis based onsimple linear regression does not provide a sound basis for calibration and prediction Amodel based on biological understanding should provide better predictions than an ldquooff-the-shelfrdquo solution
In McNeil Roberts Cockerell and Mulholland (2004) calibration of a real-time PCRassay to estimate average numbers of fungal spores carried on seed was undertaken usingsimple linear regression It was noted that this was unsatisfactory for various reasons
(a) there was evidence that DNA measurements vary systematically between runs of thePCR assay
(b) the measurements of DNA and of disease level were made on random sub-samplesfrom each seed sample and thus both are subject to sampling variation and
(c) there was evidence that the relationship between the logarithms of the DNA quantityand disease level is not linear across the whole range of interest
In this article we illustrate a more considered approach to calibration We adopt a Bayesianframework this is intended to ensure a valid inferential basis for prediction We referthroughout to an example concerning common bunt (Tilletia caries) an important fungaldisease of wheat that is transmitted via spores attached to the surface of seeds (Cockerellet al 2004)
2 A STRATEGY FOR CALIBRATION
We believe that widely used statistical principles should be adopted for the PCR cali-bration problem rather than applying standard but inappropriate theory A sensible strategywould have the following steps
(i) Consider the objectives of the calibration
(ii) Consider the properties of the PCR and conventional assays and the relationshipbetween them
(iii) Construct a calibration dataset designed to meet the objectives
(iv) Carry out an exploratory analysis of the calibration data
(v) Develop and fit an appropriate model
366 A M I Roberts C M Theobald and M McNeil
(vi) Assess the fit of the model using the calibration set and if possible a test set
(vii) Modify the model if necessary and reassess the fit
3 ASSESSMENT OF BUNT IN SAMPLES OF WHEAT SEED
Common bunt (Tilletia caries) is an important fungal disease of wheat transmitted viaspores attached to the surface of seeds It affects grain yield and more importantly quality(Cockerell et al 2004) Control of the disease relies on treatment of seed with fungicideprior to sowing A strategy of treatment according to need requires an assessment of the levelof contamination Cereal seed growers send samples from their seed lots to a seed testingstation for fungal disease testing The current standard procedure for assessing the level ofbunt contamination of wheat seed is a microscopic assay In this assay the bunt spores arewashed from a subsample of seed the wash is filtered and the spores are then counted onthe filter paper using a microscope Although the microscopic assay can produce resultswithin 24 hours the test is labor-intensive it has a low throughput and the morphologicalidentification of spores relies on the skill of experienced analysts To increase throughputa real-time PCR assay was developed using TaqMan rcopy chemistry to quantify the TilletiaDNA in a sub-sample of seed (McNeil et al 2004)
We have a calibration set based on 108 seed samples chosen to represent a wide range ofcontamination levels rather than being selected completely at random from those receivedThe set has a greater proportion of lots with high levels of contamination than has typicallybeen seen in recent years Samples from these lots were tested in 2003 using the PCR methodin three batches repeated once So in total there were six runs of the PCR assay In each runduplicate extract sub-samples were tested for each of 36 lots these duplicates were placedside-by-side on the PCR block One lot did not have microscopic assay measurements andanother lot was not tested in one PCR run these were omitted from the dataset One lotgave microscopic and PCR assay results that were inconsistent in one run and these werealso omitted from the dataset
We also have a test set to assess the model based on 169 samples received during 2004Again these samples were assessed by the microscopic and PCR assays PCR assays werecarried out over 11 runs as the samples arrived at the seed testing station during the seasonUnlike the calibration set the duplicate extract sub-samples were allocated to the PCRwells according to a two-replicate randomized complete block design This improvementin practice was intended to reduce the clear spatial effects over the PCR block observed inuniformity experiments
4 PROPERTIES OF THE ASSAYS
In developing a calibration model it is valuable to identify the essential statistical
properties of the conventional and PCR assays The form of the relationship between the
two assays should be considered along with their sampling properties and measurement-
error structure Note that in some calibration experiments the PCR assays are run on samples
Calibration of Quantitative PCR Assays 367
from prepared materials with precisely known quantities This is particularly true for GMO
examples where certified reference materials are often available (eg Berdal and Holst-
Jensen 2001) In these cases the problem is simplified since only consideration of the
properties of the PCR assay is required
In our example understanding of the biology indicates that the amount of Tilletia DNA in
a seed sample should be proportional to the number of spores present As mentioned earlier
often logarithmic transformations are used when relating DNA and corresponding pathogen
quantities The proportionality property would then translate into a linear relationship with
slope one This property is applicable to many but not all examples An exception is provided
by Microdochium nivale another fungal disease of wheat which is usually assessed in terms
of the proportion of infected seeds (Mulholland and McEwan 2000) In this case the amount
of DNA per infected seed might be expected to vary between seed lots
In many cases the PCR assay is calibrated by reference to a conventional assay rather
than through use of materials with precisely known quantities It will normally not be
possible to use the same sample for both assays because of their destructive nature Thus
the sampling properties of both need to be considered separately The samples available for
testing are usually taken from a much larger population for example the soil in a field or the
flour in a cargo hold usually it is this population about which we wish to make inferences
not the samples
In the bunt example the seed sample sent by the grower is assumed to be representative
of the seed lot usually comprising several tonnes of seed The seed testing station takes
representative sub-samples from this seed for testing For our purposes we assume that
all sampling is at random In the microscopic assay it is usual to wash and filter three
sub-samples each of 300 seeds On each filter paper the number of spores is estimated by
counting spores in 10 randomly chosen fields of view using a microscope If it is assumed
that the spores are distributed randomly through the seed that the distribution of spores
on the filter paper is random and that all spores are identified correctly then the number
of spores found has a Poisson distribution with expectation Fλ where λ is the number of
spores per seed in the lot The effective sample size factor F depends on the field of view
of the microscope here F is 1207 or 1167 as two microscopes were used
The PCR assay for bunt is also based on a sub-sample of 900 seeds Each T caries spore
contains a single diploid genome Thus the expected number of genomes in the extract
solution is 900λ The extract solution is 600 microL but the extract subsamples taken for PCR
are 1 microL Using assumptions similar to those for the microscopic assay the number of
genomes in an extract subsample will have a Poisson distribution with expectation 15λ
The PCR assay can be expected to be subject to measurement error both within and
between runs in addition to sampling variability Within each run systematic effects across
the PCR block can exist due to temperature gradients In the bunt example DNA measure-
ments are also subject to left censoring since observation of the amplification process is
limited to 40 cycles The censoring point is known but varies slightly from run to run In
the calibration set the censoring points varied from 31 times 10ndash3 ng to 41 times 10ndash3 ng in the
test set they varied from 75 times 10ndash4 ng to 70 times 10ndash3 ng In other examples censoring may
368 A M I Roberts C M Theobald and M McNeil
be a less important issue because samples containing nonzero quantities will produce DNA
measurements well above the censoring point
5 EXPLORATORY ANALYSIS
It is informative to examine the data before fitting a model We illustrate how this may
be done through the bunt example
The bunt calibration and test datasets are illustrated in Figure 1 In these plots the
variability in log DNA measurements appears to be higher at lower contamination levels
Above about 10 sporesseed the variability is more constant This variability for samples
below one sporeseed is similar to that for samples with no spores detected microscopically
These effects are consistent with the greater influence that Poisson sampling variation is
expected to have at low contamination levels under log-transformation
It is worth noting that there are more zero microscopic assay observations than cen-
sored DNA measurements in the calibration set This is unexpected given our assumptions
about the sampling properties of the two assays The probability of a zero microscopic
assay observation is exp(ndashFλ) and the probability of having no spores in the PCR tube
is exp(ndash15λ) Given that F is about 12 there should be more cases where there are no
spores in the PCR tube This indicates that such cases may not always be censored In other
experiments that we have carried out with this PCR assay samples from lots thought to
have zero bunt contamination on the basis of microscopic assay results frequently gave
uncensored DNA measurements Erroneous amplification is known to occur after a large
number of cycles (Mullis Ferreacute and Gibbs 1994) and may be the cause of such anomalies
Discreteness is apparent in the bunt microscopic assay observations at very low con-
tamination levels although modified by the slight differences in microscope fields of view
noted earlier We might expect to find a similar aggregation in the DNA measurements but
there is little evidence of this The lack of aggregation provides evidence of the level of
measurement error in the DNA measurements
Figure 2 shows the differences between the log DNA quantities for the duplicate extract
sub-samples against their means for the two datasets For this plot censored observations
are given the value of their censoring point Again it is apparent that the variability in log
DNA quantities decreases with increasing DNA concentration The variability between the
duplicates is greater for the test set due to the randomized allocation to wells The mean-
variance relationship seen in the log DNA quantities might be explained by the greater
influence of sampling variability at lower contamination levels compared to higher levels
where PCR measurement error is dominant
Evidence for systematic run-to-run variation in DNA measurements was examined by
a linear regression of DNA measurements on contamination level (both log transformed)
allowing for run There were clear differences between runs in the intercepts but not in the
slopes These run-to-run differences are apparent in Figure 1(a) Ignoring this source of
variation would result in incorrect prediction intervals for future runs
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
366 A M I Roberts C M Theobald and M McNeil
(vi) Assess the fit of the model using the calibration set and if possible a test set
(vii) Modify the model if necessary and reassess the fit
3 ASSESSMENT OF BUNT IN SAMPLES OF WHEAT SEED
Common bunt (Tilletia caries) is an important fungal disease of wheat transmitted viaspores attached to the surface of seeds It affects grain yield and more importantly quality(Cockerell et al 2004) Control of the disease relies on treatment of seed with fungicideprior to sowing A strategy of treatment according to need requires an assessment of the levelof contamination Cereal seed growers send samples from their seed lots to a seed testingstation for fungal disease testing The current standard procedure for assessing the level ofbunt contamination of wheat seed is a microscopic assay In this assay the bunt spores arewashed from a subsample of seed the wash is filtered and the spores are then counted onthe filter paper using a microscope Although the microscopic assay can produce resultswithin 24 hours the test is labor-intensive it has a low throughput and the morphologicalidentification of spores relies on the skill of experienced analysts To increase throughputa real-time PCR assay was developed using TaqMan rcopy chemistry to quantify the TilletiaDNA in a sub-sample of seed (McNeil et al 2004)
We have a calibration set based on 108 seed samples chosen to represent a wide range ofcontamination levels rather than being selected completely at random from those receivedThe set has a greater proportion of lots with high levels of contamination than has typicallybeen seen in recent years Samples from these lots were tested in 2003 using the PCR methodin three batches repeated once So in total there were six runs of the PCR assay In each runduplicate extract sub-samples were tested for each of 36 lots these duplicates were placedside-by-side on the PCR block One lot did not have microscopic assay measurements andanother lot was not tested in one PCR run these were omitted from the dataset One lotgave microscopic and PCR assay results that were inconsistent in one run and these werealso omitted from the dataset
We also have a test set to assess the model based on 169 samples received during 2004Again these samples were assessed by the microscopic and PCR assays PCR assays werecarried out over 11 runs as the samples arrived at the seed testing station during the seasonUnlike the calibration set the duplicate extract sub-samples were allocated to the PCRwells according to a two-replicate randomized complete block design This improvementin practice was intended to reduce the clear spatial effects over the PCR block observed inuniformity experiments
4 PROPERTIES OF THE ASSAYS
In developing a calibration model it is valuable to identify the essential statistical
properties of the conventional and PCR assays The form of the relationship between the
two assays should be considered along with their sampling properties and measurement-
error structure Note that in some calibration experiments the PCR assays are run on samples
Calibration of Quantitative PCR Assays 367
from prepared materials with precisely known quantities This is particularly true for GMO
examples where certified reference materials are often available (eg Berdal and Holst-
Jensen 2001) In these cases the problem is simplified since only consideration of the
properties of the PCR assay is required
In our example understanding of the biology indicates that the amount of Tilletia DNA in
a seed sample should be proportional to the number of spores present As mentioned earlier
often logarithmic transformations are used when relating DNA and corresponding pathogen
quantities The proportionality property would then translate into a linear relationship with
slope one This property is applicable to many but not all examples An exception is provided
by Microdochium nivale another fungal disease of wheat which is usually assessed in terms
of the proportion of infected seeds (Mulholland and McEwan 2000) In this case the amount
of DNA per infected seed might be expected to vary between seed lots
In many cases the PCR assay is calibrated by reference to a conventional assay rather
than through use of materials with precisely known quantities It will normally not be
possible to use the same sample for both assays because of their destructive nature Thus
the sampling properties of both need to be considered separately The samples available for
testing are usually taken from a much larger population for example the soil in a field or the
flour in a cargo hold usually it is this population about which we wish to make inferences
not the samples
In the bunt example the seed sample sent by the grower is assumed to be representative
of the seed lot usually comprising several tonnes of seed The seed testing station takes
representative sub-samples from this seed for testing For our purposes we assume that
all sampling is at random In the microscopic assay it is usual to wash and filter three
sub-samples each of 300 seeds On each filter paper the number of spores is estimated by
counting spores in 10 randomly chosen fields of view using a microscope If it is assumed
that the spores are distributed randomly through the seed that the distribution of spores
on the filter paper is random and that all spores are identified correctly then the number
of spores found has a Poisson distribution with expectation Fλ where λ is the number of
spores per seed in the lot The effective sample size factor F depends on the field of view
of the microscope here F is 1207 or 1167 as two microscopes were used
The PCR assay for bunt is also based on a sub-sample of 900 seeds Each T caries spore
contains a single diploid genome Thus the expected number of genomes in the extract
solution is 900λ The extract solution is 600 microL but the extract subsamples taken for PCR
are 1 microL Using assumptions similar to those for the microscopic assay the number of
genomes in an extract subsample will have a Poisson distribution with expectation 15λ
The PCR assay can be expected to be subject to measurement error both within and
between runs in addition to sampling variability Within each run systematic effects across
the PCR block can exist due to temperature gradients In the bunt example DNA measure-
ments are also subject to left censoring since observation of the amplification process is
limited to 40 cycles The censoring point is known but varies slightly from run to run In
the calibration set the censoring points varied from 31 times 10ndash3 ng to 41 times 10ndash3 ng in the
test set they varied from 75 times 10ndash4 ng to 70 times 10ndash3 ng In other examples censoring may
368 A M I Roberts C M Theobald and M McNeil
be a less important issue because samples containing nonzero quantities will produce DNA
measurements well above the censoring point
5 EXPLORATORY ANALYSIS
It is informative to examine the data before fitting a model We illustrate how this may
be done through the bunt example
The bunt calibration and test datasets are illustrated in Figure 1 In these plots the
variability in log DNA measurements appears to be higher at lower contamination levels
Above about 10 sporesseed the variability is more constant This variability for samples
below one sporeseed is similar to that for samples with no spores detected microscopically
These effects are consistent with the greater influence that Poisson sampling variation is
expected to have at low contamination levels under log-transformation
It is worth noting that there are more zero microscopic assay observations than cen-
sored DNA measurements in the calibration set This is unexpected given our assumptions
about the sampling properties of the two assays The probability of a zero microscopic
assay observation is exp(ndashFλ) and the probability of having no spores in the PCR tube
is exp(ndash15λ) Given that F is about 12 there should be more cases where there are no
spores in the PCR tube This indicates that such cases may not always be censored In other
experiments that we have carried out with this PCR assay samples from lots thought to
have zero bunt contamination on the basis of microscopic assay results frequently gave
uncensored DNA measurements Erroneous amplification is known to occur after a large
number of cycles (Mullis Ferreacute and Gibbs 1994) and may be the cause of such anomalies
Discreteness is apparent in the bunt microscopic assay observations at very low con-
tamination levels although modified by the slight differences in microscope fields of view
noted earlier We might expect to find a similar aggregation in the DNA measurements but
there is little evidence of this The lack of aggregation provides evidence of the level of
measurement error in the DNA measurements
Figure 2 shows the differences between the log DNA quantities for the duplicate extract
sub-samples against their means for the two datasets For this plot censored observations
are given the value of their censoring point Again it is apparent that the variability in log
DNA quantities decreases with increasing DNA concentration The variability between the
duplicates is greater for the test set due to the randomized allocation to wells The mean-
variance relationship seen in the log DNA quantities might be explained by the greater
influence of sampling variability at lower contamination levels compared to higher levels
where PCR measurement error is dominant
Evidence for systematic run-to-run variation in DNA measurements was examined by
a linear regression of DNA measurements on contamination level (both log transformed)
allowing for run There were clear differences between runs in the intercepts but not in the
slopes These run-to-run differences are apparent in Figure 1(a) Ignoring this source of
variation would result in incorrect prediction intervals for future runs
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 367
from prepared materials with precisely known quantities This is particularly true for GMO
examples where certified reference materials are often available (eg Berdal and Holst-
Jensen 2001) In these cases the problem is simplified since only consideration of the
properties of the PCR assay is required
In our example understanding of the biology indicates that the amount of Tilletia DNA in
a seed sample should be proportional to the number of spores present As mentioned earlier
often logarithmic transformations are used when relating DNA and corresponding pathogen
quantities The proportionality property would then translate into a linear relationship with
slope one This property is applicable to many but not all examples An exception is provided
by Microdochium nivale another fungal disease of wheat which is usually assessed in terms
of the proportion of infected seeds (Mulholland and McEwan 2000) In this case the amount
of DNA per infected seed might be expected to vary between seed lots
In many cases the PCR assay is calibrated by reference to a conventional assay rather
than through use of materials with precisely known quantities It will normally not be
possible to use the same sample for both assays because of their destructive nature Thus
the sampling properties of both need to be considered separately The samples available for
testing are usually taken from a much larger population for example the soil in a field or the
flour in a cargo hold usually it is this population about which we wish to make inferences
not the samples
In the bunt example the seed sample sent by the grower is assumed to be representative
of the seed lot usually comprising several tonnes of seed The seed testing station takes
representative sub-samples from this seed for testing For our purposes we assume that
all sampling is at random In the microscopic assay it is usual to wash and filter three
sub-samples each of 300 seeds On each filter paper the number of spores is estimated by
counting spores in 10 randomly chosen fields of view using a microscope If it is assumed
that the spores are distributed randomly through the seed that the distribution of spores
on the filter paper is random and that all spores are identified correctly then the number
of spores found has a Poisson distribution with expectation Fλ where λ is the number of
spores per seed in the lot The effective sample size factor F depends on the field of view
of the microscope here F is 1207 or 1167 as two microscopes were used
The PCR assay for bunt is also based on a sub-sample of 900 seeds Each T caries spore
contains a single diploid genome Thus the expected number of genomes in the extract
solution is 900λ The extract solution is 600 microL but the extract subsamples taken for PCR
are 1 microL Using assumptions similar to those for the microscopic assay the number of
genomes in an extract subsample will have a Poisson distribution with expectation 15λ
The PCR assay can be expected to be subject to measurement error both within and
between runs in addition to sampling variability Within each run systematic effects across
the PCR block can exist due to temperature gradients In the bunt example DNA measure-
ments are also subject to left censoring since observation of the amplification process is
limited to 40 cycles The censoring point is known but varies slightly from run to run In
the calibration set the censoring points varied from 31 times 10ndash3 ng to 41 times 10ndash3 ng in the
test set they varied from 75 times 10ndash4 ng to 70 times 10ndash3 ng In other examples censoring may
368 A M I Roberts C M Theobald and M McNeil
be a less important issue because samples containing nonzero quantities will produce DNA
measurements well above the censoring point
5 EXPLORATORY ANALYSIS
It is informative to examine the data before fitting a model We illustrate how this may
be done through the bunt example
The bunt calibration and test datasets are illustrated in Figure 1 In these plots the
variability in log DNA measurements appears to be higher at lower contamination levels
Above about 10 sporesseed the variability is more constant This variability for samples
below one sporeseed is similar to that for samples with no spores detected microscopically
These effects are consistent with the greater influence that Poisson sampling variation is
expected to have at low contamination levels under log-transformation
It is worth noting that there are more zero microscopic assay observations than cen-
sored DNA measurements in the calibration set This is unexpected given our assumptions
about the sampling properties of the two assays The probability of a zero microscopic
assay observation is exp(ndashFλ) and the probability of having no spores in the PCR tube
is exp(ndash15λ) Given that F is about 12 there should be more cases where there are no
spores in the PCR tube This indicates that such cases may not always be censored In other
experiments that we have carried out with this PCR assay samples from lots thought to
have zero bunt contamination on the basis of microscopic assay results frequently gave
uncensored DNA measurements Erroneous amplification is known to occur after a large
number of cycles (Mullis Ferreacute and Gibbs 1994) and may be the cause of such anomalies
Discreteness is apparent in the bunt microscopic assay observations at very low con-
tamination levels although modified by the slight differences in microscope fields of view
noted earlier We might expect to find a similar aggregation in the DNA measurements but
there is little evidence of this The lack of aggregation provides evidence of the level of
measurement error in the DNA measurements
Figure 2 shows the differences between the log DNA quantities for the duplicate extract
sub-samples against their means for the two datasets For this plot censored observations
are given the value of their censoring point Again it is apparent that the variability in log
DNA quantities decreases with increasing DNA concentration The variability between the
duplicates is greater for the test set due to the randomized allocation to wells The mean-
variance relationship seen in the log DNA quantities might be explained by the greater
influence of sampling variability at lower contamination levels compared to higher levels
where PCR measurement error is dominant
Evidence for systematic run-to-run variation in DNA measurements was examined by
a linear regression of DNA measurements on contamination level (both log transformed)
allowing for run There were clear differences between runs in the intercepts but not in the
slopes These run-to-run differences are apparent in Figure 1(a) Ignoring this source of
variation would result in incorrect prediction intervals for future runs
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
368 A M I Roberts C M Theobald and M McNeil
be a less important issue because samples containing nonzero quantities will produce DNA
measurements well above the censoring point
5 EXPLORATORY ANALYSIS
It is informative to examine the data before fitting a model We illustrate how this may
be done through the bunt example
The bunt calibration and test datasets are illustrated in Figure 1 In these plots the
variability in log DNA measurements appears to be higher at lower contamination levels
Above about 10 sporesseed the variability is more constant This variability for samples
below one sporeseed is similar to that for samples with no spores detected microscopically
These effects are consistent with the greater influence that Poisson sampling variation is
expected to have at low contamination levels under log-transformation
It is worth noting that there are more zero microscopic assay observations than cen-
sored DNA measurements in the calibration set This is unexpected given our assumptions
about the sampling properties of the two assays The probability of a zero microscopic
assay observation is exp(ndashFλ) and the probability of having no spores in the PCR tube
is exp(ndash15λ) Given that F is about 12 there should be more cases where there are no
spores in the PCR tube This indicates that such cases may not always be censored In other
experiments that we have carried out with this PCR assay samples from lots thought to
have zero bunt contamination on the basis of microscopic assay results frequently gave
uncensored DNA measurements Erroneous amplification is known to occur after a large
number of cycles (Mullis Ferreacute and Gibbs 1994) and may be the cause of such anomalies
Discreteness is apparent in the bunt microscopic assay observations at very low con-
tamination levels although modified by the slight differences in microscope fields of view
noted earlier We might expect to find a similar aggregation in the DNA measurements but
there is little evidence of this The lack of aggregation provides evidence of the level of
measurement error in the DNA measurements
Figure 2 shows the differences between the log DNA quantities for the duplicate extract
sub-samples against their means for the two datasets For this plot censored observations
are given the value of their censoring point Again it is apparent that the variability in log
DNA quantities decreases with increasing DNA concentration The variability between the
duplicates is greater for the test set due to the randomized allocation to wells The mean-
variance relationship seen in the log DNA quantities might be explained by the greater
influence of sampling variability at lower contamination levels compared to higher levels
where PCR measurement error is dominant
Evidence for systematic run-to-run variation in DNA measurements was examined by
a linear regression of DNA measurements on contamination level (both log transformed)
allowing for run There were clear differences between runs in the intercepts but not in the
slopes These run-to-run differences are apparent in Figure 1(a) Ignoring this source of
variation would result in incorrect prediction intervals for future runs
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 369
(a)
(b)
Figure 1 DNA measurement versus microscopically measured contamination level for (a) the calibration set and(b) the test set In (a) runs are distinguished by different symbols censored DNA measurements are to the left ofthe vertical dashed line In (b) censored observations are indicated by circles Zero measurements of contaminationlevel are shown separately because of the logarithmic scale
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
370 A M I Roberts C M Theobald and M McNeil
(a)
(b)
Figure 2 Absolute differences in log DNA measurements between duplicates versus their mean for (a) thecalibration set and (b) the test set Vertical dashed lines indicate the range of censoring points
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 371
6 A STATISTICAL MODEL FOR THE CALIBRATION DATA
In building a calibration model we prefer a Bayesian approach because complex models
can be relatively straightforward to fit large sample approximations are avoided prior
information can be included and predictions can incorporate multiple sources of variability
For the bunt example we propose a model that relates both the PCR and microscopic
assay measurements to the seed lot contamination level taking into account the sampling
variation associated with both assays It also allows for measurement error associated with
the PCR assay both within and between runs The adjacent location of duplicate extract
sub-samples on the PCR block in the calibration set is accommodated
The number of spores yj found by the microscopic method is related to the true
contamination level λj (in sporesseed) as described in Section 4 Thus yj is Poisson
distributed with expectation Fjλj where Fj is the effective sample size factor for lot j
We similarly assume that the unknown number of spores in the PCR tube γijk is Poisson-
distributed with expectation 15λj here i indicates the PCR run and k indicates the duplicate
extract sub-sample for seed lot j
The DNA quantity xijk is assumed to be related to γijk by
ln xijk = αi + β ln(γijk + δ
) + εij + ωijk (61)
This is essentially a linear model in ln(γijk +δ) with intercepts αi and common slope β
Although our initial understanding of the biology leads to a slope of one (and δ equal to zero)
we find that a much better fit can be achieved with this more general model The parameter
δ is included to allow an upward bias in DNA measurements at low contamination levels
(see Section 5) and is restricted to be positive Run-to-run variation is modeled by αi and
ωijk and εij represent the measurement error between adjacent wells (when duplicates are
placed side-by-side) and the wider between-well error respectively The terms αi εij and
ωijk are assumed to be N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) and to be mutually independent
As for any Bayesian analysis we need to specify prior distributions for model parame-
ters These are informed by consideration of the scales of the measurements In this article
we specify gamma distributions Ga(a b) so that the mean is ab and the variance is ab2
We follow Theobald and Talbot (2002) by specifying inverse gamma prior distributions for
the variances σ 2ε and σ 2
α in terms of prior estimates of their values with corresponding de-
grees of freedom representing our confidence in these estimates Based on past experience
we chose a value of 033 with five degrees of freedom for both variances This translates
into Ga(25 083) prior distributions for σminus2ε and σminus2
α The variance σ 2 should be some-
what less than σ 2ε so we give a beta prior distribution to σ 2σ 2
ε with parameters 15 and
3 The slope β is expected to be close to one it is given a N(1 01) prior distribution
corresponding to a prior 95 interval of (038 162) Parameters microα and ln δ are given
N(0 100) and N(ndash4 10) prior distributions
The model under consideration can be fitted using Markov chain Monte Carlo (MCMC)
methods (Gilks Richardson and Spiegelhalter 1996) We use the WinBUGS program
(Spiegelhalter Thomas Best and Lunn 2004) which is freely available from http www
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
372 A M I Roberts C M Theobald and M McNeil
Table 1 Posterior means and percentiles for model parameters
PercentilesParameter Mean 25 50 975
microα minus118 minus160 minus118 minus076β 0871 0838 0871 0905
lnδ minus913 minus1276 minus889 minus680σ 2α 0252 0097 0216 0622
σ 2ε 0220 0144 0215 0328
σ 2 00144 00078 00138 00239
mrc-bsucamacuk bugs The model can be represented in WinBUGS as a directed acyclic
graph (DAG) summarizing the conditional independence relations between the model pa-
rameters The full joint distribution for the model then has a simple factorization which
is exploited in MCMC computations the software chooses the most appropriate updating
procedure It should be possible to represent other PCR calibration models as DAGs and
then to fit them using WinBUGS or similar software
In Section 4 we noted that the PCR measurements are censored below a known DNA
quantity ci say dependent only on the run i WinBUGS allows straightforward incorporation
of such censored values
To assess convergence of the MCMC simulation we used the method proposed by
Gelman and Rubin (1992) and modified by Brooks and Gelman (1998) This compares the
chains starting from dispersed initial values We found a burn-in of 10000 iterations to be
comfortably sufficient For model fitting 100000 MCMC iterations were performed after
the burn-in
The posterior distributions for the bunt model parameters are summarized in Table 1
showing the mean the median and the 25 and 975 percentiles The posterior distribution
for β suggests that a simpler model based on proportionality between the DNA quantity
and the number of spores present (so β = 1) is not supported by the experimental data
Even with δ in the model (61) there are fewer censored PCR observations than might be
expected When the number of spores in the PCR tube γijk is zero the fitted model suggests
that the log DNA measurement would be censored with posterior probability 09975 (when
the censoring point is 37 times 10ndash3 ng) This is not consistent with the observations made in
Section 5 on the numbers censored When γijk is greater than 0 δ has little influence on
the expected log DNA quantity the DNA measurement is unlikely to be censored Even
when γijk is 1 the posterior probability that the DNA measurement is censored is less than
00001
7 PREDICTION USING THE CALIBRATION MODEL
The calibration model is fitted to allow estimation of the quantity of interest for a new
population based on single or multiple DNA measurements Specification of a predictive
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 373
model is straightforward once the calibration model is defined We show how this is done
for the bunt example For this example there would normally be DNA measurements from
duplicates within a single PCR run for simplicity the following notation is based on an
individual measurement
In the bunt example we wish to predict the true contamination level λlowast say in a new
seed lot based on a DNA measurement xlowast in a fresh PCR run We assume that the unknown
number of spores in the PCR tube γlowast is Poisson-distributed with rate 15λlowast By analogy
with (61) the DNA measurement xlowast is assumed to be related to γlowast by
ln xlowast = αlowast + βln (γlowast + δ) + εlowast + ωlowast (71)
Similarly the terms αlowast εlowast and ωlowast are N(microα σ 2α ) N(0 σ 2
ε ) and N(0 σ 2) respectively
Aitchison and Dunsmore (1975) distinguished two types of calibration experiment
ldquodesignedrdquo experiments in which the samples used are chosen by the investigator and
ldquonaturalrdquo experiments in which the recorded values represent the distribution to be expected
in future In our more complex example values are unable to be chosen for either variable
Nevertheless we treat it as a designed calibration experiment since the samples used were
chosen to cover a wider range of contamination levels than would normally be expected
In this case we must specify a prior density for λlowast in future lots for natural experiments
the data itself provides information about the distribution of λlowast We have six years of
records of microscopic assay results available from routine tests that we can use to inform
the choice of prior these cover 582 lots We assume a gamma distribution for the true
contamination levels of these lots so that the microscopic measurements have a negative
binomial marginal distribution Maximum likelihood estimates for the mean and variance of
the gamma distribution are 026 and 085 corresponding to the distribution Ga(0080 031)
However we note that disease levels were low in these six years compared to the experience
of the seed testing station over a longer period The gamma distribution gives little credence
to high contamination levels having a median of 00003 sporesseed and an upper 5 point
of 15 sporesseed Far higher levels of contamination are possible as evidenced by the
levels seen in the calibration set We therefore chose the prior distribution for λlowast by using
the same lower 5 point but increasing the upper 5 point to 1500 sporesseed This
corresponds to a Ga(0068 000025) distribution that has a median of 009 sporesseed
For prediction in the bunt example up to 5000000 MCMC iterations were performed
after the burn-in The median and the lower and upper 25 percentiles from the posterior
predictive distributions of seed lot contamination level for single DNA measurements be-
tween 10ndash3 ng and 1000 ng are shown in Figure 3 These distributions are right skewed on
the natural scale
Predictions based on duplicate DNA measurements are more difficult to illustrate They
are also dependent on the layout of extract sub-samples on the PCR block The poste-
rior predictive intervals will be narrower when the locations on the PCR block are ran-
domised compared to when the duplicates are placed side-by-side because the influence of
the between-well measurement error is reduced
As well as making predictions the model can be used to assess the probability that the
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
374 A M I Roberts C M Theobald and M McNeil
Figure 3 Posterior predictive medians and lower and upper 25 percentiles of true seed lot contamination levelsfrom a single DNA measurements showing observations from the calibration set Censored observations are tothe left of the dashed line Zero microscopic measurements are shown separately
population quantity falls below a given threshold This is straightforward using the Bayesian
approach In the bunt example seed lots with DNA measurements below 016 ng would
be 95 certain to have a seed lot contamination level below 2 sporesseed This type of
assessment might be used in decisions as to whether to treat seed with fungicide
8 ASSESSING THE FIT OF THE MODEL
Once a model has been specified it is good practice to assess how well it fits the data
We adopt the Bayesian model-checking procedure of Gelman Carlin Stern and Rubin
(2004) in which simulations from the posterior predictive distribution for the quantity of
interest are compared with actual observations In the bunt example the posterior predictive
distribution for the number of spores found by the microscopic assay yplowast say is generated
from duplicate DNA observations xlowast from an actual seed sample This is compared to
the corresponding number of spores found by the microscopic method yalowast Because of the
discreteness of the microscopic assay we define the posterior predictive p value as
Pr(y
plowast gt yalowast |xlowast x y) + 1
2Pr
(y
plowast = yalowast |xlowast x y) (81)
This is analogous to the mid-p value advocated for significance tests for categorical
data (Agresti 2002)
Figure 4 shows plots of the posterior predictive p values against the medians of the
posterior predictive distributions of yplowast for the calibration set and for the test set We think that
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 375
(a)
(b)
Figure 4 Posterior predictive p values plotted against the medians of the posterior predictive distributions formicroscopic observations of contamination level for (a) the calibration set and (b) the test set Medians which arezero are shown separately
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
376 A M I Roberts C M Theobald and M McNeil
the fit is reasonable allowing for the effect of discreteness at low concentrations However
we note in Section 5 that there should be many more cases where there are no spores in the
tube than suggested by the number of censored measurements This phenomenon may be
explained by erroneous amplification after a large number of cycles (Mullis et al 1994) We
would claim that as long as the model produces reasonable predictions of contamination
level from DNA measurements as evidenced above then it is fit for purpose In particular
the fit is most important in the range 05 to 10 sporesseed where decisions to treat are likely
to be made
Figure 3 shows that predicted contamination levels for low levels of DNA are imprecise
This is because sampling variation associated with low contamination levels dominates
other sources of variation The predicted levels are also sensitive to the choice of prior
distribution for λlowast for low levels of DNA To illustrate this consider a simplified case
where we know that there are zero spores in a tube Given a Ga(a b) prior distribution
for λlowast the corresponding posterior distribution is Ga(a b + 1) With our chosen prior
distribution the posterior distribution would be Ga(0068 100025) whereas the Jeffreyrsquos
prior distribution (proportional to λminus12lowast ) gives a Ga(05 1) posterior distribution The effect
of the choice of prior distribution diminishes as the contamination level increases
9 DISCUSSION
We have demonstrated how widely advocated principles for fitting statistical models to
data can be applied to a PCR calibration problem These include that the model should be
based on careful consideration of the statistical properties of the system generating the data
and that the fit of the model should be reviewed We have also advocated that a Bayesian
approach is to be preferred particularly for calibration with more complex models
We have used a PCR assay designed to quantify bunt contamination to illustrate the
application of these principles Although the model developed is specific to this example
many of the underlying ideas should be applicable to other calibration problems in which
PCR replaces a traditional method of measurement
Our model for predicting bunt contamination levels in seed lots takes into account
four sources of variation This certainly improves on the use of linear regression of the
logarithm of the measured contamination level on the logarithm of the DNA measurement
Such a linear regression over-predicts for low quantities of DNA and under-predicts for
high quantities It also ignores variation between runs The prediction intervals based on
simple linear regression are also fairly constant in width (on the log-scale) over the range
They are much wider at high DNA quantities than those derived from the model proposed
here which is intended to take proper account of sampling variation in both measurements
Predictions from such calibration models might be used to decide whether to treat seed
with fungicide In principle this could be formalized using Bayesian decision theory In
practice at least for the bunt example the cost of not treating seed is complex to specify
(Cockerell et al 2004) and may involve financial and environmental considerations that
cannot easily be integrated The cost of an incorrect decision also depends on the point of
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
Calibration of Quantitative PCR Assays 377
view farmers may be most interested in the grain yield and quality on their farm whereas
an agronomist taking a national view might be concerned that contamination levels do not
build up in the crop as a whole
ACKNOWLEDGMENTS
This work was partly funded by the Scottish Executive Environment and Rural Affairs Department The authorsare grateful to HGCA for permission to use the data They would also like to thank Valerie Cockerell and VinceMulholland of the Scottish Agricultural Science Agency for valuable discussions and the referees for helpfulcomments
[Received December 2005 Revised March 2007]
REFERENCES
Agresti A (2002) Categorical Data Analysis (2nd ed) New York Wiley
Aitchison J and Dunsmore I R (1975) Statistical Prediction Analysis Cambridge Cambridge University Press
Berdal K G and Holst-Jensen A (2001) ldquoRoundup Ready rcopy Soybean Event-Specific Real-Time QuantitativePCR Assay and Estimation of the Practical Detection and Quantification Limits in GMO Analysesrdquo EuropeanFood Research and Technology 213 432ndash438
Bowman J C Abruzzo G K Anderson J W Flattery A M Gill C J Pikounis V B Schmatz DM Liberator P A and Douglas C M (2001) ldquoQuantitative PCR Assay to Measure Aspergillus fumigatusBurden in a Murine Model of Disseminated Aspergillosis Demonstration of Efficacy of Caspofungin AcetaterdquoAntimicrobial Agents and Chemotherapy 45 3474ndash3481
Brinkman N E Haugland R A Wymer L J Byappanahalli M Whitman R L and Vesper S J (2003)ldquoEvaluation of a Rapid Quantitative Real-Time PCR Method for Enumeration of Pathogenic Candida Cellsin Waterrdquo Applied and Environmental Microbiology 69 1775ndash1782
Brooks S P and Gelman A (1998) ldquoGeneral Methods for Monitoring Convergence of Iterative SimulationsrdquoJournal of Computational and Graphical Statistics 7 434ndash457
Cockerell V Paveley N D Clark W S Thomas J E Anthony S McEwan M Bates J Roberts A M ILaw J R Kenyon D M and Mulholland V (2004) ldquoCereal Seed Health and Seed Treatment StrategiesExploiting New Seed Testing Technology to Optimise Seed Health Decisions for Wheatrdquo Project Report 340Home-Grown Cereals Authority London
Edwards K J Logan J M J and Saunders N A (2004) Real-time PCR an Essential Guide WymondhamHorizon Bioscience
Gallina L Dal Pozzo F Mc Innes C J Cardeti G Guercio A Battilani M Ciulli S and Scagliarini A(2006) ldquoA Real Time PCR Assay for the Detections and Quantification of Orf Virusrdquo Journal of VirologicalMethods 134 140ndash145
Gelman A Carlin J B Stern H S and Rubin D B (2004) Bayesian Data Analysis (2nd ed) Boca RatonChapman and Hall
Gelman A and Rubin D B (1992) ldquoInference From Iterative Simulation Using Multiple Sequencesrdquo StatisticalScience 7 457ndash511
Gilks W R Richardson S and Spiegelhalter D J (1996) Markov Chain Monte Carlo in Practice LondonChapman and Hall
Kunert R Gach J S Vorauer-Uhl K Engel E and Katinger H (2006) ldquoValidated Method for Quantificationof Genetically Modified Organisms in Samples of Maize Flourrdquo Journal of Agricultural and Food Chemistry54 678ndash681
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126
378 A M I Roberts C M Theobald and M McNeil
Lievens B Brouwer M Vanachter A C R C Cammue B P A and Thomma B P H J (2006) ldquoReal-TimePCR for Detection and Quantification of Fungal and Oomycete Tomato Pathogens in Plant and Soil SamplesrdquoPlant Science 171 155ndash165
McNeil M Roberts A M I Cockerell V and Mulholland V (2004) ldquoReal-Time PCR Assay for Quantificationof Tilletia caries Contamination of UK Wheat Seedrdquo Plant Pathology 53 741ndash750
Mulholland V and McEwan M (2000) ldquoPCR-Based Diagnostics of Microdochium nivale and Tilletia triticiInfecting Winter Wheat Seedsrdquo EPPO Bulletin 30 543ndash547
Mullis K B Ferre F and Gibbs R A (eds) (1994) The Polymerase Chain Reaction Boston Birkhauser
Raynor M Stephenson S Walsh D C A Pittman K B and Dobrovic A (2002) ldquoOptimisation of theRT-PCR Detection of Immunomagnetically Enriched Carcinoma Cellsrdquo BMC Cancer 2 14
Schena L Nigro F Ippolito A and Gallitella D (2004) ldquoReal-Time Quantitative PCR A New Technologyto Detect and Study Phytopathogenic and Antagonistic Fungirdquo European Journal of Plant Pathology 110893ndash908
Spiegelhalter D Thomas A Best N and Lunn D (2004) WinBUGS User Manual (Version 141) CambridgeMedical Research Council Biostatistics Unit
Theobald C M and Talbot M (2002) ldquoThe Bayesian Choice of Crop Variety and Fertilizer Doserdquo AppliedStatistics 51 23ndash36
Varma M Hester J D Schaefer F W Ware M W and Lindquist H D A (2003) ldquoDetection of Cyclosporacayetanensis using a Quantitative Real-Time PCR Assayrdquo Journal of Microbiological Methods 53 27ndash36
Winton L M Manter D K Stone J K and Hansen E A (2003) ldquoComparison of Biochemical Molecularand Visual Methods to Quantify Phaeocryptopus gaeumannii in Douglas-Fir Foliagerdquo Phytopathology 93121ndash126