Embed Size (px)
Citation preview
Journal of Immunological Methods 352 (2010) 23–31
Contents lists available at ScienceDirect
Journal of Immunological Methods
j ourna l homepage: www.e lsev ie r.com/ locate / j im
Research paper
Neurofilament ELISA validation
Axel Petzold a,⁎, Ayse Altintas b, Laura Andreoni c, Ales Bartos d,e, Achim Berthele f,Marinus A. Blankenstein g, Luc Buee h, Massimiliano Castellazzi i, Sabine Cepok f,Manuel Comabella j, Cris S. Constantinescu k, Florian Deisenhammer l, Gunnur Denizm,Gaye Ertenm, Mercedes Espiño n, Enrico Fainardi o, Diego Franciotta c, Mark S. Freedman p,Vilmantas Giedraitis y, Nils Erik Gilhus q,r, Gavin Giovannoni s, Andrzej Glabinski t, Pawel Grieb u,Hans-Peter Hartung v, Bernhard Hemmer f, Sanna-Kaisa Herukkaw, Rogier Hintzen x,Martin Ingelsson y, Samuel Jackson s, Steve Jacobsen z, Naghmeh Jafari x, Marcin Jalosinski t,Sven Jarius aa, Elisabeth Kapaki ab, Bernd C. Kieseier v, Marleen J.A. Koel-Simmelink ac,Johannes Kornhuber ad, Jens Kuhle ae, Jacek Kurzepa af, Patrice H. Lalive ag,ah, Lars Lannfelt y,Vera Lehmensiek ai, Piotr Lewczuk ad, Paolo Livrea aj, Fabiana Marnetto ak, Davide Martino aj,Til Menge v, Niklas Norgren al, Eva Papuć af, George P. Paraskevas ab, Tuula Pirttilä w,Cecília Rajda am, Konrad Rejdak af, Jan Ricny d, Daniela Ripova d, Lars Rosengren an,Maddalena Ruggieri aj, Susanna Schraen h, Gerry Shaw ao, Christian Sindic ap, Aksel Siva b,Torgny Stigbrand aq, Iva Stonebridge p, Baris Topcular ar, Maria Trojano aj, Hayrettin Tumani ai,Harry A.M. Twaalfhoven g, László Vécsei am, Vincent Van Pesch ap, Hugo Vanderstichele as,Christian Vedeler q,at, Marcel M. Verbeek au, Luisa Maria Villar n, Robert Weissert av,Brigitte Wildemann aa, Cui Yang ao, Karen Yao z, Charlotte E. Teunissen ac
a Department of Neuroinflammation, UCL Institute of Neurology, Queen Square, London WC1N 3BG, United Kingdomb Istanbul University, Cerrahpasa Medical Faculty, Department of Neurology, Istanbul, Turkeyc Lab. of Neuroimmunology, IRCCS, Neurological Institute ‘C. Mondino’, I-27100 Pavia, Italy
⁎ Corresponding author. Tel.: +44 20 7837 3611x4204; fax: +44 20 7837 8553.E-mail addresses: [email protected] (A. Petzold), [email protected] (A. Altintas), [email protected] (L. Andreoni), [email protected]
(A. Bartos), [email protected] (A. Berthele), [email protected] (M.A. Blankenstein), [email protected] (L. Buee),[email protected] (M. Castellazzi), [email protected] (S. Cepok), [email protected] (M. Comabella),[email protected] (C.S. Constantinescu), [email protected] (F. Deisenhammer), [email protected] (G. Deniz),[email protected] (G. Erten), [email protected] (M. Espiño), [email protected] (E. Fainardi), [email protected] (D. Franciotta),[email protected] (M.S. Freedman), [email protected] (V. Giedraitis), [email protected] (N.E. Gilhus),[email protected] (G. Giovannoni), [email protected] (A. Glabinski), [email protected] (P. Grieb), [email protected](H.-P. Hartung), [email protected] (B. Hemmer), [email protected] (S.-K. Herukka), [email protected] (R. Hintzen),[email protected] (M. Ingelsson), [email protected] (S. Jackson), [email protected] (S. Jacobsen), [email protected] (N. Jafari),[email protected] (M. Jalosinski), [email protected] (S. Jarius), [email protected] (E. Kapaki), [email protected](B.C. Kieseier), [email protected] (M.J.A. Koel-Simmelink), [email protected] (J. Kornhuber), [email protected] (J. Kuhle),[email protected] (J. Kurzepa), [email protected] (P.H. Lalive), [email protected] (L. Lannfelt), [email protected] (V. Lehmensiek),[email protected] (P. Lewczuk), [email protected] (P. Livrea), [email protected] (F. Marnetto), [email protected] (D. Martino),[email protected] (T. Menge), [email protected] (N. Norgren), [email protected] (E. Papuć), [email protected](G.P. Paraskevas), [email protected] (T. Pirttilä), [email protected] (C. Rajda), [email protected] (K. Rejdak), [email protected] (J. Ricny),[email protected] (D. Ripova), [email protected] (L. Rosengren), [email protected] (M. Ruggieri), [email protected] (S. Schraen),[email protected] (G. Shaw), [email protected] (C. Sindic), [email protected] (A. Siva), [email protected] (T. Stigbrand), [email protected](I. Stonebridge), [email protected] (B. Topcular), [email protected] (M. Trojano), [email protected] (H. Tumani),[email protected] (H.A.M. Twaalfhoven), [email protected] (L. Vécsei), [email protected] (V. Van Pesch),[email protected] (H. Vanderstichele), [email protected] (C. Vedeler), [email protected] (M.M. Verbeek),[email protected] (L.M. Villar), [email protected] (R. Weissert), [email protected] (B. Wildemann), [email protected](C. Yang), [email protected] (K. Yao), [email protected] (C.E. Teunissen).
0022-1759/$ – see front matter © 2009 Elsevier B.V. All rights reserved.doi:10.1016/j.jim.2009.09.014
24 A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
d AD Center, Prague Psychiatric Center, Prague, Czech Republice Charles University in Prague, Third Faculty of Medicine, University Hospital Kralovske Vinohrady, Department of Neurology, Prague, Czech Republicf Department of Neurology, Klinikum rechts der Isar, Technische Universität Muenchen, Ismaninger Strase 22, 81675 München, Germanyg Department of Clinical Chemistry, VU University Medical Center, De Boelelaan 1117, Amsterdam, The Netherlandsh Inserm U837 - JPArc Alzheimer & Tauopathies Université Lille 2 Faculté de Médecine-Pôle Recherche Institut de Médecine Prédictive et Recherche Thérapeutique 59045Lille, Francei Laboratorio di Neurochimica, Sezione di Clinica Neurologica, Dipartimento di Discipline Medico Chirurgiche della Comunicazione e del Comportamento Università degliStudi di Ferrara, Ferrara, Italyj Department of Neurology, Centre d'Esclerosi Múltiple de Catalunya, Unitat de Neuroimmunologia Clínica, Hospital Universitari Vall d'Hebron, Barcelona, Spaink Division of Clinical Neurology, University of Nottingham, Medical School, Nottingham, UKl Clinical Dept. of Neurology, Innsbruck Medical University, Anichstrasse 35, A-6020 Innsbruck, Austriam Istanbul University, Institute for Experimental Medicine, Department of Immunology, Istanbul, Turkeyn Immunology Department, Hospital Ramon y Cajal, Madrid, Spaino Neuroradiology Unit, Azienda Ospedaliero-Universitaria, Ferrara, Italyp Ottawa Hospital Research Institute, Ottawa Hospital–General Campus, Ottawa, Ontario, Canadaq Department of Clinical Medicine, University of Bergen, Norwayr Department of Neurology, Haukeland University Hospital, Bergen, Norways Queen Mary University of London, Blizard Institute of Cell Molecular Science, Barts and The London School of Medicine and Dentistry, London, UKt Department of Propedeutics of Neurology, Medical University of Lodz, Kopernik Hospital, Pabianicka Str. 62, Lodz, Polandu Medical Research Center, Polish Academy of Sciences, 02-106 Warsaw, Pawinskiego 5, Polandv Department of Neurology, Heinrich-Heine University, Duesseldorf, Germanyw Department of Neurology, University of Kuopio and Kuopio University Hospital, P.O.Box 1627, 70211 Kuopio, Finlandx The Department of Neurology, MS Centre ErasMS, Erasmus MC, Rotterdam, The Netherlandsy Uppsala University, Dept. of Public Health / Geriatrics, Rudbeck Laboratory, Dag Hammarskjölds väg 20, 751 85 Uppsala, Swedenz NINDS/NIH, Viral Immunology Section, 9000 Rockville Pike, Building 10, United Statesaa Division of Molecular Neuroimmunology, Department of Neurology, University of Heidelberg, Germanyab Department of Neurology, Athens National University, “Eginition” Hospital, 74 Vass. Sofias Ave., Athens 11528, Greeceac NeuroUnit Biomarkers for Inflammation and Neurodegeneration, Department of Molecular Cell Biology and Immunology, VU University Medical Center, Amsterdam,The Netherlandsad Department of Psychiatry and Psychotherapy, Universitaesklinikum Erlangen, Germanyae Neurology and Clinical Neuroimmunology, University Hospital, University of Basel, Switzerlandaf Department of Neurology, Medical University of Lublin, 20-954 Lublin, Jaczewskiego 8, Polandag Department of Neurosciences, Division of Neurology, Neuroimmunology Laboratory, University Hospital of Geneva and Faculty of Medicine, Geneva, Switzerlandah Department of Genetics and Laboratory Medicine, Laboratory Medicine Service, University Hospital of Geneva and Faculty of Medicine, Geneva, Switzerlandai Department of Neurology, University of Ulm, 89081, Ulm, Germanyaj Dept. of Neurological and Psychiatric Sciences, Policlinico-Piazza G. Cesare, 11 70100, University of Bari, Italyak Multiple Sclerosis Center ASO San Luigi Gonzaga Orbassano (Turin), Italyal UmanDiagnostics, Umea Biotech Incubator, Tvistevägen 48, SE-90736 Umea, Swedenam Department of Neurology, University of Szeged, Semmelweis u.6., H-6725 Szeged, Hungaryan Department of Neurology, Institute of Clinical Neuroscience, Sahlgrenska University Hospital, University of Göteborg, SE-413 45 Gothenborg, Swedenao Department of Neuroscience, McKnight Brain Institute, 100 Newell Drive, Gainesville, Florida 32610, United Statesap Neurochemistry Unit, NCHM - Avenue Hippocrate, 10 (boite 1352), 1200 BXL, Brussels, Belgiumaq Department of Immunology, University of Umea, SE-90187 Umea, Swedenar Bakirkoy Prof.Dr. Mazhar Osman Teaching and Research Hospital for Mental Health and Neurological Disorders, Istanbul, Turkeyas Innogenetics NV, BTW BE 0427.550.660, RPR Gent, Technologiepark 6, B-9052 Gent, Belgiumat Department of Neurology, Haukeland University Hospital, Bergen, Norwayau Radboud University Nijmegen Medical Centre, Donders Centre for Brain, Cognition and Behaviour, Department of Neurology, Laboratory of Pediatrics and Neurology,Alzheimer Centre Nijmegen, The Netherlandsav Division of Neurology, Department of Neurosciences, University of Geneva, Geneva, Switzerland
a r t i c l e i n f o
a b s t r a c tArticle history:Received 3 May 2009Received in revised form 28 September 2009Accepted 29 September 2009Available online 24 October 2009
Background: Neurofilament proteins (Nf) are highly specific biomarkers for neuronal deathand axonal degeneration. As these markers become more widely used, an inter-laboratoryvalidation study is required to identify assay criteria for high quality performance.
Methods: The UmanDiagnostics NF-light ®enzyme-linked immunoabsorbent assays (ELISA)for the neurofilament light chain (NfL, 68 kDa) was used to test the intra-assay and inter-laboratory coefficient of variation (CV) between 35 laboratories worldwide on 15 cerebrospinalfluid (CSF) samples. Critical factors, such as sample transport and storage, analytical delays,reaction temperature and time, the laboratories' accuracy and preparation of standards weredocumented and used for the statistical analyses.
Results: The intra-laboratory CV averaged 3.3% and the inter-laboratory CV 59%. The resultsfrom the test laboratories correlated with those from the reference laboratory (R=0.60,pb0.0001). Correcting for critical factors improved the strength of the correlation.Differences in the accuracy of standard preparation were identified as the most criticalfactor. Correcting for the error introduced by variation in the protein standards improved thecorrelation to R=0.98, pb0.0001 with an averaged inter-laboratory CV of 14%. The correctedoverall inter-rater agreement was subtantial (0.6) according to Fleiss' multi-rater kappa andGwet's AC1 statistics.
Keywords:NeurofilamentNfLNF-LNEFLBiomarkerValidation
25A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
Conclusion: This multi-center validation study identified the lack of preparation of accurate andconsistent protein standards as the main reason for a poor inter-laboratory CV. This issue is alsorelevant to other protein biomarkers based on this type of assay andwill need to be solved inorderto achieve an acceptable level of analytical accuracy. The raw data of this study is available online.
© 2009 Elsevier B.V. All rights reserved.
1. Introduction
Protein biomarkers are relevant for diagnostics andprognosis and they are increasingly used as inclusion criteriaand as a secondary outcome measure for treatment trials.With the availability of high-throughput proteomic methodslarge numbers of potential protein biomarkers have beendiscovered. However, surprisingly few withstand the test oftime and bridge the bench to bedside gap. Two exampleswithin the field of neurodegenerative diseases are the tau andamyloid-beta1–42 proteins. Both proteins were found to bekey components in the pathology of Alzheimer's disease.Several assays were developed, of which the enzyme-linkedimmunoabsorbent assays (ELISA) were employed routinely.Evidence emerged that cerebrospinal fluid (CSF) tau andamyloid-beta1–42 levels could be good diagnostic tests forAlzheimer's disease (Blennow et al., 2006; Lewczuk andWiltfang, 2008) and new diagnostic criteria including CSF tauand amyloid-beta1–42 levels were proposed (Dubois et al.,2007). Disappointingly, repeated validation studies of CSF tauand amyloid-beta1–42 levels demonstrated an unacceptableinter-laboratory coefficient of variation (CV) of up to 53%(Blankenstein et al., 2005; Lewczuk et al., 2006; Verwey et al.,2009). The factors influencing the CV remain largelyunknown.
The present study was designed to measure the quantita-tive influence of a number of controllable parameters in amulti-center study involving 35 laboratories worldwide. Therationale was to control for parameters inherent to shipmentand handling of test material and related to differentlaboratory procedures in order to optimize the inter-labora-tory CV.
The biomarker chosen for analysis was the neurofilamentlight protein (NfL) (Rosengren et al., 1996; Van Geel et al.,2005). Neurofilaments are heteropolymers composed of atleast four subunit proteins: α-internexin/Nf66 and the tripletof the neurofilament light, medium (NfM) and heavy chain(NfH) (Shaw, 1998; Petzold, 2005). Neurofilament proteinsare the most reliable protein biomarker validated forquantification of neurodegeneration (Petzold, 2005; Teunis-sen et al., 2005). Their emerging importance for basic science,clinical trials and laboratory practice cannot be overestimated.
1 The volume of stock solution for the two freeze-dried NfL standardsmounts to 480 µL.
2. Materials and methods
2.1. Kits and equipment
The NF-L kits were obtained from UmanDiagnostics, Umea,Sweden. The kit is a sandwich ELISA with two highly specificNF-L monoclonal antibodies and no cross-reactivity with otherknown brain antigens (Norgren et al., 2002, 2003, 2004).
Large thermometers, allowing easy visualisation of thetemperature (Table 1), were purchased from the LabWarehouse (TG835-15). All thermometers were calibratedin London prior to shipment.
2.2. CSF samples
Fifteen CSF samples (50 mL each sample, collected frompatients who underwent extraventricular drainage formanagement of acute hydrocephalus) were collected inpolypropylene tubes and immediately centrifuged(5000 rpm for 5min). Supernatants were aliquoted andstored at −80 °C in 1.5–2.0 mL Eppendorf tubes untilanalysis. In agreement with the Ethic Committee and theUnited Kingdom Human Tissue Act, all patient details wereanonymised.
2.3. Analytical procedures
The ELISA (UmanDiagnostics NF-light ®) was performedas previously described (Norgren et al., 2003). In brief, eachlaboratory reconstituted each of the freeze-dried NfL stan-dards (purified bovine spinal cord with a purity N98%) with240 µL of sample diluent in a 1.5 mL Eppendorf vial to give aconcentration of 40 ng/mL.1 The vial was gently mixed untilthe standard was completely solubilized and then left tostand at room temperature (RT) for 10min. A series of eightEppendorf vials was prepared, with the first vial containing900 µL of sample diluent and vials 2 to 8, 600 µL of samplediluent. Each laboratory then added 300 µL of the lyophilizedstandard to the first vial to give a concentration of 10 µg/L. Adoubling dilution using 600 µL in each step was performedand vial 8 was left only with sample diluent to give the blankreading (0 ng/mL NfL). The concentration for vials 1 to 7 issummarised in Table 3.
The freeze-dried ELISA plates were washed with 3×5-minute cycles with washing solution (300 µL). Next, 100 µL ofstandards and samples were added in quadruplicate (= 4measurements) and incubated for 1h at RT at 800 rpmagitation on a shaker. After another three 5-minute washcycles 100 µL of the biotin-labeled anti-NfL monoclonalantibody were added to each well. The plate was incubatedfor 45min (RT, 800 rpm agitation). Following another washcycle (3×5min), 100 µL of streptavidin–HRP conjugate wereadded to each well. The plate was incubated for 30min (RT,800 rpm). A last wash cycle (3×5min) was performed and100 µL of 3,3′,5,5′-tetramethylbenzidine (TMB) were added
a
Table 1Factors recorded for assay validation.
Factor Description
Number of participants n=35Sample condition on arrival (°C) −80 (n=34), 20 (n=1)Sample storage prior to analyses (°C) −80 (n=33), −40 (n=1),
−20 (n=1)ELISA kit storage prior to analyses (°C) 4 (n=33), 20 (n=2)Delay to start of analysis (days) 14 (8–30)Pipettes calibrated prior to experiment (18) 51%100–200 µL (yellow top) pipettes usedfor samples
100%
Different pipettes used for preparingstandards
84%
Temperature at which ELISAwas performed
25.3 (24.0–26.5)
Time of color reaction (minutes) 15 (12–15)
The median (IQR), number and percentage are shown.
26 A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
to each well. After incubating the plate for 15min (RT) thecolor reaction was stopped with 50 µL of stop solution(0.18 M H2SO4). The absorbance (optical density, OD) ofeach well was read at 450 nm (test) and 700 nm (blank).
All laboratories performed only one run.
2.4. Data analysis and strategy for validation
The raw data was sent to London (AP) for analysis and toAmsterdam (CT) for independent control. The concentrationof the NfL samples was calculated based on each laboratory'sstandard curve using the closest three ODs of the quad-ruplicates. The results were then discussed with eachlaboratory. One dataset was again sent to Amsterdam (CT)for independent control. Results from samples which gave ahigher OD than the highest standard were excluded as thisstudy did not allow for extrapolation. All raw data were thensubjected to statistical analysis.
SAS software (version 9.2, SAS Institute, Inc., Cary, NorthCarolina, USA) was used for all statistical analyses andgraphs. Data was presented as median and interquartilerange (IQR). The coefficient of variation (CV) was calculatedfor each measurement as the standard-deviation divided bythe mean. The result was then multiplied by 100 forexpression as a percentage. Correlation analyses wereperformed using Pearson's R for normally distributed andSpearman's R for non-Gaussian data. Partial correlationanalyses were performed on predefined variables: temper-ature at which the ELISA was performed, time of thecolorimetric reaction, time delay between shipment andanalysis of samples, accuracy of each laboratory in respect topreparation of standards and intra-assay CV. The data foreach sample was ranked in order to analyse the inter-laboratory agreement. The inter-laboratory agreement wasassessed using kappa statistics for multiple raters andmultiple samples (Fleiss, 1971) and rated as described(Landis and Koch, 1977). The Fleiss' multiple rater kappawas calculated on those samples successfully measured byeach laboratory using the SAS magree macro (version 1.2).The conditional kappa for each response category as well asthe overall agreement is presented. In addition, kappacoefficients were also calculated using the more robust
AC1 statistics introduced by Gwet (2002) using the SASinter_rater macro (version 1.0, downloaded from http://www.stataxis.com/files/sas/INTER_RATER.TXT).
3. Results
The samples were shipped on dry ice and arrived inperfect frozen storage conditions in 34 (97%) of thelaboratories (Table 1). In one case the samples were delayedat an airport for 2 days and arrived defrosted (laboratory#26). Samples were stored at −80 °C by 33 of thelaboratories until analysis. The average delay from shipmentof samples to analyses was two weeks (Table 1). Alllaboratories used yellow top (100–200 µL) pipettes forpipetting samples and standards into the ELISA plate. Thesepipettes were specifically calibrated prior to the experimentby 51% of laboratories. In all other cases pipettes werecalibrated on a routine basis as part of normal laboratorypractice. For the preparation of standards 83% of laboratoriesused a different set of pipettes, which were able to handlelarger volumes (up to 1000 µL).
The ELISA kit worked successfully in 33/35 (94%) of theparticipating laboratories. In two laboratories, the ELISA didnot work due to very low optical densities (ODb0.01) in one,probably reflecting a pipetting error due to mismatch ofantibodies. Almost uniformly high ODs were found byanother laboratory (#14), probably reflecting cross-contam-ination. The temperature at which the ELISA was performedaveraged 25 °C (Table 1). The time given for the colorimetricreaction averaged 14min (median 15min).
In nine laboratories the standard curve went intosaturation. In these cases the top standard and sampleswith high NfL concentrations (samples #1, #5, #6, #9, #10,#13 and #14, Fig. 1) had to be excluded. In this study it wasnot allowed to extrapolate and therefore we were unable tocalculate the concentration of a sample which fell outside therange of the standard curve. For this reason the NfLconcentration could not be calculated from the optical densityof 203/801 (26.5%) of all samples (Table 2). The failure ratewas highest for sample #5 (33/2, 94.3%). In contrast, the NfLconcentration could be calculated from the results of alllaboratories for sample #4 (35/35, 100%, Fig. 1). Table 2summarises the success rate for determining the OD for eachsample and the failure rate for calculating the absoluteconcentration. The percentage agreement for the ranking ofsamples was higher if the ranking was based on OD instead ofprotein concentration for 6 samples (#3, #7, #8, #10, #11,and #12). The agreement between laboratories for individualsamples ranged from fair (sample #2) to almost perfect(sample #4, Table 2). The overall agreement was substantial(Landis and Koch, 1977) using either Fleiss' multi-rater kappa(overall kappa 0.602, pb0.0001) or Gwet's AC1 statistics(overall conditional AC1 0.595, standard error 0.081,pb0.0001; unconditional AC1 0.595, standard error 0.084,pb0.0001.).
The overall ranking of the absolute protein concentrationwas identical to what was found for the OD (Table 2). But onlyfor one sample (#4) a result was obtained from alllaboratories, preventing calculation of the inter-laboratoryagreement using kappa statistics.
Fig. 1. The concentration of CSF NfL levels (µg/mL) measured per sample is shown. Individual data points are represented by dots, the box-and-whisker boxesindicate the median and interquartile (25–75%) range.
27A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
3.1. Intra-laboratory assay CV
The average intra-laboratory assay CV was 3.3% (median2.7%). The breakdown for each standard and sample issummarised in Fig. 2. The intra-assay CV was best for highconcentrations of NfL (standard 1: 10,000 ng/mL) and worstfor low concentrations of NfL (standard 8: 0 ng/mL).
3.2. Inter-laboratory assay CV
The concentration of NfL was calculated based on eachlaboratory's standard curve. Fig. 1 shows that there was a
Table 2Success rates for measuring the OD and calculating the absolute NfL protein concen
Samplenumber
OD
Success Rank % of labs Kapp
#1 34/34 (100%) 9 72% 0.47#2 34/34 (100%) 6 65% 0.38#3 34/34 (100%) 3 68% 0.47#4 34/34 (100%) 1 100% 1.00#5 26/34 (77%) – – –
#6 32/34 (94%) – – –
#7 34/34 (100%) 7 87% 0.79#8 34/34 (100%) 4 94% 0.87#9 26/34 (77%) – – –
#10 34/34 (100%) 8 74% 0.53#11 34/34 (100%) 5 62% 0.42#12 34/34 (100%) 2 68% 0.47#13 29/34 (85%) – – –
#14 31/34 (91%) – – –
#15 33/34 (97%) – – –
Overall – – – 0.60
The overall rank (1= lowest rank, 9= highest rank) of a sample is shown alongsidelaboratory (#14)measured only one sample andwas therefore not included into thisall of the 34 remaining laboratories. Fleiss' kappa and Gwet's AC1 statistics were calcshown as well as the overall agreement (there was no difference between Gwet's ccolumn is presented.). Fleiss' kappa and Gwet's AC1 could not be calculated on thesamples. Not available = –.
large discrepancy in the reported values between thelaboratories. This translated to an average inter-laboratoryassay CV of 59% (median 47%, Fig. 3).
3.3. Validation parameters
This validation study was designed to assess the influenceof quantifiable parameters on the inter-laboratory assay CV.The values of all laboratories were compared to those fromthe reference laboratory (London). Using all CSF samples,there was a weak correlation of the absolute NfL concentra-tion of R=0.60 (pb0.0001).
tration for each sample.
NfL [ng/mL]
a AC1 Success Rank % of labs
0.46 23/34 (68%) 9 77%0.36 29/34 (85%) 6 65%0.45 33/34 (97%) 3 63%1.00 34/34 (100%) 1 100%– 2/34 (6%) – –
– 14/34 (41%) – –
0.79 28/34 (82%) 7 77%0.87 30/34 (88%) 4 89%– 5/34 (15%) – –
0.52 25/34 (74%) 8 67%0.41 29/34 (85%) 5 56%0.46 32/34 (94%) 2 66%– 7/34 (21%) – –
– 11/34 (32%) – –
– 25/34 (74%) – –
0.60 – – –
the percentage of laboratories who ranked the sample equally (% of labs). Oneanalysis. The ranking was not calculated for ODs whichwere notmeasured byulated on the ranked OD. The conditional kappa for each response category isonditional and unconditional AC1 statistics for this data, therefore only oneprotein concentration due to a less than 100% success rate in 8 of 9 ranked
Fig. 2. Intra-assay CV. The bar-chart shows the intra-assay CV for allstandards and samples analyzed. The intra-assay CV averages to 4.0% (smallhorizontal bars).
28 A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
3.3.1. TemperatureThe partial correlation analysis demonstrated the influ-
ence of the temperature at which the assay was performedwith R decreasing to 0.58 (pb0.0001).
Fig. 3. Inter-laboratory assay CV. The bar-chart shows the inter-laboratoryassay CV for each individual sample. The averaged inter-laboratory assay CVis 64.8%.
3.3.2. TimeAlso the time of color development had little influence on
the results. Correcting for this variable in the partial correlationanalysis, R increased marginally to 0.62 (pb0.0001).
3.3.3. Delay in analysisThe time delay between sending out the samples and
analysis had no effect on the results (R=0.60, pb0.0001).
3.3.4. Laboratory accuracy as estimated by the intra-assay CVThe intra-assay CV (accuracy of each laboratory) had no
effect (R=0.60, pb0.0001).
3.3.5. Preparation of standardsThe greatest influence on inter-laboratory validation was
found to be variation in the concentration of the standardsused by each laboratory to calculate the CSF NfL concentrationof the samples. This is illustrated by two laboratories whichanalyzed the samples only two days apart at almost equaltemperature (26 °C and 27 °C, Table 3). The two standardcurves are not parallel and contain different concentrations ofNfL. Surprisingly the optical densities for the CSF samples arealmost identical (0–2% difference, Table 3). However, thereported concentration of CSF NfL differed considerably: 43–53% (Table 3).
It is possible that a non-systematic error was introducedbecause 84% of the laboratories used different pipettes toprepare the standards and pipette the samples (Table 1). Inorder to analyse the potential error introduced by differencesin the concentration of the NfL standard, the CSF sampleswere normalized separately to the highest optical density foreach laboratory (a one point calibration to a top value of 1).Comparing these normalized ODs, the correlation of theresults improved to R=0.72 (pb0.0001).
Fig. 1 shows that not all laboratories measured the highestvalue for the same sample and the inter-laboratory rateragreement was not perfect for all samples (Table 2). In order tostandardise we selected one sample in which the proteinconcentration was available from most of the participatinglaboratories. In addition, this sample had a sufficient highprotein concentration to allow for normalization of theremaining samples. We therefore chose to normalize forsample 7 (82% success rate for protein concentration,Table 2). Sample 7 had an average NfL concentration higherthan in six other samples (#2, #3, #4, #8, #11, and #12) withsufficient success rates to allow for a correlation analysis.Standardizing the normalization procedure to CSF sample 7further improved the correlation to R=0.98 (pb0.0001, Fig. 4).
3.4. Optimized inter-laboratory assay CV
Optimising the results to the largest effect found (standar-disation to CSF sample 7), due to a non-systematic error inpreparation of protein standards, reduced the inter-laboratoryassay CV to an average of 14%. Again, a better CV was found forCSF samples with a high NfL protein concentration (Fig. 5).
4. Discussion
This validation study identified parameters influencingthe inter-laboratory CV. In descending importance thesewere
Table 3Preparation of standards was identified as the most important parameter influencing the inter-laboratory CV.
Specimen Mean OD Difference NfL (ng/mL) Difference
Lab #26 Lab #18 OD (%) Lab #26 Lab #18 NfL (%)
Standard 1 2.79 3.12 11% 10 10 –
Standard 2 1.89 2.92 35% 5 5 –
Standard 3 1.22 1.72 29% 2.5 2.5 –
Standard 4 0.66 0.92 18% 1.25 1.25 –
Standard 5 0.40 0.47 15% 0.625 0.625 –
Standard 6 0.23 0.22 4% 0.313 0.313 –
Standard 7 0.14 0.11 11% 0.156 0.156 –
Sample 1 2.94 2.96 1% 10.899 5.075 53%Sample 7 2.35 2.31 2% 7.394 3.675 51%Sample 11 1.74 1.74 0% 4.484 2.574 43%
In this example the raw data from two laboratories closely matched for other parameters (temperature 27 °C and 26 °C, days between shipment and analysis 30and 28 days, time of color reaction 15 and 8min, respectively) are shown. Surprisingly, the optical densities for the CSF samples from these two laboratories arealmost identical (0–2% difference). For the standards the OD varies over a wider range (4–35% difference). This is the reason for the large difference in the reportedresults of CSF NfL concentrations (43 to 53%). The concentration for the NfL standards was defined as described in the Analytical procedures. Note that forlaboratory #26 the mean OD of sample 1 (2.94) was higher than the mean OD for the highest standard (2.79). Because we did not allow for extrapolation in thisstudy this sample would not have been included in the analyses.
29A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
the preparation of the standard curve, time of the colorimet-ric reaction, delay between shipment and sample analysis,intra-laboratory CV, and temperature during the experiment.The importance of identifying these factors is highlighted bythe strength of the correlation between the results from thetest and reference laboratories. For uncontrolled conditionsthere was only a weak correlation with R=0.60 with aninter-laboratory CV of 64.8%. Under controlled conditions this
Fig. 4. Correlation analysis on standardized normalization of the opticaldensity to CSF sample #7 (R=0.98, pb0.0001).
was improved to a correlation of R=0.98 with an inter-laboratory CV of 14%. Under optimal conditions an inter-laboratory CV of less than 15% should be achievable.
Themost important finding of this study is the influence ofthe preparation of the standards. Controlling for all otherparameters, this can account for a two-fold difference inreported values. This variation is in part explained by limitedaccuracy of the pipettes. Only half of the laboratoriescalibrated their pipettes prior to the experiment. The otherhalf calibrated their pipettes on a regular basis, which isgenerally regarded as good laboratory practice. A limitation ofour study design is that this information was self-reportedand could therefore be a source of unaccounted variation.
Fig. 5. The optimized inter-laboratory CV, based on the normalized mean ODof the CSF samples as a series is shown. All CSF samples (#2, #3, #4, #8, #11,and #12) were normalized to CSF sample #7. The optimized inter-laboratoryCV averages 13.8% (median 12%).
30 A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
One peculiarity of NfL is that it can self-assemble (Liem andHutchison, 1982; Lee et al., 1993; Carter et al., 1998; Perez-Olleet al., 2002). Thus the formation of aggregates is a possibility.Aggregates can be dissolved either with time or usingsonication. A difference in the solubilisation of the lyophilizedNfL referenceprobably accounts for a largepart of discrepanciesbetween laboratories. We suspect that differences betweenlaboratories in the preparation of the standards is also the likelycause of the unacceptable high inter-laboratory CV in otherstudies (Blankenstein et al., 2005; Lewczuk et al., 2006; Verweyet al., 2009). At present this key factor is not controlled for byany of the available ELISA kits for protein biomarkers. It isimportant to note that the inter-rater reliability is typicallymeasured when all raters (e.g. laboratories) operate undersimilar conditions. Therefore the identification of a newextragenous factor has two implications. Firstly, the statisticalstrength of the kappa coefficient is limited. Secondly, a futurecontrolled experiment should be conducted to evaluate theeffect of standard preparation on the agreement coefficient.
All enzymes work within a range of temperature. Theenzyme used in this experiment was horseradish peroxidase(HRP). The colorimetric reaction depends on the oxidation of3,3′,5,5′-tetramethylbenzidine (TMB) and is optimal around25 °C at pH 7.0. All experiments were performed under idealthermal conditions and correcting for temperature did notimprove the results.
The enzyme activity is calculated as moles of substrateconverted per time unit. Because the time kinetics for HRP arenot completely linear, variation in reaction time can influencethe outcome of the experiment, particularly if the reactionreaches saturation. In this study saturation was reached for anumber of samples. As our partial correlation analysis shows,correcting for time only had a very small effect on the data.
The delay between shipment of samples and analysis isimportant for protein biomarkers, which have a limitedstability. NfL has been found to be stable at −80 °C 10 yearsafter sampling (Constantinescu et al., 2009). Consistent withthis data a delay in time from shipment to analysis was notstatistically relevant.
There are substantial limitations to this study. Firstly, itwasnot possible to control for all extragenous factors. We havesummarised all extragenous factors collected in form of atable. Secondly, the effect of the now identified relevance ofstandard preparation has not been tested prospectively in acontrolled experiment. We intend to address this issue in afuture study. Thirdly, a number of observations have been self-reported (e.g. calibration of pipettes). Because the participat-ing laboratories are located in different countries and havelimited experimental time, it will be challenging a singleobserver to collect this information. Fourthly, to keep thismanuscript focused and transparent, limited use of statisticalanalyses was considered necessary. The raw data of our studyhas therefore beenmade available online.We are keen to hearabout alternative statistical approaches to this complex issue,particularly in dealing with missing ratings which will remaina problem for this type of reliability study.Many of the presentstatistical analyses were correlative. It needs to be borne inmind that a statistical correlation does not prove causality.
This large validation study on an ELISA for a proteinbiomarker demonstrated that the experience and accuracy(intra-assay CV) of each laboratory had only a minor
influence on the outcome. This is reassuring, meaning thatany laboratory should be able to use the ELISA, but will needto establish in-house reference ranges. In a post-hoc analysisthe correlation analysis of the absolute concentration of CSFNfL levels between two laboratories experienced in thedevelopment of this type of ELISA (London and Gothenburg),yielded an R of 0.99 (pb0.0001). Our conclusion from thisresult is that at present any treatment trial including NfL as aprotein biomarker may benefit from batch analyses in onelaboratory to ensure the highest degree of accuracy. The lowintra-laboratory CV is encouraging in this respect. Alterna-tively, the use of an internal laboratory quality control (suchas sample 7 in this study), is strongly recommended. Thisstudy is consistent with previous data showing that humanfactors are an important factor adding to inaccuracy (Carraroand Plebani, 2007). Our results therefore highlight theimportance of a careful adherence to the analytical procedure.The routine application of this ELISA will require carefultraining of staff (to include calibration procedures) in order tominimise the intra-laboratory inter-assay CV.
In summary this validation study on an ELISA for thequantification of NfL from human CSF revealed difficulties,due to the preparation of accurate and consistent standards.Even though an unacceptable poor inter-laboratory assay CVis known also for other proteins, such as tau and amyloid-beta1–42 (Blankenstein et al., 2005; Lewczuk et al., 2006;Verwey et al., 2009), this is not yet controlled for. Underoptimal conditions an inter-laboratory CV of 14% should beachievable for this NfL ELISA.
Acknowledgments
We should like to acknowledge the technical assistance ofMrs Ute Schulz, Mrs Brigitte Fritz, Mr Willy Deleersnijder, MsCarmine Tamborino, MS Päivi Räsänen, Mrs W. van Geel, MsAnh Dang, Ms Miliam Karamaga, Ms Martina Hoelzl, MsManjit Braitch, Mrs Olga Petropoulou, Ms Keresztes and MsLajtos. We should like to thank Dr Geoffrey Keir for discussionin the planning phase of the study and Prof Antonio Bertolottofor helpful comments on the manuscript.
This study received no financial support. All NF-light®were donated by UmanDiagnostics. Samples, packing andlogistics were provided by the London laboratory and costsfor shipment were paid by the participating laboratories.
References
Blankenstein, M.A., Clark, C., De Deyn, P.P., Galasko, D., Hartmann, T., Hampel,H., Kapaki, E., Mehta, P.D., Parnetti, L., Pirttila, T., Verbeek, M.M.,Wildfang, J., Blennow, K., Verbeek, M.M., 2005. Multicenter comparisonof assays for cerebrospinal fluid (CSF) markers for Alzheimer disease:Clin. Chim. Acta, vol. 355, p. S175.
Blennow, K., de Leon, M.J., Zetterberg, H., 2006. Alzheimer's disease. Lancet368, 387.
Carraro, P., Plebani, M., 2007. Errors in a stat laboratory: types andfrequencies 10 years later. Clin. Chem. 53, 1338.
Carter, J., Gragerov, A., Konvicka, K., et al., 1998. Neurofilament (NF)assembly; divergent characteristics of human and rodent NF-L subunits.J. Biol. Chem. 273, 5101.
Constantinescu, R., Romer, M., Oakes, D., Rosengren, L., Kieburtz, K., 2009.Levels of the light subunit of neurofilament triplet protein in cerebro-spinal fluid in Huntington's disease. Parkinsonism Relat. Disord. 15 (3),245.
Dubois, B., Feldman, H.H., Jacova, C., Dekosky, S.T., Barberger-Gateau, P.,Cummings, J., Delacourte, A., Galasko, D., Gauthier, S., Jicha, G., Meguro,
31A. Petzold et al. / Journal of Immunological Methods 352 (2010) 23–31
K., O'brien, J., Pasquier, F., Robert, P., Rossor, M., Salloway, S., Stern, Y.,Visser, P.J., Scheltens, P., 2007. Research criteria for the diagnosis ofAlzheimer's disease: revising the NINCDS-ADRDA criteria. Lancet Neurol.6 (8), 734.
Fleiss, J.L., 1971. Measuring nominal scale agreement among many raters.Psychol. Bull. 76, 378.
Gwet, A., 2002. Handbook of Inter-Rater Reliability. STATAXIS PublishingCompany, Montgomery Village, MD 20886. USA.
Landis, J.R., Koch, G.G., 1977. The measurement of observer agreement forcategorical data. Biometrics 33, 159.
Lee, M.K., Xu, Z., Wong, P.C., Cleveland, D.W., 1993. Neurofilaments areobligate heteropolymers in vivo. J. Cell Biol. 122 (6), 1337 September.
Lewczuk, P., Wiltfang, J., 2008. Neurochemical dementia diagnostics: state ofthe art and research perspectives. Proteomics 8, 1292.
Lewczuk, P., Beck, G., Ganslandt, O., Esselmann, H., Deisenhammer, F.,Regeniter, A., Petereit, H.F., Tumani, H., Gerritzen, A., Oschmann, P.,Schroder, J., Schonknecht, P., Zimmermann, K., Hampel, H., Burger, K.,Otto, M., Haustein, S., Herzog, K., Dannenberg, R., Wurster, U., Bibl, M.,Maler, J.M., Reubach, U., Kornhuber, J., Wiltfang, J., 2006. Internationalquality control survey of neurochemical dementia diagnostics. Neurosci.Lett. 409, 1.
Liem, R.K., Hutchison, S.B., 1982. Purification of individual components of theneurofilament triplet: filament assembly from the 70 000-daltonsubunit. Biochemistry 21 (13), 3221 June.
Norgren, N., Karlsson, J.E., Rosengren, L., Stigbrand, T., 2002. Monoclonalantibodies selective for low molecular weight neurofilaments. HybridHybridomics 21, 53.
Norgren, N., Rosengren, L., Stigbrand, T., 2003. Elevated neurofilament levelsin neurological diseases. Brain Res. 987 (1), 25 October.
Norgren, N., Sundstrom, P., Svenningsson, A., Rosengren, L., Stigbrand, T.,Gunnarsson, M., 2004. Neurofilament and glial fibrillary acidic protein inmultiple sclerosis. Neurology 63, 1586.
Perez-Olle, R., Leung, C.L., Liem, R.K., 2002. Effects of Charcot–Marie–Tooth-linked mutations of the neurofilament light subunit on intermediatefilament formation. J. Cell Sci. 115, 4937.
Petzold, A., 2005. Neurofilament phosphoforms: surrogate markers foraxonal injury, degeneration & loss. J. Neurol. Sci. 233, 183.
Rosengren, L.E., Karlsson, J.E., Karlsson, J.O., Persson, L.I., Wikkelso, C., 1996.Patients with amyotrophic lateral sclerosis and other neurodegenerativediseases have increased levels of neurofilament protein in CSF.J. Neurochem. 67, 2013.
Shaw, G., 1998. Neurofilaments. Springer-Verlag.Teunissen, C.E., Dijkstra, C., Polman, C., 2005. Biological markers in CSF and
blood for axonaldegeneration inmultiple sclerosis. LancetNeurol. 4, 32–41.Van Geel, W.J.A., Rosengren, L.E., Verbeek, M.M., 2005. An enzyme
immunoassay to quantify neurofilament light chain in cerebrospinalfluid. J. Immunol. Methods 296, 179.
Verwey, N.A., van der Flier, W.M., Blennow, K., Clark, C., Sokolow, S., De Deyn,P.P., Galasko, D., Hampel, H., Hartmann, T., Kapaki, E., Lannfelt, L., Mehta,P.D., Parnetti, L., Petzold, A., Pirttila, T., Saleh, L., Skinningsrud, A.,Swieten, J.C.V., Verbeek, M.M., Wiltfang, J., Younkin, S., Scheltens, P.,Blankenstein, M.A., 2009. A worldwide multicenter comparison of assaysfor CSF biomarkers in Alzheimer disease. Ann. Clin. Biochem. 46 (3), 235.
