INVITED REVIEW

Methodological requirements for valid tissue-based biomarkerstudies that can be used in clinical practice

Lawrence D. True

Received: 6 December 2013 /Accepted: 13 December 2013# Springer-Verlag Berlin Heidelberg 2014

Abstract Paralleling the growth of ever more cost efficientmethods to sequence the whole genome in minute fragments oftissue has been the identification of increasingly numerousmolecular abnormalities in cancers—mutations, amplifications,insertions and deletions of genes, and patterns of differentialgene expression, i.e., overexpression of growth factors andunderexpression of tumor suppressor genes. These abnormali-ties can be translated into assays to be used in clinical decisionmaking. In general terms, the result of such an assay is subjectto a large number of variables regarding the characteristics ofthe available sample, particularities of the used assay, and theinterpretation of the results. This review discusses the effects ofthese variables on assays of tissue-based biomarkers, classifiedby macromolecule—DNA, RNA (including micro RNA, mes-senger RNA, long noncoding RNA, protein, and phosphopro-tein). Since the majority of clinically applicable biomarkers areimmunohistochemically detectable proteins this review focuseson protein biomarkers. However, the principles outlined aremostly applicable to any other analyte. A variety ofpreanalytical variables impacts on the results obtained, includ-ing analyte stability (which is different for different analytes,i.e., DNA, RNA, or protein), period of warm and of coldischemia, fixation time, tissue processing, sample storage time,and storage conditions. In addition, assay variables play animportant role, including reagent specificity (notably but notuniquely an issue concerning antibodies used in immunohisto-chemistry), technical components of the assay, quantitation, andassay interpretation. Finally, appropriateness of an assay forclinical application is an important issue. Reference is madeto publicly available guidelines to improve on biomarker de-velopment in general and requirements for clinical use in

particular. Strategic goals are formulated in order to improveon the quality of biomarker reporting, including issues ofanalyte quality, experimental detail, assay efficiency and preci-sion, and assay appropriateness.

Keywords Immunohistochemistry . Preanalytical . Tissue .

Variables . Prognostic . Predictive

Introduction

Paralleling the growth of ever more cost efficient methods tosequence the whole genome of pieces of tissue as small as0.01 g has been identification of increasingly numerous ge-netic abnormalities in cancers—mutations, amplifications, in-sertions, deletions, and patterns of differential expression ofgenes, i.e., overexpression of growth factors and underexpres-sion of tumor suppressor genes. Knowledge of tumor-associated molecular abnormalities has stimulated the largescale screening and development of reagents (small drugs andantibody derivatives) that bind to the abnormal genes or geneproducts. Pharmaceutical agents that are specific for thesemolecular targets have greatly increased in number.Succeeding this growth industry, which produces drugs thattarget these molecular abnormalities and that have both greatertumor-specific effect than conventional cytotoxic agents withless severe and less frequent side effects, has been the impetusto develop tissue-based biomarkers. Generically defined, tis-sue biomarkers include any tissue-based feature, ranging fromthe histological grade of a cancer to point mutations. We shallfocus on molecular biomarkers in this review.

Biomarkers can be categorized as either prognostic or pre-dictive. Prognostic markers predict clinical outcome after×number of years. The number of years appropriate to evaluatethe value of a biomarker should be proportionate to the naturalhistory of the cancer. The fate of most patients with lung cancer

L. D. True (*)Department of Pathology, University of Washington Medical Center,1959 NE Pacific St., Box 356100, Seattle, WA, USAe-mail: ltrue@u.washington.edu

Virchows ArchDOI 10.1007/s00428-013-1531-0

will become apparent within 2 years; in contrast, the outcome ofthe average patient with prostate cancer would not be knownwith certainty for more than 5 years. Predictivemarkers predictresponse of a tumor to a drug that putatively targets a keyfunctional molecule involved in the biology of that tumor.Given the high cost of many targeted therapies, many earlyphase clinical trials are requiring or, for markers that are notintegral to a clinical trial, encouraging the identification andvalidation of predictive biomarkers of these novel agents. How-ever, identifying the biomarker that can serve as a detector ofabnormalities in a functionally critical step of the progression ofthe cancer can be challenging, especially if the molecularpathway contains many regulatory genes. The mandate toidentify a biomarker that can readily be assessed is particularlyhigh when the predictive biomarker is integral to the trial. Anintegral marker is essential for a clinical trial, i.e., is used toqualify a patient for a specific therapy.

The performance of biomarkers is of concern to the regu-latory agencies, the drug manufacturers, the health care deliv-erers and to patients, to maximize benefit and minimize cost,and complications of therapy based on expression of thebiomarker. However, the performance of many biomarkershas been relatively poor [1].

There is a major challenge with immunohistochemical datathat is used for tissue-based biomarkers. Biopharmaceuticalcompanies that use the biomedical literature to identify pre-dictive biomarkers or biomarkers that are integral to earlyphase clinical trials with novel drugs rely upon the literatureto identify candidate predictive biomarkers and moleculartargets for therapy. However, no more than 50 % of studiescan be replicated. Consequently, costs to develop drugs areoften wasteful, incurred by companies on drugs for which theevidence of functional efficacy is poor. A theme of this reviewis how the quality of biomarker studies can be improved.

As attention has been directed to biomarkers, the biomedcommunity has become aware of the effect of factors inde-pendent of the biologic state of the tumor that can lead toerroneous results of tissue-based assays. These can be catego-rized as preanalytical and analytical variables that affect theexpression and expression levels of biomarkers. These effects,which do not reflect the biology of the tumor, but are artifactsof tissue handling and of the way that the assay is performed,can skew results of the assays fromwhat the true biologic stateof the analyte is.

This introductory section discusses the effects of thesevariables on assays of tissue-based biomarkers, classified bymacromolecule—DNA, RNA (including micro RNA, mes-senger RNA, long noncoding RNA), protein, and phospho-protein. Many studies of the effect of preanalytical and ana-lytical variables on macromolecules in tissue have been pub-lished. A repository of manuscripts of these studies, whichwere published in peer-reviewed journals, is on the website ofthe former Office of Biospecimen and Biorepository Research

of NCI. Hundreds of studies have analyzed the effect ofpreanalytical and analytical variables. A web-accessible li-brary of publications, the Biospecimen Research Database(https://brd.nci.nih.gov/BRN/brnHome.seam), organizes thepublications in a table organized by type of specimen(blood, serum, plasma, urine, saliva, and tissue) and type ofassay for each macromolecule (DNA, RNA, protein, smallmolecule, morphology, cell count/volume, and peptide).

General approach of this review

This review is focused on the most common materials used inby pathology labs in clinical practice—human tissue obtainedat surgery, fixed in neutral buffered formalin, and embedded inparaffin (FFPE tissue). Although the main emphasis is on themethod most commonly used in clinical labs to evaluatebiomarkers in FFPE tissue—immunoperoxidase stains of pro-tein analytes assessed visually by pathologists—commentswill be provided about other analytes and methods of assess-ment of tissue-based biomarkers. Finally, examples are basedon studies of prostate cancer, the cancer with which the authoris most familiar.

Preanalytical variables

Analyte Stability

The stability of a potential biomarker varies with the type ofmacromolecule [2].

& DNA. Although relatively resistant to degradation, the infor-mativeness of DNA can be affected by tissue handling.Formaldehyde fragments DNA, resulting in shorter, lessinformative fragments wherein mutations might be missed,or, conversely, be misidentified. A relevant recent finding isthat DNA sequences differ in different somatic tissues,based on whole genome sequences [3]. This finding raisesthe prospect that even the sequence of DNAused as “normalcontrol DNA” may be misleading when used as referenceDNA for the DNA sequence of a tumor in that individual.

& mRNA. This most labile class of molecule has been used togenerate molecular signatures of the effect of environmenton tumor tissue. For example, RNA signatures of warmischemia (the length of time devascularized tissue is at bodytemperature), cold ischemia (the length of time tissue sits atambient temperature until stabilization by freezing or fixa-tion), or even preoperative diet (low vs. high protein content)have been generated. Micro RNA is a more stable nucleicacid than is messenger RNA.

& Protein. One of the most widely used biomarker assays isimmunohistochemical localization and assessment of ex-pression level of specific proteins in FFPE tissue. At least

Virchows Arch

62 preanalytical variables that might affect immunohisto-chemical reactivity (IHR) have been evaluated [4]. Thesevariables cover the time course from tissue acquisitionthrough fixation and storage of the paraffin blocks tohandling of sections prior to immunostaining. The extentthat preanalytical conditions affect protein levels varieswith the analyte and with the study. Below is a summaryof those variables that are most frequently encountered,that contribute most to preanalytical variability, and thatshould be considered in immunohistochemical assays.

Delay in acquiring the tissue from the surgery

Laparoscopic procedures have resulted in longer warm ische-mia time (the time between tissue devascularization and remov-al of tissue from the patient) than in the prelaparoscopic era.Though changes in RNA profile are measurable, few studieshave assessed the effect of warm ischemia on proteins. Anevolving consensus is that warm ischemia time not exceed 2 h.

Delay in fixation

In general, fixation can be delayed up to 12 h withoutcompromising IHR. In one study, there was no effect on estro-gen receptor (ER) or progesterone receptor (PR) IHRwith delayin fixation of up to 8 h at 19 °C or up to 12 h at 4 °C. Beyond 6 hthe effect of “cold” ischemia varies with the analyte.

Time in fixative

Fixation for at least 24 h appears to be sufficient for preser-vation of all analytes. Though loss of immunoreactivity byunderfixation is the most frequent concern, overfixation canaffect staining of some analytes. For example, there is loss ofsignal when fixation of ER exceeds 4 days.

Postfixation tissue processing

Although publications are discrepant, immunoreactivity canbe variably affected, in an analyte-independent manner, at thefollowing stages of tissue and section handling: duration andtemperature of dehydration of fixed tissue, reagents used forclearing tissue and the temperature of clearing, duration andtemperature of paraffin embedding, and the temperature andduration of heating to enhance adherence of sections to slides.

Duration of block storage

The length of time that FFPE blocks can be storedwithout lossof immunoreactivity varies by analyte and study. Some blockshave been stored as long as 25 years without loss of IHR ofER. IHR of some other analytes decreases after 2 years ofblock storage.

Duration and temperature of storage of sections

Preanalytical variables include paraffin coating of sections,desiccation, storage in nitrogen, and storage at –80 °C. Theextent of effect varies with the study and the analyte.

Other variables

Labs are unlikely to change or control for some conditionsdespite weak evidence that the variable may affect IHR. Theseinclude the nature and pH of fixative (recommended: 10 %neutral formalin buffered to pH 5–7), tissue to fixative ratio(recommended: at least 1:10), and size of tissue specimen(recommended: one dimension is no greater than 0.3 cm).The effect on IHR of other variables is minimal or negligible,such as type of tissue processor. And the effect of somevariables has not been assessed, i.e., storage of specimens indifferent fixatives. Publications concur on the effect of fixa-tion delay, fixation, dehydration, paraffin impregnation, dry-ing of slides, and storage conditions on immunostains,commenting that some recommendations by oncology interestgroups differed from published guidelines. Fox example, de-lay in fixation, recommended not to exceed 1 h, can exceed12 h without discernible effect. And sections mounted onslides should probably be stained within 6 days of preparation,though two separate guidelines cite maximum times of 1 and6 weeks, respectively.

These studies have limitations. The methods used by thedifferent investigators were not standardized. And a majority ofstudies have not been validated by independent investigations.What is needed are metrics to assess the state of retention of thein vivo state of proteins in tissue, prior to the effect on theseanalytes of preanalytical conditions. An initial set of markersthat provide correlation, albeit weak, with quality of ER havebeen published. Loss of phosphorylated tyrosine immunoreac-tivity and increased acetylated lysine and HIF1A immunoreac-tivity correlates with decreased ER staining [5]. Unknown iswhether these markers, or any set of markers, can be used toassess the degree of retention of the native state of all analytes.One suspects not, hypothesizing that host factors (variance inthe functionality and metabolism of proteolytic enzymes) andanalyte factors (primary and secondary structure, access toproteases), influence the quality of ischemia markers.

Analytical variables

The stages of assays of protein expression level are additionalsources of ex-vivo variability.

& Primary antibody. Performance characteristics of the pri-mary antibody include sensitivity and specificity for de-tecting the analyte of interest in sections of FFPE tissue.Rigorous assessment of specificity involves biochemical

Virchows Arch

and cell biological techniques that clinical labs are virtu-ally never prepared to undertake. These techniques in-clude confirmation that engineered overexpression orknockdown of the analyte in cultures and preparations ofcells (such as cell line microarrays) correlates with IHR.There are suggested guidelines for assessing the quality ofprimary antibodies for use in IHC studies [6]. However,since most clinical labs are not prepared for detailedevaluations of the performance of antibodies, directors ofIHC labs must rely on information provided by the anti-body vendors. Too often, this information lacks data rel-evant to IHC studies of FFPE tissue. For example, anti-bodies that produce a single band on Western gels and arelocalized to cells of interest in frozen tissue may notimmunoreact with cells of interest in FFPE tissue.

& Technical component of assays. As currently practiced,immunoperoxidase stains involve at least four sequentialchemical reactions. In principle, conditions should be opti-mized for each step, i.e., antibodies should saturate bindingsites and the kinetics of peroxidase catalysis of substratediaminobenzidine should be optimized for the variables thataffect the amount of substrate that is deposited on the tissuesection—temperature, pH, concentration of electron donorhydrogen peroxide, and duration of reaction. Such system-atic optimization of all steps of IHC has been exemplifiedby a study correlating a biochemically assayed proteinextract from pituitary with immunoperoxidase stains quan-tified by densitometry [7]. Such optimization of the chem-istry is virtually never done. Assays of protein in tissue thatinvolve fewer reactions promise to be more reproducibile.Quantum dot-conjugated immunostains are, in principle,subject to less variance since there is no enzyme substratereaction and fewer steps are needed [8]. Mass spectrometryanalysis of protein content in cells of interest, i.e., cancercells, is another method that is more quantitative thanimmunoperoxidase stains. However, there is currently noefficient way to isolate cells of interest from tissue in avolume sufficient for mass spectrometry.

& Interpret ive component of assays. Current ly,immunoperoxidase stains are analyzed either visually bypathologists or quantified by image analysis. The advan-tage of visual interpretation is ready accessibility of stainswithout needing to acquire and devote effort using animage analysis system. However, visual cognition of thehuman eye varies with the task. While we can readilyidentify and categorize many objects that contrast withtheir microenvironment, such as cancer cells in tissue,human vision performs poorly in assessing degrees ofdifference of elements of an image, such as grey levels(http://bcs.mit.edu/people/adelson.html). Furthermore,tissue elements adjacent to the objects being analyzedinfluence our visual cognition [9]. Finally, there arebiologic limitations to our ability to visually “measure”

light intensity. For example, the sensitivity of our retina tolight differs by the wavelength of light [10].

& Quantitation. In contrast to visual assessment, the opticaldensity of the pixels that comprise the digitized image ofan immunostain can be determined with great accuracyusing an image analysis system. However, two compo-nents of image analysis limit transferability and reproduc-ibility of the data. Both components involve a pathologist(or, plausibly, a pathology-competent individual). First,the cells of interest need to be identified and distinguishedfrom histological mimics. No current image analysis sys-tem can reliably distinguish low-grade prostate carcinomafrom such mimics as adenosis, basal cell hyperplasia, andprostatic intraepithelial neoplasia. Second, a thresholdmust be set to distinguish pixels that are deemed “posi-tive” (reaction product) from those that are “negative.”Interpathologist variance in these cognitive tasks is virtu-ally never assessed. There are additional sources of vari-ance in image analysis systems that should be considered,and either controlled for or discarded, as appropriate [11].Finally, imaging systems need to include in their evalua-tion cells of interest that are stained below the positivethreshold level. By using multiple stains (to identify nu-clei, which would have to be annotated by the pathologistas cancer nuclei, and the analyte of interest), imagingsystems can address many of these challenges [12].

& Visual scoring. How immunostain interpretations are re-duced to single values that can be used by clinicians inmaking treatment decisions is an additional source ofvariance. Immunostains are characterized by two parame-ters—the percentage of cells that express analytes at dif-ferent levels of intensity. The challenge has been incorpo-rating the two parameters of immunostains into a singlenumber. One frequently used calculation is the “HScore”(a sum of the products of level of stain intensity andpercent of cells stained at each intensity level). A conse-quence of using the HScore is that stains that differ mark-edly can have the same value. A different way to report thedata is as categories of staining [13]. Though categoricalvalues lack the pseudo precision of HScores, their repro-ducibility is, in principle, higher.

Guidelines and specifications

The literature can be made more robust by adhering to anumber of specifications, such as reporting all of the compo-nents of immunohistochemical stains. A specification termedMISFISHIE has been published. This specification includesdetailed characterization of primary antibodies used in IHCstudies. However, the performance, specificity, and sensitivityof primary antibodies is not part of the MISFISHIE specifica-tion. Publications have made the point that specificity and

Virchows Arch

sensitivity of primary antibodies should be validated. Withrespect to the quality of published IHC studies, and thus, theprobability that the results can be replicated, one cannot butthink that data on preanalytical variables that affect assayresults should be a required part of the publication. Basedupon reviews of IHC studies, publications should consideraddressing the effect (if any) of warm and cold ischemia time,length of fixation, and tumor necrosis. These parameters couldbe reported as an addendum or as an incorporated table. Thistable could also be used by the editorial staff of the journalspublishing IHC studies. A second reason for the inability toreplicate IHC studies has to do with the interpretation ofimmunostains. Little attention, in general, has been paid tohow they are interpreted. Two major sources of variance ininterpretation are the fact that majority are accessed visuallyrather than by optical density of pixels in an image digitizedfile. Visual cognition is sufficiently poor that one questionswhether no more than two, perhaps three, categories of stainintensity should be used. A second variable is that there arevirtually never a set of standard curves, referencing the inten-sity (or, luminosity, when fluorochromes are used as thedetection agents) to reference expression levels of theanalytes. A third variable has to do with heterogeneity oftumors and a question of whether the samples chosen for theimmunohistochemical analysis truly are representative of thestate of expression of analytes in tumors. For example, thesequences of DNA from different regions of a large renal cellcarcinoma differ [14]. The challenge is to identify the regionof the tumor that has genomic changes that correlate withtumor progression, i.e., growth rate, patterns of metastasis,and resistance to therapy.

Predictive biomarkers, used to select patients for targetedtherapy, will play an increasingly prominent role in cancertreatment as more drugs targeted to pathway molecules aremade available. Several additional considerations pertain tothese biomarkers—notably assay performance and validation,assay utility, and clinical utility.

Guidelines for tissue-based assays

A number of guidelines has been published within the pastdecade. MISFISHIE provides a checklist for components oftissue-based assays that are sufficiently detailed for an indepen-dent to be likely to successfully replicate the study [15]. RE-MARK criteria list the expected components of a clinical trialthat has a tissue biomarker component [16]. Recently publishedis a set of guidelines for RNA array-based data that might beused as prognostic or predictive biomarkers [17]. Despite avail-ability of these guidelines, there has been little incentive formost of the biomedical community to use them, presumablysince adherence to them requires more effort and expense, bothby the investigators who study the biomarkers and by thereviewers of manuscripts that report biomarker studies.

Use of biomarkers in practice

To effectively incorporate predictive biomarkers into practicerequires close interaction between clinical trialists/oncologists,laboratorians/pathologists, outcome statisticians, and regulatoryagencies [18]. Appreciation has grown that the level of evi-dence of the predictive power of a biomarker provides a basisfor assessing the value of the marker in clinical practice. Fourcategories of level of evidence have been proposed in studies ofbreast cancer biomarkers [19]. Levels of evidence range fromanecdotal observations to validation of the performance of thebiomarker in an independent cohort where clinicians and pa-thologists are blinded to the clinical outcome of patients in thestudy. The concept of level of evidence can be expanded toinclude evidence of clinical utility and validity in addition to thestrength of evidence that the biomarker is predictive [20]. Statedanother way, a predictive biomarker is of value if it categorizesa significant number of patients. Conversely a biomarker is oflittle value if most patients do or do not respond to the targetedtreatment.

Once the performance of an assay has been characterizedand deemed adequate for the intended purpose, it should bevalidated. Validation of assays poses challenges for clinicallabs, which may not have the specimens, resources, or experi-ence to efficiently validate a tissue-based assay [21]. An addi-tional consideration for immunohistochemical assays is the lackof a standard method for validating quantitative immunostains.In lieu of being able to generate standard deviations, interob-server variance in quantitative values is often reported as kappavalues, which is a nonparametric value [21].

A final step in accepting an assay for clinical use is approv-al by regulatory agencies, such as the FDA.

Future directions (a wish list)

A challenge that pathologists are increasingly confronted withis assay reproducibility. Publications in which tissue-basedassays provide clinical outcome data provide potentially valu-able leads for investigators who are developing and testing theutility of drugs targeting gene products in specific molecularpathways. However, at least 50 % of the data in publicationsreporting the types of preclinical studies on which tissue-based biomarker assays is based are not reproducible [22].Explanations are multifold, including insufficient detail in theoriginal study, poor study design (such as lack of separatetraining and independent test sets), and lack of incentives torobustly validate the data. Biomarker studies could presum-ably be improved based on the following wish list:

& Details of experiments. That publications that includetissue-based biomarkers contain all methodological infor-mation of the biomarker assay sufficient for an

Virchows Arch

independent investigator to conduct and interpret the re-sults of the assay identical to how the assay was done inthe original publication [23]. As discussed above, such aspecification, with spreadsheet, which incorporates mostcomponents of tissue-based assays in a checklist form(MISFISHIE) has been published [15].

& Efficiency. That assays be done and interpreted in the mosteffort-efficient manner possible for that assay.

& Precision of tissue-based assays. That variability of inter-pretation of the assay be reported in all publications. Thisgoal involves more than one investigator independentlyinterpreting the assay. Although the type of assay withpresumably the greatest interobserver variance is animmunoperoxidase assay that is visually interpreted, as-says that use image analysis should be included sincepathologists make several usually subjective decisions,as previously discussed:

• Selection of the region of interest that is digitizedand analyzed.

• Setting a threshold for binarizing results.• Amethod that evaluates objects that are “negative.”

i.e., that stain at or below the level of backgroundstaining.

Analogous to assays of analytes in solution, the results ofeach pathologist could be reported as a range of values, whichwould reflect the precision of the assay.

Ideally the level of assay precision for clinical managementof the patient should be determined and reported for all assays.

& Accuracy of tissue-based assays. That a set of standards bedeveloped that includes the dynamic range of the assayand the range of the majority of values of the type ofcancer for which the assay is used.

& Analyte quality. That internal standards that assess thequality of protein in tissue be developed.

& Assay appropriateness. That results of the assay be “fit forpurpose.” Included in this wish is the expectation that therange of interobserver and intraobserver or test variance ofthe assay that is acceptable to the clinician be reported.

Acknowledgments This work was supported in part by the NationalCancer Institute Pacific Northwest Prostate Cancer Specialized Programof Research Excellence (SPORE; P50 CA 097186-06).

Conflict of interest I declare that I have no conflict of interest.

References

1. Meshinchi S, Hunger SP, Aplenc R, Adamson PC, Jessup JM (2012)Lessons learned from the investigational device exemption review ofChildren’s Oncology Group trial AAML1031. Clin Cancer Res 18:1547–1554

2. Hewitt SM, Badve SS, True LD (2012) Impact of preanalytic factorson the design and application of integral biomarkers for directingpatient therapy. Clin Cancer Res 18:1524–1530

3. O’Huallachain M, Karczewski KJ, Weissman SM, Urban AE,Snyder MP (2012) Extensive genetic variation in somatic humantissues. Proc Natl Acad Sci U S A 109:18018–18023

4. Engel KB, Moore HM (2011) Effects of preanalytical variables onthe detection of proteins by immunohistochemistry in formalin-fixed,paraffin-embedded tissue. Arch Pathol Lab Med 135:537–543

5. Neumeister VM, Anagnostou V, Siddiqui S, England AM, ZarrellaER, Vassilakopoulou M, Parisi F, Kluger Y, Hicks DG, Rimm DL(2012) Uantitative assessment of effect of preanalytic cold ischemictime on protein expression in breast cancer tissues. J Natl Cancer Inst104:1815–1824

6. Bordeaux J, Welsh A, Agarwal S, Killiam E, Baquero M, Hanna J,Anagnostou V, Rimm D (2010) Antibody validation. Biotechniques48:197–209

7. Gross DS, Rothfeld JM (1985) Quantitative immunocytochemistry ofhypothalamic and pituitary hormones: validation of an automated,computerized image analysis system. JHistochemCytochem 33:11–20

8. Zrazhevskiy P, Gao X (2009) Multifunctional quantum dots forpersonalized medicine. Nano Today 4:414–428

9. Fandel TM, Pfnür M, Schäfer SC, Bacchetti P, Mast FW, Corinth C,Ansorge M, Melchior SW, Thüroff JW, Kirkpatrick CJ, Lehr HA(2008) Do we truly see what we think we see? The role of cognitivebias in pathological interpretation. J Pathol 216:193–200

10. Wells WA, Rainer RO, Memoli VA (1993) Equipment, standardiza-tion, and applications of image processing. J Clin Pathol 99:48–56

11. True LD (1988) Quantitative immunohistochemistry: a new tool forsurgical pathology? Am J Clin Pathol 90:324–325

12. Rubin MA, Zerkowski MP, Camp RL, Kuefer R, Hofer MD,Chinnaiyan AM, Rimm DL (2004) Quantitative determination ofexpression of the prostate cancer protein alpha-methylacyl-CoAracemase using automated quantitative analysis (AQUA): a novelparadigm for automated and continuous biomarker measurements.Am J Pathol 164:831–840

13. Etzioni R, Hawley S, Billheimer D, True LD, Knudsen B (2005)Analyzing patterns of staining in immunohistochemical studies: ap-plication to a study of prostate cancer recurrence. Cancer EpidemiolBiomarkers Prev 14:1040–1046

14. GerlingerM,RowanAJ, Horswell S, Larkin J, Endesfelder D, GronroosE, Martinez P, Matthews N, Stewart A, Tarpey P, Varela I, Phillimore B,Begum S, McDonald NQ, Butler A, Jones D, Raine K, Latimer C,Santos CR, Nohadani M, Eklund AC, Spencer-Dene B, Clark G,Pickering L, Stamp G, Gore M, Szallasi Z, Downward J, Futreal PA,Swanton C (2012) Intratumor heterogeneity and branched evolutionrevealed by multiregion sequencing. N Engl J Med 366:883–892

15. Deutsch EW, Ball CA, Berman JJ, Bova GS, Brazma A, BumgarnerRE, Campbell D, Causton HC, Christiansen JH, Daian F, Dauga D,Davidson DR, Gimenez G, Goo YA, Grimmond S, Henrich T,Herrmann BG, Johnson MH, Korb M, Mills JC, Oudes AJ,Parkinson HE, Pascal LE, Pollet N, Quackenbush J, Ramialison M,Ringwald M, Salgado D, Sansone SA, Sherlock G, Stoeckert CJ Jr,Swedlow J, Taylor RC, Walashek L, Warford A, Wilkinson DG,Zhou Y, Zon LI, Liu AY, True LD (2008) Minimum informationspecification for in situ hybridization and immunohistochemistryexperiments (MISFISHIE). Nat Biotechnol 26:305–312

16. McShaneLM,AltmanDG, SauerbreiW, Taube SE,GionM,ClarkGM,Statistics Subcommittee of the NCI-EORTCWorking Group on CancerDiagnostics (2005) Reporting recommendations for tumor marker prog-nostic studies (REMARK). J Natl Cancer Inst 97:1180–1184

17. McShane LM, Cavenagh MM, Lively TG, Eberhard DA, BigbeeWL, Williams PM, Mesirov JP, Polley MY, Kim KY, Tricoli JV,Taylor JM, Shuman DJ, Simon RM, Doroshow JH, Conley BA(2013) Criteria for the use of omics-based predictors in clinical trials.Nature 502:317–320

Virchows Arch

18. Poste G, Carbone DP, Parkinson DR, Verweij J, Hewitt SM, JessupJM (1996) Eveling the playing field: bringing development of bio-markers and molecular diagnostics up to the standards for drugdevelopment. J Natl Cancer Inst 88:1456–1466

19. Hayes DF, Bast RC, Desch CE, Fritsche H Jr, Kemeny NE, Jessup JM,Locker GY, Macdonald JS, Mennel RG, Norton L, Ravdin P, Taube S,Winn RJ (2012) Tumor marker utility grading system: framework toevaluate clinical utility of tumormarkers. Clin Cancer Res 18:1515–1523

20. Febbo PG, Ladanyi M, Aldape KD, De Marzo AM, Hammond ME,Hayes DF, Iafrate AJ, Kelley RK,Marcucci G, Ogino S, PaoW, Sgroi

DC, Birkeland ML (2011) NCCN Task Force report: evaluating theclinical utility of tumor markers in oncology. J Natl Compr CancNetw 9(Suppl 5):S1–S32

21. Williams PM, Lively TG, Jessup JM, Conley BA (2012) Bridging thegap: moving predictive and prognostic assays from research to clin-ical use. Clin Cancer Res 18:1531–1539

22. Begley CG, Ellis LM (2012) Drug development: raise standards forpreclinical cancer research. Nature 483:531–533

23. Leong AS, Leong TY (2011) Standardization in immunohistology.Methods Mol Biol 724:37–68

Virchows Arch