Methods 96 (2016) 12–26

Contents lists available at ScienceDirect

Methods

journal homepage: www.elsevier .com/locate /ymeth

A metric and workflow for quality control in the analysisof heterogeneity in phenotypic profiles and screens

http://dx.doi.org/10.1016/j.ymeth.2015.10.0071046-2023/� 2015 The Authors. Published by Elsevier Inc.This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Abbreviations: QC-KS, a heterogeneity quality control metric that uses the KSstatistic; HI, heterogeneity index; QE, quadratic entropy, an HI; nNRM, non-normality, an HI; %OL, percent outliers, an HI.⇑ Corresponding author at: University of Pittsburgh Drug Discovery Institute,

3501 Fifth Avenue, BST3 10052, Pittsburgh, PA 15260, USA.E-mail address: Gough@pitt.edu (A. Gough).

Albert Gough a,b,⇑, Tong Ying Shun a, D. Lansing Taylor a,b, Mark Schurdak a,b

aUniversity of Pittsburgh Drug Discovery Institute, 3501 Fifth Avenue, Pittsburgh, PA, USAbDept. of Computational and Systems Biology, University of Pittsburgh, 3501 Fifth Avenue, Pittsburgh, PA, USA

a r t i c l e i n f o

Article history:Received 9 August 2015Received in revised form 12 October 2015Accepted 13 October 2015Available online 4 November 2015

Keywords:HeterogeneityPhenotypic profilingHigh content screeningSystems biologyDrug discovery

a b s t r a c t

Heterogeneity is well recognized as a common property of cellular systems that impacts biomedicalresearch and the development of therapeutics and diagnostics. Several studies have shown that analysisof heterogeneity: gives insight into mechanisms of action of perturbagens; can be used to predict optimalcombination therapies; and can be applied to tumors where heterogeneity is believed to be associatedwith adaptation and resistance. Cytometry methods including high content screening (HCS), highthroughput microscopy, flow cytometry, mass spec imaging and digital pathology capture cell level datafor populations of cells. However it is often assumed that the population response is normally distributedand therefore that the average adequately describes the results. A deeper understanding of the results ofthe measurements and more effective comparison of perturbagen effects requires analysis that takes intoaccount the distribution of the measurements, i.e. the heterogeneity. However, the reproducibility ofheterogeneous data collected on different days, and in different plates/slides has not previously beenevaluated. Here we show that conventional assay quality metrics alone are not adequate for quality con-trol of the heterogeneity in the data. To address this need, we demonstrate the use of the Kolmogorov–Smirnov statistic as a metric for monitoring the reproducibility of heterogeneity in an SAR screen,describe a workflow for quality control in heterogeneity analysis. One major challenge in high throughputbiology is the evaluation and interpretation of heterogeneity in thousands of samples, such as compoundsin a cell-based screen. In this study we also demonstrate that three heterogeneity indices previouslyreported, capture the shapes of the distributions and provide a means to filter and browse big data setsof cellular distributions in order to compare and identify distributions of interest. These metrics andmethods are presented as a workflow for analysis of heterogeneity in large scale biology projects.

� 2015 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-NDlicense (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction tions of a trait can be visualized as a histogram of the cell popula-

Heterogeneity is a property of cellular systems that implies thepresence of cell-to-cell variability with respect to one or moremeasurable traits (phenotypes) at the molecular-cellular level.Because heterogeneity is a property of the population of cells, itcan be complex and therefore difficult to characterize as a singleparameter. Large scale biology projects typically span many daysor even months, and non-biological variation in heterogeneity overthe course of a study could mask the true biological heterogeneity,suggesting the need for quality control. The details of the distribu-

tion, but visualization of the large numbers of histogramsgenerated in high throughput phenotypic screens is not practicalwithout appropriate informatics tools, and qualitative comparisonsare not sufficient for quality control. In this chapter we present asimple metric for quality control of heterogeneity, and an approachto evaluating heterogeneity in large-scale biology and drug discov-ery programs consisting of measurements on thousands to millionsof samples, which can be applied to High Content Screening, HighThroughput Microscopy, flow cytometry, digital pathology or anymethod that generates large datasets of cells [1–6].

1.1. Implications of cellular heterogeneity in biomedical research, drugdiscovery and diagnostics

It is now well established that cellular heterogeneity is anintrinsic feature observed in cell systems, even when composed

A. Gough et al. /Methods 96 (2016) 12–26 13

of isogenic populations of cells. Cell-to-cell variability is not simplythe result of intrinsic noise in molecular networks; rather it is oftenthe result of deterministic regulatory molecular mechanisms thatremain largely uncharacterized [7–9]. There is growing evidencethat some heterogeneity is related to physiological and evolution-ary adaptations to new challenges [10,11]. A recent study suggeststhat heterogeneity can be decomposed into different groups ofbiomarkers that are consistent with known signaling pathways,implying a mechanistic basis for the cell-to-cell variation [12]. Ithas also been shown that patterns of signaling heterogeneity candistinguish cellular populations with different drug sensitivities[13,14]. It has been proposed that combination therapies can bedevised through computational analysis of heterogeneity [15]. Fur-thermore, the association between cellular heterogeneity andadaptation [10] suggests that ignoring the heterogeneity in thecellular response may lead to the selection of compounds to whichcells will rapidly adapt, leading to a loss of efficacy in drug devel-opment programs. Thus, it is important to incorporate heterogene-ity analysis in large scale screens in drug discovery, and phenotypicprofiles in basic biomedical research, where the goal is to relate acellular phenotype with some condition, such as a disease state,compound effect, protein knockdown, or other sample state ortreatment.

Pluripotent stem cells are a platform with tremendous potentialfor development of patient specific disease models, for modelingbiological development, and for regenerative medicine. However,stem cells exhibit heterogeneity on several levels: in the functionalcapacity to differentiate; in mRNA expression profiles; and in epi-genetic and genetic state [16]. Studies of differentiating cells havefound that heterogeneity reflects the presence of an evolving mix-ture of phenotypically distinct subpopulations, consistent with ahypothesis that differentiating cells transit through multiplerobust and discrete phenotypic states [17–19]. Improved under-standing and manipulation of the differentiation of stem cells willrequire tools to reliably characterize and monitor the evolution ofthe distributions of these subpopulations and their associatedphenotypes.

In typical cell-based phenotypic assays, compound activity ischaracterized by the well average value of a feature or a combina-tion of features, but the cell level data is usually ignored in order toachieve higher throughput. Metrics have been developed toaddress assay variability for large scale screens/profiles includingsignal-to-background and Z0-factor for inter-plate and inter-dayvariability [20–22]. However, these metrics do not address the bio-logical heterogeneity present in individual cell populations. There-fore, new tools and criteria are required for analyzing andcomparing biological heterogeneity in large scale screens/profiles,and for sorting and identifying perturbagens with potentially inter-esting or novel effects on the distribution of cell response among alarge number of patterns of cellular activity. Measuring hetero-geneity is particularly important when applying quantitative sys-tems pharmacology (QSP) to drug discovery programs. A keytenet of our implementation of QSP is the need to have phenotypicassays that reflect the heterogeneity in patients as determined by arange of ‘‘omics” profiles.

1.2. Challenges in the analysis and interpretation of heterogeneity inhigh content phenotypic screens/profiles

There are several challenges to large scale analysis of hetero-geneity and the comparison of heterogeneity between samples,assay runs, and laboratories. These challenges include: (1) largesize and complexity of the cell level data in high content screens/profiles; (2) lack of a standard approach to quality control in mea-suring heterogeneity; (3) lack of standards for the characterizationof the assay and detection system contributions to the measured

heterogeneity; and (4) lack of adequate tools to quickly quantify,compare, review and interpret heterogeneous responses.

In large scale analysis of biological activity taking the average ofa population is a quick way to reduce the data to a size that can bemore easily managed, filtered and interpreted, but neglects theinformation contained in the distribution of cellular responses.The complexity of the cell distributions and the lack of a standardanalytical approach for interpreting heterogeneity remains a bar-rier to incorporating such analyses into a project, despite evidencethat investigation of heterogeneity leads to new insights into theunderlying cellular functions [23,24]. Analyzing all the cellulardata in a compound screen, RNAi screen, genome-wide mutagene-sis [25] or other large biology project is a ‘‘big data” problem thathas not been adequately addressed. For a screen of 100,000 com-pounds and 1000 cells/well there will be 100 million data pointsfor each feature (potentially 4–20 or many more) measured inthe screen. Similarly, an SAR of �240 compounds with 10 concen-trations, triplicate wells, duplicate runs andP1000 cells/well, suchas the data set evaluated in this chapter, can easily amount to wellover 10 million cells. Data that is well organized in a database maybe easily extracted by query, but efficiently analyzing 30–500 mil-lion or more data points acquired over several months for patternsin population distributions at least requires some thought andplanning, and tools for manipulating and comparing distributions.In this chapter we present an approach to that process.

A second challenge in the heterogeneity analysis is quality con-trol. The standardization of quality control in high throughputscreening through the use of the Z0-factor [22,26], or the strictlystandardized mean difference [27], has enabled comparison ofassay performance over a wide range of assay types, routine mon-itoring of assay performance during a screen, and a statistical basisfor understanding the impact of assay variability on the selectionof hits from a screen. For dose–response assays, the V-factor, a gen-eralization of the Z-factor, provides a robust QC measure that takesinto account the complete dose–response, rather than just themaximum and minimum response [21,28]. To be able to comparecellular heterogeneity in order to interpret the biological meaning,standards and quality control are needed to ensure reproducibility.Present assay development guidelines for high content phenotypicscreens address metrics for assay variability [20,21], but do notaddress the reproducibility of the distributions within the wells.The normalization of well average data to on plate controls ade-quately corrects for plate-to-plate variability at the well level butis not typically applied to the cell level data. It is important toemphasize that in typical use, good Z0-factors do not necessarilyindicate that the cellular distributions (i.e. heterogeneity) withinthe wells are consistent from plate-to-plate, as we demonstratehere.

A third challenge in heterogeneity analysis is the lack of refer-ence standards for calibration or characterization of most high con-tent methods and in particular phenotypic methods. For example,in HCS spatial or temporal variation of illumination intensity ordetection sensitivity can contribute ‘‘detection system variation”to the measured values. While approaches to detection system cal-ibration and characterization have been published [14,29–31], sofar standards such as those routinely used in flow cytometry [32]have not been widely adopted in imaging. Typically in screening,each plate will be normalized to the control wells [20,21]. Thisapproach compensates for the average signal level on each plate;however, to compare distributions from plate-to-plate alsorequires normalization of the cell level data between plates. Opti-mally, attention to protocols during assay development, validationand screening/profiling can minimize the variation from plate-to-plate. Here we demonstrate that normalization can be an effectivemeans to compensate for plate-to-plate variation in the range ofintensities, but not necessarily the variation in the shape of the

14 A. Gough et al. /Methods 96 (2016) 12–26

distributions. Although there are many published reports ofheterogeneity in cellular systems, the measurements have beenmade using different methods and each laboratory has taken a dif-ferent approach to characterizing the heterogeneity [3,14,33–35].As a result, there are no established metrics for heterogeneity.The methods developed and presented here are based on analysisof the distributions of the cellular features and should therefore,be generally applicable to any measurement which captures thecell-to-cell variation in a parameter(s). In this chapter we suggesta workflow for quality control of the heterogeneity in phenotypicassays and metrics for monitoring heterogeneity from plate-to-plate.

A final challenge is to quickly quantify, compare, review andinterpret large numbers samples which exhibit varying hetero-geneity in cellular responses. One approach has been to segmentthe population into discrete subpopulations using Gaussian mix-ture models or k-means clustering which effectively reduce thescale of the data [12,36]. This approach can be effective when dis-crete subpopulations can be identified, but may not be effective inlarge scale projects where the changes in heterogeneity may bemore subtle, or the heterogeneity may be more complex. An alter-native approach is to characterize the heterogeneity using shapeparameters [14,34]. This approach makes no assumptions aboutthe shape of the distribution, or the number of subpopulations,but does not provide the detailed segmentation of the cell popula-tion. Here we demonstrate the use of a general set of heterogeneityindices, in combination with an informatics tool in a workflow thatorganizes large numbers of distributions to compare and identifysubsets of interest. This approach enables analysis of heterogeneityon a large scale, and could also be used to identify distributionsthat would benefit from more detailed segmentation analysis.

2. Material and methods

2.1. Sample preparation

2.1.1. Reagents and cellsFormaldehyde (37%), Triton X-100, and Tween 20 were pur-

chased from Sigma–Aldrich (St. Louis, MO). Recombinant humaninterleukin-6 (IL-6) was purchased from R & D Systems, Inc.(Minneapolis, MN). Mouse monoclonal anti-pSTAT3-Y705(Cat# 612357) primary antibodies were purchased from BectonDickenson Biosciences (San Diego, CA). The secondary goat antimouse-IgG conjugated with Alexa Fluor 488 (Cat# A11001) waspurchased from Life Technologies (Grand Island, NY). Hoechst33342 was purchased from Invitrogen (Eugene, OR). Black, clearbottom, 384 well HCS Assay Plates (Cat# 781091) were purchasedfrom Greiner Bio-One (Monroe, NC). Cal33 human head and necksquamous cell carcinoma (HNSCC) cells [37,38] were kindly pro-vided by Dr. Gerard Milano (University of Nice, Nice, France). Thecell line was maintained in Dulbecco’s modified Eagle’s medium(DMEM, Life Technologies) supplemented with 10% fetal bovineserum (Gemini Bio-Products, West Sacramento, CA), 100 U/mlpenicillin and 100 mg/ml streptomycin (HyClone, Logan, UT). Cellswere maintained in humidified incubators at 37 �C with 5% CO2.

2.1.2. Compound dilutionsCompounds were tested as 10-point, 3-fold serial dilutions over

a range from 50 lM to 2.5 nM (except as noted), in triplicate on theplate, and in duplicate runs. Master compound dilutions were pre-pared from 10 mM stock concentrations (except as noted for speci-fic compounds) in DMSO in 384 well plates by serial dilution into100% DMSO on the Bravo liquid handling platform (Velocity11,Menlo Park, CA). Replicate daughter plates containing 2 lL of theserially diluted compounds were prepared using a 384-well

transfer on the Bravo. Aluminum adhesive plate seals wereapplied and plates were stored at –20 �C. Each assay plate included32 negative (+IL-6) and 32 positive (�IL-6) control wells whichwere used to calculate the Z0-factor and for normalization.

2.1.3. Assay plate processingCal33 cells were seeded in 384-well plates using a MultiDrop

(Thermo Scientific) to dispense 2000 cells in 40 lL/well to reach50% confluence on the day of fixation. The cells were incubatedat 37 �C for 24 h, then the medium was replaced with serum freemedium and the cells were incubated for another 24 h. To establishthe optimal IL-6 concentration, STAT3 phosphorylation wasinduced by IL-6 in a 2-fold serial dilution for 10 final concentra-tions descending from 100 ng/mL. For inhibition of STAT3 phos-phorylation, compound daughter plates were withdrawn fromstorage, thawed in a 37 �C dry oven, centrifuged for 1–2 min at50 � g, and the plate seals were removed. Intermediate dilutionplates were generated using a Matrix pipette to transfer 38 mL ofserum free medium (SFM) into the wells of the daughter plates.The diluted compounds were mixed by repeated aspirate-dispense cycles on the Bravo, and then 5 lL of diluted compoundwas transferred into the assay plates. After 3 h of compound incu-bation at 37 �C, 5 lL of a 10� stock solution of IL-6 was addedusing the Bravo for a final concentration of 50 ng/mL, and theplates were spun at 50 � g for 1 min then incubated for 14 min.The plates were fixed by the addition of 50 lL of 7.4%paraformaldehyde (37 �C) containing 4 lg/mL Hoechst 33342 inPBS using the Elx405 plate washer dispenser (BioTek, Winooski,VT). After 30 min at room temperature (RT), the fixed cells werewashed twice with 80 lL PBS. PBS was aspirated, and the fixedcells were permeabilized by the addition of 50 lL of 95% ice-coldmethanol. After 30 min on ice, the 95% methanol was removedfrom the assay plates and the cell monolayers were washed twicewith Tween 20 blocking buffer, and then incubated for 30 min atRT in blocking buffer. 24.5 lL of a 1:100 dilution of mouse anti-pSTAT3-Y705 primary antibody was added using a Matrix pipetteand incubated for 1 h at RT in the dark. After washing twice withblocking buffer on the plate washer, the plates were incubated15 min. at RT, the buffer was aspirated and 24.5 lL of a 1:1000dilution of the secondary antibody added with a Matrix pipetteand incubated for 1 h at RT in the dark. The plates were thenwashed twice with 80 lL PBS, sealed with foil and stored at 4 �C.

2.2. High content screening

Images were collected on the ImageXpress Ultra (IXU) (Molecu-lar Devices LLC, Sunnyvale, CA) a point-scanning confocal HCS plat-form. The IXU was used to sequentially acquire two fluorescencechannels, Hoechst (Ch1) and Alexa 488 (Ch2), for 2 fields per wellusing a 20� (0.45-NA) ELWD objective. The Hoechst channel wasexcited using the 405 nm laser at 10% power with emission at447 ± 60 nm, and the Alexa488 channel used the 488 nm laser at10% power with emission at 514 ± 17 nm. Pixels were binned by2 with 2000 scan lines collected over an area 800 � 800 um. Imageanalysis was carried out using the Translocation Enhanced (TE)application in the MetaExpress software (Molecular Devices LLC,Sunnyvale, CA) as follows. The nuclear compartment was identifiedin Ch1 with a threshold intensity of 1500 above background and anarea of 20–1000 lm2 with auto separation of touching compart-ments (the nuclear mask). Analysis of the distributions of nuclearintensities on all of the plates in this data set showed that althoughthere was some variation in the range of intensities from plate-to-plate the vast majority of nuclei on all the plates had averageintensities that were well above the threshold used to segmentthe images, resulting in consistent nuclear regions. The nuclearmask was eroded 1 lm to avoid edge effects and to create the Inner

A. Gough et al. /Methods 96 (2016) 12–26 15

region mask in Ch2. The activation of STAT3 in each cell was mea-sured as the Mean Inner IntensityCh2 (MII) of the Alexa 488 labelwithin the Inner region mask in Ch2. For each well the averageSTAT3 activity was calculated as the average of the MII for all thecells in the well. All the calculated cell and well features were auto-matically stored in the MDCStore (Molecular Devices, Sunnyvale,CA) SQL-Server (Microsoft, Redmond, WA) database. For the analy-ses presented here, the data were extracted directly into Spotfire(TIBCO, Boston, MA) using SQL queries.

2.3. Quality control

2.3.1. Assay quality controlOn each plate, columns 1, 2, 23 and 24 were used for a total of

32 negative and 32 positive control wells, split equally in each col-umn. The assay parameter for the negative and positive controlwells, MII, was used to calculate the Z0-factor [22] and the signal-to-background (S/B = Mean[Well Average MII(negative control)]/Mean[Well Average MII(positive control)]) for each plate. Duringthe screen plates with a Z0 < 0.3, or S/B < 5 were rejected. Therewere 117 plates (384 well) that passed the QC for the screen, witha Z0 P 0.3, and a S/BP 5.

2.3.2. Heterogeneity quality control2.3.2.1. Normalization. We normalized the cell level data in eachwell to a common median to minimize the variation in the cellulardistributions across the 117 plates in this data set that is related tosample preparation, image acquisition and/or algorithm perfor-mance. We maintained a nominal assay scale by normalizing to areference distribution. We selected 8 plates to use as a virtual‘validation’ plate set and pooled the cell level data from the nega-tive control wells to establish the reference distribution and thenormalization value. The median of the reference distributionwas used for normalization of the plates. Since the positive controlin this assay was no stimulation with IL-6, resulting in essentiallyno pSTAT3 signal above background, single point normalization tothe median in the reference distribution from the validation plateswas used as follows:

NormðMIIðcellpi Þ ¼MedianðMIIðref ÞÞ

MedianðMIIðcellspnegÞÞMIIðcellpi Þ ð1Þ

where MIIðcellpi Þ is the measured value of cell i on plate p,MedianðMIIðref ÞÞ is the median MII of the reference distribution,and MedianðMIIðcellpnegÞ is the median value of the pooled cells inthe negative control wells on plate p.

2.3.2.2. Quality control metric. To measure the plate-to-plate repro-ducibility of the heterogeneity in the cellular responses, we com-pared the cellular distributions in the negative control wells onall plates to reference distributions from the selected ‘validation’plates using the Matlab function ‘kstest2()’ to calculate theKolmogorov–Smirnov (KS) statistic as a quality control metric(QC-KS):

QC-KS ¼ max jCDFpw � CDFref j ð2Þwhere CDFpw is the cumulative distribution function of the distribu-tion of the MII in a negative control well w on plate p and CDFref isthe cumulative distribution function for the reference distribution.The use of the KS statistic as a very general assay parameter is wellknown in HCS [39–44], but here we use it for quality control. Toestablish a QC-KS threshold we used the median(QC-KS)+ 3 * K * MAD for the pooled control wells on all the plates in thisdata set, where MAD is the median absolute deviation of theQC-KS value for the plates and K = r/MAD for a normal distribution(1.4286). To gain insight into the variation in the distributions we

evaluated 5 additional metrics for potential correlation with thepass/fail scoring of the plates, Cell Count, %Active, and three previ-ously published heterogeneity indices (HIs), the quadratic entropy(QE), non-normality (nNRM) and percent outliers (%OL) [14]. Cellcount was provided by the imaging algorithm as a nuclear countin Ch1, and %Active was calculated as the percent of cells withMII > 3 standard deviations above the MII of the positive controlin Ch2 on the same plate.

2.4. Indices of heterogeneity

2.4.1. Selection of the heterogeneity indicesA primary goal in the quantitation of heterogeneity was to

select indices that reflect properties of the distribution that couldbe related to the underlying biology. The selected indices reflect:(1) the overall diversity in the cell population, QE; (2) the presenceof subpopulations, nNRM; and (3) the number of outlier cells, %OL.We selected the quadratic entropy (QE) as a measure of cellulardiversity that incorporates information not only on the numberof different phenotypes, but also the magnitude of the differencesbetween phenotypes [45,46]. The QE has been shown to be usefulin quantitation of the cellular diversity in cancer tissue sections fordiagnostic applications [34].

To identify the presence of subpopulations (macro-heterogeneity as defined by Huang [7]) we adopted a measure ofNormality. The nNRM index uses the KS statistic as a measure ofthe difference between the test distribution and a Normal distribu-tion with the same mean and standard deviation. As a result, thenNRM value is 0 for a Normal distribution and increases withincreasing deviation from a Normal distribution, as would beexpected for subpopulations with distinct phenotypes. Examplesof macro-heterogeneity include the distinct states of progenitorvs. differentiated cells, the phases of the cell cycle and the timedependent changes in the intracellular distribution of proteins suchas transcription factors.

The %OL indicates the presence of outlier cells that respond dis-tinctly from the majority, which is expected to be important in thedevelopment of therapeutics, where a small number of resistantsub-clones may exist prior to treatment, and potentially result inhigh rates of relapse [48]. The %OL index is based on the standardstatistical definition of outliers used in box plots.

2.4.2. Calculation of heterogeneity indicesThe three HIs were defined previously, as were the methods

used for calculation [14]. The three HIs are defined by the followingequations:

QE ¼XNi>j¼1

dij � pi � pj ð3Þ

nNRM ¼ max jCDFwell � CDFnormj ð4Þ

%OL ¼ 1� CountðLIF : UIFÞN

� �� 100 ð5Þ

In Eq. (3) dij is a distance matrix that linearly weights the pro-duct of the subpopulation probabilities, pi and pj, by the magnitudeof the intensity difference. In Eq. (4) CDFwell is the cumulative dis-tribution in a test well, and CDFnorm is the cumulative distributionfunction for a normal distribution with the same mean andvariance as the cells in the test well. In Eq. (5) Count(LIF:UIF) isthe count of cells in the range from the lower inner fence to theupper inner fence, as defined for a standard box plot, and N isthe total number of cells in the distribution. The HIs werecalculated using a Matlab script.

16 A. Gough et al. /Methods 96 (2016) 12–26

2.5. Measuring the performance of the HIs

To evaluate the performance of the HIs in distinguishing distri-butions, two classifiers were developed to predict 8 target distribu-tions using only the HIs as inputs. The 8 target distributions wereconstructed by hierarchical clustering of the distributions from 19plates with standard compounds. The distributions were binnedinto 30 equal intensity ranges and then clustered using the Matlabfunction clusterdata. The minimum similarity threshold wasadjusted to give the 8 distributions with the highest similarity.The Random Forest (RF) and Binary Decision Tree (BTree) modelswere constructed using the R [49] package rattle [50]. The inputdata had 585 observations and the parameters for the Random For-est model were Number of Trees = 500, Number of Variables (at eachsplit) = 1 and for the BTree model were Min Split = 20, MinBucket = 7, Max Depth = 30 and Complexity = 0.01, which resultedin a tree with 8 nodes and a depth of 5.

2.6. Visualization of distributions

The cellular distributions in Figs. 1, 2, 4 and 5 are presented ashisto-box plots as previously discussed [14]. The histo-box plotswere generated in Spotfire (TIBCO, Boston, MA) by overlaying thehistogram of intensity (vertical axis) on a conventional box plot,a standard feature in the boxplot graphic in Spotfire.

2.7. Informatics tool for reviewing and comparing heterogeneity inlarge scale screens/profiles

The browser shown in Fig. 8 was developed as a dashboard inSpotfire that combines visualizations of the cell level data along

Fig. 1. Dose response characterization of the activation of STAT3 by IL-6. The graph showsD) Images of STAT3 labeling illustrate the high level of variation in STAT3 activation at coto a dynamic range of 100–40,000, then converted to 8-bit and pseudo-colored using the+ 3 Stdev(positive control) are considered Activated, and the %Activated cells is shown i

with the well level data. The views are linked so that the dataselected in one view is also selected in all other views. The browseris available for non-commercial use from the authors as a Spotfirefile with example data.

3. Results and discussion

3.1. The need for quality control for heterogeneity analysis in screens/profiles

It is important in heterogeneity analysis to clearly distinguishbetween detection system or assay variability and biologicalheterogeneity. In imaging based systems, such as HCS, spatial vari-ation of illumination intensity or detection sensitivity (or both),often seen as reduced intensity in the corners, can contribute toapparent variation in the cell-to-cell measurements. Methods forcorrecting non-uniform illumination or detection sensitivity areavailable [51,52], but often are not used because variation acrossthe field, especially when it is stable from field-to-field, has littleimpact on the population average values that are most often usedas assay readouts. However, for heterogeneity analysis it is impor-tant to understand the impact of the detection system on cell-to-cell variation. Comparative studies between imaging and flowcytometry that established the reproducibility limit for this assayon a wide field HCS system were previously reported (see forexample Fig. S3 in [14]). Because this was a retrospective studywe were not able to correct images from the IXU prior to dataextraction. However, because point scanning laser confocal sys-tems like the IXU scan a stabilized laser light source to each pixel,in general illumination uniformity is less of an issue than in wide

the activation of STAT3 by twofold serial dilutions of IL-6 starting at 100 ng/mL. (A–ncentrations from (A) 12.5 ng/mL to (D) 100 ng/mL. Original images were all scaledcolor lookup table shown. Cells with a labeling intensityPMean(positive control)n the pie charts.

Fig. 2. Standard assay quality measures are not sufficient for assessing the reproducibility of heterogeneity. Similar Z0-factors indicate consistent well-to-well assayreproducibility across plates but provide no indication of the variability of cellular response distributions (A). Even after plate-to-plate normalization there remains variationin the shapes of the distributions (B). Comparison of the cellular distributions of STAT3 activity in the IL-6+ control wells on five plates with essentially the same Z0-factorshows that the distributions can be very different from plate-to-plate. The S/B of the assay does give some indication of variation in the range of the data. As a result, theheterogeneity indices, QE, nNRM and %OL vary from plate-to-plate, with Plate 734 having the largest deviations. The QC-KS value provides a quantitative measure of thedeviation of the distribution from the reference distributions established during assay validation. In this case, Plates 926 and 1302 have the lowest deviation (0.13) from thevalidation distributions.

A. Gough et al. /Methods 96 (2016) 12–26 17

field systems. Comparison of the cell distributions from the controlwells collected on the IXU with previously published distributionscollected on intensity corrected images from a wide field HCS sys-tem showed that they were essentially the same, suggesting com-parable performance for the two detection systems.

The standard metrics used for assay quality control in screeningare designed to measure the well-to-well, plate-to-plate and day-to-day repeatability of the well level assay features (e.g. averageor total intensity, area, etc.), but do not indicate anything aboutthe distribution of cellular responses within the well. To evaluatereproducibility for the heterogeneity in the individual cellularresponses, we used data from a high content, cell-based, pheno-typic SAR screen of compounds that inhibit STAT3 activation byIL-6. The data set consisted of 117 microplates (384 wells) withtriplicate, 10 point, 3-fold dilutions of 261 compounds, some ofwhich were run multiple times. In this screen plates with aZ0 < 0.3 or a S/B < 5 were failed and repeated. The concentrationof IL-6 used for this screen, 50 ng/mL, was selected to produce amaximal activation of STAT3 without an excess of IL-6. Fig. 1 showsthe dose–response of STAT3 to IL-6. The selected concentration(1C) results in significant variation in the activation of STAT3 thatcan be seen in image C, with 84% activation (1, Pie Chart C).The higher concentration of 100 ng/mL (1D) produced similar

heterogeneity in the image (1D) and %Activation (1, Pie Chart D),indicating that 50 ng/mL IL-6 was sufficiently near the maximumvalue.

Evaluation of population distributions in the +IL-6 control wellson these plates indicated there was significant variability, indepen-dent of the Z0-factor (Fig. 2). Fig. 2A shows the raw MII of the IL-6induced cells in the negative control wells. The distributions clearlyvary in the range of intensities. After normalization of the intensityrange to themedian of the reference distribution (Fig. 2B), themed-ian MII is the same for all five plates but there are clearly some dif-ferences in the shapes of the distributions. For example, plate 734lacks the peak of STAT3 activity around 20 K that is obvious onplates 928, 1301 and 1302. Plate 729 has a smaller peak at an inten-sity of just over 20 K. There are some differences in the S/B mea-sured during the course of the screen that are perhaps indicativeof the differences in the distributions, but because the S/B > 5 onall the plates, no distinction was made between them during thescreen. We calculated the three HIs for these five plates to evaluatethe potential impact of this variability on measurements of hetero-geneity. On plate 734 the QE, NRM, and %OL are all much higher,indicating a broader distribution that is less normal and has moreoutliers than the other plates. Plate 729 also has a somewhat differ-ent distribution when compared with plates 928, 1301 and 1302;

18 A. Gough et al. /Methods 96 (2016) 12–26

however the differences in the HIs on this plate are lower, as mightbe expected, with only the nNRM indicating a substantial differencein normality. The variation in these distributions and measured HIssuggested a need for a QCmetric to quantitate the reproducibility ofthe cellular distributions on the plates. Because the KS statistic isdesigned for comparing distributions, we adopted it as a new met-ric, the quality control-KS (QC-KS) that, as described below and canbe easily used to monitor the population distributions over time inlarge scale biology projects.

3.2. Workflow for quality control of heterogeneity in phenotypicscreens/profiles

It is clear from Fig. 2, that a similar Z0-factor does not imply con-sistency in cell level distributions from plate-to-plate. It is impor-tant though to note that the high reproducibility of this assay, asindicated by a robust Z0-factor, suggested that good reproducibilityin the distributions could be achieved, as was the case here. Opti-mally, the data acquisition would be controlled to minimize varia-tion in the assay data range and allow comparison of biologicalheterogeneity across plates, and especially over long times, how-ever, here it was essential to normalize the data prior to applyingthe quality control.

To correct for the plate-to-plate variation in the intensity range,the data from each well was normalized to the reference distribu-tion using equation 1. The resulting normalized distributions forthe +IL-6 controls on all 117 plates are shown in Fig. 3. After nor-malization the median of the distributions were consistent,although there were still small differences in the shapes of someof the distributions. A component of those differences is in the rel-ative height of the activated cells (upper peak) vs non-activatedcells (lower peak). As previously reported, a fraction of the cellsin this assay, ranging from about 7–12% of the cells in a well arenot activated by IL-6, even though essentially all the cells wereshown to be capable of STAT3 activation when stimulated by

Fig. 3. Normalization of the distributions for all 117 plates. The plates were normalizeddistributions in the pooled IL-6+ control wells indicate that the normalization is effectiv

Oncostatin-M [14]. The variation in distributions remaining afternormalization was evaluated quantitatively using the QC-KS.

The QC-KS statistic was selected as a quantitative QC measureof the difference between the distributions in the control wellson each plate and a reference distribution. To evaluate perfor-mance of the QC-KS metric the distributions in the negative con-trols on all 117 plates (Fig. 3) were reviewed and 9 plates wereidentified that appeared to have ‘‘abnormal” distributions(Fig. 3): plates 726–735 in the first row and plate 893 in the fourthrow. In addition, a tenth plate, 888 in the fourth row, appears tohave a ‘‘clipped” distribution, with no intensities above about25 K, but otherwise the distribution is reasonably consistent withthe other plates. Based on this qualitative comparison, these 10plates were labeled as failed and used to evaluate the QC-KS statis-tic. Fig. 4 compares the distributions in passed (blue) and failed(red) plates for several metrics. For the QC-KS metric (Fig. 4A)passed plates have a QC-KS < 0.2, while failed plates generally havea QC-KS > 0.25. Only plate 888 (green), which was failed for aclipped distribution, has a QC-KS < 0.2. Because cell density mighthave contributed to the difference between passed and failedplates we compared the distributions of cell density (Fig. 4B).The distributions in cell count on passed and failed plates wereessentially the same, suggesting that the abnormal distributionswere not associated with a variation in cell density. Another factorthat might have contributed to the differences in the distributionsis the degree of activation by IL-6. Fig. 4C shows that the percentactive cells was consistently lower in failed plates, except for plate888, and one passed plate, number 759. Further evaluation of plate759 found that it had a 2-fold higher background in the positivecontrol, leading to a lower determination of percent active, eventhough the negative control distribution looks normal. This lowerlevel of activation of the cells on the failed plates suggested thatit was a contributing factor in the difference.

An important question though, was whether the differences indistribution on the failed plates would significantly impact the

to the median of the pooled reference validation plate controls. Histograms of thee in establishing a consistent signal range across all plates.

Fig. 4. Selection of a QC metric of reproducibility in heterogeneity analysis. Each pair of histograms (A–F) indicates the distribution of plates labeled as Fail (red) or Pass(blue). A single Failed plate (888) with a clipped distribution is highlighted in green for reference. (A) The KS statistic, QC-KS, provides a clear interpretation and gooddiscrimination between Pass and Fail, except for plate 888. (B) Although cell count might contribute to the distribution of cellular response on a plate, in this case there is nosignificant difference in cell count between Passed and Failed plates. (C) Differences in the percent activated cells also might result in variation in the distributions, and thatdoes appear to be the case in this assay. (D) The Failed plates have a much broader distribution as indicated by the increase in QE. (E) The Failed plates have a less normaldistribution as indicated by the increased nNRM. (F) Some of the Failed plates have an increased number of outliers indicated by the increased %OL.

Fig. 5. Quality Control workflow for heterogeneity analysis. A. To establish and quality control heterogeneity in high throughput imaging projects, the distributions in thecontrol wells are evaluated during validation to assess the reproducibility of the heterogeneity. B. The distributions from the control wells on the validation plates are pooledto establish reference distributions. C. Each control well on the validation plates is compared with the reference distribution using the KS statistic (QC-KS), results are shownhere as a heatmap, to quantitatively assess reproducibility. D. The QC-KS statistic is used to monitor the distributions during a screening campaign. The solid horizontal line isthe median(QC-KS), and the dashed lines represent the median ±3 * MAD * K, the selected QC limit. E. The heatmap of the control wells on all the plates shows the variation inthe QC-KS values from well-to-well and plate-to-plate, and the 10 plates that were Failed. In a new project, the Failed plates would be flagged for review.

A. Gough et al. /Methods 96 (2016) 12–26 19

measurement of the HIs. Fig. 3D–F shows the histograms of the QE,nNRM and %OL for passed and failed plates. Both QE and nNRMshow clear differences between passed and failed plates, whileonly some of the failed plates show a level of %OL that is consistentwith the passed plates. These results suggest that comparisons of

HI measurements between passed and failed plates would not bemeaningful. Furthermore, the results suggest that the QC-KS is auseful metric to QC plates for heterogeneity analysis. To select athreshold QC-KS value by which to flag plates we used the medianQC-KS value +3 * K * MAD (see materials and methods) of the

Fig. 6. Evaluating the performance of the Heterogeneity Indices in predicting the shape of the cellular distributions. A. Hierarchical clustering was used to group distributionsfrom all the wells on 19 plates into 8 distinct classes. B. The distributions were then split into a training set and test set to construct a Random Forest classifier to predict thecluster number using only the 3 heterogeneity indices. C. A simpler decision tree model was also constructed to predict the distribution class. Although the performance waspretty good, Cluster 3 was not successfully separated from clusters 1, 2 and 4.

Table 1Distribution of target clusters for classifier development.

Observations Class from clustering

1 2 3 4 5 6 7 8

Number 335 218 75 75 25 17 59 33Fraction of total 0.40 0.26 0.09 0.09 0.03 0.02 0.07 0.04

20 A. Gough et al. /Methods 96 (2016) 12–26

QC-KS values for all 117 plates, which gave a threshold value of0.19. Applying that threshold in Fig. 2, for example, the QC-KS val-ues for the plates show that plates 729 and 734 would be flaggedby this criterion. In Fig. 4A, all 9 plates that were failed for abnor-mal distributions would be flagged, while plate 888 which wasfailed for an artifact would not be flagged.

Fig. 5 illustrates the workflow that was adopted for quality con-trol of biological heterogeneity. Since this was a retrospective anal-ysis, we did not have true validation plates generated during assaydevelopment, so we pooled the negative control wells from 8plates prepared on 4 different days to establish the reference distri-butions of the negative controls. In new projects we use a set ofvalidation plates to establish the reference distributions, alongwith the signal-to-background, Z0-factor and other assay variability

Table 2Random forest confusion matrix.

Actual Predicted

1 2 3 4 5 6 7 8

1 319 17 3 0 0 0 0 02 24 199 3 0 0 0 0 03 8 2 60 0 0 0 0 04 0 0 1 70 1 0 0 05 0 0 0 0 23 0 0 06 0 0 0 0 0 17 0 07 0 0 0 0 0 0 50 38 0 0 0 0 0 0 0 37

metrics. Fig. 5A illustrates the consistency of the distributions onthose 8 plates after normalization. We specifically selected platesprepared on 4 different days so the variability would be similarto a set of validation plates prepared on different days in a de novoscreening project. The positive and negative controls are pooled toconstruct the reference distributions (Fig. 5B) and then the QC-KSvalues are calculated for all the individual control wells on the val-idation plates and visualized as a heat map (Fig. 5C). The predom-inant blue color in the heat map indicates consistent QC-KS values,which are generally below 0.2, and therefore distributions that aresimilar to the reference distributions, as expected. There are sev-eral (0–5) wells on each plate where the QC-KS value is between0.2 and 0.3 (blue–gray), but the average in each case is well below0.2. Using the distribution of QC-KS values on the validation platesa pass/fail threshold of 0.2 was selected and used to score the aver-age QC-KS on the 117 plates in the project (Fig. 5D). Nine plateswere flagged by this criterion, and examination of the distributionson those plates (Fig. 3) and the heat map of the wells on those

Table 3Variable importance in random forest classifier.

HI Class from clustering

1 2 3 4 5 6 7 8

QE 57.11 31.86 7.93 40.33 22.79 13.81 73.29 38.39nNRM 37.12 19.21 12.07 58.12 26.17 �0.97 8.03 43.85%OL 31.79 49.31 6.58 5.32 20.67 13.32 41.87 17.26

Table 4Binary decision tree confusion matrix.

Actual Predicted

1 2 3 4 5 6 7 8

1 297 42 0 0 0 0 0 02 76 146 0 4 0 0 0 03 31 29 0 10 0 0 0 04 0 1 0 60 11 0 0 05 0 0 0 0 23 0 0 06 0 0 0 0 2 13 2 07 0 0 0 0 0 7 42 48 0 0 0 0 0 0 0 37

A. Gough et al. /Methods 96 (2016) 12–26 21

plates (Fig. 5E) confirmed that the distributions were not consis-tent with the other plates, so they were not used in the remaininganalysis. In general, the distributions on the failed plates appear toshow a lower level of induction of STAT3. One other plate, number888, was failed due to an apparent data processing error thatclipped the distribution, leaving a total of 107 plates for hetero-geneity analysis.

3.3. Use of the heterogeneity quality control work flow in HCS, highthroughput microscopy and other high throughput cytometry methods

The KS statistic is a standard metric for comparing distributionsand has been used in different ways in high throughput micro-scopy, flow cytometry and high content assays and screens. TheKS statistic measured between the distribution in a sample andthe distribution of the plate controls has been used in cytometryas a very general assay metric that avoids any assumption aboutthe nature of the distribution of cellular responses [39,44,53].The KS statistic has also been used as a normality test by compar-ing a sample distribution to a normal distribution with the samemean and standard deviation [14,44,47] and is used in the non-Normality (nNRM) heterogeneity index [14]. Here we presentanother application of the KS statistic as a QC metric (QC-KS)for measuring the plate-to-plate or day-to-day reproducibility of

Fig. 7. Filtering and drilling into 22,000 distributions using the Heterogeneity Indices. A. Tto sort the cell distributions in the horizontal and vertical directions, respectively. This pris composed of all the cells from all the wells for which the QE and KS-norm indices fell wthat bin, but are now displayed with finer resolution on the binning for more detailed r

cellular distributions in HCS, high throughput microscopy and highthroughput flow cytometry. The integration of this measure andworkflow into a screening or profiling project would be greatlyfacilitated by the incorporation of KS analysis in the software forHCS and other cellular analysis platforms, which would allow theQC-KS value to be produced as a standard feature, along withZ0-factor, V-factor, S/B and other assay metrics. The QC-KS statisticcan also be easily be calculated in most statistics and numericalanalysis packages. While it would be useful to establish generalQC limits for the QC-KS metric, the extent of variation in the cellu-lar distributions and the tolerance for variability in the projectshould be established during assay development and validation,along with the QC-KS threshold. In this study we found that athreshold of 0.2 worked well to reject plates with clearly differentdistributions.

3.4. Application of heterogeneity indices to large scale analysis ofheterogeneity

One of the challenges in large scale analysis of heterogeneity issorting through large numbers of distributions to identify a subsetof interest. On the 107 plates in this SAR project there are morethan 20,000 compound test wells and therefore more than20,000 distributions to evaluate. Although the heterogeneityindices were previously shown to be useful in characterizing cellu-lar distributions as heterogeneous or not, micro vs macro hetero-geneity, and with or without outliers [14], we wanted to testwhether these three indices captured sufficient information aboutthe distributions to enable their use as filters to identify common,or unusual patterns of responses. To construct a classifier that usedonly the three heterogeneity indices to predict the shapes of thecellular distributions, we selected a subset of plates that containedcontrol compounds with known STAT3 activity, to ensure thatthere were a significant fraction of active wells. Hierarchical clus-tering of the 837 distributions on the standard plates was usedto create 8 target distribution classes (Fig. 6A). Although the com-pounds on the standard plates were all active, because the plates

he QE and KS-norm indices were binned into 10 uniformly sized bins, and then usedovides an overview of the general distribution shapes. Each distribution in this viewithin that bin. B. Selecting a single bin to zoom into the distributions that compriseeview of the distributions within that subset of wells.

Fig. 8. Heterogeneity Browser for High Throughput Cytometry. The filtering concept in Fig. 7 was expanded into an interactive Heterogeneity Browser in which the cellularheterogeneity in the wells can be identified and reviewed in a variety of ways. All data in the browser is linked, so that selection of data in one graph highlights the data in allother graphs. Here, the distributions with %Activation = 45–55% have been selected in D, and are highlighted in cyan in all the views. A. 2D matrix of distributions withincreasing QE index on the horizontal axis and increasing normality (decreasing nNRM) on the vertical axis. B. 2D matrix with increasing % OL on the horizontal axis and againdecreasing nNRM on the vertical axis. C. The dose–response of distributions for a two compounds. The interface allows scrolling through all the compounds in the selection.D–G. Histograms of the percent activated cells and the 3 HIs for the pooled replicate wells over the whole data set. D. %Activated cells. E. QE index. F. nNRM index. G. %OLindex. H. List of all compounds highlighting one in the current selection.

Fig. 9. The recommended workflow for heterogeneity analysis in screens or largescale biology projects. For new projects, heterogeneity analysis should be incorpo-rated early in the development of the assay. For retrospective projects, step 1 shouldbe to evaluate the range of the distributions of the positive and negative controls inthe whole data set, and step 2 would be to normalize the data if necessary toestablish consistent distributions from plate-to-plate. Steps 3–6 would remain thesame.

22 A. Gough et al. /Methods 96 (2016) 12–26

were from a concentration series there were also many inactivewells. Table 1 shows the number of distributions that contributeto each target class, clearly indicating that 2/3 of the distributionswere of class 1 or 2 (activated STAT3) (Fig. 6A). The data weredivided into training (535 observations) and test (302 observa-tions) sets and used to construct 2 classifiers, a Random Forest(RF) and a Binary Decision Tree (BTree).

Fig. 6B illustrates the results of the classification of the distribu-tion in each well on the standard plate using the RF classifier usingonly the three heterogeneity indices as inputs. As can be seen bycomparison of the distributions in Fig. 5A and B, and the confusionmatrix for the RF classifier (Table 2), the RF model does an excel-lent job of classifying the distributions. The classifier performs beston classes 4–7, having the least confusion where the distributionsindicate various degrees of inhibition. The performance on theseclasses of intermediate inhibition, which are near the IC50s, areof significant interest, as it is these patterns of inhibition that arehypothesized to be associated with specific mechanisms of action[13,54]. The other classes, where the distributions are more likethe negative or positive controls, have about a 10% error rate.The relative importance of the three HIs in the RF classifier isshown in Table 3 and indicates that all three HIs contribute signif-icantly to the majority of the classes.

Fig. 6C illustrates the results of classifying the distributionsusing the BTree classifier. The BTree uses a series of binary nodesto split the distributions into classes based on a threshold at eachnode. Comparison of the distributions in Fig. 5A and C, and the con-fusionmatrix in Table 4, indicates that although the performance ofthe BTree is not as good at the RF, it still provides good classificationof the distributions. The BTree also provides a simple set of rules forusing the HI values to identify a specific class of distribution, for

example, most of the Class 6 distributions (64%) can be found withQE = 0.016–0.032 and nNRMP 0.055. As with the RF classifier theBTree performs best on classes 4–7, missing class 3 completely. Itis clear from the confusion matrix in Table 4 that most Class 3 dis-tributions were assigned equally to classes 1 & 2, with about 15%assigned to class 4. So although the BTree is not as efficient at

A. Gough et al. /Methods 96 (2016) 12–26 23

distinguishing classes 1–3, it works reasonably well for the mostinteresting classes, 4–7.

The results of the classification indicate that the three HIs usedin this study effectively distinguish distributions in the rangewhere there is partial inhibition of STAT3. The performance ofthe HIs in predicting population distributions suggests that theycan be used effectively to filter population distributions and iden-tify distribution classes of interest in HCS, high throughput imag-ing and other cytometry projects.

3.5. Comparisons of compound heterogeneity in large data sets

A goal of this project was to develop a method to evaluateheterogeneity in large data sets. The data set in this study consistedof over 20,000 distributions comprised of over 30 million cells.Clearly some means of organizing or filtering the data is needed.Fig. 7A illustrates one approach that we have found to be useful.The data is arranged and binned using the HIs in order to providean overview of the range of distributions in the data set. In thisgraphical view, generated in Spotfire (TIBCO), the QE and nNRMindices are divided into 10 equal size bins over the full range ofthe HIs in this data set, and the histograms of the distributionsare displayed as a 10 � 10 matrix with QE increasing in the hori-zontal direction (left to right) and the nNRM increasing in the ver-tical direction (top to bottom). Each histogram in this view iscomprised of all the cells in all the wells in that HI bin, and thearrangement exposes the range of distributions in the dataset.For example, in the lower right, high QE, high nNRM, there is a ser-ies of bimodal distributions, which is a clear indication of differen-tial compound sensitivity among subpopulations of cells. Incolumn 2 row 4, the distribution highlighted in red is clearly moreunimodal, although the width of the distribution suggests micro-heterogeneity in that population. We expect that the individualdistributions that comprise the highlighted distribution all have asimilar shape because they are in the same row and column in thistable so they all have a similar QE and nNRM, and this is supportedby examination of the individual distributions. Zooming into theunderlying distributions (Fig. 7B) confirms that they are consistentin median value, width and shape and therefore exhibit a similardegree of micro-heterogeneity.

The use of the HIs to arrange distributions for comparison isexpected to be a valuable approach to the analyzing the evolutionof population distributions in the differentiation of stem cells. Evo-lution of the distribution of phenotypes may not always lead todiscrete subpopulations, therefore direct comparisons of the distri-bution shape using the method described here, coupled with aquantitative measure of the magnitude of the difference usingthe KS statistic, should be a useful approach.

3.6. Browsing for patterns of heterogeneity in compound screens

Although using the HIs to create multiscale views of hetero-geneity provides a quick means to select and review distributionsby shape, we wanted an interface that would also enable the selec-tion of distributions of interest by other characteristics, such ascompound ID, compound series, inhibition state, or the HIs, andthen to view the patterns of response in individual concentration– response series. Fig. 8 shows a view of the Heterogeneity Browsertool that was developed for interactive analysis and review of cel-lular distributions. All the views (Fig. 8A–H) are linked so thatselection of data in one view selects the same data in all the otherviews. Fig. 8A is essentially the same view of the data as shown inFig. 7A reduced from a 10 � 10 matrix of distributions to 6 � 8.This view shows the range of distributions in the data set andselection of one or more distributions in this view will highlightthe selection in all views and allow the user to zoom into the

component distributions as shown in Fig. 7B. Fig. 8B is similar toFig. 8A except the distributions are organized by increasing %OLon the horizontal axis and increasing normality on the vertical axis.The view in Fig. 8C enables the comparison of the distributions ofactivity over the dose–response around the IC50 value. In thisview, the dose–responses are aligned by ‘relative IC50’, with thedose closest to the IC50 (�IC50) in the center, the dose below(<IC50) or above (>IC50) that dose on the left or right respectively,and the combined distributions for dosesP 2 dilutions below(�IC50) or above (�IC50) the IC50 on the far left and right respec-tively. This view enables the comparison of heterogeneity indepen-dent of compound potency. The views in Fig. 8D–G showhistograms of the distributions of percent activity (8D), QE (8E),nNRM (8F) and %OL (8G) for the combined replicate wells acrossall the plates in the data set. These histograms show the range ofthe metrics for the data currently displayed in the browser, in thiscase the full data set, and allows selection of a range of interest.The last view in the browser is the list of compounds (8H). Thisview allows the selection of specific compound(s) for evaluationor comparison, as well as displaying the list of compounds selectedin other views.

As an example of the use of the browser, in Fig. 8D all distribu-tions with a %Active Cells in the range from 45% to 55% wereselected (highlighted in cyan). In all other views the same distribu-tions and the compound names are also highlighted in cyan.Fig. 7A and B indicates the wide range of distributions, from uni-modal to bimodal that can be seen around the 50% activation. ThenNRM histogram in Fig. 8F shows that in the selected activationrange the distributions are mostly non-normal. Fig. 8H shows oneof the compounds and the concentration which inhibited about50% of the cells, compound 10491, and Fig. 8C shows the distribu-tion of activity for that compound around its IC50 value. Thebrowser illustrated in Fig. 8 enables the user to:

� Select compounds to view the distributions as a function ofconcentration.

� Select distribution shapes to view the compounds andconcentrations with that shape.

� Select a range of activity within a well to identify thedistributions in that activity range.

� Select a range of heterogeneity using the heterogeneity indicesto identify distributions with macro-heterogeneity (high QE),non-normal distributions (high nNRM) or unusually high num-ber of outliers (high %OL).

3.7. A workflow for heterogeneity analysis

Fig. 9 summarizes the workflow established for heterogeneityanalysis that includes QC to ensure consistent distributions ofactivity over the course of a screen. Optimally, heterogeneity anal-ysis starts early in a project, during assay development (Step 1).Knowledge of the phenotype distributions induced by the controlcompounds will inform the selection of the best control compound,as well as establish reference distributions for assay validation. Tomaintain reproducibility in the distributions throughout a project,the range of assay values in the controls must be consistent fromplate-to-plate (Step 2). In HCS this can be accomplished by settingthe acquisition parameters based on the signal and background inthe control wells. Depending on the platform this may require con-trolling the intensity of the light source, the gain of the detector,the integration time and any averaging over time or pixels. Nor-mally, assay validation will include the production of replicate allpositive and all negative plates. The data from these plates shouldbe used to create reference set of distributions by combing thereplicate control wells as illustrated in Fig. 5 (Step3). These refer-ence distributions should be used to evaluate the reproducibility

24 A. Gough et al. /Methods 96 (2016) 12–26

of the distributions from well-to-well and plate-to-plate, and rela-tive to the reference development distributions, using the QC-KSstatistic. The variability in the distributions on the validation platesshould be used to establish a threshold for the QC-KS metric to beused to flag plates where the control distributions are outside theacceptable range. We used a threshold of the median +3 * K * MAD(see details in Methods). Once the reference distributions and QCthresholds are established they can be routinely applied duringthe project (Step 4) to enable the reliable calculation and compar-ison of heterogeneity from plate-to-plate and day-to-day for theduration of a large scale biology project (Steps 5–6).

1 Manuscript in preparation.

4. Conclusions

4.1. Quality control for heterogeneity analysis in HCS, high throughputmicroscopy, flow cytometry and other high throughput cytometrymethods

When performing heterogeneity analysis on large data sets indrug discovery and development or biomedical research using arange of cytometric analysis platforms, it is important to evaluateand to control the consistency of the biological distributions fromplate-to-plate and over the duration of the project. We demon-strated that good Z0-factors are not sufficient for ensuring the dis-tributions are reproducible, especially plate-to-plate. We adaptedand applied the Kolmogorov–Smirnov statistic to measure thereproducibility of the cellular heterogeneity (QC-KS). Referencedistributions should be established early in the project to trackthe performance of the assay in terms of the reproducibility ofthe distributions. Normalization of the median on each plate tothe median of the reference distribution appeared to work wellfor correcting variations in the intensity range of the distributions.There are many possible sources for this variation in range includ-ing variation in reagents, assay timing, acquisition factors, instru-ment factors, or image analysis algorithms. As with other largedata methods, such a DNA or protein microarrays, normalizationis an effective approach to correcting for many of these assay vari-ables [55]. Here we show that the KS statistic is an effective QCmetric for quality control where the distribution of the data isimportant, as well as the population average.

4.2. Performance of the heterogeneity indices in classifyingheterogeneity

The three heterogeneity indices were previously shown to beuseful in categorizing heterogeneity [14] but here they wereshown to be effective in classifying heterogeneity into discretedistributions defined by hierarchical clustering of the cellular dis-tributions, based on histogram shape. The performance of theheterogeneity indices in predicting the shape of the distributionsindicates that they successfully capture the important features ofthe distributions and that they can effectively be used to filter orarrange distributions for analysis.

4.3. Discovering patterns of heterogeneity in large data sets

The heterogeneity indices were used to create and demonstratea hierarchical approach to drilling down into the distributions in alarge data set. A summary of over 20,000 distributions in a 2Dmatrix (Fig. 7), ordered by the QE and nNRM indices, displays therange of distribution shapes in the data set. Drilling down intothe summary data provides a more detailed view of the distribu-tions with that general shape. This hierarchical approach to brows-ing the distributions allows access to a very large number ofhistograms, in this case each level represents on the order of 100

fold reduction in the number of distributions, giving access tonearly all the 20,000 distributions in this data set in 2 levels.Browsing 1 million distributions would require just 3–4 levels. Thisvisualization was incorporated into an interactive browser (Fig. 8)to simplify selecting and filtering distributions. Using this browser,with a couple of clicks the user could see: the evolution of theheterogeneity for a compound series in an SAR; the variation inthe distribution of phenotypes in a stem cell differentiation mediaseries; the rank order compounds by similarity of the cell inhibi-tion distribution; and many other analyses that relate phenotypedistributions to cell treatments.

4.4. Impact of heterogeneity analysis in HCS, high throughput imagingand other methods for high throughput cytometry

It is well established that heterogeneity is common in cellularsystems, yet the majority of cell based assays assume a normal dis-tribution and report an average value, or compensate for ‘‘outliers”by reporting the median [44]. By ignoring the cell level data anddepending on the well level average, most studies miss the oppor-tunity to understand the effects of candidate therapeutics or otherperturbagens on essential subpopulations. There is a growing bodyof evidence that analysis of heterogeneity reveals novel insightsinto cellular functions and response to perturbation, of criticalimportance in drug discovery, research and development ofdiagnostics and stem cell differentiation and maturation[2,9,11,17,56,57]. Studies have shown that analysis of heterogene-ity provides insights into network topology [58,59] and can revealinteractions between independent networks [60]. Heterogeneityis also a key component of biological adaptation [10], which canlead to reduced or even loss of efficacy. In cancer for example,one study found that in the majority of cases of relapse in acutemyeloid leukemia, the dominant clone at relapse was a minor clonefound at diagnosis that was resistant to treatment and underwentfurther expansion to form the dominant clone [24].

Heterogeneity can be inherent to a population of cells [61],induced by the microenvironment [62], or induced by compoundor reagent treatment [14,60,63,64]. The differential sensitivity todrugs by subpopulations of cells, or cells of different types, can pro-vide an indication of compound mechanism of action [2,7–9,12,13,65].1 In one study it was shown that cells treated with drugsof similar mechanism of action induce similar heterogeneity [63],while in another study, the analysis of the heterogeneity inducedby a drug led to the development of a hypothesis of interacting net-works [60]. One major advantage of phenotypic screens is the abilityto screen many targets and pathways in situ, in one screen, however,in the end, little is known about the targets of the compounds. Anal-ysis of heterogeneity in those assays, especially in comparison withcompounds of known mechanism, may provide a useful startingpoint for identifying the targets of hits in a screen, or for evaluatingwhether activity in an SAR driven by a phenotypic assay remains ontarget. Heterogeneity can also directly indicate other compoundproperties that may be useful for prioritization. For example, the firstpopulation of cells that respond to a compoundmay be a robust indi-cation of critical compound properties, such as the ability of thecompound to penetrate cells and hit the target [44].

Analysis of heterogeneity early in drug development wouldallow drug design that addresses complex cellular interactionsand differential drug sensitivity [10,13] or that adopts an approachthat uses combination therapy [14,15]. The tools described herewill facilitate the identification and analysis of heterogeneity inlarge-scale biology projects and therefore help drive the

A. Gough et al. /Methods 96 (2016) 12–26 25

application of heterogeneity analysis in basic biomedical researchand drug discovery.

Acknowledgements

This work was funded by a Pennsylvania Department of HealthCURE research Grant (SAP# 4100054875) and used the Universityof Pittsburgh Cancer Institute (UPCI) Chemical Biology Facility thatis supported in part by award P30CA047904, an NIH-National Can-cer Institute, Cancer Center Support Grant to the UPCI. The dataanalyzed in this project was generated under the NExT-CBC Agree-ment Number 29XS131-TO6 funded by the National Cancer Insti-tute to D.L. Taylor. The funders had no role in study design, datacollection and analysis, decision to publish, or preparation of themanuscript.

References

[1] K.A. Giuliano, J.R. Haskins, D.L. Taylor, Advances in high content screening fordrug discovery, Assay Drug Dev. Technol. 1 (4) (2003) 565–577.

[2] A. Gough, et al., High-content analysis with cellular and tissue systemsbiology: a bridge between cancer cell biology and tissue-based diagnostics, in:J. Mendelsohn et al. (Eds.), The Molecular Basis of Cancer, Saunders/Elsevier,Philadelphia, PA, pp. 369–392, 2015.

[3] B. Balluff et al., De novo discovery of phenotypic intratumour heterogeneityusing imaging mass spectrometry, J. Pathol. 235 (1) (2015) 3–13.

[4] C.B. Black et al., Cell-based screening using high-throughput flow cytometry,Assay Drug Dev. Technol. 9 (1) (2011) 13–20.

[5] D.G. Spiller et al., Measurement of single-cell dynamics, Nature 465 (7299)(2010) 736–745.

[6] P. Liberali, B. Snijder, L. Pelkmans, Single-cell and multivariate approaches ingenetic perturbation screens, Nat. Rev. Genet. 16 (1) (2015) 18–32.

[7] S. Huang, Non-genetic heterogeneity of cells in development: more than justnoise, Development 136 (23) (2009) 3853–3862.

[8] B. Snijder, L. Pelkmans, Origins of regulated cell-to-cell variability, Nat. Rev.Mol. Cell. Biol. 12 (2) (2011) 119–125.

[9] S.J. Altschuler, L.F. Wu, Cellular heterogeneity: when do differences make adifference?, Cell 141 (4) (2010) 559–563

[10] D.S. Tawfik, Messy biology and the origins of evolutionary innovations, Nat.Chem. Biol. 6 (10) (2010) 692–696.

[11] C.E. Meacham, S.J. Morrison, Tumour heterogeneity and cancer cell plasticity,Nature 501 (7467) (2013) 328–337.

[12] R.J. Steininger et al., On comparing heterogeneity across biomarkers,Cytometry Part A 87 (6) (2015) 558–567.

[13] D.K. Singh et al., Patterns of basal signaling heterogeneity can distinguishcellular populations with different drug sensitivities, Mol. Syst. Biol. 6 (2010)369.

[14] A.H. Gough et al., Identifying and quantifying heterogeneity in high contentanalysis: application of heterogeneity indices to drug discovery, PLoS One 9 (7)(2014) e102678.

[15] B. Zhao et al., Addressing genetic tumor heterogeneity throughcomputationally predictive combination therapy, Cancer Disc. 4 (2) (2014)166–174.

[16] P. Cahan, G.Q. Daley, Origins and implications of pluripotent stem cellvariability and heterogeneity, Nat. Rev. Mol. Cell Biol. 14 (6) (2013) 357–368.

[17] H.H. Chang et al., Transcriptome-wide noise controls lineage choice inmammalian progenitor cells, Nature 453 (7194) (2008) 544–547.

[18] L.H. Loo et al., Heterogeneity in the physiological states and pharmacologicalresponses of differentiating 3T3-L1 preadipocytes, J. Cell Biol. 187 (3) (2009)375–384.

[19] S. Huang, D.E. Ingber, A non-genetic basis for cancer progression andmetastasis: self-organizing attractors in cell regulatory networks, Breast Dis.26 (2006) 27–54.

[20] W. Buchser, et al., Assay development guidelines for image-based high contentscreening, high content analysis and high content imaging, in: G.S.Sittampalam et al., (Eds.), Assay Guidance Manual [Internet], October 1[Updated 2014 Sep 22], Eli Lilly & Company and the National Center forAdvancing Translational Sciences, Bethesda (MD), 2012 <http://www.ncbi.nlm.nih.gov/books/NBK100913/>.

[21] M. Bray, A. Carpenter, Advanced assay development guidelines for image-based high content screening and analysis, in: G.S. Sittampalam et al., (Eds.),Assay Guidance Manual [Internet], November 16 [Updated 2013 May 1], EliLilly & Company and the National Center for Advancing Translational Sciences,Bethesda (MD), 2012 <http://www.ncbi.nlm.nih.gov/books/NBK126174/>.

[22] J.-H. Zhang, T.D.Y. Chung, K.R. Oldenburg, A simple statistical parameter foruse in evaluation and validation of high throughput screening assays, J.Biomol. Screen. 4 (2) (1999) 67–73.

[23] K. Polyak, Tumor heterogeneity confounds and illuminates: a case fordarwinian tumor evolution, Nat. Med. 20 (4) (2014) 344–346.

[24] M. Kleppe, R.L. Levine, Tumor heterogeneity confounds and illuminates:assessing the implications, Nat. Med. 20 (4) (2014) 342–344.

[25] H. Koike-Yusa et al., Genome-wide recessive genetic screening in mammaliancells with a lentiviral CRISPR-guide RNA library, Nat. Biotechnol. 32 (3) (2014)267–273.

[26] G.S. Sittampalam, et al., Assay Guidance Manual [Internet], Eli Lilly & Companyand the National Center for Advancing Translational Sciences: Bethesda (MD),2004 <http://www.ncbi.nlm.nih.gov/books/NBK53196/>.

[27] X.D. Zhang, A pair of new statistical parameters for quality control in RNAinterference high-throughput screening assays, Genomics 89 (4) (2007) 552–561.

[28] I. Ravkin et al., Multiplexed high-throughput image cytometry using encodedcarriers, Proc. SPIE 5322 (2004) 52–63.

[29] M.-A. Bray et al., Workflow and metrics for image quality control in large-scalehigh-content screens, J. Biomol. Screen. 17 (2) (2012) 266–274.

[30] R.F. Stack et al., Quality assurance testing for modern optical imaging systems,Microsc. Microanal. 17 (4) (2011) 598–606.

[31] D. Grunwald et al., Calibrating excitation light fluxes for quantitative lightmicroscopy in cell biology, Nat. Protoc. 3 (11) (2008) 1809–1814.

[32] R.A. Hoffman, Standardization, calibration, and control in flow cytometry, Curr.Protoc. Cytom., 2005 (Chapter 1, Unit 1 3).

[33] L.H. Loo et al., An approach for extensibly profiling the molecular states ofcellular subpopulations, Nat. Meth. 6 (10) (2009) 759–765.

[34] S.J. Potts et al., Evaluating tumor heterogeneity in immunohistochemistry-stained breast cancer tissue, Lab. Invest. 92 (9) (2012) 1342–1357.

[35] R.F. Schwarz et al., Phylogenetic quantification of intra-tumour heterogeneity,PLoS Comput. Biol. 10 (4) (2014) e1003535.

[36] A. Heindl, S. Nawaz, Y. Yuan, Mapping spatial heterogeneity in the tumormicroenvironment: a new era for digital pathology, Lab. Invest. 95 (4) (2015)377–384.

[37] J. Gioanni et al., Two new human tumor cell lines derived from squamous cellcarcinomas of the tongue: establishment, characterization and response tocytotoxic treatment, Eur. J. Cancer Clin. Oncol. 24 (9) (1988) 1445–1455.

[38] V.L. Bauer et al., Establishment and molecular cytogenetic characterization of acell culture model of head and neck squamous cell carcinoma (HNSCC), Genes(Basel) 1 (3) (2010) 388–412.

[39] I.T. Young, Proof without prejudice. Use of the Kolmogorov–Smirnov test forthe analysis of histograms from flow systems and other sources, J. Histochem.Cytochem. 25 (7) (1977) 935–941.

[40] K.A. Giuliano, Y.-T. Chen, D.L. Taylor, High-content screening with siRNAoptimizes a cell biological approach to drug discovery: defining the role of P53activation in the cellular response to anticancer drugs, J. Biomol. Screen. 9 (7)(2004) 557–568.

[41] Z.E. Perlman et al., Multidimensional drug profiling by automated microscopy,Science 306 (5699) (2004) 1194–1198.

[42] S. Huang, Statistical issues in subpopulation analysis of high content imagingdata, J. Comput. Biol. 17 (7) (2010) 879–894.

[43] A.Y. Ng et al., A cell profiling framework for modeling drug responses from HCSimaging, J. Biomol. Screen. 15 (7) (2010) 858–868.

[44] S.A. Haney, Rapid Assessment and Visualization of Normality in High-Contentand Other Cell-Level Data and its Impact on the Interpretation of ExperimentalResults, J. Biomol. Screen. (2014).

[45] C.R. Rao, Diversity and dissimilarity coefficients – a unified approach, Theoret.Popul. Biol. 21 (1) (1982) 24–43.

[46] D. Schleuter et al., A user’s guide to functional diversity indices, Ecol. Monogr.80 (3) (2010) 15.

[47] H.W. Lilliefors, On the Kolmogorov–Smirnov test for normality with mean andvariance unknown, J. Am. Statist. Assoc. 62 (318) (1967) 399–402.

[48] L.A. Garraway, P.A. Janne, Circumventing cancer drug resistance in the era ofpersonalized medicine, Cancer Disc. 2 (3) (2012) 214–226.

[49] R Core Team, R: A Language and Environment for Statistical Computing, 2015,R Foundation for Statistical Computing, Vienna, Austria <http://www.R-project.org/>.

[50] G.J. Williams, SpringerLink (Online service), Data mining with Rattle and R theart of excavating data for knowledge discovery, in: Use R! 2011, Springer, NewYork (p. 1 online resource xx, 374 p.).

[51] S. Singh et al., Pipeline for illumination correction of images for high-throughput microscopy, J. Microsc. 256 (3) (2014) 231–236.

[52] J.M. Zwier et al., Quantitative image correction and calibration for confocalfluorescence microscopy using thin reference layers and SIPchart-basedcalibration procedures, J. Microsc. 231 (Pt 1) (2008) 59–69.

[53] K.A. Giuliano et al., Systems cell biology knowledge created from high contentscreening, ASSAY Drug Develop. Technol. 3 (5) (2005) 501–514.

[54] S.D. Rice et al., Analysis of chemotherapeutic response heterogeneity and drugclustering based on mechanism of action using an in vitro assay, Antican. Res.30 (7) (2010) 2805–2811.

[55] R.A. Irizarry et al., Exploration, normalization, and summaries of highdensity oligonucleotide array probe level data, Biostatistics 4 (2) (2003)249–264.

[56] A. Marusyk, V. Almendro, K. Polyak, Intra-tumour heterogeneity: a lookingglass for cancer?, Nat Rev. Cancer 12 (5) (2012) 323–334.

[57] L. Pelkmans, Cell biology. Using cell-to-cell variability–a new era in molecularbiology, Science 336 (6080) (2012) 425–426.

[58] S. Farkash-Amar et al., Noise genetics: inferring protein function by correlatingphenotype with protein levels and localization in individual human cells, PLoSGenet. 10 (3) (2014) e1004176.

[59] Y. Wang et al., Identifying network motifs that buffer front-to-back signalingin polarized neutrophils, Cell Rep. 3 (5) (2013) 1607–1616.

26 A. Gough et al. /Methods 96 (2016) 12–26

[60] K.E. Gascoigne, S.S. Taylor, Cancer cells display profound intra- and interlinevariation following prolonged exposure to antimitotic drugs, Cancer Cell 14 (2)(2008) 111–122.

[61] D.R. Sisan et al., Predicting rates of cell state change caused by stochasticfluctuations using a data-driven landscape model, Proc. Natl. Acad. Sci. USA109 (47) (2012) 19262–19267.

[62] F. Chen et al., New horizons in tumor microenvironment biology: challengesand opportunities, BMC Med. 13 (2015) 45.

[63] M.D. Slack et al., Characterizing heterogeneous cellular responses toperturbations, Proc. Natl. Acad. Sci. USA 105 (49) (2008) 19306–19311.

[64] N.M. Toriello et al., Integrated microfluidic bioprocessor for single-cell geneexpression analysis, Proc. Natl. Acad. Sci. USA 105 (51) (2008) 20173–20178.

[65] M.J. Gerdes et al., Emerging understanding of multiscale tumor heterogeneity,Front Oncol. 4 (2014) 366.