Appendix 1: Stream Biological Objective California Stream … · 2019-02-28 · ample, replacement of native taxa with invasive species may reduce taxonomic completeness, even if

Appendix 1: Stream Biological Objective California Stream Condition Index Documentation

Table of Contents Appendix 1: Stream Biological Objective California Stream Condition Index Documentation .............................................................................................................................. 1

California Stream Condition Index Scientific Publication............................................................ 2

California Stream Condition Index Reference Sites Scientific Publication ................................ 32

Table of Reference Sites for the California Stream Condition Index ........................................ 51

State of California Technical Memorandum on the California Stream Condition Index .......... 63

DRAFT - FOR PUBLIC REVIEW 1 of 76

California Stream Condition Index Scientific Publication


Bioassessment in complex environments: designing anindex for consistent meaning in different settings

Raphael D. Mazor1,2,5, Andrew C. Rehn2,6, Peter R. Ode2,7, Mark Engeln1,8, Kenneth C. Schiff1,9,Eric D. Stein1,10, David J. Gillett1,11, David B. Herbst3,12, and Charles P. Hawkins4,13

1Southern California Coastal Water Research Project, 3535 Harbor Boulevard, Suite 110, Costa Mesa, California 92626 USA2Aquatic Bioassessment Laboratory, California Department of Fish and Wildlife, 2005 Nimbus Road, Rancho Cordova,

California 95670 USA3Sierra Nevada Aquatic Research Laboratory, University of California, 1016 Mt. Morrison Road, Mammoth Lakes,

California 93546 USA4Department of Watershed Sciences, Western Center for Monitoring and Assessment of Freshwater Ecosystems,

and the Ecology Center, Utah State University, Logan, Utah 84322-5210 USA

Abstract: Regions with great natural environmental complexity present a challenge for attaining 2 key properties of an idealbioassessment index: 1) index scores anchored to a benchmark of biological expectation that is appropriate for the range of naturalenvironmental conditions at each assessment site, and 2) deviation from the reference benchmark measured equivalently in allsettings so that a given index score has the same ecological meaning across the entire region of interest. These properties areparticularly important for regulatory applications like biological criteria where errors or inconsistency in estimating site-specificreference condition or deviation from it can lead to management actions with significant financial and resource-protection con-sequences. We developed an index based on benthic macroinvertebrates for California, USA, a region with great environmentalheterogeneity. We evaluated index performance (accuracy, precision, responsiveness, and sensitivity) throughout the region todetermine if scores provide equivalent ecological meaning in different settings. Consistent performance across environmentalsettings was improved by 3 key elements of our approach: 1) use of a large reference data set that represents virtually all of the rangeof natural gradients in the region, 2) development of predictive models that account for the effects of natural gradients on biologicalassemblages, and 3) combination of 2 indices of biological condition (a ratio of observed-to-expected taxa [O/E] and a predictivemultimetric index [pMMI]) into a single index (the California Stream Condition Index [CSCI]). Evaluation of index performanceacross broad environmental gradients provides essential information when assessing the suitability of the index for regulatory ap-plications in diverse regions.Key words: bioassessment, predictive modelling, predictive multimetric index, reference condition

A major challenge for conducting bioassessment in envi-ronmentally diverse regions is ensuring that an index pro-vides consistent meaning in different environmental set-tings. A given score from a robust index should indicate thesame biological condition, regardless of location or streamtype. However, the performance (e.g., accuracy, precision,responsiveness, and sensitivity) of an index may vary in dif-ferent settings, complicating its interpretation (Hughes et al.1986, Yuan et al. 2008, Pont et al. 2009). Effective bioassess-ment indices should account for naturally occurring varia-tion in aquatic assemblages so that deviations from refer-ence conditions resulting from anthropogenic disturbanceare minimally confounded by natural variability (Hughes et al.1986, Reynoldson et al. 1997). When bioassessment indi-ces are used in regulatory applications, such as measuring

compliance with biocriteria (Davis and Simon 1995, Coun-cil of European Communities 2000, USEPA 2002, Yoderand Barbour 2009), variable meaning of an index score maylead to poor stream management, particularly if the envi-ronmental factors affecting index performance are unrec-ognized. Those who develop bioassessment indices or thepolicies that rely on them should evaluate index perfor-mance carefully across the different environmental gradientswhere an index will be applied.

A reference data set that represents the full range ofenvironmental gradients where an index will be used iskey for index development in environmentally diverse re-gions. In addition, reference criteria should be consistentlydefined so that benchmarks of biological condition areequivalent across environmental settings. Indices based on

E-mail addresses: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

DOI: 10.1086/684130. Received 21 September 2014; Accepted 28 April 2015; Published online 22 October 2015.Freshwater Science. 2016. 35(1):249–271. © 2016 by The Society for Freshwater Science. 249


benthic macroinvertebrates (BMI) for use in California weredeveloped with reference data sets that used different crite-ria in different regions (e.g., Hawkins et al. 2000, Herbst andSilldorff 2009, Rehn 2009). For example, several referencesites used to calibrate an index for the highly urbanizedSouth Coast region had more nonnatural land use than anyreference site used to develop an index for the rural NorthCoast region (Ode et al. 2005, Rehn et al. 2005). Further-more, lower-elevation settings were poorly represented inthese reference data sets. In preparation for establishingstatewide biocriteria, regulatory agencies and regulated par-ties desired a new index based on a larger, more consistentlydefined reference data set that better represented all envi-ronmental settings. Considerable effort was invested to ex-pand the statewide pool of reference sites to support de-velopment of a new index (Ode et al. 2016). The diversityof stream environments represented in the reference poolnecessitated scoring tools that could handle high levels ofcomplexity.

Predictive modeling of the reference condition is an in-creasingly common way to obtain site-specific expectationsfor diverse environmental settings (Hawkins et al. 2010b).Predictive models can be used to set biological expecta-tions at test sites based on the relationship between bio-logical assemblages and environmental factors at referencesites. Thus far, predictive modeling has been applied al-most exclusively to multivariate indices focused on taxo-nomic completeness of a sample, such as measured by theratio of observed-to-expected taxa (O/E) (Moss et al. 1987,Hawkins et al. 2000, Wright et al. 2000), or location of sitesin ordination space (e.g., BEnthicAssessment of SedimenT[BEAST]; Reynoldson et al. 1995). Applications of predic-tive models to multimetric indices (i.e., predictive multi-metric indices [pMMIs]) are relatively new (e.g., Cao et al.2007, Pont et al. 2009, Vander Laan and Hawkins 2014).MMIs include information on the life-history traits ob-served within an assemblage (e.g., trophic groups, habitatpreferences, pollution tolerances), so they may provide use-ful information about biological condition that is not in-corporated in an index based only on loss of taxa (Gerritsen1995). Predictive models that set site-specific expectationsfor biological metric values may improve the accuracy, pre-cision, and sensitivity of MMIs when applied across diverseenvironmental settings (e.g., Hawkins et al. 2010a).

A combination of multiple indices (specifically, a pMMIand an O/E index) into a single index might provide moreconsistent measures of biological condition than just oneindex by itself. Variation in performance of an index wouldbe damped by averaging it with a 2nd index, and poor per-formance in particular settings might be improved. For ex-ample, an O/E index may be particularly sensitive in moun-tain streams that are expected to be taxonomically rich,whereas a pMMI might be more sensitive in lowland areas,where stressed sites may be well represented in calibrationdata. Moreover, pMMIs and O/E indices characterize as-

semblage data in fundamentally different ways. Thus, theyprovide complementary measures of stream ecological con-dition and may contribute different types of diagnostic infor-mation. Taxonomic completeness, as measured by an O/Eindex, and ecological structure, as measured by a pMMI,are both important aspects of stream communities, and cer-tain stressors may affect these aspects differently. For ex-ample, replacement of native taxa with invasive species mayreduce taxonomic completeness, even if the invaders haveecological attributes similar to those of the taxa they dis-placed (Collier 2009). Therefore, measuring both taxo-nomic completeness and ecological structure may providea more complete picture of stream health.

Our goal was to construct a scoring tool for perennialwadeable streams that provides consistent interpretationsof biological condition across environmental settings inCalifornia, USA. Our approach was to design the tool tomaximize the consistency of performance across settings,as indicated by evaluations of accuracy, precision, respon-siveness, and sensitivity. We first constructed predictivemodels for both a taxon loss index (O/E) and a pMMI. Sec-ond, we compared the accuracy, precision, responsiveness,and sensitivity of the O/E, pMMI, and combined O/E +pMMI index across a variety of environmental settings.Our primary motivation was to develop biological indicesto support regulatory applications in the State of California.However, our broader goal was to produce a robust assess-ment tool that would support a wide variety of bioassess-ment applications, such as prioritization of restoration proj-ects or identification of areas with high conservation value.

METHODSStudy region

California contains continental-scale environmental di-versity within 424,000 km2 that encompass some of themost extreme gradients in elevation and climate found inthe USA. It has temperate rainforests in the North Coast,deserts in the east, and chaparral, oak woodlands, andgrasslands with a Mediterranean climate in coastal regions(Omernik 1987). Large areas of the state are publicly owned,but vast regions have been converted to agricultural (e.g.,the Central Valley) or urban (e.g., the South Coast and theSan Francisco Bay Area) land uses (Sleeter et al. 2011). For-estry, grazing, mining, other resource extraction activities,and intensive recreation occur throughout rural regions ofthe state, and the fringes of urban areas are undergoingincreasing development. For convenience, we divided thestate into 6 regions and 10 subregions based on ecoregional(Omernik 1987) and hydrologic boundaries (CaliforniaStateWater Resources Control Board 2013) (Fig. 1).

Compilation of dataWe compiled data from >20 federal, state, and regional

monitoring programs. Altogether, we aggregated data from

250 | Bioassessment in complex environments R. D. Mazor et al.


4457 samples collected from 2352 unique sites between1999 and 2010 into a single database. We excluded BMIsamples with insufficient numbers of organisms or taxo-nomic resolution (described below) from analyses. Wetreated observations at sites in close proximity to each other(within 300 m) as repeat samples from a single site. For siteswith multiple samples meeting minimum requirements, werandomly selected a single sample for use in all analysesdescribed below, and we withheld repeat samples from allanalyses, except where indicated below. We used 1318 sitessampled during probabilistic surveys (e.g., Peck et al. 2006)to estimate the ambient condition of streams (describedbelow).

Biological dataFifty-five percent of the BMI samples were collected fol-

lowing a reach-wide protocol (Peck et al. 2006), and theother samples were collected with targeted riffle protocols,which produce comparable data (Gerth and Herlihy 2006,Herbst and Silldorff 2006, Rehn et al. 2007). For most sam-ples, taxa were identified to genus, but this level of effortand the total number of organisms/sample varied among

samples, necessitating standardization of BMI data. Weused different data standardization approaches for thepMMI and the O/E. For the pMMI, we aggregated iden-tifications to ‘Level 1’ standard taxonomic effort (most in-sect taxa identified to genus, Chironomidae identified tofamily) as defined by the Southwest Association of Fresh-water Invertebrate Taxonomists (SAFIT; Richards and Rog-ers 2011) and used computer subsampling to generate500-count subsamples. We excluded samples with <450 in-dividuals (i.e., not within 10% of target). For the O/E index,we used operational taxonomic units (OTUs) similar toSAFIT Level 1 except that we aggregated Chironomidae tosubfamily. We excluded ambiguous taxa (i.e., those identi-fied to a higher level than specified by the OTU). We alsoexcluded samples with >50% ambiguous individuals fromO/E development, no matter howmany unambiguous indi-viduals remained. We used computer subsampling to gener-ate 400-count subsamples, and we excluded samples with<360 individuals. A smaller subsample size was used forthe O/E index than for the pMMI because exclusion ofambiguous taxa often reduced sample size to <500 indi-viduals. A final data set of 3518 samples from 1985 sitesmet all requirements and was used for development andevaluation of both the O/E and pMMI indices.

Environmental dataWe collected environmental data frommultiple sources

to characterize natural and anthropogenic factors knownto affect benthic communities, such as climate, elevation,geology, land cover, road density, hydrologic alteration,and mining (Tables 1, 2). We used geographic informationsystem (GIS) variables that characterized natural, unalter-able environmental factors (e.g., topography, geology, cli-mate) as predictors for O/E and pMMI models and var-iables related to human activity (e.g., land use) to classifysites as reference and to evaluate responsiveness of O/Eand pMMI indices to human activity gradients. We calcu-lated most variables related to human activity at 3 spatialscales (within the entire upstream drainage area [water-shed], within the contributing area 5 km upstream of a site[5 km], and within the contributing area 1 km upstreamof a site [1 km]) so that we could screen sites for localand catchment-scale impacts. We created polygons defin-ing these spatial analysis units using ArcGIS tools (version9.0; Environmental Systems Research Institute, Redlands,California).

Classification of sites along a human activity gradientWe were unable to measure stress directly with this data

set, so instead, we used a human activity gradient underthe assumption that it was correlated with stress (Yatesand Bailey 2010). We divided sites into 3 sets for develop-ment and evaluation of indices: reference (i.e., low activity),moderate-, and high-activity sites. We defined reference

Figure 1. Regions and subregions of California. Thick graylines indicate regional boundaries, and thin white lines indicatesubregional boundaries. NC = North Coast, CHco = CoastalChaparral, Chin = Interior Chaparral, SCm = South Coastmountains, SCx = South Coast xeric, CV = Central Valley,SNws = Sierra Nevada-western slope, SNcl = Sierra Nevada-central Lahontan, DMmo: Desert/Modoc-Modoc plateau,DMde =Desert/Modoc-deserts.

Volume 35 March 2016 | 251


Table

1.Natural

gradientsandtheirim

portance

(Gini=meandecrease

inGiniindex),M

SE=%

increase

inmeansquarederror)forrand

om-forestmod

elsfortheob

served

(O)/expected

(E)taxa

indexandeach

metricused

inthepredictive

multimetricindex(pMMI).P

redictorsthat

wereevaluatedbu

tno

tselected

foranymod

elinclud

e%

sedimentary

geology,nitrogenou

sgeology,soilhydraulic

cond

uctivity,soilperm

eability,S-bearinggeology,calcite-bearinggeology,andmagnesium

oxide-bearinggeology.

Sources:A

=NationalElevation

Dataset

(http://ned.usgs.gov/),B

=PRISM

clim

atemapping

system

(http://www.prism

.oregonstate.edu

),C=generalized

geology,mineralogy,

andclim

atedata

derivedforacond

uctivity

prediction

mod

el(O

lson

andHaw

kins

2012).Dashesindicate

that

thepredictors

wereno

tused

tomod

elthemetric.

Variable

Description

O/E

Taxon

omic

richness

MSE

%intolerant

MSE

#Sh

redd

ertaxa

MSE

Clin

ger%

taxa

MSE

Coleoptera%

taxa

MSE

EPT%

taxa

MSE

Data

source

Gini

MSE

Location

New

lat

Latitud

e90.5

0.09

18.8

0.0063

1.26

0.0054

0.00079

0.0027

New

long

Lon

gitude

––

25.3

0.0058

0.99

0.0030

–0.0024

SITE_E

LEV

Elevation

89.5

0.11

11.8

––

–0.00231

–A

Catchmentmorph

ology

LogW

SALog

watershed

area

86.6

0.06

–0.0020

1.23

––

–A

ELEV_R

ANGE

Elevation

rang

e–

–2.4

––

0.0026

––

A

Clim

ate

PPT

10-y

(2000–

2009)average

precipitationat

thesamplingpo

int

74.8

0.07

8.4

0.0063

0.92

––

0.0016

B

TEMP

10-y

(2000–

2009)averageair

temperature

atthesamplingpo

int

81.9

0.09

9.3

0.0052

–0.0023

–0.0019

B

SumAve_P

MeanJune

toSeptem

ber1971–2000

mon

thlyprecipitation,

averaged

across

thecatchm

ent

––

5.5

––

––

0.0033

B

Geology

BDH_A

VE

Average

bulk

soildensity

––

5.7

––

0.0021

––

C

KFC

T_A

VE

Average

soilerod

ibility

factor

(k)

––

6.2

––

0.0027

–0.0025

C

Log_P

_MEAN

Log

%Pgeology

––

3.7

––

––

–C


sites as ‘minimally disturbed’ sensu Stoddard et al. (2006)and selected them by applying screening criteria based pri-marily on landuse variables calculated at multiple spatialscales (i.e., 1 km, 5 km, watershed; Table 2). We calculatedsome screening criteria at only 1 spatial scale (e.g., in-streamgravel mine density at the 5-km scale and W1_HALL, aproximity-weighted index of human activity based on fieldobservations made within 50 m of a sampling reach; Kauf-mann et al. 1999). We excluded sites thought to be affectedby grazing or recreation from the reference data set, evenif they passed all reference criteria. Identification of high-activity sites was necessary for pMMI calibration (describedbelow) and for performance evaluation of both pMMI andO/E. We defined high-activity sites as meeting any of thefollowing criteria: ≥50% developed land (i.e., % agricul-tural + % urban) at all spatial scales, ≥5 km/km2 road den-sity, or W1_HALL ≥ 5. We defined sites not identified aseither reference or high-activity as moderate-activity sites.We further divided sites in each set into calibration (80%)and validation (20%) subsets and stratified assignment tocalibration and validation sets by subregion to ensure repre-sentation of all environmental settings in both sets (Fig. 1).

Only 1 reference site was found in the Central Valley, so thatregion was combined with the Interior Chaparral (whoseboundary was within 500 m of the site) for stratificationpurposes.

Development of the O/E indexDevelopment of an O/E index or pMMI follows the

same basic steps: biological characterization, modeling ofreference expectations from environmental factors, selec-tion of metrics or taxa, and combining of metrics or taxainto an index. pMMI development has an additional inter-mediate step to set biological expectations for sites withhigh levels of activity (Fig. 2). Taxonomic completeness,as measured by O/E, quantifies degraded biological condi-tion as loss of expected native taxa (Hawkins 2006). E rep-resents the number of taxa expected in a specific sample,based on its environmental setting, and O represents thenumber of those expected taxa that were actually observed.We developed models to calculate the O/E index follow-ing the general approach of Moss et al. (1987). First, we de-fined groups of reference calibration sites based on their

Table 2. Stressor and human-activity gradients used to identify reference sites and evaluate index performance. Sites that did notexceed the listed thresholds were used as reference sites. Sources A = National Landcover Data Set (http://www.epa.gov/mrlc/nlcd-2006.html), B = custom roads layer, C = National Hydrography Dataset Plus (http://www.horizon-systems.com/nhdplus), D = Na-tional Inventory of Dams (http://geo.usace.army.mil), E = Mineral Resource Data System (http://tin.er.usgs.gov/mrds), F = predictedspecific conductance (Olson and Hawkins 2012), G = field-measured variables. WS = watershed, 5 km = watershed clipped to a 5-kmbuffer of the sampling point, 1 km = watershed clipped to a 1-km buffer of the sampling point, W1_HALL = proximity-weightedhuman activity index (Kaufmann et al. 1999), Code 21 = landuse category that corresponds to managed vegetation, such as roadsides,lawns, cemeteries, and golf courses. * indicates variable used in the random-forest evaluation of index responsiveness.

Variable Scale Threshold Unit Data source

* % agricultural 1 km, 5 km, WS <3 % A

* % urban 1 km, 5 km, WS <3 % A

* % agricultural + % urban 1 km, 5 km, WS <5 % A

* % Code 21 1 km and 5 km <7 % A

* WS <10 % A

* Road density 1 km, 5 km, WS <2 km/km2 B

* Road crossings 1 km <5 crossings B, C

* 5 km <10 crossings B, C

* WS <50 crossings B, C

* Dam distance WS <10 km D

* % canals and pipelines WS <10 % C

* Instream gravel mines 5 km <0.1 mines/km C, E

* Producer mines 5 km 0 mines E

Specific conductance Site 99/1a prediction interval F

W1_HALL Reach <1.5 NA G

% sands and fines Reach % G

Slope Reach % G

a The 99th and 1st percentiles of predictions were used to generate site-specific thresholds for specific conductance. The model underpredicted athigher levels of specific conductance (data not shown), so a threshold of 2000 μS/cm was used as an upper bound if the prediction interval included1000 μS/cm.



Figure 2. Summary of steps in developing the predictive multimetric index (pMMI) and observed (O)/expected (E) taxa index.Pc = probability of observing a taxon at a site, CSCI = California State Condition Index.


taxonomic similarity. Second, we developed a random-forestmodel (Cutler et al. 2007) to predict group membershipbased on naturally occurring environmental factors mini-mally affected by human activities. We used this model topredict cluster membership for test sites based on theirnatural environmental setting. The probability of observ-ing a taxon at a test site (i.e., the capture probability) wascalculated as the cluster-membership-probability-weightedfrequencies of occurrence summed across clusters:

Pc j ¼ ∑ki¼1ðGiFiÞ; (Eq. 1)

where Pcj is the probability of observing taxon j at a site,Gi is the probability that a site is a member of group i, Fi isthe relative frequency of the taxon in group i, and k is thenumber of groups used in modeling. The sum of the cap-ture probabilities is the expected number of taxa (E) in asample from a site:

E ¼ ∑mj¼1Pcj; (Eq. 2)

where m is the number of taxa observed across all refer-ence sites. We used Pc values ≥ 0.5 when calculating O/Ebecause excluding locally rare taxa generally improves pre-cision of O/E indices (Hawkins et al. 2000, Van Sickle et al.2007). This model was used to predict E at reference andnonreference sites based on their natural environmentalsetting.

We used presence/absence-transformed BMI data fromreference calibration sites to identify biologically similargroups of sites. We excluded taxa occurring in <5% ofreference calibration samples from the cluster analysis be-cause inclusion of regionally rare taxa can obscure patternsassociated with more common taxa (e.g., Gauch 1982,Clarke and Green 1988, Ostermiller and Hawkins 2004).We created a dendrogram with Sørensen’s distance mea-sure and flexible β (β = −0.25) unweighted pair groupmethod with arithmetic mean (UPGMA) as the linkagealgorithm in R (version 2.15.2; R Project for StatisticalComputing, Vienna, Austria) with the cluster package(Maechler et al. 2012) and scripts written by J. Van Sickle(US Environmental Protection Agency, personal commu-nication). We identified groups containing ≥10 sites andsubtended by relatively long branches (to maximize differ-ences in taxonomic composition among clusters) by visualinspection of the dendrogram. We retained rare taxa thatwere excluded from the cluster analysis for other steps inindex development.

We constructed a 10,000-tree random-forest modelwith the randomForest package in R (Liaw and Wiener2002) to predict cluster membership for new test sites.We excluded predictors that were moderately to stronglycorrelated with one another (|Pearson’s r| ≥ 0.7). When

we observed correlation among predictors, we selectedthe predictor that was simplest to calculate (e.g., calcu-lated from point data rather than delineated catchments)as a candidate predictor. We used an initial random-forestmodel based on all possible candidate predictors to iden-tify those predictors that were most important for pre-dicting new test sites into biological groups as measuredby the Gini index (Liaw and Wiener 2002). We evaluateddifferent combinations of the most important variables toidentify a final, parsimonious model that minimized thestandard deviation (SD) of reference site O/E scores at cal-ibration reference sites with the fewest predictors.

We evaluated O/E index performance in 2 ways. First,we compared index precision with the lowest and high-est precision possible given the sampling and sample-processing methods used (Van Sickle et al. 2005). SD ofO/E index scores produced by a null model (i.e., all sitesare in a single group, and capture probabilities for eachtaxon are the same for all sites) estimates the lowest pre-cision possible for an O/E index. SD of O/E values basedon estimates of variability among replicate samples (SDRS)estimates the highest attainable precision possible for theindex. Second, we evaluated the index for consistency byregressing O against E for reference sites. Slopes close to 1and intercepts close to 0 indicate better performance.

Development of the pMMIWe followed the approach of Vander Laan and Haw-

kins (2014) to develop a pMMI. In contrast to traditionalMMIs, which typically attempt to control for the effectsof natural factors on biological metrics via landscape clas-sifications or stream typologies, a pMMI accounts for theseeffects by predicting the expected (i.e., naturally occurring)metric values at reference sites given their specific environ-mental setting. A pMMI uses the difference between theobserved and predicted metric values when scoring biolog-ical condition, whereas a traditional MMI uses the rawmetric for scoring. Traditional approaches to MMI devel-opment may reduce the effects of natural gradients on met-ric values through classification (e.g., regionalization or ty-pological approaches; see Ode et al. 2005 for a Californiaexample), but they seldom produce site-specific expecta-tions for different environmental settings (Hawkins et al.2010b).

We developed the pMMI in 5 steps (Fig. 2): 1) metriccalculation, 2) prediction of metric values at referencesites, 3) metric scoring, 4) metric selection, and 5) assem-bly of the pMMI. Apart from step 2, the process for de-veloping a pMMI is comparable to that used for a tradi-tional MMI (e.g., Stoddard et al. 2008). We developed anull MMI based on raw values of the selected metrics toallow us to estimate how much predictive modeling im-proved pMMI performance. The process was intended toproduce a pMMI that was unbiased, precise, responsive,



and able to characterize a large breadth of ecological at-tributes of the BMI assemblage.

Metric calculation We calculated biological metrics thatcharacterized the ecological structure of BMI assemblagesfor each sample in the data set. We used custom scripts inR and the vegan package (Oksanen et al. 2013) to calcu-late a suite of 48 widely used bioassessment metrics, cho-sen because they quantify important ecological attributes,such as taxonomic richness or trophic diversity (a subsetof which is presented in Table 3). Many of these metricsare widely used in other bioassessment indices (e.g., Royer

et al. 2001, Stribling et al. 2008). Different formulationsof metrics based on taxonomic composition (e.g., Dipterametrics) or traits (e.g., predator metrics) were assigned tothematic metric groups representing different ecologicalattributes (Table 3). These thematic groups were used tohelp ensure that the metrics included in the pMMI wereecologically diverse.

Prediction of metric values at reference sites We usedrandom-forest models to predict values for all 48 metricsat reference calibration sites based on the same GIS-derived candidate variables that were used for O/E devel-

Table 3. Metrics evaluated for inclusion in the predictive multimetric index (pMMI). Only metrics that met all evaluation criteria areshown. EPT = Ephemeroptera, Plecoptera, and Trichoptera; Resp = direction of response; I = metric increases with human-activitygradients; D = metric decreases with human-activity gradients; Var Exp = % variance explained by the random-forest model; r 2

(cal) = squared Pearson correlation coefficient between predicted and observed values at reference calibration sites; r2 (val) = squaredPearson correlation coefficient between predicted and observed values at reference validation sites; t (null) = t-statistic for thecomparison of the raw metric between the reference and high-activity samples within the calibration data set; t (mod) = t-statistic forthe comparison of the residual metric between the reference and high-activity samples within the calibration data set; F = F-statisticfor an analysis of variance of metric residual values from reference calibration sites among regions shown in Fig. 1; S :N = signal-to-noise ratio; Freq = frequency of the metric among the best-performing combinations of metrics. Tolerance, functional feeding group,and habit data were from CAMLnet (2003). * indicates metric selected for inclusion in the pMMI.

Metric Resp Var Exp r2 (cal) r2 (val) t (null) t (mod) F S :N Freq

Taxonomic diversity

*Taxonomic richness D 0.27 0.27 0.15 21.6 23.7 1.0 6.7 0.83

Functional feeding group

Scrapers

No. Scraper taxa D 0.40 0.40 0.29 15.3 19.1 1.2 7.6 0.17

Shredders

% Shredder taxa D 0.27 0.27 0.46 17.6 10.6 1.0 4.1 0.33

* No. Shredder taxa D 0.39 0.39 0.35 19.2 15.2 1.9 5.4 0.50

Habit

Clingers

* % Clinger taxa D 0.34 0.34 0.42 21.7 14.6 0.2 4.8 1.00

No. Clinger taxa D 0.39 0.40 0.32 26.0 25.3 0.5 11.1 0

Taxonomy

Coleoptera

* % Coleoptera taxa D 0.30 0.31 0.22 10.3 15.8 1.0 5.0 0.83

No. Coleoptera taxa D 0.34 0.34 0.29 13.6 20.9 0.6 6.2 0.17

EPT

* % EPT taxa D 0.31 0.32 0.46 30.0 23.1 0.4 6.0 0.67

No. EPT taxa D 0.40 0.40 0.31 27.8 25.3 1.4 10.0 0.17

Tolerance

* % Intolerant taxa D 0.23 0.23 0.15 21.7 15.6 0.5 5.1 0.67

% Intolerant taxa D 0.51 0.51 0.58 32.7 25.3 1.5 6.9 0.17

No. Intolerant taxa D 0.52 0.52 0.53 28.4 21.8 1.5 9.6 0

Tolerance value I 0.22 0.25 0.20 −21.5 −17.0 0.4 5.0 0

% Tolerant taxa I 0.22 0.24 0.38 −26.1 −22.3 1.4 4.9 0.17



opment (Table 1). Manual refinement was impracticalbecause of the large number of models that were devel-oped, so we used an automated approach (recursive fea-ture elimination [RFE]) to select the simplest model (themodel with the fewest predictors) whose root meansquare error (RMSE) was ≤2% greater than the RMSE ofthe optimal model (the model with the lowest RMSE).We considered only models with ≤10 predictors. Limit-ing the complexity of the model typically reduces over-fitting and improves model validation (Strobl et al. 2007).We implemented RFE with the caret package in R usingthe default settings for random-forest models (Kuhn et al.2012). We used the randomForest package (Liaw andWiener 2002) to create a final 500-tree model for eachmetric based on the predictors used in the model selectedby RFE. We then used these models to predict metric val-ues for all sites. We used out-of-bag predictions for thereference calibration set (an out-of-bag prediction is basedonly on the subset of trees in which a calibration site wasexcluded during model training). To evaluate how welleach model predicted metric values, we regressed raw ob-served values against predicted values for reference sites.Slopes close to 1 and intercepts close to 0 indicate bettermodel performance. If the pseudo-R2 of the model (calcu-lated as 1 – mean squared error [MSE]/variance) was >0.2,we used the model to adjust metric values (i.e., observed –predicted), otherwise we used the observed metric values.Hereafter, ‘metric’ is used to refer to both raw and adjustedmetric values.

Metric scoring Scoring is required for MMIs becausemetrics have different scales and different responses tostress (Blocksom 2003). Scoring transforms metrics to astandard scale ranging from 0 (i.e., most stressed) to 1(i.e., identical to reference sites). We scored metrics fol-lowing Cao et al. (2007). We scored metrics that de-crease with human activity as

ðObserved−MinÞ=ðMax−MinÞ; (Eq. 3)

where Min is the 5th percentile of high-activity calibrationsites and Max is the 95th percentile of reference calibra-tion sites. We scored metrics that increase with humanactivity as

ðObserved−MaxÞ=ðMin−MaxÞ; (Eq. 4)

where Min is the 5th percentile of reference calibrationsites, and Max is the 95th percentile of high-activity sites.We trimmed scores outside the range of 0 to 1 to 0 or 1.We used 5th and 95th percentiles instead of minimum ormaximum values because they are more robust estimatesof metric range than minima and maxima (Blocksom 2003,Stoddard et al. 2008).

Metric selection We selected metrics in a 2-phase pro-cess: 1) based on their individual performance, and 2) basedon their frequency in high-performing prototype pMMIs.Evaluating the performance of many prototype pMMIs avoidsselection of metrics with spuriously good performance andis preferable to selecting metrics or pMMIs based on per-formance evaluations conducted 1 metric at a time (Hugheset al. 1998, Roth et al. 1998, Angradi et al. 2009, Van Sickle2010). Initial elimination of metrics based on their indi-vidual performance alleviates the computational challengeof evaluating large numbers of prototype pMMIs.

We used several performance criteria to eliminate met-rics from further analysis. We assessed responsiveness tohuman activity by computing t-statistics based on com-parisons of mean metric values at reference sites and siteswith high levels of activity and eliminated metrics with at-statistic < 10. We assessed bias by determining whethermetric values varied among predefined geographic regions(Fig. 1). We considered metrics with an F-statistic > 2derived from analysis of variance (ANOVA) by geographicregion to have high regional bias and eliminated them.Other screening criteria were modified from Stoddard et al.(2008). We excluded metrics with >⅔ zero values acrosssamples and richness metrics with range < 5. We also elimi-nated metrics with a signal-to-noise ratio (ratio of between-site to within-site variance estimated from data collectedat sites withmultiple samples) < 3.

We further screened metrics by evaluating the perfor-mance of all possible combinations as prototype pMMIsand selecting metrics that were frequent among proto-types with the best performance. First, we assembled allnonredundant combinations of metrics that met mini-mum performance criteria into prototype pMMIs. Lim-iting the redundancy of metrics increases the number ofthematic groups included in prototypes, thereby improv-ing the ecological breadth of the pMMI. Redundant com-binations of metrics included those with multiple metricsfrom a single metric group (e.g., tolerance metrics; Table 3)or correlated metrics (|Pearson’s r ≥ |0.7|). Prototype pMMIsranged in size from a minimum of 5 to a maximum of10 metrics, a range that is typical of MMIs used for streambioassessment (e.g., Royer et al. 2001, Fore and Grafe 2002,Ode et al. 2005, Stoddard et al. 2008, Van Sickle 2010). Wecalculated scores for these prototype pMMIs by averagingmetric scores and rescaling by the mean of reference cali-bration sites, which allows comparisons among prototypepMMIs.

Subsequently, we ranked prototype pMMIs to identifythose with the best responsiveness and precision. Biasedmetrics already had been eliminated from consideration,and none of the prototypes exhibited geographic bias (re-sults not shown), so we did not use accuracy to rank proto-type pMMIs. We estimated responsiveness as the t-statisticbased on mean scores at reference and high-activity cali-



bration sites and precision as the SD of scores from refer-ence calibration sites. We identified the best subset of pro-totype pMMIs as those appearing in the top quartile forboth criteria. Therefore, prototype pMMIs in the best sub-set possessed several desirable characteristics: ecologicalbreadth, high responsiveness, and high precision.

We assembled the final pMMI by selecting metrics inorder of their frequency in the best subset of prototypepMMIs. We added metrics in order of decreasing fre-quency and avoided adding metrics from the same the-matic group or correlated (Pearson’s r ≥ 0.7) metrics. Weexcluded metrics that appeared in <⅓ of the best proto-type pMMIs from the final pMMI.

Aggregation of the pMMI We calculated scores for thefinal pMMI by averaging metric scores and rescaling bythe mean of reference calibration sites (as for prototypepMMIs). Rescaling of pMMI scores ensures that pMMIand O/E are expressed in similar scales (i.e., as a ratio ofobserved to reference expectations) and improves com-parability of the 2 indices.

We calculated scores for a combined index (the Califor-nia Stream Condition Index [CSCI]) by averaging pMMIand O/E scores. We calculated a null combined index byaveraging null MMI and null O/E scores.

Performance evaluation Evaluation of index performancefocused on accuracy, precision, responsiveness, and sensi-tivity (Table 4). We compared the performance of each in-dex to that of its null counterpart. Many of our approachesto measuring performance also have been used widely inindex development (e.g., Hawkins et al. 2000, 2010a, Clarke

et al. 2003, Ode et al. 2008, Cao and Hawkins 2011). Wescored all indices on similar scales (i.e., a minimum of 0,with a reference expectation of 1), so no adjustments wererequired to make comparisons (Herbst and Silldorff 2006,Cao and Hawkins 2011). We conducted all performanceevaluations separately on calibration and validation datasets.

We regarded indices as accurate if scores at referencesites were not influenced by environmental setting or timeof sampling. Precise indices were those with low variabil-ity among reference sites and among samples from re-peated visits within sites. Responsive indices were thosethat showed large decreases in response to human activ-ity. Sensitive indices were those that frequently found non-reference sites to be below an impairment threshold (e.g.,10th percentile of scores at reference sites).

Performance of the indices along a gradient of expectednumbers of common taxa (E) The performance of anideal index should not vary with E. For example, indexaccuracy should not be influenced by the expected rich-ness of a site. We evaluated the accuracy, precision, andsensitivity of the indices against E by grouping sites intobins that ranged in the number of expected taxa (bin size =4 taxa). We chose this bin size because it was the smallestnumber that allowed analysis of a wide range of values of Ewith large numbers of sites in each bin (i.e., ≥37 sites foraccuracy and precision estimates and 15 sites for sensitiv-ity estimates). We measured accuracy as the proportion ofreference sites in each bin with scores ≥10th percentile ofreference calibration sites. We measured precision as theSD of reference sites in each bin and sensitivity as the

Table 4. Summary of performance evaluations. SD = standard deviation.

Aspect Description Indication of good performance

Accuracy and bias Scores are minimally influencedby natural gradients

• Approximately 90% of validation reference sites have scores>10th percentile of calibration reference sites

• Landscape-scale natural gradients explain little variability inscores at reference sites, as indicated by a low pseudo-R2

for a 500-tree random-forest model• No visual relationship evident in plots of scores at reference

sites against field measurements of natural gradientsPrecision Scores are similar when measured

under similar settings• Low SD of scores among reference sites (1 sample/site)• Low pooled SD of scores among samples at reference sites

with multiple sampling eventsResponsiveness Scores change in response to

human activity gradients• Large t-statistic in comparison of mean scores at reference

and high-activity sites• Landscape-scale human activity gradients explain variability

in scores, as indicated by a high pseudo-R2 for a 500-treerandom-forest model

Sensitivity Scores indicate poor conditionat high-activity sites

• High percentage of high-activity sites have scores <10th

percentile of calibration reference sites



proportion of high-activity sites within each bin withscores <10th percentile of reference calibration sites. Werepeated all analyses with scores from indices based onnull models.

Unlike accuracy and precision, the sensitivity of an idealindex (if measured as described above) may vary with E,but only to the extent that stress levels vary with E. How-ever, how stress levels truly varied with E is unknown be-cause human activity gradients were used to approximatestressor gradients, and direct, quantitative measures of stresslevels are not possible. Even direct measures of water chem-istry or habitat-related variables are at best incomplete es-timates of the stress experienced by stream communities,and these data were not available for many sites in ourdata set. Therefore, we supplemented analyses of sensitiv-ity against E by evaluating the difference in sensitivity be-tween the pMMI and O/E against E. We calculated thedifference as the adjusted Wald interval for a difference inproportions with matched pairs (Agresti and Min 2005)with the PropCIs package in R (Scherer 2013). The differ-ence between the indices should be constant if E has noinfluence on sensitivity, or if E affects both indices in thesame way. In the absence of direct measures of stress levels,these analyses provide a good measure of the influence of Eon index sensitivity.

Establishment of biological condition classes,and application to a statewide assessment

We created 4 condition classes based on the distribu-tion of scores at reference calibration sites, with a rec-ommended interpretation for each condition class: likelyto be intact (>30th percentile of reference calibration siteCSCI scores), possibly altered (10th–30th percentiles), likelyto be altered (1st–10th percentile), and very likely to be al-tered (<1st percentile). We used the qnorm() function in Rto estimate thresholds from the observed mean and SD ofreference calibration site CSCI scores. We explored otherapproaches to setting thresholds, such as varying thresh-olds by ecoregion or setting thresholds from environmen-tally similar reference sites, but rejected these approachesbecause of their added complexity and minimal benefits(Appendix S1).

We applied thresholds to a subset of sites from proba-bilistic surveys (n = 1318 sites) to provide weighted esti-mates of stream condition in California and for each ma-jor region. We also used the thresholds to make unweightedestimates of reference, moderate-activity, and high-activitysites for each region of the state. We used unweighted es-timates because few reference probabilistic samples wereavailable in certain regions. For weighted estimates, we cal-culated site weights by dividing total stream length in eachstratum by the number of sampled sites in that stratum(these strata were defined as the intersections of strata fromeach contributing survey). All weight calculations were con-

ducted using the spsurvey package (Kincaid and Olsen2013) in R (version 2.15.2). We used site weights to estimateregional distributions for environmental variables using theHorvitz–Thompson estimator (Horvitz and Thomson 1952).Confidence intervals for estimates of the proportion of Cal-ifornia’s stream length meeting reference criteria were basedon local neighborhood variance estimators (Stevens andOlsen 2004).

RESULTSBiological and environmental diversity of California

Biological assemblages varied markedly across naturalgradients in California, as indicated by cluster analysis. Weidentified 11 groups that contained 13 to 61 sites (Fig. 3).A few of these groups were geographically restricted, butmost were distributed across many regions of the state. Forexample, sites in group 10 were concentrated in the Trans-verse Ranges of southern California, and sites in group 7were entirely within the Sierra Nevada. In contrast, sites ingroups 1 and 4 were broadly distributed across the north-ern ⅔ of California.

Environmental factors differed among several groups.Groups 8 through 11, all in the southern portions of thestate, were generally drier and hotter than other groups,whereas groups 1 through 5, predominantly in mountain-ous and northern regions, were relatively wet and cold.Expected number of taxa also varied across groups. Forexample, the highest median E (i.e., sum of capture prob-abilities > 0.5) (17.2) was observed in group 3, whereasthe lowest (7) was observed in group 8. The median Ewas <10 for 3 of the 11 groups (groups 8, 10, and 11).Sites in low-E groups were preponderantly (but not ex-clusively) in the southern portions of the state.

Development of predictive modelsPredicting the number of locally common taxa for the O/Eindex The random-forest model selected to predict as-semblage composition used 5 predictors: latitude, eleva-tion, watershed area, mean annual precipitation, and meanannual air temperature (Table 1). The model explained 74and 64% of the variation in O at calibration and validationsites, respectively. Regression slopes (1.05 and 0.99 at cal-ibration and validation sites, respectively) and intercepts(–0.36 and 0.52) were similar to those expected from un-biased predictions (i.e., slope = 1 and intercept = 0, p >0.05). The random-forest model was modestly more pre-cise (SD = 0.19) than the null model (SD = 0.21) but sub-stantially less precise than the best model possible (SD =0.13).

Predicting metric values and developing the pMMI Pre-dictive models explained >20% of variance in 17 of the48 metrics evaluated for inclusion in the pMMI (a subsetof which are shown in Table 3). For 10 metrics, ≥30% of



the variance was explained, and for 2 metrics (no. intol-erant taxa and % intolerant taxa), >50% of the variancewas explained. Squared correlation coefficients (r2) be-tween predicted and observed metric values ranged fromnear 0 (e.g., Simpson diversity) to >0.5 (no. and % intol-erant taxa metrics). Results for validation reference siteswere consistent with results for calibration sites, but r2 val-ues differed markedly between calibration and validationdata sets for some metrics (Table 3). In general, modelsexplained the most variance for %-taxa metrics, and theleast for %-abundance metrics, but this pattern was notconsistent for all groups of metrics.

Metrics selected for the pMMI Of the 48 metrics evalu-ated, 15 met all acceptability criteria (Table 3). The biascriterion was the most restrictive and eliminated 21 met-rics, including all raw metrics and 2 modeled metrics (%climber taxa and % predators). The discrimination crite-rion eliminated 15 metrics, most of which were alreadyeliminated by the bias criterion. Other criteria eliminatedfew metrics, all of which were already rejected by othercriteria. The 15 acceptable metrics yielded 28,886 possi-ble prototype pMMIs ranging in size from 5 to 10 met-rics, but only 234 prototype pMMIs contained uncorre-lated metrics or metrics belonging to unique metric groups(data not shown). All of these prototype pMMIs contained≤7 metrics. Of these 234 prototypes, only 6 were in the topquartile for both discrimination between reference and high-activity calibration samples and for lowest SDs among ref-erence calibration samples.

The final pMMI included 1 metric from each of 6 met-ric groups (Table 3). Some of the selected metrics (e.g.,Coleoptera % taxa) were similar to those used in regionalindices previously developed in California (e.g., Ode et al.2005). However, other widely used metrics (e.g., noninsectmetrics) were not selected because they were highly cor-related with other metrics that had better performance(pairwise correlations not shown).

The random-forest models varied in how much of thevariation in the 6 individual metrics they explained (Pseudo-R2 range: 0.23–0.39). Regressions of observed on predictedvalues for reference calibration data showed that severalintercepts were significantly different from 0 and slopeswere significantly different from 1 (i.e., p < 0.05), but thesedifferences were small. The number of predictors used ineach of the 6 models ranged from 2 (for no. Coleoptera

Figure 3. Dendrogram and geographic distribution of eachgroup identified during cluster analysis. Numbers next to leavesare median values for expected number of taxa (E), elevation(Elev, m), precipitation (PPT, mm), and air temperature (Temp,°C).



taxa) to 10 (for taxonomic richness) (Table 1). Predictorsrelated to location (e.g., latitude, elevation) were widelyused, with latitude appearing in every model. In contrast,predictors related to geology (e.g., soil erodibility) or catch-ment morphology (e.g., watershed area) were used less of-ten. In general, the most frequently used predictors alsohad the highest importance in the predictive models, asmeasured by % increase in mean square error. The leastfrequently used predictor (i.e., % P geology) was used in1model (taxonomic richness).

Performance of predictive modelsEffects of predictive modeling on metrics For most met-rics, reducing the influence of natural gradients throughpredictive modeling reduced the calculated difference be-tween high-activity and reference sites, a result suggestingthat stressor and natural gradients can have similar andconfounded effects on many metric values (Table 3). Forexample, for 27 of the 48 metrics evaluated, the absolutet-statistic was much higher (difference in |t| > 1) for theraw metric than for the residuals. In contrast, the absolutet-statistic for residuals was higher for only 12metrics.

Performance evaluation of the O/E, pMMI, and combinedindices By all measures, predictive indices (whetherused alone or combined) performed better than their nullcounterparts, particularly with respect to accuracy/bias(Table 5). For example, mean regional differences in nullindex scores at reference sites were large and significant(Fig. 4A, C, E), and responses to natural gradients were

strong (Fig. 5A–O). In contrast, all measures of biaseswere greatly reduced for predictive indices (Fig. 4B, D, F).

Predictive modeling improved several aspects of preci-sion. Variability of scores among reference sites was lowerfor all predictive indices than for their null counterparts,particularly for the pMMI (Table 5). Regional differencesin precision were larger for the pMMI than O/E (bothpredictive and null models), and combining these 2 indi-ces into the CSCI improved regional consistency in preci-sion (Fig. 4B, D, F). Predictive modeling had a negligibleeffect on within-site variability (Table 5).

In contrast to precision and accuracy, responsivenesswas more affected by index type than whether predictiveor null models were used. Both predictive and null MMIsappeared to be slightly more responsive than the com-bined indices, which in turn were more responsive thanO/E indices. This pattern was evident in all measures of re-sponsiveness, such as magnitude of t-statistics, variance ex-plained by multiple human-activity gradients in a random-forest model, and steepness of slopes against individualgradients (Table 5, Fig. 6A–I).

Analysis of sensitivity indicated stronger sensitivity ofthe pMMI than the O/E, and the combined index hadintermediate sensitivity. Overall, 47% of nonreference siteshad scores <10th percentile of reference calibration sitesfor the CSCI, in contrast with 52% of the pMMI and 35%of the O/E. Despite the overall difference between thepMMI and the O/E, agreement was relatively high (76%)when the 10th percentile was used as an impairment thresh-old (i.e., O/E ≥ 0.76 and pMMI ≥ 0.77). When the 1st per-centile was used to set thresholds (i.e., O/E ≥ 0.56 andpMMI ≥ 0.58), the agreement rate was 90%.

Table 5. Performance measures to evaluate California State Condition Index (CSCI), MMI = multimetric index, and observed (O)/expected (E) taxa index at calibration (Cal) and validation (Val) sites. For accuracy tests, only reference sites were used. Ref mean =mean score of reference sites (* indicates value is mathematically fixed at 1), F = F-statistic for differences in scores at calibrationsites among 5 regions (shown in Fig. 1, Central Valley excluded; residual df = 467), Var = variance in index scores explained bynatural gradients at reference sites, among sites = standard deviation of scores at reference sites, within sites = standard deviation ofwithin-site residuals for reference Cal (n = 220 sites) and Val (n = 60) sites with multiple samples, t = t-statistic for difference betweenmean scores at reference and high-activity sites, var = variance in index scores explained by human-activity gradients at all sites.

Index Type

Accuracy Precision Responsiveness

Ref mean F Var Among sites Within sites t Var

Cal Val Cal Val Cal Val Cal Val Cal Val Cal Val Cal Val

CSCI Predictive 1.01 1.01 1.3 1.4 −0.08 −0.13 0.16 0.17 0.11 0.1 28.5 13 0.49 0.42

Null 1* 1 52.9 4.7 0.41 0.12 0.21 0.2 0.11 0.11 28.6 14.8 0.64 0.58

MMI Predictive 1* 0.98 0.8 1.3 −0.15 −0.09 0.18 0.19 0.12 0.12 30.9 14.4 0.54 0.48

Null 1* 1 62.2 8.7 0.46 0.2 0.24 0.24 0.12 0.12 29.2 15.3 0.67 0.61

O/E Predictive 1.02 1.03 1.2 1 0.01 −0.12 0.19 0.2 0.16 0.13 21.0 9.3 0.31 0.25

Null 1* 1 23.5 0.9 0.23 −0.03 0.21 0.22 0.15 0.13 24.1 11.8 0.48 0.41



Effect of E on performance By most measures, perfor-mance was better at high-E than at low-E sites, but pre-dictive indices were much more consistent than theirnull equivalents. For example, the accuracy of null indiceswas very poor at low-E sites (0.46–0.54 at E = 5; Fig. 7A),whereas predictive indices were much more accurate (0.73–0.86 at E = 5; Fig. 7E. At high-E sites, accuracy was >0.90for both predictive and null indices. Precision was better athigh-E sites for the pMMI and O/E index, but the CSCIhad better and more consistent precision than the otherindices at all values of E (Fig. 7B, F). For example, preci-sion ranged from 0.22 to 0.15 (range = 0.07) for boththe pMMI and the O/E, whereas it ranged from 0.18 to0.14 (range = 0.04) for the CSCI.

In contrast to the weak associations between E and ac-curacy and precision, E was very strongly associated withsensitivity, as measured by the percentage of high-activitysites with scores <10th percentile threshold (Fig. 7C, G).

The pMMI classified a larger proportion of sites as in non-reference condition across nearly all values of E than theO/E index did, but the difference was largest at low-E sites(Fig. 7D, H). For example, at the lowest values of E ana-lyzed (5), the pMMI identified 87% of high-activity sites asbiologically different from reference, whereas O/E identi-fied only 47% of sites as in nonreference condition. As Eincreased, the difference between the 2 indices in propor-tion of sites classified as nonreference decreased. Wald’sinterval test indicated significant differences between theindices for values of E up to 13. At low-E sites, the sensi-tivity of the CSCI was between the 2 indices, but at high-Esites, CSCI was more similar to pMMI. All 3 indices showedthat low-E sites were more pervasively in nonreferencecondition than high-E sites, and the proportion of siteswith scores <10th percentile of reference calibration sitesdecreased as E increased. In contrast to precision and ac-curacy, sensitivity was more consistent across settings for

Figure 4. Box-and-whisker plots for distribution of scores for null (A, C, E) and predictive (B, D, F) models for the observed(O)/expected (E) taxon index (A, B), multimetric index (MMI) (C, D), and the combined index (CSCI) (E, F) scores by geographicregion (see Fig. 1 for codes). White boxes indicate scores at calibration sites, and gray boxes indicate scores at validation sites. Thehorizontal dashed lines indicate the expected value at reference sites (= 1). Lines in boxes are medians, box ends are quartiles,whiskers are 1.5× the interquartile range, and dots are outliers (i.e., values >1.5× the interquartile range).



null than predictive indices. For all analyses of perfor-mance relative to E, validation data yielded similar results(not shown).

Establishment of biological condition classesand application to a statewide assessment

We established 4 biological condition classes based onthe distribution of CSCI scores at reference calibrationsites. Statewide, 52% of streams were likely to be intact (i.e.,CSCI ≥ 0.92 [30th percentile of reference calibration sites]).Another 18% were possibly altered (i.e., CSCI ≥ 0.79 [10th

percentile]), 11% were likely to be altered (i.e., CSCI ≥ 0.63[1st percentile]), and 19% were very likely to be altered (i.e.,CSCI < 1st percentile) (Table 6). Although many (i.e., 49%)high-activity sites were very likely to be altered, this num-ber varied considerably by region. Few high-activity siteswere in this condition class in the more forested regions(e.g., 24% in the North Coast, 15% in the Sierra Nevada),whereas higher numbers were observed in relatively arid re-gions (e.g., 100% in the Desert/Modoc region and 68% in

the Central Valley). In contrast, the percentage of refer-ence sites in the top 2 classes varied much less acrossregions, from a low of ∼85% in the South Coast and Des-ert/Modoc regions to a high of 98% in the North Coast(Table 6).

DISCUSSIONOur evaluation of index performance across different

environmental settings demonstrates that, to the greatestextent possible with existing data, we have designed anindex with scores that have comparable meanings for dif-ferent stream types in an environmentally heterogeneousregion of the USA. Each site is benchmarked against ap-propriate biological expectations anchored by a large andconsistently defined reference data set, and deviationsfrom these expectations reflect site condition in a consis-tent way across environmental settings. Thus, the indexcan be used to evaluate the condition of nearly all peren-nial streams in California, despite the region’s consider-able environmental and biological complexity. Three ele-

Figure 5. Relationships between observed (O)/expected (E) taxon index (A–E), multimetric index (MMI) (F–J), and the combinedindex (CSCI) (K–O) scores and slope (A, F, K), % fast water (area of reach with riffle, run, cascade, or rapid microhabitats) (B, G, L),% sand and fines (C, H, M), sampling date (D, I, N), and day of the year (E, J, O) at reference sites for predictive (black symbols, solidlines) and null (gray symbols, dashed lines) indices. The dotted line indicates a perfect relationship without bias.



ments of the design process contributed to the utility ofthis index in an environmentally complex region: a robustreference data set, predictive modeling, and the combina-tion of multiple endpoints into a single index.

Large, representative reference data setsThe 1st element was the large, representative, and rig-

orously evaluated reference data set (Ode et al. 2016). Nat-ural factors that influence biological assemblages mustbe adequately accounted for to create an assessment toolthat performs well across environmental settings (Cao et al.2007, Schoolmaster et al. 2013). The strength of relation-ship between natural factors and biology varies with geo-graphic scale (Mykrä et al. 2008, Ode et al. 2008), andrepresenting locally important factors (such as unusual ge-ology types with limited geographic extent, e.g., Campbellet al. 2009) contributes to the ability of the index to distin-guish natural from anthropogenic biological variability inthese environmental settings. Our reference data set wasspatially representative and encompassed >10 y of sam-pling. Long-term temporal coverage improves the repre-

sentation of climatic variability, including El Niño-relatedstorms and droughts. The spatial and temporal breadth ofsampling at reference sites provides confidence in the ap-plicability of the CSCI for the vast majority of wadeableperennial streams in California.

Predictive modelingThe 2nd element of the CSCI’s design, predictive mod-

eling, enabled the creation of site-specific expectationsfor 2 indices, and these models created indices superiorto those created by null models in nearly every aspect,particularly with respect to bias in certain settings. Theseresults are consistent with a large body of literature show-ing similar results for indices that measure changes intaxonomic composition (e.g., Reynoldson et al. 1997, Haw-kins et al. 2000, Van Sickle et al. 2005, Hawkins 2006,Mazor et al. 2006). However, few studies to date showedthat the benefits extend to MMIs (e.g., Bates Prins andSmith 2007, Pont et al. 2009, Hawkins et al. 2010b, School-master et al. 2013, Vander Laan and Hawkins 2014).

Figure 6. Relationships between observed (O)/expected (E) taxon index (A–C), multimetric index (MMI) (D–F), and the combinedindex (CSCI) (G–I) scores and % developed area of the watershed (WS) (A, D, G), riparian activity (B, E, H), and % sand and fines(C, F, I) for predictive (black symbols, solid lines) and null indices (gray symbols, dashed lines). The dotted line indicates the referenceexpectation of 1.



Our preference for predictive over traditional MMIs isnot based only on the superior performance the pMMIrelative to its null counterpart. The null MMI evaluated inour study was simplistic and did not reflect typical typo-logical approaches to MMI development, which includeregionalization in metric selection (e.g., Stoddard et al.2008), regionalization in scoring (e.g., Ode et al. 2005), ornormalization to watershed area (e.g., Klemm et al. 2003)to account for variability across reference sites. However,traditional MMIs based on regionalization usually lackmetric and scoring standardization, which complicates in-terregional comparisons. Even if typological approachesprovided equivalent performance to predictive indices, thelatter would be preferred because of their ability to setsite-specific management goals because predictive indicescan better match the true potential of individual sites(Hawkins et al. 2010b). Thus, a watershed manager couldtake action to maintain a level of diversity a stream can

truly support, rather than a level typical of potentially dis-similar reference sites.

Combining multiple indicesThe 3rd element of the CSCI’s design that contributed

to its utility in different stream types was inclusion of boththe pMMI and the O/E index. Regulatory agencies ex-pressed a strong preference for a single index to supportbiocriteria implementation, and we thought that theCSCI was preferable to either the pMMI or O/E index.The different sensitivities of the 2 components shouldenhance the utility of the CSCI across a broad range ofdisturbances and settings. Together, they provide multiplelines of evidence about the condition of a stream andprovide greater confidence in the results than a singleindex that might be biased in certain settings. Use of bothmetric and multivariate indices is widespread in assess-ments of coastal condition (e.g., the M-AMBI index; Mu-xika et al. 2007) specifically because the combination takesadvantage of the unique sensitivities of each index in dif-ferent habitat types (Sigovini et al. 2013). Applications ofa multiple-index approach in stream assessment pro-grams are uncommon, but the need has been suggested(e.g., Reynoldson et al. 1997, Mykrä et al. 2008, Collier2009).

The decision to use both the pMMI and O/E index wasbased, at least partly, on observations that they had dif-ferent sensitivities in different settings, particularly at low-E sites. The difference between the 2 indices might meanthat the O/E index correctly indicates a greater resilienceto stress at certain stream types or that the pMMI is morefinely tuned to lower levels of stress simply because it wasspecifically calibrated against high-activity sites in similarsettings. Mechanistically, the difference probably occurredbecause O/E index scores are mainly affected by the lossof common taxa. For example, in low-E sites (which werecommon in dry, low-elevation environments in southernand central coastal California), the O/E index predictedoccurrence of only a small number of highly toleranttaxa (e.g., baetid mayflies) because only these tolerant taxaoccur with high probability in these naturally stressful en-vironments. Sensitive taxa also occur at reference sites indrier, low-elevation settings, but they were typically toorare to affect the O/E index (Appendix S2).

The interpretive value of rare, sensitive taxa in estima-tion of biological integrity of an individual site is unclear,but the ability of a site to support these taxa may be im-portant to the health of a dynamic metacommunity, whererare taxa occupy only a small subset of suitable sites atany one time. Although several investigators have shownthat exclusion of rare taxa usually enhances precision ofO/E indices (e.g., Ostermiller and Hawkins 2004, VanSickle et al. 2007), our results suggest that in certain set-tings, this exclusion may obscure an important response to

Figure 7. Effect of expected number of taxa (E) on accuracy(A, E), precision (B, F), sensitivity (C, G), and difference insensitivity between the predictive multimetric index (pMMI)and the observed (O)/expected (E) taxa indices (D, H) for null(A–D) and predictive (E–H) index performance. The gray bandsin the bottom panels C and G indicate the 95% confidenceinterval around the difference. Accuracy = proportion of refer-ence calibration sites in reference condition (i.e., score >10th per-centile of reference calibration sites) for each index. Precision =standard deviation of reference calibration sites for each index.Sensitivity = proportion of high-activity sites not in referencecondition.



stress. Including rare taxa in certain environmental set-tings while excluding them in others may improve theconsistency of an O/E index in complex regions, but wedid not explore this option. The observation that sensitivityof all indices was lowest where E was highest was unex-

pected, and may be attributed to several potential causes.Most probably, anthropogenic stress was less severe athigh-E than at low-E sites. High-activity sites were identi-fied via indirect measures based on stressor sources (e.g.,development in the watershed) rather than direct measures

Table 6. Percentage of sites in different condition classes by region and site status. Percentiles refer to the distribution of scores atreference calibration (Cal) sites. Overall estimates are based on sites from probabilistic surveys and are not split into Cal or validation(Val) sets. For reference, moderate-, and high-activity sites, numbers in the last 6 columns are percentage of sites. For overall assessments,these numbers are percentage of stream miles. Dashes indicate that no sites were analyzed.

Region

Total sites

Likely to beintact ≥30th

percentile(CSCI ≥ 0.92)

Possiblyaltered 30th–10th percentile(CSCI ≥ 0.79)

Likely to bealtered 1st–10th

percentile(CSCI ≥ 0.63)

Very likely tobe altered <1st

percentile(CSCI < 0.63)

Cal Val Cal Val Cal Val Cal Val Cal Val

Statewide

Reference 473 117 75 74 15 16 8 8 1 3

Moderate activity 626 156 53 56 20 20 18 17 8 7

High activity 497 122 13 18 13 14 25 22 49 46

Overall 919 52 18 11 19

North Coast

Reference 60 16 85 63 13 31 0 6 2 0


High activity 45 9 29 67 33 33 13 0 24 0

Overall 162 58 23 10 9

Chaparral

Reference 74 19 68 63 20 26 9 0 3 11


High activity 126 28 18 21 13 7 18 11 50 61

Overall 147 34 16 17 33

South Coast

Reference 96 23 70 70 16 9 14 22 1 0


High activity 241 60 5 10 12 13 32 27 52 50

Overall 387 44 16 16 24

Sierra Nevada

Reference 221 55 77 82 14 11 7 5 1 2


High activity 27 8 56 25 11 38 19 13 15 25

Overall 106 70 19 6 5

Central Valley

Reference 1 0 100 – 0 – 0 – 0 –


High activity 47 13 0 0 4 8 28 38 68 54

Overall 60 2 8 18 71

Desert/Modoc

Reference 21 4 71 75 14 25 14 0 0 0


High activity 5 4 0 50 0 0 0 50 100 0

Overall 57 48 14 9 30



of water or habitat quality, so we could not ensure homog-enous levels of disturbance among this set of sites. Alter-natively, high-E settings might be more resilient to stress,perhaps because of their greater diversity (Lake 2000).Thus, the indices may have different responses to thesame level of stress in different settings, depending on E.

Despite the lower sensitivity of the O/E index at low-Esites, we think that including it in a combined index waspreferable to using the more sensitive pMMI by itself.Combining the 2 indices was a simple way to retain highsensitivity at low-E sites, while retaining the advantages ofthe O/E as a measure of biodiversity (Moss et al. 1987, Haw-kins et al. 2000). The ability of the O/E index to measuretaxonomic completeness has direct applications to con-servation of biodiversity and makes it particularly sensitiveto replacement of native fauna by invasive species. Fur-thermore, because it is calibrated with only reference sites,the O/E index is not influenced by the distribution or qual-ity of high-activity sites. In contrast, we used the pMMIunder the assumption that the set of high-activity sites ad-equately represented the types of stressors that might beencountered in the future. Inclusion of the O/E index inthe CSCI provides a degree of insurance against faulty as-sumptions about the suitability of the high-activity site setfor pMMI calibration.

We combined the 2 indices as an unweighted mean forseveral technical reasons, but primarily because this wasthe simplest approach to take without stronger supportfor more complicated methods. As we demonstrated, theCSCI has less variable performance across stream typesthan its 2 components. Approaches that let the lowest (orhighest) score prevail are more appropriate when the com-ponents have similar sensitivity, but in our case would betantamount to using the pMMI alone and muting the in-fluence of the O/E index. Approaches that weight the 2 com-ponents based on site-specific factors (e.g., weighting thepMMI more heavily than the O/E index at low-E sites) areworthy of future exploration. Evaluating the pMMI and O/E indices independently to assess biological condition at asite might be useful, particularly at low-E sites, but the com-bined index is preferred for applications where statewideconsistency is important, such as designation of impairedwaterbodies.

Unexplained variabilityIn our study, predictive models were able to explain only

a portion of the variability observed at reference sites—sometimes a fairly small portion. For example, the SD ofthe predictive O/E was only slightly lower than the SD ofthe null O/E (0.19 vs 0.21) and much larger than thatassociated with replicate samples (0.13). None of the se-lected random-forest models explained >39% (for the no.shredder taxa metric) of the variability at reference cali-bration sites. The unexplained variability may be relatedto the additional effects of environmental factors that are

unsuitable for predicting reference condition (e.g., alter-able factors, like substrate composition or canopy cover),environmental factors unrelated to those used for model-ing (e.g., temporal gradients, weather antecedent to sam-pling), field and laboratory sampling error, metacommu-nity dynamics (Leibold et al. 2004, Heino 2013), or neutralprocesses in community assembly that are inherently un-predictable (Hubbell 2001, Rader et al. 2012). The relativecontribution of these factors is likely to be a fruitful areaof bioassessment research. Given the number and breadthof environmental gradients evaluated for modeling, we thinkit unlikely that additional data or advanced statistical meth-ods will change the performance of these indices.

Setting thresholdsSome investigators have suggested that thresholds for

identifying impairment in environmentally complex re-gions may require different thresholds in different settingsbased on the variability of reference streams in each set-ting. For example, Yuan et al. (2008) proposed ecoregionalthresholds for an O/E index for the USA based on theobservation that index scores at reference sites were twiceas variable in some ecoregions as in others. Alternatively,site-specific thresholds could be established based on thevariability of a subset of environmentally similar referencesites. We rejected both of these approaches in favor ofuniform thresholds based on the variability of all referencecalibration sites. We rejected ecoregional thresholds orother typological approaches because the validity of eco-regional classifications may be questionable for sites nearboundaries. We rejected site-specific thresholds based on en-vironmentally similar reference sites because they did notimprove accuracy or sensitivity relative to a single statewidethreshold when predictive indices are used (Appendix S1).These results are consistent with those of Linke et al.(2005), who showed that indices calibrated with environ-mentally similar reference sites had similar performanceto indices based on predictive models that were calibratedwith all available reference sites. Other approaches, suchas direct modeling of the SD of index scores as a functionof natural factors, also might improve comparability ofscores across settings (R. Bailey, Cape Breton University,personal communication).

Conclusions and recommended applicationsMany recent technical advances in bioassessment have

centered on improving the performance of tools used toscore the ecological condition of water bodies. Much ofthe progress in this area has come from regional, national,and international efforts to produce overall condition assess-ments of streams in particular regions (e.g., Simpson andNorris 2000, Van Sickle et al. 2005, Hawkins 2006, Heringet al. 2006, Stoddard et al. 2006, Paulsen et al. 2008). A keychallenge in completing these projects has been incompat-



ibility among scoring tools designed to assess streams inmultiple regions, each calibrated for unique and locally im-portant environmental gradients (Cao and Hawkins 2011).This issue has been well documented for large-scale pro-grams in which investigators have attempted to integratescores from a patchwork of assessment tools built for smallersubregions (Heinz Center 2002, Hawkins 2006, Meadoret al. 2008, Pont et al. 2009), but far less attention hasbeen paid to the meaning of index scores at individualstream reaches (Herlihy et al. 2008, Ode et al. 2008). As-sessment of CSCI performance across the range of envi-ronmental settings in California was essential because theCSCI is intended for use in regulatory applications thataffect the management of individual reaches, and consis-tent meaning of a score was a key requirement of regu-latory agencies and stakeholders.We attempted tomaximizeconsistency of the CSCI by using a large and representativereference set and by integrating multiple indices based onpredictive models. Consistent accuracy was attained throughthe use of predictive models, whereas the consistency ofprecision and sensitivity was improved through the use ofmultiple endpoints.

The CSCI was designed for condition assessments, butwe think it has broad application to many aspects of streammanagement. For example, it could be used to select com-parator sites with similar biological expectations to test sitesfor use in causal assessments (e.g., CADDIS; USEPA 2010)or to prioritize streams that can support rare or threatenedassemblages for restoration or conservation (Linke et al.2011). The predictions generated by the index can informmanagement decisions about streams for which no biolog-ical data are available. Predictive indices, such as the CSCI,are powerful additions to the stream manager’s tool kit,especially in environmentally complex areas. We recognizethe challenges in enabling the general public to calculatean index as complex as the one presented here. Fortunately,online automation of many of the steps is possible. For ex-ample, much of the GIS analysis can be simplified by usingpublicly available resources like StreamStats (US Geolog-ical Survey 2012). An automated tool is in development,but people who are interested in using the CSCI or examin-ing its component models are encouraged to contact theauthors.

ACKNOWLEDGEMENTSWe thank Dave Buchwalter, Rick Hafele, Chris Konrad,

LeRoy Poff, John Van Sickle, Lester Yuan, Jason May, LarryBrown, Ian Waite, Kevin Lunde, Marco Sigala, Joseph Furnish,and Betty Fetscher for contributions to the approach describedin this paper. Financial support was provided by grants from theUS EPA Region IX and the California State Water ResourcesControl Board. Most of the data used in this analysis were pro-vided by the California Surface Water Ambient Monitoring Pro-gram, the California Department of Fish and Wildlife, the Sierra

Nevada Aquatic Research Lab, the Stormwater Monitoring Coa-lition of Southern California, the US Forest Service, and theGeographic Information Center at California State UniversityChico. We thank Kevin Collier, Bob Bailey, and 1 anonymousreferee for their valuable feedback on the manuscript. This pa-per is dedicated to the memory of Miriam D. Mazor.

LITERATURE CITEDAgresti, A., and Y. Min. 2005. Simple improved confidence inter-

vals for comparing matched proportions. Statistics in Medi-cine 24:729–740.

Angradi, T. R., M. S. Pearson, D. W. Bolgrein, T. M. Jicha, D. L.Taylor, and B. H. Hill. 2009. Multimetric macroinvertebrateindices for mid-continent US great rivers. Journal of theNorth American Benthological Society 28:785–804.

Bates Prins, S. C., and E. Smith. 2007. Using biological metricsto score and evaluate sites: a nearest-neighbour reference con-dition approach. Freshwater Biology 52:98–111.

Blocksom, K. A. 2003. A performance comparison of metricscoring methods for a multimetric index for Mid-Atlantichighland streams. Environmental Management 42:954–965.

California State Water Resources Control Board. 2013. RegionalWater Quality Control Board boundaries. California State Wa-ter Resources Control Board, Sacramento, California. (Availablefrom: http://www.waterboards.ca.gov/waterboards_map.shtml)

CAMLnet. 2003. List of California macroinvertebrate taxa andstandard taxonomic effort. California Department of Fish andGame, Rancho Cordova, California. (Available from: www.safit.org)

Campbell, R. H., T. H. McCulloh, and J. G. Vedder. 2009. TheMiocene Topanga group of southern California—a 100-yearhistory of changes in stratigraphic nomenclature. Open-FileReport 2007-1285. Version 1.1. US Geological Survey, Reston,Virginia.

Cao, Y., and C. P. Hawkins. 2011. The comparability of bioassess-ments: a review of conceptual and methodological issues. Jour-nal of the North American Benthological Society 30:680–701.

Cao, Y., C. P. Hawkins, J. Olson, and M. A. Kosterman. 2007.Modeling natural environmental gradients improves the ac-curacy and precision of diatom-based indicators. Journal ofthe North American Benthological Society 26:566–585.

Clarke, K. R., and R. H. Green. 1988. Statistical design and anal-ysis for a “biological effects” study. Marine Ecology ProgressSeries 46:213–226.

Clarke, R. T., J. F. Wright, and M. T. Furse. 2003. RIVPACSmodels for predicting the expected macroinvertebrate faunaand assessing the ecological quality of rivers. Ecological Mod-elling 160:219–233.

Collier, K. J. 2009. Linking multimetric and multivariate ap-proaches to assess the ecological condition of streams. Envi-ronmental Monitoring and Assessment 157:113–124.

Council of European Communities. 2000. Establishing a frame-work for community action in the field of water policy. Direc-tive 2000/60/EC. Official Journal of European Communities.L327(43):1–72.

Cutler, D. R., T. C. Edwards, K. H. Beard, A. Cutler, and K. T.Hess. 2007. Random forests for classification in ecology. Ecol-ogy 88:2783–2792.



Davis, W. S., and T. P. Simon. 1995. Biological assessment andcriteria: tools for water resource planning and decision mak-ing. Lewis Press, Boca Raton, Florida.

Fore, L. S., and C. Grafe. 2002. Using diatoms to assess thebiological condition of large rivers in Idaho (U.S.A.). Fresh-water Biology 47:2014–2037.

Gauch, H. G. 1982. Multivariate analysis in community ecology.Cambridge University Press, Cambridge, UK.

Gerritsen, J. 1995. Additive biological indices for resource man-agement. Journal of the North American Benthological So-ciety 14:451–457.

Gerth, W. J., and A. T. Herlihy. 2006. The effect of samplingdifferent habitat types in regional macroinvertebrate bio-assessment surveys. Journal of the North American Bentho-logical Society 25:501–512.

Hawkins, C. P. 2006. Quantifying biological integrity by taxo-nomic completeness: its utility in regional and global assess-ments. Ecological Applications 16:1277–1294.

Hawkins, C. P., Y. Cao, and B. Roper. 2010a. Method of predictingreference condition biota affects the performance and interpre-tation of ecological indices. Freshwater Biology 55:1066–1085.

Hawkins, C. P., R. H. Norris, J. N. Hogue, and J. W. Feminella.2000. Development and evaluation of predictive models formeasuring the biological integrity of streams. Ecological Ap-plications 10:1456–1477.

Hawkins, C. P., J. R. Olson, and R. A. Hill. 2010b. The refer-ence condition: predicting benchmarks for ecological andwater-quality assessments. Journal of the North AmericanBenthological Society 29:312–343.

Heino, J. 2013. The importance of metacommunity ecology forenvironmental assessment research in the freshwater realm.Biological Reviews 88:166–178.

Heinz Center (H. John Heinz III Center for Science and theEnvironment). 2002. The state of the nation’s ecosystems:measuring the lands, waters, and living resources of theUnited States. Cambridge University Press, New York.

Herbst, D. B., and E. L. Silldorff. 2006. Comparison of the perfor-mance of different bioassessment methods: similar evaluationsof biotic integrity from separate programs and procedures. Jour-nal of the North American Benthological Society 25:513–530.

Herbst, D. B., and E. L. Silldorff. 2009. Development of a benthicmacroinvertebrate index of biological integrity (IBI) for streamassessments in the Eastern Sierra Nevada of California. SierraNevada Aquatic Research Lab, Mammoth Lakes, California.(Available from: http://www.waterboards.ca.gov/lahontan/water_issues/programs/swamp/docs/east_sierra_rpt.pdf )

Hering, D., R. K. Johnson, S. Kramm, S. Schmutz, K. Szoszkiewicz,and P. F. Verdonschot. 2006. Assessment of European streamswith diatoms,macrophytes,macroinvertebrates andfish: a com-parative metric-based analysis of organism response to stress.Freshwater Biology 51:1757–1785.

Herlihy, A. T., S. G. Paulsen, J. Van Sickle, J. L. Stoddard, C. P.Hawkins, and L. Yuan. 2008. Striving for consistency in anational assessment: the challenges of applying a referencecondition approach on a continental scale. Journal of the NorthAmerican Benthological Society 27:860–877.

Horvitz, D. G., and D. J. Thompson. 1952. A generalization ofsampling without replacement from a finite universe. Jour-nal of the American Statistical Association 47:663–685.

Hubbell, S. P. 2001. The unified neutral theory of biodiversity.Monographs in Population Biology 32:1–392.

Hughes, R. M., D. P. Larsen, and J. M. Omernik. 1986. Regionalreference sites: a method for assessing stream potentials.Environmental Management 10:629–635.

Hughes, R. M., P. R. Kauffmann, A. T. Herlihy, T. M. Kincaid,L. Reynolds, and D. P. Larsen. 1998. A process for developingand evaluating indices of fish assemblage integrity. CanadianJournal of Fisheries and Aquatic Sciences 55:1618–1631.

Kaufmann, P. R., P. Levine, E. G. Robinson, C. Seeliger, andD. V. Peck. 1999. Surface waters: quantifying physical habi-tat in wadeable streams. EPA/620/R-99/003. Office of Researchand Development, US Environmental Protection Agency,Wash-ington, DC.

Kincaid, T., and A. Olsen. 2013. Spsurvey: spatial survey designand analysis. R package version 105. R Project for StatisticalComputing, Vienna, Austria.

Klemm, D. J., K. A. Blocksom, F. A. Fulk, A. T. Herlihy, R. M.Hughes, P. R. Kaufmann, D. V. Peck, J. L. Stoddard, and W. T.Thoeny. 2003. Development and evaluation of a macroinver-tebrate biotic integrity index (MBII) for regionally assessingMid-Atlantic Highlands streams. Environmental Management31:656–669.

Kuhn, M., J. Wing, S. Weston, A. Williams, C. Keefer, and A.Engelhardt. 2012. caret: classification and regression training.R package, version 5.15-045. R Project for Statistical Com-puting, Vienna, Austria. (Available from: http://www.epa.gov/nheerl/arm)

Lake, P. S. 2000. Disturbance, patchiness, and diversity instreams. Journal of the North American Benthological Soci-ety 19:573–592.

Leibold, M. A., M. Holyoak, N. Mouquet, P. Amarasekare, J. M.Chase, M. F. Hoopes, R. D. Holt, J. B. Shurin, R. Law, D.Tilman, M. Loreau, and A. Gonzalez. 2004. The metacommu-nity concept: a framework for multi-scale community ecology.Ecology Letters 7:601–613.

Liaw, A., and M. Wiener. 2002. Classification and regression byrandomForest. R News 2:18–22.

Linke, S., R. H. Norris, D. P. Faith, and D. Stockwell. 2005.ANNA: a new prediction method for bioassessment programs.Freshwater Biology 50:147–158.

Linke, S., E. Turak, and J. Nel. 2011. Freshwater conservationplanning: the case for systematic approaches. Freshwater Bi-ology 56:6–20.

Maechler, M., P. Rousseeuw, A. Struyf, M. Hubert, and K.Hornik. 2012. cluster: cluster analysis basics and extensions.R package version 1.14.3. R Project for Statistical Comput-ing, Vienna, Austria.

Mazor, R. D., T. B. Reynoldson, D. M. Rosenberg, and V. H.Resh. 2006. Effects of biotic assemblage, classification, andassessment method on bioassessment performance. CanadianJournal of Fisheries and Aquatic Sciences 63:394–411.

Meador, M. R., T. R. Whittier, R. M. Goldstein, R. M. Hughes,and D. V. Peck. 2008. Evaluation of an index of biotic integ-rity approach used to assess biological condition in westernUS streams and rivers at varying spatial scales. Transactionsof the American Fisheries Society 137:13–22.

Moss, D., T. Furse, J. F. Wright, and P. D. Armitage. 1987. Theprediction of macro-invertebrate fauna of unpolluted running-



water sites in Great Britain using environmental data. Fresh-water Biology 17:41–52.

Muxika, I., Á. Borja, and J. Bald. 2007. Using historical data,expert judgment and multivariate analysis in assessing refer-ence conditions and benthic ecological status, according tothe European Water Framework Directive. Marine PollutionBulletin 55:16–29.

Mykrä, H., J. Aroviita, J. Kotanen, H. Hämäläinen, and T.Muotka. 2008. Predicting the stream macroinvertebrate faunaacross regional scales: influence of geographical extent onmodel performance. Journal of the North American Bentho-logical Society 27:705–716.

Ode, P. R., C. P. Hawkins, and R. D. Mazor. 2008. Comparabilityof biological assessments derived from predictive models andmultimetric indices of increasing geographic scope. Journal ofthe North American Benthological Society 27:967–985.

Ode, P. R., A. C. Rehn, and J. T. May. 2005. A quantitative toolfor assessing the integrity of southern coastal Californiastreams. Environmental Management 35:493–504.

Ode, P. R., A. C. Rehn, R. D. Mazor, K. C. Schiff, E. D. Stein, J. T.May, L. R. Brown, D. B. Herbst, D. Gillett, K. Lunde, and C. P.Hawkins. 2016. Evaluating the adequacy of a reference-poolsite for ecological assessments in environmentally complex re-gions. Freshwater Science 35:237–248.

Oksanen, J., F. G. Blanchet, R. Kindt, P. Legendre, P. Minchin,R. B. O’Hara, G. L. Simpson, P. Solymos, M. H. H. Stevens,and H. Wagner. 2013. vegan: community ecology package.R package version 2.0-6. R Project for Statistical Computing,Vienna, Austria.

Olson, J. R., and C. P. Hawkins. 2012. Predicting natural base-flow stream water chemistry in the western United States.Water Resources Research 48:W02504.

Omernik, J. M. 1987. Ecoregions of the conterminous UnitedStates. Map (scale 1:7,500,000). Annals of the Association ofAmerican Geographers 77:118–125.

Ostermiller, J. D., and C. P. Hawkins. 2004. Effects of samplingerror on bioassessments of stream ecosystems: applicationto RIVPACS-type models. Journal of the North AmericanBenthological Society 23:363–382.

Paulsen, S. G., A. Mayio, D. V. Peck, J. L. Stoddard, E. Tarquinio,S. M. Holdsworth, J. Van Sickle, L. L. Yuan, C. P. Hawkins,A. T. Herlihy, P. R. Kaufmann, M. T. Barbour, D. P. Larsen,and A. R. Olsen. 2008. Condition of stream ecosystems in theUS: an overview of the first national assessment. Journal ofthe North American Benthological Society 27:812–821.

Peck, D. V., A. T. Herlihy, B. H. Hill, R. M. Hughes, P. R.Kaufmann, D. J. Klemm, J. M. Lazorchak, F. H. McCormick,S. A. Peterson, S. A. Ringold, T. Magee, and M. Cappaert.2006. Environmental Monitoring and Assessment Program—Surface Waters Western Pilot study: field operations manualfor wadeable streams. EPA/620/R-06/003. Office of Researchand Development, US Environmental Protection Agency,Corvallis, Oregon.

Pont, D., R. M. Hughes, T. R. Whittier, and S. Schmutz. 2009.A predictive index of biotic integrity model for aquatic–vertebrate assemblages of western U.S. streams. Transactionsof the American Fisheries Society 138:292–305.

Rader, R. B., M. J. Keleher, E. Billman, and R. Larsen. 2012. His-tory, rather than contemporary processes, determines varia-

tion in macroinvertebrate diversity in artesian springs: theexpansion hypothesis. Freshwater Biology 57:2475–2486.

Rehn, A. C. 2009. Benthic macroinvertebrates as indicators ofbiological condition below hydropower dams on West SlopeSierra Nevada streams, California, USA. River Research andApplications 25:208–228.

Rehn, A. C., P. R. Ode, and C. P. Hawkins. 2007. Comparisonof targeted-riffle and reach-wide benthic macroinvertebratesamples: implications for data sharing in stream-conditionassessments. Journal of the North American BenthologicalSociety 26:332–348.

Rehn, A. C., P. R. Ode, and J. T. May. 2005. Development of abenthic index of biotic integrity (B-IBI) for wadeable streamsin northern coastal California and its application to regional305(b) assessment. Report to the State Water Resources ControlBoard. California Department of Fish, Rancho Cordova, Califor-nia. (Available from: http://www.waterboards.ca.gov/water_issues/programs/swamp/docs/reports/assess_nocal2005.pdf )

Reynoldson, T. B., R. C. Bailey, K. E. Day, and R. H. Norris.1995. Biological guidelines for freshwater sediment based onBEnthic Assessment of SedimenT (the BEAST) using a multi-variate approach for predicting biological state. AustralianJournal of Ecology 20:198–219.

Reynoldson, T. B., R. H. Norris, V. H. Resh, K. E. Day, and D. M.Rosenberg. 1997. The reference condition: a comparison ofmultimetric and multivariate approaches to assess water-quality impairment using benthic macroinvertebrates. Journalof the North American Benthological Society 16:833–852.

Richards, A. B., and D. C. Rogers. 2011. List of freshwatermacroinvertebrate taxa from California and adjacent statesincluding standard taxonomic effort levels. Southwest Associa-tion of Freshwater Invertebrate Taxonomists, Chico, California.(Available from: www.safit.org)

Roth, N., M. Southerland, J. Chaillou, R. Klauda, P. Kayzak, S.Stranko, S. Weisberg, L. Hall, and R. Morgan. 1998. Mary-land biological stream survey: development of a fish indexof biotic integrity. Environmental Monitoring and Assess-ment 51:89–106.

Royer, T. V., C. T. Robinson, and G. W. Minshall. 2001. Devel-opment of macroinvertebrate-based index for bioassessmentof Idaho rivers. Environmental Management 27:627–636.

Scherer, R. 2013. PropCIs: various confidence interval methodsfor proportions. R package, version 0.2-4. R Project for Sta-tistical Computing, Vienna, Austria.

Schoolmaster, D. R., J. B. Grace, E. W. Schweiger, B. R. Mitchell,and G. R. Guntenspergen. 2013. A causal examination of theeffects of confounding factors on multimetric indices. Eco-logical Indicators 29:411–419.

Sigovini, M., E. Keppel, and D. Tagliapietra. 2013. M-AMBI-revisited: looking inside a widely-used benthic index. Hydro-biologia 717:41–50.

Simpson, J. C., and R. H. Norris. 2000. Biological assessment ofriver quality: development of AusRivAS models and outputs.Pages 125–142 in J. F. Wright, D. W. Sutcliffe, and M. T.Furse (editors). Assessing the biological quality of freshwaters:RIVPACS and other techniques. Freshwater Biological Associ-ation, Ambleside, Cumbria, UK.

Sleeter, B. M., T. S. Wilson, C. E. Soulard, and J. Liu. 2011.Estimation of late twentieth century land-cover change in



California. Environmental Monitoring and Assessment 173:251–266.

Stevens, D. L., and A. R. Olsen. 2004. Spatially balanced sam-pling of natural resources. Journal of the American Statisti-cal Association 99:262–278.

Stoddard, J. L., A. T. Herlihy, D. V. Peck, R. M. Hughes, T. R.Whittier, and E. Tarquinio. 2008. A process for creating multi-metric indices for large-scale aquatic surveys. Journal of theNorth American Benthological Society 27:878–891.

Stoddard, J. L., D. P. Larsen, C. P., Hawkins, R. K. Johnson, andR. H. Norris. 2006. Setting expectations for the ecologicalcondition of streams: the concept of reference condition.Ecological Applications 16:1267–1276.

Stribling, J. B., B. K. Jessup, and D. L. Feldman. 2008. Precisionof benthic macroinvertebrate indicators of stream conditionin Montana. Journal of the North American BenthologicalSociety 27:58–67.

Strobl, C., A.-L. Boulesteix, A. Zeileis, and T. Hothorn. 2007.Bias in random forest variable importance measures: illustra-tions, sources, and a solution. BMC Bioinformatics 8:25.

USEPA (US Environmental Protection Agency). 2002. Summaryof biological assessment programs and biocriteria develop-ment for states, tribes, territories, and interstate commissions:streams and wadeable rivers. EPA-822-R-02-048. Office of En-vironmental Information and Office of Water, US Environ-mental Protection Agency, Washington, DC.

USEPA (US Environmental Protection Agency). 2010. CausalAnalysis/Diagnosis Decision Information System (CADDIS).Office of Research and Development, US EnvironmentalProtection Agency, Washington, DC. (Available from: http://www.epa.gov/caddis)

US Geological Survey. 2012. The StreamStats program. USGeological Survey, Reston, Virginia. (Available from: http://streamstats.usgs.gov)

Vander Laan, J. J., and C. P. Hawkins. 2014. Enhancing the perfor-mance and interpretation of freshwater biological indices: an ap-plication in arid zone streams. Ecological Indicators 36:470–482.

Van Sickle, J. 2010. Correlated metrics yield multimetric indi-ces with inferior performance. Transactions of the AmericanFisheries Society 139:1802–1817.

Van Sickle, J., C. P. Hawkins, D. P. Larsen, and A. T. Herlihy.2005. A null model for the expected macroinvertebrate as-semblage in streams. Journal of the North American Ben-thological Society 24:178–191.

Van Sickle, J., D. P. Larsen, and C. P. Hawkins. 2007. Exclusionof rare taxa affects performance of the O/E index in bio-assessments. Journal of the North American BenthologicalSociety 26:319–331.

Wright, J. F., D. W. Sutcliffe, and M. T. Furse. 2000. Assessing thebiological quality of freshwaters: RIVPACS and other techniques.Freshwater Biological Association, Ambleside, Cumbria, UK.

Yates, A. G., and R. C. Bailey. 2010. Selecting objectively definedreference stream sites for stream bioassessment programs.Environmental Monitoring and Assessment 170:129–140.

Yoder, C. O., and M. T. Barbour. 2009. Critical elements of statebioassessment programs: a process to evaluate program rigorand comparability. Environmental Monitoring and Assess-ment 150:31–42.

Yuan, L. L., C. P. Hawkins, and J. Van Sickle. 2008. Effects ofregionalization decisions on an O/E index for the US na-tional assessment. Journal of the North American BenthologicalSociety 27:892–905.



Appendix S1. Nearest-neighbor thresholds do not improve performance of predictive indices.

Variable impairment thresholds may be useful when the precision of an index varies

greatly across settings (Death and Winterbourn 1994). For example, Yuan et al. (2008) observed

2-fold differences in variability at reference sites across ecoregions in an observed (O)/expected

(E) taxa index for the USA, results that justified different thresholds for each region. In such

circumstances, a uniform threshold may increase the frequency of errors in the more variable

settings. Reference sites with scores below a uniform threshold may be disproportionately

common in settings where the index is less precise. A variable threshold that is lower in more

variable settings may reduce this error rate (i.e., the reference error rate).

To determine if variable impairment thresholds based on site-specific characteristics

could lead to an unbiased distribution of errors across regions, we evaluated 2 approaches to

establishing thresholds: 1) a traditional approach, where a single number (based on variability in

scores at all reference calibration sites) was used as a threshold, and 2) a site-specific approach,

where thresholds were based on only a subset of the most environmentally similar reference

calibration sites. In both cases, we considered sites to be in reference condition if their index

score was >10th

percentile of the relevant set of reference calibration site values. We measured

environmental similarity as standard Euclidean distances along all environmental gradients used

in predictive models (Table 1). We evaluated several different sizes of reference-site subsets (25,

50, 75, 100, and 200, and the full set of 473). We calculated the error rate for all regions (except

for the Central Valley, which had only 1 reference site) as the proportion of sites with scores

below the threshold. We plotted these regional error rates against the number of reference sites

used to calculate the threshold (Fig. S1) and transformed scores at test sites into percentiles

relative to each of these distributions. We used the predictive California Stream Condition Index


(CSCI) and its null equivalent in this analysis.

Variable thresholds greatly reduced the regional bias of the error rate of the null index,

but had a negligible effect on the predictive index. For example, the null index had a very high

error rate (0.30) in the South Coast when a uniform threshold was used, but this error rate

dropped to 0.10 when variable thresholds based on 25 or 50 reference sites were used. In

contrast, the regional error rate of the predictive index was always <0.15 and was not highly

influenced by the number of reference sites used to establish thresholds.

We recommend a uniform threshold used in conjunction with a predictive index because

of the added complexity and minimal benefits provided by the variable, site-specific thresholds.

Literature cited

Death, R. G., and M. J. Winterbourn. 1994. Environmental stability and community persistence:

a multivariate perspective. Journal of the North American Benthological Society 13:125–

139.

Yuan, L. L., C. P. Hawkins, and J. Van Sickle. 2008. Effects of regionalization decisions on an

O/E index for the US national assessment. Journal of the North American Benthological

Society 27:892–905.


Fig. S1. Effects of nearest neighbor thresholds on error rates, calculated as the proportion of

reference calibration sites below the threshold for null (A) and predictive (B) indices. Each point

represents a different region. The highest number of reference sites is equivalent to the uniform

threshold used in the main study.


Appendix S2. Index responsiveness as a function of predicted % sensitive taxa: a comparison of

a predictive metric approach and the observed (O)/expected (E) taxa index.

The responsiveness of a bioassessment index depends on its ability to change in response

to stress, and the loss of sensitive taxa is typically one of the strongest responses to stress

(Rosenberg and Resh 1993, Statzner et al. 2004). To see if the ability to detect the loss of

sensitive taxa depends on number of common taxa (E), we compared the proportion of sensitive

taxa expected by an O/E index and a predictive multimetric index (pMMI) under different values

of E. For the pMMI, this proportion was calculated as the predicted % intolerant taxa metric, as

described in the accompanying manuscript. For the O/E, this proportion was calculated as the %

of expected operational taxonomic units (OTUs) that are sensitive (OTUs with tolerance value <

3. For OTUs consisting of multiple taxa with different tolerance values, we used the median

tolerance value). CAMLnet (2003) was the source of tolerance values. Estimates from both the

O/E and pMMI were plotted against E to see whether the 2 indices allowed consistent ranges of

response across values of E. These predictions were compared with the observed % intolerant

taxa at reference sites to confirm the validity of these estimates.

At high-E sites (E > 14), both the pMMI and O/E had a consistent capacity to detect loss

of sensitive taxa (Fig. S2A, C). Furthermore, both indices estimated similar proportions of

sensitive taxa (~40%), suggesting that the 2 indices have similar sensitivity in these settings.

Both indices also predicted a decline in the proportion of sensitive taxa at low-E sites, indicating

that E affects the sensitivity of the pMMI and O/E. However, at the lowest levels of E, the O/E

had no capacity to detect loss of sensitive taxa, whereas the pMMI predicted ~20% sensitive taxa

at these sites, preserving a limited capacity to respond to loss of sensitive taxa. This capacity

explains why the pMMI was more sensitive than the O/E at low-E sites.


Inspection of the data at reference sites indicates that sensitive taxa were truly present at

these low-E sites (Fig. S2B, D) and that modeling the metric directly sets more accurate

expectations for sensitive taxa in these settings (metric prediction vs observed R2 = 0.80; O/E

prediction vs observed R2 = 0.55). However, these taxa were excluded from the index because of

the minimum capture probability (i.e., 50%). Therefore, the predictive metric and not the O/E

will be able respond to the loss of sensitive taxa at low-E sites.

Literature Cited

CAMLnet. 2003. List of California macroinvertebrate taxa and standard taxonomic effort.

California Department of Fish and Game, Rancho Cordova, California. (Available from:

www.safit.org)

Rosenberg, D. M., and V. H. Resh. 1993. Freshwater biomonitoring and benthic

macroinvertebrates. Chapman and Hall, New York.

Statzner, B., S. Dolédec, and B. Hugueny. 2004. Biological trait composition of European stream

invertebrate communities: assessing the effects of various trait filter types. Ecography

27:470–788.


Fig. S2. Proportion of sensitive taxa predicted by a predictive multimetric index (pMMI) (A, B)

and an observed (O)/expected (E) taxa index (C, D) at all sites (A, C), or observed at reference

calibration (B, D) sites. Dark triangles represent sites with high (>15) numbers of expected taxa,

gray circles represent sites with moderate (10–15) numbers of expected taxa, and white squares

represent sites with low (<10) numbers of expected taxa. The solid line represents a smoothed fit

from a generalized additive model.


California Stream Condition Index Reference Sites Scientific Publication


Evaluating the adequacy of a reference-site poolfor ecological assessments in environmentallycomplex regions

Peter R. Ode1,7, Andrew C. Rehn1,8, Raphael D. Mazor1,2,9, Kenneth C. Schiff2,10, Eric D. Stein2,11,Jason T. May3,12, Larry R. Brown3,13, David B. Herbst4,14, David Gillett2,15, Kevin Lunde5,16,and Charles P. Hawkins6,17

1Aquatic Bioassessment Laboratory, California Department of Fish and Wildlife, 2005 Nimbus Road, Rancho Cordova, California95670 USA

2Southern California Coastal Water Research Project, 3535 Harbor Boulevard, Suite 110, Costa Mesa, California 92626 USA3US Geological Survey, 6000 J Street, Sacramento, California 95819 USA4Sierra Nevada Aquatic Research Laboratory, 1016 Mount Morrison Road, Mammoth Lakes, California 93546 USA5San Francisco Bay Regional Water Quality Control Board, 1515 Clay Street, Oakland, California 94612 USA6Western Center for Monitoring and Assessment of Freshwater Ecosystems, Department of Watershed Sciences,

and the Ecology Center, Utah State University, Logan, Utah 84322-5210 USA

Abstract: Many advances in the field of bioassessment have focused on approaches for objectively selecting thepool of reference sites used to establish expectations for healthy waterbodies, but little emphasis has beenplaced on ways to evaluate the suitability of the reference-site pool for its intended applications (e.g., compli-ance assessment vs ambient monitoring). These evaluations are critical because an inadequately evaluated ref-erence pool may bias assessments in some settings. We present an approach for evaluating the adequacy of areference-site pool for supporting biotic-index development in environmentally heterogeneous and pervasivelyaltered regions. We followed common approaches for selecting sites with low levels of anthropogenic stress toscreen 1985 candidate stream reaches to create a pool of 590 reference sites for assessing the biologicalintegrity of streams in California, USA. We assessed the resulting pool of reference sites against 2 performancecriteria. First, we evaluated how well the reference-site pool represented the range of natural gradients presentin the entire population of streams as estimated by sites sampled through probabilistic surveys. Second, we eval-uated the degree to which we were successful in rejecting sites influenced by anthropogenic stress by comparingbiological metric scores at reference sites with the most vs fewest potential sources of stress. Using this approach,we established a reference-site pool with low levels of human-associated stress and broad coverage of environmen-tal heterogeneity. This approach should be widely applicable and customizable to particular regional or program-matic needs.Key words: reference condition, bioassessment, environmental heterogeneity, performance measures, benthicmacroinvertebrates

Many of the refinements to biological monitoring tech-niques over the past 30 y have centered on strengtheningthe theoretical and practical basis for predicting the bio-logical expectation for a given location in the absence ofhuman-derived disturbance, i.e., the ‘reference state’ or‘reference condition’ (Hughes et al. 1986, Reynoldson et al.1997, Stoddard et al. 2006, reviewed by Bonada et al. 2006,Hawkins et al. 2010b, Dallas 2013). The need to anchorbiological expectations to a reference condition is now of-ten regarded as essential. However, discussion regarding

how to evaluate whether the properties of a pool of ref-erence sites are adequate for its intended uses has beenrare (Herlihy et al. 2008).

Authors of many recent treatments of the reference-site selection process recognize that objective criteria cangreatly enhance the consistency of reference-condition de-terminations (Whittier et al. 2007, Herlihy et al. 2008,Yates and Bailey 2010, Dobbie and Negus 2013, Lundeet al. 2013), and examples of objective site-selection pro-cesses are increasingly common (e.g., Hawkins et al. 2000,

E-mail addresses: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

DOI: 10.1086/684003. Received 4 November 2014; Accepted 3 February 2015; Published online 24 September 2015.Freshwater Science. 2016. 35(1):237–248. © 2016 by The Society for Freshwater Science. 237


Stoddard et al. 2006, Collier et al. 2007, Whittier et al.2007, Sánchez-Montoya et al. 2009, Yates and Bailey 2010).Several different approaches reflecting philosophical dif-ferences of practitioners and the varied monitoring ques-tions each program addresses exist for identifying refer-ence sites (e.g., Herlihy et al. 2008, Sánchez-Montoya et al.2009, Yates and Bailey 2010). Programs in which biologicalintegrity is assessed often call for a ‘minimally disturbed’or ‘least disturbed’ standard (sensu Stoddard et al. 2006)for selecting reference sites because truly pristine streamsare rare or nonexistent throughout the world. The mainchallenge is to choose site-selection criteria that retainsites with the highest biological integrity possible, therebymaintaining the philosophical ideal of the reference con-dition. However, geographic variability in the importanceof different stressors that affect biological condition cancomplicate the establishment of uniform reference defini-tions (Statzner et al. 2001, Herlihy et al. 2008, Mykrä et al.2008, Ode et al. 2008).

Robust reference-site selection involves balancing 2potentially conflicting goals: 1) reference criteria shouldselect sites that uniformly represent the least disturbedconditions throughout the region(s) of interest, minimiz-ing the effects of anthropogenic stress on the indicator ofinterest, and 2) reference sites should represent the fullrange of environmental settings in the region in suffi-cient numbers to adequately characterize natural vari-ability in the indicator(s) of interest. Restrictive criteria mayminimize anthropogenic stress within the reference poolat the expense of spatial or environmental representative-ness, particularly in regions with diverse environmentalsettings or pervasive alteration (Mapstone 2006, Osenberget al. 2006, Yuan et al. 2008, Dallas 2013, Feio et al. 2014).On the other hand, relaxing criteria to allow enough sitesin the reference-site pool weakens the ability to detect de-viation from the natural biological state. The consider-ation of environmental representativeness is especially crit-ical in regulatory applications where errors in estimatingsite-specific reference expectations may have significantfinancial and resource-protection consequences. Evaluat-ing the influence of the selected reference criteria on char-acteristics of the reference-site pool allows scientists andresource managers to make informed decisions about thisbalance.

We describe an approach used to evaluate the adequacyof a reference-site pool for assessing biological conditionof streams in California, an environmentally complex re-gion of the USA overlain with large areas of pervasive de-velopment. Our work is built on previous efforts to identifyreference conditions in similarly complex regions (e.g.,Collier et al. 2007, Herlihy et al. 2008, Sánchez-Montoyaet al. 2009, Falcone et al. 2010). We followed common ap-proaches to identify reference sites, then conducted an ex-tensive characterization of the pool of reference sites, withan emphasis on assessing how well the natural diversity at

streams in the region was represented by the reference-site pool and whether the biological integrity of the poolwas reduced when maximizing representativeness.

METHODSWe assembled a set of 1985 candidate reference sites

representing a wide range of stream types to supportdevelopment of screening criteria. We characterized eachsite with a suite of landuse and land-cover metrics thatquantified both its natural characteristics and potentialanthropogenic stressors at the site or in its upstreamdrainage area. We then screened sites with a subset oflanduse metrics (e.g., road density and % urban land usein the upstream watershed) based on thresholds that rep-resented low levels of anthropogenic activity (least dis-turbed sensu Stoddard et al. 2006). We evaluated the poolof reference sites that passed screening criteria to assesswhether the objectives of balancing naturalness and rep-resentativeness were achieved to a degree sufficient tosupport the development and defensible application of bi-ological scoring tools and condition thresholds (i.e., bio-criteria).

SettingCalifornia’s stream network is ∼280,000 km long, and

30% of the length is perennial according to the NationalHydrography Dataset (NHD) medium-resolution (1 ∶ 100k)stream hydrology data set (http://nhd.usgs.gov/data.html).These streams drain a large (424,000 km2) and remark-ably diverse landscape. California spans latitudes between33 and 42°N, and its geography is characterized by ex-treme natural gradients. It encompasses both the highestand lowest elevations in the conterminous US, temperaterainforests in the northwest, deserts in the northeast andsoutheast, and a Mediterranean climate in most remainingregions. California’s geology is also complex, with recentlyuplifted and poorly consolidated marine sediments in theCoast Ranges, alluvium in its broad internal valleys, gra-nitic batholiths along the eastern border and recent volcaniclithology in the northern mountains. The state’s environ-mental heterogeneity is associated with a high degree ofbiological diversity and endemism in the stream fauna(Erman 1996, Moyle and Randall 1996, Moyle et al. 1996).

California’s natural diversity is accompanied by anequally complex pattern of land use. The natural land-scapes of some regions of the state have been nearly com-pletely converted to agricultural or urban land uses (e.g.,the Central Valley and the South Coast) (Sleeter et al.2011). Other regions are still largely natural but containpockets of agricultural and urban land use and supporttimber harvest, livestock grazing, mining, and intensiverecreational uses. Our analyses generally treated environ-mental variation as continuous, but to facilitate some as-sessments, we divided the state into 6 regions, 4 of which

238 | Reference-site-network performance P. R. Ode et al.


were further divided into subregions (north coast, Cen-tral Valley, chaparral [coastal and interior], Sierra Ne-vada [western and central Lahontan], south coast [xericand mountains], deserts + Modoc) based on modified eco-regional (Omernik 1987) and hydrological boundaries(Fig. 1).

Aggregation of site dataWe inventoried >20 federal, state, and regional moni-

toring programs to assemble the data sets used forscreening reference sites. Candidate data sets were mostlyrestricted to wadeable, perennial streams, but some non-wadeable rivers were included, as were some nonpe-rennial streams, because of unavoidable imprecision in theassignment of flow status to stream reaches. All 1985unique sites (Fig. 1) were sampled between 1999 and 2010,and resulting data were compiled into a single database.We considered sites sampled within 300 m of one anotherto be replicates and used only the most recent sample.

We used physical habitat data to characterize gradientsrelated to natural (e.g., slope) and anthropogenic (e.g., ri-parian disturbance) factors (Tables S1, S2). All physicalhabitat data were collected with standard protocols fromthe US Environmental Protection Agency’s [EPA’s] Envi-ronmental Monitoring and Assessment Program protocol

(EMAP; Peck et al. 2006) or California’s modification ofEMAP protocols (Ode 2007). For calculation of reach-scale physical-habitat metrics, we followed Kaufmann et al.(1999).

Most benthic macroinvertebrate (BMI) data used toevaluate the extent of degradation within the reference-site data set were collected following the EMAP reach-wide protocol, but some older data were collected follow-ing the EMAP targeted-riffle protocol (Peck et al. 2006).Previous studies have shown these protocols produce sim-ilar bioassessments in the western USA (Ode et al. 2005,Gerth and Herlihy 2006, Herbst and Silldorff 2006, Rehnet al. 2007, Mazor et al. 2010). Prior to all analyses, BMIdata were converted to standard taxonomic effort levels(generally genus-level identifications except chironomidmidges were identified to subfamily; see Richards and Rog-ers 2006) and subsampled when necessary to 500-count.

Combination of probability data setsWe used data from a subset of sites (919 of 1985 sites)

that were sampled under probabilistic survey designs toevaluate whether our final pool of reference sites ade-quately represented the full range of natural stream set-tings in California. Probability data sets provide objectivestatistical estimates of the true distribution of populationparameters (in this case, natural characteristics of Califor-nia’s perennial stream network) (Stevens and Olsen 2004).First, we created a common sample frame such that therelative contribution of each site to the overall distributionof stream length (the site’s weight) could be calculated inthe combined data set. All probabilistic sites were regis-tered to a uniform stream network (NHD Plus version 1;http://www.horizon-systems.com/nhdplus/) and attributedwith strata defined by the design parameters of all inte-grated programs (e.g., land use, stream order, survey bound-aries, etc.). Second, we calculated site weights for each siteby dividing total stream length in each stratum (e.g., all 2nd-order streams draining agricultural areas in the north coastregion) by the number of sampled sites in that stratum us-ing the spsurvey package (Kincaid and Olsen 2009) in R(version 2.11.1; R Project for Statistical Computing, Vienna,Austria).

Geographical information system (GIS)data and metric calculation

We assembled a large number of spatial data sourcesto characterize natural and anthropogenic gradients thatmay affect biological condition at each site, e.g., land coverand land use, road density, hydrologic alteration, mining,geology, elevation, and climate (Table S1). We evaluateddata sets for statewide consistency and excluded layerswith poor or variable reliability. All spatial data sourceswere publicly available except for the roads layer, whichwas customized for this project by appending unimproved

Figure 1. Distribution of 1985 candidate sites screened forinclusion in California’s reference pool. White circles representpassing sites and black circles represent sites that failed ≥1screening criteria. Thick solid lines indicate boundaries of ma-jor ecological regions referred to in the text. Lighter dashedlines indicate subregional boundaries.



and logging road coverages obtained from the US ForestService and California Department of Forestry and FireProtection to a base roads layer (ESRI 2009).

We converted land cover, land use, and other measuresof human activity into metrics (Table S2) expressed withinthe entire upstream drainage area of the site (watershed),within 5 km upstream (5k) and within 1 km upstream (1k).We created polygons defining these spatial units withArcGIS tools (ESRI 2009). Local polygons were created byintersecting a 5-km- or 1-km-radius circle centered at thestream site with the primary watershed polygon. We calcu-lated metrics associated with sampling location (e.g., meanannual temperature, elevation, NHD+ attributes, etc.), basedon each site’s latitude and longitude. Data for all screeningvariables were available for all sites, except for W1_HALL(Kaufmann et al. 1999), a field-based quantitative measureof anthropogenic stressors at the reach scale that was avail-able for ∼½ of the sites (Table S2). We also calculated pre-dicted electrical conductivity (Olson and Hawkins 2012), asite-specific estimate of natural background conductivitybased on modeled relationships between observed conduc-tivity and a suite of natural geographic, geological, climatic,and atmospheric variables (Table S2).

Selection of stressor screening variables and thresholdsTo maximize the naturalness of the reference-site pool,

we eliminated sites that exceeded specific thresholds forhuman activity (Table 1). Failure of any one screen wassufficient to eliminate a candidate site from the reference

pool. Strict initial screening criteria (human influencevariables set at 0) resulted in a set of only ∼100 referencesites that occurred almost exclusively in mountainous re-gions (Sierra Nevada and North Coast Mountains) and thatpoorly represented most streams in California. We then re-laxed thresholds after consulting reports from prior ref-erence development projects (Ode et al. 2005, Rehn et al.2005, Stoddard et al. 2006, Rehn 2009) and the literature(e.g., Collier et al. 2007, Angradi et al. 2009, Falcone et al.2010) for previously established criteria, except for specificconductance. For specific conductance, we rejected siteswhose observed conductance values fell outside the 1 and99% prediction interval. If the prediction was >1000 μS/cm,we used a fixed rejection threshold of >2000 μS/cm. Thenew goal for the screening criteria was maximum repre-sentativeness in all regions of the state with the least re-laxation of human-influence criteria possible.

Sensitivity of site counts to different screening thresh-olds The relative dominance of different stressors andtheir contribution to overall disturbance at candidate sitesvary regionally. To explore the effect of threshold ad-justments on site counts in different regions, we adjustedthresholds for each primary metric individually while allothers were held constant and plotted the number ofpassing sites (i.e., threshold sensitivity) for each region.These partial-dependence curves were used to evaluatethe number of reference sites potentially gained by relaxingthresholds for each screening metric in each region (see

Table 1. Distribution of reference and nonreference sites (number [n] and %) failing different numbers of thresholds (each screen andspatial-scale combination is counted independently). Number of streams and extent of stream length estimated to be reference byregion (% ref ± 1 SE) based on probability data only.

RegionTotal streamlength (km)

Reference Nonreference

% ofnonreferencesites failing(thresholds)

n % of sites % of stream length SE n % of sites 1–2 3–5 ≥5

North Coast 9278 76 31 26 3 168 69 26 57 18

Chaparral 8126 93 22 19 4 334 78 44 17 39

Coastal Chaparral 5495 61 18 14 5 275 82 47 16 37

Interior Chaparral 2631 32 35 28 6 59 65 34 22 44

South Coast 2945 119 18 23 4 555 82 22 10 68

South Coast Mountains 1123 86 42 53 7 121 58 62 23 15

South Coast Xeric 1821 33 7 3 1 434 93 11 6 83

Central Valley 2407 1 1 2 2 69 99 1 7 91

Sierra Nevada 11,313 276 56 43 5 218 44 56 26 18

Western Sierra Nevada 8577 131 53 34 6 118 47 58 29 14

Central Lahontan 2736 145 59 76 5 100 41 54 23 23

Deserts + Modoc 2531 25 33 32 10 51 67 51 29 20

Total 36,599 590 30 29 2 1395 70 33 20 47



Hill et al. 2013 for a similar example). We distinguishedstressor variables with thresholds whose adjustment had alarge influence on the number of accepted reference sites(and therefore, might improve overall environmental rep-resentativeness of the reference pool) from variables whosethreshold adjustment had little influence on final site num-bers. In the latter case, thresholds could be kept strict tominimize the risk of compromising biological integrity.

Performance measuresEvaluation of reference-site-pool representativeness Weevaluated 2 aspects of representativeness: 1) the numberof reference sites identified statewide and within majorregions of California (i.e., adequacy, Diamond et al. 2012),and 2) the degree to which those reference sites repre-sented the range of natural variability in physical andchemical gradients associated with California streams (i.e.,environmental representativeness). We compared the dis-persion of reference sites to the distribution of naturalgradients in each region (as estimated from our probabil-ity distributions) and in multivariate environmental spacedescribed by a principal components analysis (PCA)-basedordination. We used 11 natural gradients that are associ-ated with benthic invertebrate composition in Californiastreams (Mazor et al. 2016; listed in Table S2) in the PCAanalysis.

Evaluation of anthropogenic stress in the reference-sitepool All thresholds allowed at least some degree of up-stream human activity, so we determined if these stresslevels were biologically important by assessing the re-sponsiveness of a set of common BMI metrics to differ-ent stress-related variables. We used t-tests to determineif means of BMI metrics at a subset of sites passing morestringent screens were different from those at sites pass-ing only ‘standard’ screens (Table 2). The more stringentscreens were: <1% nonnatural land use (agricultural, urban,or Code 21 [a development-associated vegetation class inthe NLCD data set that corresponds to lawns and recrea-tional grasses in urban areas and roadside vegetation inrural and exurban areas]) at all spatial scales; road density<1 km/km2 for all spatial scales; W1_HALL < 0.5; and allother criteria as listed in Table 2.

BMI metric values indicative of healthy biological con-dition vary naturally in different environmental settings.Natural variation could reduce our ability to discern bio-logically meaningful differences between stringent andless stringent reference groups. To correct for this poten-tial confounding influence and to apply a more conserva-tive test of the null hypothesis (no difference betweengroups), residuals from random-forest models of metricresponse to natural gradients were used in t-tests insteadof raw metrics. Lack of significant differences in residualsbetween the high and low threshold groups was taken as

evidence that biological integrity was not sacrificed by theless strict thresholds.

RESULTSReference status by region

Of the 1985 sites evaluated for potential use as refer-ence sites, 590 passed all screening thresholds (Table 1).The number of reference sites varied by region, with thehighest concentrations in mountainous regions (e.g., theSierra Nevada, North Coast, and South Coast Mountains).Lower elevation, drier subregions had fewer referencesites (South Coast Xeric = 33, Interior Chaparral = 32),and only a single reference site was identified in the Cen-tral Valley (near the boundary of the Interior Chaparral).

Based on probability survey data, 29 ± 2% (SE) of Cal-ifornia’s stream length was estimated to meet our refer-ence criteria (Table 1). Streams that met reference criteriawere most extensive in mountainous regions, contribut-ing ∼76 and 53% of the stream length in the Central

Table 2. Thresholds used to select reference sites. Scale refersto spatial area of analysis (WS = upstream watershed, 1k =watershed area within 1 k of site, 5k = watershed area within5 k of site). NA = not applicable, W1_HALL = Index of humandisturbance.

Variable Scale Threshold Unit

% agriculture 1k, 5k, WS 3 %

% urban 1k, 5k, WS 3 %

% agriculture +% urban 1k, 5k, WS 5 %

% Code 21 1k, 5k 7 %

WS 10 %

Road density 1k, 5k, WS 2 km/km2

Road crossings 1k 5 crossings

5k 10 crossings

WS 50 crossings

Dam distance WS 10 km

% canalsand pipelines WS 10 %

In-streamgravel mines 5k 0.1 mines/km

Producer mines 5k 0 mines

Specificconductance site 99/1a Prediction interval

W1_HALL site 1.5 _

a The 99th and 1st percentiles of predictions were used to generatesite-specific thresholds for specific conductance. Because the predictedconductivity model (Olson and Hawkins 2012) was observed tounderpredict at higher levels of specific conductance (data not shown),a threshold of 2000 μS/cm was used as an upper bound if theprediction interval included 1000 μS/cm.



Lahontan and South Coast Mountain subregions, respec-tively. Only 2 to 3% of stream length in the Central Valleyand the South Coast Xeric regions was estimated to meetreference criteria, whereas 43 and 32% of the Sierra Ne-vada and Deserts + Modoc stream length met our refer-ence criteria, respectively. Despite the large number of ref-erence sites in the North Coast (n = 76) and Western Sierra(n = 131) regions, relatively limited extents of stream lengthmet reference criteria in these regions (26 and 34% of streamlength, respectively). These levels are similar to levels seenin Chaparral regions, suggesting that the abundance of ref-erence sites in some regions is caused more by the large ex-tent of perennial streams than lack of anthropogenic stress-ors in the region.

Sensitivity of site counts to threshold levelsLarge regional differences were present in the number

and types of stressor metrics that contributed to the re-moval of candidate sites from the reference pool (Table 1).For example, most nonreference sites in the Sierra Nevada,South Coast Mountains, Chaparral, and Desert + Modocregions failed only 1 or 2 thresholds (typically road densityand Code 21), but a large majority (i.e., >80%) of non-reference sites in the Central Valley and the South CoastXeric regions failed ≥5 thresholds. For other regions, thepercentage of streams failing 1 or 2 thresholds ranged from26 to 47%.

Similar regional patterns were reflected in thresholdsensitivity plots (Fig. 2A–D; all example metrics werecalculated at the watershed scale). For example, adjustingthresholds for the landuse metrics, % agricultural and %urban (Fig. 2A, D), had little influence on the proportionof sites that passed reference screens in most regions,indicating that other screening thresholds were limiting.In contrast, even a modest increase of the threshold forCode 21 greatly increased the number of passing sites inmost regions, especially in the North Coast, Chaparral,and South Coast (Fig. 2B). Threshold adjustments for roaddensity had similarly large impacts in the North Coast,Chaparral, and Desert + Modoc regions (Fig. 2C). Thissensitivity allowed us to selectively relax screening thresh-olds for road density and Code 21, thereby increasing thenumber of passing sites and improving representation inseveral regions, particularly in the Chaparral, a region withrelatively few sites prior to the adjustment. We would havehad to adjust many other metric thresholds concurrently toachieve a comparable result had we not identified this pat-tern of differential threshold sensitivity.

Reference-site representativenessThe large number of sites in our data set that came

from probabilistic surveys (n = 919) allowed us to pro-duce well resolved distribution curves for a suite of natu-ral environmental factors in each region (Fig. 3A–F). For

nearly all natural factors and regions examined, the dis-persion of reference-site values along environmental gra-dients matched the overall distribution of values for thesegradients well. However, small but potentially importantgaps were evident. For example, streams with very largewatersheds (i.e., >500 km2; Fig. 3A) and very high-elevationstreams (i.e., >2500 m; Fig. 3B) and were represented byonly a few reference sites. Most of the other gaps wereassociated with a class of streams that represented the tailsof distributions for several related environmental variables(low-elevation, low-gradient, low-precipitation, large water-sheds; Fig. 3B, D, E).

PCA of environmental variables provided additional evi-dence that the reference-site pool represented natural en-vironmental gradients well (Fig. 4). Gaps generally wererestricted to the extremes of gradients. For example, in a2-dimensional plot of PCA Axes 1 and 2, a cluster of sitesthat lacked reference-site coverage was evident at the rightside of PCA Axis 1 (Fig. 4), which corresponds to portionsof the Central Valley and a group of low-gradient, low-elevation, low-precipitation, and large watershed streamsin southern coastal California. Other axis combinations in-dicated similarly good coverage of natural environmentalgradients and identified similar gaps.

Figure 2. Example threshold sensitivity (partial dependence)curves showing the relationship between proportion of poten-tial reference sites and thresholds for % agricultural (A), %Code 21 (B), road density (C), and % urban (D). All otherstressors were held constant using the thresholds listed in Ta-ble 2. Vertical lines indicate reference thresholds for eachmetric.



Biological response to stressorsBMI metric scores at reference sites that passed the

most stringent screening criteria (n = 294) were indistin-guishable from scores at those reference sites that passedmore relaxed standard screens (Fig. 5) and were clearly

different from scores at nonreference sites. All t-tests fordifferences in mean BMI metric scores between the 2 setsof reference sites were not significant (Fig. 5), indicatingthat we did not sacrifice biological integrity to achieveadequate representation of natural gradients.

Figure 3. Comparison of reference-site representation along biologically influential natural gradients of watershed area (A), siteelevation (B), conductivity (C), % reach slope (D), annual precipitation (E), and CaO geology in the watershed (F). Full distributions ofnatural gradients estimated from probabilistic sampling surveys within major regions of California are shown as kernel densityestimates. Values of individual reference sites are shown as small vertical lines. Regions (see Fig. 1) are abbreviated as: SN = SierraNevada, SC = South Coast, NC = North Coast, DM = Deserts + Modoc, CV = Central Valley, CH = Chaparral.



DISCUSSIONRigorous consideration of reference concepts can en-

hance multiple components of watershed-managementprograms, including development and application of bio-

logical (e.g., indices of biotic integrity) and nonbiological(e.g., streambed substrate composition) endpoints. To en-sure optimal use of reference-condition-based tools, pro-gram personnel need to evaluate whether selection criteria

Figure 4. Principal components analysis (PCA) ordination of 1985 sites based on natural environmental characteristics (geologyand climate variables listed in Table S2), showing the 2 primary principal component axes. Larger outlined circles indicate referencesites and smaller dots indicate nonreference sites. Colors represent regions shown in Fig. 1. The inset depicts vectors of selectednatural variables as estimated from correlation with the PCA axes.

Figure 5. Boxplots comparing benthic macroinvertebrate (BMI) metric scores at a subset of reference sites that passed very strictscreens (n = 292) and reference sites that passed less strict screens (n = 298). Significant differences (p < 0.05) were not observed forany comparison of the reference groups. Boxplots of nonreference, stressed sites (n = 613) are included for visual comparison. Linesin boxes are medians, box ends are quartiles, whiskers represent 1.5× the interquartile distance, and dots are outliers. EPT =Ephemeroptera, Plecoptera, Trichoptera.



produce a pool of reference sites suited to intended uses(Bailey et al. 2004, 2012, Herlihy et al. 2008). Our selectionprocess yielded 590 unique reference sites that, except forthe Central Valley, represented nearly the full range of allnatural gradients evaluated. Thus, we have confidence thatanalyses and assessment tools developed from this pool ofreference sites will be representative for most perennialstreams in California. Our thresholds did not eliminate allanthropogenic disturbances from the reference-site pool,but the influence of these disturbances on the reference-sitefauna was minimal, and the balance we achieved betweenenvironmental representativeness and biological integrityis sufficient to support robust regulatory applications forwadeable perennial streams in California. Furthermore,although we anticipated that we would need to make re-gional adjustments in either the choice of stressors or spe-cific thresholds used for screening reference sites, we wereable to achieve adequate reference-condition representationfor most regions of the state with a common set of stressorsand thresholds, thereby maintaining interregional compa-rability. Thus, no need exists for region-specific thresholdadjustments, and the complications they create for man-agement interpretation can be avoided (see Herlihy et al.2008, Yuan et al. 2008).

In the terminology of Stoddard et al. (2006), ourreference-site pool was initially identified based on a leastdisturbed definition, but the sites probably are minimallydisturbed given the limited response to watershed alter-ation in BMI metrics. The selective and systematic relax-ation of reference screens allowed us to achieve broadrepresentation of most perennial, wadeable streams in Cal-ifornia with a single set of statewide reference criteria.

Applications of the reference-condition approachA robust reference-site pool is needed to achieve several

stream and watershed management objectives. Reference-condition concepts provide defensible regulatory frame-works for protecting and managing aquatic resources anda consistent basis for combining multiple biological indi-cators (e.g., algal, fish, and BMI assemblages) in integratedassessments. The process of defining reference criteria alsocan be used to help identify candidate streams and water-sheds deserving of special protections and application ofantidegradation policies, which are often under-applied inthe USA and elsewhere (Collier 2011, Linke et al. 2011).

The reference-condition approach also is potentiallyuseful in: 1) establishing objective regulatory thresholds fornonbiological indicators and 2) providing context for in-terpreting targeted and probabilistic nonbiological monitor-ing data. Establishment of regulatory standards for water-quality constituents that vary naturally in space and time(e.g., nutrients, Cl–, conductivity, and fine sediment) canbe arbitrary and contentious, especially compared to theprocess for establishing objectives for manufactured pol-

lutants, like pesticides. The range of concentrations oc-curring at reference sites could be used to guide criteriadevelopment for physical and chemical pollutants withnon-0 expectations (Hawkins et al. 2010a, b, Yates andBailey 2010, Vander Laan et al. 2013). The physiochemicalconditions found at reference sites can be used to predictthe condition of test sites in a natural state (e.g., VanderLaan et al. 2013). Furthermore, the range of values ex-pected in the natural reference state can give managementprogram personnel the perspective needed to distinguishrelatively small differences in pollutant concentration fromenvironmentally meaningful differences. Ultimately, thebroad success of these nonbiological applications will de-pend on rigorous evaluations of the reference data set,just as they do for biological applications of the referenceconcept.

Limits of this analysisAt least 3 types of data limitations can influence the

adequacy of a reference-site pool: 1) inadequate or inac-curate GIS layers, 2) limited or imprecise informationabout reach-scale stressors, and 3) inadequate or unevensampling effort. Improvements in the availability and ac-curacy of spatial data over the last 2 decades have greatlyenhanced our ability to apply consistent screening crite-ria across large areas, but reliance on these screens canunderestimate the amount of biological impairment thatactually exists at a site (Herlihy et al. 2008, Yates and Bailey2010). The most accurate and uniformly available spatialdata tend to be associated with urban stressors (e.g., landcover, roads, hydrologic alteration), but estimates of rec-reation, livestock grazing, timber harvest, mining, and theirprobable effects on biota typically are under- and morevariably estimated (Herbst et al. 2011). Other potentialstressors, such as climate change and aerial deposition ofnutrients or pollutants, are even more challenging to quan-tify and use to screen reference sites. Reach-scale (prox-imate) alterations can have a large influence on aquaticassemblages (e.g., Waite et al. 2000, Munn et al. 2009), butare difficult to assess unless adequate quantitative dataare collected along with biological samples. We includedreach-scale anthropogenic disturbance data (W1_HALL)in our screens when available (∼50% of sites), but we un-doubtedly missed disturbance at sites where reach-scaledata were lacking. Unintentional inclusion of stressed sitesprobably affected biota in our reference-site pool, but weanticipate these effects can be reduced over time as avail-ability and quality of stressor data sets improve.

Highly heterogeneous regions like California are likelyto contain unique environmental settings (Erman 1996,Moyle and Randall 1996) that are infrequently sampledand might not be included in reference-site screeningunless intentionally targeted. For example, we added ad-ditional reference sites with naturally high conductivity



when we identified a lack of sites at the high end of thisgradient. We attempted to include as much environmen-tal diversity as possible, but some stream types withunique physical or chemical characteristics probably wereunder-sampled (e.g., mountain streams >2500 m asl). How-ever, our framework provides a means of explicitly testingthe degree to which such stream types are represented bythe overall pool. Applicability of existing assessment toolsto sites in these gaps may require further investigation,and additional targeted sampling (e.g., in high-elevationheadwater streams) is likely to yield needed data. In con-trast, some data gaps occur in pervasively disturbed re-gions (e.g., the Central Valley) that are unlikely to yieldadditional sites.

We used objective reference criteria based largely onGIS-measured variables, but the approach we used forevaluating performance of the reference-site pool couldbe applied to other strategies for selecting reference sites,such as one that emphasizes field-measured criteria (e.g.,Herlihy et al. 2008), or best professional judgment (e.g.,Lunde et al. 2013). The approach outlined in our paperis general and can be used to evaluate the suitability of areference-site pool for a wide range of habitat types, in-cluding nonperennial streams, lakes, depressional wet-lands, and estuaries (e.g., Solek et al. 2010). For appli-cations where different reference criteria are applied todifferent regions or stream types (e.g., Herlihy et al. 2008,Yuan et al. 2008, Yates and Bailey 2010), these analysesprovide essential context for performing multiregion com-parisons.

ConclusionsIncreased attention has been paid in recent years to the

importance of quantifying the performance of variouscomponents of bioassessment (Diamond et al. 1996, 2012,Cao and Hawkins 2011), particularly as they relate to com-parability among data sets. This attention to performancevalidation is likely to facilitate the adoption of biologicalendpoints in water-quality programs worldwide. Similarattention to measuring the performance of reference-sitepools relative to their intended uses also will be of signifi-cant benefit. In particular, explicit attention to environ-mental representativeness should help improve overall ac-curacy of condition assessments and reduce predictionbias (see Hawkins et al. 2010b) in all reference-conditionapplications.

ACKNOWLEDGEMENTSThe analyses reported here were developed in support of

California’s biological assessment program and serve as thefoundation of the state’s regulatory biological criteria. Financialsupport for this effort was provided by grants from the US EPARegion IX and the California State Water Resources Control

Board’s Surface Water Ambient Monitoring Program. The de-velopment process was supported by stakeholder and regulatorydevelopment advisory groups, whose contributions strongly in-fluenced the objectives and approach we used. We are especiallygrateful to the considerable contributions of members of our sci-entific advisory panel who provided constructive guidance through-out the project: Dave Buchwalter, Rick Hafele, Chris Konrad,LeRoy Poff, John Van Sickle, and Lester Yuan. We further thankJohn Van Sickle for his valuable editorial contributions and Jo-seph Furnish for constructive advice throughout the process. Wethank Tony Olsen, Tom Kincaid, and Kerry Ritter for help com-bining the multiple probability surveys. We also thank ChristophMatthaei and 2 anonymous referees for helpful comments thatimproved this manuscript.

This work would not have been possible without 10 y ofeffort by field crews, taxonomists, and program staff from mul-tiple state and federal monitoring programs, including the USForest Service, US EPA, California Department of Fish and Wild-life, State and Regional Water Resources Control Boards, andthe Stormwater Monitoring Coalition.

LITERATURE CITEDAbramson, M. 2009. Tracking the invasion of the New Zealand

mudsnail, Potamopyrgus antipodarum, in the Santa MonicaMountains. Urban Coast 1:21–27.

Angradi, T. R., M. S. Pearson, T. M. Jicha, D. L. Taylor, D. W.Bolgrien, M. F. Moffett, K. A. Blocksom, and B. H. Hill. 2009.Using stressor gradients to determine reference expectationsfor great river fish assemblages. Ecological Indicators 9:748–764.

Bailey, R. C., R. H. Norris, and T. B. Reynoldson. 2004. Bio-assessment of freshwater ecosystems: using the referencecondition approach. Kluwer Academic Publishers, Boston,Massachusetts.

Bailey, R. C., G. Scrimgeour, D. Coté, D. Kehler, S. Linke, andY. Cao. 2012. Bioassessment of stream ecosystems enduringa decade of simulated degradation: lessons for the real world.Canadian Journal of Fisheries and Aquatic Sciences 69:784–796.

Baker, M. E., M. J. Wiley, and P. W. Seelbach. 2001. GIS-basedhydrologic modelling of riparian areas: implications for streamwater quality. Journal of the American Water Resources Asso-ciation 37:1615–1628.

Bonada, N., N. Prat, V. H. Resh, and B. Statzner. 2006. Devel-opments in aquatic insect biomonitoring: a comparative anal-ysis of recent approaches. Annual Review of Entomology 51:495–523.

Cao, Y., and C. P. Hawkins. 2011. The comparability of bio-assessments: a review of conceptual and methodological is-sues. Journal of the North American Benthological Society30:680–701.

Collier, K. J. 2011. The rapid rise of streams and rivers in con-servation assessment. Aquatic Conservation: Marine andFreshwater Ecosystems 21:397–400.

Collier, K. J., A. Haigh, and J. Kelly. 2007. Coupling GIS andmultivariate approaches to reference site selection for wade-able stream monitoring. Environmental Monitoring and As-sessment 127:29–45.



http://www.journals.uchicago.edu/action/showLinks?crossref=10.1002%2Faqc.1196

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1002%2Faqc.1196

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1007%2Fs10661-006-9256-z

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1007%2Fs10661-006-9256-z

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1016%2Fj.ecolind.2008.09.007

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1111%2Fj.1752-1688.2001.tb03664.x


http://www.journals.uchicago.edu/action/showLinks?crossref=10.1139%2Ff2012-010

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1146%2Fannurev.ento.51.110104.151124

http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F10-067.1

Dallas, H. 2013. Ecological status assessment in Mediterraneanrivers: complexities and challenges in developing tools forassessing ecological status and defining reference conditions.Hydrobiologia 719:483–507.

Diamond, J., M. Barbour, J. B. Stribling. 1996. Characterizingand comparing bioassessment methods and their results: aperspective. Journal of the North American BenthologicalSociety 15:713–727.

Diamond, J., J. B. Stribling, L. Huff, and J. Gilliam. 2012. Anapproach for determining bioassessment performance andcomparability. Environmental Monitoring and Assessment184:2247–2260.

Dobbie, M. J., and P. Negus. 2013. Addressing statistical andoperational challenges in designing large-scale stream con-dition surveys. Environmental Monitoring and Assessment185:7231–7243.

Erman, N. 1996. Status of aquatic invertebrates. Pages 987–1008 in Sierra Nevada Ecosystem Project: final report toCongress, Vol. II. Centers for Water and Wildland Resources,University of California Davis, Davis, California.

ESRI (Environmental Systems Research Institute). 2009. Businessanalyst. Environmental Systems Research Institute, Redlands,California. (Available from: http://www.esri.com/software/businessanalyst)

Falcone, J. A., D. M. Carlisle, and L. C. Weber. 2010. Quantify-ing human disturbance in watersheds: variable selection andperformance of a GIS-based disturbance index for predict-ing the biological condition of perennial streams. EcologicalIndicators 10:264–273.

Feio, M. J., F. C. Aguiar, S. F. P. Almeida, J. Ferreira, M. T.Ferreira, C. Elias, S. R. S. Serra, A. Buffagni, J. Cambra,C. Chauvin, F. Delmas, G. Dörflinger, S. Erba, N. For,M. Ferréol, M. Germ, L. Mancini, P. Manolaki, S. Marcheg-giani, M. R. Minciardi, A. Munné, E. Papastergiadou, N. Prat,C. Puccinelli, J. Rosebery, S. Sabater, S. Ciadamidaro,E. Tornés, I. Tziortzis, G. Urbanic, and C. Vieira. 2014. Leastdisturbed condition for European Mediterranean rivers. Sci-ence for the Total Environment 476/477:745–756.

Gerth, W. J., and A. T. Herlihy. 2006. The effect of samplingdifferent habitat types in regional macroinvertebrate bio-assessment surveys. Journal of the North American Benthol-ogical Society 25:501–512.

Hawkins, C. P., Y. Cao, and B. Roper. 2010a. Method of pre-dicting reference condition biota affects the performanceand interpretation of ecological indices. Freshwater Biology55: 1066–1085.

Hawkins, C. P., R. H. Norris, J. Gerritsen, R. M. Hughes, S. K.Jackson, R. H. Johnson, and R. J. Stevenson. 2000. Evalua-tion of landscape classifications for biological assessment offreshwater ecosystems: synthesis and recommendations. Jour-nal of the North American Benthological Society 19:541–556.

Hawkins, C. P., J. R. Olson, and R. A. Hill. 2010b. The refer-ence condition: predicting benchmarks for ecological water-quality assessments. Journal of the North American Ben-thological Society 29:312–343.

Herbst, D. B., M. T., Bogan, S. K. Roll, and H. D. Safford. 2011.Effects of livestock exclusion on in-stream habitat and ben-thic invertebrate assemblages in montane streams. Freshwa-ter Biology 57:204–217.

Herbst, D. B., and E. L. Silldorff. 2006. Comparison of the per-formance of different bioassessment methods: similar evalua-tions of biotic integrity from separate programs and proce-dures. Journal of the North American Benthological Society25:513–530.

Herlihy, A. T., S. G. Paulsen, J. Van Sickle, J. L. Stoddard, C. P.Hawkins, and L. Yuan. 2008. Striving for consistency in anational assessment: the challenges of applying a referencecondition approach on a continental scale. Journal of theNorth American Benthological Society 27:860–877.

Hill, R. A., C. P. Hawkins and D. M. Carlisle. 2013. Predictingthermal reference conditions for USA streams and rivers.Freshwater Science 32:39–55.

Hughes, R. M., D. P. Larsen, and J. M. Omernik. 1986. Regionalreference sites: a method for assessing stream potentials.Environmental Management 10:629–635.

Kaufmann, P. R., P. Levine, E. G. Robinson, C. Seeliger, andD. V. Peck. 1999. Quantifying physical habitat in wadeablestreams. EPA/620/R-99/003. Research Ecology Branch, USEnvironmental Protection Agency, Corvallis, Oregon.

Kincaid, T., and T. Olsen. 2009. SPSurvey package for R. Na-tional Health and Environmental Effects Research Labora-tory, Western Ecology Division, US Environmental ProtectionAgency, Corvallis, Oregon. (Available from http://www.epa.gov/nheerl/arm)

Linke, S., E. Turak, and J. Neale. 2011. Freshwater conservationplanning: the case for systematic approaches. Freshwater Bi-ology 56:6–20.

Lunde, K. B., M. R. Cover, R. D. Mazor, C. A. Sommers, andV. A. Resh. 2013. Identifying reference conditions and quan-tifying biological variability within benthic macroinvertebratecommunities in perennial and non-perennial Northern Cal-ifornia streams. Environmental Management 51:1262–1273.

Mapstone, B. D. 2006. Scalable decision criteria for environ-mental impact assessment: effect size, Type I, and Type IIerrors. Pages 67–82 in R. J. Schmitt and C. W. Osenberg(editors). Detecting ecological impacts. Academic Press, NewYork.

Mazor, R. D., K. Schiff, K. Ritter, A. C. Rehn, and P. R. Ode. 2010.Bioassessment tools in novel habitats: an evaluation of indicesand sampling methods in low-gradient streams in California.Environmental Monitoring and Assessment 167:91–104.

Mazor, R. D., A. C. Rehn, P. R. Ode, M. Engeln, K. Schiff, E. D.Stein, D. J. Gillett, D. B. Herbst, and C. P. Hawkins. 2016.Bioassessment in complex environments: designing an indexfor consistent meaning in different settings. Freshwater Sci-ence 35:249–271.

Meinshausen, N. 2006. Quantile regression forests. Journal ofMachine Learning Research 7:983–999.

Moyle, P. B., and P. J. Randall. 1996. Biotic integrity of water-sheds. Pages 975–986 in Sierra Nevada Ecosystem Project:final report to Congress, Vol. II. Centers for Water and Wild-land Resources, University of California Davis, Davis, Cali-fornia.

Moyle, P. B., R. M. Yoshiyama, and R. A. Knapp. 1996. Statusof fish and fisheries. Pages 953–974 in Sierra Nevada Eco-system Project: final report to Congress, Vol. II. Centers forWater and Wildland Resources, University of CaliforniaDavis, Davis, California.





http://www.journals.uchicago.edu/action/showLinks?crossref=10.1007%2Fs10661-009-1033-3



http://www.journals.uchicago.edu/action/showLinks?crossref=10.1016%2Fj.scitotenv.2013.05.056


http://www.journals.uchicago.edu/action/showLinks?crossref=10.1016%2Fj.scitotenv.2013.05.056

http://www.journals.uchicago.edu/action/showLinks?system-d=10.1086%2F684130

http://www.journals.uchicago.edu/action/showLinks?system-d=10.1086%2F684130

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1007%2FBF01866767

http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282006%2925%5B501%3AEOSDHT%5D2.0.CO%3B2


http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282006%2925%5B501%3AEOSDHT%5D2.0.CO%3B2

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1111%2Fj.1365-2427.2009.02357.x

http://www.journals.uchicago.edu/action/showLinks?system=10.2307%2F1467818













http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282006%2925%5B513%3ACOTPOD%5D2.0.CO%3B2

Munn, M. D., I. R. Waite, D. P. Larsen, and A. T. Herlihy. 2009.The relative influence of geographic location and reach-scalehabitat on benthic invertebrate assemblages in six ecoregions.Environmental Monitoring and Assessment 154:1–14.

Mykrä, H., J. Aroviita, J. Kotanen, H. Hämäläinen, and T.Muotka. 2008. Predicting the stream macroinvertebrate faunaacross regional scales: influence of geographical extent onmodel performance. Journal of the North American Ben-thological Society 27:705–716.

Ode, P. R. 2007. Standard operating procedures for collectingbenthic macroinvertebrate samples and associated physicaland chemical data for ambient bioassessment in California.Surface Water Ambient Monitoring Program, Sacramento,California. (Available from http://www.swrcb.ca.gov/water_issues/programs/swamp/docs/swamp_sop_bio.pdf)

Ode, P. R., C. P. Hawkins, and R. D. Mazor. 2008. Comparabilityof biological assessments derived from predictive models andmultimetric indices of increasing geographic scope. Journal ofthe North American Benthological Society 27:967–985.

Ode, P. R., A. C. Rehn, and J. T. May. 2005. A quantitative toolfor assessing the integrity of Southern California coastalstreams. Environmental Management 35:493–504.

Olson, J. R., and C. P. Hawkins. 2012. Predicting natural base-flow stream water chemistry in the western United States.Water Resources Research 48:W02504.

Omernik, J. M. 1987. Ecoregions of the conterminous UnitedStates. Map (scale 1:7,500,000). Annals of the Association ofAmerican Geographers 77:118–125.

Osenberg, C. W., R. J. Schmitt, S. J. Holbrook, K. E. Abu-Saba,and A. R. Flegal. 2006. Detection of environmental impacts:natural variability, effect size, and power analysis. Pages 83–108 in R. J. Schmitt and C. W. Osenberg (editors). Detectingecological impacts. Academic Press, New York.

Peck, D. V., A. T. Herlihy, B. H. Hill, R. M. Hughes, P. R.Kaufmann, D. J. Klemm, J. M. Lazorchak, F. H. McCormick,S. A. Peterson, S. A. Ringold, T. Magee, and M. Cappaert. 2006.Environmental Monitoring and Assessment Program—surfacewaters western pilot study: field operations manual for wade-able streams. EPA/620/R-06/003. Office of Research and De-velopment, US Environmental Protection Agency, Corvallis,Oregon.

Rehn, A. C. 2009. Benthic macroinvertebrates as indicators ofbiological condition below hydropower dams on west slopeSierra Nevada streams, California, USA. River Research andApplications 25:208–228.

Rehn, A. C., P. R. Ode, and C. P. Hawkins. 2007. Comparisonof targeted-riffle and reach-wide benthic macroinvertebratesamples: implications for data sharing in stream-conditionassessments. Journal of the North American BenthologicalSociety 26:332–348.

Rehn, A. C., P. R. Ode, and J. T. May. 2005. Development of abenthic index of biotic integrity (B-IBI) for wadeable streamsin northern coastal California and its application to regional305(b) assessment. Report to the State Water Resources Con-trol Board. Aquatic Bioassessment Laboratory, CaliforniaDepartment of Fish and Game, Rancho Cordova, California.(Available from: http://www.waterboards.ca.gov/water_issues?programs/swamp/docs/reports/final_north_calif_ibi.pdf)

Reynoldson, T. B., R. H. Norris, V. H. Resh, K. E. Day, and D. M.Rosenberg. 1997. The reference condition: a comparison ofmultimetric and multivariate approaches to assess water qualityimpairment using benthic macroinvertebrates. Journal of theNorth American Benthological Society 16:833–852.

Richards, A. B., and D. C. Rogers. 2006. List of freshwater macro-invertebrate taxa from California and adjacent states includingstandard taxonomic effort levels. Southwest Association ofFreshwater Invertebrate Taxonomists, Chico, California. (Avail-able from: http://www.safit.org/ste.html)

Sánchez-Montoya, M. M., M. R. Vidal-Abarca, T. Puntí, J. M.Poquet, N. Prat, M. Rieradevall, J. Alba-Tercedor, C. Zamora-Muñoz, M. Toro, S. Robles, M. Álvarez, and M. L. Suárez.2009. Defining criteria to select reference sites in Mediterra-nean streams. Hydrobiologia 619:39–54.

Sleeter, B. M., T. S. Wilson, C. E. Soulard, and J. Liu. 2011. Esti-mation of the late twentieth century land-cover change inCalifornia. Environmental Monitoring and Assessment 173:251–266.

Solek, C. W., E. Stein, J. N. Collins, L. Grenier, J. R. Clark, K.O’Connor, C. Clark, and C. Roberts. 2010. Developing astatewide network of reference wetlands for California: con-ceptual approach and process for prioritizing and selectingreference sites. Southern California Coastal Water ResearchProject, Costa Mesa, California. (Available from: http://www.sccwrp.org)

Statzner, B., B. Bis, S. Dolédec, and P. Usseglio-Polatera. 2001.Perspectives for biomonitoring at large spatial scales: a uni-fied measure for the functional classification of invertebratecommunities in European running waters. Basic and Ap-plied Ecology 2:73–85.

Stevens, D. L., and A. R. Olsen. 2004. Spatially balanced sam-pling of natural resources. Journal of the American Statisti-cal Association 99:262–278.

Stoddard, J. L., P. Larsen, C. P. Hawkins, R. K. Johnson, andR. H. Norris. 2006. Setting expectations for the ecologicalcondition of running waters: the concept of reference condi-tion. Ecological Applications 16:1267–1276.

Vander Laan, J. J., C. P. Hawkins, J. R. Olson, and R. A. Hill.2013. Linking land use, in-stream stressors, and biologicalcondition to infer causes of regional ecological impairmentin streams. Freshwater Science 32:801–820.

Waite, I. R., A. T. Herlihy, D. P. Larsen, and D. L. Klemm.2000. Comparing strengths of geographic and nongeographicclassifications of stream benthic macroinvertebrates in theMid-Atlantic Highlands, USA. Journal of the North Ameri-can Benthological Society 19:429–441.

Whittier, T. R., J. L. Stoddard, D. P. Larsen, and A. T. Herlihy.2007. Selecting reference sites for stream biological assess-ments: best professional judgment or objective criteria. Journalof the North American Benthological Society 26:349–360.

Yates, A. G., and R. C. Bailey. 2010. Selecting objectively definedreference streams for bioassessment programs. Environmen-tal Monitoring and Assessment 170:129–140.

Yuan, L. L., C. P. Hawkins, and J. Van Sickle. 2008. Effects ofregionalization decisions on an O/E index for the nationalassessment. Journal of the North American BenthologicalSociety 27:192–905.



http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282007%2926%5B349%3ASRSFSB%5D2.0.CO%3B2

http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282007%2926%5B349%3ASRSFSB%5D2.0.CO%3B2

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1198%2F016214504000000250


http://www.journals.uchicago.edu/action/showLinks?crossref=10.1198%2F016214504000000250





http://www.journals.uchicago.edu/action/showLinks?crossref=10.1890%2F1051-0761%282006%29016%5B1267%3ASEFTEC%5D2.0.CO%3B2





http://www.journals.uchicago.edu/action/showLinks?crossref=10.1029%2F2011WR011088









http://www.journals.uchicago.edu/action/showLinks?crossref=10.1002%2Frra.1121


http://www.journals.uchicago.edu/action/showLinks?crossref=10.1002%2Frra.1121


http://www.journals.uchicago.edu/action/showLinks?crossref=10.1078%2F1439-1791-00039

http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282007%2926%5B332%3ACOTARB%5D2.0.CO%3B2

http://www.journals.uchicago.edu/action/showLinks?crossref=10.1078%2F1439-1791-00039

http://www.journals.uchicago.edu/action/showLinks?system=10.1899%2F0887-3593%282007%2926%5B332%3ACOTARB%5D2.0.CO%3B2

Table S1. Sources of spatial data used for screening of reference sites and evaluation of reference-site characteristics. Codes refer to

application in Table S2.

Type of spatial data Source or model Reference Code

Climate PRISM http://www.prism.oregonstate.edu a

Geology and mineral content Generalized geology and mineralogy data Olson and Hawkins (2012) c

Atmospheric deposition National Atmospheric Deposition Program National Trends Network http://nadp.sws.uiuc.edu/ntn/ d

Predicted surface-water conductivity Quantile regression forest model (Meinshausen 2006) Olson and Hawkins (2012) e

Soil properties Generalized soil properties data Olson and Hawkins (2012) f

Ground water MRI-Darcy Model (Baker et al. 2001) Olson and Hawkins (2012) h

Waterbody location and attribute data NHD Plus http://www.horizon-systems.com/nhdplus/ i

Dam location, storage National Inventory of Dams http://geo.usace.army.mil/ j

Land cover, imperviousness National Land Cover Dataset (2001) http://www.epa.gov/mrlc/nlcd-2006.html k

Elevation National Elevation Dataset http://ned.usgs.gov/ m

Mine location and attribute data Mineral Resource Data System http://tin.er.usgs.gov/mrds/ n

Discharge location and attribute data California Integrated Water Quality System http://www.swrcb.ca.gov/ciwqs/ o

Road location and attribute data California State University Chico Geographic Information Center CSU Chico Geographic Information Center q

Railroad location and attribute data California State University Chico Geographic Information Center CSU Chico Geographic Information Center r

Invasive invertebrate records California Aquatic Bioassessment Lab http://www.dfg.ca.gov/abl/ u

University of Montana http://www.esg.montana.edu/aim/mollusca/nzms/index.html


http://nadp.sws.uiuc.edu/ntn/

http://geo.usace.army.mil/

http://ned.usgs.gov/

Santa Monica Baykeeper Abramson 2009

USGS Non-indigenous Aquatic Species Database http://nas.er.usgs.gov


Table S2. Natural and stressor variables used for screening reference sites and evaluating reference-site characteristics. Unless noted

in column n (= sample size), metrics were calculated for 1985 sites. Source(s) codes refer to sources listed in Table 1. Scale refers to

spatial area of analysis (WS = upstream watershed, 1k = watershed area within 1 km of site, 5k = watershed area within 5 km of site).

Variables preceded by an asterisk (*) were used in the calculation of predicted conductivity (CondQR50) and variables preceded by a

dagger (†) were used in the principal components analysis. NLCD = National Land Cover Dataset.

Metric Description n Source(s) Unit

Scale

Point WS 5k 1k

Natural gradient

Location

*†logWSA Area of the unit of analysis m2 X

ELEV Elevation of site m m X

MAX_ELEV Maximum elevation in catchment m m X

*†ELEV RANGE Elevation range of catchment m m X

*†New_Lat Latitude X

*†New_Long Longitude X

Climate

*†PPT_00_09 10-y (2000–2009) average annual precipitation a mm X

*†TEMP_00_09 10-y (2000–2009) average monthly temperature a °C X

*AtmCa Catchment mean of mean 1994–2006 annual precipitation-weighted mean Ca+ d mg/L X


concentration

*AtmMg Catchment mean of mean 1994–2006 annual precipitation-weighted mean Mg+

concentration

d mg/L X

*AtmSO4 Catchment mean of mean 1994–2006 annual precipitation-weighted mean SO4–2

concentration

d mg/L X

*LST32AVE Average of mean 1961–1990 first and last day of freeze d days X

*MINP_WS Catchment mean of mean 1971–2000 minimum monthly precipitation d mm/mo X

*MEANP_WS Catchment mean of mean 1971–2000 annual precipitation d mm/mo X

*†SumAve_P Catchment mean of mean June–September 1971–2000 monthly precipitation d mm/m X

*TMAX_WS Catchment mean of mean 1971–2000 maximum temperature d °C X

*XWD_WS Catchment mean of mean 1961–1990 annual number of wet days d days X

*MAXWD_WS Catchment mean of 1961–1990 annual maximum number of wet days d days X

Geology

CaO_Avg Calcite mineral content c % X

MgO_Avg Magnesium oxide mineral content c % X

ᵡN_Avg Nitrogenous mineral content c % X

P_Avg P mineral content c % X

PCT_SEDIM Sedimentary geology in catchment c % X

S_Avg Sulfur mineral content c % X

*UCS_Mean Catchment mean unconfined compressive strength f MPa X


*LPREM_mean Catchment mean log geometric mean hydraulic conductivity h 10–6

m/s X

*†BDH_AVE Catchment mean bulk density f g/cm3 X

*†KFCT_AVE Catchment mean soil erodibility (K) factor f None X

*PRMH_AVE Catchment mean soil permeability f In/h X

CondQR50 Median predicted conductivity 1155 e µS/cm X

Stressor

Hydrology

PerManMade Percent canals or pipes at the 100-k scale i % X

InvDamDist Inverse distance to nearest upstream dam in catchment j km X

Land use

Ag % Agricultural (row crop and pasture, NLCD 2001 codes 81 and 82) k % X X X

Urban % Urban (NLCD 2001 codes 21–24) k % X X X

CODE_21 % Urban/Recreational Grass (NLCD code 21) k % X X X

Mining

GravelMinesDensL Linear density of gravel mines within 250 m of stream channel n mines/km X X X

MinesDens Number of mines (producers only) n mines X

Transportation

PAVED_INT Number of paved road crossings q, r Count X X X

RoadDens Road density (includes rail) q, r km/km2 X X X

Habitat


Embeddedness Average % cobble embeddedness 576 Field

measurements

% X

P_SAFN % sands and fines 1191 Field

measurements

% X

W1_HALL Weighted human influence 964 Field

measurements

None X


Table of Reference Sites for the California Stream Condition Index Station Latitude Longitude Sample Date CSCI

103FS0405 41.78889527 -124.0778375 9/14/2010 1.073788446 103WER026 41.84998645 -124.0332179 7/29/2003 1.112196311 103WER029 41.80976601 -124.1120887 7/30/2003 0.985982087 105CE0149 41.48313406 -123.7437815 9/27/2005 0.933622317 105CE0329 41.30817529 -123.0339192 7/17/2007 1.074196666 105FS0415 41.61202148 -123.2922522 9/21/2010 1.191011342 105FS0418 41.5728015 -123.5446885 7/27/2000 0.837966726 105FS0423 41.7369287 -123.548724 8/10/2010 0.987356493 105FS0425 41.35826851 -123.0745063 8/10/2011 1.157492483 105FS0428 41.4410528 -123.3582948 8/3/2011 1.007754759 105PS0043 41.72939189 -123.2367063 9/17/2008 1.144407253 105PS0083 41.36469401 -123.1871691 9/8/2009 1.114592916 105PS0090 41.07846793 -123.031717 8/25/2009 1.016029253 105PS0097 41.74157187 -123.355488 9/8/2009 1.170430741 105PS0154 41.09812487 -123.0733997 8/9/2011 0.994237798 105PS0540 41.62001951 -122.0692366 8/5/2010 1.144353095 105WE0583 41.7084139 -123.132001 8/1/2001 1.113892184 105WE0592 41.36252018 -122.5200748 9/12/2000 1.070591558 105WE0833 41.86158299 -122.209015 8/15/2001 0.989155466 105WE1091 41.05030234 -122.9345206 8/13/2002 0.981472821 105WER052 41.31565299 -123.060801 8/27/2003 1.078410302 106CE0521 41.04590374 -123.5394091 7/21/2006 1.024264983 106FS0040 40.16912433 -123.0251519 6/26/2001 0.815460898 106PS0115 41.17290127 -122.8695878 9/21/2010 0.993714816 106PS0124 40.17054647 -123.0286717 8/16/2010 1.216462084 106PS0166 41.04411616 -123.6117049 8/2/2011 1.196756318 106WE0505 40.86234647 -122.9030219 8/9/2000 0.98748657 106WE1055 40.90343109 -123.0151421 7/11/2002 1.164797135 106WE1066 41.15436642 -123.5702606 7/17/2002 1.085361514 106WE1079 41.13485769 -122.8050878 7/9/2002 0.871947271 106WER011 40.93588454 -123.154861 6/28/2001 0.980564245 106WER016 40.48965069 -123.0294576 9/16/2009 1.193645743 107PS0021 41.32412093 -124.0079587 9/3/2008 1.095297132 107WER092 41.39995614 -124.0579036 8/12/2003 0.882726171 111BBRBCC 39.69950839 -122.9244532 9/23/2009 1.015829442 111BGCAER 40.02232507 -123.061481 10/6/2009 1.212189312 111CE0569 40.34675705 -123.9924738 7/18/2006 0.909613526 111FS0001 39.32709256 -122.8311121 6/12/2001 0.951344402 111FS0007 39.47122705 -122.8337823 6/21/2001 0.98555429


Station Latitude Longitude Sample Date CSCI 111FS0008 39.31929726 -122.8454015 6/22/2001 0.910317346 111FS0158 39.39568533 -122.8637371 9/12/2000 1.119633897 111HSCAER 39.52503385 -122.8527579 7/26/2011 1.053819223 111PS0110 39.35141964 -122.8762117 6/24/2009 1.186680448 111PS0144 39.90017962 -123.0171393 8/18/2010 0.987184458 111PS0160 39.44611995 -123.0292647 7/25/2011 1.239887813 111PS0204 39.27693366 -122.8658082 8/17/2010 0.782561041 111WE0816 39.58740561 -122.9181733 8/14/2001 1.130188315 111WE0837 39.8692887 -123.0522717 7/9/2002 0.84815047 111WE0844 40.31663763 -123.9987389 8/8/2002 1.071032683 111WE1050 39.71147016 -122.8419278 6/25/2002 1.254663084 111WE1071 39.70053373 -122.8125516 6/26/2002 0.367827622 111WER008 39.49365678 -122.8639231 6/21/2001 0.881780224 111WER046 39.7403222 -123.6311019 10/1/2003 1.048173202 111WER047 39.72965635 -123.6323576 10/1/2003 1.08679564 111WER159 39.42936794 -122.8451843 6/29/2004 1.192803007 112WE1063 40.16930752 -124.0410581 9/10/2002 1.131272395 114DLCURx 39.12873514 -123.2313281 6/20/2002 1.143966731 114PS0145 38.8936121 -123.0701698 6/20/2011 0.98163653 114WE0614 38.53916922 -123.0719085 8/20/2003 0.835051825 114WE0810 38.87517623 -122.8828949 7/26/2001 1.257162812 115EBAG01 38.34613327 -122.9153477 7/1/2010 0.619487632 200COYCOI 37.12621772 -121.4811805 6/28/2010 0.826264993 201LAG335 37.9917933 -122.661945 4/19/2001 0.837931575 201MRS020 37.9196808 -122.6695183 9/13/2010 1.258287657 202BUT040 37.24131773 -122.316955 4/9/2002 1.329877463 202BUT050 37.20923557 -122.3326697 6/16/2009 1.032878339 202PES170 37.26388691 -122.2393177 4/10/2002 0.977220655

204AHOACR 37.46158312 -121.7690186 5/13/2004 0.973251813 204CE0116 37.40849637 -121.7103221 8/9/2005 0.913604636 206BEA035 38.45060179 -122.4382199 4/25/2001 1.034604226 206CON130 38.48539888 -122.2956245 5/3/2000 1.066129335 206DRY060 38.38855251 -122.4093756 4/23/2001 1.244608368 206DRY100 38.41223805 -122.4342513 5/3/2000 1.239219643 206NOV080 38.06236324 -122.5811128 4/15/2004 0.655963362 206TUL120 38.28461447 -122.2175407 6/13/2011 1.134949056

304WADRED 37.11373244 -122.26962 6/29/2010 1.153702723 305LGCACR 36.34910146 -120.821041 7/16/2008 0.94571438 305SWACRO 37.08478483 -121.794114 6/29/2010 1.005721089 305UVASWA 37.08669726 -121.7944539 6/29/2010 1.022020427


Station Latitude Longitude Sample Date CSCI 307CARLOS 36.37219615 -121.6630073 6/22/2010 1.221677924 307SANCLE 36.43072607 -121.798394 6/21/2010 0.91723023

308BSU 36.24559854 -121.773391 7/13/2010 0.882807306 308SALHY1 35.81439054 -121.3589594 6/23/2010 0.877294234

308SOBHW1 36.45626997 -121.9252228 6/21/2010 1.041071921 308WLOxxx 35.8935202 -121.4616141 4/7/2004 0.817591388 309CAW126 36.23071982 -121.5515624 6/17/2003 1.08883073 309CAW130 35.98618668 -121.3570484 6/12/2003 1.184058606 309CAW174 36.09276821 -121.434252 5/20/2003 0.795149715 309CAW178 36.08531517 -121.3941823 5/21/2003 0.689010578 309PS0116 36.06400288 -121.3261577 6/16/2011 1.007767316 309SARANF 36.06447703 -121.3852597 8/26/2008 1.19046094 309UASIND 36.1198069 -121.4688849 6/23/2010 1.251596124 310COOxxx 35.2549357 -120.887212 3/29/2002 1.108639806 310LPCBPC 35.26539078 -120.5226007 6/9/2009 1.075577022

312MANNIR 34.77057782 -119.9364104 6/14/2010 0.997575483 312PS0099 34.77957633 -119.9579578 6/14/2010 0.649713107 312WE1028 34.74724787 -119.716677 5/29/2002 1.194915868 312WE1035 34.73255449 -119.6771076 5/30/2002 1.03161103 314WE0677 34.74761088 -120.0543524 5/9/2002 1.123478983 314WE0779 34.67015892 -119.754612 7/11/2001 1.170676522 314WE0796 34.68354218 -119.8011571 7/10/2001 1.235261218 402BA0095 34.46260229 -119.3460184 5/29/2008 1.2729175 402BA0287 34.50283671 -119.365789 6/9/2008 1.079462565 402S02591 34.51784045 -119.3784676 7/21/2011 1.013337058 402S04795 34.47988903 -119.1706761 7/14/2011 1.027308156 402S05407 34.50125108 -119.3504389 7/21/2011 1.127727499 402WE0536 34.51967468 -119.4063132 6/14/2000 1.143326429 402WE0803 34.4212966 -119.3817335 6/12/2001 0.947629675 403BA0015 34.53173072 -119.1836741 5/19/2008 0.986316738 403BA0171 34.58578564 -119.2833858 5/19/2008 1.061592963 403BA0960 34.59121797 -118.4606154 5/21/2008 1.161570351 403S01164 34.35078916 -118.5796872 6/24/2011 1.020572381 403S01536 34.5690899 -118.3921259 7/12/2011 0.864260853 403S01979 34.45205498 -119.136245 7/14/2011 0.991078019 403STC021 34.64779466 -119.0679133 6/4/2003 0.980190875 403STC026 34.55916274 -119.2691552 2/5/2003 0.73750977 403STC029 34.61532768 -118.5595799 2/4/2003 1.107726648 403STC030 34.5675607 -119.1544287 2/6/2003 1.2142365 403STC064 34.55814811 -119.1045921 6/8/2011 1.038886403


Station Latitude Longitude Sample Date CSCI 403STC065 34.63111461 -118.5364072 2/4/2003 1.138201897 403STC085 34.64270211 -119.0783376 6/4/2003 1.000816261 403STC086 34.55624729 -119.205918 2/6/2003 0.974817008 403WE0501 34.54139913 -118.774337 6/7/2001 1.219975402 403WE0540 34.65096861 -119.1203168 6/21/2000 0.914825176 403WE0795 34.64086852 -119.0820458 6/6/2001 1.121789534 403WE1027 34.72655489 -119.0294278 5/2/2002 0.779963124 405BRCAMS 34.27376523 -117.8990056 6/7/2011 1.298672786 405S01248 34.25895681 -117.744993 7/26/2011 1.173898189 405WER318 34.26818848 -117.8929433 7/7/2009 1.186548118 504PS0227 39.98925583 -121.9698086 8/12/2009 0.973563876 504WE0527 40.10751946 -122.031096 8/20/2001 1.009461462 505PS0110 41.09655559 -122.2015036 8/3/2010 0.987578764 505PS0174 40.99700689 -122.0590657 9/27/2011 1.153543871 506PS0062 40.86097847 -122.4622287 9/15/2008 1.009820211

508ACNFPW 40.28075595 -121.7662868 10/9/2002 0.89716628 509ACNFPP 40.24151768 -121.8625736 10/10/2002 1.021286937 509ACSFPP 40.24141277 -121.8621056 10/10/2002 1.137845799 509ATCINC 40.23309871 -121.8784189 7/14/2008 1.094115766 509BSCADC 40.10205747 -121.6636106 8/10/2000 1.082192867 509CE0162 40.06017231 -121.9478016 9/6/2005 0.817280199 509FCA054 40.21812279 -121.9988993 9/21/2009 0.913903573

509PCDTWR 40.31173177 -121.8865247 9/3/2008 0.942972204 509PS0085 40.25515869 -121.5589888 9/7/2011 1.095573282 509PS0170 40.26505186 -121.7681249 7/26/2010 1.068052062 509PS0234 40.23194797 -121.8989795 7/21/2011 0.802818462 513PS0008 38.93238602 -122.9322325 6/11/2009 0.617584714

514BMCARC 38.87262994 -120.7436056 8/1/2000 0.946439601 514ICEUPx 38.82007635 -120.3139587 9/8/2005 1.091246429 514LNCASC 38.81356525 -120.2260949 8/30/2011 0.669786779 514SCCAAR 38.79109538 -120.1048071 7/24/2008 1.091234926 517FCA046 39.46546447 -120.96739 7/20/2009 0.983350996 517PS0054 39.44579799 -121.0335499 7/28/2010 1.068325463 517PS0074 39.6237242 -120.7248071 9/20/2011 1.132679387 518CE0047 40.37539338 -121.1049458 6/14/2004 0.293582159 518CE0338 39.76503016 -120.8314678 8/21/2007 1.236922748 518CPCRCR 40.00465798 -121.2706029 8/14/2008 0.915376123 518FCA075 39.9061276 -121.0467675 9/23/2008 1.055193706 518GZCUPX 39.88992526 -121.2779301 8/20/2008 1.009911202 518KNCAWC 40.45028219 -121.3672013 8/7/2000 0.816286315


Station Latitude Longitude Sample Date CSCI 518MCGRRx 39.65351001 -121.3950561 10/10/2002 0.933752518 518PS0033 39.90979932 -121.0706296 6/29/2009 1.138966111 518SED086 39.80622272 -121.0421036 8/17/2011 1.133722681 518WE0521 39.81297147 -120.6821148 8/30/2000 1.133405254 518YLCAFR 40.01206027 -121.2474584 8/17/2011 1.187916608 522CE0027 39.29118144 -122.5590115 6/3/2004 0.863584731 523PS0172 39.93458556 -122.8273985 8/10/2011 1.172434441

523TMCATG 39.85365304 -122.6709404 10/5/2009 0.747921328 524ACMFPW 40.26751694 -121.7756355 10/9/2002 1.06174095 525WE0755 41.33707374 -122.4087735 8/22/2001 1.028786655 526CE0061 41.37625045 -120.2918331 8/3/2004 0.902976402 526PS0072 41.13510762 -120.8002516 9/23/2008 1.041688186 526PS0076 41.27434167 -120.2636819 8/26/2008 0.981199051 526PS0396 41.37827034 -120.4141952 9/1/2009 1.081116087 526WER123 41.27280981 -120.2733868 8/4/2004 1.23856222 532WE0760 38.57710132 -120.5451141 8/9/2001 1.047875635 532WER222 38.49592822 -119.8948968 7/28/2004 0.985688895 534MLC108 38.30048971 -119.9347228 9/14/2011 1.077834392 534RSCAGG 38.13263387 -120.2184873 9/19/2011 0.851420294 536CE0106 37.80111368 -119.8476773 8/17/2005 1.250447935 536CE0330 38.03672444 -120.0899836 6/27/2007 0.833149484 536CLV004 37.97643254 -120.051348 10/13/2005 1.134983221 536SED006 37.90553891 -119.9969611 8/16/2006 1.130496137 536SED108 37.86881216 -119.9043549 9/21/2011 1.090336631 536WER237 37.87780048 -119.2520771 8/10/2004 1.057183083 537BLCABG 37.70457761 -119.9741714 9/15/2011 0.844408461 537CE0238 37.68463228 -119.6854873 8/15/2006 1.03626815 537FCA009 37.68448695 -119.5401621 8/12/2009 0.847538018 537WER236 37.85294961 -119.5748096 8/11/2004 1.06885087 540CE0078 37.21963497 -118.8614809 8/17/2004 1.031758246 540GNCBTC 37.56419334 -119.3154568 9/13/2011 1.239742814 540SED003 37.24198496 -118.9465535 6/20/2007 0.952300459 540SJRUPx 37.24041128 -118.9388794 9/15/2005 1.086204175 552FCA017 36.96367359 -119.2313961 8/19/2008 0.663134338 552MFCAKR 36.85384177 -119.0974007 6/11/2008 0.717908584 552PS0028 36.84306028 -119.0971832 6/18/2008 1.137798314 552PS0284 36.90905851 -119.1272701 8/29/2011 1.240293486

553KRMAPC 36.51986649 -118.7597473 8/22/2011 0.977859411 553KRNAYC 36.54645788 -118.897798 8/14/2000 1.094162226 553WER208 36.64784736 -118.8022696 7/8/2004 1.037625969


Station Latitude Longitude Sample Date CSCI 553WER225 36.35116471 -118.7513701 7/8/2004 1.13654784 554CLCALK 36.21272141 -118.5166538 8/23/2011 1.128220782

554NYCGWD 36.02064002 -118.5670574 7/19/2000 1.093390201 554PMCABC 35.95286412 -118.4154333 7/20/2000 1.211122895

554PMCGWD 36.08357346 -118.5475494 7/19/2000 0.803437341 555PS0064 36.18223438 -118.7880582 6/17/2008 1.104254159 601LVC002 37.97301582 -119.1102743 7/2/2003 0.950262713

603DDM001 37.7512698 -118.9429195 9/14/2000 0.906127576 603DDM004 37.70821134 -119.056999 7/6/1999 1.023494518 603DDM005 37.70826681 -119.0531977 7/1/1999 0.985001967 603DDM006 37.70537702 -119.032801 9/13/2011 1.026505444 603FCA145 37.56513646 -118.7544542 8/11/2009 1.194842623 603GLS001 37.74707335 -118.9862384 9/7/2006 0.687048917 603OHR001 37.74498783 -118.7556297 9/13/2000 0.967866345 603PS0008 37.48579392 -118.7604659 7/30/2008 0.99354364 603PS0045 37.55124585 -118.8015769 8/29/2011 0.905671538 603PS0056 37.34173611 -118.7337829 8/31/2011 0.917515881 603PS0073 37.70933961 -119.0580324 9/13/2011 1.093641004 603PS0109 36.59494492 -118.1448532 7/29/2008 0.938174922 603WE0758 37.54963915 -118.7610019 7/2/2001 1.08099741 603WER242 37.55109121 -118.8730709 8/31/2004 0.866298022 603WER245 37.0109348 -118.3298285 7/29/2004 0.897713657 603WKR001 36.24400803 -118.0553363 7/26/2005 0.670121197 604WER244 37.55472829 -118.1715316 7/27/2004 0.571260204 626PS0171 34.44758034 -118.015911 5/15/2008 1.089687361 626PS0619 34.4187649 -117.9747059 6/3/2010 1.123450808 626PS0827 34.3865167 -117.8250169 6/3/2010 1.217640188 628CE0088 34.30576495 -116.9426962 5/12/2004 0.871879457 630BDY001 38.27033777 -119.3085893 7/7/2005 1.242744222 630BUC001 38.27562088 -119.2584217 7/24/2000 0.820719504 630FCA097 38.27129636 -119.3305513 8/10/2009 1.181393791 630PMT001 38.30374868 -119.0328702 7/6/2005 0.677863597 630PS0005 38.06609598 -119.2250465 7/31/2008 1.206173011 630RBS002 38.26904327 -119.2608541 9/11/2000 0.767458505 630RBS004 38.19188307 -119.3198422 7/12/2002 0.930558893 630SWA001 38.35817076 -119.34264 7/16/2002 1.197112467 630SWA002 38.37018312 -119.3475864 8/13/1999 0.978647602 630WE0860 38.31089784 -119.3440752 8/21/2008 1.102109841 631CE0074 38.45073836 -119.4881965 8/23/2004 1.117025421 631COT001 38.44022039 -119.4246821 9/28/2000 0.990558544


Station Latitude Longitude Sample Date CSCI 631KIR001 38.34522076 -119.5207102 9/28/2000 0.983039758 631KIR002 38.33582244 -119.5134524 7/18/1999 0.679446889 631KIR003 38.33714447 -119.5204724 7/22/2002 1.00736902 631POR002 38.33189907 -119.5331485 7/18/2002 0.796499284 631PS0023 38.47176646 -119.3530753 8/4/2009 1.065205261 631PS0033 38.28757702 -119.5474199 8/30/2010 0.908971186 631SLK001 38.60058445 -119.5685004 7/27/2000 0.812600793 631SRD001 38.31520277 -119.5972931 7/13/2005 1.017061082

631WWF001 38.36615405 -119.5640402 7/12/2005 1.154295631 631WWK002 38.37232061 -119.4543162 7/8/2002 1.192114845 631WWK005 38.3091402 -119.5505771 8/15/2000 0.922709906 631WWK006 38.29114835 -119.5459791 7/17/2002 1.124730349 632BGV002 38.59508678 -119.6493374 7/8/1999 0.669077678 632BGV003 38.59837991 -119.6482284 7/24/2002 1.03155841 632CHV001 38.67824961 -119.9008069 8/19/2004 0.883281181 632COY001 38.48078376 -119.5991313 9/5/2007 1.207986388 632COY002 38.4656221 -119.5795678 9/5/2007 1.260027828 632CRV001 38.49839051 -119.6046153 9/6/2007 1.242295614 632ECR006 38.5910931 -119.656858 9/7/2000 1.121955872 632ECR008 38.57040906 -119.6279319 9/8/2007 0.957186707 632ELD001 38.49939776 -119.7408887 7/10/2003 1.117172912 632LEV004 38.68851296 -119.6485606 7/9/1999 1.117630797

632MUR001 38.49204842 -119.7020096 9/7/2011 1.165107074 632NOB001 38.57394592 -119.7906001 9/1/2010 1.052818807 632PS0007 38.53334752 -119.5937108 7/23/2008 1.050486019 632PS0014 38.56549454 -119.6234842 9/8/2007 0.885838818 632PS0058 38.60611127 -119.6900209 8/15/2011 1.124226606 632SND001 38.52111146 -119.5776001 6/22/2005 1.039379944 632SPT001 38.68450082 -119.8085037 9/6/2000 1.227420851 632SVK002 38.55183553 -119.6072492 7/23/2002 1.128223005 632SVR001 38.60380314 -119.7703631 6/28/2007 1.236606477 632WE0629 38.69556861 -119.8764616 6/27/2002 1.119749895 632WLF001 38.57309679 -119.6964221 9/7/2000 1.228392674 633PS0003 38.81741232 -119.9094677 7/22/2008 0.835085387

633WCR003 38.68076254 -119.9330804 8/31/1999 0.925979201 633WE0991 38.73224238 -119.9296218 9/10/2003 1.101368577 634CAS001 38.92503119 -120.1096291 8/15/2002 0.968650168 634CE0482 39.08457235 -120.2154141 7/14/2004 0.963474439 634GEN001 39.04425231 -120.1437994 8/30/2000 1.127806658 634GRF001 39.26550198 -120.0268055 7/19/2005 1.206163548


Station Latitude Longitude Sample Date CSCI 634HHCATC 38.85368262 -119.947937 10/4/2011 0.944991562 634MEK001 39.01533059 -120.1573212 8/14/2002 0.962844368 634PS0035 38.87154117 -120.0993532 9/1/2010 1.078211683

634R10BMW 38.7796377 -119.9980634 9/29/2011 0.876592665 634R10GNL 39.02996227 -120.1603969 10/3/2011 0.716391427 634R10UTR 38.77826055 -120.0293634 9/28/2011 1.272912546 634SAX001 38.86080587 -119.9847693 8/22/2001 1.141515786 634SAX003 38.87181611 -119.9828069 9/11/2007 0.973684089 634SAX004 38.86952333 -119.9810723 9/11/2007 1.191341529 634SAX005 39.8680118 -119.9576079 9/11/2007 1.214095404 634TRT001 38.85699255 -119.9443614 8/23/2001 1.023139019 634UTR008 38.7937118 -120.0202984 9/21/2000 1.269765312 634WRD001 39.13981535 -120.1910775 7/21/2006 1.065823979 634WSN001 39.22204047 -120.1017828 7/20/2005 1.184357571 635BRC001 39.38390927 -120.0208554 9/8/2005 0.937870376 635CLS001 39.30027953 -120.2521883 9/1/2000 1.06024449 635GRY001 39.37332861 -120.0293459 9/7/2005 1.154967062 635MAR003 39.27536517 -120.1720755 7/27/2004 1.119337328 635POL001 39.23573299 -120.2331556 8/31/2000 1.192645869 635PRN001 39.38574829 -120.2523958 7/11/2001 1.062106899 635PS0038 39.38237649 -120.2262368 8/9/2010 1.0160885 635SQN001 39.20374898 -120.2501294 7/9/2001 1.088255574 636LCY001 39.46467416 -120.4265801 7/12/2001 1.167794052 636LTR001 39.48479987 -120.3734233 8/31/2000 1.190419365 636LTR002 39.49345792 -120.3464502 7/13/2001 0.996121272 636PRZ001 39.47595781 -120.3825208 7/12/2001 0.957610781 636PS0070 39.49469726 -120.3337301 9/21/2011 0.708809823 636SAG001 39.43728013 -120.229495 7/12/2001 1.026872596 637CE0511 40.56708241 -121.1093011 8/10/2004 1.029837851 719FCA001 33.76056054 -116.549231 6/24/2008 1.091254053 719WE0864 33.76066574 -116.549863 5/8/2001 0.958117623 801CCWFAC 34.18951448 -117.1818622 5/20/2004 0.550647174 801CE0152 34.18869908 -117.1818586 7/20/2005 0.891081842 801WE0787 34.14540971 -116.8791779 6/27/2001 1.089826005 801WE1008 34.18097468 -116.9124942 6/6/2002 1.184298363 801WE1132 34.13378343 -116.8431432 3/26/2009 1.148163962 802FMCAIP 33.80700075 -116.74394 6/14/2005 0.846125238 802WE0658 33.79828795 -116.7492155 5/9/2001 0.936273857 802WE0794 33.72931676 -116.674799 5/10/2001 1.081405774 901CSCADC 33.49388103 -117.430283 5/26/2011 1.050200761


Station Latitude Longitude Sample Date CSCI 901DCCSMC 33.47362209 -117.4655951 5/16/2011 1.275247096 901S01849 33.55523906 -117.3977969 6/1/2010 1.151291979 901S02873 33.54344001 -117.3973932 6/1/2010 1.236699166 901SCCA74 33.61853643 -117.5098416 6/8/2010 1.101917864 903SLFRCx 33.34178027 -116.8774618 5/22/2001 1.127907042

903WE0798 33.33686513 -116.8293442 5/17/2001 0.904777171 905CE0512 33.11107103 -116.7431147 4/20/2004 1.076698742 905S02561 33.12974258 -116.6359163 5/25/2011 1.270228371 905WE0679 33.13162498 -116.6551708 4/24/2002 0.788305154 907S03210 33.00312855 -116.7291857 5/18/2011 0.989849401 911KCKCRX 32.78740357 -116.4509978 5/19/2001 1.125978736 911S00858 32.90282372 -116.493371 5/31/2010 1.071750352 911S01142 32.73547774 -116.6526831 5/24/2011 0.937273678 911TJIND2 32.89738775 -116.5043406 6/5/2007 0.87466397 911TJWIL3 32.69398597 -116.6957022 5/6/2008 1.038247635

CAT08722-017 38.75327624 -120.0282365 8/5/2009 0.754560916 CAT08722-024 39.25791883 -120.0126331 7/29/2009 1.069545967 CAT08722-025 38.87548027 -120.0834864 8/18/2009 0.771611523 CAT08722-036 39.06611231 -119.9342582 7/27/2009 1.091982778 CAT08722-039 39.08895429 -120.1657312 8/13/2009 1.04767914 CAT08722-047 39.13958936 -119.9235974 7/20/2009 0.676059511 CAT08722-049 38.78029505 -120.0285593 9/2/2009 0.801650797 CAT08722-054 38.89909303 -119.9040375 8/21/2009 0.910061674 CAT08722-055 39.00570809 -120.1651446 8/25/2009 1.254320321 CAT08722-056 39.27978207 -119.9127368 7/28/2009 1.167252873 CAT08722-228 39.07307177 -119.9200509 8/10/2010 0.970328894

JonLee6 41.32697454 -124.023873 9/13/2010 1.054469009 PC1 34.41189317 -118.9009827 8/12/2010 1.016981913 PC4 34.42310857 -118.8848637 8/16/2010 1.054210995 PC5 34.42767307 -118.8847086 8/13/2010 1.130450995 PC6 34.43046892 -118.8867539 8/12/2010 1.105978023 PC7 34.4327607 -118.8910331 8/12/2010 1.096179327

R5BIO-014 38.64733915 -119.9989484 8/16/2001 0.977102039 R5BIO-015 38.7784003 -119.997197 7/12/2001 0.793221607 R5BIO-022 39.01894738 -120.153041 7/24/2001 1.042116768 R5BIO-024 38.72996701 -120.0177647 7/25/2001 0.8880305 R5BIO-027 38.85406534 -119.9482172 7/30/2001 0.967334731 R5BIO-029 39.14052517 -120.1970268 7/31/2001 0.707202417 R5BIO-033 38.14916501 -120.1223999 8/29/2001 0.91533553 R5BIO-035 41.09204922 -122.1126819 9/11/2001 1.037886317


Station Latitude Longitude Sample Date CSCI R5BIO-038 38.4161343 -119.7399543 8/22/2001 0.974600157 R5BIO-042 41.9159344 -120.887592 6/26/2001 0.668536768 R5BIO-047 41.44264631 -121.0153884 6/27/2001 0.762690968 R5BIO-048 41.36145387 -120.2821169 6/28/2001 0.99658012 R5BIO-049 41.09566689 -122.1359859 8/4/2010 1.144150714 R5BIO-053 39.13877156 -120.4756121 9/7/2010 1.227320603 R5BIO-108 39.72102787 -120.6394208 6/28/2000 0.998476955 R5BIO-109 39.73210882 -120.7249355 6/29/2000 0.976732865 R5BIO-111 40.21188667 -121.2724044 7/5/2000 0.702136836 R5BIO-112 40.38537866 -121.340849 7/5/2000 0.94538793 R5BIO-116 40.20222187 -121.5090257 8/24/2010 0.99884899 R5BIO-117 41.15743124 -120.2622529 7/11/2000 0.796419244 R5BIO-118 41.30003671 -120.8612791 7/11/2000 1.056798783 R5BIO-127 39.58466265 -120.810289 7/24/2000 0.921001176 R5BIO-131 39.06366327 -120.2683467 7/26/2000 0.801992449 R5BIO-132 39.38095934 -120.2383368 7/26/2000 0.992082694 R5BIO-150 38.51109304 -120.3374329 8/18/2000 1.15992233 R5BIO-155 40.11766094 -120.4740452 9/6/2000 0.623139306 R5BIO-157 39.37761871 -122.6532221 9/18/2008 1.0560797 R5BIO-162 40.18694974 -121.3068337 9/19/2000 1.09435306 R5BIO-201 37.42500022 -119.5962201 10/8/2009 0.956727438 R5BIO-204 37.52424451 -119.4645009 6/29/2000 1.019429586 R5BIO-206 37.68936741 -119.7050909 7/6/2000 0.983668636 R5BIO-212 36.14557602 -118.4917868 7/20/2000 1.123663073 R5BIO-215 35.48086346 -118.6802004 6/10/2008 0.883769658 R5BIO-221 37.41317728 -118.9079622 9/29/2010 1.184817084 R5BIO-225 36.3498534 -118.7507626 8/15/2000 0.964864231 R5BIO-227 36.61141529 -118.6998413 8/16/2000 0.858633577 R5BIO-230 37.88272443 -119.7946137 8/23/2000 1.059170864 R5BIO-231 37.53798272 -119.6168397 8/29/2000 0.902510058 R5BIO-232 37.75931535 -119.5289737 7/16/2008 0.927200456 R5BIO-233 37.80705579 -119.8534456 8/30/2000 1.094198767 R5BIO-240 37.73974293 -119.0499841 8/25/2010 0.711015867 R5BIO-248 35.97744185 -118.4857809 7/21/2000 0.888866377 R5BIO-302 34.20161233 -117.2285196 5/2/2007 0.992561605 R5BIO-304 34.16404977 -116.8330076 6/21/2000 1.017983568 R5BIO-308 34.18395889 -117.6269622 6/13/2005 0.861830835 R5BIO-309 33.68274449 -117.5016991 6/14/2000 0.892396666 R5BIO-311 34.24961498 -117.5179965 6/14/2005 0.841683549 R5BIO-315 34.26557823 -117.7475447 5/28/2008 1.205659683


Station Latitude Longitude Sample Date CSCI R5BIO-319 34.25156861 -117.8835433 5/28/2008 1.075047561 R5BIO-320 34.53901008 -119.1611891 6/15/2010 0.996857134 R5BIO-321 34.67439327 -119.2975368 6/16/2010 1.243809651 R5BIO-323 34.5175101 -119.3791833 8/9/2000 1.178540835 R5BIO-326 36.21301111 -121.5350733 6/22/2010 1.261076008 R5BIO-327 34.74644254 -119.70575 8/23/2000 1.061803531 R5BIO-417 40.21081729 -120.6345839 7/27/2000 0.74890452 R5BIO-430 40.63805326 -123.3220037 10/1/2008 1.199752665

R5BIO-6484 32.76157981 -116.4514524 5/19/2001 0.792429633 R5BIO-6498 33.38820309 -117.3255485 6/6/2001 0.951686792 R5BIO-6514 33.00107907 -116.7097707 8/24/2010 0.713637154 R5BIO-6516 33.12631878 -116.8049529 4/21/2009 0.864905977 R5BIO-6528 33.35116829 -116.9131115 5/22/2001 0.816588817 SA-C-06-035 34.07771232 -116.8766197 6/22/2006 0.920858103 SEDTMDL_1 38.40735543 -119.7993421 8/31/2010 1.014849

SEDTMDL_10 37.4111117 -119.5952149 8/14/2006 1.205970489 SEDTMDL_103 38.81353832 -120.2575245 7/12/2007 1.214485819 SEDTMDL_106 37.64226733 -119.0827001 7/3/2007 1.219575876 SEDTMDL_107 38.40328038 -119.7955075 8/18/2006 1.150641237 SEDTMDL_108 37.86877335 -119.9041865 8/15/2006 0.912926516 SEDTMDL_114 37.39678749 -119.5654995 6/18/2007 1.040208141 SEDTMDL_14 36.00815097 -118.5059527 6/4/2007 1.097983132 SEDTMDL_16 36.13487062 -118.4728818 6/6/2007 0.811228652 SEDTMDL_19 36.05549985 -118.1317604 6/7/2007 1.097661278 SEDTMDL_20 36.75293395 -118.8167064 7/8/2008 1.217256995 SEDTMDL_21 40.34786078 -121.5917501 9/17/2008 1.268825214 SEDTMDL_25 35.99761016 -118.5355973 6/4/2007 1.187525066 SEDTMDL_26 36.06754475 -118.4911458 6/5/2007 1.230898903 SEDTMDL_27 35.9006252 -118.3695008 8/24/2011 1.142948454 SEDTMDL_28 35.96904448 -118.5210689 6/5/2007 0.897685138 SEDTMDL_29 36.75796259 -118.8954995 6/21/2007 0.897428628 SEDTMDL_3 37.24243778 -118.9465541 8/30/2011 0.92892372

SEDTMDL_31 39.10839976 -120.1889879 7/20/2006 0.904859835 SEDTMDL_32 38.56653968 -119.6270868 6/29/2007 1.110416768 SEDTMDL_33 39.04347 -120.1477479 7/19/2006 1.131448183 SEDTMDL_34 38.13438453 -119.2353239 7/2/2007 1.000526338 SEDTMDL_36 39.48765098 -120.3678075 6/29/2007 0.996136591 SEDTMDL_38 38.32829536 -119.4494959 7/1/2007 1.228125072 SEDTMDL_5 37.2410335 -119.2436192 8/8/2006 1.088273088

SEDTMDL_52 38.31275402 -119.5482306 9/14/2006 1.082966445


Station Latitude Longitude Sample Date CSCI SEDTMDL_54 37.75387057 -118.9653414 9/13/2006 1.019785587 SEDTMDL_6 37.90501576 -119.9966407 8/16/2006 1.163075534

SEDTMDL_62 36.00338972 -121.3914057 5/19/2007 0.836071691 SEDTMDL_77 41.62399958 -122.0643658 9/16/2008 1.009292674 SEDTMDL_78 39.15233858 -120.4006295 7/11/2007 0.918437612 SEDTMDL_84 41.83256737 -120.3000025 8/7/2007 1.002775436 SEDTMDL_86 39.8063146 -121.0426623 7/18/2007 1.016546061 SEDTMDL_89 39.85235124 -120.7263527 8/13/2008 1.011278075 SEDTMDL_9 37.46040568 -119.4163099 7/9/2008 0.950499942

SEDTMDL_91 39.71175216 -120.5342662 7/19/2007 0.996761776 SGUR003 34.27283483 -117.8896685 6/14/2005 0.668176581

SGUR00428 34.24597199 -118.0486816 8/3/2011 0.996146919 SGUR006 34.25081661 -117.8226547 6/16/2005 0.655574296 SGUR010 34.19269141 -117.8651493 6/18/2010 0.811964809 SGUR103 34.24276754 -118.0497193 8/3/2011 1.182188991

SMC00271 34.54893177 -119.1656226 6/4/2009 1.108269201 SMC00469 33.52987943 -117.4090544 5/13/2009 0.782205995 SMC00538 32.78101413 -116.6330177 5/5/2009 0.985655217 SMC00960 34.59076021 -118.4617574 6/2/2010 0.789209616 SMC01196 34.27837802 -118.0296019 7/14/2010 0.655296245 SMC01424 34.24011836 -117.8161687 7/1/2009 0.933917944 SMC01705 33.60305489 -117.5097303 5/14/2009 0.820711938 SMC01883 34.7162595 -119.0188461 6/21/2010 0.985856656 SMC03643 34.56208805 -119.1288699 6/24/2010 1.096844145 SMC04383 34.50221069 -119.3558715 6/9/2010 1.072155545 SMC05243 34.59248244 -119.2512309 6/10/2010 0.690501412 SMC06467 33.79117928 -117.7176845 6/28/2010 0.954521966

SMCR8_057 34.25073228 -117.5485094 5/18/2008 1.00878903 SMCR8_068 34.27166708 -117.6471041 6/10/2008 0.846594419 SMCR8_074 34.08807939 -116.8839219 6/25/2008 0.732436894 SMCR8_080 34.15055585 -116.8842658 6/24/2008 1.066251778 SMCR8_118 34.28336603 -117.4820757 6/11/2008 0.631837646 VENTURA13 34.50144493 -119.3480016 9/11/2006 0.830223653 VENTURA14 34.50465551 -119.3741874 9/11/2006 0.925247071


State of California Technical Memorandum on the California Stream Condition Index


Page 1

The California Stream Condition Index (CSCI):

A New Statewide Biological Scoring Tool for

Assessing the Health of Freshwater Streams

Prepared by: Andrew C. Rehn1, Raphael D. Mazor2,1, Peter R. Ode1

1California Department of Fish and Wildlife 2Southern California Coastal Water Research Project

SWAMP Technical Memorandum

SWAMP-TM-2015-0002

September 2015

TABLE OF CONTENTS Objective ................................................................................................................................................................................. 1 Overview ................................................................................................................................................................................. 1

Introduction ........................................................................................................................................................................ 2 Compilation of Data Sets .................................................................................................................................................. 3 Quantifying Natural and Anthropogenic Gradients Across Sites .............................................................................. 4 Building Predictive Models for O/E and MMI ............................................................................................................ 5 Setting Scoring Thresholds for the CSCI ....................................................................................................................... 6 CSCI Performance ............................................................................................................................................................. 7 Calculating the CSCI.......................................................................................................................................................... 8

References .............................................................................................................................................................................. 9 Suggested Citation............................................................................................................................................................. 10 Appendix 1. Stressor and human activity gradients ........................................................................................................ 11 Appendix 2. Natural gradients used as predictors for development of O/E and MMI indices. ........................... 12 Appendix 3. Summary of performance evaluations from Mazor et al. (in press) ..................................................... 13

OBJECTIVE The objective of this technical memo is to summarize the development, features and use of SWAMP’s next-

generation index for monitoring stream health in California.

OVERVIEW California’s dramatic environmental diversity supports a broad array of natural stream types throughout the

state. Bioassessment of freshwater stream and rivers is especially challenging in such a region because the

reference condition, or the benchmark of biological condition expected when human disturbance in the

environment is absent or minimal, varies greatly among natural stream types. Previous indices used by

monitoring programs were developed on a regional basis to help partition the state’s environmental diversity,

but statewide assessments were confounded by different criteria used in different regions. The CSCI, which

translates complex data about individual benthic macroinvertebrates (BMIs) found living in a stream into an


California Stream Condition Index (CSCI) Technical Memorandum

Page 2

overall measure of stream health, was developed specifically to address some of the shortcomings of earlier

indices. First, the CSCI was developed with a much larger, more representative data set that makes it applicable

statewide and that covers the broad range of environmental variability among natural stream types. Second, the

CSCI sets biological benchmarks for a site based on its site-specific environmental setting. Finally, the CSCI

combines two separate types of indices, each of which provides unique information about the biological

condition of a stream: a multi-metric index (MMI) that measures ecological structure and function, and an

observed-to-expected (O/E) index that measures taxonomic completeness. Together they provide multiple

lines of evidence about the condition of a stream, providing greater confidence in results than a single index.

Figure 1. Extreme natural gradients in California result in a high degree of natural variation in biological expectations among stream types.

Introduction California contains continental-scale environmental diversity within its borders, encompassing some of the

most extreme gradients in elevation, climate and geology found in the United States (Figure 1). It supports

temperate rainforests in the North Coast, alpine forests and meadows in the mountains, deserts in the east, and

chaparral, oak woodlands, and grasslands with a Mediterranean climate in most remaining parts of the state.

Such great physiographic complexity correspondingly supports a broad array of natural stream types, which in

turn hosts a rich diversity of aquatic organisms. Bioassessment, which is the science of using aquatic organisms

as indicators of stream health and function, is greatly complicated in such regions because the reference

condition varies greatly among natural stream types (Figure 2).

Previous indices used by stream monitoring programs to rank or “score” biological condition at sampling sites

relative to reference conditions were developed for specific subregions of California as a means of partitioning

the state’s environmental variability (e.g., Ode et al. 2005, Rehn 2009). While this approach allowed the

establishment of defensible impairment thresholds within regions, comparison among regions was confounded

for two closely related reasons: 1) the criteria used to define reference conditions varied among regions, and 2)

each index was composed of different metrics so that deviation from the reference benchmark was not

equivalently measured in all settings and did not have the same ecological meaning across the entire state.



Page 3

Moreover, some portions of the state and certain stream types were unrepresented. To support the ongoing

development of California’s statewide Biological Integrity Plan, the State Water Board funded the development

of a new index that was applicable statewide, encompassed as much natural environmental variability as

possible, and allowed consistent and equivalent scoring thresholds in all regions of the state.

Figure 2. Bioassessment is complicated in regions with natural environmental complexity because the reference condition varies greatly among natural stream types.

The California Stream Condition Index (CSCI) is a new statewide biological scoring tool that translates complex

data about benthic macroinvertebrates (BMIs) found living in a stream into an overall measure of stream health.

Finalized in 2013 and recently accepted for publication (Mazor et al. in press), the CSCI represents the latest

generation of biological indicators for assessing stream health in California. The CSCI combines two separate

types of indices, each of which provides unique information about the biological condition at a stream: a multi-

metric index (MMI) that measures ecological structure and function, and an observed-to-expected (O/E) index

that measures taxonomic completeness. Unlike previous MMI or O/E indices that were applicable only on a

regional basis or under-represented large portions of the state, the CSCI was built with a statewide dataset that

represents the broad range of environmental conditions across California. The CSCI provides consistency and

accuracy in the interpretation of biological data collected by both statewide and regional monitoring programs

and will be the basis of the new statewide Biological Integrity Plan. This memo summarizes the development

and key features of the CSCI, including its performance characteristics, recommended scoring thresholds, and

data requirements for its use. Full details of CSCI development can be found in Mazor et al. (in press).

Compilation of Data Sets Benthic datasets for CSCI development were compiled from more than 20 federal, state, and regional

monitoring programs that sampled streams sites between 1999 and 2011. Standardization of BMI data was

necessary because the level of taxonomic effort used to identify organisms and the number of specimens

identified per sample varied among programs. Somewhat different data standardization approaches were used

for the MMI and the O/E, but to accommodate data reduction that occurs during standardization, 600-count

BMI samples identified to “Level 2a” as defined by the Southwest Association of Freshwater Invertebrate

Taxonomists (SAFIT, Richards and Rogers 2011) were required1. BMI samples with insufficient numbers of

organisms or taxonomic resolution were excluded from analyses. A final data set from 1,985 sites met all

requirements and was used for development and evaluation of both the O/E and MMI indices (Figure 3).

1 SAFIT Level 2a identifies most taxa to species and Chironomidae to subfamily.



Page 4

Quantifying Natural and Anthropogenic Gradients Across Sites Environmental data were gathered from multiple sources to characterize natural and anthropogenic factors

known to affect benthic communities such as climate, elevation, geology, land cover, road density, hydrologic

alteration, and mining. GIS variables that characterized natural and relatively stable environmental factors (e.g.,

topography, geology, climate) were used as predictors for O/E and MMI models, whereas variables related to

human activity (e.g., land use, road density, etc.) were used to classify sites as reference and to evaluate

responsiveness of O/E and MMI indices to human activity gradients. Most variables related to human activity

were calculated at three spatial scales: within the entire upstream drainage area (watershed), within the

contributing area 5 km upstream of a site, and within the contributing area 1 km upstream of a site

(Appendix 1). Quantifying human activity at multiple spatial scales allowed sites to be screened for both local

and catchment-scale impacts. By contrast, variables used as predictors for O/E and MMI indices were

calculated at either the site (i.e., “point”) scale or the watershed scale, but not at the local (1k and 5k) scales

(Appendix 2).

Figure 3. Distribution of 1,985 sampling sites used in development and validation of the CSCI. White circles are sites that passed reference screens (n = 590; see text) and black circles are sites that failed one or more screening criteria. Major ecological regions are those used as reporting units for the Perennial Streams Assessment (PSA).

Sites were divided into three sets for development and evaluation of indices: reference (i.e., low-activity),

moderate-activity, and high-activity sites. Uniform statewide criteria for defining reference sites were recently

established by Ode et al. (in press; also see Appendix 1) with an emphasis on achieving a balance between

thorough environmental representativeness while still maintaining a pool of “minimally-disturbed” sites sensu



Page 5

Stoddard et al. (2006). Nearly 600 of the 1,985 sites included in the data set for CSCI development passed

reference screening criteria (Figure 3), a fairly high success rate due to an emphasis being placed on data sets

likely to contain high-quality reference sites during data compilation. In addition to good geographic coverage,

the final reference pool also represented several biologically important natural gradients (Figure 4).

Identification of high-activity sites was necessary for MMI calibration (described below) and for performance

evaluation of both MMI and O/E. High-activity sites were defined as meeting any of the following criteria:

≥50% developed land (i.e., % agricultural + % urban) at any spatial scale; ≥ 5 km/km2 road density at any

spatial scale; or riparian disturbance index (W1_HALL of Kaufmann et al. 1999) ≥ 5. Sites not identified as

either reference or high-activity were designated as moderate-activity.

Figure 4. Examples of natural gradients that are important drivers of biological variability and are well-represented by the reference site pool. Unbiased estimates of natural gradient distributions in California were derived from probabilistic surveys conducted between 2000 and 2011 and are shown as kernel density estimates. Values of individual reference sites are shown as small vertical lines. Regions (see Figure 2) are abbreviated as follows: SN = Sierra Nevada, SC = South Coast, NC = North Coast, DM = Deserts + Modoc, CV = Central Valley, CH = Chaparral.

Building Predictive Models for O/E and MMI The CSCI combines two different types of indices that have traditionally been used separately in stream

assessments and provide unique information about the biological condition of a stream; an observed-to-

expected (O/E) index that measures taxonomic completeness, and a multi-metric index (MMI) that measures

ecological structure and function. Predictive modeling has been used in the development of O/E indices since

their inception (Moss et al. 1987), but its use in the development of MMIs is relatively new (e.g., Pont et al.

2009). In each case, modeling improves index performance, but the process through which modeling helps

achieve better performance differs somewhat between the approaches.

O/E indices assess the taxonomic completeness of a site by comparing observed and expected taxa lists. The

taxa expected at a new assessment site, or a “test” site, are predicted by statistical modeling of relationships

between taxonomic composition and natural environmental gradients at reference sites. Biological condition at

a test site is then measured as the number of expected taxa (E) that are actually observed (O), and degradation

of biological condition is quantified as loss of expected native taxa. Modeling relationships between taxonomic

composition and natural environmental gradients produces indices that are more precise compared to null

models where all taxa are assumed to have an equal probability of occurrence at all sites. In addition, the



Page 6

statistical modeling process in development of an O/E index produces site-specific expectations for each

assessment site.

A multi-metric index aggregates several measures of BMI attributes, or metrics (e.g., percent predators, number

of pollution tolerant species, etc.), into a single measure of biological condition. Metrics include measures of

assemblage richness, composition, and diversity, and are chosen based on their responsiveness to human

disturbance gradients and/or their ability to discriminate between reference and degraded condition. The

challenge is that expected metric values at reference sites vary greatly among natural stream types, and natural

gradients often co-vary with human disturbance gradients, thereby confounding metric response to disturbance.

Previous MMIs developed for use in various subregions of California utilized regionalization approaches to

control for the effects of natural variation in biological expectations, where “one size fits all” expectations were

developed within large, mostly geographically defined areas (e.g., chaparral vs. mountains). Regionalization

approaches have often been shown to poorly account for natural variation among sites (Hawkins et al. 2010).

Therefore, models were developed to predict expected metric values at reference sites based on multiple natural

environmental gradients (Appendix 2). Metric residuals (the difference between observed and expected values)

were used as new metric values instead of raw metrics because they measure the range of metric variation after

removing the effect of natural environmental variables. The models developed for reference sites were then

used to predict expected metric values and calculate residuals at moderate- and high-activity sites. The

advantages of this approach are twofold: 1) the expected metric values for any given assessment site are site-

specific; and 2) metric residuals provide a more accurate evaluation of metric response to human disturbance

gradients because they model out the effects of variation across natural environmental gradients. Use of

modeled metrics produced an MMI with much better performance characteristics than un-modeled (null)

metrics.

O/E indices do not require scoring because, as a simple ratio of observed-to-expected taxa, they are already

scaled so that the mean score at reference sites is 1. Scoring is required for MMIs because individual metrics

have different scales and different responses to stress, i.e., as human activity increases, some metrics decrease

while others increase (Blocksom 2003). Scoring transforms metrics to a standard scale ranging from 0 (i.e., most

stressed) to 1 (i.e., similar to reference sites). After scoring, final metrics2 were chosen based on their ability to

discriminate between reference and high-activity sites and by lack of bias among PSA regions (Figure 3). Scores

for the final MMI at each site were then calculated by averaging the scores of the final selected metrics and

rescaling (dividing) by the mean of reference calibration sites. Rescaling of MMI scores ensures that MMI and

O/E are expressed in similar scales (i.e., as a ratio of observed to reference expectations), improving

comparability of the two indices. A combined index (the California Stream Condition Index, CSCI) was

calculated by averaging final MMI and O/E scores.

Setting Scoring Thresholds for the CSCI The CSCI was calibrated during its development so that the mean score of reference sites is 1. Scores that

approach 0 indicate great departure from reference condition and degradation of biological condition.

2 Six metrics representing different aspects of assemblage composition (richness, trophic structure, tolerance, etc.) were chosen for inclusion in the final MMI: Taxonomic Richness, Shredder Taxa Richness, Percent Clinger Taxa, Percent Coleoptera Taxa, Percent EPT Taxa, and Percent Intolerant Individuals.



Page 7

Scores > 1 can be interpreted to indicate greater taxonomic richness and more complex ecological function

than predicted for a site given its natural environmental setting. In practice, CSCI scores observed from nearly

2000 study reaches sampled across California range from about 0.1 to 1.4. For the purposes of making

statewide assessments, three thresholds were established based on the 30th; 10th; and 1st percentiles of CSCI

scores at reference sites3. These three thresholds divide the CSCI scoring range into 4 categories of biological

condition as follows: ≥0.92 = likely intact condition; 0.91 to 0.80 = possibly altered condition; 0.79 to 0.63 =

likely altered condition; ≤0.62 = very likely altered condition.

Figure 5. Distribution of CSCI scores at reference sites with thresholds and condition categories.

CSCI Performance The CSCI had better performance than its null (un-modeled) counterpart in terms of accuracy and bias,

precision, responsiveness, and sensitivity (see Appendix 3 for definitions of performance criteria). For example,

mean regional differences in null CSCI scores at reference sites were large and significant, but were mostly

absent in predictive CSCI scores (Figure 6a). The CSCI also was strongly responsive to human disturbance

gradients, and the response was not confounded by the effects of natural gradients because those effects were

modeled out by the use of metric residuals (Figure 6b).

3 The rationale for these thresholds was to balance Type 1 errors (inferring degradation when it does not exist) and Type II errors

(inferring a site is in reference condition when it is degraded). Similar thresholds have a precedent in bioassessment literature, but other methods for setting thresholds are possible, and if applied, might be equally valid.



Page 8

Figure 6. Examples of CSCI performance: a) Distribution of scores for predictive (black boxes) and null models (red dashed boxes) for the CSCI by geographic region. The horizontal dashed line indicates the expected value at reference sites (i.e., 1). Boxes represent the median, first, and third quartiles. Whiskers represent 1.5 times the interquartile range. Circles and X’s represent outliers. b) Relationship between predictive CSCI score (open circles and solid line) and null CSCI score (gray symbols and dashed line) and percent development in the watershed (= % urban + % ag). Note that the null CSCI has a steeper slope than the predictive CSCI because un-modeled metrics partially respond to natural gradients. By contrast, the predictive CSCI provides a more accurate response to disturbance gradients because the effects of metric variation across natural gradients have been modeled out.

Calculating the CSCI Two types of data are required to calculate the CSCI: biological data and environmental data. Biological data are

generated from benthic macroinvertebrate samples collected in accordance with standard SWAMP protocols

(Ode 2007) and identified to the required taxonomic level of effort. SWAMP currently recommends a 600-

count sample identified by a qualified taxonomist to at least SAFIT level 2a (Richards and Rogers 2015), with

most taxa identified to species and Chironomidae identified to subfamily. Environmental data (e.g., watershed

area, geology, precipitation) are generated by a specialist following standard geographic information system

(GIS) protocols. Interim instructions (Mazor et al. 2015) that describe all steps in calculating the CSCI can be

found at the SWAMP Bioassessment Program website. The first section describes the process for using GIS to

delineate catchment polygons and then calculate environmental predictors (see Appendix 2 for required

predictors). The second section describes the process for using the environmental predictors in conjunction

with taxonomic data to calculate CSCI scores using custom libraries and scripts in the R statistical programming

language. SWAMP is currently developing online tools to generate CSCI scores from user-supplied biological

data and site coordinates, requiring minimal technical expertise.

More information about the SWAMP Bioassessment Program can be found at:

http://www.waterboards.ca.gov/water_issues/programs/swamp/bioassessment. Those wishing to arrange

training in CSCI calculation should contact Calvin Yang: [email protected]


http://www.waterboards.ca.gov/water_issues/programs/swamp/bioassessment/docs/csci_scoring_instruct.pdf

http://www.waterboards.ca.gov/water_issues/programs/swamp/bioassessment/

http://www.waterboards.ca.gov/water_issues/programs/swamp/bioassessment

mailto:[email protected]


Page 9

REFERENCES Blocksom, K. A. 2003. A performance comparison of metric scoring methods for a multimetric index for Mid-Atlantic highland streams. Environmental Management 42: 954-965. Hawkins, C. P., Y. Cao, and B. Roper. 2010b. Methods of predicting reference condition biota affects the performance and interpretation of ecological indices. Freshwater Biology. 55: 1066-1085. Kaufmann, P. R., P. Levine, E. G. Robinson, C. Seeliger, and D. V. Peck. 1999. Surface waters: Quantifying physical habitat in wadeable streams. EPA/620/R-99/003. US EPA. Office of Research and Development. Washington, DC. Mazor, R.D. 2015. Bioassessment of streams in southern California: A report on the first 5 years of the Stormwater Monitoring Coalition’s regional stream survey. Southern California Coastal Water Research Project Technical Report 844. R. D. Mazor, P. R. Ode, A. C. Rehn, M. Engeln, T. Boyle, E. Fintel, and S. Verbrugge. 2015. The California Stream Condition Index (CSCI): Guidance for calculating scores using GIS and R. SCCWRP Technical Report #883. SWAMP-SOP-2015-0004. Mazor, R.D., A.C. Rehn, P.R. Ode, M. Engeln, K.C. Schiff, E.D. Stein, D. Gillett and C.P. Hawkins. In press. Bioassessment in complex environments: designing an index for consistent meaning in different settings. Freshwater Science. Moss, D., T. Furse, J. F. Wright, and P. D. Armitage. 1987. The prediction of macro-invertebrate fauna of unpolluted running-water sites in Great Britain using environmental data. Freshwater Biology 17: 41-52. Ode, P.R. 2007 Standard operating procedures for collecting benthic macroinvertebrate samples and associated chemical and physical data for ambient bioassessments in California. California State Water Resources Control Board Surface Water Ambient Monitoring Program (SWAMP) Bioassessment SOP 001. Ode, P. R., A. C. Rehn, and J. T. May. 2005. A quantitative tool for assessing the integrity of southern coastal California streams. Environmental Management 35: 493-504. Ode, P.R., T.M. Kincaid, T. Fleming and A.C. Rehn. 2011. Ecological Condition Assessments of California’s Perennial Wadeable Streams: Highlights from the Surface Water Ambient Monitoring Program’s Perennial Streams Assessment (PSA) (2000-2007). A collaboration between the State Water Resources Control Board’s Non-Point Source Pollution Control Program (NPS Program), Surface Water Ambient Monitoring Program (SWAMP), California Department of Fish and Game Aquatic Bioassessment Laboratory, and the U.S. Environmental Protection Agency. Ode, P.R., A.C. Rehn, R.D. Mazor, K.C. Schiff, E.D. Stein, J.T. May, L.R. Brown, D.B. Herbst, D. Gillett, K. Lunde and C.P. Hawkins. In press. Evaluating the adequacy of a reference site pool for the ecological assessment of streams in environmentally complex regions. Freshwater Science. Olson, J. R., and C. P. Hawkins. 2012. Predicting natural base-flow stream water chemistry in the western United States. Water Resources Research 48. W02504. doi: 10.1029/2011WR011088 Pont, D., R. M. Hughes, T. R. Whittier, and S. Schmutz. 2009. A predictive index of biotic integrity model for aquatic-vertebrate assemblages of Western U.S. streams. Transactions of the American Fisheries Society. 138: 292-305.



Page 10

Rehn, A. C. 2009. Benthic macroinvertebrates as indicators of biological condition below hydropower dams on West Slope Sierra Nevada streams, California, USA. River Research and Applications 25: 208-228. Richards, A. B., and D. C. Rogers. 2011. List of freshwater macroinvertebrate taxa from California and adjacent states including standard taxonomic effort levels. Southwest Association of Freshwater Invertebrate Taxonomists. Chico, CA. Available from www.safit.org. Stoddard, J. L., D. V. Peck, S. G. Paulsen, J. Van Sickle, C. P. Hawkins, A. T. Herlihy, R. M. Hughes, P.R. Kaufmann, D. P. Larsen, G. Lomnicky, A. R. Olsen, S. A. Peterson, P. L. Ringold, and T. R. Whittier. 2005. An Ecological Assessment of Western Streams and Rivers. EPA 620/R-05/005, U.S. Environmental Protection Agency, Washington, DC.

SUGGESTED CITATION

Rehn, A.C., R.D. Mazor and P.R. Ode. 2015. The California Stream Condition Index (CSCI): A New Statewide

Biological Scoring Tool for Assessing the Health of Freshwater Streams. Swamp Technical Memorandum

SWAMP-TM-2015-0002.


http://www.safit.org/


Page 11

APPENDIX 1. STRESSOR AND HUMAN ACTIVITY GRADIENTS USED TO IDENTIFY

REFERENCE SITES AND EVALUATE INDEX PERFORMANCE.

Sites that did not exceed the listed thresholds were used as reference sites. WS: Watershed. 5 km: Watershed

clipped to a 5-km buffer of the sample point. 1 km: Watershed clipped to a 1-km buffer of the sample point.

Variables marked with an asterisk (*) indicate those used in the random forest evaluation of index

responsiveness. W1_HALL: proximity-weighted human activity index (Kaufmann et al. 1999). Sources are as

follows: A: National Landcover Data Set. B: Custom roads layer. C: National Hydrography Dataset Plus.

D: National Inventory of Dams. E: Mineral Resource Data System. F: Predicted specific conductance (Olson

and Hawkins 2012). G: Field-measured variables. Code 21 is a land use category that corresponds to managed

vegetation, such as roadsides, lawns, cemeteries, and golf courses.

Variable Scale Threshold Unit Data source

* % Agriculture 1 km, 5 km, WS <3 % A

* % Urban 1 km, 5 km, WS <3 % A

* % Ag + % Urban 1 km, 5 km, WS <5 % A A * % Code 21 1 km and 5 km <7 %

* WS <10 % A B * Road density 1 km, 5 km, WS <2 km/km2

* Road crossings 1 km <5 # crossings B, C

* 5 km <10 # crossings B, C B, C * WS <50 # crossings

* Dam distance WS <10 km D

* % Canals and pipelines WS <10 % C C, E * Instream gravel mines 5 km <0.1 mines/km

* Producer mines 5 km 0 mines E

Specific conductance Site 99/1** prediction interval F G W1_HALL Reach <1.5 NA

% Sands and fines Reach % G Slope Reach % G

** The 99th and 1st percentiles of predictions were used to generate site-specific thresholds for specific conductance. Because the model was observed to under-predict at higher levels of specific conductance (data not shown), a threshold of 2000 µS/cm was used as an upper bound if the prediction interval included 1000 µS/cm.



Page 12

APPENDIX 2. NATURAL GRADIENTS USED AS PREDICTORS FOR DEVELOPMENT

OF O/E AND MMI INDICES.

Variable Data Source

Site (i.e., “point”)

Latitude Longitude Elevation A

Catchment Morphology Log watershed area A Elevation Range

A

Climate 10-year (2000-2009) average precipitation at the sample point

B

10-year (2000-2009) average air temperature at the sample point

B

Mean June to September 1971-2000 monthly precipitation, averaged across the catchment

B

Geology Average bulk soil density C Average soil erodibility factor (k) C Log % phosphorus-bearing geology C

Sources:

A. National Elevation Dataset (http://ned.usgs.gov/)

B. PRISM climate mapping system (http://www.prism.oregonstate.edu)

C: Generalized geology, mineralogy, and climate data derived for a conductivity prediction model (Olson and Hawkins 2012)

Predictors that were evaluated but not selected for any model include percent sedimentary geology, nitrogenous geology, soil hydraulic conductivity, soil permeability, sulfur-bearing geology, calcite-bearing geology, and magnesium oxide-bearing geology.


http://ned.usgs.gov/

http://www.prism.oregonstate.edu/


Page 13

APPENDIX 3. SUMMARY OF PERFORMANCE EVALUATIONS FROM MAZOR ET AL.

(IN PRESS)

Aspect Description Indication of good performance

Accuracy and Bias Scores are minimally influenced by natural gradients

- Approximately 90% of validation reference sites have scores above the 10th percentile of calibration reference sites.

- Landscape-scale natural gradients explain little variability in scores at reference sites, as indicated by a low pseudo-R2 for a 500-tree random forest model.

- No visual relationship evident in plots of scores at reference sites against field measurements of natural gradients.

Precision Scores are similar when measured under similar settings

- Low standard deviation of scores among reference sites (one sample per site)

- Low pooled standard deviation of scores among samples at reference sites with multiple sampling events.

Responsiveness Scores change in response to human activity gradients

- Large t-statistic in comparison of mean scores at reference and high-activity sites.

- Landscape-scale human activity gradients explain variability in scores, as indicated by a high pseudo-R2 for a 500-tree random forest model.

Sensitivity Scores indicate poor condition at high-activity sites

- High percentage of high-activity sites have scores below the 10th percentile of calibration reference sites.


Documents

Appendix 1: Stream Biological Objective California Stream … · 2019-02-28 · ample, replacement of native taxa with invasive species may reduce taxonomic completeness, even if