How Much Pollution Is My Program How Much Pollution Is My Program Preventing?Preventing?

Betsy ElzufonBetsy Elzufon

Larry Walker AssociatesLarry Walker Associates

June 13, 2002June 13, 2002

Tools for Measuring Source Control Tools for Measuring Source Control Program EffectivenessProgram Effectiveness

Water Environment Research FoundationWater Environment Research Foundation

Project #98-WSM-2Project #98-WSM-2

www.werf.org 703-684-2470www.werf.org 703-684-2470

Project GoalsProject Goals

• Identify and develop evaluation tools for residential Identify and develop evaluation tools for residential and commercial pollution prevention programsand commercial pollution prevention programs

• Address misconceptions regarding effectiveness Address misconceptions regarding effectiveness measurementmeasurement

• Develop a framework for incorporating effectiveness Develop a framework for incorporating effectiveness measurement into pollution prevention programsmeasurement into pollution prevention programs

• Assess framework and evaluation tools through Assess framework and evaluation tools through demonstration projectsdemonstration projects

Project conducted in 2 PhasesProject conducted in 2 Phases

• Phase 1 - Develop framework and identify Phase 1 - Develop framework and identify measurement toolsmeasurement tools– Tools to Measure Source Control Program Tools to Measure Source Control Program

Effectiveness (2000)Effectiveness (2000)

• Phase 2 - Assess Tools through Demonstration Phase 2 - Assess Tools through Demonstration ProjectsProjects– Controlling Pollution at Its Source: Wastewater Controlling Pollution at Its Source: Wastewater

and Stormwater Demonstration Projects (2001)and Stormwater Demonstration Projects (2001)

WERF Demonstration ProjectsWERF Demonstration Projects

• SFWPPP Mercury Reduction ProjectSFWPPP Mercury Reduction Project

• Santa Monica New Development Program Santa Monica New Development Program AssessmentAssessment

• Woodland Oil and Grease ReductionWoodland Oil and Grease Reduction

• LACSD Lindane ReductionLACSD Lindane Reduction

• LA County Used Oil CampaignLA County Used Oil Campaign

• Davis Healthy Gardens Program EvaluationDavis Healthy Gardens Program Evaluation

Getting StartedGetting Started

• What are the specific issues to be addressed?What are the specific issues to be addressed?

• What do I already know about these issues?What do I already know about these issues?

• What are my goals, what are we trying to accomplish?What are my goals, what are we trying to accomplish?

• How will I know if I have achieved my goals?How will I know if I have achieved my goals?

• How will I know if my program is having an impact?How will I know if my program is having an impact?

• Can influent concentration changes be attributed solely to Can influent concentration changes be attributed solely to our P2 program?our P2 program?

• What other things can I measure besides concentration What other things can I measure besides concentration changes?changes?

• Are there intermediate steps for which change can be Are there intermediate steps for which change can be measured?measured?

Stages to Environmental Stages to Environmental ImprovementImprovement

• Design and implement programDesign and implement program

• Increase awareness of pollution issuesIncrease awareness of pollution issues

• Change behaviorChange behavior

• Reduce inputs of targeted pollutantsReduce inputs of targeted pollutants

• Improve water quality of dischargesImprove water quality of discharges

• Improve environmental conditionImprove environmental condition

Framework for Developing an Framework for Developing an Effective ProgramEffective Program

• Identify the issueIdentify the issue

• Identify and assess sourcesIdentify and assess sources

• Determine available control strategiesDetermine available control strategies

• Evaluate and prioritize control strategiesEvaluate and prioritize control strategies

• Establish a goalEstablish a goal

• Implement the programImplement the program

• Evaluate effectivenessEvaluate effectiveness

• Modify programModify program

Program Stages Program Stages

• Project planningProject planning– Audience characterizationAudience characterization– Baseline informationBaseline information– Existing program reviewExisting program review

• ImplementationImplementation– Tracking progressTracking progress– Achieving milestonesAchieving milestones– Meeting expectationsMeeting expectations

• CompletionCompletion– Goals achievedGoals achieved– Most successful strategiesMost successful strategies– Future needs/ next stepsFuture needs/ next steps

Effectiveness Measurement Tools Effectiveness Measurement Tools TestedTested

• Planning ToolsPlanning Tools– Estimated Load ReductionEstimated Load Reduction– Review existing informationReview existing information– SurveysSurveys

• Measuring increased awarenessMeasuring increased awareness– Phone surveysPhone surveys– Mailed surveysMailed surveys– Intercept surveysIntercept surveys

• Measuring behavior changeMeasuring behavior change– Sales TrackingSales Tracking– Participation TrackingParticipation Tracking– SurveysSurveys– Site visitsSite visits

• Measuring pollutant load reductions (monitoring)Measuring pollutant load reductions (monitoring)

Planning ToolsPlanning Tools

Estimated Load ReductionsEstimated Load Reductions

Existing Information ReviewExisting Information Review

SurveysSurveys

Guidelines for Developing and Guidelines for Developing and Evaluating a ProgramEvaluating a Program

• Identify and prioritize issues/pollutants Identify and prioritize issues/pollutants

• Prioritize sourcesPrioritize sources

• Baseline informationBaseline information

• Develop a strategy for each identified sourceDevelop a strategy for each identified source

• Assign a goal to each strategyAssign a goal to each strategy

• Identify an effectiveness measure for each sourceIdentify an effectiveness measure for each source

SF Estimated Hg Load ReductionSF Estimated Hg Load Reduction

Estimated Load

(lb/year)% of

Influent Control Strategy

Partici-pation Factor

Loading

Factor

Effective-ness Rating

Estimated Reduction

(lb/yr) Residential SourcesHuman waste (1) 18 20% Uncontrollable 0% 0% 0% 0Laundry Graywater 12.1 14% Graywater Systems 2% 100% 2% 0.2Thermometers 5.9 7% Outreach/ Turn-in Campaign20% 100% 20% 1.2Contact Lens Solution 2.2 2% Public Education 10% 100% 10% 0.2

2.2 2% Business Outreach 30% 100% 30% 0.7Other Residential Sources 0.36 0.40% Public Education 10% 100% 10% 0.036Total Residential 38.6 43% 2.3

Commercial/ Industrial SourcesDental Facilities 26.2 29% Outreach-Selected BMPs50% 90% 45% 11.8

26.2 29% Regulatory Program (4)80% 90% 72% 7.1Medical Facilities/ Schools 5.51 6% Business Outreach 80% 100% 80% 4.4

5.51 6% Sewer Line Cleaning 40% 30% 12% 0.7Laundry 0.74 1% Graywater Systems 30% 100% 30% 0.2Other Commercial 0.4 0.44% Business Outreach 40% 100% 40% 0.2Total Commercial 33 37% 24.3Other Sources 17.5 20%Total Influent Loading 89

Estimated Reductions (% of Influent) All Strategies Implemented 30%Dental Outreach, thermometer turn-in, Medical outreach only 17%

SFWPPP Mercury Reduction ProjectSFWPPP Mercury Reduction Project

• Project planningProject planning– Estimated load reductionEstimated load reduction– Previous dental outreach evaluation (mailed Previous dental outreach evaluation (mailed

survey)survey)

• Dental outreach program site visitsDental outreach program site visits

• Thermometer exchange programThermometer exchange program– participation trackingparticipation tracking– phone surveyphone survey– intercept questionsintercept questions

Santa Monica New Development Santa Monica New Development ProgramProgram

• Co-permittee in LA County Stormwater permitCo-permittee in LA County Stormwater permit

• New Development requirement that projects New Development requirement that projects submit an Urban Runoff Mitigation Plansubmit an Urban Runoff Mitigation Plan

• Goal was to assess program and determine future Goal was to assess program and determine future directionsdirections

• Assessment of process, submitted plans, and Assessment of process, submitted plans, and finished project finished project

Santa Monica Urban Runoff Mitigation Santa Monica Urban Runoff Mitigation PlansPlans

BMP URMP ReviewSubsurface pit 72Retention basin 40Dry well 21Depression pit 19Landscaping 15Permeable pavement 15Planters 12French drain 9Perforated pipeline toleach field

8

Gravel pit 7Gravel trench 5Trees 4Subsurface gravel 3End of pipe treatment 3Catch basin 2Parking lot retention 1Sump 1

New Development Program New Development Program Assessment Results Assessment Results

• Agency staff and building community could be Agency staff and building community could be better educated on BMP design and better educated on BMP design and implementation and URMP requirementsimplementation and URMP requirements

• Building community may benefit from information Building community may benefit from information on wider variety of BMPson wider variety of BMPs

• Workshop developed to address these issuesWorkshop developed to address these issues

Woodland Oil and Grease ReductionWoodland Oil and Grease Reduction

• Existing restaurant control program with high Existing restaurant control program with high compliance ratecompliance rate

• Oil and grease collection system upsetsOil and grease collection system upsets

• Existing program review to identify other sourcesExisting program review to identify other sources

Woodland Oil and Grease AssessmentWoodland Oil and Grease Assessment

Call-outs (1996-1999)

site description

25 Industrial21 low income housing20 low income housing, apartments,restaurant17 residential16 apartments, school, pizza place16 Restaurant, apartments, shopping center16 apartments16 apartments, residential7 warehouses, apartments7 residential7 restaurants6 mall, houses, restaurant6 low income housing4 apartments4 Industrial4 restaurants, hotels3 warehouses3 hospital, school

Residential Outreach Program on

Proper Oil and Grease Handling

Mailed SurveysMailed Surveys

• One page - front and backOne page - front and back

• Pre-addressed, stamped envelopePre-addressed, stamped envelope

• One word answers, check boxesOne word answers, check boxes

• Colorful layoutColorful layout

Dental Survey ResultsDental Survey Results

• 27% of 843 Dental surveys received27% of 843 Dental surveys received

• Best source of environmental information is CDA, Best source of environmental information is CDA, SFDS (80%) NOT government agencies (31%)SFDS (80%) NOT government agencies (31%)

• 60% recalled 1997 brochure60% recalled 1997 brochure

• 66% recycle scrap amalgam. ~50% recycle traps, 66% recycle scrap amalgam. ~50% recycle traps, ~27% recycle vacuum filter waste~27% recycle vacuum filter waste

• Over half of respondents recommend non-Over half of respondents recommend non-amalgam fillingsamalgam fillings

Dental Outreach Program Dental Outreach Program DevelopmentDevelopment

• Work with CDA and SFDS, they co-signed all outreach Work with CDA and SFDS, they co-signed all outreach lettersletters

• Focus on increasing implementation of BMPs regarding Focus on increasing implementation of BMPs regarding recycling of trap and filter waste and waste logrecycling of trap and filter waste and waste log

• Increase awareness of amalgam separatorsIncrease awareness of amalgam separators

• Used targeted outreach/ site visits as outreach strategyUsed targeted outreach/ site visits as outreach strategy

Measuring Increased Awareness Measuring Increased Awareness

Quantitative Surveys (phone, mail)Quantitative Surveys (phone, mail)

Non-quantitative (Intercept)Non-quantitative (Intercept)

Davis Healthy Gardens ProgramDavis Healthy Gardens Program

• Purpose to educate residents regarding impact of Purpose to educate residents regarding impact of pesticide use on local waterways and on general pesticide use on local waterways and on general stormwater issuesstormwater issues

• Outreach included signs, theatre slides, Outreach included signs, theatre slides, workshops, newspaper articles, fan brochureworkshops, newspaper articles, fan brochure

• Awareness SurveysAwareness Surveys– 1996 survey on stormwater, wastewater, 1996 survey on stormwater, wastewater,

hazardous waste disposal awarenesshazardous waste disposal awareness– 1999 phone survey and mail survey 1999 phone survey and mail survey

1999 Phone Survey1999 Phone Survey

• Phone survey repeated questions from 1996 Phone survey repeated questions from 1996 survey to measure changes in awarenesssurvey to measure changes in awareness

• Phone survey added questions on Healthy Gardens Phone survey added questions on Healthy Gardens Program awarenessProgram awareness– Recall of program materialsRecall of program materials– Recall of program messageRecall of program message– Connection between pesticides and local water Connection between pesticides and local water

pollutionpollution

• 140 surveys completed (goal was 250)140 surveys completed (goal was 250)

1999 Mailed Survey1999 Mailed Survey

• Survey repeated Healthy Gardens questions from Survey repeated Healthy Gardens questions from phone surveyphone survey

• Mailed to 500 randomly selected residents w/ a Mailed to 500 randomly selected residents w/ a pre-addressed stamped envelope pre-addressed stamped envelope

• Received 198 responses in 2 weeksReceived 198 responses in 2 weeks

• Combined survey response was 340 responsesCombined survey response was 340 responses

• Mailed survey responses and phone survey Mailed survey responses and phone survey responses were statistically the same so responses were statistically the same so responses could be combinedresponses could be combined

Healthy Gardens Program Healthy Gardens Program AssessmentAssessment

• 47 - 50% of respondents recalled program47 - 50% of respondents recalled program

• 18% remember the programs message18% remember the programs message

• 4 - 17% of survey respondents reported using less 4 - 17% of survey respondents reported using less pesticidespesticides

• Items recalled most often - Logo, North Pond map, Items recalled most often - Logo, North Pond map, theatre slide, demonstration signstheatre slide, demonstration signs

• Locations recalled most often - North Pond, Farmer’s Locations recalled most often - North Pond, Farmer’s Market, movies, storesMarket, movies, stores

Pollution Prevention Program Pollution Prevention Program Assessment - Changes since 1996Assessment - Changes since 1996

• Decrease in respondents reporting using City’s disposal Decrease in respondents reporting using City’s disposal programs (47% to 38% for used oil)programs (47% to 38% for used oil)

• Increase from 2% to 26% of respondents reporting they Increase from 2% to 26% of respondents reporting they don’t use pesticidesdon’t use pesticides

• Increase in awareness of wastewater treatment plant Increase in awareness of wastewater treatment plant (37% to 82%)(37% to 82%)

• Increased awareness that stormwater goes to Davis Increased awareness that stormwater goes to Davis ponds (3% to 14%)ponds (3% to 14%)

• Significant portion still believe stormwater receives Significant portion still believe stormwater receives treatment (39% to 34%)treatment (39% to 34%)

What is an Intercept Survey?What is an Intercept Survey?

• Short interview conducted at a fixed location where Short interview conducted at a fixed location where target audience is likely to gathertarget audience is likely to gather

• Non-quantitativeNon-quantitative

• Important design factorsImportant design factors– Appropriate venueAppropriate venue– Day and time of dayDay and time of day– Oral or written surveyOral or written survey– Person who conducts surveyPerson who conducts survey– IncentiveIncentive

Grocery Store Intercept SurveyGrocery Store Intercept Survey

• Events conducted outside local grocery stores in Events conducted outside local grocery stores in problem areasproblem areas

• Provided outreach materials (potholders, can lids, Provided outreach materials (potholders, can lids, brochures) brochures)

• Conducted survey on grease handling in Spanish Conducted survey on grease handling in Spanish and Englishand English

• Distributed materials at local apartment Distributed materials at local apartment complexescomplexes

• Conducted follow-up intercepts to measure change Conducted follow-up intercepts to measure change in awareness after 3 monthsin awareness after 3 months

Grease Survey ResultsGrease Survey Results

• 19% - 24% pour used cooking oil down the drain19% - 24% pour used cooking oil down the drain

• Apartment dwellers more likely to dispose of oil Apartment dwellers more likely to dispose of oil improperly than in single family homesimproperly than in single family homes

• Low recall of outreach materials (4%)Low recall of outreach materials (4%)

LA County Used Oil CampaignLA County Used Oil Campaign

• Annual campaign to promote used oil recyclingAnnual campaign to promote used oil recycling– Targets Latino & African American men 16-35Targets Latino & African American men 16-35– Bill boards, theatre slides, radio, bus stop signsBill boards, theatre slides, radio, bus stop signs

• Intercept survey conducted at auto parts stores to determine Intercept survey conducted at auto parts stores to determine campaign recall and used oil disposal practicescampaign recall and used oil disposal practices

– Conducted in English and SpanishConducted in English and Spanish

– Saturday morningSaturday morning

– Conducted by males fitting target groupConducted by males fitting target group

– Participants received two lottery ticketsParticipants received two lottery tickets

• 188 of 328 approached completed survey188 of 328 approached completed survey

Intercept Survey ResultsIntercept Survey Results

• 90% disposing of used oil and oil filters properly90% disposing of used oil and oil filters properly

• 65-70% dispose of other automotive fluids properly65-70% dispose of other automotive fluids properly

• 40% recalled advertising regarding used oil recycling40% recalled advertising regarding used oil recycling

• 20% recalled advertising as radio or billboards20% recalled advertising as radio or billboards

• 60% recalled incorrectly TV or auto parts store 60% recalled incorrectly TV or auto parts store advertisingadvertising

Measuring Behavior ChangesMeasuring Behavior Changes

Site Visits Site Visits

Particpation TrackingParticpation Tracking

SurveysSurveys

Sales TrackingSales Tracking

Behavior Change Measurement may Behavior Change Measurement may be very project specificbe very project specific

• Commercial/ business audience - Site VisitsCommercial/ business audience - Site Visits

• Residential - tracking participation in collection Residential - tracking participation in collection campaign, changes in sales patternscampaign, changes in sales patterns

• Workshop, training - quizWorkshop, training - quiz

• Surveys still usefulSurveys still useful

Dental Outreach Materials

Dental Site VisitsDental Site Visits

• 50% of general dental practices visited (34) 50% of general dental practices visited (34)

• 74% recycle scrap amalgam - higher than mail 74% recycle scrap amalgam - higher than mail survey results (67%)survey results (67%)

• 15% used no amalgam - lower than mail survey 15% used no amalgam - lower than mail survey results (23%)results (23%)

• 16% recycle traps properly16% recycle traps properly

• 38% dispose of traps as medical waste38% dispose of traps as medical waste

• 40% report placing >90% non-amalgam fillings40% report placing >90% non-amalgam fillings

San Francisco Thermometer San Francisco Thermometer Collection CampaignCollection Campaign

• Purpose to increase awareness regarding mercury Purpose to increase awareness regarding mercury as a water pollutant and reduce use of mercury as a water pollutant and reduce use of mercury thermometersthermometers

• Outreach included newspaper articles, PSAs, Outreach included newspaper articles, PSAs, media event, direct mail, water bill insertmedia event, direct mail, water bill insert

• Evaluation through participation tracking and Evaluation through participation tracking and phone surveyphone survey

Thermometer Turn-in - Participation Thermometer Turn-in - Participation TrackingTracking

• 2 questions regarding where participants lived and 2 questions regarding where participants lived and

where they heard about programwhere they heard about program

• ResultsResults

– 3300 people turned in 4700 thermometers3300 people turned in 4700 thermometers

– Most effective outreach - Newspaper (42%), Most effective outreach - Newspaper (42%),

Radio (26%), Word-of-mouth (18%)Radio (26%), Word-of-mouth (18%)

– Best collection locations -40% collected at 1 of 9 Best collection locations -40% collected at 1 of 9

fire stationsfire stations

New Development Workshop GoalsNew Development Workshop Goals

• Increase understanding of ordinance and SUSMP Increase understanding of ordinance and SUSMP requirementsrequirements

• Improve knowledge regarding effective post Improve knowledge regarding effective post construction BMPSconstruction BMPS

Santa Monica Workshop EvaluationSanta Monica Workshop Evaluation

• Participants knowledgeable regarding general Participants knowledgeable regarding general Stormwater issues before workshopStormwater issues before workshop

• Increased understanding of certain regulationsIncreased understanding of certain regulations

• >100% increase in number of correctly identified >100% increase in number of correctly identified post-construction BMPspost-construction BMPs

• Increase in types of BMPs mentionedIncrease in types of BMPs mentioned

Workshop Follow-Up Survey Mailed to Workshop Follow-Up Survey Mailed to ParticipantsParticipants

Post-ConstructionBMP

Number in 46Santa Monica

Submittals

Number in 11responses from

workshopparticipants

Drywells 36 1Depression Basin 4 0Planter/Landscaping

4 3

French Drain 2 1Gravel Bed 1 0Catch Basin Insert 1 3Interceptors 0 2Permeable Paving 1 2Cistern 0 1Retention Basin 0 1Swale 0 1Total 49 15

LACSD Lindane ReductionLACSD Lindane Reduction

• Goal to reduce lindane levels in wastewaterGoal to reduce lindane levels in wastewater

• EPA grant to conduct pilot programEPA grant to conduct pilot program

• Conducted outreach targeting health care Conducted outreach targeting health care professionalsprofessionals

• Evaluated using surveys, water quality monitoring Evaluated using surveys, water quality monitoring and sales trackingand sales tracking

Survey Results - Awareness ChangeSurvey Results - Awareness Change

0

20

40

60

80

100

Before

After

Per

cen

t

Before

After

Highly Toxic to Humans Pollutes Environment

Survey Results - Survey Results - Physician Behavior ChangePhysician Behavior Change

0

20

40

60

80

100

Doctors Prescribing Lindane

Before

After

Per

cen

t

Pharmacy Study Results - Pharmacy Study Results - Sales RatiosSales Ratios

0

1

2

3

4

Rat

io L

ind

ane

to N

on

-L

ind

ane

Rem

edie

s

Control Outreach

Bonus Bonus Result!Result!

• AB 2318 LowenthalAB 2318 Lowenthal

• Ban medical uses of lindane in CaliforniaBan medical uses of lindane in California

• Signed by Governor Davis in Sept. 2000Signed by Governor Davis in Sept. 2000

• In effect as of January 1,2002In effect as of January 1,2002

Davis KioskDavis Kiosk Survey Survey

Pollutant Level Reductions Pollutant Level Reductions

Water quality monitoringWater quality monitoring

LACSD Lindane Reduction Project -LACSD Lindane Reduction Project -Wastewater MonitoringWastewater Monitoring

0

10

20

30

40

Control Outreach

Before Before

After

After

Med

ian

Lin

dan

e, p

pt

Davis Healthy Gardens ProgramDavis Healthy Gardens Program

• Stormwater monitoring identified pesticides as Stormwater monitoring identified pesticides as a potential concerna potential concern

• Major source identified as residential useMajor source identified as residential use

• Program developed to educate public regarding Program developed to educate public regarding impact of pesticide use on local waterwaysimpact of pesticide use on local waterways

0

1

2

Nov/23/1995 Feb/28/1997 Jun/6/1998 Sep/12/1999

Rainfall (inches)

diazinon (µg/L)

Chlorpyrifos (µg/L x10)

Healthy Gardens Program

North Pond DataNorth Pond Data

Bottom Line - PLANNING is CriticalBottom Line - PLANNING is Critical

Other useful informationOther useful information

Pollution Prevention Program CostsPollution Prevention Program Costs

Demonstration Project ProgramCosts*

Eval-uationCosts

Evaluation ToolsUsed

SFWPPP ThermometersOutreach Campaign

$75,000* $17,000* Phone surveys,tracking

SFWPPP Dentists – 1997Brochure production and 2000site visits

$75,000* $14,000 Mail surveys, sitevisits

Woodland Oil and GreaseResidential Outreach

$12,000 $4000 2 interceptsurveys

Santa Monica NewDevelopment Workshops

$27,000 $5000 Quizzes

Davis Healthy Gardens $50,000* $22,000* Phone survey, mailsurvey, kiosksurvey, wqmonitoring

LACSD Lindane Reduction PilotProject

$160,000*

$19,000* wq monitoring,sales tracking

LA County 199 Used OilRecycling Campaign

$360,000*

$13,000 Intercept survey

* Includes agency (non-WERF grant costs)

Cost of Effectiveness MeasurementCost of Effectiveness Measurement

Effectiveness Measurement Tool Cost % ofProjectCost

Intercept Survey – Used Motor Oil $14,000 5% Intercept Survey –Kitchen Oil & Grease $4000 20% Mail Surveys – dental practices $10,000 10% Mail Surveys– pesticide education program $5,000 10% Mercury source ID and P2 Plan $8500 NA New Development Program Assessment $10,000 NA Oil & Grease Program Assessment $7600 NA Phone Surveys – mercury thermometers $7000 10% Phone Surveys – pesticide education program $5000 10% Sales Tracking $9000 6% Site visit analysis $3000 20% Thermometer collection tracking $5000 7% Workshop Quizzes $5000 15% WQ Monitoring – (stormwater, pesticides) $7000 14% WQ Monitoring – (wastewater, lindane) $10,000 7%

Fears associated with Effectiveness Fears associated with Effectiveness MeasurementMeasurement

• ExpensiveExpensive

• Negative resultsNegative results

• Ambiguous resultsAmbiguous results

Approaches to reducing survey costsApproaches to reducing survey costs

• Ask less questionsAsk less questions

• Mailed surveyMailed survey

• Intercept surveysIntercept surveys

Approaches to reducing monitoring Approaches to reducing monitoring costscosts

• Less constituentsLess constituents

• Less locationsLess locations

• Grabs only or composites onlyGrabs only or composites only

Avoiding negative resultsAvoiding negative results

• Pre-project planningPre-project planning

• Use more than one evaluation approachUse more than one evaluation approach

• Information gathering Information gathering

• Make comparisonsMake comparisons

Thermometer Phone SurveyThermometer Phone Survey

• 400 thermometer owners surveyed (69% of residents 400 thermometer owners surveyed (69% of residents contacted) before and after campaigncontacted) before and after campaign

• No change in awarenessNo change in awareness

• Good audience characterizationGood audience characterization

• Most likely to own Hg thermometer (over 65)Most likely to own Hg thermometer (over 65)

• 52% of thermometer owners own Hg thermometer52% of thermometer owners own Hg thermometer

• 2 - 3% dispose of Hg down the drain (1.6% from RMS)2 - 3% dispose of Hg down the drain (1.6% from RMS)

• Alternate evaluation tool (participation tracking) Alternate evaluation tool (participation tracking) provided useful informationprovided useful information

Healthy Gardens - What we learnedHealthy Gardens - What we learned

• Reduction in pesticide levels - monitoring results Reduction in pesticide levels - monitoring results ambiguousambiguous

• Survey results provided information about Survey results provided information about intermediate stepsintermediate steps– Increases in awareness of some topics and Increases in awareness of some topics and

apparent behavior changeapparent behavior change– Best locations for outreachBest locations for outreach– Best types of outreachBest types of outreach

Successful Effectiveness Successful Effectiveness MeasurementMeasurement

• Planning is criticalPlanning is critical• Ask the right questionsAsk the right questions• Use more than one approachUse more than one approach• Water quality may lack statistical significance if based on Water quality may lack statistical significance if based on

small samplesmall sample• Sufficient time is necessary to see measurable results Sufficient time is necessary to see measurable results

particularly for voluntary programs and for the general particularly for voluntary programs and for the general publicpublic

• Effectiveness measurement most successful for awareness, Effectiveness measurement most successful for awareness, behavior change, reduction of inputsbehavior change, reduction of inputs

• More work is necessary to make connection between More work is necessary to make connection between source control and discharge reductions or environmental source control and discharge reductions or environmental improvementimprovement

Guidelines for Developing a ProgramGuidelines for Developing a Program

• Identify and prioritize issues/pollutants Identify and prioritize issues/pollutants – Most immediate concernMost immediate concern– Previous experiencePrevious experience

• Prioritize sourcesPrioritize sources– Biggest sourceBiggest source– Easiest to work withEasiest to work with– Best potential for reductionBest potential for reduction

• Baseline informationBaseline information– Existing records reviewExisting records review– Potential partnersPotential partners– Characterize AudienceCharacterize Audience

Guidelines for Evaluating ProgramGuidelines for Evaluating Program

• Develop a strategy for each identified sourceDevelop a strategy for each identified source– Available staff and contract budgetAvailable staff and contract budget

• Assign a goal to each strategyAssign a goal to each strategy

• Identify an effectiveness measure for each source Identify an effectiveness measure for each source – Is there baseline information that can be usedIs there baseline information that can be used– Is there something built into the strategy that Is there something built into the strategy that

we can work withwe can work with– Audience characteristics may be usefulAudience characteristics may be useful

Add table here?Add table here?

Reading MaterialReading Material

• Tools to Measure Source Control Program Tools to Measure Source Control Program Effectiveness (2000)Effectiveness (2000)

• Controlling Pollution at Its Source: Controlling Pollution at Its Source: Wastewater and Stormwater Demonstration Wastewater and Stormwater Demonstration Projects (2001)Projects (2001)

– www.werf.orgwww.werf.org– 703-684-2470703-684-2470