EMSs and Performance Change: What Happens?18% reported positive net quantifiable benefits Improved compliance: 71% eliminated violations 5/29/2002 [email protected] 9 UNC-Chapel Hill Environmental

5/29/2002 [email protected] 1UNC-Chapel Hill

EMSs and Performance EMSs and Performance Change: What Happens?Change: What Happens?

Daniel (David) Edwards, Jr.Deborah Amaral

Richard (Pete) Andrews

University of North Carolina at Chapel HillMSWG Moving to Higher Environmental Performance Using EMS

Workshop, Orlando, Florida, June 3, 2002 http://ndems.cas.unc.edu/


OverviewOverviewProject Objectives and Design

Environmental Performance Changes

Economic Performance Changes

Compliance Changes

Explaining Differences Among Facilities’ Performance

Summary and Conclusions


Project Objectives and DesignProject Objectives and Design


Objectives of StudyObjectives of StudyWhat effects does EMS have on environmental performance, economic performance, regulatory compliance?

For what kinds of facilities, and what kinds of EMSs?

Who are these facilities? – large or small, public or private

What are their prior experiences? – management practices, compliance historiesWhat is the character of the EMS? – outside involvement, transparency, ISO certification


Longitudinal Design: facility-level data, three phases

Project DesignProject Design

Baseline: pre-EMS characteristics, performance, management practices.

EMS Design: Baseline: Performance Updates:

3 Years 1 Year 2 Years

EMS Design: characteristics and process of each facility’s EMS.

Post-EMS Updates (2x): changes in performance and in evolution of EMS itself.


How Many Facilities? How Many Facilities? 91 facilities provided baseline data

61 facilities provided EMS design data

34 facilities provided first update data

Second update in progress (19 facilities to date, more expected)


The 34 facilities that have reported first update data to NDEMS

Facilities in various stages of EMS development and implementation

Median number of years, last baseline year reported to first update: 2.5 years

25 facilities reported environmental performance data

34 facilities reported economic performance data

32 facilities reported compliance data

9 states represented

20 sectors represented

Sample for this AnalysisSample for this Analysis


Highlights of FindingsHighlights of FindingsImproved environmental performance

56% improved at least half of their environmental indicators

64% improved at least half of their environmental indicators related to EMS objectives

Improved economic performance

85% perceived benefits of EMS adoption and implementation

45% improved at least half of their economic indicators

18% reported positive net quantifiable benefits

Improved compliance: 71% eliminated violations


Environmental Performance Environmental Performance ChangesChanges


NDEMS facilities reported six broad categories of EPIs used:

Air EmissionsWastewaterNatural Resource UseProduction Elements (e.g. byproducts, material use)Waste Generation and DisposalSustainability Practices

Environmental PerformanceEnvironmental PerformanceIndicators (EPIs)Indicators (EPIs)


Evaluating Changes in Evaluating Changes in EPIs EPIs (1)(1)

Identified best baseline EPI value: if update value is better, performance has improvedimproved

Identified worst baseline value: if update value is worse, performance has deteriorateddeteriorated

If the update value is neither better than the best, nor worse than the worst, performance is similarsimilar


Evaluating Changes in Evaluating Changes in EPIsEPIs (2)(2)

Two scores for each facility, based on proportion of EPIs with improved performance:

One score for all EPIs

One score for only EPIs related to facility’s EMS Objectives and Targets


EPIs: ResultsEPIs: ResultsAlmost all facilities had some improved indicatorsMore than half of the facilities (56%) -- at least half of all indicators improvedAlmost 2/3 of the facilities (64%) -- at least half the indicators related to objectives and targets improved

• fewer indicators in this group deteriorated • but this is a much smaller group of indicators


Environmental Performance: ALL EPIsEnvironmental Performance: ALL EPIs

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

152

2225139

1085

217

1224184

162320196

1131

1417

Faci

lity

Proportion of Indicators in Performance Category

Worse Same Better

ID Score n17 1.00 614 1.00 31 1.00 23 1.00 2

11 0.75 86 0.75 4

19 0.67 320 0.64 1123 0.60 516 0.60 54 0.53 17

18 0.50 424 0.50 412 0.50 27 0.46 11

21 0.45 95 0.38 248 0.34 65

10 0.31 299 0.26 47

13 0.23 1325 0.20 1522 0.15 132 0.00 4

15 0.00 17


EPI Performance: Objective & Target RelatedEPI Performance: Objective & Target Related

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

9158

1372

25225

121016212446

203

1418191

231117

Faci

lity

Proportion of OBJECTIVE AND TARGET Associated Indicators in PerformanceCategory

Worse Same Better

ID Score N17 1.00 411 1.00 423 1.00 2

1 1.00 219 1.00 118 1.00 114 1.00 1

3 1.00 120 0.80 5

6 0.75 44 0.57 14

24 0.50 421 0.50 216 0.50 210 0.50 212 0.50 2

5 0.33 322 0.25 425 0.17 6

2 0.00 37 0.00 1

13 0.00 88 0.00 10

15 0.00 109 0.00 2


EPI Performance: Case StudyEPI Performance: Case StudyFacility: Privately held auto supplier (rubber products). < 300 employees. Experience with a number of management systems during the baseline period. No non-compliances during baseline.

EMS: Influenced by market pressures. ISO 14001 certified. Template for other facilities within the organization. Developedinternally by all levels of employees and with input from government and local citizens.

Environmental indicators: Eight indicators in four categories (sustainable practices, air releases, waste generation/disposal and natural resource use). Half objectives and targets focused.

Performance: 75% of all EPIs improved, + 100% O&T EPIs


EPI Performance: Case ResultsEPI Performance: Case Results

NoSimilarrecyclingNoSimilarwater/sewer useNoImprovedindustrial waste generationNoImprovednatural gas useYesImprovedelectrical energy usedYesImprovedhazardous waste generationYesImprovednon-HAP emissionsYesImprovedHAP emissions

O&T FOCUSO&T FOCUSPERFORMANCE PERFORMANCE INDICATORINDICATOR


Economic Performance Economic Performance ChangesChanges


Economic Performance : Benefits Economic Performance : Benefits and Costs of EMS Adoptionand Costs of EMS Adoption

Economic Impact Indicators (EIIs) were evaluated and a performance index developed

Quantified costs reported by each facility were subtracted from quantified benefits

Benefits that were perceived but not quantified were also evaluated


Economic Impact Indicators (EII)Economic Impact Indicators (EII)20 facilities reported baseline values for EIIs that could be compared to Update values.

Nearly half (45%) showed improvement in at least half of their economic indicators

More than half (55%) exhibited some improved EIIs, but a few reported worse economic performance

Most EII improvements represented changes in economic analogues to EPIs (e.g. cost of waste disposal, cost of materials, cost of natural resources, recycling dollars generated)


Economic Performance : EIIsEconomic Performance : EIIs

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

18824916151412221035136721191117

Faci

lity

Proportion of TOTAL Indicators in Performance Category

Worse Same Better

ID Score n17 1.00 611 1.00 619 1.00 121 0.80 157 0.80 56 0.67 3

13 0.67 35 0.50 43 0.50 2

10 0.38 822 0.29 72 0.00 31 0.00 1

14 0.00 115 0.00 116 0.00 19 0.00 4

24 0.00 78 0.00 1

18 0.00 1


Quantified Benefits and CostsQuantified Benefits and Costs88% of facilities reported quantifiable costs associated with EMS design and/or implementation. 25% reported quantifiable benefits.

EMS Benefits n Median

Net EMS design benefits 30 : ($54,000)

Net EMS implementation benefits 32 : $ 0

Net benefits 30 : ($40,000)

Six NDEMS facilities (18%) reported positive quantified net benefits. The median net benefit reported was $129,000.


Net Quantified BenefitsNet Quantified BenefitsNet Benefits

(650)(550)(450)(350)(250)(150)(50)50

150250350450

1116172119 1 2 3414 5 6 18 3 1230102831 8 291327222426321525 4 7

Facility

USD

$1,0

00

ID Net11 39916 24517 13221 12619 51

1 52 (3)

34 (13)14 (19)

5 (21)6 (22)

18 (24)3 (30)

12 (33)30 (35)10 (45)28 (63)31 (77)

8 (81)29 (86)13 (105)27 (109)22 (117)24 (129)26 (170)32 (180)15 (232)25 (260)

4 (273)7 (628)


Quantified Benefits: Case Study Quantified Benefits: Case Study Facility: Privately held metal finisher, ~100 employees; some experience with management systems during the baseline period. Several instances of non-compliance during baseline.

EMS: Influenced mainly by marketing, but seeking regulatory benefits. Intends ISO 14001 certification. Developed with input from government, consultants and by employees at all levels.

Design Costs: $10,000 -- staff time, consulting fees

Implementation Costs: $70,000 – staff time, materials, equipment

Total Costs: $80,000


Quantified Benefits: Case Study Quantified Benefits: Case Study Design Benefit: $0 – none reported.

Implementation Benefit: $203,000 – reduced raw materials, waste disposal, fines, monitoring, natural resource use

Net Benefit: $203,000 - $80,000 = $123,000

How were net benefits achieved?Eliminated Chemical Use – new equipment: $8,000 saved/yr.

Material Substitution – decreased rejects: $128,000 saved/yr.

Decrease wastewater – added sensors: $4,500 saved/yr.

Decreased waste volume – added sensors: $12,000 saved/yr.


Benefits Perceived, Not QuantifiedBenefits Perceived, Not Quantified

29 facilities (85%) also reported perceived but unquantified economic benefits associated with EMS design and implementation

Six broad categories of benefits identified (including 27 more specific types)

5/30/02 [email protected] 27

UNC-Chapel Hill

Types of NonTypes of Non--quantified Benefitsquantified Benefits

14%Improved Community Relations

21%Improved Supply Chain Relations

55%Reduced Liability

59%Improved Regulatory Performance

76%Increased Operational Efficiency

93%Increased Management Efficiency

Percent Reporting Benefit Percent Reporting Benefit (n=29)(n=29)

BenefitBenefit


NonNon--quantified Benefits: Examplesquantified Benefits: ExamplesDesign: Increase in management commitment, and in employee awareness and sense of responsibility; process changes, disposal costs, electrical consumption; maintaining customers

Implementation: More involvement by personnel external to the EHS function; decrease in hazardous materials, chemical purchases, waste disposal, power requirements, permit fees, direct labor handling; better utilization of resources for all applications; air permit requirement eliminated; maintained market share


Regulatory Compliance Regulatory Compliance ChangesChanges


Regulatory ComplianceRegulatory Compliance32 facilities reported information on regulatory compliance during the baseline and update period.

57% had no violations either in the baseline or the update

71% of the 14 facilities that had a violation during the baseline had none during the update

11% of the 18 facilities that had no violations during the baseline received at least one NOV during the update

Only 1 facility had a major violation during the baseline and update periods; one additional facility had a major violation during the update. All other violations were classified as minor.


Violation Activity: BaselineViolation Activity: Baseline--UpdateUpdate

4

10BASELINE

No Violations

UPDATENo Violations BASELINE

Violations

UPDATEViolations

14

26

2

16

18

6


Summary : Performance ChangesSummary : Performance ChangesEnvironmental performance: Almost all improved on some indicators, and more than half improved a majority of their indicators

Economic performance: More than half showed improvement in some economic impact indicators; quantified costs > benefits for most but not all; most also noted important unquantified benefits

Compliance: Significant reduction in number of facilities experiencing violations

Variability: Differences in outcomes among facilities


Explaining Differences Among Explaining Differences Among FacilitiesFacilities’’ PerformancePerformance


Explaining Differences: DemographicsExplaining Differences: DemographicsThe following demographic characteristics were associated with improved performance:

SIZE

Large facilities (>300 employees) – higher EPI

More than 50 but less than 1,000 employees – higher EII

OWNERSHIP

Publicly traded and government facilities – higher EPI

Publicly traded and privately held facilities – higher EII


Management System ExperienceManagement System ExperienceFacilities with some baseline experience with management systems

(e.g. TQM, ISO 9000) scored higher on both EPI and EII indexes

n= 7

n=18

0.000n= 70.255NoNo

0.500n=130.564Yes Yes

MedianMedianEII ScoreEII Score

MedianMedianEPI ScoreEPI Score

Prior System Prior System ExperienceExperience


Other Prior EM ExperienceOther Prior EM ExperienceFacilities that used other EM techniques during the baseline period (e.g. P2 Plans) had higher scores on EPIs related to EMS objectives and targets.

0.500n= 12NoNo

0.750n=13YesYes

Median EPI OT ScoreMedian EPI OT ScoreP2 PlanP2 Plan


Explaining Differences: Explaining Differences: Compliance HistoryCompliance History

Of the independent variables examined, a compliance history freeof violations, fines or non-compliances was positively associated with the greatest number of variables that measured subsequent performance.

Facilities with these outstanding compliance histories:

Scored higher on the overall EPI index

Scored higher on the Objective- and Target-Related EPIs

Had higher net benefits


Explaining Differences: ComplianceExplaining Differences: Compliance

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Compliant Non-Compliant

EPI Score

EPI OT Score

n=15

n=8


Explaining Differences: Explaining Differences: Compliance History Compliance History –– cont.cont.

These results do NOT mean that EMSs are unlikely to produce significant improvement by more problematic facilities

Few facilities with major violations included in study

Results show compliance improvement

Only first update data, non-compliant facilities may simply take longer to improve


Explaining Differences: EMS DesignExplaining Differences: EMS DesignEMS design and implementation is a flexible process that allows for variation in process and content. Examples include variations in:

ISO certification intentions

Third party audits

Outside Party involvement

Use of consultants in EMS design

EMS transparency

These variations might also contribute to differing levels of environmental and economic performance.


ISO Certification and EMS AuditsISO Certification and EMS Audits

69% of the facilities were certified or intended to certify to ISO 14001

94% of the facilities audit their EMS, and 72% of those that do audit use third-party auditors

No association with performance results


Outside Party InvolvementOutside Party Involvement85% of the facilities involved external individuals or groups in the design of their EMS

53% used consultants

41% involved government institutions or organizations

12% involved environmental organizations

Facilities developing EMSs without using consultants had median EPI-OT scores nearly twice as high as those using consultants

Involvement by other outside parties was not associated with performance results


EMS TransparencyEMS TransparencyPublic disclosure of the results of the EMS process may increasepublic trust and regulator confidence.

75% of the 34 facilities planned to release objectives and targets to interested parties

39% intended to disclose the results of EMS audits

24% intended to disclose both objective and target lists andEMS audit results

Facilities intending to disclose O&T lists and audit results had higher median EPI-OT scores than non-disclosers


Outside Parties, TransparencyOutside Parties, Transparency

0.50018No Disclosure

0.8007Disclose O&T and Audit Results

16

9nn

0.409

0.636

Median Median EPI scoreEPI score

0.800Developed EMS Internally

0.416Used Consultants

Median Median EPI OT EPI OT scorescoreEMS DESIGN VARIABLEEMS DESIGN VARIABLE


Summary and ConclusionsSummary and Conclusions


SummarySummaryAnalysis of first update data for 34 NDEMS pilot facilities

Overall, improvements in environmental and economic performance and regulatory compliance are evident

In general, facilities with the most prior system management experience and the best compliance histories had the best performance scores

EMS transparency was associated with higher performance scores, and use of a consultant during EMS design was associated with lower scores


Strongest FindingsStrongest FindingsImproved environmental performance : 92% had some improved indicators

Environmental indicators focused on EMS Objectives and Targets were more successful: 64% improved at least half

Compliance improved: 71% eliminated violations reported during the baseline period

Previous management system experience: these facilities had better environmental performance scores

Positive compliance histories: these facilities had better environmental and economic indicator scores


LimitationsLimitationsSmall sample (34 facilities in 20 sectors)

Short time period (first update data)

Cooperating facilities bias

Volunteers: pride and confidence in their performance

Facilities recruited by states and EPA

Varied government incentives for participation (e.g technical and financial assistance, regulatory flexibility)


Continuing ResearchContinuing ResearchSecond update data: summer 2002

Final report & databases publicly available: Fall/Winter 2002-03

Follow-on study in progress:Larger sample of facilitiesRelationships between external incentives, facility motivations, and EMS outcomes and subsequent performance


Special ThanksSpecial ThanksWe are deeply grateful to the participating facilities and statemanagers who have provided data for these analyses

We are also grateful for the sponsorship of the Office of Wastewater Management and other units of the U.S. Environmental Protection Agency

http://ndems.cas.unc.edu

Documents

EMSs and Performance Change: What Happens?18% reported positive net quantifiable benefits Improved compliance: 71% eliminated violations 5/29/2002 [email protected] 9 UNC-Chapel Hill Environmental