Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
A Federal R&D Evaluation Framework for Influencing Safety Culture Change
in the U.S. Rail Industry
Strengthening Safety Culture Symposium
October 1-2, 2014 Halifax, Nova Scotia
MICHAEL COPLEN Senior Evaluator Office of Research and Development Office of Railroad Policy and Development Federal Railroad Administration
JOYCE RANNEY Safety Culture, Senior Evaluator
Surface Transportation Human Factors Volpe National Transportation Systems Center
1999 Study: Compliance with Railroad Operating Rules and Corporate Culture Influences
3
• Identify, develop, and implement innovative safety culture pilot projects in U.S. railroad industry
• Develop safety culture interventions applicable across different organizations and environments
• Evaluate utilization, impact, and effectiveness of pilot projects
• Where successful, support broad-scale adoption and implementation across industry
Develop a “business case” for safety culture in the railroad industry
Safety Culture in U.S. Railroad Industry Research and Evaluation Strategy, 2001
Approach Carriers Start Date Functions Outcomes
Participative Safety Rules Revision
ACBL, CSXT, KCS, CN-IC
1999 All Operating 30% reduction in reportable injuries Drop in liability claims
Root-Cause Analysis Problem Solving Canadian Pacific 2003 Mechanical
50% drop in injury rates (all injuries)
Clear Signal for Action (CSA)
• Peer-to-Peer Feedback
• Continuous Improvement
• Safety Leadership
Amtrak 2001 Station Services 76% drop in injury rates 71% drop in reportable injuries
Union Pacific 2005 Road Crews 79% drop in L.E. decertification rates 81% drop in derailments
Union Pacific 2006 Yard Crews
65% drop in yard-derailment rates
Confidential Close Call Reporting System (C³RS)
Union Pacific Canadian Pacific New Jersey Transit Amtrak
2007 2008 2009 2011
Road & Yard Crews
31% reduction in derailments at 1 site 90% drop in discipline cases 48% drop in excess-speed reports
Safety Culture Demonstration Pilot Impact Evaluations in U.S. Railroad Industry
4 4
Safety Culture Rail Industry Policy Influences
Organization Policy Changes Union Pacific “Total Safety Culture” Program
Toronto Transit System-wide safety culture change
Canadian Pacific Re-committed to ISROP
RSIA of 2008 Requires “Risk Reduction Programs” Relies on pilot programs for promulgate regulations
Amtrak Began implementing “Safe-2-Safer” Program; Joined C3RS
New Jersey Joined C3RS
BNSF Began major safety culture change initiative
*Body of evidence suggests R&D pilots strongly influenced industry wide
changes.
5
From Pilots to Carrier-wide and Industry-wide Change
6
• Union Pacific - TSC • BNSF - AO • Amtrak – S2S • Norfolk Southern
• Passenger Rail CSA • C3RS • Short Line Safety Institute
Responses
CSA • Yard • Road • Mechanical
C3RS • Yard • Road • Mechanical • Engineering
Pilots Carrier-wide
Industry-wide
FRA’s Confidential Close Call Program (C3RS)
7
8
FRA’s PAX Industry Clear Signal for Action (CSA) Program • No-cost customizable CSA software and training
materials • Low-cost implementation support
• Objective - Enhance Safety and Security in entire company by: • Safety leadership • Peer-to-peer feedback • Continuous improvement
• Simultaneous implementation • All crafts, all locations nationwide
• System Safety Organization
Restructuring
• Safe Align® (Safety Leadership Development)
• BAPP® (Behavioral Accident Prevention Process -- continuous improvement)
• “Approaching Others” (Employee-led training)
- Identifying exposure and risk - Cues for pausing work - providing effective feedback)
Responses
BNSF’s Strategic Safety Culture Initiatives
• Align Field and Safety Functions
• Provide managers with safety leadership skills
• Strengthen . . .
• Engaging workforce
Needs Responses
10
HOW CAN AN EVALUATION FRAMEWORK INFLUENCE SAFETY CULTURE?
11
CIPP Evaluation Model: (Context, Input, Process, Product)
• Context • Input
• Implementation • Impact
Daniel L. Stufflebeam's adaptation of his CIPP Evaluation Model framework for use in guiding program evaluations of the Federal Railroad Administration's Office of Research and Development. For additional information, see Stufflebeam, D.L. (2000). The CIPP model for evaluation. In D.L. Stufflebeam, G. F. Madaus, & T. Kellaghan, (Eds.), in Evaluation models (2nd ed.). (Chapter 16). Boston: Kluwer Academic Publishers.
Stakeholder engagement is key
Types of Evaluation
12
FORMATIVE SUMMATIVE When: • Before or during R&D
projects/programs • After R&D projects/programs
Purpose: To guide: • Program planning • Program design • Implementation strategies
To assess: • Completed projects or project
lifecycles • Accomplishments • Impacts To meet accountability requirements
Primary Focus:
• To improve programs • To prove program merit or worth
13
Evaluation Framework: Roles of Evaluation
Context Inputs Implementation Impact
Formative Evaluation
Identifies:
- Needs - Problems - Assets
Helps set:
- Goals - Priorities
Assesses:
- Alternative approaches
Develops:
- Program plans
- Designs - Budgets
Monitors:
- Implementation - Documents issues
Guides:
- Execution
Assesses: +/- outcomes
Reassess: - Project/program plans
Informs: - Performance metrics - Strategic planning - Policy development
Summative Evaluation
Assesses:
Original program goals and priorities
Assesses:
Original procedural plans and budget
Assesses:
- Execution
Assesses:
- Outcomes - Impacts - Side effects - Cost-effectiveness
Roles and Types of Evaluation
14
• Utility (useful): to ensure evaluations serve the information needs of the intended users.
• Feasibility (practical): to ensure evaluations are realistic, prudent, diplomatic, and frugal.
• Propriety (ethical): to ensure evaluations will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.
• Accuracy (valid): to ensure that an evaluation will reveal and convey valid and reliable information about all important features of the subject program.
• Accountability (professional): to ensure that those responsible for conducting the evaluation document and make available for inspection all aspects of the evaluation that are needed for independent assessments of its utility, feasibility, propriety, accuracy, and accountability.
* The Program Evaluation Standards were developed by the Joint Committee on Standards for Educational Evaluation and have been accredited by the American National Standards Institute (ANSI).
15
Evaluation Standards* Guiding principles for conducting evaluations
Evaluative Framework – Safety Culture Illustrative Questions
Context Inputs Implementation Impact
Formative Evaluation
What are the highest priority needs for improving safety and safety culture? What is the existing culture and the context of that culture?
Given the need for safety culture change, what are the most promising alternatives? How do they compare (potential success, costs, etc.)? How can this strategy be most effectively implemented? What are some potential barriers to implementation?
To what extent is the program proceeding on time, within budget, and effectively? Is the program being implemented as designed? If needed, how can the design be improved?
To what extent are intended users (states, organizations, public) using the program? What other indicators of use, if any, have emerged that indicate the program is being used and behavioral change is occurring? What are some emerging outcomes (positive or negative)? How can the implementation be modified to maintain and measure long-term success?
Summative Evaluation
To what extent did the program address this high priority need?
What strategy was chosen and why, compared to other viable strategies (re. prospects for success, feasibility, costs)?
To what extent was the program carried out as planned, or modified with an improved plan?
To what extent did this program effectively address the need to improve safety and safety culture? Were there any unanticipated negative or positive side effects? What conclusions and lessons learned can be reached (i.e., cost effectiveness, stakeholder engagement, program effectiveness)?
16
Context Inputs Implementation Impact
Formative Evaluation
What is the context leading ASLRRA to commit to the SLSI? What are the industry’s high priority safety needs?
What are the possible tools and alternative approaches for improving safety culture and safety compliance?
What aspects of the original SLSI are being implemented as planned? To what extent are the safety culture and safety compliance assessment tools being used as designed? How can they be improved?
What are the emerging effects and impacts of the SLSI? Are the safety culture and safety compliance assessments being used? Why or why not?
Summative Evaluation
To what extent did the SLSI effort address ASLRRA’s high priority safety needs and goals?
What were the actual activities and outcomes delivered as part of the SLSI? How well did the assessment tools and processes map to the industry safety needs?
To what extent was the SLSI carried out as planned? How strong was the implementation, in collaboration, confidentiality, facilitating use, etc.?
To what extent has the SLSI been successful? How well did the safety culture and safety compliance assessment tools work? What were the outcomes?
Example Core Evaluation Questions: Short Line Safety Institute
Mission: to enhance safety culture and safety compliance of short line and regional railroads through voluntary, non-punitive partnerships
17
Context
Short Line Safety Institute Evaluation Framework
Inputs Outputs
• Short line and regional railroads
• Employees/labor unions
• Management • FRA RRS • Other railroads • DOT at-large • Congress • Citizenry
FRA R&D Team Assessment Tools: • Safety culture • Safety compliance • Interview protocols
Educational Materials: • Employee • Manager
Organizational Plan
Outcomes
• Fully functioning Safety Institute
• Safety action plans • Ongoing assessments • Site-based training
programs • Improved safety culture • Increased safety
compliance • Use of Institute
repository resources • Reduced accidents and
injuries
Priority Improve crude-by-rail transportation safety
Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.
Situation Rapid increase in crude oil production and related incidents
Activities • Needs
assessments • Job analyses • Literature
reviews • Stakeholder
engagement strategies
• Organizational planning
What we invest What we get For whom What we see
FRA R&D Funding • ASLRRA grant and
matching funds • UCONN grant • Volpe IAA
ASLRRA Team • Onsite assessments • Assessment reports • Participant feedback • Buy-in
Context, Input, Implementation, and Impact Evaluation 18
FRA Safety Culture Selected Bibliography • Zuschlag, M., Ranney, J., Coplen, M. (2012). Impact Evaluation of an Organizational Change Program for Union Pacific Road and Yard
Operations Shows Improved Safety Performance and Safety Culture. Safety Science (submitted for publication). • Zuschlag, M., Ranney, J., Coplen, M., Harnar, M. (2012, November).Transformation of Safety Culture on the San Antonio Service Unit
of Union Pacific Railroad. U.S. DOT Federal Railroad Administration. DOT-FRA-ORD-12-16. http://www.fra.dot.gov/eLib/details/L04121.
• Kath, L., Marks, K. & Ranney, J. (2010). Safety climate dimensions, leader–member exchange, and organizational support as predictors of upward safety communication in a sample of rail industry workers. Safety Science, doi:10.1016/j.ssci.2010.01.016 http://www.fra.dot.gov/eLib/details/L03035
• Morrow, S. L., McGonagle, A. K., Dove-Steinkamp, M. L., Walker, C. T. Jr., Marmet, M., and Barnes-Farrell, J. L. (2010). Relationships between psychological safety climate facets and safety behavior in the rail industry: A dominance analysis. Accident Analysis and Prevention, 42(5), 1460-1467. http://www.fra.dot.gov/eLib/Details/L04247
• Coplen, M., Ranney, J. and Zuschlag, M. (2009, September). Decreases in Collision Risk and Derailments Attributed to Changing At-Risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR09-20]. http://www.fra.dot.gov/eLib/details/L01342
• Coplen, M., Ranney, J. and Zuschlag, M. (2009, September). Improved Safety Culture and Labor-Management Relations Attributed to Changing At-Risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR09-19]. http://www.fra.dot.gov/eLib/details/L01462
• Coplen, M. and Ranney, J. (2009, May). Safe Practices, Operating Rule Compliance, and Derailment Rates Improve at Union Pacific Yards with STEEL Process, U.S. DOT Federal Railroad Administration [RR09-08]. http://www.fra.dot.gov/eLib/Details/L04248
• Raslear, T., Ranney, J. and Multer, J. (2008, December). Confidential Close Call Reporting System: Preliminary Evaluation Findings, U.S. DOT Federal Railroad Administration, December 2008 [RR08-33]. http://www.fra.dot.gov/eLib/Details/L04249
• Ranney, J., Wu, S., Austin, C., and Coplen, M. (2008, June). Positive Safety Outcomes of Clear Signal for Action Program at Union Pacific Yard Operations, U.S. DOT Federal Railroad Administration [RR08-09]. http://www.fra.dot.gov/eLib/Details/L04250
• Zuschlag, M., Ranney, J. and Coplen, M. (2008, June). Promising Evidence of Impact on Road Safety by Changing At-risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR08-08]. http://www.fra.dot.gov/eLib/details/L03483
• Ranney, J. and Nelson, C. (Coplen, M. COTR ). (2007).The Impact of Participatory Safety Rules Revision on Incident Rates, Liability Claims, and Safety Culture in the U.S. Railroad Industry. U.S. DOT Federal Railroad Administration, Washington, DC [Report No. DOT/FRA/ORD-07/14]. http://www.fra.dot.gov/eLib/details/L01613
• Zuschlag, M., Ranney, J. and Coplen, M. (2007, February). Clear Signal for Action Program Addresses Locomotive Cab Safety Related to Constraining Signals, U.S. DOT Federal Railroad Administration [RR07-08]. http://www.fra.dot.gov/eLib/details/L03505
• Coplen, M., Ranney, J. and Zuschlag, M. (2007, February). Behavior-Based Safety at Amtrak-Chicago Associated with Reduced Injuries and Cost, U.S. DOT Federal Railroad Administration [RR07-07]. http://www.fra.dot.gov/eLib/details/L03506
• Coplen, M. (2007, January). Proactive Risk Management Safety Approaches for Managing Human-Factors-Caused Accidents in the Railroad Industry: Alternatives to Compliance. Pre-conference Human Factors Workshop #141. Transportation Research Board Annual Conference, Washington, D.C. http://www.fra.dot.gov/eLib/details/L03506
• Lee, M. and Ranney, J. (2006). Example of Investigation Best Practices: Interim Findings from an Evaluation of Canadian Pacific Railways' Investigation of Safety-Related Occurrences Protocol (ISROP), U.S. DOT Volpe National Transportation Systems Center, Cambridge, MA. http://www.fra.dot.gov/eLib/Details/L04251
• Ranney, J. and Lee, M. T. (2006, September). Canadian Pacific Railway Services’ 5-Alive Safety Program Shows Promise in Reducing Injuries, U.S. DOT Federal Railroad Administration [RR06-14]. http://www.fra.dot.gov/eLib/details/L03517
• Ranney, J. and Lee, M. T. (2006, September). Canadian Pacific Railway Investigation of Safety-Related Occurrences Protocol Considered Helpful by both Labor and Management, U.S. DOT Federal Railroad Administration [RR06-13]. http://www.fra.dot.gov/eLib/details/L03518
FRA Safety Culture Selected Bibliography (cont.)
• Ranney, J. and Zuschlag, M. (2006, September). Behavior-Based Safety at Amtrak-Chicago Associated with Reduced Injuries and Costs, U.S. DOT Federal Railroad Administration [RR06-12].
• Coplen, M. (2006, January). Danger, people working: multi-modal lessons on improving safety through work process observations and process improvement methods. Pre-Conference Human Factors Workshop. Transportation Research Board Annual Conference, Washington, D.C.
• Ranney, J., Nelson, C. and Coplen, M. (2005). The Efficacy of Behavior-based Safety in the U.S. Railroad Industry: Evidence from Amtrak-Chicago. Transportation Research Board 85th Annual Meeting, Pp. P06-0633. http://www.fra.dot.gov/eLib/details/L04252
• Ranney, J. and Nelson, C. (2004). Impacts of Participatory Safety Rules Revision in U.S. Railroad Industry: An Exploratory Assessment. Transportation Research Record: Journal of the Transportation Research Board of the National Academies, Washington, D.C., pp 156-163. http://www.fra.dot.gov/eLib/Details/L04253
• Ranney, J. (2004, January). Safety Rules Revisions Impact on Safety Culture, Incident Rates, and Liability Claims in the U.S. Railroad Industry: A Summary of Lessons-Learned, U.S. DOT Volpe National Transportation Systems Center, Cambridge, MA. http://www.fra.dot.gov/eLib/Details/L04258
• Coplen, M. (2003, August). FRA R&D Pilot Safety Initiative: Proactive Preventative Safety. Presentation on behavior-based safety at BNSF labor/management meeting. Alliance, NE.
• Coplen, M. and Ranney, J. (2003, January). The Impact of Safety Rules Revisions on Safety Culture, Incident Rates, and Liability Claims in the U.S. Railroad Industry, U.S. DOT Federal Railroad Administration [RR03-03]. http://www.fra.dot.gov/eLib/details/L03554
• Ranney, J. (2003). Safety Culture: TRB Human Factors in Transportation Workshop 105. Pre-conference workshop presented at the 82nd Transportation Research Board Meeting, January 12, 2003, Washington, DC. http://www.fra.dot.gov/eLib/details/L04259
• Ranney, J. and Coplen, M. (2000). Assessing At-Risk Behavior in Railroad Operations. Presentation at Transportation Research Board Annual Conference. January, 2000.
• Coplen, M. (1999).Compliance with Railroad Operating Rules and Corporate Culture Influences: Results of a Focus Group and Structured Interviews. U.S. DOT Federal Railroad Administration, Washington, DC [Report No. DOT/FRA/ORD-99/09]. http://www.fra.dot.gov/eLib/details/L04185
FRA Safety Culture Selected Bibliography (cont.)
Conclusion: Research and development programs in federal agencies can play a major role influencing industry safety culture, but it is much more than simply publishing results. Incorporate Evaluation as a Key Strategy Tool.
• Ask, then answer, questions that matter.
About processes, products, programs, policies, and impacts Helped identify, develop, and design pilot safety culture implementation projects
• Monitor the extent to which, and the ways, in which projects and programs are being
implemented. What’s working, and why, or why not? Monitored pilot implementations for ongoing improvement
• Measure the outcomes and impacts.
Inform others about lessons learned, progress, and program impacts Documented safety and safety culture outcomes from pilot implementations
• Refine program strategy, design, and implementation.
Where successful programs are confirmed, supports broad-scale adoption across the industry Helped identify industry partners and inform strategy for company and industry-wide scale-up
• Systematically engage key stakeholders to improve program success.
Identify and actively involve intended users Clarify intended uses and potential misuses Increased the utilization, impact, and effectiveness of pilot safety culture project outcomes for broader scale
adoption and sustainability
Contact Information
MICHAEL COPLEN • Senior Evaluator • Office of Research and Development • Office of Railroad Policy and Development • Federal Railroad Administration [email protected] JOYCE RANNEY • Safety Culture, Senior Evaluator • Surface Transportation Human Factors • Volpe, The National Transportation Systems Center [email protected]
23
EXTRA SLIDES
24
Evaluation Standards *Guiding principles for conducting evaluations
Utility (useful)
Feasibility (practical)
Propriety (ethical)
Accuracy (valid)
Evaluation Accountability (professional)
• Evaluator Credibility
• Attention to Stakeholders
• Negotiated Purposes
• Explicit Values • Relevant
Information • Meaningful
Processes & Products
• Timely & Appropriate Reporting
• Concern for Consequences & Influence
• Project Management
• Practical Procedures
• Contextual Validity
• Resource Use
• Responsive & Inclusive Orientation
• Formal Agreements
• Human Rights & Respect
• Clarity & Fairness • Transparency &
Disclosure • Conflicts of Interest • Fiscal
Responsibility
• Justified conclusions & decisions
• Valid Information • Reliable
Information • Explicit Program &
Context Description
• Information Management
• Sound Design & Analyses
• Explicit Evaluation Reasoning
• Communication & Reporting
• Evaluation Documentation
• Internal Metaevaluation
• External Metaevaluation
Note: The Program Evaluation Standards were developed by the Joint Committee on Educational Evaluation and have been accredited by the American National Standards Institute (ANSI). 25
American Evaluation Association (http://www.eval.org) •3000 members in 2001 •over 7700 members today • all 50 states •over 60 countries •$95/year membership, includes
– American Journal of Evaluation – New Directions in Evaluation – online access to full journal articles
Evaluation Resources
26
• Affiliate Evaluation Associations – Washington Research and Evaluation Network (WREN) – Federal Evaluator’s Network
• Evaluation Journals
– American Journal of Evaluation (AJE) – New Directions for Evaluation (NDE) – Evaluation Review – Evaluation and the Health Professions
• The Evaluator’s Institute (http://tei.gwu.edu/courses_dc.htm)
– George Washington University
• The Evaluation Center (http://www.wmich.edu/evalctr/) – Western Michigan University
Evaluation Resources
27
• Commitment from all key stakeholders
• Voluntary, confidential/anonymous, non-punitive participation
• Systematic and objective data gathering, analysis, and reporting
• Problem solving, barrier identification and removal, corrective action process
• Long-term sustaining mechanisms
Common Elements of Successful Safety Culture Change
28
Corrective actions were not just focused on the individuals involved in the event
ISROP results led to system-wide improvements
Safety Alert issued across company in 2004
Updated jacking guidelines prepared in 2006
Safety Culture Rail Industry Policy Influences: ISROP Case Study – System Wide Impact
29
‘‘§ 20156. Railroad safety risk reduction program ‘‘(1) PROGRAM REQUIREMENT.— ... the Secretary of Transportation . . . shall require each railroad carrier ... ‘‘(A) to develop a railroad safety risk reduction program under subsection (d) that systematically evaluates railroad safety risks on its system and manages those risks . . . ‘‘(2) RELIANCE ON PILOT PROGRAM.—The Secretary may conduct behavior-based safety and other research, including pilot programs, before promulgating regulations under this subsection and thereafter. The Secretary shall use any information and experience gathered through such research and pilot programs under this subsection in developing regulations under this section.”
110TH CONGRESS of the United States of America
H. R. 2095 Rail Safety Improvement Act of 2008
Policy Influence at U.S. Congress
Policy Influence at U.S. DOT Safety Council
Safety Culture Action Team
–Safety Culture Research Paper
–DOT Safety Policy Statement
31
DOT Safety
Culture
DOT Actions
Safety Culture:
Industry Organizations
States Public
Actions: Industry
Organizations States Public
Safety Outcomes
The traditional view of Federal intervention
The prevailing view of safety culture
U.S. DOT and Safety Culture
CSA Overview What is CSA?
Safer Culture
Peer-to-peer Feedback
Safer Practices
Continuous Improvement
Safer Workplace
Confidential Data
Safety Leadership
33
Program Evaluations: Company and Industry-wide Safety Culture Change Initiatives*
Organization/Program Company or Industry-wide Changes Confidential Close Call Reporting System (C3RS)
Transitioning from pilots to systemwide program
CSA PAX Operations No-cost training and educational materials
Amtrak “Safe-to-Safer” Program
BNSF
Safe Align® “Approaching Others” (EST/AO) Behavioral Accident Prevention Process (BAPP)® Safety System Organizational Restructuring
ASLRRA Short Line Safety Institute * Ongoing FRA R&D Evaluation Projects
34
• Vision: for the short line and regional railroad industry to be recognized as one of the safest in the world
• Mission: to enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships
• Strategic Goals: • to enhance and improve safety practices • to increase the short line and regional railroad
industry’s culture of commitment to safety
Short Line Safety Institute Vision, Mission, and Strategic Goals Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.
35
• FRA R&D Team (FRA, Volpe, UCONN) Activities: – Tool development: Pilot test, design, and implement protocols, tools, and
procedures for assessing safety culture and safety compliance – Program development: Conduct organizational structure needs assessment,
and recommend appropriate and effective organizational structure for Safety Institute
– Evaluate project development, implementation processes, and ongoing outcomes to inform efforts for larger-scale Safety Institute
• ASLRRA/Safety Institute Team Activities: – Conduct safety culture and safety compliance assessments – Provide safety education, training, and development to managers and
employees – Implement the Safety Institute’s organizational development plan
• Transition: Following Pilot Project evaluation, transition to a permanent, expanded Short Line Safety Institute with embedded continuous improvement
Short Line Safety Institute Pilot Project Overview Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.
36