Assessing Safety Climate and Culture Examples Aviation ... · Assessing Safety Climate and Culture Examples Aviation, ... Senior Commissioned Junior Enlisted. ... Senior management

2012 Fifth International (HRO) Conference

Anthony [email protected]

May 2012

Assessing Safety Climate and CultureExamples Aviation, Aerospace, and Healthcare

mailto:[email protected]


AGENDA

• Classic Organizational Accidents• Sources of Human Error• High - Reliability Organizations• Organizational and Safety Culture• Assessing Safety “culture” – Climate• HRO – Culture status of Selected Aviation,

Aerospace and US Hospitals• Validation of HRO – Survey Culture Instrument


Aviation and Aerospace Accidents

• Air Florida Washington DC “start up airline”

• ValuJet Florida Everglades “outsourced cargo services”

• Challenger Space Shuttle “risk perception -- decision”

• Columbia Space Shuttle “risk perception--decision”

• TIROS weather satellite “procedural compliance”


Key Organizational Accident Theories

• Incubation Periods (Turner,1978)• The Normal Accident (Perrow, 1984)• The High-Reliability Organization (Roberts, 1993)• Organizational Culture (Shein, 1990)• Safety Culture (Reason, 1990)• Normalization of Deviance (Vaughn, 1997)• Organizational Sense-making (Weick, 1999,

Weick and Sutcliffe 2007)


Sources of Human Errors

• Attitudes about job safety and performance• Culture of Safety is deficient in key areas, such as

procedure compliance and reporting• Inadequate standards, training and monitoring of

practitioner qualifications • Poor communications and information flow• Non-compliance with rules, best practices• Stress, production pressure, inadequate resources


• Failure to instill strong safety values/culture• Pressure to complete a job or meet schedule• Failure to establish or enforce standards• Over-tasking front line personnel- understaffing• Failure to manage fatigue or stress• Inadequate resources to perform job safely• Poor equipment status/reliability• Failure to manage known risks, including high-risk workers

Supervisory Contributions


Angled DecksAviation Safety Center

Naval Aviation Maintenance ProgramRAG (FRS) Concept Initiated

NATOPS ProgramSquadron Safety Program

System Safety Aircraft DesignCRM (crew-resource-mgmt)

Aircrew reviewsRisk management (ORM)Safety climate-culture

776 aircraftdestroyed in1954

Naval Aviation Major (Class A) Flight Mishaps

Fiscal Year

7 aircraftdestroyed

inFY-11

all in flightmishaps

2000-2011


Naval Aviation Risk Mitigation

• Recruitment and selection• Aeromedical screening• Training standardization and qualifications• Command supervision and risk management• Aircrew performance reviews• Human Factors reviews (Boards & Councils)• Crew Resource Management (CRM)• Safety Climate and Culture Assessments


Safety Culture: The simple-minded View

Human Factors, Inc.


High - Reliability Organization (HRO)

• A culture of trust, shared values, and risk mitigating communication processes.

• Communication that provides opportunities for open discussion and improvement.

• Distributed decision-making – Risk decisions made by most qualified

• Individual responsibility, “where the buck for safety stops everywhere.”

(Roberts, 1993)


Emergence of Just Culture Culture Concept

The phrase just culture refers to the principles for achieving a culture in which frontline personnel feel comfortable disclosing errors—including their own—while maintaining professional accountability... A just culture recognizes that individual practitioners should not be held accountable for system failings over which they have no control.

A just culture also recognizes anyone can produce errors as a consequence of predictable interactions between human operators and the systems in which they work. However, in contrast to a culture that turns to blame as its governing principle, a just culture does not tolerate conscious disregard of rules, reckless behavior or gross misconduct (AHRQ 2008)


Organizational Culture

Shared Values (What is important) and Beliefs(How things work) that interact with an organizations structures and control systems to produce Behavioral Norms (The way things work around here).

(Reason 1998)


Safety Culture

• Shared values about what is safe and unsafe• Common beliefs about how to conduct safe

operations• Behavioral norms that govern risk-taking,

everyday procedures and precautions• Transmission of values, beliefs and accepted

practices to others.


A Concise Definition of Safety Culture


High Reliability Organization (HRO) Model1. Safety Process auditing (SPA) – conducts adequate audits and reviews of safety

processes to ensure they are working as intended.

2. Safety Culture & Reward system (SCRS) – creates a “just culture” policy and procedures for open reporting and rewards safe behavior.

3. Risk management (RSK) – has risk management processes in place and continuous risk assessment by employees at all levels.

4. Quality Assurance (QA) best practices – promotes and monitors use of standard procedures and best practices

5. Leadership and Supervision (LDSHP)-- Command Control – clearly communicates safety policies, objectives, and provides active leadership and resources to promote safe operations.

(Adapted from Libuser 1994)


Survey is based on HRO PrinciplesMEASUREMENT AREAS: HRO MODEL

1. Safety Process auditing (SPA)

2. Safety Culture & Reward system (SCRS)

3. Quality assurance (QA)- best work practices

4. Risk management (RSK)

5. Leadership and Supervision (LDSHP)-- Command Control

SAMPLE SURVEY ITEMS

• My [organization] conducts adequate safety reviews and updates.

• Anyone who intentionally violates a safety rule is swiftly corrected.

• My [organization] has a reputation for high quality performance.

• I believe that our leaders have a clear picture of operational risks.

• My [organization] is genuinely concerned about safety.

HRO model is supported by validation tests.


Survey based on HRO PrinciplesRoberts and Libuser HRO Model

(Adapted by Ciavarelli, 1997-2007)

1. Safety Process auditing

2. Safety Culture & Reward system

3. Quality assurance- best work practices

4. Risk management

5. Leadership and Supervision-- Command Control

Weick and Sutcliffe HRO Model (2007)

Sensitivity to operations- Awareness of operations – Mindfulness

Preoccupation with failure Leadership Commitment - promotion of a “just culture”

Reluctance to simplify – practical work processes and valid procedures

Deference to expertise – most qualified make risk decisions – management listens to workers as part of risk mitigation

Resilience – trained to detect and respond to failure and error recovery


High Reliability Organizational (HRO) Model

SPA SCRS RSKM QA LDSHP


Copyright © HFA

SPA QA SCRS RSK MNGT LDSHP

Copyright 2001-2007 Human Factors Associates, Inc.

Diagnostic Feedback: Fictitious Data


Normative Data: Aviation or Aerospace

3.05%6.60%

14.23%

42.56%

33.47%

9.65% Problematic

76.03% Favorable

Survey Return Rate: 50%

2012 Fifth International (HRO) ConferencePREDICTIVE VALIDITY: Navy Safety Climate SurveyRelationship between Safety Climate and Accidents

• A study conducted by the US Navy in 2006, found that the survey item average for Risk Management, a category of the High Reliability Organization (HRO) and survey-climate scale for CSA, was a good indicator of accident risk.

• Those squadrons that took the survey were divided into four quartiles based upon their average ratings for the survey’s Risk Management items. The figure on the right shows that squadrons with the lowest HRO-climate ratings had significantly more accidents. Lowest

ClimateRatings

Highest ClimateRatings

MiddleClimateRatings

Schimpf and Figlock 2006

2012 Fifth International (HRO) ConferenceExample of Demographic Statistical Analysis

Higher Ranking Personnel Give Higher Ratings

Rank differences statistically significant (p<.001)

Over-committedShort

Resources

Survey Items

Military RankSenior Commissioned

Junior Enlisted


BUT -- What about HRO and Healthcare?

2012 Fifth International (HRO) ConferenceProblematic %: Hospital Aviator Study Data Taken 1989 – 2001

Gaba, Singer, Sinaiko, Bowen and Ciavarelli (2003)

(Gaba, Singer et al 2003)

Survey Item

14. Senior management supports a climate that promotes patient safety.20. Senior management has a clear picture of the risks associated with patient care.22. My unit take the time to identify and assess risks to ensure patient safety.11. Patient safety decisions are made by the most qualified people, regardless of rank or hierarchy,17. I have received sufficient training to enable me to address patient safety problems.8. My performance is evaluated against defined safety standards.

OVERALL COMPARISON

NavalAviation

2.71.9

1.6

4.0

2.2

5.1

5.6

Hospitals

16.121.9

10.1

18.5

11.3

16.0

17.5


Survey Item

3. Senior management supports a climate that promotes patient safety.

4. Senior management has a clear picture of the risks associated with patient care.

5. My unit take the time to identify and assess risks to ensure patient safety.

24. Patient safety decisions are made by the most qualified people, regardless of rank or hierarchy,

30. I have received sufficient training to enable me to address patient safety problems.

31. My performance is evaluated against defined safety standards.

OVERALL COMPARISON:

Naval Aviation

2.6

2.2

1.7

3.2

1.4

3.1

4.1

Hospitals

9.6

14.2

6.7

22.1

6.9

10.8

12.1

Problematic %:Hospital Aviator Study Data Taken 2006-2007(Singer, Rosen, Zhao, Ciavarelli, and Gaba 2010)

2012 Fifth International (HRO) ConferenceNormative Benchmark Comparison Across Domains

Comparison % Problematic MeanMajor Airlines [Data restricted by Airline Clients]Commuter Airlines [Future data needs]Freight Transport [Future data needs]Business Jet [Future data needs]Financial Services [Future data needs]

US Hospitals 12.1 unavailableUS Navy 4.10 unavailableOil and Gas helicopter transport 18.5% 3.5Air medical One 8.9% 4.0Air medical Two 7.5% 4.0Energy/Power 10.6% 3.8Aerospace 6.4% 4.1


• Survey process for use as a risk assessment tool – validation of survey instrument is continuous.

• Approach incorporates a complete measurement, analysis and diagnostic display “dashboard” system for supervisors that provides diagnostic feedback and normative “benchmark” comparisons for a given company, industry or domain.

• Survey process was successfully used by military and civilian organizations, including helicopter transport, aerospace, airline, electric power, and critical medical care facilities. Selected survey items were used in hospital studies.

• Approach applies to other high-risk industries, that desire to asses their

• HRO performance effectiveness and risk exposure.

Demonstration Web Site: https://www.hfa-clients.com/demosite/login.html

Summary and Conclusions

https://www.hfa-clients.com/demosite/login.html


US Navy Safety Climate SurveyStatistical Validation

OPTIONAL SLIDES

• Measurement Reliability Test

• Construct Validity

• Predictive Validity


Reliability Tests for US Navy Command Safety Assessment (CSA)

• Cronbach-Alpha -- .98

• Guttman Split-half -- .95

CSA survey is highly reliable.

Sengupta 2000

2012 Fifth International (HRO) ConferenceConstruct Validity: Factor Analysis of US Navy Command Safety Assessment Survey

Purpose: Establish factor structure of survey instrument -- key components:

• Factor 1: Safety Climate (46% of variance)• Factor 2: Hi-risk aviator management (4%)• Factor 3: Safety program effectiveness (4%)• Factor 4: Resource adequacy (3%)• Factor 5: Risk management effectiveness (3%)

About 60% of variance accounted for by less than 50 items.

Ciavarelli 2005, Sengupta 2000


MCAS verses US Navy Accident Frequency

Safety climate Metric comparing MCAS results to individual unit mishap occurrence within 24 months after survey completion

• 168 squadrons

• August 2000-April 2004

31

Lowest quartile had nearly twice number of accidents as top quartile

(Schimpf and Figlock 2006)


Summary of Relationship• Units with survey average scores one sigma less than average, the

risk of mishap increases 28 %

• Units with survey average scores two sigma less than average, the risk of mishap increases 65 %

• Units with survey average scores three sigma less than average, the risk of mishap increases over a factor of 2

• Units with survey average scores one sigma above average, the risk of mishap decreases 22 %

• Units with survey average scores two sigma's above average, the risk of mishap decreases 39 %

Kinzer 2004


Civil Aviation Survey Validation Testing

• Civilian Aviation Organizational Safety Effectiveness Survey (OSES) -- derived from US Navy CSA

• Data Selected from large civilian aviation database

• Over 3000 survey takers included in a Factor Analysis Study of OSES completed in 2009.

• Results supported a five factor solution -- consistent with HRO Model used in OSES (see slide 15).

2012 Fifth International (HRO) ConferenceReferences and Source Documents


REFERENCES AND SOURCE DOCUMENTS

Deal, T.E., Kennedy, A.A. (1982). Corporate cultures: The rites and rituals of corporate life. Reading, MA: Addison-Wesley Publishing Company.Flin, R., Mearns, K., O'Connor, P., and Bryden, R., (2000). Measuring safety climate: Identifying the common features. Safety Science, 34, 177-192.Gaba, D., Singer, S., Sinaiko, A.D., Bowen, J.D., and Ciavarelli, A.P. (2003). Differences in safety climate between hospital personnel and naval aviators. Human Factors and Ergonomics, 45,(2), 173-185.Gaba, D.M. (2000). Structural and organizational issues in patient safety: A comparison of health care to other high-hazard industries. California Management Review, 43, 83-102.Helmreich, R, and Merritt, A. (1998). Culture at work in aviation and medicine. Aldershot, UK: Ashgate Publishing, Ltd.Hofstede, G. (1994). Cultures and Organizations: Software of the mind. NY: Harper-Collins.Hofstede, G., Neuijen, B., Ohayv, D.D., Sanders, G. (1990). Measuring organizational cultures: A qualitative and quantitative study across 20 cases. Administrative Science Quarterly, 35, 286-317.


Lauber, J.K. (1993, April) A safety culture perspective. Proceedings of the Flight Safety Foundation 38th Annual Corporate Aviation Safety Seminar, pp. 11-17.Libuser, C.B. (1994). Organizational structure and risk mitigation (Ph.D. Dissertation). Los Angeles, CA: University of California at Los Angeles (UCLA).Mearns, J., and Flin, R. (1999). Assessing the state of organizational safety -- Culture or climate? Current Psychology: Research Reviews, 18, (1), 5-17.Merritt, A. & Helmreich, R.L. (1996). Human factors on the flight deck. The influences of national culture. Journal of Cross-Cultural Psychology, 27, (1), 5-24.Perrow, C. (1984). Normal accidents: Living with high-risk technologies. NY: Basic Books.Reason, J. (1998). Achieving a safe culture: Theory and practice. Work and Stress, 12, (3), 293-306.Reason, J. (1997). Managing the risks of organizational accidents. Brookfield: AshgateRoberts, K. H. (1993). Culture characteristics of reliability enhancing organizations. Journal of Managerial Issues, 5, 165-181.Roberts, K.H., Rousseau, D, La Porte, T., (1994). The culture of high reliability: Quantitative and qualitative assessment aboard nuclear powered aircraft carriers. Journal of High Technology Management Research, 5, 141-161.


REFERENCES AND SOURCE DOCUMENTS

Sagan, S. (1993). The Limits of Safety. Princeton: Princeton University Press.Schein, E.H. (1999). The corporate culture survival guide. San Francisco: Jossey-BassSengupta, (2000, March). Factor Analysis of US Navy Command Safety Assessment Survey. Monterey, CA. Naval Postgraduate School.Turner, B.A., (1991). The development of a safety culture. Chemistry and Industry. pp. 241-243.Turner, B.A. (1978). Man-made disasters. NY: Crane-Russak & Co.Vaughn, D. (1996). The Challenger Launch Decision: Risky Technology, Culture and Deviance at NASA. Chicago: University of Chicago Press.Weick, K.E. (1987). Organizational culture as a source of reliability.. California Management Review, 19, 112-127Weick, K.E. (1993). The Collapse of sense making in organizations: The Mann Gulch disaster. Administrative Science Quarterly, 38, 628-652.Weick, K.E. and Sutcliffe, K.M.(2007). Managing the unexpected. John Wiley & Sons.


Westrum R. and Adamski, A.J., (1999). Organizational factors associated with safety and mission success in aviation environments. In, D.J. Garland, J.A. Wise, and V. David. Hopkin (eds.), Handbook of Human Factors. NJ: Lawrence Erlbaum.Weigmann, D.A., Zhang.H, von Thaden, T., Sharma,G. and Mitchell.A. ((2002,June). A synthesis of safety culture and safety climate research (Technical Report: ARL-02-3/FAA-02-2). Federal Aviation Administration, Atlantic City.Zohar, D. (1980). Safety climate in industrial organizations: Theoretical and applied implications. Journal of Applied Psychology, 65, (1), 96-102.Zohar, D. (2000). A group-level model of safety climate: Testing the effect of group climate on micro-accidents in manufacturing jobs. Journal of Applied Psychology, 85, 587-596.Zohar, D. (2001, July). Safety climate: Conceptual and measurement issues. In, Quick, J.C. & Tetrick, L.E. (Eds.) Handbook of occupational health psychology. Washington, DC: American Psychological Association .