Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
IAEA
Why surveys?
• To capture attitudes and impressions of a large
population
• To make sure that everyone in a organization
has had an opportunity to make his/her voice
heard
• To establish a baseline and be able to track
changes over time
IAEA
Advantages of surveys
• Everyone has an opportunity to provide their view
• The survey itself is a message – ’we care about your view’!
• Data can be processed statistically to identify differences
between groups (e.g. functional groups or hierarchical
levels)
• Areas for further investigation, with other methods, can be
identified
• Survey data gives a
’baseline’ for comparisons
over time
IAEA
Disadvantages of surveys
• It is resource-intensive to plan, distribute, analyse and
communicate survey results
• Surveys identify symptoms rather than causes
• The information collected is about what employees think
they think – this is not the same as how they really act!
• Surveys are subject to response bias, e.g. respondents
may feel that they should respond in certain ways
• Questionnaires say more about what the person asking
questions thinks is important than what the respondent
feels is important!
IAEA
Integrity is important
Integrity must be assured
from two perspectives:
Integrity in
collecting and
handling data
Integrity in analysis
and presentation of
data
IAEA
Distribution strategy
Response rate is very important for statistical validity
• Communication strategy, how results will be taken care of =
increased motivation to take part
• Management championing the survey
• Allocate work time for filling it out
• Rewards for filling in the questionnaire on time
• Reminders
• Response rate can be increased by monitoring the progress within
different departments (’competition’)
IAEA
Distribution strategy (cont’d)
• Group sessions to fill out questionnaire can be arranged
• Ensure that the integrity of the situation for filling out the
questionnaire is maintained for all respondents and for the
entire response period
• E.g. paper and electronic versions available and accessible;
arrangements for shift workers
IAEA
Measurement levels (1)
1. Nominal level• Equivalence ‘=’ or ‘≠’
• Many demographics, e.g. sex, religion, state/ province, etc.
2. Ordinal level• Rank order ‘≤’ or ‘≥’
• Preference, agreement, etc.
3. Interval level (scale has no fixed zero point)• Ratios between differences have meaning
• Transformations permitted: y=a(x) + b (b≠0)
4. Ratio level (scale has fixed zero point)• Ratios (absolute) have meaning
• Transformations permitted: y=a(x)
IAEA
Measurement levels (2)
Limited or unlimited amount of scale values:
discrete or continuous
• Nominal level: discrete
• Ordinal level: discrete or continuous
• Interval- and ratio level: continuous
IAEA
Analysing survey data
• It is easy to treat numbers as absolutes – but we should
only see them as relative!
• Hence, comparisons over time are the most interesting ones.
• Do not settle for simple means and standard deviations.
Use regression analysis, cluster analysis, factor analysis,
etc. to understand the data
• This is a skill and requires interpretation and expertise!
• Ensure that no analysis on too small groups or other losses
of anonymity can happen
• E.g. a department with only one manager makes him/her easy to
identify
IAEA
Pitfalls
• Being deceived by precision
• Pre-determined levels of what is acceptable or not; red-
yellow-green indicators
• Failing to understand that this is a question of interpretation
just as much as qualitative approaches are
IAEA
Pitfalls
4. I think the communication of safety culture
values is…
• We know nothing about
response rate. Is this 27
out of 3000?
• We know nothing about
standard deviation. Are
the differences
statistically significant?
• What is a ’good’ level
here?
IAEA
Handling the results
• Communicate results by various media, e.g. intranet and
face-to-face meetings – highlights relevance to groups
• Be careful about how statistical results are communicated
in order to prevent over-confidence
• If we consider all items with a response of 70 % or higher as
strengths…
.... what to make of the fact that 70 % of the employees find it
important to follow rules and procedures?
• It also means that 30 % do not!
....which may or may not be a problem depending on the
circumstances.
IAEA
Handling the results (cont’d)
• ‘Waking the sleeping giant’ – using open fields where
suggestions will be made. How to handle these?
• A low response rate can indicate a safety culture issue
• Survey results should not be communicated
separately from the rest of the self-assessment
findings
• Remember, the organization’s responses to the
communication of results is another good input for the
safety culture self assessment!
IAEA
Summary
• Valid surveys are difficult to develop
• Surveys are a good way to gather perceptions
from large populations
• Surveys cannot be used alone because they say
as much about what the surveyor thinks as what
the respondents think
• Surveys are most useful for watching trends in
responses over time
• Surveys must be analyzed by people with
expertise in statistics and social sciences