Upload
patience-carroll
View
215
Download
0
Tags:
Embed Size (px)
Citation preview
DEPARTMENT OF HEALTHERIC REVIEW
OCTOBER 2014
2
Document structure
• Background• Profile• Overall Perceptions of the ERIC Dataset• Providers of Information• Users of Information• Changes and Improvements to the ERIC Dataset• Reports and Analysis Tools• The Way Forward
3
BACKGROUNDDH – ERIC Review – October 2014
4
Introduction
• The Estate & Facilities is critical to the delivery of high quality and safe clinical services by the NHS• It covers areas such as cleaning, food, environment quality, energy and utility usage• Annually £7.2 billion of revenue is spent on its estates & facilities by the NHS• Circa. £5 billion of capital is spent to maintain and improve it
• The Estates Related Information Collection (ERIC) is the sole source of data on the NHS Estates & Facilities asset base and patient and business critical services in England and is critical to many stakeholders supporting many functions including:• The NHS constitution pledge ‘The NHS commits to ensure that services are provided in a
clean and safe environment that is fit for purpose, based on national best practice’ through providing key data on the healthcare environments
• Informing Estates and Facilities policy development within the Department of Health• Contributing to the Governments ‘Efficiency, Reform and Growth’ Agenda by driving up
estates and facilities efficiency through the use of key performance indicators and peer benchmarking
5
Overview of Project Purpose
• The data collection covers 18 topic areas and comprises approximately 100 data fields and questions
• Data is collected at both organisation and individual site level where the site meet a given criteria
• Individual organisations can provide information for a number of different sites under their control
• The most recent data collection provided over 140,000 individual data items
• In addition to a light touch annual review a more fundamental review of ERIC is undertaken periodically to ensure data collection remains fit for purpose and meets the needs of the Department of Health, NHS bodies and other stakeholders
• The fundamental review has two key research elements that will define any changes to the ERIC data collection:
• Market research to collect and analyse the views of NHS users and other stakeholders
• Research undertaken by the Department to define their own internal requirements and those from other government departments
6
• Existing Dataset• What organisations use ERIC for and any benefits it brings to the running of the
organisation• The extent to which users are aware of the benefits of ERIC• The extent to which users are supportive of ERIC in supporting public accountability and
transparency• The extent to which data requested is readily and easily available• The level of confidence in data accuracy against topic areas• Are users aware of the analysis produced by DH and if so, do they find the analysis useful• The extent to which organisations use ERIC to benchmark and improve estates and facilities
performance• ERIC IT System
• Data entry and user friendly – Could the way data is entered be improved?• Reporting – Which reports are used most and why? Can their format be improved?
• Improvements• Any questions/topic areas users wish to be removed and why?• Any questions/topic areas users wish to be added and why?• Any existing questions that would benefit from being broken down into a more detailed
level of granularity and why?
Objectives
7
Methodology Summary
• Stakeholders from NHS bodies were invited to take part in an online survey• Stakeholders that completed the survey were given the option to take part in a follow up
telephone depth interview• A selection of those willing to be recontacted were called to discuss their views on ERIC in
more detail• Other stakeholders (e.g. Regulators, arm’s length bodies, professional bodies, etc) were also
invited to take part in a telephone depth interview
493 NHS body invites
33 Qualitative Interviews
178 Survey Completions
Other Stakeholders
8
Quantitative Methodology - Sampling
• OMB were provided with 504 individual contacts covering 255 different organisations– Contacts were either a DoE Contact or an ERIC contact
• Contacts were sent a pre contact email from the DoH project manager in order to introduce the research as well as giving them an opportunity to opt-out, update the survey contact to someone more appropriate, etc.
• Following the pre contact email exercise there were 493 contacts remaining to be sent the survey link
Total
Total contacts provided 504Unusable – pre contact email opt-outs 11Total contacts invited 493Unusable – incorrect email 9Total useable sample 484Interviews achieved 178Response rate 37%% Organisations covered 63%
9
Quantitative Methodology – Survey Progress
15-Sep
16-Sep
17-Sep
18-Sep
19-Sep
20-Sep
21-Sep
22-Sep
23-Sep
24-Sep
25-Sep
26-Sep
27-Sep
28-Sep
29-Sep0
5
10
15
20
25
30
35
40
21
35
84 3
36
18
11
3 4
35
Number of Interviews
• The online survey was open for two weeks from 15th September to 29th September• Responses peaked in the first 24 hours (15th/16th) and again following the two survey
reminder emails (22nd and 29th)
10
Qualitative Methodology Details
• Following completion of the online quantitative survey, 33 telephone depth interviews were conducted with stakeholders
• The sample was made up of• NHS Bodies that had taken part in the online
quantitative survey• Contacts were selected based on their
response to the online survey to achieve a balance of both providers/users and positive/negative views
• Other Stakeholders including regulators, arm’s length bodies, professional bodies, other governmental departments and commercial organisations
• The discussion guide was drafted alongside the DH project manager and aimed to explore stakeholder’s views on ERIC in greater depth in order to enable a more detailed understanding of the overall of experience of providers and users
11
PROFILEDH – ERIC Review – October 2014
12
Respondent Profile - Online Survey
The online survey covered a wide range of organisations, with the majority of respondents employed by Acute Foundation Trusts and Acute NHS Trusts
Base: All Respondents, Other (178, 2%)
Organisation Type
Acute Foundation Trust
Acute NHS Trust
Mental Health Foundation
Mental Health Trust
Ambulance Foundation Trust
Ambulance NHS Trust
Care Foundation Trust
Care Trust
Community Trust
Social Enterprise
42%
26%
12%
5%
1%
2%
1%
1%
7%
2%
13
None (0%); 56%
Some (1%-50%), 37%
Most (51%-99%); 4% All (100%); 1%
Sites under a PFI contract?
Base: All Respondents (178)
Respondent Profile - Online Survey
14
Respondents to the online survey were typically in an ‘Estates and/or Facilities’ role
Base: All Respondents (178)
Job Type
Manager of Estates/Facilities
Estates/Facilities Officer
Director of Estates and/or Facilities
Assistant Director of Estates and/or Facilities
Information Officer
Director of Finance or similar
Other
37%
20%
14%
12%
11%
2%
4%
Respondent Profile - Online Survey
15
0-2 years; 18%
3-4 years; 15%
5 or more years; 67%
Length of Relationship
A large proportion of the sample have been a provider/user of ERIC for 5 or more years.The majority of stakeholders surveyed classified themselves as both a provider and a user.
Base: All Respondents (178)
Provider; 22%
User; 2%
Both; 75%
Provider or User?
Respondent Profile - Online Survey
16
Respondent Profile - Qualitative Interviews
Organisation type No. of interviews
NHS Trusts
Acute NHS Trusts 7
Acute Foundation Trusts 4
Mental Health Foundation Trusts 7
Mental Health Trusts 1
Ambulance Foundation Trusts 1
Ambulance NHS Trusts 1
Care Foundation Trusts -
Care Trusts -
Community Trusts 2
Social Enterprise 1
Other stakeholders
Arm’s Length NHS Bodies 4
Professional Bodies 1
Commercial 4
Total 33
Among NHS Trusts, mix of:• Users and providers of ERIC
data (most were both)• Broadly supportive vs. less
supportive of ERIC (based on survey responses)
• Trust with and without PFI contracts
17
OVERALL PERCEPTIONS OF THE ERIC DATASET
DH – ERIC Review – October 2014
18Perceptions of the ERIC dataset depend on a range of inter-related factors
Perceptions of the ERIC dataset
• These factors all contribute to the trade-off between burden of collection and usefulness of data
• They also impact one another:• E.g. lack of awareness of potential
use/value can drive disengagement and less reliable data collection
• Increasing support for ERIC and improving overall impressions needs to be holistic• Dealing with both practical issues with
the collection process and with awareness/understanding of the data’s potential
19The Big Picture: Support for ERIC is strong, but it is not considered perfect
• General support for the idea of ERIC– Delivers accountability– Useful for policy makers and for
Trusts– It is the best we have and has
the potential to be very useful• Usability of the system and burden
of providing data has improved in recent years
BUT
• Problems with the realities of delivery degrade trust in the data and likelihood to use
I am fully supportive of ERIC
I am somewhat supportive of ERIC
I am somewhat unsupportive of ERIC
I am very unsupportive of ERIC
43%
41%
9%
2%
Thinking about your opinion of ERIC overall, which of the following statements best applies?
Base, Don’t know: All Providers/Users (176, 5%)
To drive greater support, usage and impact, improvements are required to ‘tighten up’ the quality and consistency of data and to increase depth of understanding of how it is used and why it is important.
Rationalisation of the dataset would help, but only limited agreement on areas to cut back.
20
Overall perceptions of ERIC
“It provides information that is key to decision making both centrally and
locally. It’s an important starting point to gain insight into our own and
other organisations.”
“We have insufficient confidence that the data is accurate, making
benchmarking and comparisons of doubtful value”
“It can be like a big black cloud hanging over you from April to June
where you know you have to get things done, but if you plan you can
take the cloud away.”
“We certainly need to keep it. It is the best we have got and has got a lot
better. We need to work on improving it even more.”
21
It allows for better benchmarking
Provides us with organisation data/behaviour
Do not trust data interpretation/data quality
Useful - general
Strategic planning
Useful for FOI requests
Helps inform government policy
Other
46%
37%
20%
13%
7%
3%
3%
9%
Highlights the importance of data quality and the fact that support for ERIC is somewhat qualified
Base: All supportive of ERIC (147)
Thinking about why you are supportive of ERIC, please could you provide us with more details?
(Top mentions - 3%+)
Reasons for support of ERIC highlight some key strengths but also reveal concerns about data quality
Spontaneous mention of poor data quality even when asked for reasons for support
22
Do not trust data interpretation/data quality
Outdated/needs updating
Time consuming/onerous
Isn't relevant to our organisation
Other
40%
20%
20%
10%
15%
Further evidence that data quality is a key concern
Base: All not supportive of ERIC (20)
Thinking about why you are not supportive of ERIC, please could you provide us with more details?
Concerns regarding data quality are the primary reason for lack of support
23
Perceptions vary by organisation type
Based on the qualitative interviews, certain types of organisation within the NHS benefit less from ERIC and are more aware of problems and difficulties completing the return…
Community Trusts
Ambulance Trusts
Social Enterprises
• ERIC not set up to record data in a way which suits their Estates– Multiple smaller building– Varied and dynamic usage within sites
• The burden of data collection/collation is greater
• The reliability and ‘meaning’ of the data is limited
• Benchmarking is not useful, as not comparing like for like (vs. other types of Trust) and the pool of similar organisations in so small
24
PROVIDERS OF INFORMATIONDH – ERIC Review – October 2014
25
How Trusts manage the process
Receiving and checking new
data definitions
Collecting and collating data
Checking and chasing data
Inputting and submitting the
return
• Single Estates (usually) professional responsible
• Most share new definitions with contributors
• But draft definitions can confuse, so may wait for final version
• Some attending workshops (but not all)
• Emailing contributors (other departments and external providers) in advance and then sharing definitions
• Some re-work spreadsheet to increase clarity for some staff
• Mining and re-working own databases (some collecting data on a monthly basis)
• Responsible individual required to chase (usually by email) some departments or external organisations
• Data checked before inputting against own internal systems and in some cases historic data
• Inputting usually the sole responsibility of one individual
• Undertaken sporadically over a few weeks
• Sign off comes from Estates or Finance Director
26Overview of perceptions of the process are mixed, but majority are satisfied
Things have improved, but concerns remain and Trusts generally agree that some improvements should be made
x Potential to misinterpret remains a core problem
x Time taken to receive final definitions delays process
x Data not always reliable/trustedx Takes time to receive all data (esp. from
3rd parties)x Staff providing data not always aware of
the real purpose/importance of ERIC
x Validation could be smoother and quicker
x Data entry still quite long-winded (additional fixes could smooth the process)
Many have been tightened up in recent years
Receiving draft definitions in advance useful to some
Number of fields reduced, so easier and quicker than it used to be
Staff become used to what is required and some Trusts have processes in place for streamlining collection
New interface an improvement (faster scrolling, fewer pop-ups)
27
Data collection satisfaction
Please could you rate your satisfaction with the following aspects of the data collection process...
Ease of use of the ERIC system
Managing the process of the submission
Overall Experience
Data collection at Site level
Data entry, validation and submission
Data collection for aggregate sites
Interpretation of the data definitions
13%
9%
9%
6%
9%
5%
4%
61%
63%
62%
56%
50%
42%
37%
17%
20%
22%
25%
24%
20%
35%
6%
8%
6%
10%
15%
14%
21%
2%
1%
1%
2%
2%
3%
3%
0.006
0.01
0.146
Very satisfied Satisfied Neither Dissatisfied Very dissatisfied Not relevant
Base: All Providers (172)
% Satisfied
74%
72%
71%
62%
59%
47%
41%
A high proportion are satisfied with the ease of use, submission process and overall experience of the data collection process. However, only two fifths are satisfied with the interpretation of data definitions.
28Moving forward: Summary of improvements and changes to the submission process overall
• Tighten data definitions• Speed up delivery of final definitions• Make definitions spreadsheet easily
adaptable (sharing with others)• More integrated calculations• Ability to deactivate irrelevant fields• Ability to annotate return with caveats
and issues• Enable validation ‘as you go’• Carry data forward/view last year’s values• Import from Excel
• Reinstate forums• Reinstate/refresh/publicise roadshows
and seminars• Produce introductory literature/manual• Raise awareness of ERIC’s importance
and potential benefit within Trusts– Encourage adoption of business
processes which are integrated/aligned with ERIC data requirements
…relating to dataset and process…
…relating to provision of support and information…
29
Difficulties in completing the return
Have you had any issues with the following when completing your ERIC return...?
Obtaining information from within your organisation
Data not available in time for submission
Time required to manipulate data to align with ERIC
Time taken to validate data
Time required to interpret data definitions
Obtaining information from Private Finance partner/outside organisations
Obtaining information from NHS Property Services
Anything else?
70%
51%
49%
45%
41%
30%
27%
8%
7 in 10 stakeholders have had issues in obtaining information from within their organisation.5 in 10 have struggled to get data in time for submission.
Base: All Providers (172)
30
Improvements to submission process
Would you like to see any improvements made to the overall submission process?Would like to see improvements to sub-
mission process?44%
34%
22%
Yes No Don't know
Of those giving a ‘valid’ response, the majority would like to see improvements made to the submission process.A large proportion (22%) of ‘don’t know’ responses suggests a lack of clear understanding of what needs to change but an
underlying sense that the system isn’t perfect.Base: All Providers (172)
“The data definitions are quite complex and could
do with being a little simpler in explanation”
“The input system could be easier, there’s a lot of stepping through screens
and waiting for validation”
“There remains too much guesswork – there is a lack
of standardisation in interpreting the data definitions between
organisations”
31
Make definitions clearer/more detailed
Improve software/navigation/make more user friendly
Add tools/resources for collecting data e.g. Notes, a manual, a forum
Smarter validation/Improve validation
Ability to blank/deactivate fields which aren't relevant
Ensure consistency/accuracy of data input
Ability to report on individual sites
Reduce required information required for submission
Carry data forward where it is unlikely to change
Break down sections to make collection simpler
Provider training in how to complete ERIC
Other
29%
20%
11%
9%
7%
7%
4%
4%
4%
4%
4%
20%
The most commonly suggested improvement to the submission process is improving the clarity/detail of data definitions
Base: All who would like to see improvement (75)
Improvements to submission process
Would you like to see any improvements made to the overall submission process?(Top mentions - 4%+)
32The Challenges of the Process: Trusts report a number of challenges associated with ‘getting the ERIC return in’
Concerns• New staff= inconsistency in data and in
slowing down of the process• Junior staff not engaged in ERIC and
do not see it a priority• Dealing with multiple sites and landlords is
very time-consuming (esp. Community Trusts and Ambulance Trusts)• Dealing with NHS Property Services
considered particularly difficult this year
• Final definitions arrive late, delaying start of process
Potential solutions/improvements• Address issues with ambiguity around
NHS Property Services’ role and remit
• Provision of ERIC introduction literature or training (bring back or better promote roadshows and seminars) to drive understanding and engagement to staff in other departments
• Speed up delivery of final definitions
“We get some draft definitions early on, but it seems to take a long
time to authorise them for use.”
33Data Definitions: The most commonly mentioned problem area
Concerns• Often thought to be the root of
inconsistencies and inaccuracies in the data as a whole
• Often definitions leave too much room for interpretation (e.g. large cost measures not stipulating what is/is not included)
• Trusts choose what suits their existing systems and approaches, and what suits their strategic aims
• Definitions can be long and complicated• Unclear which external providers/internal
department’s data to include• Complex, key metrics are tricky to calculate
Potential solutions/improvements• Greater clarity in some key definitions
• Provide ‘include…; don’t include…’ examples
• Take care to maintain brevity• Provide integrated systems to
calculate complex metrics• Reinstate online forums / FAQs section
to help with interpretation issues• Make definitions spreadsheet easier to
adapt for end users
“There used to be online forums to help with the definitions, but they stopped them. They were helpful, and we need
them back.”
34
Data definitions
“Take transport miles. Which miles belong to whom? A&E blue light are
ours, but patient transport services…are they ours or the commissioning body’s
Are these things getting double counted?”
“As the years have gone by it has got easier.”
“There is just too much ambiguity. They need to spell out what should and what should not be included.”
35Data collection: General concerns are raised about collecting data
General Concerns• Takes time to receive all data back• Collecting from 3rd parties can be difficult
• E.g. PFI partners, contractors may not collect all data in format required
• Finance data not always collected at a site level or in required format
• Providing aggregated data is counter-intuitive for sites with multiple activities in different buildings
• Trust’s internal organisation of sites not matching ERICs (e.g. number of sites)
• In shared buildings data may be duplicated or missed
• System does not reflect building/site usage
Potential solutions/improvements• Some types of organisation should be
excluded from providing certain types of data (needs to be clearly set out in definitions and in the online system)
• Ability to annotate return and include caveats about data which may not be complete (e.g. when provided by external contractors etc.)
• Training and instruction to encourage Trusts to build requirements for particular data into contracts with suppliers
“ERIC is useless for us in a Community setting as it does not
take into account usage.”
36Data collection: Specific parts of the dataset are cause for concern
Quality of Buildings/Backlog Maintenance• Complex calculations required• Up-to-date surveys not always
completed for all buildings
Transport• Business miles not recorded
consistently• What/who to include not always clear
Waste• Contractors not collecting data on
individual sites/buildings
Finance• Ambiguity about what to include (e.g.
capital costs)• Ambiguity in terms of which staff to
include in hard/soft FM costs
Food Services• Staff vs. visitor catering costs hard to
define in canteens• Are sub-contracted staff canteens an
income or cost?
Mor
e co
mm
only
men
tione
d
Contracted Out Services• Ambiguity in terms of which staff to
include in hard/soft FM costs
37Limited agreement regarding the most problematic data areas
Greatest confidence in reliability is placed against Fire Safety, Trust Profile and Age Profile.There is low confidence in the reliability of data for Transport Services and Contracted Out Services.
Trust Profile
Contracted Out Services
Finance
Staff
Transport Services
Fire Safety
Cleanliness
Laundry and Linen
Areas
Function and Space
Age Profile
Quality of Buildings
CHP EnergyWater Services
Waste
Car Parking
Food Services
0%
10%
20%
30%
40%
50%
60%
70%
0% 10% 20% 30% 40% 50% 60% 70%
% V
ery
Confi
dent
inA
ccur
acy
% Very Easy to Provide
Ease of collection vs. Perceived Reliability (% Very Easy/Reliable)
• Few areas are considered very easy to collect
• Trusts are more confident about the accuracy of their data, but often concerned about data which they rely on third parties to deliver (esp. NHS PS and some PFI partners)
• Other broad areas of consistent concern are transport services, backlog maintenance and some aspects of finance
• Some admit making guestimates for some of this data in the past (and to an extent now too)
38
Ease of Data Collection
How easy is it to collate the data for each section of the ERIC return?Fire Safety
Age Profile
Trust Profile
Areas
Car Parking
Water Services
Energy
CHP
Cleanliness
Laundry and Linen
Staff
Function and Space
Waste
Food Services
Quality of Buildings
Transport Services
Finance
Contracted Out Services
26%
23%
30%
20%
12%
14%
13%
19%
10%
9%
9%
10%
8%
8%
10%
3%
6%
6%
49%
44%
33%
34%
40%
36%
33%
26%
35%
35%
34%
27%
28%
27%
24%
24%
20%
18%
20%
26%
30%
33%
36%
33%
33%
13%
38%
34%
42%
38%
30%
35%
40%
33%
45%
41%
2%
4%
3%
11%
7%
14%
17%
3%
10%
13%
9%
19%
26%
20%
18%
26%
22%
26%
1%
2%
1%
2%
3%
1%
2%
1%
3%
3%
3%
4%
5%
3%
6%
7%
4%
4%
1%
1%
2%
1%
1%
1%
1%
36%
2%
3%
1%
1%
1%
2%
1%
5%
1%
2%
Very easy Easy Average Difficult Very Difficult Not relevantBase: All Providers (172)
Stakeholders find collating the information for Fire Safety, Age Profile and Trust Profile easiest whereas information on Transport Services, Finance and Contracted Out Services present more of a challenge.
% Very Easy/Easy75%
67%
63%
54%
52%
50%
46%
45%
45%
44%
43%
37%
35%
35%
34%
27%
25%
24%
39
Accuracy of data collection
How confident are you in the accuracy of data you provide within ERIC?
Trust Profile
Fire Safety
Age Profile
Staff
Areas
Car Parking
Finance
Water Services
Cleanliness
Energy
Laundry and Linen
Waste
Quality of Buildings
Contracted Out Services
Function and Space
Food Services
Transport Services
CHP
52%
61%
51%
42%
41%
40%
26%
40%
36%
39%
31%
27%
23%
19%
24%
24%
16%
40%
45%
34%
44%
51%
51%
50%
63%
48%
52%
48%
55%
53%
55%
59%
53%
51%
58%
22%
1%
3%
3%
4%
7%
8%
8%
8%
8%
8%
9%
16%
17%
17%
18%
16%
18%
3%
2%
1%
1%
1%
1%
1%
1%
1%
2%
1%
3%
1%
1%
3%
1%
5%
4%
34%
0.012
0.006
0.006
0.023
0.029
0.006
0.017
0.023
0.029
0.023
0.023
Very confident Quite confident Column1 Not at all confident Not relevant Base: All Providers (172)
% Very Confident/Quite Confident
97%
95%
94%
93%
91%
90%
89%
88%
88%
87%
86%
80%
78%
78%
77%
75%
73%
61%
40
Accuracy of data collection
Thinking about the areas where you are less confident in the accuracy of data provided, why is this...?
We rely on others for some of the information
Accuracy of data is hard to verify/hard to get
Information required is too detailed/hard to quantify/interpret
Data isn't collected in a format suitable for ERIC
Collecting data is resouce heavy/time consuming
Historical data is provided to us
Colleagues don't understand the importance
Age of site/buildings
Question definitions changing
Information asked for isn't relevant to us
Other
45%
29%
18%
15%
11%
8%
3%
3%
2%
2%
6%
Stakeholders are least confident in the data they provide when they rely on others for this information
Base: All not confident in accuracy for at least one area (89)
41
Accuracy of data collection
Thinking about the areas where you are less confident in the accuracy of data provided, why is this...?
“Gaining access to the information across the trust for certain
information can be difficult due to a number of reasons like staff turnover,
etc.”
“Departments outside of my control i.e. Finance, Contracting, HR, etc. do
not see the relevance of the information and the importance of its
accuracy.”
“There is difficulty in getting reliable or even any data from building
owners where we are occupiers/tenants”
“As many of these services are outsourced to third parties, we are not able to verify the quality of the
information provided by them”
42
Data entry and and validation
Concerns• Validation:
• Time consuming to correct red flags (awkward to view data on single screen and only get alert at end of process)
• Not always ‘true’ errors (esp. for small Trusts)
• Cannot see last year’s data (for sense check)• Need to key in by hand – can’t upload from
other data sets• Insufficient built-in calculations to deliver
speed and efficiency (e.g. CHP metrics)• Need to enter values into irrelevant fields
Potential solutions/improvements• Enable validation checks ‘as you go’ to
speed up the process• Variable red flag tolerance for different
type/size of Trust• Annotate fields for self or colleagues to
refer to next year• Ability to view last year’s data alongside
this year’s or pre-filled field for static data (building sizes etc.)
• Enable importing of data from Excel• More auto calculations to produce complex
metrics• Use of a database which is completed an an
ongoing basis, generating ERIC data
43
USERS OF INFORMATIONDH – ERIC Review – October 2014
44
Usage of ERIC data: Overall picture
• Usage varies across Trusts, but most see some benefit
• Depends on personal preferences and level of engagement, as well as level of suitability of data for Trust types
• ERIC data provides value and has an impact on decision-making in many cases
• But reliability of data a concern for most and the primary barrier for not using more (or not relying on ERIC)
• General awareness of the potential uses of ERIC quite high but does not always mean a deep level of understanding of capabilities
• Some scope to reduce size of dataset, but lack of agreement on specific measures to cut
Increasing confidence in ERIC and raising awareness of capabilities and usefulness through education/communication should be a priority. Changes to the dataset itself likely to be relatively moderate
45
Moderate Users
Trusts describe different levels of overall use of ERIC data
Overall, Trusts are deriving some value from ERIC, and some are finding the data very valuable. There is scope to increase this through communication around its potential uses and improvements to reliability
• Using ERIC data quite regularly, for a number of purposes• But limited awareness and understanding of the range of
potential uses• ERIC not well known throughout the organisation
Enthusiastic Users
Avoiders/non-users
• Reliant on ERIC for strategic planning and other decision-making
• New applications and uses often sought• Known throughout the organisation (requests for supporting
info from ERIC quite common)
Mor
e co
mm
on in
sam
ple
• Doubts about benefits/suitability and reliability limit use• Seen as purely a data gathering exercise for DH• Most commonly smaller Trusts and Community Trusts
46
Benchmarking/Performance Management
ERIC data is used in the following ways within Trusts (1)
• Keeping track of performance against Trust KPIs and trends • Comparing key cost measures/performance against peer group• Identifying best practice in other Trusts
Reporting
Strategies and planning
• Inputting into annual reports to the Trust Board on KPIs or specific issues (energy consumption, carbon footprint)
• Informing the development of the Estates strategy (contributing to evidence for decision-making, along with other data)
Mor
e co
mm
on in
sam
ple
Enquiries & FOI requests
• Media enquires (most recently car parking and catering)• FOI requests linked to high profile topics or often by prospective
suppliers looking for a sales angle
Benchmarking and reporting on performance are considerably more prevalent uses. Trusts often use what ERIC makes easily available to contribute to their understanding and evidence base
47
Business Cases
ERIC data is used in the following ways within Trusts (2)
Using ERIC to drive operational decision-making is less common. Internal data often considered more appropriate and reliable
• To support plans for capital investment (assessing condition of existing estate to support decisions to refurbish or replace sites/buildings)
Contracts
Operational Policies &
Procedures
• Cost monitoring to inform procurement decision making (waste, energy, catering)
• Charge setting for other users of Estate (energy costs etc.)
• Benchmarking and change over time on metrics informing and supporting decisions such as energy efficiency measures or parking policy
Less
com
mon
in s
ampl
e
48
Use of ERIC
Have you used ERIC data in your organisation for any of the following...?Benchmarking/Performance Man-
agement
Reporting
Strategies and Planning
Public/Media Enquiries or FOI Requests
Business Cases
Contracts
Operational Polices and Procedures
Anything else?
91%
87%
69%
69%
59%
28%
28%
5%
Base: All Users (137)
49
Business case appraisal
Usage by other organisations: Arm’s Length and Professional Bodies*
• Contributing to data to evaluate business cases for (mostly) capital expenditure
*Note: Categories for purposes of anonymity
Understanding carbon reduction
performance
Monitoring and reporting performance on other sustainability
metrics
• Exploring, monitoring and reporting on carbon footprint for NHS at regional and national levels
• Encouraging and helping Trusts meet their carbon reduction goals and report on their performance in annual reports
• Water use and waste (recycling)• Travel plans and adaptions
Providing Estates advice to Trusts
• Benchmarking performance on particular areas to advise on operational decisions
50
Preparing commercially
available reports
Usage by other organisations: Commercial organisations
• Analysing the raw ERIC data and using to create relevant and usable benchmarking reports
• Trend analysis• Add commentary and descriptions which help Trust
identify any problems with unreliable data etc.• Preparing general Estates report on wider NHS
Services to specific Trusts
• Brokering deals for capital investment (new buildings, change of use)
• Specific projects to help Trusts deal with risk analysis• Helping with strategic planning and cost-cutting• Imbedding in Trusts in temp roles (Estates Directors, Fire
Officers, external auditor roles)• (also helping complete the ERIC return)
51
High impact / influence
Impact and benefits of using ERIC data: Impact on decision-making and strategic planning
Demonstrates that ERIC can have some impact and has the potential to be highly valued
Mor
e co
mm
on
Contributory impact / influence
No impact or influence
• Most common• Supports decision making (e.g.
contract negotiations) and inputs into strategic planning
• Other data also used in conjunction with ERIC
• ERIC data can kick start investigations into performance and expenditure
• Relying on ERIC data for some key decisions and strategic planning (high level within Estates dpt.)
• Used as basis for strategic business planning
• Moderately common
• May access ERIC data, but not said to have had impact on decisions or strategy
• Inferior to alternative data (internally collected)
• Lack of confidence in data• Least common
Increasing impact and benefit
52Impact and benefits of using ERIC data: Other benefits
These more ‘peripheral’ benefits could be promoted more as part of a wider communications strategy
• Need to collect data and provide reports (or similar) keeps Estates teams in touch with the wider Trust teams
• Encourages understanding and cooperation– Formally via benchmarking clubs or
informally through local contacts
Promoting best practiceDriving communication with other departments
and Trusts
• Benchmarking can flag up examples of top performing Trusts, triggering exploration of practices/site visits and senior level dialogue
• Requirement to complete return can encourage timely and accurate data collection
“It is useful to know what is going on with the Trust. I like to know how we
are doing and how we compare.”
“We have developed processes and a system to collect the data on an
on-going basis, that is just the right thing to do.”
53Impacts and benefits felt by the wider NHS and other stakeholders
Overall, other stakeholders support ERIC as it is established and unique. However, they agree that the system has the potential to be of even greater benefit to themselves and Trusts
• Provides unique breadth of data• A long historic dataset enabling useful trend
analysis• Although other sources of data exist, they do not
have the same breadth/length• Covers high level metrics of relevance to policy
makers
Enables departments to meet their statutory
requirements
Enables large scale, high level analysis across the
whole NHS Estate
Speeds up processes
Helps verify business cases
• Centralised, pre-collected data removes the need for targeted investigation and research when assessing business cases/strategies
• Provides a layer of assurance which otherwise might be missing from assessments
54Barriers to (and problems encountered when) using ERIC data
Awareness of what it can do
Lack of time to investigate/use
Concerns about reliability and accuracy
Other systems in place which are more suitable
Timing of data delivery too slow
Insufficient/unsuitable data to meet needs
Driving up awareness of ERIC’s capabilities should increase overall engagement and help contribute to increasing reliability and accuracy of data (which should also be addressed directly)
Wider System-centric
Two types of barrier
work together
and influence
one another
55Accuracy, reliability and completeness of data remains a challenge
Increasing reliability and accuracy is the key priority for the majority of stakeholders. There is acknowledgement that this is not a straightforward task
• While Trusts are fairly confident in the accuracy of their own returns (with some important exceptions), they often doubt the reliability and accuracy of the rest of the dataset
• Other stakeholders (NHS central departments, professional bodies and commercial users) are also somewhat sceptical, feeling that inaccuracies exist
• However, participants across the sample (especially other stakeholders) pointed out that at a macro level the data was mostly acceptable in terms of accuracy– Some added that the quality of data has improved over
time
“I think there are doubts about the accuracy right
across the data.”
“It is such a big dataset, you know that for high level
measures the mean figures are accurate.”
56Lack of accuracy in the data occurs in different ways…
While some high level changes to how ERIC is planned and delivered are mentioned, the need to retain continuity is also acknowledged. Ironing out data input issues is often seen as priority no.1
Inconsistencies Inaccuracies/omissions Systemic problems
Data input issues Inherent issues
• Difficulty interpreting data definitions
• Different internal approaches to data monitoring
• Difficulty collating correct data
• Trusts manipulating data to ‘look good’ or support strategic plans
• Lack of care collating, checking and inputting
• Not using audited/verified data
• Closed dataset = inability to correct inaccuracies in past data
• A snap-shot, so not up-to-date
• Lacks granularity• Missing chunks of data (NHS
PS)• Underpinned by a lack of engagement and understanding of
ERIC by those providing data• Exacerbated by churn in Estates teams and fragmentation of
NHS structure
57Improving ERIC for those using it is primarily about improving the data itself and raising awareness of benefits and importance
Some users also suggest that Trusts should be more accountable for the accuracy and completeness of their returns
• Roadshows, local meetings, forums• Training for new staff (and non-Estates staff)• Pro-active publication of case studies showing high
performance• A re-launch of ERIC to refresh perceptions• Tap into existing networks and organisations…gain
commitment of HeFMA, NHSPS, CHP, P21 to drive message to others
• Publicise (and act on) this review
Increasing accuracy
Driving up awareness of benefits/uses
Encouraging greater
engagement
58Specific improvements to the dataset from a user perspective
• Faster delivery of data• Consider how to deliver more frequent data collection, or ideally real time data
collection• Aligning more closely with…
– Other data collection systems and procedures– Govt/legal requirements around reporting on issues such as sustainable
development• Capture change in activity and usage data• Update definitions around renewables in line with latest recommendations from DECC
and Environment Agency• Users (those not involved in preparing return) would ideally like more data if possible
– Not always able to pinpoint what this is in interviews, but would welcome more granularity
59
CHANGES AND IMPROVEMENTS TO THE DATASET
DH – ERIC Review – October 2014
60Which parts of the ERIC dataset are most/least useful?
Suggest consideration given to cutting back some specific questions in conjunction with other internal evidence
• There is agreement from within all sample groups that ERIC could benefit from further ‘pruning’ to make it less complex and reduce the burden on Trusts
• But, there is no clear agreement on areas of data to remove on a large scale
• Trusts and other stakeholders are using a wide range of data for different purposes
• Removing some of the less useful and less reliable specific questions across the dataset is the recommended approach
“Those question on mileage are just not accurate and
not useful to anyone.”
• Transport the most consistently mentioned area suitable for removal of questions
• Contracted Out Services also more likely to be mentioned as unnecessary
61Suggested changes to data by section – Trust level data
Trust Profile• No consistent suggestions recorded, but
overall usefulness to Estates often questioned
Contracted Out Services• Increase granularity (break-out by
contractor type)• Or remove altogetherFinance• No consistent suggestions recorded
Staff• Break down staff costs further within both
soft and hard FM costs
Transport Services• Remove business mileage (T06 03) and
patient mileage (T06 04)• Remove number of claims through HTCS
(T06 02)Fire Safety• Section considered a duplication of other
returns…is this needed?
Cleanliness• Split out pay/non-pay costs (T0801)• Greater links to clinical issues such as
infection rate/risks
Laundry and linen• Low area of interest in non-acute orgs
Other suggestions:• Does the Trust have a sustainable development action plan/how developed is it?• Capture incidence of overheating in wards• Capture use of nitrous oxide used (greenhouse gas)
62Suggested changes to data by section – Site level data
Area• Break down occupancy costs• Site heated volume (S0104) very
inaccurate, could be removed
Function and space• S0201/2 deliver unreliable
result…clarify/simplify or remove
Age Profile• No consistent suggestions recorded
Quality of Buildings• Risk backlog difficult to calculate, but
potentially useful data. Clearer definitions a priority
CHP• Should be filtered and only asked to
relevant Trusts
Water Services• Separate surface water and highway drainage
Waste• Reduce options for different types (S082-5)
Car Parking• Include more granularity on parking charging
structures• Collapse staff and patient parking as often not
separatedFood Services• Include relevant PLACE scores
Energy• Record who energy provided by/how rates
negotiated• Ability to deactivate irrelevant energy
consumed fields• Harmonise with CRC requirements• Include gas consumed locally
63
Usefulness of ERIC
How do you rate the usefulness of the information provided within each section of ERIC?Energy
Water Services
Waste
Areas
Quality of Buildings
Cleanliness
Function and Space
Food Services
Finance
Car Parking
Laundry and Linen
Age Profile
Fire Safety
Staff
Trust Profile
Contracted Out Services
Transport Services
CHP
50%
42%
39%
43%
42%
39%
39%
40%
39%
33%
30%
26%
26%
24%
18%
20%
14%
14%
42%
47%
48%
45%
44%
46%
46%
43%
43%
44%
43%
47%
46%
46%
52%
47%
39%
31%
4%
7%
9%
8%
10%
12%
12%
11%
14%
17%
20%
21%
21%
23%
20%
23%
36%
16%
2%
3%
2%
3%
3%
1%
2%
1%
2%
4%
4%
4%
4%
4%
7%
5%
8%
7%
1%
1%
1%
1%
1%
1%
1%
4%
2%
1%
2%
2%
2%
2%
2%
4%
2%
32%
Very useful Fairly useful Not particularly useful Not at all useful Not relevantBase: All Users (137)
64
Use of ERIC
Which section of ERIC do you use for each of the areas you listed? (Top 5 Sections)
Strategies and Planning
Base 75
Quality of Buildings 57%Function and Space 56%Areas 55%Age Profile 51%Trust Profile 41%
Benchm’ing/Perf. Mgmt
Base 109
Cleanliness 57%Waste 55%Laundry and Linen 54%Finance 50%Food Services 50%
Reporting
Base 96
Fire Safety 44%Energy 36%Water Services 34%Areas 32%Function and Space 31%
Business Cases
Base 43
Quality of Buildings 35%Finance 30%Age Profile 30%CHP 19%Function and Space 16%
Public Enq’s/FOI Requests
Base 42
Car Parking 40%Food Services 36%Staff 24%Energy 19%Trust Profile 17%
Contracts
Base 18
Contr’d Out Serv’ 39%Laundry and Linen 22%Finance 17%Cleanliness 17%Energy 17%
Operational Pol’s/Proc’s
Base 75
Water Services 38%Fire Safety 25%Waste 25%Car Parking 25%Staff 13%
65
REPORTS AND ANALYSIS TOOLSDH – ERIC Review – October 2014
66
Awareness of reporting and analysis tools
Trusts would benefit from further information and guidance regarding the potential uses and differences between different types of reports and analysis on offer
• Trusts are generally aware of most benchmarking reports (and the majority use most or all)
• However, Trusts are not always fully aware of what the reports offer, or their potential benefits
• Awareness of KPI analysis is slightly lower, and lack of understanding/ confusion with benchmark reports is evidentAw
aren
ess
Understanding
Benchmarking Reports
KPI Analysis
“I know I work with ERIC all the time, but I’m not really sure what the differences are.”
67
Awareness of reports and analysis tools
Which of the following benchmarking reports are you aware of...?Site Comparison
Trust Key Performance Indicators
Organisation Wide
Region Trusts
Region Wide
Organisation Trend
93%
92%
87%
84%
82%
81%
Base: All Users (137)
Estates & Facilties KPI's 66%
Are you aware of the ‘Estates and Facilities KPI’ analysis produced by DH?
68
Usage of Benchmarking Reports and KPI analysis
• Vast majority of Trusts use at least one benchmarking report annually
• Most run the reports once per year, soon after the publication of the data (often included in annual reports)
• Some also run reports more frequently to meet specific operational requests within the Estates or Finance departments
• KPI analysis less frequently described, and sometimes the difference between benchmarking reports and KPI analysis not understood
“We always use ERIC KPI’s if challenged with anything to
prove our efficiency. If it does not answer my needs I look
elsewhere.”
Smaller Trusts and Community trusts describe
less interest in using reports
69
Usage of Benchmarking Reports
How frequently do you use these reports...?
The majority of stakeholders use the benchmarking reports no more than 2-3 times a year
Site Comparison
Trust Key Performance Indicators
Organisation Wide
Region Trusts
Region Wide
Organisation Trend
0.01
0.01
0.01
3%
6%
4%
2%
1%
1%
14%
17%
14%
13%
12%
13%
36%
37%
33%
36%
34%
34%
35%
36%
36%
34%
36%
37%
10%
3%
10%
12%
13%
14%Weekly Monthly 4-6 times a year 2-3 times a yearOnce a year Never
Base: All Aware
70Perceptions of using ERIC data to benchmark are generally positive but with some variations
• Useful means of making high level comparisons
• Degree of flexibility in what to benchmark against (but can result in comparing apples and pears)
• Ability to export data in spreadsheet format delivers flexibility (for those able to manipulate it)
What is working? What is not working?
• Insufficient detail/flexibility:– Unable to compare just specific metrics – Unable to select specific Trusts to compare
against– Unable to generate bespoke reports
• Radar charts unintuitive/difficult to interpret• Benchmarking only a snapshot in time• Content and purpose of reports not always
obvious until you have read the contents
“We want to be able to select specific Trusts, not just ones that the system suggests, and not just
locally.”
“It is particularly useful as it shows how we perform
nationally but also specifically within the local area.”
71
Benchmarking between organisations
How useful do you think the current ERIC system is for benchmarking performance against comparable organisations for each of the topic areas ERIC covers?
Energy
Water Services
Cleanliness
Waste
Food Services
Laundry and Linen
Quality of Buildings
Areas
Finance
Fire Safety
Function and Space
Car Parking
Staff
Trust Profile
Age Profile
Contracted Out Services
Transport Services
CHP
36%
31%
31%
30%
30%
26%
23%
29%
31%
20%
22%
24%
18%
19%
14%
20%
13%
9%
50%
53%
53%
53%
53%
54%
55%
47%
45%
54%
47%
44%
50%
47%
50%
42%
47%
37%
11%
12%
13%
13%
12%
15%
16%
18%
20%
20%
24%
26%
27%
23%
26%
31%
28%
22%
3%
2%
2%
3%
3%
4%
5%
4%
3%
5%
5%
5%
3%
9%
9%
5%
10%
17%
Very useful Fairly useful Not particularly useful Not at all useful Base: All Users (137)
The Energy, Water Services, Cleanliness, Waste and Food Services sections are most useful for benchmarking. Contracted Out Services and Transport are less likely to be seen as useful for benchmarking, presumably linked to their perceived
reliability
72Estates and Facilities KPI Analysis is seen as moving in the right direction
Links to clinical data such as infection rates would make KPI analysis more useful
“It is going the right way, but I think that ERIC could look to what is going on in the commercial world and
learn something.”
• An interesting output• Delivery via dashboard monitoring helpful• But KPIs not linked to clinical data and not always intuitive/relevant
73A number of specific improvements to the access and delivery of ERIC data are suggested
Base: All Users (137)
• Benchmarking to include a toolkit to interrogate the data– More detailed benchmarking bandings (e.g. ability to just focus on FM costs)– Ability to choose specific Trusts to compare against
• Clearer titling and description of report contents and potential benefits• Ability to benchmark against other, non-NHS organisations • Multifunctional rather than broad brush reporting (via a tool kit for users)• Template reports which could be applied to different sections and modified• Access to the ERIC online system/functionality for wider NHS departments• Greater access to more information via guest log-in (e.g. all potential FOI requests)• More commentary and signaling of implications within benchmark reports (more of a ‘so
what’ element)• Adherence to latest open data principles
74
Access improvements
Would you like to see any improvements in how information from ERIC can be accessed?
Whilst the proportion suggesting improvements is relatively low (12%) a significantly high proportion (31%) say ‘don’t know’ suggesting a lack engagement and clear understanding of what needs to change
“Any of the information that would be available on a freedom of information request should be available via a guest
login.”
Improvements to Access of ERIC Info' Required? 12% 57% 31%
Yes No Don't know
Base: All Users (137)
“I would like to be able to print off comparisons from neighbouring
Trusts.”
“More intuitive user reports. Currently you have to open most reports before understanding what they are telling you in order to find the right one to
suit your needs”
75
Benchmarking improvements
Are there any ways in which you think the benchmarking potential of ERIC could be improved?
“I think the information within ERIC should identify where top 20%
performance is being achieved and seek a narrative/case study from the
providers to share best practice.”
“Training should be given as to exactly what is required in the data fields.
The written guidance is often confusing and difficult to understand.”
“The data granularity or the way it is presented does not lend itself to
useful benchmarking. I feel that with the right metrics, it could be slicker
and much more useful.”
“All the data collected by ERIC is very useful, but the difficulty comes to
comparing apples with apples. Too many decisions are made when
comparing apples with oranges.”
Base: All Users(137)
76
THE WAY FORWARDDH – ERIC Review – October 2014
77
The Way Forward (1)
• Support for, and satisfaction with ERIC are reasonably high• But improvements to the data definitions and collection process along with increased
engagement, understanding and awareness of the system in general/what it is capable of are required
• Increasing accuracy and consistency in the data will be a challenge and requires both improvements to the definition and support/education/training for those tasked with collecting the information
• Some further improvements to the ERIC IT systems (inputting and reporting) will be helpful, but recent improvements are already noticed
• A slimmed-down, less complex dataset is desirable in general. But, given recent cuts in the size of ERIC, identifying clear areas to make large scale cuts is challenging– Transport Services and Contracted Out Services present the most obvious opportunity
for cuts• Some additional detail to the data is also required to make ERIC more usable• At a broad level, there is agreement that ERIC could deliver greater value if it were more
clearly integrated with other data collection and reporting systems, and focused more on providing a holistic picture for Trusts (making greater linkages to clinical outcomes)
78
The Way Forward (2)
• The future in terms of coverage of Community Trusts should be considered carefully– While stakeholders recognise that a consistent, fully comprehensive dataset is desirable,
important questions are raised regarding the fit between ERIC data definitions and the way in which these Trusts operate
• It will also be important to clarify the situation regarding NHS PS going forward to ensure Trusts know what to do and that users of data can be confident that the dataset is representative
• While awareness of many aspects of ERIC (reporting options, usage by government departments) is numerically high, further work is required to increase depths of understanding and engagement with ERIC overall– This will deliver greater reliability of data, as the importance of completing the return
consistently and accurately will be better appreciated– And will drive up usage of, and advocacy for ERIC as a source of useful data
79