Methods of RegulationInquiry Seminar

“We live life forwards but understand it backwards”

Peter HomaChief executive

13th October 2011

Agenda• Biography and caveat emptor• Creating an inspectorate• Lessons from healthcare regulation• Setting, monitoring, improving and enforcing healthcare

standards• A chief executive’s perspectives: how inspectors’

findings are best implemented• Coda

Biography• 1989 - 1998: CE, Leicester Royal Infirmary• 1998 - ’99: Head, National Patients Access Team,

Department of Health• ’99 - 2003: Inaugural CE, Commission for Health

Improvement• 2003 - ’06: CE, St George’s Hospital, London• 2006 - present: CE Nottingham University Hospitals

Creating an inspectorate

• Substantial, complex task to design and build an inspectorate including method development, staff recruitment and training and excellent governance structures and process

• Develop inspectorate’s values, strategic and operational plans

• Design and embed quality assurance processes to ensure inter-rater reliability

• Establish effective relationships with internal and external stakeholders including (sometimes with MOUs) patient and public groups, NHS, DH, 10 Downing Street, Treasury, other regulators, Royal Colleges and media

Creating an inspectorate

• Government often assumes that newly created inspectorates can operate more quickly than practical

• Considerable risk during a new inspectorate’s early days due to new legal duties, staff, methods untested QA and internal and external relationships

Lessons from healthcare regulation

• Focus on patient outcome, safety and experience c.f. The “Apple Ipad approach”: design methods “outside in” not “inside out”

• Inspection methods should be subject to the same rigorous continuous improvement as the organisations that are inspected. Avoid inspectorial methodological rigidity and develop effective change procedures for inspectorate staff. Benchmark inspectorate performance against global best practice

Lessons from healthcare regulation• NHS and provider Boards are the focus of governance accountability

and responsibility. They operate in a complex context that must be understood by the inspectorate

• Inspections should form part of an overall process for improvement lead by the Board and not an isolated event

• Establish open, transparent and published inspection methods and standards that Boards can use to guide and measure their own work to improve the quality of patient care

Lessons from healthcare regulation

• Well prepared peer reviews of organisations against agreed standards by lay people, practicing clinicians and managers help to improve the quality of patient care. Such peer review can be separate or integrated to inspections. Multiple benefits include providing inspections with a deep understanding of everyday healthcare practice and this experience is taken back to peer reviewers’ own organisations

• Inspectorates should try and anticipate and avoid unintended consequences of their inspection methods/standards for example…

Lessons from healthcare regulation

• Inspectorates should develop predictive analyses to try and anticipate major organisation healthcare failure before they occur e.g. small specialist geographically isolated units (See CHI’s Lakeland NHS Trust investigation 2000)

• There should be as few inspectorates as possible and those that exist should have a duty to operate as a coherent system where the whole is greater than the sum of the parts. Given inspectorates’ legal duties this is often challenging to achieve but essential if patient care is to improve as much as possible (See Michael Power, Inspection Society)

Lessons from healthcare regulation

• Inspectorate judgement of organisations’ performance should take into account the organisational context including PCTs, CCGs, SHA

• One of the Inspectorate's main contributions is providing a Board with a thoughtful, authentic, assessment of their organisation’s comparative performance. “Holding a mirror to the organisation”

• High quality data about the quality of clinical outcome, process and patient experience comprise the oxygen for service improvement. Data quality assessment should be made at all levels (see Nottingham case study)

Data Quality Kitemark

12

Data Quality IndicatorDefinition Sufficient Insufficient

What does it mean for a Board member if this indicator is insufficient

Owner action if the indicator is insufficient

Granularity

Can the data be broken down into smaller parts? For example, can the data be shown at Trust level down to the level of ward?

Yes the data can be broken down to sub Trust level

No the data is only available at Trust level It is not possible to dril l down to

understand the cause of the issue

Create an action plan to enable the data item to be broken down sub trust parts

Completeness

Does the data demonstrate the expected number of records for that month? For example a 10% increase or decrease on last month. Are there any known reasons why the data is not complete, for example staff on leave and have not entered all data onto a system. Are all of the required fields complete on the database?

The data demonstrates no significant change month on month, or where it does there is an explanation.

The data available are significantly different to expected but no explanation can be given

True performance cannot be assured because all of the data has not been presented, or there is an unknown cause for significant variation

Carry out a root cause analysis as to why the data was not complete and ensure data is complete the following month

Validation

Prior to publication, does the data get validated in accordance with the Trust's Data Validation Policy? For example, spot checks, random sample checks. Has the data been signed off by the indicator owner outlined in the Information Frameworks? In addition, has a clinician been involved in the validation of the data (where appropriate)?

The data is validated against a secondary source in l ine with the data validation policy. The indicator owner can assure the data is a true reflection of performance.

No validation has taken place. The owner cannot assure that the data truly reflects the performance. A random sample may reveal errors.

The data has not been validated and therefore true performance cannot be assured

Follow the guidelines set out in the Data Quality Validation policy

TimelinessIs the data the most up-to-date and validated available from the system?

Yes the data is the most up to date available

Data is not available for the current month due to problems with the system or process

The data is not the most up-to-date therefore decision made at Board may be incorrect

Create an action plan to show when timely data will be available

Source

Is the source the data is obtained from fully documented? For example, do all users know how to extract the data in a consistent way, to ensure that errors in performance reporting do not occur?

All users understand how the to extract data and what data is available on the system. All have up to date training on system usage.

The data source is a poorly documented bespoke system. Inconsistent extracts are common.

If the source is poorly documented users may extract data in an inconsistent way and therefore the data presented cannot be relied on

Create an action plan to show when the data source will be fully documented and users fully trained.

Audit

Has the system or processes used to collect the data been internally or externally audited in the last 12 months?

The system and processes involved in the collection, extraction and analysis of the data have been audited to give “significant assurance.”

No formal audit has taken place in the last 12 months

The system and processes have not been audited in the last 12 months, therefore assurance cannot be given about the quality of the data

Ensure that a formal audit is carried out in the next 3 months

Assessment of Executive Director

Does the lead Director believe the data used for this indicator to be a true reflection of actual performance?

The Executive Director can give significant assurance about the quality of the data

The Executive Director cannot give assurance about the quality of the data

Assurance cannot be given to the quality of the data and therefore the performance displayed

Executive Director must investigate why the data quality is poor and feedback to CET

This page gives guidance on how to update the Data Quality Kitemark for the Integrated Board report. For all indicators please state whether they adhere to the following measures or whether the measure is not relevant to the indicator.

Data Quality Kite Mark

Setting, monitoring, improving and enforcing healthcare standards

• Standards should draw on global best practise e.g. “matching Michigan”, infection control, stroke services and must incorporate assessments of the humanity of patient care. See Jocleyn Cornwell, The Point of Care, King’s Fund, and the Institute of Healthcare Improvement, Boston MA.

• Boards’ assessments against agreed standards should be published at least annually supplemented where possible by the inspector's judgements.

• Standards against which Boards’ self assessments and inspectorate’s assessments are made should be both “core” and “aspirational”.

Setting, monitoring, improving and enforcing healthcare standards

• Boards should have locally owned action plans to improve patient care delivery and these should be regularly reviewed. Progress should be measured through improvements in clinical outcome, process and patient experience.

• Inspectorates should have a repertoire of interventional instruments to require and where necessary compel organisations to improve patient care. These should be a clear, well understood interventional escalator. The intervention selected will be proportionate to the scale of performance failure. These instruments should be consonant with other inspectorate’s legal duties and requirements. Boards should be held to account to deliver improvements.

A chief executive’s perspective: how inspectors’ findings are best implemented

• Boards provide a vital role in setting the values, direction, tone and public accountability and responsibility for the organisation’s delivery of high quality patient care. The regulatory environment should be designed to take this into account.

• Boards must demonstrate that patient safety is the top priority.

• On occasions e.g. investigations into major service failure, inspectorates must examine the respective contributions of all levels of NHS/DH management

• Boards should compare their own organisation against others to understand their relative performance across the breadth of patient services. This requires high quality data and information and investment in information systems and analytical capacity.

A chief executive’s perspective: how inspectors’ findings are best implemented

• Boards should demonstrate the capacity to deliver high quality care using available resources wisely. Boards should avoid asymmetrical attention to the domains of money and patient safety and patient experience.

• There should be movement of staff between NHS providers and inspectors to ensure that both have a good understanding of the other’s perspective. This should help avoid “provider capture” but achieve “provider understanding” and vice versa.

• Inspectorates need to earn credibility among multiple stakeholders through sound, wise, thoughtful and proportionate judgements and responses. Such credibility is to a degree correlated to the inspectorate's access to contemporary provider working experience.

A chief executive’s perspective: how inspectors’ findings are best implemented

• Welcome the opportunity for staff to become peer reviewers as this develops skills and experience in the organisation to improve patient care (“trickle down” effect).

• Inspectors should recognise the interorganisational opportunities to improve patient care and ensure that all appropriate elements of the local healthcare system are assessed.

• Inspectorates must operate as a coherent system even though they may be statutorily separate. This is to avoid a conflicting inspectorial requirements upon providers.

Coda• Excellent healthcare Boards focus on how to deliver high quality patient

care within available resources. Excellent Boards do not describe this as mutually exclusive requirements but rather as a complex simultaneous equation that must be solved using the considerable commitment, energy and ingenuity that so often define healthcare staff.

• Constant NHS and inspectorial reorganisations inhibit development of well established relationships and inspectorial methods and improvement plans. This reduces momentum to improve services.

Coda• The separation of financial/governance and quality

functions between Monitor and CQC creates additional complexity for NHS providers and both regulators.

• The further a commentator from the healthcare front line the more time is devoted to speculating about “gaming” of inspectorial regimes.

Coda

• Regulators’ legal structures and duties provide the regulatory anatomy. However, it is the leadership of inspectorates that provide the physiology i.e how well they work.

• Sound legal structures can be confounded by insular leadership behaviours. Suboptimal legal arrangements can be made to work through well intended leadership.

• The tough economic environment in the NHS is going to place increasing pressure on providers, commissioners and inspectorates.