SURGICAL AUDIT AND

CLINICAL RESEARCH

“Surgery without audit is like playing cricket without keeping the score.”(Hugh brendon devlin 1932-1998)”

Definition

Clinical audit is a quality improvement

process that seeks to improve patient care

and outcomes through systematic review of

care against explicit criteria and the

implementation of change.

the word ‘auditing’ has been derived from

Latin word “audire” which means “to hear”.

Surgical audit is a systematic, critical analysis

of the quality of care that is reviewed by peers

against explicit criteria or recognised

standards, and then used to further improve

the surgical practice with the ultimate goal of

improving the care of patients.

Surgical Audit –why do it?

To identify whether standards are being met,

and evidence from research are being used in

practice.

To identify baselines for development of

standards.

To reduce clinical risk.

To ensure cost effective use of resources, and

effectiveness of a service.

To highlight problems, and help in the solution.

To improve team working and communication.

To improve patient care and outcomes.

HISTORY

One of first ever clinical audits was

undertaken by Florence Nightingale during

the Crimean War of 1853-1855.

Aspects of patient care in audit

1. Structure – what is in place.

2. Process – what you do

3. Outcome – the results you get

1. Structure – what is in place The people,

Their training,

Their knowledge,

The way they are led,

The equipment,

Their organization,

The way they are paid, etc.

2. Process – what you do How referrals are processed,

What diagnostic tests are done,

The antibiotics that are used,

The thromboembolic prevention that is customary,

The use of intensive care,

The policy of feeding & mobilization after surgery,

The discharge policy, etc.

3. Outcome – The Results

You Get

Wound dehiscence rate,

Readmission rates,

Mortality,

Freedom from progression,

Reduction in symptoms,

Improvement in quality of life,

Return to work, etc.

Explicit Criteria: An individual who needs training

An instrument that needs replacing

At team level e.g. nurses undertaking procedures instead of, or

in addition to, doctors

At institutional level e.g. new antibiotic policy

At regional level e.g. provision of a tertiary referral centre

At national level e.g. screening programmes & health

education campaigns

Audit – can be done in different forms

A personal surgical audit (total/ practice/

selected);

Group/ hospital / specialty audit (focused or

generic).

Total Practice or Workload Audit: This is an audit

that covers all the surgical operations performed.

Selected Audit from Surgical Practice:

This is an audit that covers all patients who

undergo a selected procedure, or an audit that

covers all procedures conducted within a selected

time-frame.

A Clinical Unit Audit:

This is an audit conducted by a clinical unit in

which individual surgeons may participate.

Group or Specialty Audit:

This is an audit conducted by or under the auspices of a

group or Specialty Society

e.g. the National Breast Cancer Audit,

A Focused Audit:

This is an audit which looks at one or more Council for

Health Care Standards indicators and the factors which

influence it:

e.g. what is the wound infection rate after large bowel

surgery – emergency/ elective procedure, type of surgery,

antibiotic prophylaxis blood loss

There are a number of types of audit that take place within

an institution, including:

• Morbidity and mortality meetings

• Local/regional audit

• National or international comparative audit.

Purpose

Selected cases are presented at mortality and

morbidity review (M&M) meetings for the

purpose of:

• Discussing management decisions

• Providing a learning opportunity

• Identifying opportunities to improve patient

safety and quality of care

Preparation

Identify clinicians who will form the core group for

the department M&M meetings

Appoint a senior consultant to be the chair and to

have responsibility for meeting management.

Appoint a registrar or fellow with responsibility for

case coordination

Book a regular meeting time. It is a requirement that

meetings are held monthly.

Case identification

Cases presented at department M&M meetings

will be identified from a range of sources

including:

• all deaths which occur under the unit

• any case referred by a clinician

• Any case referred via the organisation.

standardized format for M&M presentation

Conduct of meeting

Encourage a focus on patient care.

Establish a ‘safe’ environment

M & m meetings are not convened for the purpose

of focussing on an individual’s performance.

Promote participation from those attending the

meeting

Ensure brief minutes and action items are taken

Focus the discussion on identifying what went

wrong, why it went wrong, what could be done

differently in the future and what action is

required

The Clinical Audit Cycle

Choose Topic

In this stage the audit team will identify a problem, or an area of healthcare service

that is to be compared to standards.

When Choosing A Topic It Is Recommended To Focus On Areas Where

Standards and guidelines exist, and there is conclusive

evidence about effective clinical practice.

Problems have been encountered in practice.

There have been recommendations or complains from the

patients or public.

There is high volume, high risk, or high cost.

There is clear potential of improvement of service

Determining scope

Common areas in the scope of an audit include:

30 day mortality and significant morbidity;

Length of hospital stay;

Positive and negative outcomes

Operation-specific complications

Use of investigations

Justification of management

Patient satisfaction.

Choose Topic

Decisions regarding the overall purpose of the audit, either as what should happen as a result of the audit, or what question you want the

audit to answer, should be written as a series of statements or tasks that the audit will focus on. Collectively, these form the audit criteria.

A criterion is a measurable outcome of care. For example,

‘Patients in the ICU should receive GIT bleeding

prophylaxis’.

A standard is the threshold of the expected compliance for

each criterion (these are usually expressed as a percentage).

For the above example an appropriate standard would be:

‘GIT bleeding prophylaxis received in 90% of cases’.

Criteria can be classified as Structure criteria.

Process criteria.

Outcome criteria.

Remember, Criteria of an audit should be:

S

M

A

R

T

Clear, not vague

Objectively

Realistic

To the topic

As well as Theoretically sound

Choose Topic

•Data collection sheets or questioners developed by the audit team.

COLLECT DATA

The information necessary to answer the audit question.

Collect prospectively or retrospectively.

Follow up data collected.

Data can be collected from a register, medical records

data, review of referrals, or from previous appointment

schedules.

Choose Topic

•Simple Statistical analysis based on the type of data collected.•Graphical presentation.

Analysis stage, whereby the results of the

data collection are compared with criteria

and standards.

The end stage of analysis is concluding

how well the standards were met and, if

applicable, identifying reasons why the

standards weren't met in all cases.

These reasons might be agreed to be

acceptable, i.e. could be added to the

exception criteria for the standard in

future, or will suggest a focus for

improvement measures.

In theory, any case where the standard was not met in 100% of

cases suggests a potential for improvement in care.

In practice, where standard results were close to 100%, it

might be agreed that any further improvement will be difficult

to obtain.

And that other standards, with results further away from

100%, are the priority targets for action.

This decision will depend on the topic area, in some ‘life or

death’ type cases, it will be important to achieve 100%, in

other areas a much lower result might still be considered

acceptable.

Choose Topic

Discuss your results…

Conclude if standards were met, and if not,

because of what.

What needs to be done about that?

Is the solution practical?

Was the standard itself applicable ?

Share your results…

Once the results of the audit have been published

and discussed, it is time to formulate and action

plan, that should include;

o What needs to be done or changed.

o Who is going to do it.

o When it is going to be done.

o How it will be done.

o Involve higher authorities.

Choose Topic

RE –AUDITING CYCLE… After an agreed time frame, the audit should be

repeated.

The same methods and data analysis are used to ensure

comparability.

The re-audit should demonstrate that the changes have

been implemented and that improvements have been

made.

This stage is critical to the successful outcome of the

audit process as it verifies whether the changes

implemented have had an effect.

Summary – The five stage approach toclinical audit

Step-By-Step Guide for Doing An Audit

Stage 1 – preparing for audit

Think broadly.

Funding..

Skills.

Time.

Teamwork

Stage 2 – selecting criteria

Think big.

Check guidelines.

Systematic reviews..

Process or outcome.

Stage 3 – measuring the level of performance

Routine data..

Electronic data.

Medical records.

Abstract data.

Legalities.

Stage 4 – making improvements

Barriers..

Feedback..

Discussion..

Implementation methods.

Stage 5 – sustaining improvement Re-audit.

Structural change.

Cultural change. .

What Makes Effective Audit?1. Promotion of a culture of audit: audit is undertaken in an

atmosphere that highlights educational aspects, is regarded as

non-threatening or ‘safe’, and is carried out in a culture of ‘no-

blame’. This atmosphere enables open discussion of findings,

and participants will be able to discuss their feelings concerning

audit reviews.

2. Allocate time and resources: Audit should not be allowed

to become a burden, as this will make participation difficult. It

should be considered as part of normal clinical practice.

3. Oversee and verify data collection

The data should be accurate and complete, with clinical

details provided by clinicians.

Don’t forget to look for:

• The complication that didn’t occur;

• The death that was missed;

• The house surgeon’s diagnosis that was misconceived;

• The misinterpreted pathology report; and

• The reason for the misdiagnosis

4. Productive peer review:

Good follow up and implementation of change requires the surgeons to

work closely with management and putting in place systems for quality

improvement and risk management

Privacy

Confidentiality in audit process is essential.

It is also important to reassure participating surgeons

and other team members that peer review discussions

constitute confidential professional peer review rather

than a witch hunt.

Educational opportunities that can arise from audit

include:

• Encouraging collaboration, modifying attitudes and approaches to

clinical problems;

• Enhancing critical approaches and giving a rational basis to local changes

in clinical practice;

• Encouraging learning about new technologies and procedures and

auditing their introduction to provide justification;

• Indicating deficiencies in knowledge and skills, which leads to

development of educational activities to address these; and

• By developing required standards of care, giving guidance as to what is

expected

Is Clinical Audit A research

Research asks:Are we singing the right song?

Audit asks:Are we singing this song right?

Patient registry

A patient registry gathers data on patients

in a systematic way. Such registries can

be very helpful in understanding the

levels of service provision and can have a

place in research and audit.

Service evaluations Evaluations usually assess the details of the

service provided. Unless the evaluation

assesses the service against explicit criteria

it cannot be regarded as clinical audit.

Research study and study design Research is designed to generate new

knowledge and might involve testing a new

treatment or regimen. COMPONENTS OF RESEARCH:

1. IDENTIFYING A RESEARCH TOPIC.

2. PROJECT DESIGN.

3. STATISTICAL ANALYSIS.

4. ANALYSING A SCIENTIFIC ARTICLE.

5. PRESENTING AND PUBLISHING AN ARTICLE

WHAT IS THE PURPOSE OF THE STUDY?

A good clinical study starts with

a good question based on good hypothesis that is based on good and

comprehensive review of the available evidence from pre-clinical and

clinical data

Type of design depends on the question to be answered

Most studies can be placed into one of two general categories, descriptive

(or exploratory) and analytic.

Good research study -FINER criteria

Feasible

Interesting

Novel

Ethical

Relevant

WHAT IS BEING COMPARED?

1. Complications,

2. Cost, efficacy,

3. Effectiveness,

4. Quality of life,

5. Functional status,

6. Patient satisfaction

WHAT IS THE OUTCOME OF INTEREST?

1. Safety.

2. Effectiveness and Efficacy.

3. Patient-Reported Outcomes.

4. Resource Utilization.

5. Costs

SafetySafety end points capture

The inherent risks of an operation (e.., Surgical site

infection),

Natural history of the underlying disease process in the

context of therapy (e.g, Malignancy-associated deep

venous thrombosis in a postoperative patient), and/or

Operative mortality and postoperative complications

(morbidity) are the most commonly measured markers of

safety

Effectiveness and Efficacy

Efficacy refers to the extent to which a treatment intervention

achieves its purported benefit and the durability of that result.

Efficacy is usually, requires comparison of the selected

intervention to a control group, may include randomization, and

usually necessitates longer follow-up.

Efficacy studies are more challenging to execute and more

expensive to fund than simple descriptive studies

Effectiveness relates to outcomes in real-world practice.

Patient-Reported Outcomes Patient-reported outcomes measure subjective outcomes of care

reported by the patient directly, without further interpretation of

this response by a provider or researcher.

PRO data are collected through the use of survey instruments.

Examples of common PRO concepts are

Health-related quality of life (HRQOL),

Satisfaction with care,

Functional status,

Well-being, and health status.

Resource Utilization

Resource utilization refers to the use of health

services related to an intervention.

Includes

Length of stay,

Hospital readmission,

Use of outpatient,

Pharmacy, and durable medical equipment (e.G.,

Wheelchairs and oxygen) services, and

Emergency room use

Costs

Charges are the amount of money requested for health

services and supplies. By comparison, costs are the actual

amount of money required to deliver care.

A cost-utility analysis quantifies health benefits in terms of

quality-adjusted life-years (QALYs). Utilities are a measure

of overall quality of life, usually scaled between 0 and 1,

with 1 being perfect health,

Incremental cost-effectiveness ratio (ICER), which is the

difference in costs between two competing therapeutic

options divided by the difference in health outcome.

WHAT IS THE STUDY DESIGN?

67

Clinical Study Types

Observational Studies Cohort (Incidence, Longitudinal)

Case-Control

Cross-Sectional (Prevalence)

Case Series

Case Report

Experimental Studies Uncontrolled Trials

Controlled Trials

Most descriptive studies should be considered

hypothesis-generating rather than causality-focused

whereas analytic studies test a prespecified

hypothesis.

Important issues in Study Design

Validity: Truth

External Validity:

Can the study be generalized to the population

Internal Validity:

Results will not be due to chance, bias or

confounding factors

Symmetry Principle: Groups are similar

Misclassification

Confounding: distortion of the effect of one risk factor by

the presence of another

Bias: Any effect from design, execution, & interpretation

that shifts or influences results

Confounding bias: failure to account for the effect of

one or more variables that are not distributed equally

Measurement bias: measurement methods differ

between groups

Sampling (selection) bias: design and execution errors

in sampling

Misclassification Misclassification is the incorrect categorization of a subject into a

study group.

There are two types of misclassification,

1. Non-differential

2. differential.

Non-differential misclassification indicates an equal and random

chance that any one subject will be misclassified (or included as

part of the wrong study group).

Differential misclassification, the chance a subject is misclassified

is nonrandom.

Stage migration, also known as the Will Rogers

phenomenon, is a classic example of

misclassification.

Patients only assessed clinically might be

understaged—categorized as early-stage cancer will

have less survival rate or if overstaged patients were

considered with truly late-stage patients, survival

would be better than in actuality.

Characteristics of observational studies

Can study risk factors that have serious consequences

Study individuals in their natural environment (>> extrapolation)

Possibility of confounding.

Evaluate the effect of a

suspected risk factor

(exposure) on an outcome(e.g.

disease) define

‘exposure’ and ‘disease.

Describe the impact of the risk

factor on the frequency of

disease in a population

Cross - Sectional Study

Exposure and disease measured once, i.e. at the same point in time

present futurepast

n

exposed ?diseased ?

Cross - Sectional Study

Random sample from population

i.e. results reflect reference population

Estimates the frequencies of both exposure and

outcome in the population

Measuring both exposure and outcome at one point

in time

Typically a survey

Cross - Sectional Study

Can study several exposure factors and outcomes

simultaneously

Determines disease prevalence

Helpful in public health administration & planning

Quick

Low cost (e.g. mail survey)

Limitation:

Does not determine causal relationship

Not appropriate if either exposure or outcome is rare

Cohort studies Follow-up studies; subjects selected on presence or

absence of exposure & absence of disease at one point

in time. Disease is then assessed for all subjects at

another point in time.

Typically prospective but can be retrospective,

depending on temporal relationship between study

initiation & occurrence of disease.

Cohort Study

Individuals selected by exposure status and future occurrence of disease measured

present futurepast

n

Exposed yes no

disease ?disease ?

n

Exposed yes no

disease ?disease ?

Cohort studies

More clearly established

temporal sequence between

exposure & disease

Allows direct measurement of

incidence

Examines multiple effects of a

single exposure (OC and breast,

ovarioan cancers)

•Limitations:

• time consuming and

expensive

• loss to follow-up &

unavailability of data

• inefficient for rare diseases

Case-Control Study

Good for rare disease (e.g. cancer)

Can study many risk factors at the

same time

Usually low cost

Confounding likely

Case-Control Study Design

Cases

Controls

Exposed

Unexposed

Exposed

Unexposed

TimeDatacollection

Direction of inquiry

Q: What happened?

81

Case-Control study

• Study subjects selected on basis of whether

they have (case) or do not have (control) a

disease

• Useful for disease with long latency period

• Efficient in terms of time & costs

• Particularly suited for rare diseases

• Examines multiple exposures to a single

disease

Case-control study

Limitations:

(1) Susceptible to bias (particularly selection &

recall)

(2) Difficulties in selection of controls

(3) Ascertainment of disease & exposure status

(4) Inefficient for rare exposures unless

attributable risk is high

Randomized Controlled TrialsAn experimental comparison study where participants are allocated

to

treatment/intervention or control/placebo groups using a random

mechanism. Best for studying the effect of an intervention.

Advantages:

unbiased distribution of confounders

blinding more likely

randomisation facilitates statistical analysis

Disadvantages:

expensive: time and money

volunteer bias

ethically problematic at times

Meta-Analysis Meta-analysis is a technique that pools available published

data in an effort to increase the statistical power of an

analysis.

QUOROM (Quality of Reporting of Meta-Analyses)and

MOOSE (Meta-Analysis of Observational Studies in

Epidemiology) guidelines have been developed to ensure the

quality and validity of results obtained through meta-analysis

HOW WERE THE DATA ANALYZED?

A continuous variable is one that can take on an infinite number of values. Age and length of stay are examples of a continuous variable.

Categorical variables have discrete values.

The simplest categorical variable is a binary variable that can only take on one of two values, such as sex [male, female]).

Ordinal variables are ordered categorical variables. Cancer stage is a classic example of an ordinal categorical variable.

Nominal variables are unordered categorical variables, such as race.

Hypothesis Testing Hypothesis testing uses comparative or analytic statistics to

determine whether observed differences between two or more

groups are real or are attributable to chance.

The P value is a statistical summary measure for hypothesis

testing.

A significance level of 5% (P = .05) is widely used to indicate a

statistically significant finding, although this value is arbitrary.

P value is interpreted as the probability that the observed

difference in outcomes between groups is the result of chance.

For smaller studies: Hypothesis testing is also

possible by examining 95% confidence interval

A wide CI indicates a lack of precision, whereas

a tight (small) interval would be indicative of

minimal uncertainty

Two types of errors can occur with hypothesis testing.

An alpha (or type I) error occurs when one

observes a difference in outcomes when one does

not actually exist.

A beta (or type II) error occurs when no difference

in outcomes is observed when a difference truly

exists (a false-negative finding).

The unpaired t-test is used to compare two independent

groups that have continuous outcome variables.

Paired t-test is used to compare two dependent groups that

have continuous outcome variables.

An analysis of variance (ANOVA) is used when comparing

more than two groups with a continuous outcome variable.

The chi-square statistic is often used to compare the

distributions of two or more groups with categorical

outcome variables.

Fisher’s exact test is more appropriate for such

comparisons when the sample size is small

Multivariable Analysis Multivariate regression models are among the most

commonly used methods to evaluate the relationship between

variables and outcomes in the absence of the influence of

other measured variables.

Linear regression is used to evaluate the relationship

between factors potentially associated with a continuous

outcome variable, such as length of stay.

Logistic regression is used when the outcome variable is

binary (e.g., operative mortality)

Propensity Score Analysis

Propensity score analysis is an alternative method of

risk adjustment. When two groups are being

compared, logistic regression is used to calculate a

given subject’s risk or probability (or propensity) of

having an exposure of interest (e.g., minimally

invasive as compared with open surgery).

propensity scores are not superior to multivariate

techniques, but are simply an adequate alternative

Instrumental Variable Analysis

Instrumental variable analysis is another method of

accounting for unmeasured confounding and controlling

bias.

The principle underlying this type of analysis is that there

are unmeasured, or immeasurable, confounders that might

bias the study’s results.

The best example of an instrumental variable is

randomization.

ARE THERE ETHICAL CONSIDERATIONS?

In the first instance, common sense is

the best guide to whether or not a study

is ethical.

Still important to seek advice from an

ethics committee whenever research is

contemplated.

ANALYSING A SCIENTIFIC ARTICLE

Take Home Message

Keep audits simple

Get everyone involved

Do not confuse clinical audit with research.

Take care with statistics – errors can lead to inaccurate conclusions

Share learning - tell everyone about your audit

Tell ‘The Organisation’ about your audit

Re-audit to ensure improvement in clinical care

Only by becoming more critical

evaluators of the surgical

literature will the next

generation of surgeons be able

to embrace fully the promise of

evidence based surgery.

References

Bailey and love's short practice of surgery 26th ed.

Sabiston textbook of surgery, 19th edition.

Principles for best practice in clinical audit-national institute

for clinical excellence.

Clinical audit: handbook.

A practical guide to clinical audit-the national clinical audit

advisory group