@Lauren_Houston

Lauren Houston¹, Dr Yasmine Probst¹, Dr Allison Humphries¹

¹School of Medicine, Faculty of Science, Medicine and Health,

University of Wollongong

lah993@uowmail.edu.au

Measuring data quality

through a source data

verification audit

@Lauren_Houston

Health data has long been

scrutinised1,2

A large proportion of errors

are from transcribing data3,4

No “gold standard” method exists

to measure data quality error rates.

Background

1. Y.W. Lee et al. (2006) 3. M. Mealer et al. (2013)

2. M.N. Zozus et al. (2014) 4. M.L. Nahm et al. (2008)

@Lauren_Houston

What is source data

verification?

Source data

Electronic

record

Case report

form

@Lauren_Houston

What are the gaps in the knowledge? ICH GCP guidelines are non specific to amount, timing

and frequency of monitoring5

Cost-effectiveness of SDV6

No single definition to define data quality or universally

accepted method to measure error rates7,8

Audits may be published but not for public viewing8

5. ICH GCP (1996) 7. R.Rostami et al. (2009)

6. C. Baigent et al. (2008) 8. R.V. Gómez-Rioja et al. (2013)

@Lauren_Houston

Aim

To monitor data quality

by developing and conducting

source data verification

audits to achieve

quality assurance.

@Lauren_Houston

Study Background

UOW and IHMRI

Healthy-lifestyle blinded-RCT

12 months

5 clinical Accredited Practising Dietitians

To limit bias the audit was blinded

@Lauren_Houston

100% SDV on the 10% random sample

Quality assurance rule developed whereby if,

>5% of data variables were incorrect a second

10% random sample was extracted

Manual verification checks conducted

Calculation of error rate

Method

@Lauren_Houston

Participants

n=210

Random sample

n=21

Total of 820 data points

from 21 participants

If >41 (5%) data points

are incorrect

Second 10% random

sample of original

participants

Procedure of data audit

Participants

n=210

Random sample

n=21

Total of 685 data points

from 21 participants

If >34 (5%) data points

are incorrect

100% SDV of all data

points

10%

random

sample

100%

SDV

@Lauren_Houston

Audit 1

Source data verification

Anthropometric

Physiological

Medications

Electronic

spreadsheet

record

Coded

electronic

spreadsheet

record

Source

documents

@Lauren_Houston

Audit 2

Source data verification

Anthropometric

Medications

Electronic

spreadsheet

record

Coded

electronic

spreadsheet

record

Source

documents

@Lauren_Houston

Coded electronic

spreadsheet

@Lauren_Houston

Statistical Analysis

Total error = (code 2+3+4) / (code 1+2+3+4)

Data “not entered” (code 5) – excluded

Chi square, p<0.05

Post-hoc; adjusted standardised

residuals and z test of column

proportions

@Lauren_Houston

0

0.5

1

1.5

2

2.5

3

Anthropometric Physiological Medications

%Minor error

Audit 1 Audit 2

0

2

4

6

8

10

12

14

16

Anthropometric Physiological Medication

%

Major error

Audit 1 Audit 2

Results

@Lauren_Houston

0

10

20

30

40

50

60

Anthropometric Physiological Medication

%Not recorded data

Audit 1 Audit 2

0

5

10

15

20

25

30

Anthropometric Physiological Medication

%Not entered data

Audit 1 Audit 2

Results

@Lauren_Houston

0

10

20

30

40

50

60

70

80

90

100

Anthropometric audit 1

Anthropometricaudit 2

Medicationaudit 1

Medicationaudit 2

Totalaudit 1

Totalaudit 2

% Correct Incorrect

Correct and Incorrect

@Lauren_Houston

ResultsChi square had a significant difference;

χ2 (4, 1293) = 672.405, p = 0.000

Adjusted standardised residuals determined audit

sections were significantly different

From the z test of column proportions

anthropometric audit 1 and 2; medications audit 1

and 2 do not differ.

All other sections differed from each other.

@Lauren_Houston

SummaryAudit 1 physiological section <5% error

Average total error anthropometric (9%), medications

(76%) and overall (34.5%)

Proportion of error trended upward as length of study

increased

“Not recorded” (code 4) data had the greatest

contribution to total error

@Lauren_Houston

DiscussionImportance of identifying errors and determining

solutions

If >10% of a clinical dataset is erroneous the data

may be considered unreliable

Developed a 5% quality assurance rule

Data quality variations

@Lauren_Houston

Source documents considered the “gold standard”

Source document-to-electronic spreadsheet

Audits cannot guarantee 100% free from error

Clinical research setting and trial design

Did not determine the impact of audit findings

Limitations

@Lauren_Houston

Examine the 10% snapshot model with a 5% quality

assurance error rate

Standardise a SDV audit process

Assess the frequency and

cost-effectiveness

Overcome barriers and increase awareness

Recommendations

@Lauren_Houston

How many

errors are too

many?

@Lauren_Houston

AcknowledgementsDr. Yasmine Probst

Dr. Allison Humphries

Sr/Prof. Linda Tapsell

A/Prof. Marijka Batterham

Illawarra Health and Medical

Research Institute

Smart Foods Centre

All participants and staff

involved