Verification of Results

Group Work

Petra Vergeer | Senior Health Specialist

Overview

1

• Operational data from RBF programs is of limited use

unless it is accurate.

• A cross-case analysis was carried out to expand knowledge

about verification processes and practices. o Afghanistan, Argentina, Burundi, and Rwanda

• Introduction of verification “pathway tool” o Initial consultation to obtain feedback to further develop tool

Verification Pathway Tool

Examine quantity verification error rates

1

Facility-level patterns

Indicator-level patterns

Generalized

across all

contracted

parties

Localized to specific

facilities (by

geographic area)

Localized to

specific types

of facilities

Indicators with

complex

compliance

criteria

Indicators that are

rewarded more

frequently and/or

have higher patient

volume

Indicators

rewarded at a

higher level

Examine quantity

verification error rates

Activities to

explore

Activities to

explore

Examine patient tracing error rates

Examine patient tracing error

rates

Facility-level patterns

Indicator-level patterns

Generalized

across all

contracted

parties

Localized to specific

facilities (by

geographic area)

Localized to

specific types

of facilities

Indicators with

complex

compliance

criteria

Indicators that are

rewarded more

frequently and/or

have higher patient

volume

Indicators

rewarded at a

higher level

Activities to

explore

Activities to

explore

Examine counter-verification error rates Verifier-level patterns

Indicator-level

patterns

Generalized

across all

verified

parties

Localized

to a

specific

verifier

Localized

to a

specific

type of

facility Examine quantity

and quality counter-verification error

rates (Assume data are at level

of contracted party.)

Indicators with

complex

compliance

criteria

Indicators that are

rewarded more

frequently and/or

have higher patient

volume

Indicators

rewarded at a

higher level

Activities to

explore

Activities to

explore

Factors influencing verification: a

conceptual framework Context

Verification Characteristics Impact on accuracy,

cost, sustainability

RBF Characteristics RATIONALE FOR RBF CONTRACT TYPE

USE OF RBF RESULTS

Improving health outcomes/HSS Relational

Payment, improving performance

Financial accountability/Cost control Classic

Transparency, Naming and Shaming

Monthly Annual

Yes

Large

Whole universe Risk-based approach Internal

Verification Results and Their Use

FREQUENCY

ALLOWABLE ERROR MARGIN

SAMPLE SIZE INSTITUTIONAL SETUP ADVANCE WARNING No

Small

Third party

Learning, Error correction Cost recovery, Sanction

PAYMENT FREQUENCY Monthly Annual

POLITICAL ENVIRONMENT GOVERNANCE CULTURE

Potential causes of generalized error rates:

verification • Knowledge and tools

o Insufficient training for providers, data entry clerks

o Data collection tools insufficiently clear

o Explore validity of verification methods

• Allowable error margin o Assess error margin

• Advance warning

• Institutional setup o There is a conflict of interest between the verifier and the contracted party

• Sample size o Data are “noisy”

• Frequency o Verification/counter-verification occurs too infrequently to deter gaming

• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting

Question: What are the causes in your country?

Back to

main slide

Potential causes of generalized error rates:

patient tracing • Knowledge and tools

o Insufficient training for providers, data entry clerks

o Data collection tools insufficiently clear

o Explore validity of verification methods

• Allowable error margin o Assess error margin

• Advance warning

• Institutional setup o There is a conflict of interest between the verifier and the contracted party

• Sample size o Data are “noisy”

• Frequency o Verification/counter-verification occurs too infrequently to deter gaming

• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting

Question: What are the causes in your country?

Back to

main slide

Potential causes of generalized error rates:

counter-verification • Knowledge and tools

o Insufficient training for providers, data entry clerks

o Data collection tools insufficiently clear

o Explore validity of verification methods

• Allowable error margin o Assess error margin

• Advance warning

• Institutional setup o There is a conflict of interest between the verifier and the contracted party

• Sample size o Data are “noisy”

• Frequency o Verification/counter-verification occurs too infrequently to deter gaming

• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific geographic area: verification

• Knowledge and tools o Insufficient support for contracted parties

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency o Verification occurs too infrequently to deter gaming

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific geographic area: patient tracing

• Knowledge and tools o Insufficient support for contracted parties

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency o Verification occurs too infrequently to deter gaming

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific verifier: counter-verification

• Knowledge and tools o Insufficient support for verifiers

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency o Verification occurs too infrequently to deter gaming

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific type of facility: verification

• Knowledge and tools

• Allowable error margin o Explore whether error margin is appropriate

• Advance warning

• Institutional setup

• Sample size o Biased data due to insufficient sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific type of facility: patient tracing

• Knowledge and tools

• Allowable error margin o Explore whether error margin is appropriate

• Advance warning

• Institutional setup

• Sample size o Biased data due to insufficient sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of error rates localized to a

specific type of facility: counter-verification

• Knowledge and tools

• Allowable error margin o Explore whether error margin is appropriate

• Advance warning

• Institutional setup

• Sample size o Biased data due to insufficient sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators with complex compliance criteria:

verification

• Knowledge and tools o Confusing data collection tools

o Inadequate training for providers and data entry clerks

• Allowable error margin o Indicator too complex for accurate reporting

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators with complex compliance criteria:

patient tracing

• Knowledge and tools o Confusing data collection tools

o Inadequate training for providers and data entry clerks

• Allowable error margin o Indicator too complex for accurate reporting

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators with complex compliance criteria:

counter-verification

• Knowledge and tools o Confusing data collection tools

o Inadequate training for providers and data entry clerks

• Allowable error margin o Indicator too complex for accurate reporting

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among indicators that

are rewarded more frequently or have higher patient

volume: verification

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among indicators that

are rewarded more frequently or have higher patient

volume: patient tracing

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among indicators that

are rewarded more frequently or have higher patient

volume: counter-verification

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators that are rewarded at a higher level:

verification

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup o Incentive amount encourages gaming

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators that are rewarded at a higher level:

patient tracing

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup o Incentive amount encourages gaming

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Potential causes of high error rates among

indicators that are rewarded at a higher level:

counter-verification

• Knowledge and tools

• Allowable error margin

• Advance warning

• Institutional setup o Incentive amount encourages gaming

• Sample size

• Frequency

• Verification results and use

Question: What are the causes in your country?

Back to

main slide

Examples of options to address high error rates

• Visit small number of facilities to investigate underlying

causes of error rate patterns

• Increase sanction amount/proportion

• Conduct additional trainings for providers, data entry clerks

• Provide additional support on an ongoing basis for

providers, data entry clerks

Thank You