25
ETA Data Validation ETA Data Validation July 2003 July 2003

ETA Data Validation July 2003. Overall ETA Data Validation Project Goals Develop a comprehensive, systematic data validation system to ensure data integrity

Embed Size (px)

Citation preview

ETA Data ValidationETA Data Validation

July 2003July 2003

Overall ETA Data ValidationProject Goals

Develop a comprehensive, systematic data validation system to ensure data integrity across programs

Increase uniformity in data definitions and data Increase uniformity in data definitions and data collection across similar programscollection across similar programs

Strike the proper balance between data integrity and burden to achieve acceptable, sustainable level of error

Coordinate closely with DOL Dept. of Inspector General on methods and approach

Validity and VerificationDept. of Labor Perspective

Develop reputation for reliable and accurate Develop reputation for reliable and accurate program dataprogram data

Administration’s focus on management and Administration’s focus on management and accountabilityaccountability

Improve basis for incentives and sanctionsImprove basis for incentives and sanctions

Basis for continuous improvementBasis for continuous improvement

Programs Included

Unemployment Insurance Benefits and Tax (UI)Unemployment Insurance Benefits and Tax (UI) Workforce Investment Act (WIA)Workforce Investment Act (WIA) Trade Adjustment Assistance (TAA and NAFTA-TAA)Trade Adjustment Assistance (TAA and NAFTA-TAA) Labor ExchangeLabor Exchange Migrant and Seasonal Farm Worker Program (MSFW)Migrant and Seasonal Farm Worker Program (MSFW) Division of Indian and Native American Programs Division of Indian and Native American Programs

(DINAP)(DINAP) Senior Community Service Employment (SCSEP)Senior Community Service Employment (SCSEP) Office of Apprenticeship, Training, Employment, and Office of Apprenticeship, Training, Employment, and

Labor Services (OATELS)Labor Services (OATELS)

Stages of the Project

1.1. Reporting, performance and validation Reporting, performance and validation requirements analysis and specificationsrequirements analysis and specifications

2.2. Develop validation toolsDevelop validation tools

3.3. Pilot validation methodologyPilot validation methodology

4.4. TrainingTraining

5.5. Technical assistanceTechnical assistance

1. Requirements Analysisand Specifications

Requirements analysis and specifications Requirements analysis and specifications document the reporting and performance needs document the reporting and performance needs of each relevant ETA programof each relevant ETA program

Documentation is organized in the ETA Reporting Documentation is organized in the ETA Reporting and Performance Databaseand Performance Database

Database defines each data element and reporting Database defines each data element and reporting specification for each report and performance specification for each report and performance itemitem

2. Develop Validation Tools – Handbooks

Handbooks contain reporting specs and Handbooks contain reporting specs and validation instructions, including validation instructions, including acceptable source documentationacceptable source documentation

SCSEP awaiting final specs SCSEP awaiting final specs

LX has no handbook, just software and LX has no handbook, just software and user’s guide, no case-record level data user’s guide, no case-record level data validation at this timevalidation at this time

2. Develop Validation Tools – Software

Software completed for LX, WIA, and TAASoftware completed for LX, WIA, and TAA

Software under development for MSFW and Software under development for MSFW and DINAPDINAP

Distribution of handbooks and software via Distribution of handbooks and software via ETA websitesETA websites

– LX: www.uses.doleta.gov/rptvalidation.aspLX: www.uses.doleta.gov/rptvalidation.asp

– WIA and TAA: www.uses.doleta.gov/dv/WIA and TAA: www.uses.doleta.gov/dv/

3. Pilot – State Programs

Pilot state programs – two formal state pilotsPilot state programs – two formal state pilots

– Texas – WIATexas – WIA– Washington State – WIA, TAA, LX Washington State – WIA, TAA, LX – Utah and West Virginia have been trainedUtah and West Virginia have been trained

LX was implemented in August 2002LX was implemented in August 2002

Other states are testing WIA Other states are testing WIA

4. Training

Regional training sessions are being held in Regional training sessions are being held in the summer of 2003 for WIA, LX, TAA the summer of 2003 for WIA, LX, TAA

Other programs Other programs –– determine training determine training strategy individuallystrategy individually

– Tie into national meetingsTie into national meetings– 2-3 sessions per program2-3 sessions per program

5. Technical Assistance

Phone and e-mail TA available Phone and e-mail TA available – Installing softwareInstalling software– Building and loading extract filesBuilding and loading extract files– Conducting report validationConducting report validation– Conducting data element validation Conducting data element validation – Contact information in software user’s guide and Contact information in software user’s guide and

help menu of softwarehelp menu of software

TA e-mail addressesTA e-mail addresses– For WIA: [email protected] WIA: [email protected]– For LX: [email protected] LX: [email protected]– For TAA: [email protected] TAA: [email protected]

How Data Validation Systems Improve Data Quality

Improve communication from ETA to programmersImprove communication from ETA to programmers

Provide a blueprint or roadmap to understand Provide a blueprint or roadmap to understand reporting and performance measurementreporting and performance measurement

Minimize burden of interpreting specifications Minimize burden of interpreting specifications

Provide clear standards for assessing validityProvide clear standards for assessing validity

Provide detailed diagnostic data for correcting Provide detailed diagnostic data for correcting problemsproblems

Report Validation

Given the data that are stored, is the software Given the data that are stored, is the software generating the correct countsgenerating the correct counts

Develop an audit trail to support the Develop an audit trail to support the numerators and denominators for each numerators and denominators for each performance outcomeperformance outcome

• Classifying participant records into Classifying participant records into performance outcome groups enables non-performance outcome groups enables non-technical staff to validate and analyze technical staff to validate and analyze program outcomesprogram outcomes

Data Element Validation

Report will not be accurate if the data being used by Report will not be accurate if the data being used by the software are wrongthe software are wrong

Requires checking data elements against source Requires checking data elements against source documentation to verify compliance with federal documentation to verify compliance with federal definitionsdefinitions

Handbooks contain instructions and examples of Handbooks contain instructions and examples of acceptable source documents for each data element acceptable source documents for each data element validatedvalidated– States identify state-specific source documentation to reflect States identify state-specific source documentation to reflect

the variability of state MIS systems and state/local the variability of state MIS systems and state/local documentation standardsdocumentation standards

Self-reported elements such as race, gender, and Self-reported elements such as race, gender, and ethnicity are not validatedethnicity are not validated

ETA Data Validation

Entered Employment Rate 6 8

Sample Portion WIA Annual Report

Adults

Numerator DenominatorState orGrantee

Database

DetailParticipant

RecordExtract

Case Files

Compare validationdata to source data

Data Element Risks

Low risk data elementsLow risk data elements

– Computer generated – wage recordsComputer generated – wage records– Human input with:Human input with:

Minimal judgement (e.g. dates)Minimal judgement (e.g. dates) Low performance impactLow performance impact

High risk data elementsHigh risk data elements– Human input with:Human input with:

Considerable judgement (interpreting rules)Considerable judgement (interpreting rules) High performance impact – supplemental High performance impact – supplemental

employment data employment data

Software Selects Samples for Data Element Validation

Sampled records are displayed on Sampled records are displayed on automated worksheetsautomated worksheets

Participant records with positive Participant records with positive outcomes not based on wage records are outcomes not based on wage records are over-sampledover-sampled

SoftwareSoftware– Adjusts error rates based on weights Adjusts error rates based on weights – Produces a detailed data element Produces a detailed data element

validation report with error rates for validation report with error rates for each data elementeach data element

Data Element Validation by Program

For WIA, TAA, LX, MSFW, DINAP, and SCSEP For WIA, TAA, LX, MSFW, DINAP, and SCSEP software generates worksheets for sampled software generates worksheets for sampled recordsrecords

For WIA and TAA cluster sampling used to For WIA and TAA cluster sampling used to reduce the number of offices to be visitedreduce the number of offices to be visited

For LX, no data validation against source For LX, no data validation against source documents — 25 cases are reviewed to documents — 25 cases are reviewed to ensure that file was built correctlyensure that file was built correctly

Benefits of Performance and Analysis Software

Provides technical assistance to states Provides technical assistance to states

Reduces burden on local offices and small statesReduces burden on local offices and small states

Clear and easy analysis of outcomes Clear and easy analysis of outcomes

– For example, impact of zero pre-program For example, impact of zero pre-program earningsearnings

Makes underlying performance data accessible to Makes underlying performance data accessible to managersmanagers

Breaks out performance by many factors and Breaks out performance by many factors and checks for errorschecks for errors

Software Allows forFlexible Data Analysis

Software will report by user-selected time period Software will report by user-selected time period (weekly, monthly, quarterly, annually)(weekly, monthly, quarterly, annually)

Users can also select reports by state or sub-state Users can also select reports by state or sub-state breakouts, including WIB, office, or case managerbreakouts, including WIB, office, or case manager– Not multiple offices per participant unless state loads Not multiple offices per participant unless state loads

separate filesseparate files– Software may be enhanced to allow multiple countsSoftware may be enhanced to allow multiple counts

Users can sort participant records by any field within Users can sort participant records by any field within performance outcome groups — will have 3 tiered performance outcome groups — will have 3 tiered sortsort

Users can also export participant groups for analysis, Users can also export participant groups for analysis, local feedback, or WRIS requestslocal feedback, or WRIS requests

Reporting of Validation Results

Software produces Software produces

– Report validation summaryReport validation summary

– Data element validation summary and analytical Data element validation summary and analytical reportsreports

WIA and LX software creates files with the annual WIA and LX software creates files with the annual report validation values for upload to ETAreport validation values for upload to ETA

Visual Basic Applications

Software requires any Windows operating Software requires any Windows operating system system

No other software requiredNo other software required

For large files, MS SQL Server is an option if For large files, MS SQL Server is an option if the state has a license (for UI and LX only)the state has a license (for UI and LX only)

Front-end edit checks ensure proper format of Front-end edit checks ensure proper format of recordsrecords

Next Generation Reporting and Performance System

In Fall 2004, states may use federal software In Fall 2004, states may use federal software to:to:

– Generate reportsGenerate reports– Perform and report on data validationPerform and report on data validation– Edit and transmit individual participant Edit and transmit individual participant

recordsrecords

Software likely to be developed as part of Software likely to be developed as part of new EIMS software development effortnew EIMS software development effort

Web-Based StateInternal Audit Tool

States want capability to perform data element States want capability to perform data element validation at sub-state levelvalidation at sub-state level

Proposed design:Proposed design:

– Software would generate samples for any Software would generate samples for any level (WIB, office) upon request from level (WIB, office) upon request from authenticated user (through web)authenticated user (through web)

– User can complete worksheets and generate User can complete worksheets and generate reports on-linereports on-line

– One sample per WIB or office per imported One sample per WIB or office per imported filefile

– Will be able to report multiple offices per Will be able to report multiple offices per participantparticipant

Benefits of Internal Audit Tool

States and federal government are States and federal government are dependent upon data quality at the local dependent upon data quality at the local levellevel

Increase the efficiency and precision of Increase the efficiency and precision of existing state monitoring effortsexisting state monitoring efforts

Potential cost savings for the system as Potential cost savings for the system as a wholea whole