Changing Perspectives on Workforce Changing Perspectives on Workforce System PerformanceSystem Performance

Data ValidationData Validation

Workforce InnovationsWorkforce InnovationsSan AntonioSan Antonio

July, 2004July, 2004

Purpose of Today’s Session

Update states and grantees on what is Update states and grantees on what is new in data validationnew in data validation

Provide information on the findings to Provide information on the findings to date from the first round of validationdate from the first round of validation

Allow states and grantees to provide Allow states and grantees to provide feedback to ETA on how to improve the feedback to ETA on how to improve the DV processDV process

Role of Data Validation in ETA Performance Assessment

Data validation is a key component in overall Data validation is a key component in overall performance strategyperformance strategy

Program funding is being directly tied to reliable Program funding is being directly tied to reliable performance outcomes (performance budget performance outcomes (performance budget integration)integration)

Data validation required by OIG and now being Data validation required by OIG and now being reviewed by GAOreviewed by GAO

Data validation is integrated into reporting Data validation is integrated into reporting Validation tools are evolving to meet state needsValidation tools are evolving to meet state needs

Programs Included in the Data Validation Effort

Unemployment Insurance Benefits and Tax (UI)Unemployment Insurance Benefits and Tax (UI) Workforce Investment Act (WIA)Workforce Investment Act (WIA) Trade Adjustment Assistance (TAA and NAFTA-TAA)Trade Adjustment Assistance (TAA and NAFTA-TAA) Labor ExchangeLabor Exchange National Farmworker Jobs Program (NFJP)National Farmworker Jobs Program (NFJP) Indian and Native American Programs (INA)Indian and Native American Programs (INA) Senior Community Service Employment (SCSEP)Senior Community Service Employment (SCSEP) Office of Apprenticeship, Training, Employment, and Office of Apprenticeship, Training, Employment, and

Labor Services (OATELS)Labor Services (OATELS)

How Does Validation Work? Two separate processes are required to Two separate processes are required to

ensure that performance data is reliableensure that performance data is reliable

– Report ValidationReport Validation

– Data Element ValidationData Element Validation

ETA provides software to states and ETA provides software to states and grantees that analyzes participant grantees that analyzes participant records records

Report Validation

Ensures that performance calculations are Ensures that performance calculations are accurateaccurate

DV software creates an audit trail for the DV software creates an audit trail for the numerator and denominator for each numerator and denominator for each performance measure performance measure

• Classifying participant records into Classifying participant records into performance outcome groups enables non-performance outcome groups enables non-technical staff to validate and analyze technical staff to validate and analyze program outcomesprogram outcomes

Data Element Validation

Report will not be accurate if the data being Report will not be accurate if the data being used by the software are wrongused by the software are wrong

Requires checking data elements against Requires checking data elements against source documentation to verify compliance source documentation to verify compliance with federal definitionswith federal definitions

Handbooks contain instructions and Handbooks contain instructions and examples of acceptable source documents for examples of acceptable source documents for each data element validatedeach data element validated– States identify state-specific source States identify state-specific source

documentation to reflect the variability of state MIS documentation to reflect the variability of state MIS systems and state/local documentation standardssystems and state/local documentation standards

Reporting of Validation Results

Data validation software produces Data validation software produces

– Report validation summaryReport validation summary

– Data element validation summary and analytical Data element validation summary and analytical reportsreports

WIA and LX software creates files with the annual WIA and LX software creates files with the annual report validation values for upload to ETAreport validation values for upload to ETA

Validation Efforts to Date

Many states have shared their validation Many states have shared their validation resultsresults

First round of validation was a valuable First round of validation was a valuable learning experience for alllearning experience for all

ETA has not set standards for acceptable ETA has not set standards for acceptable data qualitydata quality

Standards will be set for PY 2004 data Standards will be set for PY 2004 data validationvalidation

What is New for PY 2003 Validation

New schedule for reporting and New schedule for reporting and validationvalidation

New softwareNew software

New policies for collection and retention New policies for collection and retention of source documentationof source documentation

Schedule for Reporting of Validation Results

WIA RV will be due October 1, 2004 when the WIA RV will be due October 1, 2004 when the annual report is dueannual report is due

WIA and TAA data element validation will be WIA and TAA data element validation will be due February 1, 2005due February 1, 2005

LX report validation will be due November 15, LX report validation will be due November 15, 2004 with the report.2004 with the report.

Data Validation for National Programs

NFJP validation to begin in FebruaryNFJP validation to begin in February– Reports due in JuneReports due in June– Pilot of process was conducted in spring Pilot of process was conducted in spring – Software will be tested further in fallSoftware will be tested further in fall– Training session scheduled for NovemberTraining session scheduled for November

Data validation to be added to SCSEP in late Data validation to be added to SCSEP in late 20052005

Indian and Native American validation will be Indian and Native American validation will be incorporated into existing reporting softwareincorporated into existing reporting software

Revised Software

WIA Version 3.0 to be released in mid-WIA Version 3.0 to be released in mid-AugustAugust

New versions TAA (version 1.3) and LX New versions TAA (version 1.3) and LX (version 1.8) validation software(version 1.8) validation software

All will include automated upload of DV All will include automated upload of DV reports to ETAreports to ETA

WIA Software Changes

Calculate performance for the new reporting Calculate performance for the new reporting periodsperiods

Calculate Table OCalculate Table O More complete edit checksMore complete edit checks Ability to filter source table and performance Ability to filter source table and performance

outcome groups to provide greater analytical outcome groups to provide greater analytical flexibilityflexibility

Accept records for participants served only by Accept records for participants served only by NEGsNEGs

Software Upgrades for WIA Data Element Validation

Revised Data Element Validation Worksheets to Revised Data Element Validation Worksheets to reflect reduction in elements reflect reduction in elements

Improved ability to identify records that have Improved ability to identify records that have not been validatednot been validated

Ability to identify sampled records that have are Ability to identify sampled records that have are missing, invalid, wrong SSN, or whose location missing, invalid, wrong SSN, or whose location is unknown.is unknown.

Ability to trace exported samplesAbility to trace exported samples

States Experiences with Data Validation

States had to determine staff to be States had to determine staff to be responsible for data validation responsible for data validation

Communication of expectations and Communication of expectations and requirements to local areasrequirements to local areas

Mode of data element validation – onsite, Mode of data element validation – onsite, centralized or bothcentralized or both

State and Local Roles and Responsibilities

States had varying experiences in States had varying experiences in identifying validation assignmentsidentifying validation assignments– Some states had no problemSome states had no problem– Some states took time to sort through Some states took time to sort through

roles of different unitsroles of different units– Some states still have not clarified Some states still have not clarified

assignments (particularly for TAA)assignments (particularly for TAA)

Organization of case files at local areas Organization of case files at local areas was often not standardized or adequatewas often not standardized or adequate

Improving the Clarity of Source Documentation Requirements

ETA has not had clear and specific ETA has not had clear and specific policies for collection and retention of policies for collection and retention of source documentationsource documentation

States need to provide clear guidance to States need to provide clear guidance to local areaslocal areas

ETA will clarify requirements in change 1 ETA will clarify requirements in change 1 to TEGL 3-03 to TEGL 3-03 – Currently in clearanceCurrently in clearance– To be issued in AugustTo be issued in August

Streamlined Data Element Validation Requirements

As a result of state feedback, ETA As a result of state feedback, ETA reviewed and reduced the number of reviewed and reduced the number of elements to be validatedelements to be validated

All elements directly related to All elements directly related to performance or eligibility performance or eligibility

Detail for reduction in elements

ProgramProgram # of Elements for # of Elements for PY02PY02

# of Elements for # of Elements for PY03PY03

AdultAdult 4141 2626

Dislocated WorkerDislocated Worker 4444 3030

Older YouthOlder Youth 4848 3535

Younger YouthYounger Youth 100100 2929

TradeTrade 3535 2323

NFJPNFJP 1717 1414

TotalTotal 285285 157157

Various Methods for Data Element Validation

Onsite validation is essential to preserve the Onsite validation is essential to preserve the integrity of the processintegrity of the process

Ideal for state staff to perform validation onsiteIdeal for state staff to perform validation onsite– Promotes communication and mutual Promotes communication and mutual

understandingunderstanding In some cases, onsite validation is impracticalIn some cases, onsite validation is impractical

– Distances are too greatDistances are too great– Small number of recordsSmall number of records

States can therefore pursue a combination of States can therefore pursue a combination of onsite and remote validation if necessaryonsite and remote validation if necessary

Findings for WIA Report Validation

States had problems in two areasStates had problems in two areas

– Problems with extract file imported Problems with extract file imported into softwareinto software

– Problems with calculationsProblems with calculations

WIA Report Validation -- File Problems

Extract file imported into the software is Extract file imported into the software is incorrectincorrect– Data in extract file does not match Data in extract file does not match

data in state’s data systemdata in state’s data system– Inconsistent dataInconsistent data

File is different from the file used to File is different from the file used to calculate the report submitted to ETAcalculate the report submitted to ETA– Missing recordsMissing records– Changed/Updated DataChanged/Updated Data

WIA Report Validation – Calculation Problems

States excluded older Youth in advanced States excluded older Youth in advanced training/post-secondary school from training/post-secondary school from performance, even if the youth is performance, even if the youth is employed.employed.

Failure to distinguish pre-dislocation Failure to distinguish pre-dislocation earnings from pre-registration earnings earnings from pre-registration earnings for dislocated workersfor dislocated workers

Exclusion of records for earnings Exclusion of records for earnings calculations due to 99,999.99.calculations due to 99,999.99.

Data Element Validation Findings

Significant number of errors – error rates Significant number of errors – error rates exceeded 20% for some elements exceeded 20% for some elements

Many errors can be explained by lack of Many errors can be explained by lack of clarity in expectations for local source clarity in expectations for local source documentationdocumentation

Problems with changing wage records Problems with changing wage records and WRIS dataand WRIS data

WIA Data Element Validation Results for Adults

>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%

Individual with Individual with DisabilityDisability

Date of First Training Date of First Training ServiceService

Date of BirthDate of Birth Employed 1Employed 1stst qtr after qtr after ExitExit

Veteran StatusVeteran Status Date of First Date of First Intensive ServiceIntensive Service

Employment Status Employment Status at registrationat registration

Wages second Wages second quarter prior to quarter prior to registrationregistration

Other Reasons for Other Reasons for ExitExit

Source of Source of Supplemental Data Supplemental Data for 1for 1stst and 3 and 3rdrd Quarter Quarter

Low IncomeLow Income Wages first qtr Wages first qtr following exit quarterfollowing exit quarter

Wages third quarter Wages third quarter prior to registrationprior to registration

TANFTANF Wage third quarter Wage third quarter following exitfollowing exit

Type of recognized Type of recognized certificate/credentialcertificate/credential

Date of WIA ExitDate of WIA Exit

WIA Data Element Validation – Results for Dislocated Workers

>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%

Displaced Displaced HomemakersHomemakers

Individual with a Individual with a disabilitydisability

Date of BirthDate of Birth Employed 1Employed 1stst qtr after qtr after ExitExit

Veteran StatusVeteran Status Date of actual Date of actual qualifying dislocationqualifying dislocation

Wages 2Wages 2ndnd and 3 and 3rdrd quarter prior to quarter prior to registrationregistration

Date of First Training Date of First Training ServiceService

Date of WIA Date of WIA registrationregistration

Wages 2nd and 3rd Wages 2nd and 3rd quarter prior to quarter prior to dislocationdislocation

Source of Source of Supplemental Data Supplemental Data for 1for 1stst and 3 and 3rdrd Quarter Quarter

Date of WIA ExitDate of WIA Exit Wages 1Wages 1stst, 2, 2ndnd, and , and 3rd quarter following 3rd quarter following exit quarterexit quarter

Type of recognized Type of recognized certificate/credentialcertificate/credential

Employed in the 3Employed in the 3rdrd quarter after exitquarter after exit

WIA Data Element Validation – Results for Older Youth

>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%

In post-secondary In post-secondary education or education or advanced training 3advanced training 3rdrd quarter after exitquarter after exit

Veteran StatusVeteran Status Employment status at Employment status at registrationregistration

Date of birthDate of birth

Date of First Training Date of First Training ServiceService

Date of WIA ExitDate of WIA Exit Wages 2Wages 2ndnd and 3 and 3rdrd quarter prior to quarter prior to registrationregistration

Type of recognized Type of recognized certificate/credentialcertificate/credential

Employed in the 3Employed in the 3rdrd quarter after exitquarter after exit

Wages 1Wages 1stst, 2, 2ndnd, and , and 3rd quarter following 3rd quarter following exit quarterexit quarter

In post-secondary In post-secondary education/advanced education/advanced training 1training 1stst quarter quarter after exitafter exit

Source of Source of supplemental 3supplemental 3rdrd quarter after exitquarter after exit

Source of Source of supplemental 1supplemental 1stst quarter after exitquarter after exit

Other reasons for exitOther reasons for exit

WIA Data Element Validation Results for Younger Youth

>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%

Date of WIA ExitDate of WIA Exit Education Status at Education Status at RegistrationRegistration

Date of birthDate of birth

Other reasons for exitOther reasons for exit

Date goals attainedDate goals attained

Attained secondary Attained secondary sechool diplomasechool diploma

Date attained diplomaDate attained diploma

Youth Placement and Youth Placement and Retention InformationRetention Information

TAA Data Element Validation – Results by the Numbers

>20%>20% 10-20%10-20% 5-10%5-10% <5%<5%

Most recent Most recent qualifying separationqualifying separation

Date of registrationDate of registration Employed in the 1Employed in the 1stst and 3and 3rdrd quarters after quarters after exitexit

Date of ExitDate of Exit Earnings 1Earnings 1stst, 2, 2ndnd, and , and 33rdrd quarters after exit quarters after exit

Earnings 2Earnings 2ndnd and 3 and 3rdrd quarters prior to quarters prior to most recent most recent qualifying separationqualifying separation

Continuing Challenges

State wage record files are always changingState wage record files are always changing

– One solution is to “freeze” the file to avoid changes One solution is to “freeze” the file to avoid changes

– States should track changes in order to validate States should track changes in order to validate wageswages

Confidentiality of WRIS dataConfidentiality of WRIS data

– WRIS has rules restricting access to informationWRIS has rules restricting access to information

– Software allows states to suppress display of wage Software allows states to suppress display of wage valuesvalues

Future of Data Validation

Standards for acceptable error rates to Standards for acceptable error rates to be establishedbe established

ETA is moving toward a consolidated ETA is moving toward a consolidated reporting systemreporting system

Data Validation will be integrated into the Data Validation will be integrated into the new reporting systemnew reporting system

For More Information Contact InformationContact Information

Traci Di MartiniTraci Di Martini 202-693-3698202-693-3698 Dimartini.traci@dol.Dimartini.traci@dol.govgov

MPR Technical AssistanceMPR Technical Assistance William Borden – 609-275-2131William Borden – 609-275-2131 Jonathan Ladinsky – 609-275-2250Jonathan Ladinsky – 609-275-2250 WIATAWIATA@@mathematicamathematica--mprmpr.com.com TAATA@mathematica-mpr.comTAATA@mathematica-mpr.com ESTA@mathematica-mpr.comESTA@mathematica-mpr.com

http://www.doleta.gov/Performance/reporting/http://www.doleta.gov/Performance/reporting/tools_datavalidation.cfmtools_datavalidation.cfm

We Need Your Feedback

Tell us about your experiences with Data Tell us about your experiences with Data ValidationValidation

What did you learn that may help others What did you learn that may help others

What improvements can be madeWhat improvements can be made