Validating Self-Reported Education: Results of a Pilot Study

Jesse RothsteinCecilia RouseAshley Miller

How many high school graduates are there?

Current Population SurveyGives “status completion” rate:

# of 25-year-old graduates / 25-year-old population

Common Core of DataGives “school-based completion rate”:

# of graduates in year t / # of 9th graders in t-3.

CPS estimates much higher than CCD estimates 2000/2 cohorts: 84% CPS vs. 70% CCD

CPS vs. CCD-based graduation rates

60%

65%

70%

75%

80%

85%

90%

95%

1975 1980 1985 1990 1995 2000 2005

CPS (age 25)

CCD (9th grade demoninator)

Two easy issues HS diploma attainment rises with age in CPS

Delayed completion? Adult education / GEDs? Misreporting?

“9th grade bulge” CCD estimates usually use 9th grade enrollment as denominator Many students are held back in 9th grade 9th grade enrollment consistently about 7-10% higher than 8th

grade enrollment through 1990s These reduce the problem, but don’t eliminate it.

CPS vs. CCD-based graduation rates

60%

65%

70%

75%

80%

85%

90%

95%

1975 1980 1985 1990 1995 2000 2005

CPS (age 25)

CCD (9th grade demoninator)

CCD (8th grade denominator)

Not very sensitive to definitions CCD-based estimates all tend to yield similar

answers. Mishel and Roy (2006): Get same answers

with CPS, Census, NELS, NLSY.

Alternative graduation rates, existing data

60%

65%

70%

75%

80%

85%

90%

95%

1975 1980 1985 1990 1995 2000 2005

CPS (age 25) CCD (9th grade demoninator)CCD (8th grade denominator) Swanson (CCD)Greene/ Forster (CCD) Warren/ Halpern-Manners (CCD)Mishel/ Roy (NELS) Mishel/ Roy (NLSY97)Mishel/ Roy (Census) Mishel/ Roy (CPS)

Not very sensitive to definitions CCD-based estimates all tend to yield similar

answers. Mishel and Roy (2006): Get same answers

with CPS, Census, NELS, NLSY. Except!

Some evidence that GEDs can account for a lot of the discrepancy (Heckman and LaFontaine 2007)

Alternative graduation rates, existing data

60%

65%

70%

75%

80%

85%

90%

95%

1975 1980 1985 1990 1995 2000 2005

CPS (age 25) CCD (9th grade demoninator)CCD (8th grade denominator) Swanson (CCD)Greene/ Forster (CCD) Warren/ Halpern-Manners (CCD)Mishel/ Roy (NELS) Mishel/ Roy (NLSY97)Mishel/ Roy (Census) Mishel/ Roy (CPS)Heckman/ LaFontaine (Census)

Potential explanations

1. Attainment increase with age is poorly understood2. Proxy responses in CPS may overstate graduation3. Respondents may overstate own graduation4. “Some college” may not have graduated HS5. Regular diplomas vs. GEDs, certificates of attendance,

adult ed., etc.6. Public vs. private schools7. Immigrants/emigrants since HS8. Grade retention9. Divergent definitions in CCD10. CPS excludes armed forces, prisoners11. Poor coverage in CPS

Potential explanations that we will address

1. Attainment increase with age is poorly understood2. Proxy responses in CPS may overstate graduation3. Respondents may overstate own graduation4. “Some college” may not have graduated HS5. Regular diplomas vs. GEDs, certificates of attendance,

adult ed., etc.6. Public vs. private schools7. Immigrants/emigrants since HS8. Grade retention9. Divergent definitions in CCD10. CPS excludes armed forces, prisoners11. Poor coverage in CPS

Our Sample….

Sample frame = Households likely to have an 18-25 year old member (complied by a commercial vendor).

We purchased 1,000 (randomly selected) phone numbers.

Our target was 50 “completed” inteviews.

Survey implementation consisted of three steps. Step 1: Mimic the CPS Conduct a “CPS-style” interview with a HH member

who was at least 18 years old (the “household respondent”).

Note: We ended up with 2 versions (“A” and “B”) where version “A” asked for a full HH roster while version “B” asked initially if an 18-25 year old lived in the HH; only asked questions about one 18-25 year old.

Probe more on education questions, particularly regarding alternative types of high school degrees.

Ask for the name and location of the last high school attended by the young adult, as well as year of high school graduation.

Step 2: Verify Proxy Report and Respondent Understanding of CPS-style Education Question Ask similar “CPS-style” education questions

of the young adult him or herself (“initial report”).

Probe further on education questions (“final report”).

Also ask for the name and location of last high school attended as well as year of high school graduation.

Step 3: Verify Proxy and Self-reports with Administrative Data Send a letter to the high school requesting

verification of reported high school graduation information.

Follow-up with a telephone call for a subset of schools that did not respond within 10-12 days.

Table 1: Sample sizes (Part I) Total Version A Version B

(1) (2) (3)

N % N % N %

Number of phones called 1000 413 587

Non-working/non-residential 132 13% 93 23% 39 7%

Unable to make contact 223 22% 16 4% 207 35%

Made contact 645 65% 304 74% 341 58%

Made contact 645 304 341

Language problem 10 2% 5 2% 5 1%

Unresolved after max calls 73 11% 2 1% 71 21%

Refusal 418 65% 228 75% 190 56%

Completed HH interview 144 22% 69 23% 75 22%

Table 1: Sample sizes (Part II) Total Version A Version B

(1) (2) (3)

N % N % N %

Completed HH interview 144 69 75

No eligible youth 52 36% 25 36% 27 36%

1+ eligible youth in HH 92 64% 44 64% 48 64%

All eligible youth 117 39 78

Youth was HH respondent 10 9% 4 10% 6 8%

Failed to contact 41 35% 6 15% 35 45%

Refused 18 15% 9 23% 9 12%

Turned out ineligible 3 3% 2 5% 1 1%

Completed self-report 45 38% 18 46% 27 35%

Table 1: Sample sizes (Part III) N %

Completed self or proxy reports 117

Ineligible (outside age range/ambiguous age) 5 4%

Did not attend high school 3 3%

Has GED 4 3%

Still in high school 15 13%

Not enough identifying information 11 9%

Sent letter to high school 79 68%

All letters sent to high schools 79

No response 27 34%

Declined to provide information 6 8%

Provided information 46 58%

Conclusion

We find a high level of agreement between proxy respondents, the young adults themselves and high school administrative records on whether individual has a high school diploma (or equivalent).

We see less agreement between proxy respondents and young adults on college attendance.

Questions about modes of high school completion should be asked of those with a college education as well as those with only a high school education.