Quality Assurance Practices associated with Producing ...programs.ccsso.org/content/pdfs/Examining Graduation Indicators-10252007.pdfQuality Assurance Practices associated with Producing

Council of Chief State School Officers Accountability and State Reporting

Robin Taylor (Chair) Associate Secretary

Delaware Department of Education

J. P. Beaudoin Research in Action, Inc.

Pete Goldschmidt

California State University, Northridge/CRESST

Quality Assurance Practices associated with Producing Cohort Graduation Rates

August 2007

Council of Chief State School Officers The Council of Chief State School Officers (CCSSO) is a nonpartisan, nationwide, nonprofit organization of public officials who head departments of elementary and secondary education in the states, the District of Columbia, the Department of Defense Education Activity, and five U.S. extra-state jurisdictions. CCSSO provides leadership, advocacy, and technical assistance on major educational issues. The Council seeks member consensus on major educational issues and expresses their views to civic and professional organizations, federal agencies, Congress, and the public. State Education Indicators The Council is a strong advocate for improving the quality and comparability of assessments and data systems to produce accurate indicators of the progress of our elementary and secondary schools. The CCSSO education indicators project is providing leadership in developing a system of state-by-state indicators of the condition of K-12 education. Indicators activities include collecting and reporting statistical indicators by state, tracking state policy changes, assisting with accountability systems, and conducting analyses of trends in education. The CCSSO reports on state education policies inform education leaders and educators about the current status and trends in policies across the 50 states that define and shape elementary and secondary education in public schools. The report is part of a continuing biennial series produced by the Council’s education indicators project. We report 50-state information on policies regarding teacher and leader preparation and certification, graduation requirements, state content standards, student assessment programs, school time, and student attendance. The work of CCSSO is possible because of the excellent cooperation and coordination by staff in each state department of education as well as by funding from the U.S. Department of Education.

2007 Council of Chief State School Officers

Elizabeth Burmaster (Wisconsin), President Rick Melmer (South Dakota), President-Elect

Gene Wilhoit, Executive Director

Rolf K. Blank, Director of Education Indicators

Council of Chief State School Officers Attn: Publications

One Massachusetts Ave., NW, Suite 700 Washington, DC 20001

202-336-7016 Fax: 202-408-8072

www.ccsso.org

ISBN: 1-933757-08-6

Copyright © 2007 by the Council of Chief State School Officers. All rights reserved.

Quality Assurance Practices associated with Producing Cohort Graduation Rates

Strategies for Increasing Reliability and Validity of School Accountability Data and Decisions Workgroup

Robin Taylor (DE)-Chair Wes Bruce (IN)

Steve Hebbler (MS) Keith Kameoka (HI)

Pat McCabe (CA) Matthew Pakos (MA) Thomas Spencer (LA) Rachelle Tome (ME) Cathy Wagner (MN)

John Weiss (PA) John Wickizer (KY)

Comments and Suggestions: Deborah Newby, CCSSO; Nancy Smith, Data Quality Campaign; ASR SCASS members and guests

Table of Contents

Executive Summary .........................................................................................................................1

Section 1: The graduation rate issue for states.................................................................................2

1.1: Current graduation rate calculations .............................................................................4

1.2: Why a cohort method?..................................................................................................8

Section 2: What is needed to produce cohort rates? ......................................................................12

2.1: Data collection ............................................................................................................12

2.2: Report production .......................................................................................................19

Section 3: Quality assurance practices in producing cohort graduation rates ...............................23

3.1: Applying QADM to graduation rates .........................................................................25

3.2: Capacity stages ...........................................................................................................28

3.3: Actions prior to public release ....................................................................................31

3.3.1: Internal auditing...........................................................................................31

3.3.2: Data collector verifications..........................................................................32

3.3.3: Rate assignments..........................................................................................32

3.3.4: Trend analysis ..............................................................................................33

3.3.5: Screening Report Templates........................................................................34

3.4: Actions after public release.........................................................................................34

3.4.1: After action reviews.....................................................................................34

3.4.2: Risk assessments..........................................................................................34

3.4.3: External audits .............................................................................................35

3.4.4: On-site monitoring.......................................................................................35

3.4.5: Technical manuals .......................................................................................36

3.4.6: Data collector trainings................................................................................36

Section 4: Conclusion ....................................................................................................................38

References......................................................................................................................................40

Appendix A: AYP graduation rate designs across ASR member states........................................43

Appendix B: Graduation rate design comparisons ........................................................................49

Appendix C: Selected auditing practices of selected states ...........................................................52

Appendix D: Mississippi Department of Education Report ..........................................................53

Appendix E: Graduation rate proxies: Utah and Minnesota..........................................................71

Appendix F: External auditing: Utah.............................................................................................77

List of Tables

Table 1: Proxy limitations summary................................................................................................7

Table 2: Data elements and conditional codes for production.......................................................15

Table 3: Error control issues (school-level)...................................................................................30

List of Figures

Figure 1: Graduation process controls ...........................................................................................17

Figure 2: Indiana Department of Education graduation rates........................................................21

Figure 3: Indiana Department of Education graduation business rules .........................................21

Figure 4: Delaware’s graduation rate diagnostic matrix for SY 2005-06......................................26

Quality Assurance Practices associated with Producing Cohort Graduation Rates, CCSSO ASR SCASS 1

Executive Summary

The Council of Chief State School Officer’s State Collaborative on Accountability

Systems and Reporting (ASR SCASS) has spent two years examining issues of data validity in

state accountability systems. This paper is the second in a series exploring methods of

implementing quality assurance practices in these systems, this one focusing on the calculation

of cohort graduation rates.

As of 2007, states are using a combination of many different methods to determine

graduation rates from local high schools, but most are preparing to move to a cohort calculation,

one that tracks students from entrance to exiting high school. Sensitive to the data requirements

and potential data pitfalls of calculating a cohort graduation rate, Quality Assurance Practices

Associated with Producing Cohort Graduation Rates explores how graduation rates are

calculated in many states, the need for calculation of a cohort graduation rate, and details several

quality assurance practices states can use to produce valid and reliable cohort graduation rates.

Focused on linking the data quality issues to accountability results from a practitioner’s

viewpoint, the paper lays out several quality assurance steps states could implement at various

points. First, states can utilize the expanded Quality Assurance Data Matrix (QADM), developed

for Validity Threats: Detection and Control Practices for State and Local Education Officials, as

a self-evaluation and tool for mitigating error. Other steps to implement both during the

calculation process and after reporting include internal auditing, data collector verifications, rate

assignments, trend analysis, risk assessments, external audits, on-site monitoring, and data

collector trainings. Examples from states are provided to demonstrate the production sequence

and potential reporting formats. Finally, states are reminded to constantly evaluate the efficacy

of their quality assurance practices to ensure they are producing the intended goals.


Section 1

The Graduation Rate Issue for States

The 2001 reauthorization of the Elementary and Secondary Education Act of 1965

(ESEA), known as the No Child Left Behind Act of 2001 (NCLB), heightened data demands of

every state. The law required every state to report disaggregated student assessment results at the

school and district level as well as each high school and district’s graduation rate.

Given the status of state student data systems in 2001, few states or districts were

prepared to provide anything but a proxy rate for their students, estimates of the number of

Graduation rate, as defined in NCLB, 200.19 Other Academic Indicators

(a) Each state must use the following other academic indicators to determine AYP:

(1) High Schools.

(i) The graduation rate for public high schools, which means–

(A) The percentage of students, measure from beginning high school, who graduate

from high school with a regular diploma (not including an alternative degree that is not fully

aligned with the State’s academic standards, such as a certificate or a GED) in the standard

number of years; or

(B) Another definition, developed by the State and approved by the Secretary in the

State plan, that more accurately measures the rate of students who graduate from high school

with a regular diploma as defined above

(ii) In defining the graduation rate, the State must avoid counting a dropout as a

transfer.


students graduating from high school compared to the number beginning high school four years

earlier. There was little capacity to track individual students through their high school careers to

determine when, and if, they graduated, producing a cohort graduation rate.

Since 2001 several national organizations as well as the U. S. Department of Education

(ED) have developed alternative rates to the various state rates used for calculating AYP, such as

the Averaged Freshman Graduation Rate (AFGR), the Exclusion-Adjusted Cohort Graduation

Indicator (EACGI), the NGA graduation rate, and others. As the nation’s education reform focus

shifted to include high school success, a common, accurate calculation of student success in high

school (as defined by graduation in four years) was desired, both to measure the efficacy of high

schools as well as to help target resources to schools in need.

As of 2007, states are using a combination of many different methods to determine

graduation rates from local high schools, but most are preparing to move to a cohort calculation.

Sensitive to the data requirements and potential data pitfalls of calculating a cohort graduation

rate, one that follows students across their high school careers, the Council of Chief State School

Officer’s Accountability System and Reporting State Collaborative (ASR SCASS) began

examining the issues involved with producing valid and reliable graduation rate results in 2006.

The resulting document explores how graduation rates are calculated in many states, the need for

calculation of a cohort graduation rate, and focuses on several quality assurance practices states

can use to produce valid and reliable cohort graduation rates.


1.1. Current graduation rate calculations

One of the most basic questions in evaluating the effectiveness of high schools is whether

students who enter high school as freshman progress through the curriculum and graduate

approximately four years later with a high school diploma. Two fundamental approaches can be

used to answer the question:

1. Use cross-sectional data and estimate, or

2. Establish longitudinal data system and produce differing rates.

Until recently, most states did not have the necessary student-level information system to

accurately monitor student progression across multiple years. State officials used proxy methods

to estimate graduation, completion, and dropout rates based on the most current data available.

Many of these estimates were fraught with data errors produced by user inputs, system default,

and limited quality assurance practices.

It is largely due to technology limitations that many states rely on proxy indicators to

determine graduation rate. (See Appendix A, Table A1 for a description of graduation rate

designs for selected states.) As described by the NISS (2004) study, tracking the progress of

students across time requires, at minimum, a student-level information management system.

These systems must be sophisticated enough to track student movement individually and in

cohorts throughout the state during the current school year and across multiple years.

A student cohort, by definition, is a group of individuals who start at a given point in time

and are then examined again at future intervals. The typical progression is from one grade-level

to the next, thus producing “natural” intervals to measure changes in cohort characteristics.

Students are assigned to progressively higher grades according to those criteria found within

state policy (e.g., high stakes testing) and local pupil progression plans. Students are expected to


attain certain educational benchmarks for progression into higher grades. For many students,

these criteria are not met during their K-12 education resulting in students being retained in one

or more grade levels, thus exiting their original cohort.

States and districts have made good faith efforts to accurately report graduation data to

the general public. Yet graduation proxy estimates requires several assumptions that become null

and void for particular groups of students. First, students typically earn enough credits to be

reclassified into the next higher grade on an annual basis. Some students will remain at a

particular grade until they have accumulated enough credits to progress. Other students will earn

credits at faster rates, thus completing high school in less than the standard four years.

A second assumption is that students attend high school at a school offering the complete

set of courses or grade ranges. Many districts are reorganizing the high school experience into

several different schools (e.g., 9th grade or career academies). This phenomenon complicates

how one school can be held solely accountable (for NCLB purposes) for the student graduating

on-time.

A third assumption that students’ status is known and accurately reported by local

officials. Students may

(a) transfer from one school and enter another without notifying school officials,

(b) reach an legal age to drop out of the system,

(c) move to another state or country,

(d) be placed in a juvenile center, incarcerated, or become wards of the state, or

(e) exit the system only to reenroll at a latter date.

These conditions along with numerous others require detailed reporting by local officials

and quality assurance procedures.


However, proxy indicators have numerous design limitations when used to address on-

time graduation rates. In some cases, proxies can estimate graduation rates, but rarely at the

school-level. The Mississippi Cohort method, detailed in Appendix D, avoids numerous

limitations by aggregating student-level data to higher units of analysis. This procedure also

produces other rate indicators, such as four and five year completers, along with on-time

graduates. The need for multiple years of data remains an implementation barrier for newly

established (e.g., Oklahoma’s WAVE) or emerging (e.g., Nebraska’s NSSRS) student

information system. These systems will require several years of implementation data before they

will be able to produce graduation rates based on a cohort approach.

Most states have used cross-sectional data to estimate on-time graduation rates. These

cohort-type “proxies” are currently being used to meet graduation rate reporting requirements

prescribed by federal regulations (i.e. 34 CFR Part 200). Unlike Louisiana, Tennessee,

Delaware, Texas and a few others, many states are in their first decade of operating a student

information system able to: (a) assign unique student identifiers, (b) track student movement

over time by using data mining tools (typically within a data warehouse infrastructure), (c)

implement a comprehensive series of exit codes, (d) screen and audit data elements, and (e) build

LEA capacity to ensure data integrity. Without these system components, state-level reporting is

based on aggregate data submitted by LEAs. Dropout estimates were readily used as proxies for

graduation rates (for example, see Appendix D, Utah’s dropout definitions); however, these data

contained numerous anomalies that produced, at best, graduation estimates. Further, few states

apply a cohort method when calculating graduation rates for AYP purposes.

A critical review of these graduation proxies demonstrates clearly their inability to meet

the rigorous design necessary to produce school-level, cohort graduation rates. At best, the


proxies allow for estimates at the district or state level but even at these units of analysis, critical

production business rules are not defined, reducing the reliability and validity of these

calculations over time. Table 1 summarizes the limitations of various proxy methods.

Table 1. Proxy Limitations Summary

Limitation* AFGR MN HSGR SCPI GGI MSC

Ignores dropouts Yes

No

No

Yes

No

Includes retained 9th-graders Yes

Yes

No

Yes

No

Includes late and early graduates Yes

No

Yes

Yes

No

Includes alternative completers No

Yes

No

No

No

Ignores out migration Yes

Yes

Yes

Yes

No

Ignores in migration Yes

Yes

Yes

Yes

No

Ignores students still enrolled in school No

Yes

No

No

No

Does not represent a 4-year time span No

No

Yes

No

No

* Adapted from Seastrom et al. (2006) ** Average Freshman Graduation Rate [AFGR]; Minnesota High School Graduation Rate [MN-HSGR]; Swanson Cumulative Promotion Indicator [SCPI]; Greene Graduation Indicator [GGI]; and Mississippi Cohort [MSC]

Perhaps the most significant business rule missing in proxy methods for determining

graduation rates is associated with student mobility. Typically described as “migration,”

students moving into or out of the cohort must be addressed. Simply excluding these data will

result in an ever growing number of students, whose graduation status is never accounted for in

public reporting or AYP determinations. This phenomenon has some similarities with the full

academic year (FAY) provision that allows student performances at the school-level to be


removed if they have not attended the entire school year. Yet, these business rule limitations and

differing report methods do not equate to students not being accounted for within the education

system.

1.2. Why a Cohort Method?

A cohort approach to calculating graduation rates can provide clear and concise

information on how many high school freshmen complete their educational experience and

graduate within a specific period of time. Unlike other approaches that estimate the number of

students who complete their high school experience, the cohort approach makes these

determinations for each student. This is accomplished by examining multiple years of data and

tracking the student’s enrollment and exit information.

Since data is tracked at the individual student level, the cohort method allows for both

data aggregation, analysis at the district, state, or national level, and data disaggregation, data on

the graduation rate of student subpopulations at the school level. Detailed characteristics, such as

student mobility, retention, drop-out, and other factors can be examined at each level. Contrast

this to proxy methods, which generally produce data at the district or state level and cannot move

to lower units of analysis. This limitation was obvious when federal requirements outlined in

NCLB required using graduation rates for AYP determinations.

In 2005 the National Center for Education Statistics (NCES) at the United States

Department of Education announced a new calculation of graduation rates for each state. Called

the Averaged Freshman Graduation Rate (AFGR), the calculation uses existing data submitted

by states to NCES through the Common Core of Data. Essentially, the percentage of graduates is

figured as follows:


The calculation also allows for ungraded students to be allocated across grades. The

authors point out that though this rate is “not as accurate as an on-time graduation rate computed

from a cohort of students using student record data, this estimate of an on-time graduation rate

can be computed with currently available data” (Seastrom, M., et al., 2007). However, based on

the calculations available for cross-state graduation rate comparisons, this was considered the

best alternative “based on a technical review and analysis of [other possible] estimates”

(Seastrom, 2007). The AFGR for all states for 2002-03 and 2003-04 was published in 2007,

though it is possible to figure the rate for several previous years based on the NCES published

formula.

The National Institutes of Statistical Sciences/Education Statistics Services Institute’s

Task Force on Graduation, Completion, and Dropout Indicators published its final report

(NISS/ESSI, 2004) on technical issues associated with calculating graduation (also completion

and dropout) indicators. A task force comprised of researchers, state and federal officials, and

others produced a series of recommendations to accurate calculate an on-time graduation rate.

The group reported the following:

1. No single indicator of graduation, completion, or dropouts can serve all purpose;

2. No indicator allowing for “exclusions” can be free of perverse incentives;

3. Uniformity across states in reporting (along with methods for calculating

indicators) graduation, completion, and dropout indicators;

4. Data quality and availability; and,

5. Verification of interstate transfers.

[(students in 8th grade in year y-1) + (9th grade in year y) + (10th grade in year y+1)]/3 graduates with regular diplomas in year y+4


These findings were reported in conjunction with a recommendation that the graduation

indicator must have three key properties: (a) data are derived from student-level collections and

are functions of entry and year of graduation, (b) entry and graduation year is in cumulative

form, and (c) exclusions are defined by a narrow set of business rules. These principles

supported the creation of the Exclusion-Adjusted Cohort Graduation Indicator (EACGI), which

has now been adopted by the NGA. This cohort approach uses student-level data and a series of

data elements and conditions that records student status across multiple years.

In 2005, the nation’s governors, through the National Governor’s Association, adopted

the EACGI rate, represented as follows:

Governors from all 50 states agreed to work toward calculating the rate, adjusting data

systems to collect the necessary indicators if necessary. This calculation is referred to as the

“NGA Graduation Rate.” This formula applies only to student who receive standard diplomas.

In 2006, NGA and CCSSO convened a technical panel to create technical guidelines to

allow this calculation to be utilized in all states, realizing the governors’ aim of a comparable

graduation rate utilizing high-quality, student-level longitudinal data. A selection of the panel’s

recommendations follow:

1. Calculate the 4-year rate, as specified in the NGA rate formula, and also a 5-year

graduation rate.

2. Define, document, and implement a detailed student-level exit data collection process

to account for students who leave the public school during or between school years.

On-time graduates by year X [(first time 9th graders in year X-4) + (transfers in) – (transfers out)]


3. Establish a process by which the state educational agency (SEA) reviews statistical

trends of exit data within and across school years to identify potentially erroneous

data.

4. A student whose IEP allows an extra year to graduation, or who is receiving services

for limited English proficiency who is allowed extra time to graduate, should be

placed in the cohort with which that student is expected to graduate.

5. States should clearly describe by component how they are calculating the rate.

(Smith, 2006).

Many states redoubled their efforts to develop systems to allow them to figure the cohort

graduation rate following the NGA agreement. At the same time, many continued to find

themselves reeling form the sheer quantity of data generated and required by NCLB assessment

and data calculations. A breakdown in the reliability of the data at any point from input at the

school level to data transfer at the district or state level, to calculations, could produce spurious,

or at least unreliable, results for any of these cohort calculations.

Though each state education agency is working to follow each individual student

throughout his or her high school career, in order to report when and if successful high school

completion takes place reporting cohort graduation rate calculations, agency staff need to ensure

1. the proper tools are in place to collect information necessary to complete the

calculation, and

2. make sure the quality assurance practices are in place so that the reported figures are

valid and reliable.

The second part of this paper focuses on these two topics.


Section 2

What is needed to Produce Cohort Rates?

There are three fundamental activities necessary to produce cohort graduation rates in a

manner consistent with the NISS or NGA recommendations. These activities are to: (a) collect

data across multiple years, (b) create business rules and design logic for production, and (c)

implement quality assurance practices. Within each of these macro-level activities are numerous

tasks requirements for both local and state-level officials.

Data inputs describing student characteristics, enrollment and exit status, and other

information must be acquired from the data collectors. Local and state information managers

must implement data integrity and verification processes, while policy makers must establish the

overarching context, business rules, and design logic used to report targeted results. The

production process requires migration of data into the programmer’s code, execution of the code,

and validation checks before finalization. Once finalized, the data are placed in the report

production cycle and disseminated to data collectors as required by state and federal policies and

procedures.

During the entire work cycle, quality assurance practices used to verify and validate the

data are continuously being implemented. These data are used to correct errors, reduce the

probability of their occurrences, and verify alignment between the underlying data and that

reported. These numerous activities must adhere to a rigid, standardized set of operating

procedures or run the risk of producing inaccurate information.

2.1. Data collection

Perhaps the most critical infrastructure necessary to produce cohort graduation rate is a

centralized student information system (SIS) that uses a unique identification system to identify


students. Without a data system that can accurately track a student’s entry, progression, and exit,

no calculation of cohort rates will be accurate.

Control efforts within the SIS must have a comprehensive set of internal verification

processes to detect and correct irregularities such as duplicate records, missing data fields,

illogical data, and invalid enrollment and/or exit codes. These error mitigation efforts are

typically done prior to score production because they are used to establish the master file set.

Validation efforts in some states, such as Nebraska, Delaware, and Louisiana, use quantitative

approaches to examine the reasonableness of the data. Data cells identified as beyond

established thresholds are “flagged” for qualitative review. These validation activities in other

state settings can range from informal reviews done by internal staff to comprehensive auditing

by third-parties.

Business rules: Front loading investments in human capital

Verification and validation actions must be carefully planed to ensure limited fiscal

and human resources are effectively allocated while adhering to established production and

reporting schedules. State and local officials must not only develop and maintain the

necessary hardware and software infrastructure, but also must direct investments toward

human capital development. “Front loading” their investments in the personnel (data

collectors) who will enter, review, and validate data indicates a proactive strategy to improve

data quality. In other words, improvement efforts should seek to maximize resource use by

focusing on high risk data elements and production processes requiring human interactions.

For better or worse, data entry and upkeep is ultimately up to individuals, data collectors at

the school or district level.


The “human factor” (HF) can contribute to error production and its removal during each

sequence of the production phase. The first HF must provide information detailing student

characteristics such as the school they are attending, their assigned grade, and the types of

programs they participate in. The HF is complicated when the data collectors use complex

business rules and IT taxonomies (NFES, 2006) in which they may have never received training

or inadequate technical support. Data are provided by data collectors who must assume

responsibility for their accuracy. This responsibility may be shared across several

program/school staff (e.g., attendance officer, counselor, front office secretary, local IT manager)

and readily results in data being duplicated or modified without knowledge of all responsible

parties. Many of these data remain static over time or change in highly predictable patterns. For

example, gender and date of birth are clearly fixed characteristics that do not change over time,

yet school assignment can readily change within any given school year.

Students contribute to the HF by changing schools, attending unique programs, becoming

incarcerated or progressing from one grade to the next in atypical ways. Data collectors update

the records of these students using the available codification within the information system. This

information can require updates within a given school year or across years, especially with

highly mobile students.

Data input errors entered into the early stages of graduation rate score production will

obviously produce spurious results. Data verification and automated edit checks of data

elements like those provided within Table 2 will correct much of the faulty information before it

enters into the production phase.


Table 2: Data Elements and Conditional Codes for Production

Data Element Description Needed for Cohort

District ID Name and NCES code Yes School ID Name and NCES code Yes School Year Year associated with data Yes Student Name First, middle, and last name No Student Age Chronological age No Student ID Unique identifier Yes Grade Assigned grade including “ungraded” programs members Yes Gender Student’s sex No-but Ethnicity Student race No-but ED Economically disadvantaged flag No-but SWD Students with disabilities flag No-but LEP English language learner flag No-but Migrant Migrant flag No Title I Title I flag No Resident School School assigned for accountability No Resident District District assigned for accountability No Enrollment Date Entered school Yes Enrollment Type Public, Non-public, Private, Attending in another state, Resident

attending non-public school Yes

Exit Date Left school Yes Exit Reason Reason codes clusters (NFES, 2006)

1. Still enrolled 2. Transferred 3. Dropped out 4. Completed 5. Not Enrolled, eligible to return 6. Exited-neither completed nor dropped out

Yes

Completion Type Types of diplomas Graduate with diploma, Industry based certificate, GED only, Certificate of Achievement

Yes

Cohort Assigned Cohort identifier Yes

The second HF are the activities associated with the constructing of student cohorts.

Student cohorts are developed through a series of data manipulations and linking across differing

years. Although this is accomplished through computer software programs, it requires the

specialized skills of humans to write, manipulate, and execute the code. Validation checks used

to produce error reports and/or to reject data outside of prescribed tolerances must also be

conducted by staff members. Data quality checks by information managers may result in

submitted data being changed in the master files. For example, a student may have an exit date


Once created, the programming code is manipulated to reflect changes in policy,

easily detailed for clarity in flowcharts.

and condition showing him/her as a dropout; however, a valid enrollment date is identified

keeping the student within the cohort used for the district but not at the school-level. These types

of record corrections are also needed in assigning and reassigning students to particular cohort

groups. Enrollment and assigned grades are needed to determine if the student is repeating the

grade, although these data may be collected using a customized data element. For example,

students who graduate prior to the standard year must be maintained in their original cohort, so

students who do not progress to higher grades can be identified.

The third HF occurs within the report production cycle. Report production processes

require detailed business rules articulating how data are to be manipulated, errors resolved,

variables created, and other operational tasks executed. Flowcharting the decision logic prior to

software programming assists in graphically organizing the decision being executed within

underlying software code. This simple technique is a helpful tool in linking the programming

and policy logic as the lexicon for each system is very different. Modularizing the scoring code

can quicken internal auditing and other validating activities, while reducing the time necessary to

change and test additional coding. The results must undergo a series of quality assurance checks

to verify the results meet the specifications outlined in the business rules and design logic. These

validation checks can range from simply examining the frequency distributions across differing

units of analysis to conducting score replication by external entities.

In Louisiana, and other states, preliminary graduation (and dropout) data are provided to

LEAs prior to report production. This process allows local officials to evaluate outcomes from

their score production processes and correct data errors. Once reviewed, the final data is


migrated into the reporting production sequences used to manufacture information for the

selected audiences. Customized reports may be required for improvement efforts or to conduct

auditing functions. Overall, the HF exists throughout the graduation process reflected in Figure

1. Efforts to ensure reported data are accurate and credible requires addressing the HF’s

influence within the system.

Figure 1. Graduation Process Controls

Graduation Process Controls

Input Cluster A

Student Characteristics[Demographics &

Program Membership ]

Input Cluster B

Student Status[Enrollment &

Completer]

Input Cluster C

Student Cohort[Assignment &Reassignment]

Public Reports

AYP

StateAccountability

NGA

Strategic Plan

End

Use

r

IT

After PublicRelease

On-SiteMonitorings

End-User Trainings

External Audits

Technical Manuals

Risk Assessments

After ActionReviews

IT Validation

*Field Specs*Error Report

*ID Assignment

Screening

* Duplicates*Enrollment Status*Completer Type

*Transfers*Missing

*Cohort Year

Post Production QA

Rate Assignments

Trend Analysis

Screening ReportTemplates

End-User (LEA)Verifications

Internal Audits

Rate Production

Business RuleApplication

Policy Alignment

Decision andOperational Logic

End

-Use

r

The production of graduation rates requires a series of complex data manipulations and

computations. To illustrate this point, the procedures used by the Mississippi Department of

Education to calculate cohort graduation, completer, and dropout rates (see Appendix D for the

complete February 12, 2007 report) are presented in the excerpt below.


Background This information provides details on the procedure developed for tracking a full cohort of students in the

Mississippi Student Information System (MSIS) and calculating estimated dropout, completion, and graduation rates for the full cohort. The objectives were to ensure that all students would be included in an appropriate cohort (i.e., no students would be systematically left out) and to use data available in MSIS to calculate accurate counts and rates for dropouts, completers, and graduates.

Since MSIS was implemented statewide at the beginning of the 2001/2002 school year, the first four-year full cohort could be tracked using data from the end of school year 2004/2005 updated with the results of the 2005 summer activity procedure. All the required data were available in April 2006. Using the most appropriate analysis techniques, there are still certain errors that will be present in the calculated counts and rates. Those include coding errors in the data transmitted to MSIS from the district student administrative packages (SAPs), a small amount of data inconsistency (generally in the first school year) due to edit checks that were implemented later, and students whose final disposition is unknown because they left the system during summer 2002 or summer 2003 (before the summer activity procedures were implemented).

*Technical Note: The cohort includes retained students. The Mississippi Department of Education is

currently working on the next cohort (beginning with 9th graders in 2002/2003) and will be able to include in the full cohort only the first-time 9th graders. Rates reported in this initial work will differ from future values excluding repeaters.

Decision Logic The decision and programming logic for implementing a full cohort tracking system was approved and

data files were built using the data in MSIS. The steps used for tracking a cohort and analyzing the data are as follows:

1. All students who entered MSIS as ninth graders during school year 2001/2002 were identified and their data written to a data file. For each student, a variety of data variables were extracted from MSIS, including the last know disposition for students who were not still enrolled somewhere in Mississippi at the end of 2004/2005. [N=41,160]

2. All students who entered MSIS as tenth graders during school year 2002/2003, and were not already in the data file, were identified and added to the file. [N=4,384]

3. All students who entered MSIS as eleventh graders during school year 2003/2004, and were not already in the data file, were identified and added to the file. [N=2,344]

4. All students who entered MSIS as twelfth graders during school year 2004/2005, and were not already in the data file, were identified and added to the file. [N=1,304]

5. All students coded grade 56 or 58 (self-contained special education) who were the age of typical ninth graders during school year 2001/2002 were identified and their data written to a separate data file. [N=1,310]

6. All students coded grade 56 or 58 (self-contained special education) who were the age of typical tenth graders during school year 2002/2003, and were not already in the data file, were added to the file. [N=452]

7. All students coded grade 56 or 58 (self-contained special education) who were the age of typical eleventh graders during school year 2003/2004, and were not already in the data file, were added to the file. [N=331]

8. All students coded grade 56 or 58 (self-contained special education) who were the age of typical twelfth graders during school year 2004/2005, and were not already in the data file, were added to the file. [N=164]

9. Student records in the regular grade file and the grade 56/58 file were matched by MSIS ID to identify any duplicates. Of the 2,257 grade 56/58 students, 58 students had records in both files. An examination of the duplicate data records revealed that the only difference was in the cohort flag (year of entry into the cohort). Therefore, a procedure was run to eliminate one of the duplicate records and merge the regular grade and grade 56/58 data files. Unduplicated records comprised the full cohort. [N=51,391]

10. A tentative solution was used for student data records that contained both a completion code and a transfer code. This problem reflected data from the earliest year, prior to the addition of certain edit checks in MSIS. The solution was to blank out the transfer code for any student with a completion code (T, O, G, OD).


11. The cohort flag, transfer codes, and completion codes in the data file were used to set unique binary (0,1) variables that could be used for aggregating the student data at the school, district, and state levels.

12. The full cohort data file was matched by MSIS ID to the Month 1 2001/2002 enrollment file to determine which students coded as ninth graders in 2001/2002 were "true beginning" cohort students rather than students that had entered later in the school year. Of the 41,160 students, 38,833 were true cohort students and 2,327 had been added.

13. The full cohort data file was matched to the Month 9 2004/2005 enrollment file to determine the status for students who had no transfer code and no completion code. There were 145 "lost" T1 and T2 transfer students and 2,961 students enrolled in some grade.

14. For students in the full cohort data file whose disposition was still unknown after the above steps, summer activity codes were applied. Of the 2,741 students with summer activity codes from 2004 or 2005, 6.6% had completed all requirements except for a passing score on one or more tests needed for graduation, 58.5% had been coded as dropouts, and 35.0% had been coded as transfers or deaths.

15. Ultimately, there were 4,648 students in the full cohort data file whose final disposition was unknown. Since they were probably students who were lost during summer 2002 and summer 2003 (prior to the implementation of the summer activity process), it was decided to apply apportioning constants to the aggregate counts based on the actual percentage of students coded as summer dropouts and transfers during 2004 and 2005.

16. All the full cohort data records were summarized (aggregated) at the state level based on tentative logic outlined in the steps that follow.

17. Binary values (0 or 1) were accumulated across all student records for each of the following variables. Total Full Cohort N-Count, Cohort 0 (true cohort), Cohort 1, Cohort 2, Cohort 3, Cohort 4, T1, T2, T3, T4, T5, T7, T8, Z1, IS1, IS2, IS3, ST4, ST5, ST7, ST8, SZ1, Dropout, Dropsumm, Diploma, Trad, Occu, GED, Cert, ABT, Stillenrl, and Unknown.

18. The unknown student aggregate count was multiplied by 0.585 (58.5%) to yield the statewide number estimated to have been summer dropouts (SD1 through SD21).

2.2. Report Production

Reporting graduation rates has continued to be a challenge for national, state, and local

agencies. One complexity is in defining the audience receiving the information, then

customizing the reported data in a manner that meets the data collectors’ needs. The general

public typically receives information about graduation and dropout rates from local media

outlets. These data may or may not accurately reflect the official information provided by the

state agency. Explaining the plethora of business rules, differing methodologies, and reporting

formats often results in misinformation being disseminated. In worst case scenarios, politicians

and other policy-makers act on spurious information and make inappropriate causal links

between policy goals and reported outputs.


The passage of NCLB and its mandated reporting requirements has increased the

communication burden of state officials. Multiple graduation definitions and disaggregating

graduation data of subpopulations has resulted in many states using web-based decision-support

systems (DSS). Decision-support systems typically afford data collectors the ability to develop

customized queries for comparative purposes. In their simplest forms, the public can download

static data tables in a format that can be manipulated using commercial software packages (e.g,

Excel, SPSS, SAS). More sophisticated DSS allow the data collectors to define data elements

and extract files from the data warehouse to produce longitudinal data sets. Query-building tools

allow the user to link student data across multiple years without providing personally identifiable

records. In Indiana, the Indiana Department of Education has created a series of graduation and

associated data figures and charts for each of its schools and districts. These displays (see Figure

4) provide data collectors with easily understandable statistics on graduation rates based on

cohort calculations. The data for other types of students such as (a) dropouts, (b) GED

recipients, and (c) students still enrolled are compared to the percent of on-time graduates. The

state further disaggregates the graduation data by those reporting categories outlined by NCLB.

Additional information is provided to the data collectors by moving their cursers over the

selected data element, while hyperlinks connect data users with the applicable business rules (see

Figure 3).


Figure 2. Indiana Department of Education Graduation Rates

Figure 3. Indiana Department of Education Graduation Business Rules


It is clear that some states have the robust data systems necessary to produce cohort

graduation rates, and more. The tools described above allow for in-depth analysis not

possible with proxy graduation calculations. However, especially with the vast quantity of

publicly-reported data, as in the Indiana example, it is necessary to institute layers of quality

assurance practices to ensure the data generated is valid and reliable.


Section 3

Quality Assurance Practices in Producing Cohort Graduation Rates

Quality assurances practices are critical in producing valid and credible graduation rates.

Whether the agency is reporting the NGA, cohort, status, and/or completion rates, the processes

used to examine data inputs prior to score and report production is essential.

Quality assurance practices in most states are a combination of externally reported data,

internal review procedures (field specifications and error reports), and data verification

techniques. These processes allow agency officials to assert their graduation and dropout

determinations are valid representations of events in local schools and districts.

The judicious application of control measures (Wheeler & Lyday, 1989) is one approach

within a quality assurance design. Control procedures are used to evaluate targeted data

elements during the production cycle and mitigate unwanted error. Without such control

procedures, valid inferences about performance cannot be made for the given year. The controls

must be sensitive to detect slight changes in graduation (including non-graduate) indicators,

while discerning actual change from natural variability and non-systematic error.

Progress in creating clear and concise operating rules has been made at the national and

state-level. Graduation, completer, and dropout data being collected and reported from student

information systems has significantly improved the quality of the results. States are developing

and implementing quality assurance practices at critical points in the score development process.

Some states, such as Indiana, Utah, and Delaware, are continuously examining their internal

practices and capacity used to produce graduation rates. Other states, such as Mississippi and

Louisiana, conduct extensive quantitative design testing prior to policy adoption.


All state data contain data anomalies; some are valid while others do not represent actual

facts. Several critical areas, such as primary keys used to link multiple years of data for a

student, require differing levels of error detection and controls. State and local officials have

limited time and human resources to investigate each and every data point in their information

systems. However, some data elements require more effort than others because of their overall

influence on the final score. A better understanding those critical data associated with applying a

cohort approach can assist state and local leaders directing quality assurance efforts.

A state applies quality assurance practices to document calculations for external review

Delaware’s schools and districts operate within the public sector; thus their activities

must reflect both transparency and stakeholder input. Accountability activities, part of the

greater education system, must ensure that scarce resources are being used in the best interest

of the public. Continuous improvement practices at both the state and district-level strive to

maintain high quality through transparent practices and credible decision-making to promote

consumer satisfaction. One approach used by the Delaware Department of Education (DDOE)

is to document the entire accountability subsystem (DDOE, 2006), including the production of

graduation rates, in sufficient detail so as to facilitate external reviews. Recognizing how the

accountability system is dependent upon information management and assessment subsystems;

the agency must annually examine its current practices. Anticipating having to calculate the

National Governors Association’s (NGA) graduation rate and other completer information,

state officials evaluated areas needing additional capacity development. Because of the

interdependency among subsystems within the agency, specifically information management

and accountability programs, special consideration was been given to understanding how

improvement efforts will create spillover effects (i.e., additional benefits at no additional cost).


3.1. Applying the QADM to Graduation Rates

The Quality Assurance Diagnostic Matrix (QADM) was designed through research for a

previous paper on data validity issues for CSSSO’s ASR SCASS, and presented as a tool for

SEAs to examine the quality assurance capabilities in producing cohort graduation rates. The

QADM was derived from the conceptual work found within the Systems Security Engineering

Capability Maturity Model® (Carnegie-Mellon, 2003) and Six Sigma’s DMAIC® process

(Pyzdek, 2003). The QADM for calculating graduation and dropout rates comprise a set of

quality assurance indicators for two critical educational subsystems: (a) information

management, and (b) accountability [including other public reporting]. These performance

indicators are aligned within each of the targeted subsystems and evaluated across a capacity

continuum. The continuum is organized into four stages that reflect increased organizational

capacity and quality assurance sophistication.

The performance indicators within each subsystem represent a finite set within the

universe of potential indicators. For use in graduation and dropout rate production, several

indicators were modified from the original design (CCSSO, 2006). The graduation rate

indicators focused on validating data, training data collectors, improving quality practices, and

evaluating system controls. This set would most likely be refined by state and local officials to

match the operational activities unique to each subsystem.


Figure 4. Delaware’s Graduation Rate Diagnostic Matrix for SY 2005-06

Quality Assurance Diagnostic Matrix (QADM)-2006 Stage 4 4. Leading

3b. Streamlining Stage 3 3a. Improving Pr

oact

ive

2b. Documenting Stage 2 2a. Standardizing

1b. Reviewing Stage 1

1a. Planning

0b. Completing

Rea

ctiv

e

Stage 0 0a. Omitting

E

duca

tiona

l Sy

stem

s •

IT (1

.1 th

ru 1

.5)

• A

ccou

ntab

ility

(3.2

thru

3.

5)

1.1-

Inte

grat

es p

olic

y ch

ange

s

1.2-

Val

idat

es p

rogr

am

mem

bers

hip

1.3-

Trai

ns d

ata

prov

ider

s

1.4-

Exam

ines

fiel

d sp

ecs a

nd

asso

ciat

ed b

usin

ess r

ules

1.5-

Det

erm

ines

dat

a qu

ality

ne

eds

3.2-

Ope

ratio

naliz

es p

olic

y

3.3-

Supp

orts

acc

urat

e in

terp

reta

tions

3.4-

Alig

ns sy

stem

co

mpo

nent

s

3.5-

Eval

uate

s TO

A

beha

vior

alch

ange

s

The QADM reflects the capacity stages at a specific moment in time, rather than a

forward looking (goal setting) approach. The vertical axis provides a continuum of quality

process characteristics organized into stages. These stages range from quality practices not being

addressed (Stage 0) to standardized procedures used to maximize efficiencies (Stage 4). The

rating process requires the users to rate each quality indicator found across the X-axis using the

capability scale reflected on the Y-axis. The Y-axis continuum reflects the degree of capacity

necessary for task implementation (see Figure 4).

The first component of the QADM, on the X-axis, requires the agency to select targeted

indicators associated with programmatic aspects of the agency. Within these subsystems, time

and money resources are used to produce targeted outputs. Often several subsystems must work

on individual parts of the targeted output. For example, statewide assessments used student-level


information to determine critical administrative tasks necessary to produce test scores for each

eligible student. Critical quality indicators are organized within two key subsystems:

Information Management and Accountability. It should be noted that these indicators would

need be converted into operational terms (objectives, tasks, resources, milestones, measures, and

performance standards) prior to implementation (see Maine Department of Education’s Quality

Assurance Plan, 2006).

Information Management

This subsystem comprises the following quality indicators:

• Integrates policy changes into the student information system.

• Validates enrollment and program membership with data collectors and coordinates

information within the agency.

• Trains data providers on error detection using screening and auditing functions.

• Examines field specifications against business rule narratives.

• Determines data quality needs and implements improvement activities.

Accountability

This subsystem comprises the following quality indicators:

• Validates score and report production.

• Operationalizes and standardizes policy.

• Supports accurate interpretations.

• Promotes credible judgment supported by additional evidence.

• Aligns system components.

• Evaluates Theory of Action (TOA) behavioral changes.

3.2. Capacity Stages


The second component of the QADM is the continuum of quality capacities, represented

on the Y-axis of Figure 4. These capacities are dependent on the time and human resources

available to the entity under review. External demands (e.g., NCLB) create additional resource

demands that otherwise could be directed to improve existing operations. This “Catch-22” has

been a significant challenge to both state and local agencies as they attempt to implement new

programs prior to establishing sufficient capacity for those activities currently operating. In the

QADM, the capabilities are represented as follows:

Stage 0

0a. Omitting - Not addressed by the entity (we don’t do it here)

0b. Completing - Informal and random (someone does some part somewhere)

Stage 1

1a. Planning - Planned but undocumented (someone does it, but it is not in writing)

1b. Reviewing - Monitored but undocumented (someone does it, it [the outcome of the review]

is in a written report but not in the process)

Stage 2

2a. Standardizing - Mixed formality (done, written reports, some documentation and

standardized procedures)

2b. Documenting - Formal yet compartmentalized (done, standardized, documented, subsystem

specific)

Stage 3

3a. Improving - Standardized and validated (done, standardized, improved upon, checked,

subsystem specific)

3b. Streamlining - Systemic and dynamic (done everywhere, continuously improved upon)


Stage 4

4. Leading - Efficient and transparent (how we do business, industry-leading practices)

The QADM modified for graduation and dropout rates provides a simple graph that

defines the quality assurance needs across the agency. At this point, officials are able to

visualize quality assurance areas needing development and/or augmentation. In Maine, the state

officials at the Department of Education tailored the QADM approach in developing their quality

assurance plan (MDE, 2006). Beyond modifying the indicators, the Maine Department of

Education created performance descriptors reflecting the differing stages of capacity.

Accounting for every student within a state is a complex task made more difficult by

diverse programs, student mobility, changing policies, and political demands. Graduation

information continues to provide an indicator of educational quality and is readily understood by

the general public. The underlying data and the differing methods used to produce graduation

statistics is in of itself complex and diverse.

Three basic graduation rate components are demographic, membership, and

enrollment/exit data. Among the specific data elements within these components, errors can

occur that can have a significant influence on determining the graduation status of the individual

and when aggregating data to higher units of analysis. Errors can be created by either data

collectors, exit code taxonomies (see National Forum on Education Statistics, 2006 publication

for additional information), or a combination of both. Even with the best quality assurance

mechanisms, errors will remain in the master data file used for score production. Other errors

enter into the system because of the complexity of including the information using the existing

business rules. Table 3 outlines some of the technical issues states must address when

calculating cohort and NGA graduation rates. Issues are rated by the potential influence on the


reported graduation rate. A rating of 1 means data with this type of error produces minor errors

in the overall results. Issues with a rating of 2 contain readily detectable mistakes in the final

scores, while a rating of 3 invalidates the results. For example, Issue #13 would be rated a 2 for

calculating completion rates and a 3 for the NGA graduation rate. The decision logic being that

adding the GED data into the NGA rate is a substantive violation of the business rules. For

completer data, GED data would most likely be a reported category, thus errors would be readily

detected by simple visual inspection of the data.

Table 3: Error Control Issues (School-level)

Potential Impact Issue Completer Rate NGA Rate

1. Changing school grade configurations 2 2 2. Alternate learning centers 1 1 3. Retained students 1 3 4. Summer dropouts 1 1 5. Unknown status 2 2 6. Nongraded students 1 1 7. Juvenile centers 1 1 8. Redistricting 2 2 9. Multiple feeding patterns 1 1 10. “Super seniors” 1 1 11. Transfer to non-public school 2 2 12. Transfer out of state 1 1 13. Transfer to GED program 2 3 14. Transfer to home school 2 2 15. Grade 8 summer dropout 1 1 16. Residence programs 1 1 17. Regional schools 1 1 18. Grade skipping 1 2 19. Enrollment grade is less than prior year 2 2 20. Duplicate records 3 3 21. Completion date precedes transfer data 1 1

1 = Minor errors 2 = Detectable 3 = Invalidates results


3.3. Actions Prior to Public Release

The quality of any product or service must adhere to minimum quality standards while

seeking ways to improve either quality and/or efficiencies associated with production. In

examining the production processes necessary to report cohort graduation rates, state agencies

must examine those controls being used in other production processes. Controls are those efforts

used to identify and reduce unwanted error. As outlined in Figure 1, quality control can be

organized into three general areas: (a) those associated with verifying score accuracy (post

production), (b) internal reports created within the agency, and (c) reports and activities external

to the agency. Appendix C provides a sample of quality assurance practices for several states.

3.3.1. Internal Auditing (IA)-Internal auditing typically explores its use of screening

standards, production controls, and pre-release quality assurance efforts. The QADM can

function as a tool to accomplish the “where are we now” task. High risk areas and those data

elements within particular schools and school districts that were beyond established tolerances

are flagged for further IA activities. Automated subroutines are written, disseminated, and their

findings are reviewed within the agency. These results may constitute policy changes, additional

control development, or the use of external activities (i.e., on-site monitoring) prior to

accountability score production.

Internal auditing tasks are also focused on the data collection, validation, and production

procedures. These activates are almost always conducted by a third-party not affiliated with the

agency. The findings are typically used to formulate improvement efforts and build staff

capacity. Costs associated with IA activities are moderate to high regardless of whether out-

sourcing approaches are used. Timing is critical for IA activities, especially if it is associated


with auditing programming codes used to extract, validate, and produce scores. The importance

comes from having IA results available in enough time to implement corrective actions.

The IA tasks use test samples produced from both random and purposeful sampling

techniques. Beyond random selections, a conscious effort is made to include large and small

schools, new schools, and idiosyncratic situations. The typical sample represents approximately

10% of all public schools and districts within the state. When score anomalies are encountered,

additional schools are checked and rechecked until the internal auditor and quality assurance

team member are satisfied that the problem has been corrected.

3.3.2. Data Collector Verifications-Data collectors, those inputting the information and

those persons receiving the final reports are often some of the best resources available to state

officials. Having these individuals support data clean-up and validation activities can

significantly reduce undetected errors. For agencies operating decision support system (secure),

local staff can examine very granular data points at the student-level and make any necessary

corrections. Advanced data validation and score examination tools can be made available to data

collectors attempting parallel data runs.

Preliminary data can serve as a verification benchmark prior to final score acceptance and

public release. Data errors and programming inconsistencies are readily detected by individuals

who have the ability to independently produce their own graduation rates. However, training

costs are high for these activities. Standardized procedures amenable to third-party replication

and robust professional development initiatives are needed for even mature systems. Costs can

be reduced by manipulating existing subsystems and modifying training materials currently in

operation.

3.3.3. Rate Assignments- Examining the graduation, completion, and dropout rates can be


accomplished using simple comparative checks to validate that the results appear “reasonable.”

This is typically done when prior experiences and knowledge of schools and districts confirm

pre-established expectations. Time and labor costs are relatively low, the technique detects minor

variances within high risk data elements. Newly emerging systems should not rely exclusively on

this approach. Further, state officials would benefit from placing analytical emphasis to each unit

of analysis.

3.3.4. Trend Analysis- A standardized approach used extensively in the manufacturing

and petrochemical markets is the application of statistical process controls (Pyzdek, 2003,

Wheeler, 2003) that monitor variances during production. Delaware (DDOE, 2006) applied this

principle by establishing prevalence thresholds based on several data points. Evaluating the

production results, data elements, and screening results using standardized thresholds allows for

automated quantitative analysis. This approach provides critical information about whether the

underlying data are of sufficient quality to provide credible results.

The trend analysis approach does require multi-wave data, four or more years of data and

the additional time (often six months or more) to finalize the “standard year window.” Thus is

often limited to more “mature” systems. Time and labor costs to establish performance

thresholds, create the necessary programming code, document the procedures, and train staff on

For newly emerging system, another approach is to examine the year to year

matriculation and dropout rates. Over a four-year period the accumulative percentage moving

will always be greater than the amount estimated by a four-year graduation rate. The

difference between the two indicators increases inversely with the proportion of students who

fail to continue to the next grade.


the how to implement methodology are typically high. Modifying procedures used in other

educational subsystems (i.e., assessment) combined with outsourcing are ways to reduce costs.

3.3.5. Screening Report Templates- Screening paper, web-based and oral presentation

materials prior to public release are essential controls for systems of all levels of maturity.

Internal, external or a combination of controllers evaluate the data and narrative language to

ensure it is accurate. Many entities develop simple checklists and spreadsheets that guide the

reviewers’ efforts, while concurrently providing the necessary documentation for future quality

assurance activities.

This approach assumes the underlying data quality and production results are within

acceptable tolerances. Costs in developing standardized procedures and instrumentation is

moderate to low. Implementation costs are moderate and can be reduced by increasing internal

capacity. The resulting efforts reduce the likelihood that senior officials will present information

and supportive background data that differs from the actual production results. Further, it

provides assurances that the publicly disseminated, and data typically reported by the media, are

credible and able to withstand public scrutiny.

3.4. Actions after Public Release

3.4.1. After Action Reviews (Reflections) - This qualitative process is conducted with the

key operational staff, including programmers, policy, public affairs, and accountability

personnel. The basic intent is to highlight the work flow and task accomplishments that worked

as designed and those that failed and required additional time and effort to resolve.

3.4.2. Risk Assessments - Risk assessments are conducted on data elements and

production procedures in which small errors can significantly degrade output quality. These

types of analysis can range from informal meetings within the agency based upon experience to


formal reports by a risk management entity. All production procedures have some degree of

risk. Risk, the potential for reporting information that does not reflect truth, must be addressed

after each production sequence. Information from these tasks is often used to initiate other

control activities such as on-site monitoring and training. Time and labor costs are typically low

in most situations.

3.4.3. External Audits - Auditing as a control is the most intrusive and confrontational.

External entities conduct auditing activities to verify the accuracy of data reported to the agency.

Further, the auditors will examine the system of controls to prevent fraud, waste, and abuse.

Their tasks are to evaluate and report deviations from established procedures and policies.

Legislative and fiscal auditors have standardized sampling and review procedures that allow for

comparisons across groups. Some auditing functions (see Utah external auditing - Appendix E)

are required by policy; however, this is typically associated with fiscal accountability.

3.4.4. On-Site Monitoring - The agency uses data from IA, risk assessments, and other

control activities to conduct on-site reviews. The agency notifies each local superintendent

regarding the on-site monitoring of selected schools and the district’s central office. Standardized

interview protocols, document review checklists, and other information collection tools are

developed and assigned to each team member. The monitoring design calls for the verification

of data reported by the school and school district. On-site information is collected from data

providers, counselors, administrators, and program (including information managers) staff at the

central office.

On-site monitoring has high costs to both the agency and data collectors, mostly in time

to prepare, conduct, and respond to the findings. This control differs from “pure audit” because

the monitoring team is attempting to examine current practices with the intent to resolve poor


practices, rather than apply sanctions and consequences. Similar to the Title I monitoring

approach used in many states, the data collectors benefit by being provided with a series of

recommendations and operational details necessary to resolve the underlying problem.

For example, districts and schools should maintain documentation on all of the students

who leave the district. This documentation on who, when, and why the student left should be

reflected in the data used by the agency for graduation calculations. Inaccuracy in the exit codes

should be resolved at the source, especially when it reflects brinksmanship. On-site monitoring

costs often provide helpful information about where additional professional development efforts

should be focused.

3.4.5. Technical Manuals- These documents can provide detailed business rules, design

logic, and previous rate distributions necessary to validate task fidelity. Also, technical manuals

are one approach used to document quality assurance efforts. Technical manuals organize the

business rules, design logic, and operational programming in such a manner as to promote

transparency and increase system credibility. Business rules are often triangulated against policy

and computer languages to ensure alignment between these interactive subsystems. This

detection procedure continues to be implemented each year, with particular focus on policy-

driven changes.

Costs associated with the initial development can range from low to moderate based upon

how well the agency documents its internal procedures. Typically, the documentation resides in

numerous staff computers (including the programming code) on the agency’s servers, at worst, it

resides solely in the institutional knowledge of a few staff members.

3.4.6. Data collector Trainings - This control should focus on the data provider at the

school and the information managers within the district. Information systems are complicated by


their extensive codifications and jargon not usually used by school-level personnel. Data

providers at high schools are often inputting thousands upon thousands of entries over a given

school year, which leads to coding errors. Also, data providers may not have first hand

information about a given student and must rely on teachers and counselors. This phenomenon

increases the likelihood of errors produced through ineffective communications (“JP didn’t

dropout, he moved to Mexico or was that Mexico, Maine?”).

Training costs for data collectors should focus not only on software interface but data

quality. These costs are high for emerging systems and moderate for mature systems. Typically,

expenditures are assigned to activities focus on personnel using the interfaces correctly, rather

than on verifying the data input is accurate. Costs can be managed when teacher-led resources

are utilized in building capacity. Information management assets should provide the content and

then assign professional development experts to provide the in-services.


Section 4

Conclusion

Quality assurance practices for educational agencies must define those specific actions

necessary to ensure credibility. Local and state agencies must have robust and standardized

controls that ensure non-systematic errors are not reported to the public. Graduation rates

calculated using a cohort method require high-quality inputs from information management

subsystems prior to score production. This interrelationship exists because student

characteristics, membership and cohort inputs are necessary to determine high school graduation,

completion, and dropout rates. These rates provide one indicator of the school’s effectiveness in

producing students ready to enter into the workforce or to begin additional schooling.

The QADM provides a series of indicators by which to monitor improvement in quality

assurance practices. These practices require the capacity (time, money, and people) necessary to

implement sufficient detection and control procedures. Areas that were initially determined to be

below Stage 2 are at an increased risk of unwanted variance entering into their graduation,

completer, and dropout results. These areas have the highest potential to introduce error and/or

misinformation into the results available for public consumption.

Controls must be established to mitigate risk through pre-screening data, standardizing

production procedures, conducting internal audits, and providing training. These efforts should

differ based on the maturity and capacity development actions of the agency. Communication

efforts must focus on the public’s understanding that graduation estimates are just that, estimates.

Rates determined using a cohort approach will produce higher quality information about high

schools.


The movement toward reporting a cohort graduation rate is clear and progressing. With

all 50 governors on board, state education agencies are aiming to produce valid and reliable

reports. Toward this end, as state data systems are being built or refined, it is important to build

in quality assurance practices, which can take many forms, to ensure that the best possible data

are reported. Quality assurance efforts must seek to prevent misclassification from poor data and

design flaws. These tasks must be accomplished with scarce resources and limited time for

capacity development before the next production cycle. At the same time, SEA officials realize

that implementing quality assurance practices, any of the methods outlined in this paper, will

improve the quality of the data used in producing cohort graduation rates. The result, when

implemented carefully and systematically, will be a more valid and reliable report.


References

Council of Chief State School Officers (2006). Validity threats: Detection and control

practices for state and local education officials. Washington, DC: Author.

Delaware Department of Education (2006). Technical and operational manual for 2006.

Dover, DE: Author.

Greene, J.P., and Winters, M.A. (2005). Public high school graduation and college-

Readiness rates. Working Paper No.8. New York: The Manhattan Institute.

Maine Department of Education (2006). Standards, assessment, and accountability

quality assurance plan. Augusta, ME: Author.

Mississippi Department of Education-Office of Research and Statistics (2007). Estimated

graduation completion and dropout counts and rates based on approved procedures for tracking

a cohort of students over 4 years. Jackson, MS: Author.

National Forum on Education Statistics. (2006). Accounting for every student: A

taxonomy for standard student exit codes (NFES 2006-804). U.S. Department of Education.

Washington, DC: National Center for Education Statistics.

National Governors Association (2005). Graduation counts: A report of the National

Governors Association task force on state high school graduation data. Washington, DC:

Author.

National Governors Association (2006). Graduation counts: Compact and task force

report-Guidance on state implementation and reporting. NGA Center for Best Practices.

Washington, DC: Author.

National Institute of Statistical Sciences and Education Statistics Services Institute

(2004). National Institute of Statistical Sciences/Education Statistics Services Institute Task


Force on Graduation, Completion, and Dropout Indicators (NCES 2005-105). U.S. Department

of Education, Washington, DC: National Center for Education Statistics.

No Child Left Behind Act of 2001, P.L. 107-110, 115 Stat. 1425 (2002); and 34 CFR Part

200 Title I-Improving the Academic Achievement of the Disadvantaged, Final Regulations

Section 200.19(a)(1).

Pyzdek, T. (2003). The Six Sigma handbook. New York, NY: McGraw-Hill.

Seastrom, M., Hoffman, L., Chapman, C. and Stillwell, R. (2007). The Averaged

Freshman Graduation Rate for Public High Schools From the Common Core of Data: School

Years 2002–03 and 2003-04 (NCES 2006–606rev). U.S. Department of Education, National

Center for Education Statistics. Washington, DC.

Seastrom, M., Chapman, C., Stillwell, R., McGrath, D., Peltola, P., Dinkes, R., and Xu,

Z. (2006).User’s Guide to Computing High School Graduation Rates, Volume 1: Review of

Current and Proposed Graduation Indicators (NCES 2006-604). U.S. Department of Education,

National Center for Education Statistics. Washington, DC: U.S. Government Printing Office.

Smith, N. J (2006). Implementing the NGA graduation rate compact: State-level issues.

Washington, DC: CCSSO.

Swanson, C.B. (2003). Keeping and losing count: Calculating graduation rates for all

students under NCLB accountability. Washington, DC: The Urban Institute, Education Policy

Center.

Swanson, C.B. (2004). Who graduates? Who doesn’t? A statistical portrait of public high

school graduation, Class of 2001. Washington, DC: The Urban Institute, Education Policy

Center.


Wheeler, D.J. (2003). Making sense of data: SPC for the service sector. Knoxville, TN:

SPC Press.


APPENDIX A

AYP Graduation Rate Designs across ASR Member States (NISS, 2004)

State Indicator Type

Algorithm Data Elements

AL Departure

G = High School Graduates Students receiving a standard diploma for completion of state-developed requirements for a public secondary education program in year y. Includes: recipients of Alabama High School Diploma, Alabama Occupational Diploma, and Alternate Adult High School Diploma; and summer graduates. Excludes: recipients of non-standards-based completion certificates or GED. A = Alternative High School Completers Students receiving non-diploma high school completion or exit documentation (e.g., graduation certificates). Excludes: recipients of GED or other high school equivalency credential. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)

AK Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers Students completing a public secondary education program in year y without receiving a standard diploma. Includes: nonstandard diplomas, certificates (e.g., Certificates of Achievement), and GEDs. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)

AZ Cohort

G = High School Graduates Students from an entering freshman class who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). E = Entering Cohort Number of students who started high school (i.e., ninth grade) in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death. Excludes: dropouts.




CA Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: students who receive high school equivalency certificate or GED. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State adopted CCD dropout definition in 2003.)

CT Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: special education students who have until age 21 to earn a regular diploma. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

DE Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: students who receive a GED certificate. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

HI Cohort

G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard (Board of Education or Department of Education) diploma. Excludes: other program completion credentials offered in lieu of a standard diploma (e.g., IEP certificate) and equivalency credentials (e.g., GED). E = Entering Cohort Number of first-time ninth grade students in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system. Excludes: dropouts (as defined by CCD).

IA Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: students receiving regular diplomas from an alternative placement within the district, or who have had the requirements modified in accordance with a disability. Excludes: other program completion credentials offered in lieu of a standard diploma (e.g., certificate of attendance) D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Includes: GED recipients. (State uses CCD dropout definition.)




IL Cohort

G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard diploma. Excludes: students with too few credits to graduate, other program completion credentials offered in lieu of a standard diploma (e.g., IEP certificate) and equivalency credentials. E = Entering Cohort Number of first-time ninth grade students in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who graduated in year y but were not members of the original entering cohort. Includes: students transferring into the local school system at cohort grade-level, and students who graduated in fewer or more than four years. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death. Excludes: students who drop out or are expelled.

IN Other D = High School Dropouts Students who were enrolled in school during school year y or the previous summer recess and leave the educational system during that period without graduating from high school. Excludes: death, temporary absence due to suspension or a school-excused absence, and transfer to a public or nonpublic school. (State does not use CCD dropout definition.) E= Enrollment Students enrolled in grade g during year y. DR = Dropout Rate Calculated for year y by dividing number of students who drop out of grade g by number of students enrolled in that grade at the beginning of the school year.

LA Other

D = High School Dropouts Students who were enrolled in grades 9-12 at some time during school year y who were no longer in enrollment in school on October 1 of the following year. Includes: students expected to be in membership in year y+1 but have not graduated from high school or completed a state- or district-approved secondary educational program. Excludes: transfer to another public school district, private school, or state- or district-approved education program which might include a GED preparation program; temporary school-recognized absence due to suspension or illness; and death. (State uses CCD dropout definition with alternative reporting calendar.) E = High School Enrollment Cumulative student enrollment in grades 9–12 for school year y. Includes: any dropouts not included in cumulative enrollment (e.g., summer dropouts).

MA Cohort G = High School Graduates Students who took the tenth grade state assessment in year y-2 and graduated with a competency determination in year y. Excludes: students who have transferred into or out of the school system since the time of the tenth grade assessment. E = Enrollment Base Number of students enrolled in grade 10 in year y-2 (i.e., enrollment at the time of the tenth grade assessment). I = Inflow Students who joined the graduating class since the time of the tenth grade assessment by transferring into the local school system at grade-level. O = Outflow Students who leave the graduating class since the time of the tenth grade assessment by transferring out.




ME Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: students receiving a diploma after an approved fifth year of extended study. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Excludes: students enrolled in an approved fifth year.

MI Cohort

G = High School Graduates Students from an entering freshman class who receive a standard diploma for completion of a public secondary education program within four years of starting ninth grade. Includes: recipients of a standard diploma. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials. E = Entering Cohort Number of students who started high school (i.e., ninth grade) in year y-3. Cohort membership (m) is defined as individuals who were enrolled for the first time in a particular grade (9) at a given point in time (y-3) within a public school system (e.g., school, district). I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system or death.

MN Departure

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

MS Cohort G = High School Graduates Students who receive a standard diploma for completion of a public secondary education program in year y. E = Enrollment Base Number of ninth grade students in year y-3. I = Inflow to Cohort Students who joined the original cohort by transferring into the local school system at cohort grade-level. O = Outflow from Cohort Students who leave the original cohort due to transfer from the local school system. F = Failing Students Number of students at cohort grade-level failing over the four-year period. (Note: Based on review of state documents, “failing” is understood to mean students retained in grade.)

NC Other

G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). m = Graduating Class Indication of student’s expected graduating class, determined on time elapsed since taking the state’s eighth grade assessment. For the on-time graduation class in year y (i.e., m=1), students would have taken the eighth grade test in year y-4.




NE Departure G = High School Graduates Students receiving a standard diploma for on-time completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

OH Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

OK Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Includes: summer graduates. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers (GED) Students receiving GED credentials. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition with alternative reporting calendar.)

PA Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). A = Alternative High School Completers Students receiving nonstandard diplomas. D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

RI Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)

UT Departure G = High School Graduates Students receiving a standard diploma for completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. Includes: students’ program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). (State uses CCD dropout definition.)




WV Departure G = High School Graduates Students receiving a standard diploma for on-time completion of a public secondary education program in year y. Excludes: other program completion credentials offered in lieu of a standard diploma and equivalency credentials (e.g., GED). D = High School Dropouts Students in grade g who leave school during year y and have not graduated from high school or completed a state- or district-approved secondary educational program. (State uses CCD dropout definition.)


APPENDIX B

Graduation Rate Design Comparisons

Designer Description Unique Business Rules Limitations

National Governor’s Association (NGA) Compact Rate

The rate is an adjusted cohort graduation rate based on the number of on-time graduates in Y1 divided by first time 9th graders in Y-4 + transfers in – transfers out.

4yr Graduates- Student receives diploma in four years, excluding GED, certificates or other school-based credit. Late Completers: IEP and Immigrants assigned to later cohort.

Designed for high schools with at least grades 9-12 Standard number of years is defined as four years 4 yr vs. 5 yr rates

Louisiana’s NGA Hybrid

The rate is an adjusted cohort graduation rate based on the number of on-time graduates, early graduates, and IEP 4+graduates divided by first time 9th graders in Y-4 + transfers in – transfers out.

Establish denominator For schools, start with first-time freshmen to establish the cohort. Remove legitimate exits. Add anyone transferring into the school prior to Oct. 1, 11th-grade. Add anyone transferring from another school within the district prior to Oct. 1, 12th-grade. Remove students with IEPs that indicate they will take longer than 4 years to graduate. Add any “IEP” students that graduate or that turned 22 without graduating. Establish numerator-All those graduating with a diploma in 4 years. Any early graduates from the specific cohort. IEP graduates (4+) under age 22. District will be the same, except we add transfers into the district prior to Oct. 1, 12th-grade.

“Legitimate” leavers are those students with exit the system but cannot be directly monitored within the operational parameters of the state’s student information system. For example, expelled or students transferring to private or home school settings. State will adjust exit reason if the student is found (transferred) in another school district; conversely students coded as transferred to another district but are “no shows” are redesigned as “dropouts”.



*Average Freshman Graduation Rate (AFGR)

The rate is an estimate of the percentage of high school students who graduate on time by dividing the number of graduates with regular diplomas by the size of the incoming freshman class 4 years earlier, expressed as a percent

Average Freshman Enrollment- Y-5 Grade 8 fall enrollment + Y-4 Grade 9 fall enrollment + Y-3 Grade 10 fall enrollment / 3 Graduating-12 grader receiving a regular diploma in Y-Total number of students graduating with a regular diploma in a single year (y)

Includes all graduates (repeaters, early graduators)-not truly on-time; Rate does not adjustments for movement in or out of the cohort. Requires grade 8 data with unclear handling rules for complex “feeding” pattern within the district. Total number of first-time 9th-grade students from three years prior to the chosen year (y-3)

Cohort Graduation Indicator (CGI)

The rate of students starting in C-4 and graduating with a regular diploma in four years + IEP five + years

9th Grade Cohort Y-4 First time 9th grades – Transfers out- GED Year and grade level a student transfers into a cohort. Year of graduation.

Underestimates the EACGI graduation indicator because it does not explicitly adjust for exclusions Requires the year a student enters the 9th grade and whether he/she enters 9th grade for the first time. A reason for a student's departure prior to graduation and documentation related to the departure.

Exclusion-Adjusted Cohort Graduation Indicator (EACGI)

The rate of students from one freshman class (first time freshman) and new entrants to that class (over the four years of high school) who graduate. Includes new members to the cohort and adjusts for documented transfers.

Annual data is collected for each student. School report/track students as they progress through high school Repeating 9th graders are differentiated from first time 9th graders. “Qualified departures” are (1) transfers to other institutions offering a state-designated diploma-granting program (2) imprisonment, and (3) death).

Requires the year a student enters the 9th grade and whether he/she enters 9th grade for the first time, year and grade level a student transfers into a cohort, and the year of graduation Schools must document reason for a student's departure prior to graduation and documentation related to the departure.

Mississippi Cohort The rate tracks a full cohort of students using the state’s student information system (MSIS) who graduate in four years. All students are included in a cohort.

Students entering as ninth graders one year and adds students entering the next higher grade in each subsequent year. Students assigned to special education self-contained grade code 56 or 58 enter the cohort based on "peer grade" (calculated using birth dates). Summer activity codes from 2004 and 2005 are included in the procedure and unknown students (predominantly students lost during the summers of 2002 and 2003) are apportioned into estimated dropouts and estimated transfers1.



MN HSGR The rate is calculated for a particular year by counting the number of graduates divided by the number of graduates plus selected dropouts from that year and previous three years. The most recent fiscal year is referred to as Year 4 while the previous fiscal years are referred to as years 3, 2, and 1.

Establish numerator The count of students graduating in Year 4. Establish denominator Add grade 9 students dropping out in year 1-grade 10 students dropping out in year 2-grade 11 students dropping out in year 3-grade 12 students dropping out in year 4-students graduating in year 4.

Graduates from any other year (beyond the current) are not included. Only those dropouts reported in certain grades in certain years are included in the denominator with the graduates. Requires data from grades 8-12, which produce an unduplicated record set.

*Greene Graduation Indicator (GGI)

The rate is an estimate of the percentage of grade 12 students receiving a regular diploma in Year 1 divided by the adjusted grade 8 enrollment Y-4.

Adjusted 8th Grade Enrollment- Y-4 Grade 8 fall enrollment * % enrollment change Y and Y-4. Step 1-Estimate the rate of change in the high school population form the time a freshman class started to the time it was to have graduated. Step 2. This rate of change is multiplied by the estimate of first-time freshmen that takes changes due to migration into account. Step 3. Number of diplomas awarded in the graduating year is divided by the estimated number of potential on-time seniors. Calculation of GGI requires information on:

Includes all graduates (repeaters, early graduators)-not truly on-time; not adjustments for movement in or out; reduces the estimate of first-time freshman by ignoring 8th and 9th grade drop outs; assumes changes in the population over the 4 years is due to migration. Design treated cases with less than 200 9th graders as “missing”, also they cautioned against its use with highly mobile populations. Requires the number of graduates receiving a regular diploma in a specified year, the high school population (grades 9 through 12) for a specified year for three years prior, the number of 8th graders enrolled four years prior, and the 10th grade students enrolled two years prior.

*Swanson’s Cumulative Promotion Indicator (SCPI)

The index is an estimate of the probability that a student will complete within the standard four year period.

Cumulative product of the proportion of students who progress from one grade to the next at the end of the school year for grades 9, 10, and 11 multiplied by the proportion of seniors who graduate at the end of the school year.

Requires the number of students in 9th, 10th, 11th, and 12th grades in the year of analysis and the umber of students in 10th, 11th, and 12th grades in the year after the year of analysis.


APPENDIX C

Selected auditing practices of selected states

Arizona Delaware Hawaii Indiana Louisiana Maine Nebraska Utah Internal auditing x Data collector verifications x x Rate assignments x x Trend analysis x Screening report templates x x Risk assessments x External audits x On-site monitoring x Technical manuals x x x x Data collector trainings x x

Quality Assurance Practices associated with Producing Cohort Graduation Rates, , CCSSO ASR SCASS 53

Appendix D

Mississippi Department of Education Report Estimated Graduation, Completion and Dropout Counts and Rates Based on Approved1

Procedures for Tracking a Cohort of Students Over 4 Years

(Final Report)

This report presents information on procedures used for tracking a cohort of students using data entered by school districts into the Mississippi Student Information System (MSIS). The target cohort comprised students entering ninth grade during the 2001/2002 school year plus students entering one grade level higher in each subsequent school year through 2004/2005. Thus, the study tracked the full cohort of students over a four-year period. The statistics presented in this report are "4-year" estimates of graduation, completion, and dropout results for the cohort. Work on this study could not begin until June 2005 -- after Month 9 (end of year) data had been received from all school districts. Since MSIS was implemented statewide in 2001/2002, that was the point at which four full years of student level data were available. The results presented in this report also include data from 2004 and 2005 "summer activity." Interpretations of data appearing in this report must acknowledge the following caveats and limitations: • The results are only as accurate as the data entered into MSIS. • The results include data from the initial implementation of MSIS. • The full cohort used in this report included repeaters. • There was no opportunity for districts to enter final disposition codes for students leaving school during the summers of 2002 and 2003. • Statistical procedures were used for apportioning students with unknown final dispositions into estimated dropouts and transfers. 1Approved through the state APA (public comment) process and State Board of Education action. Office of Research and Statistics

February 12, 2007


Introduction Information about dropouts, high school completers, and high school graduates is important to educators and the public for assessing the success of students and their educational experiences. Although the concept of a school dropout, completer, or graduate may seem simple, there are many variables that must be considered. Some states have not attempted to calculate and report one or more of the above counts or rates, and the procedures and formulas used by states has varied considerably with no standardization. Of the three rates, graduation rate is the one that has been reported by most states. Over the years, several studies have argued that graduation rates reported by states are inaccurate and are almost always too high. Some of those studies used reported enrollment to estimate state level (and, in some cases, district level) graduation rates. Although estimates based on enrollment, with or without adjustments for population changes, are not as accurate as rates based on tracking individual students over time, few states have had the data required for tracking individual students. Three sets of independent state level graduation rate estimates for Mississippi are shown in Table 1. Although the estimated rates differ and represent different student cohorts, all are considerably lower than the graduation rates reported by the state for the same years (80.5% for 2002 and 83.7% for 2004). Table 1 Independent Graduation Rate Estimates for Mississippi

Averaged Freshman Graduation Rate for the Graduating Class of 20041

Manhattan Institute Estimate Using the "Greene Method" for the Graduating Class of 20022

Urban Institute Projection Using the Cumulative Promotion Index for the Graduating Class of 20043

62.7% 60.0% 58.0% 1The averaged freshman graduation rate is based on a simple calculation using enrollment data and a count of graduates. This calculation is part of the more elaborate "Greene Method" cited below, but it does not require an adjustment for overall population change. Averaging the 8th, 9th, and 10th grade enrollments over three years helps eliminate the peak in 9th grade enrollment attributable to retention in that grade. This method approximates an ultimate graduation rate. The U.S. Department of Education adopted this procedure for reporting state level graduation rates beginning in fall 2005. See Seastrom, M., Hoffman, L., Chapman, C. and Stillwell, R. (2006) The Averaged Freshman Graduation Rate for Public High Schools From the Common Core of Data: School Years 2002-03 and 2003-04. Washington, DC. U.S. Department of Education [NCES 2006-606]. 2Greene, J. P. & Winters, M. A. (2005). Public High School Graduation and College Readiness Rates: 1991-2002 [Education Working Paper No. 8] New York: Manhattan Institute for Policy Research. This method approximates an ultimate graduation rate.


3Swanson, C. B. (2005) Projections of 2003-2004 High School Graduates: Supplemental Analyses based on findings from Who Graduates? Who Doesn't? Washington, DC: The Urban Institute, Education Policy Center. This method approximates a 4-year graduation rate. State graduation rates have taken on increased significance due, in part, to the inclusion of graduation rate as a mandatory "other academic indicator" for Adequate Yearly Progress (AYP) under the No Child Left Behind Act of 2001 (NCLB). NCLB provides some information regarding which students can be counted as graduates and it indicates that the calculations are to represent graduation in a "standard" number of years. The "NGA Graduation Rate" A major push for standardization in the way states calculate and report graduation rates resulted from a year-long initiative on high school redesign that culminated during the July 2005 meeting of the National Governors Association. During (and following) that meeting all states agreed to develop and implement a standardized method of calculating and reporting graduation rates. The method conforms to recommendations developed by the NGA Task Force on State High School Graduation Data and presented in its report, Graduation Counts. The report, available on the NGA Center for Best Practices web site at www.nga.org/Center, included the following recommendations. 1. Immediately adopt, and begin taking steps to implement, a standard four-year, adjusted cohort graduation rate using the following formula: Graduation rate = [on-time graduates in year x] ÷ [(first-time entering ninth graders in year x – 4) + (transfers in) – (transfers out)] 2. Build the state’s data system and capacity to ensure that the system can collect, analyze, and report the adopted indicators and other important information. 3. Adopt additional, complementary indicators to provide richer context and understanding about outcomes for students and how well the system is serving them, including five- and six-year cohort graduation rates; a college-ready graduation rate; a dropout rate; completion rates for those earning alternative completion credentials from the state or a GED; in-grade retention rates; and percentages of students who have not graduated but are still in school or who have completed course requirements but failed a state exam required for graduation. At the time the above report was released, the Mississippi Department of Education (MDE) had just begun working on procedures for tracking a student cohort in MSIS for purposes of calculating accurate dropout, completion, and graduation rates. MSIS satisfies the second NGA recommendation (a comprehensive statewide student level database) and contained four years of data by the end of the 2004/2005 school year. Although summer activity for 2005 would not be available until early spring 2006, preliminary estimates of dropout, completion, and graduation rates were compiled by the Office of Research and Statistics (ORS) and presented to the State Board of Education at its meeting in August 2005.The estimated rates calculated for the "true cohort" of students entering 9th grade at the beginning of the 2001/2002 school year were: Dropouts – 26.0%, Graduates (4-Year) – 60.6%, Graduates (ultimately) – 63.2%.


The above graduation rate estimates agreed closely with the independent estimates presented in Table 1. Additional work by ORS over the next 12 months would include the development of procedures for constructing a "full cohort" of students, incorporation of summer activity, and procedures for accounting for students whose final disposition (transfer, dropout, completion) was unknown. CCSSO Technical Panel The Council of Chief State School Officers gathered representatives from several states and organizations, the National Governors Association Center for Best Practices, and the Data Quality Campaign in June 2006 to evaluate and discuss implementation issues related to the NGA Graduation Rate. The goal of the technical panel was to identify standard implementation practices that could be adopted by all states to make the subsequent rates as comparable as possible. In its recently released report, Implementing the NGA Graduation Rate Compact: State-level Issues (available on the CCSSO web site at www.ccsso.org), the Technical Panel stated, "While initially the formula appears very straightforward, there are so many differences among state data systems as well as in state-specific rules for accounting for and providing services to students with special needs that cross-state comparability is not an easy feat with this one definition. This graduation rate is applied to students who receive a standard diploma, not a certificate of completion or attendance or a General Educational Development (GED) certificate." In addition to addressing data collection issues, the report discussed definitions and procedural issues such as cohort length, cohort definition, and special populations (e.g., students with disabilities and limited English proficient students). Working with the NGA compact guidelines, the CCSSO Technical Panel made specific recommendations for dealing with various data issues. The recommendations included the following (cited verbatim). • Because there may only be a small portion of the student body who is allowed additional time to graduate in their IEPs, states may choose to not include them in the NGA graduation rate cohort, and include them in a five- or six-year graduation rate… • Calculate the NGA rate for cross-state comparability, but consider calculating at least a five-year rate for students who receive a regular or advanced diploma within five years, without receiving the fifth year as a result of a special dispensation from the school, in addition in order to provide a more complete graduation picture for the state. • For the purposes of the NGA graduation rate, the panel recommends defining first time 9th graders as any student who was enrolled at least one day in grade 9 in Year X. • The panel recommends that the count of transfers in includes every student who enters the cohort on grade-level at any point during the four year period and does not exclude students who arrive late in the 12th grade (or any grade).


• Define, document, and implement a detailed student-level Exit Code data collection process to account for students who leave the public school during or between school years. • Establish a process by which the SEA reviews statistical trends of exit data within and across school years to identify potentially erroneous data. Establish a detailed review and validation process for samples of district, school, and/or student data. Establish clearly defined consequences for schools and districts that do not maintain clear and accurate documentation and validation processes that meet the states guidelines and for submitting erroneous data to the state. Clearly communicate each of the processes and consequences to districts and schools. • Students who ‘vanish’ (i.e., cannot be found in another location, no documentation exists for where they went, etc.) should be counted as dropouts, not as transfers out. • 4th-year summer graduates should be counted as graduates in the NGA rate. • Create a way to collect student-level data that identifies which students are allowed additional time to graduate by their IEPs. Calculate the NGA rate without these students the year after the end of the cohort and then again the following year with these students included. Be transparent in reporting the impact these students had on the rates and why the rate for a given year was recalculated and republished. • Follow the same general guidelines used for students receiving special education services [to deal with issues involving students with limited English proficiency]. • Follow existing state policies and practices regarding tracking and accounting for incarcerated students, but be very clear in how those students are included in the calculation of the NGA graduation rate. • Students retained in grades 9-12 remain in the cohort to which they were originally attributed. In its conclusion, the report states, "The NGA graduation rate is not intended to be used as an accountability tool by states or the U.S. Department of Education. The purpose of the NGA graduation rate is to provide governors and other policymakers with a standard definition based on student-level longitudinal and high quality data that enables comparability across states. Having a standard definition across states helps policymakers across the country communicate with and learn from each other as future research and policies are designed and undertaken. The Graduation/Completion/Dropout (GCD) Definition Committee In December 2005, the State Superintendent designated a committee to continue work on procedures for tracking cohorts of students in MSIS and calculating accurate graduation, completion, and dropout counts and rates. The committee met five times between December


2005 and October 2006 to discuss issues and review results based on data produced by the Office of Management Information Systems (MIS) and the Office of Research and Statistics. Updates of the state level results were presented to the Mississippi State Board of Education during July 2006 and October 2006. The final state level results are presented later in this paper. Although the CCSSO Technical Panel Report was only released in October 2006, the procedures developed by the GCD Definition Committee and used for estimating the state and district level counts and rates are in close agreement with the Technical Panel's recommendations. For example, the tracked cohort comprises students entering the ninth grade at the beginning of school year 2001/2002 ("true cohort" students) plus other ninth grade students entering during that school year plus students entering grade 10 in 2002/2003, students entering grade 11 in 2003/2004, and students entering grade 12 in 2004/2005. The full cohort also includes self-contained special education students who were added to the cohort based on "peer grade" (calculated using the student's age). All students remained in the identified cohort even if they were retained in a grade (9 through 12). Student transfers and deaths, as well as dropouts, were identified using school year and "summer activity" codes in MSIS. The system was designed to track the full cohort assessing the final disposition of each student at selected points (at the end of 4 years, the end of 5 years, etc.). Certain students with disabilities are counted differently in rates representing different time spans. Tracking a Cohort of Students in MSIS: The Big Picture As stated earlier in this report, preliminary tracking of the "true cohort" (students in ninth grade at the beginning of school year 2001/2002) produced an estimated 4-year dropout rate of 26.0%, an estimated 4-year graduation rate of 60.6%, and an estimated ultimate graduation rate of 63.2%. After much work developing procedures for constructing and tracking a "full cohort" and determining the final disposition for each student, the updated 4-year estimates were very similar. The steps included in the data analysis and calculations, and the intermediate results, are shown in Figure 1 (Computer Output – Step by Step). Establishing the Full Cohort Beginning with 38,833 students who were in grade 9 at the beginning of 2001/2002, 3,600 incoming ninth graders were added that year followed by 4,826 tenth graders in 2002/2003, 2,668 eleventh graders in 2003/2004, and 1,464 twelfth graders in 2004/2005. That resulted in a full cohort comprising 51,391 students. The total cohort included 2,199 self-contained special education students who had been added to the cohort based on "peer grade." Within the cohort, 7,023 students were students with disabilities (eligible under IDEA) sometime within the 4-year tracking period. Identifying Appropriate Denominators Using the transfer/death codes in MSIS (for each school year and for "summer activity" in 2004 and 2005), 7,742 students were identified as "coded" transfers/deaths. There were 4,648 students whose final disposition was unknown. Most of those students probably left (transferred, died, or


dropped out) during the summer in 2002 or 2003 – however, there was no way for districts to code the disposition for those students. Therefore, a procedure was developed for estimating the number of unknown disposition students who were probably transfers/deaths and the number who were probably dropouts. Using summer activity codes entered by districts for 2004 and 2005, it was determined that 35.0% of the students had been coded as transfers/deaths and 58.5% had been coded as dropouts. Those percentages were used to apportion the unknown disposition students into estimated transfer/death and dropout counts. Applying 35.0% to the 4,648 unknowns resulted in an estimate of 1,625 additional transfers and deaths. Thus, the total estimated transfer/death count was 9,367. That count was subtracted from the total cohort N-Count to yield a dropout denominator – 42,024. As explained later, a slightly different denominator must be used for calculating completion and graduation statistics. Estimated Dropout Rate Calculation Using the dropout codes in MSIS (for each school year and for "summer activity" in 2004 and 2005), 8,306 students were identified as "coded" dropouts. Applying the 58.5% summer activity dropout percentage value to the 4,648 unknowns resulted in an estimate of 2,718 additional dropouts. Thus, the total estimated dropout count was 11,169. Dividing that value by the dropout denominator (42,024) resulted in a 4-year dropout rate estimate of 26.6%.


Figure 1. Computer Output – Step by Step

COHORT STATISTICS FOR SY0102G09 -- OVER 4 YEARS NOTE1: APPORTIONING OF UNKNOWNS INTO DROPOUT AND TRANSFER/DEATH ESTIMATES WAS BASED ON DISTRICT SUMMER ACTIVITY CODING FOR SUMMER 2004 AND 2005. NUMBER OF STUDENTS WITH UNKNOWN FINAL DISPOSITION = 4648 NOTE2: ESTIMATED DROPOUT RATE WAS APPLIED TO STUDENTS STILL IN SCHOOL TO GET AN ESTIMATE OF POSSIBLE COMPLETERS BEYOND THE 4-YEAR PERIOD. NOTE3: VALUES MARKED WITH AN ASTERISK (*) ARE SUBTOTAL VALUES THAT ARE ALREADY INCLUDED IN ANOTHER LISTED VALUE. ------------------------------------------------------------------------------ N-COUNT COHORT 0 ( 9TH GRADERS MONTH 1 2001/2002) 38833 N-COUNT COHORT 1 ( 9TH GRADERS ADDED IN 2001/2002) 3600 N-COUNT COHORT 2 (10TH GRADERS ADDED IN 2002/2003) 4826 N-COUNT COHORT 3 (11TH GRADERS ADDED IN 2003/2004) 2668 N-COUNT COHORT 4 (12TH GRADERS ADDED IN 2004/2005) 1464 N-COUNT TOTAL (TRUE COHORT + ALL ADDED STUDENTS) 51391 ADDED GRADE 56/58 (BASED ON STUDENT AGE/PEER GRADE) 2199* SPED STUDENTS IDENTIFIED FROM 2001/2002 THROUGH 2004/2005 7023* ------------------------------------------------------------------------------ LOST T3 (MAY REPRESENT MSMS, MSA, MSB, MSD, ETC.) 1068 TRANSFER T4 ( TRANSFER TO NON-PUBLIC SCHOOL) 2893 TRANSFER T5 ( TRANSFER TO SCHOOL OUT OF STATE) 2521 TRANSFER T7 ( TRANSFER TO APPROVED GED PROGRAM) 138 TRANSFER T8 ( TRANSFER TO HOME SCHOOL) 99 TRANSFER ST4 (SUMMER TRANSFER TO NON-PUBLIC SCHOOL) 217 TRANSFER ST5 (SUMMER TRANSFER TO SCHOOL OUT OF STATE) 541 TRANSFER ST7 (SUMMER TRANSFER TO APPROVED GED PROGRAM) 118 TRANSFER ST8 (SUMMER TRANSFER TO HOME SCHOOL) 70 DEATH Z1 (DEATH) 65 DEATH SZ1 (SUMMER DEATH) 12 TRANSFER/DEATH PERCENTAGE USED FOR ESTIMATE (SEE NOTE 1) 35.0% ESTIMATED TRANSFER/DEATH FOR UNKNOWN STUDENTS (SEE NOTE 1) 1625 TOTAL TRANSFER/DEATH (THIS IS AN ESTIMATE) 9367 DROPOUT DENOMINATOR (ALL UNKNOWNS CONSIDERED TRANSFERS) 39306 DROPOUT DENOMINATOR (TOTAL MINUS TRANSFER/DEATH ESTIMATE) 42024 DROPOUT DENOMINATOR (ALL UNKNOWNS CONSIDERED DROPOUTS) 43649 CODED DROPOUTS (CODED DURING THE FOUR SCHOOL YEARS) 6703 CODED DROPOUTS (CODED IN SUMMER ACTIVITY 2004 & 2005) 1603 CODED DROPOUTS WITH REASON 12 (XFER TO GED OR NON-EDUC) 2388* DROPOUT PERCENTAGE USED FOR ESTIMATE (SEE NOTE 1) 58.5% ESTIMATED DROPOUTS FOR UNKNOWN STUDENTS (SEE NOTE 1) 2718 LOST T1 (DROPOUT: NO ENTRY IN ANOTHER GRADE IN SCHOOL) 6 LOST T2 (DROPOUT: NO ENTRY IN ANOTHER SCHOOL IN THIS DISTRICT) 139 ALL DROPOUTS OVER THE FOUR YEARS (THIS IS AN ESTIMATE) 11169 ESTIMATED DROPOUT RATE (LOW: UNKNOWNS CONSIDERED TRANSFERS) 21.5% (- 5.1%) ESTIMATED DROPOUT RATE (CODED PLUS ASSUMED-SEE NOTE 1) 26.6% ESTIMATED DROPOUT RATE (HIGH: UNKNOWNS CONSIDERED DROPOUTS) 29.3% (+ 2.7%)

Continued on Next Page


Figure 1. Computer Output – Step by Step (continued)

UNKNOWNS THAT WERE NOT APPORTIONED 305 NOT APPORTIONED (PERCENTAGE OF THE DROPOUT DENOMINATOR) 0.7% ------------------------------------------------------------------------------ GRADE 56/58 STILL AROUND AT END OF 2004/2005 827 ANY GRADE LEP STILL AROUND AT END OF 2004/2005 9 COMPL/GRAD DENOM (ALL UNKNOWNS CONSIDERED TRANSFERS) 38470 COMPL/GRAD DENOM (TOTAL MINUS TRANSFER/DEATH/SA56/SA58) 41188 COMPL/GRAD DENOM (ALL UNKNOWNS CONSIDERED DROPOUTS) 42813 COMPLETERS (CODED COMPLETERS OVER THE 4 YEARS) 27589 COMPLETED WITH A TRADITIONAL DIPLOMA (COUNTS AS A GRADUATE) 25057* COMPLETED WITH AN OCCUPATIONAL DIPLOMA (CHANGE:NOT A GRADUATE) 128* COMPLETED ALL REQUIREMENTS EXCEPT FOR SATP PASSING SCORE 180* COMPLETED WITH A CERTIFICATE OF ATTENDANCE 1481* COMPLETED WITH A DISTRICT PROGRAM GED 743* ESTIMATED COMPL RATE (LOW: UNKNOWNS CONSIDERED DROPOUTS) 64.4% (- 2.6%) ESTIMATED 4-YEAR COMPLETION RATE (WITH APPROTIONED UNKNOWNS) 67.0% ESTIMATED COMPL RATE (HIGH: UNKNOWNS CONSIDERED TRANSFERS) 71.7% (+ 4.7%) GRADUATES (CODED DIPLOMA [TRAD] RECIPIENTS OVER THE 4 YEARS) 25057 ESTIMATED 4-YR GRAD RATE (LOW: UNKNOWNS CONSIDERED DROPOUTS) 58.5% (- 2.3%) ESTIMATED 4-YEAR GRADUATION RATE (WITH APPORTIONED UNKNOWNS) 60.8% ESTIMATED 4-YR GRAD RATE (HIGH: UNKNOWNS CONSIDERED TRANSFERS) 65.1% (+ 4.3%) ESTIMATED 4-YR GRADUATION RATE (INCLUDING GED RECIPIENTS) 62.6% ------------------------------------------------------------------------------ STILL ENROLLED AT THE END OF SCHOOL YEAR 2004/2005 2961 STILL ENROLLED IN GRADE 12 (AND HAS NOT COMPLETED) 467* STILL ENROLLED IN GRADE 11 1030* STILL ENROLLED IN GRADE 10 422* STILL ENROLLED IN GRADE 9 93* STILL ENROLLED IN GRADE 8 (PROBABLE MSIS DATA ERROR) 2* STILL ENROLLED IN GRADE 7 (PROBABLE MSIS DATA ERROR) 1* STILL ENROLLED IN GRADE 56 (PROBABLE MSIS DATA ERROR) 13* STILL ENROLLED IN GRADE 58 (NOTE: 56/58 NOT IN DENOMC) 814* STILL ENROLLED IN GRADE 78 62* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS1) 28* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS2) 18* STILL ENROLLED ASSUMPTION (UNKNOWN WITH SA CODE IS3) 11* STILL ENROLLED (AS A % OF THE COMPLETION DENOMINATOR) 7.2% POSSIBLE FUTURE COMPLETERS (SEE NOTE 2) 5.3%


Estimated Completion Rate Calculation The number of students actually coded by districts as school completers (traditional graduates, occupational diploma graduates, district GED program completers, special education certificate of attendance recipients, and students completing all requirements for graduation except for a passing score on one of the Subject Area tests) was used as the numerator for calculating the 4-year completion rate. Using the special grade code (58) for self-contained special education students, 827 students with that code who were still enrolled in school at the end of 2004/2005 (plus 9 LEP students who were still in school) were subtracted from the dropout denominator yielding a completion/graduation denominator – 41,188. Dividing 27,589 coded completers by the completion/graduation denominator (41,188) produced a 4-year completion rate estimate of 67.0%. That rate is an estimate because the denominator includes an estimated count of transfers/deaths. Estimated Graduation Rate Calculation The number of students actually coded by districts as high school graduates (traditional graduates only) was used as the numerator for calculating the 4-year graduation rate. Dividing 25,057 coded graduates by the completion/graduation denominator (41,188) produced a 4-year graduation rate estimate of 60.8%. That rate is an estimate because the denominator includes an estimated count of transfers/deaths. Future Completers and Rates Beyond Four Years Although the NGA graduation rate is explicitly the rate for students completing high school in four years, both the NGA Task Force on State High School Graduation Data and the CCSSO Technical Panel acknowledged that such a rate provides only a partial picture regarding high school completion. Until data are available for individual students with disabilities concerning the number of years their IEP team determines is needed for graduation (or high school completion), it will not be possible to accurately adjust the completion/graduation denominator. For the 4-year completion and graduation rate estimates presented in this report, students with disabilities instructed in a self-contained secondary environment and still enrolled at the end of four years were not included in the completion/graduation denominator when calculating the estimated 4-year rates. A valid assumption is that those students would not be expected to graduate from high school within four years. In fact, many of those students' IEPs state that the student will complete high school with a certificate of attendance rather than a regular or occupational diploma. In addition to self-contained students with disabilities who would not be expected to complete high school within four years, there are other students (disabled and non-disabled) who were retained in some grade 9-12 and were still in school at the end of 2004/2005. Although those students are included in the denominator and are not counted as completers or graduates in the estimated 4-year completion and graduation rates, most will eventually complete high school. Of the students in the total cohort, 2,961 (7.2% of the students in the completion/graduation denominator) were still enrolled in school at the end of 2004/2005. Applying the 4-year dropout


estimate (26.6%) to the students who were still enrolled resulted in an estimate of 2,173 students who will probably complete high school in future years.

The 2,173 additional students represent 5.3% of the completion/graduation denominator. Adding 5.3% to the 4-year completion estimate yields an estimated ultimate completion rate of about 72%.


State Level 4-Year Dropout, Completion and Graduation Rate Estimates Once the procedure for tracking a full cohort of students in MSIS was approved, it was run separately for certain student subgroups to produce disaggregated estimates. Table 2 presents estimated 4-year dropout, completion, and graduation counts and rates by gender and ethnicity. Table 2 Disaggregated 4-Year Dropout, Completion, and Graduation Data

Group

Total Cohort N-Count

Dropout Denominator (Transfers and Deaths1

Subtracted)

Estimated Dropouts1

And Estimated 4 Year Dropout Rate2

Completion/ Graduation Denominator (Transfers, Deaths1 and Grade 58 Subtracted)

Completers and Estimated 4-Year Completion Rate

Graduates and Estimated 4-Year Graduation Rate3

Possible Future Completers4

All Students 51,391 42,024 11,169 26.6% 41,188 27,589

67.0% 25,057 60.8%

5.3%

Female 23,895 19,803 4,174 21.1% 19,524 14,427

73.9% 13,529 69.3

4.2%

Male 27,496 22,226 7,002 31.5% 21,669 13,162

60.7% 11,528 53.2%

6.1%

Asian 438 311 43 13.8% 308 255

82.8% 245 79.5%

3.1%

Black 26,094 22,437 6,748 30.1% 21,897 13,389

61.1% 11,839 54.1%

6.6%

Hispanic 508 286 74 25.9% 273 189

69.2% 174 63.7%

6.0%

Native American 107 65 26

40.0% 64 33 51.6%

30 46.9%

5.6%

White 24,244 18,838 4,227 22.4% 18,559 13,723

73.9% 12,769 68.8%

3.6%

Black Female 11,971 10,514 2,484 23.6% 10,343 7,166

69.3% 6,640 64.2%

5.5%

White Female 11,398 8,929 1,619 18.1% 8,828 7,012

79.4% 6,651 75.3%

2.7%

Black Male 14,123 11,923 4,266 35.8% 11,554 6,223

53.9% 5,199 45.0%

7.3%

White Male 12,846 9,913 2,613 26.4% 9,735 6,711

68.9% 6,118 62.8%

4.4%

Note: This table represents the official final run on 02/11/2007 and supersedes all earlier versions.

1Based on actual statewide 2004 and 2005 summer activity coding, 58.5% of unknown students were classified as dropouts and 35.0% were classified as transfers/deaths. 2Includes all coded school year and summer activity dropouts plus "lost" T1 and T2 transfers. This represents a 4-year "9-12" dropout rate. The customary "7-12" cohort dropout rate would be higher. 3Graduates include only traditional diploma recipients. Occupational diploma recipients, district GED recipients, special education certificate of attendance recipients, and students who completed all requirements except for a passing score on one or more tests required for graduation are completers, but not graduates. Note: Occupational diploma classification was changed on the 02/11/2007 run.


4Possible future completion percentage was calculated by applying the estimated statewide dropout rate to students who were still enrolled at the end of 2004/2005. Add the percentage in this column to estimate the ultimate completion rate; the estimated ultimate graduation rate will be somewhat lower.


A Procedure for Tracking a Full Cohort in MSIS: The Details This section of the paper provides details concerning the procedure developed for tracking a full cohort of students in MSIS and calculating estimated dropout, completion, and graduation rates for the full cohort. The objectives were to ensure that all students would be included in an appropriate cohort (i.e., no students would be systematically left out) and to use data available in MSIS to calculate accurate counts and rates for dropouts, completers, and graduates. Since MSIS was implemented statewide at the beginning of the 2001/2002 school year, the first four-year full cohort could be tracked using data from the end of school year 2004/2005 updated with the results of the 2005 summer activity procedure. All the required data were available in April 2006. Using the most appropriate analysis techniques, there are still certain errors that will be present in the calculated counts and rates. Those include coding errors in the data transmitted to MSIS from the district student administrative packages (SAPs), a small amount of data inconsistency (generally in the first school year) due to edit checks that were implemented later, and students whose final disposition is unknown because they left the system during summer 2002 or summer 2003 (before the summer activity procedures were implemented). Logic for implementing a full cohort tracking system was approved and data files were built using the data in MSIS. The steps used for tracking the cohort and analyzing the data follow.

1. All students who entered MSIS as ninth graders during school year 2001/2002 were identified and their data written to a data file. For each student, a variety of data variables were extracted from MSIS, including the last know disposition for students who were not still enrolled somewhere in Mississippi at the end of 2004/2005. [N=41,160]

2. All students who entered MSIS as tenth graders during school year 2002/2003, and were not already in the data file, were identified and added to the file. [N=4,384]

3. All students who entered MSIS as eleventh graders during school year 2003/2004, and were not already in the data file, were identified and added to the file. [N=2,344]

4. All students who entered MSIS as twelfth graders during school year 2004/2005, and were not already in the data file, were identified and added to the file. [N=1304]

5. All students coded grade 56 or 58 (self-contained special education) who were the age of typical ninth graders during school year 2001/2002 were identified and their data written to a separate data file. [N=1,310]

6. All students coded grade 56 or 58 (self-contained special education) who were the age of typical tenth graders during school year 2002/2003, and were not already in the data file, were added to the file. [N=452]

7. All students coded grade 56 or 58 (self-contained special education) who were the age of typical eleventh graders during school year 2003/2004, and were not already in the data file, were added to the file. [N=331]

8. All students coded grade 56 or 58 (self-contained special education) who were the age of typical twelfth graders during school year 2004/2005, and were not already in the data file, were added to the file. N=[164]


9. Student records in the regular grade file and the grade 56/58 file were matched by MSIS ID to identify any duplicates. Of the 2,257 grade 56/58 students, 58 students had records in both files. An examination of the duplicate data records revealed that the only difference was in the cohort flag (year of entry into the cohort). Therefore, a procedure was run to eliminate one of the duplicate records and merge the regular grade and grade 56/58 data files. Unduplicated records comprised the full cohort. [N=51,391]

10. A tentative solution was used for student data records that contained both a completion code and a transfer code. This problem reflected data from the earliest year, prior to the addition of certain edit checks in MSIS. The solution was to blank out the transfer code for any student with a completion code (T, O, G, OD).

11. The cohort flag, transfer codes, and completion codes in the data file were used to set unique binary (0,1) variables that could be used for aggregating the student data at the school, district, and state levels.

12. The full cohort data file was matched by MSIS ID to the Month 1 2001/2002 enrollment file to determine which students coded as ninth graders in 2001/2002 were "true beginning" cohort students rather than students that had entered later in the school year. Of the 41,160 students, 38,833 were true cohort students and 2,327 had been added.

13. The full cohort data file was matched to the Month 9 2004/2005 enrollment file to determine the status for students who had no transfer code and no completion code. There were 145 "lost" T1 and T2 transfer students and 2961 students enrolled in some grade.

14. For students in the full cohort data file whose disposition was still unknown after the above steps, summer activity codes were applied. Of the 2,741 students with summer activity codes from 2004 or 2005, 6.6% had completed all requirements except for a passing score on one or more tests needed for graduation, 58.5% had been coded as dropouts, and 35.0% had been coded as transfers or deaths.

15. Ultimately, there were 4,648 students in the full cohort data file whose final disposition was unknown. Since they were probably students who were lost during summer 2002 and summer 2003 (prior to the implementation of the summer activity process), it was decided to apply apportioning constants to the aggregate counts based on the actual percentage of students coded as summer dropouts and transfers during 2004 and 2005.

16. All the full cohort data records were summarized (aggregated) at the state level based on tentative logic outlined in the steps that follow.

17. Binary values (0 or 1) were accumulated across all student records for each of the following variables. Total Full Cohort N-Count, Cohort 0 (true cohort), Cohort 1, Cohort 2, Cohort 3, Cohort 4, T1, T2, T3, T4, T5, T7, T8, Z1, IS1, IS2, IS3, ST4, ST5, ST7, ST8, SZ1, Dropout, Dropsumm, Diploma, Trad, Occu, GED, Cert, ABT, Stillenrl, and Unknown.

18. The unknown student aggregate count was multiplied by 0.585 (58.5%) to yield the statewide number estimated to have been summer dropouts (SD1 through SD21).


19. The unknown student aggregate count was multiplied by 0.350 (35.0%) to yield the statewide number estimated to have been summer transfers/deaths (ST4, ST5, ST7, ST8, SZ1).

20. All transfer and death counts (coded and estimated) were accumulated and subtracted from the total full cohort N-count to yield the denominator for calculating an estimated 4-year dropout rate. [Dropout Denominator Value=42,024]

21. Accumulated counts for students coded as dropouts during the school year, those coded as dropouts during 2004 and 2005 summer activity, those estimated as dropouts during summer 2002 and 2003, and "lost" T1 and T2 transfers to yield an estimate of dropouts over the 4-year period. [Value=11,169]

22. Divided the dropouts count by the denominator to yield the estimated 4-year dropout rate. [Value=26.6%]

23. Subtracted from the dropout denominator value the count of certain “still enrolled” students who would not be expected to complete high school within four years. Since the IEP for students with disabilities under IDEA do not currently contain a specific IEP committee recommendation regarding the number of years those students need to successfully complete high school, a “proxy” measure was used. Cohort students who were still enrolled in self contained special education environments (i.e. MSIS grade code “58”) at the end of 2004/2005 were assumed to be students for whom an IEP committee would recommend extended time for school completion [Value = 827]. In addition, 9 LEP students who were still enrolled at the end of 2004/2005 were considered students who would be expected to take longer than four years to complete high school. Subtracting those 836 students from the dropout denominator (42024) yielded the denominator for calculating estimated 4-year completion and graduation rates. [Completion/Graduation Denominator Value=41,188]

24. Accumulated counts for traditional diploma recipients, occupational diploma recipients, GED recipients, SPED certificate of completion recipients, and students lacking only a passing score on a test required for graduation to yield an estimate of school completers over the 4-year period. [Value=27,589].

25. Divided the completers count by the completion/graduation denominator to yield the estimated 4-year completion rate. [Value=67.0%]

26. Divided the traditional diploma count [Value=25,057] by the completion/graduation denominator to yield the estimated 4-year graduation rate. [Value=60.8%]

27. Applied the estimated dropout rate to the count of still-enrolled students to estimate the percentage of students in the denominator who might be expected to complete school in future years. [Value=5.3%] Note: Some of the students who were still enrolled were self-contained special education students. Therefore, the possible completion rate can be added to the estimated 4-year completion rate to estimate an ultimate completion rate [about 72%], but the estimated ultimate graduation rate would be somewhat lower.

28. Calculated disaggregated dropout, completion, and graduation rates by going back to step #16 and running the analyses separately by gender (Female, Male), ethnicity (Asian, Black, Hispanic, Native American, White) and gender/ethnicity (Black Female, White Female, Black Male, White Male).


Procedure for Calculating District Level Estimates

29. District level cohort statistics were calculated using the procedures described above. Beginning at Step #16, the full cohort data were aggregated to the district level based on the last known district in which the student was enrolled.

30. In step #18, the percentage of summer activity records coded as dropouts by the last known district during the 2004 and 2005 summer activity process was applied to the unknown student aggregate count to yield an estimated number of dropouts.

31. In step #19, the percentage of summer activity records coded as transfers/deaths by the last known district during the 2004 and 2005 summer activity process was applied to the unknown student aggregate count to yield an estimated number of transfers/deaths.

Issues Related to Calculating School Level Estimates (No expected completion time available)

The calculation of school level dropout, completion, and graduation rates will require special school-to-school linking. For example, some students begin in a school containing a 9th grade, but end in a different school containing a 12th grade. Students whose final disposition (transfer, dropout, death) occurs in 9th grade would have a "last known school" differing from the rest of the cohort. Procedures will have to be developed for handling such situations. Differences in Graduation Rate Calculations The large variation in graduation rates produced by the traditional Mississippi formula and the procedures presented in this paper is due, primarily, to differences in the way that the data are managed and the calculations are performed. The differences are outlined in the following table.

Traditional Formula Cohort Tracking Procedure

1. Students included in a cohort Only students who enter grade 9 are entered into a cohort. Therefore, self-contained students with grade code "58" are never included in a cohort and never appear in the denominator for calculating the graduation rate. That reduces the size of the denominator and increases the graduation rate.

All students are eventually included in a cohort. Grade 58 students are included in the denominator for calculating dropout rates, however, they are handled differently for calculating completion and graduation rates for different time spans (4 years, 5 years, 6 years).


Traditional Formula Cohort Tracking Procedure

2. Students leaving during the summer

A procedure was implemented beginning in 2004 to account for students who were present at the end of one school year and did not show up at the beginning of the next school year. However, all graduation rate calculations that included students leaving during earlier summers had those students subtracted from the cohort. That reduced the size of the denominator and resulted in an increased graduation rate.

Although the procedures used in this study included the same students that left during the summers of 2002 and 2003, those students weren't excluded from the cohort. Instead, a procedure was implemented for apportioning the students into estimated transfers and estimated dropouts based on summer activity coding for 2004 and 2005.

3. Students completing school with a GED, an occupational diploma, or a certificate of attendance.

While not counted as graduates, students completing high school with a GED, an occupational diploma, or a certificate of attendance are subtracted from the denominator. That is virtually the same as actually counting them as graduates and it increases the graduation rate.

Students completing high school in any way are counted as school completers. However, only those earning a traditional diploma are counted as graduates. The same denominator is used for calculating the completion rate and the graduation rate.

4. Students retained in a high school grade.

Students retained in a grade 9-12 are subtracted from their original cohort count and added to the cohort count for the next later graduating class. Although this does not change the ultimate graduation rate, it makes it impossible to calculate a true "4-year" graduation rate.

Students remain in the same cohort. Students taking longer than four years to complete will not count in the 4-year graduation rate, but will count in the rate for their graduation year (e.g., the 5-year or 6-year rate). Note: Certain students with disabilities and certain LEP students are excluded from the denominator when calculating certain rates (e.g., the 4-year graduation rate).


APPENDIX E

Graduation Rate Proxies: Utah and Minnesota

Graduation, completers, dropout proxies: Utah

This definition implies a cohort rate. Utah is exercising the option under 200.19(a)(1)(b)

to adjust this definition slightly in order to ensure statewide comparability and reduce error in

measurement by restricting the cohort to grades ten through twelve, since Utah high schools,

which may implement any of three different grade spans, have only these three grades in

common; in fact, nearly half have only these three grades -- Grades 7-12 (21% of high schools),

Grades 9-12 (33%), and Grades 10-12 (46%).

Specifically, the cohort graduation rate is operationalized by Utah, according to the

recommendation of the NCES (U.S. Dept. of Ed., August 2002, p. 3), to simulate the movement

of a class through high school: The number of students who graduated from 12th

grade in the

current year divided by the sum of: (1) these same graduates, and (2) the number of students who

dropped out of 12th

grade in the current year, (3) 11th

grade in the prior year, and (4) 10th

grade in

the year before that.

In order to continue applying official NCES definitions (U.S. Dept. of Education, January

2003, pp. 25, 79-81) -- in distinguishing "graduates" from "other completers", and "dropouts"

from "transfers" [see Note below] -- which have already been incorporated into Utah State Board

of Education rule (R277-419), Utah also lags the rate by one year; thus, the 2003 report includes

the rate for the 2002 cohort.

Regular diploma graduates may include students with disabilities who can be retained as

"seniors" until the age of 22. As long as such students are retained, their cohort year is adjusted,


so that their completion status is included in the calculation of the graduation rate for the

graduating class of the year in which it is finally determined.

Since graduation counts are derived from student level detail, they can be disaggregated

by all of the programmatic and demographic categories identified in NCLB legislation. Dropout

counts, however, are only available in disaggregated form by gender and ethnicity, as required

by the NCES in its Common Core of Data (CCD) surveys, so graduation rates cannot be

calculated for all subgroups.

To address the discrepancy between NCES/CCD and NCLB/AYP in disaggregation

requirements, Utah will begin collecting dropout counts at the student level during the 2004-05

school year, when the Class of 2007 is in 10th grade.

Graduation, completers, dropout proxies: Minnesota

AYP Calculation: Graduation Measurement

The Graduation measurement is one of the Secondary Indicators. The Graduation

computation is based on MARSS enrollment data reported over a five year period; End of Year

data from the previous four years and Fall data from the current year. The graduation measure is

computed for all disaggregated groups, but is only used as an AYP Secondary Indicator measure

for the ‘All’ group where required. The disaggregated group AYP marks are only used to

determine Safe Harbor status when computing Proficiency.

Target

80% (0.8000) or 0.1% (0.0010) improvement over the prior year.

Minimum Cell Size for Measurement

40


Student record selection

Generally, students in grades 8-12 within a school are evaluated to compute the

Graduation measurement. Records with the following MARSS characteristics are excluded from

the Graduation measurement:

• MARSS Status equals 1

• Grade not equal to 08 - 12

Additionally, students with the following State Aid Categories are excluded from the

Graduation measurement:

• 14 – Attending in another State

• 15 – Attending in MN but tuition paid by another State

• 16 – Shared Time

• 17 – Shared Time

• 18 – Shared Time/Tuition

• 25 – Adult Student

• 28 – Resident Attending Non-Public School

Determining the unduplicated count (using the last reported student record in the

MARSS system)

An unduplicated count of student records over multiple years is required. To do so, only

the last record reported for any particular MARSS Number is selected. All other records are

ignored in the Graduation measurement.

To find the last reported record for an individual MARSS Number, the selected records

are evaluated in order by Fiscal Year and then the Status End Date. If two records for the same

MARSS Number end on the same date, the lower Status End code is used. If both have the same


status end code, the record with the higher documentID is used. Only that final record is

included in the computation with its corresponding demographics and status end code (graduate

or dropout code). The computation uses the following MARSS elements when selecting records:

From the student table:

• Submission Code

• Fiscal Year

• Status End Date

• MARSS Number

• Grade

• Status End Code

Once the last record is determined, the status end codes are then evaluated. Only those

records with a graduation or dropout code are retained. All other records are ignored.

Graduate Status End Codes

• 08, 09

Dropout Status End Codes

• 06, 14, 15, 16, 17, 18, 19, 31, 32, 33, 34, 35, 37

General Formula

To compute the entity’s Graduation rate for a particular year, the count of graduates in

that year is divided by the count of graduates plus selected dropouts from that year and previous

three years. Multiply this by 100 to provide the percentage where applicable. The most recent

fiscal year is referred to as Year 4 while the previous fiscal years are referred to as years 3, 2, and

1.


Numerator

The count of students graduating in Year 4.

The computation uses the Fiscal Year in each selected MARSS record to determine if it

should be included in the count. If computing the Graduation Rate for 2005, the count of

graduates reported in 2005 is used as the numerator. Graduates from any other year are not

included.

Denominator

Count of grade 9 students dropping out in year 1

+ Count of grade 10 students dropping out in year 2



+ Count of students graduating in year 4

Only those dropouts reported in certain grades in certain years are included in the

denominator with the graduates. Dropouts or graduates from any other year are not included.

Formula Example

This example is illustrates how the Graduation rate for 2004 is computed.

• MARSS EOY data 2001 through 2004 is evaluated with MARSS Fall data from

2005.

• Only grade 8 – 12 records are evaluated eliminating certain conditions and state

aid categories described above.

• The last record for any particular MARSS number is found by using Fiscal Year

and Status End Date – all other records for that MARSS number are removed.


• Only records where the Status End Code is either a graduate code or a dropout

code are used – all other records are removed.

• The remaining records are evaluated counting the graduates and dropouts within

each entity.

Count of Students Graduating in 2004

Count of grade 9 students dropping out in 2001

+ Count of grade 10 students dropping out in 2002



+ Count of students graduating in 2004


APPENDIX F

External Auditing: Utah

Utah External Auditing

2.1. To assure users of aggregate membership, fall enrollment, and dropout data reported

by districts to the USOE via the Data Clearinghouse that such data are reasonably accurate and

supported by adequate local documentation. To identify sources of errors in recording and

reporting nonfiscal data for the purpose of making continuous improvements to the quality of

student accounting systems.

STANDARDS FOR RECORDING AND REPORTING STUDENT DATA

3.1. Standards for organizing and maintaining a student accounting system and for

reporting pertinent to these agreed upon procedures are found principally in two documents:

3.2. State Board of Education rule R277-419 on “Pupil Accounting” contains the legal

standards and is found at http://www.rules.utah.gov/publicat/code/r277/r277-419.htm.

3.3. The USOE “Data Clearinghouse Update Transactions” file layout contains the

technical standards and is found at

http://www.schools.utah.gov/computerservices/Clearinghouse/Clearinghouse.htm.

3.4. The following parts of the Data Clearinghouse document are the most relevant to the

purposes of these agreed upon procedures:

• S1 Exit Code field — this is the reason why the student left school before the end of

the school year

• S1 School Membership field — this is “regular” membership

• S2 SCRAM Membership field — this is special education membership


• S2 SCRAM Time/Setting field — note the distinction between “self contained” and

“resource” types of special education students

In addition to reading the above mentioned documentation, the independent accountant

should also become familiar with the district’s data management policies and practices,

especially as these impact the district’s production and submission of the Year End and Fall

Clearinghouse files to the USOE.

AGGREGATE MEMBERSHIP:

4.1. Select schools in the district such that each school is included in the sample at least

once every five years; if feasible, a shorter cycle, such as once every three years, would be

preferable.

4.2. Visit each school in the sample.

4.3. Select students in the schools such that the total number of students in the sample is

equal to or greater than the following size according to the enrollment of the district on the

previous October 1:

Enrollment Sample Size

40,000 or greater 70

20,000 to 39,999 50

10,000 to 19,999 40

1,000 to 9,999 30

Less than 1,000 20

Students sampled should include an appropriate representation from each compliance rule

[see 4.4(a) through (g)] with a focus on potential or identified risk of noncompliance, i.e.,


students who enter or exit during the school year and students with significant absences.

4.4. For each student in the sample, study the student’s records and supporting

documentation (particularly for entry and exit dates), identify the student’s aggregate

membership as recorded in the records, and determine whether the following rules from R277-

419 were properly applied in calculating the student’s aggregate membership:

(a) Ten Day Rule [R277-419-1(O)] If the student had 10 consecutive school days

of unexcused absences, the student’s exit date is not later than the school day after the 10th

day of

such absences and, consequently, the student did not generate membership from that day on. An

“unexcused absence” means an absence charged to a student when the student was not physically

present at school at any of the times attendance checks were made during the day in accordance

with R277-419-3(B)(4), and the student’s absence could not be accounted for by evidence of a

legitimate excuse in accordance with the local board of education policy on truancy as defined in

Utah Code 53A-11-101.

(b) Maximum 990 Hours Rule [R277-419-3(A)] This comprises three related

equations: (i) the sum of the student’s regular (K-12) membership and special education self

contained membership does not exceed 180 days; (ii) the sum of the student’s special education

self contained and special education resource membership does not exceed 180 days; and (iii)

the sum of the student’s regular membership and special education resource membership does

not exceed 360 days.

(c) Individualized Education Plan Rule [R277-419-3(B); R277-750]

If the student was enrolled in a special education program, there is an appropriately completed

IEP for the student justifying the service.


(d) Homebound/Hospitalized & Suspension Rule [R277-419-4(C)] If the student was in

membership while homebound, hospitalized or suspended, (i) the student received a minimum of

two hours of instructional contact by a licensed educator each week and

(ii) the circumstances requiring this instructional arrangement are clearly documented.

(e) Part Time Proration Rule [R277-419-4(E)(1)] If the student was enrolled for

only part of the school day and/or only part of the school year, the student’s membership was

prorated according to the number of hours or periods the student was actually enrolled in relation

to the number of hours or periods the student could have been enrolled. As one example, if the

student was in membership 4 periods each day in a 7-period school day for all 180 school days,

the student’s aggregate membership is 103 days; as another example, if the student was in

membership for 7 periods each day in a 7-period day for 103 school days, the student’s

membership is 103 days.

(f) Released Time Rule [R277-419-4(E)(2)] If the student was released for

religious instruction or individual learning activity, (i) there is a Student Education/Occupation

Plan (SEOP) signed by the student, the student’s parent or guardian, and a representative of the

school indicating the use of released time for this purpose is consistent with the plan and (ii)

released time did not exceed the equivalent of one period per school day.

(g) Youth In Custody Rule [R277-419-4(G)]

If the student was enrolled in YIC classes for two to four hours a day (YIC Program Code = ISI

1), regular membership cannot exceed half of total possible regular membership for the student.

If the student was enrolled in YIC classes for more than four hours a day (YIC Program Code =

ISI-2), regular membership must be zero.

4.5. For any student whose reported aggregate membership is based on a violation of one


or more of these rules, recalculate aggregate membership to determine the correct figure.

4.6. Use the appropriate illustrative report at the end of this document as a model for

writing a formal summary of your findings and report complete details of your findings for each

student in the sample in the format specified under the “Aggregate Membership” tab of the APP

C-5:

TRANSFER STUDENT DOCUMENTATION:

6.1. Select secondary schools in the district such that each school is included in the

sample at least once every four years; if feasible, a shorter cycle, such as once every three years,

would be preferable. These may be the same secondary schools which were selected for the

purpose of applying agreed-upon procedures for Fall Enrollment in 5.1.

6.2. Obtain a copy of the Transfer Students List (from the prior Year End upload of the

Clearinghouse), which contains students, organized by school, who were:

(a) enrolled in grades 7 through 12; but

(b) not enrolled on the last day of the school year; and

(c) not classified by the district as either high school completers or dropouts.

6.3. Select a sample of students from this list equal to or greater than 5% of the total

number students on the list; however, the sample size should not be fewer than 10 and need not

exceed 30.

6.4. For each student in the sample, determine whether adequate documentation exists to

support the district's claim that the student was not a dropout. The following constitute adequate

documentation for the possible types of students on the list:

(a) of “transfer” to another school (TD, TO, TP, TS) — an official request for the


student’s records by the receiving school as evidenced in a formal written communication from

the school; or an entry in a log systematically maintained for the purpose showing either (i) the

name of the student, the name of the person and the name and address of the school making the

request, and the date the request was made; or (ii) especially in cases where the parent requests

the student’s record at the time of exiting, an effort by the school to confirm the reenrollment of

the student in another school.

(b) of “transfer” out of the United States (TC, perhaps some instances of TO) — a

statement by a member of the community who has personal knowledge that the student has

moved to another country.

(c) of “transfer” to home schooling (TH) — evidence that the student was issued

a certificate exempting them from public school attendance for the purpose of home schooling in

accordance with Utah Code 53-11-102.

(d) of “withdrawal” (WD) — evidence based on a source external to the school

explaining the student’s situation and justifying withdrawal without continuing provision of

educational services in accordance with R277-419-4(C) as a reasonable response.

(e) of “death” (DE) — a copy of the death certificate or an obituary as published

in a commercial newspaper.

6.5. Use the appropriate illustrative report at the end of this document as a model for

writing a formal summary of your findings and report complete details of your findings for each

student in the sample in the format specified under the “Transfer Student” tab of the APP C-5:

Sample Schedules spreadsheet available from Emily Eyre.

Do not estimate dropout counts for any school or for the district as a whole. Any

adjustments to dropout counts in light of the compliance findings for the purpose of


accountability reporting will be made at the discretion of the USOE.

Documents

Quality Assurance Practices associated with Producing ...programs.ccsso.org/content/pdfs/Examining Graduation Indicators-10252007.pdfQuality Assurance Practices associated with Producing