General Supervision: Highlights on Monitoring

and the 09-02 Memo

Western Regional Resource Center APR Clinic 2010 • November 1-3, 2010 • San Francisco,

California

What is General Supervision?• A requirement:

IDEA 34 CFR §300.600The “State” must monitor the implementation of this part, enforce this part… and annually report on performance under this part.

• A way to organize what you do

• An important component of OSEP verification visits and determinations

• A method of tracking and encouraging continuous improvement

Components of General Supervision Helpful Guidance From TA Providers(NCSEAM)

State Performance

Plan

Policies, Procedures & Effective

Implementation

Data on Processes

and Results

Targeted Technical

Assistance & Professional

Development

Effective Dispute

Resolution

Integrated Monitoring Activities

Improvement, Correction,

Incentives & Sanctions

Fiscal Management

According to OSEP

Five Critical Elements of General Supervision:

1. A system to identify noncompliance in a timely manner using its different components

2. A system to ensure correction of identified noncompliance in a timely manner

3. Procedures and practices to implement the dispute resolution requirements of IDEA

Western Regional Resource Center APR Clinic 2010

According to OSEP

Five Critical Elements of General Supervision:

4. Procedures and practices to improve educational results and functional outcomes for all children with disabilities

5. Procedures and practices to implement other requirements, i.e., fiscal requirements, private schools, NIMAS, assessments, etc.

Building Effective General Supervision

General Supervision Foundation

Step 1 – Identifying an Issue

• What components are used to identify noncompliance?

• Which issues to look for: How few are too few? How many is too many?

• Determining and prioritizing what to look for.

Components (examples):On-site Activities

• IEP/Record Reviews• Interviews (Families/Providers/Teachers)• Others ?

Off-site Activities• Database (includes SPP/APR data collections and

analysis)• Self Assessment• Desk Audit• Surveys (Families/Providers/Teachers)• Contracts• Dispute Resolution (formal and informal)• Local reporting• Others?

99

Which Issues to Look For

• SPP/APR Indicators

• Related Requirements

• Your Indicators

1010

Using SPP/APR to Identify Issues

• Compliance and performance

• Self-identified issues (discussion of progress and slippage, improvement activities)

• OSEP-identified issues (response table)

1111

• Coordination of APR data collection and monitoring data

• Database data vs. census data vs. monitoring data

Monitoring Data and the SPP/APR

1212

• If the SEA/LA receives data through its database that show noncompliance, the SEA/LA must: Make a finding, AND Require correction as soon as possible,

and in no case later than one year after the SEA/LA’s notification

From OSEP on Databases

1313

• SEA/LA may identify one or more points in time during SPP/APR reporting period when it will review compliance data from database and identify noncompliance

• In making compliance decisions, SEA/LA should then review all data received since the last time SEA/LA examined data from database and made compliance decisions. SEA/LA may either examine: All data in database, OR

Statewide representative sample

From OSEP on Databases

1414

From OSEP on Identifying Noncompliance• Findings must be made based on data

collected through any method that demonstrates noncompliance (e.g., when the level of compliance is less than 100%)

• “Substantial compliance” (e.g., 95% compliance) or other “thresholds” (e.g., 3 of 4 children had a timely transition conference) do not apply to identification or correction of noncompliance

1515

Option 1

Make a finding of noncompliance.

Option 2

Verify whether data demonstrate noncompliance, and then issue finding if data do demonstrate noncompliance.

Option 3

Verify LEA has corrected noncompliance before State issues written findings of noncompliance, in which case State not required to issue written finding of noncompliance.

Slide prepared by OSEP

Scenario - Identification

9-21-10

* In this scenario, written notification is issued three months from discovery. Written notification may be issued at any time up to three months from discovery.

Scenario A is an example of how States can use data from the entire FFY 2008 to identify noncompliance in FFY 2009. As a result, a period of two years occurs between the fiscal year when data are reported on indicator performance in the APR (FFY 2008) and the fiscal year when data on correction of findings are reported in the APR (FFY 2010).

Scenario A – Using Data from a Complete Fiscal Year to Identify Noncompliance

Identification of Noncompliance in FFY 2009

August 15, 2009: Review and analyze all FFY 2008 (July 1, 2008 thru June 30, 2009) data to determine compliance (identify finding(s) of noncompliance).

Do I make a finding?

Maybe

OPTIONAL September 15, 2009: Review most current data from FFY 2009 (e.g., 30 days of data) to determine whether the LEA or EIS program corrected the noncompliance before the State issues written findings of noncompliance, in which case the State would not be required to make a finding of noncompliance. Document verification of correction of noncompliance.

OPTIONAL September 15, 2009: Verify whether the data demonstrate noncompliance (e.g., check for data accuracy or if data reflects data entry errors).

Is there still a need to make a finding?

November 15, 2009: Provide written notification of finding(s) of noncompliance for FFY 2009 (based on FFY 2008 data).*

Yes No

No Yes

1717

Scenario #1• On October 9, the Lead Agency monitored

Oleander Infant/Toddler Daycare Program and found that 2 of 18 files did not have justification for not being in natural environments.

• On October 23rd the program faxed the IFSP pages to the LA, with justification statements based on the Medical Team at UNC pediatric services, that two children with active foot/mouth disease were in pediatric medical rehab class for 3 months, after which they would be moved to the regular toddler class.

• The Lead Agency issued the monitoring report on November 1st.

1818

Scenario #1

• Should the Lead agency issue a finding of noncompliance for the two files that did not have adequate natural environment justifications?

• What options does the Lead Agency have prior to issuing the monitoring report?

1919

Prioritizing Issues – Monitoring with a Focus

How can you make this process manageable and still get the data you need?

Putting it TogetherNEW JERSEY GENERAL SUPERVISION SYSTEM

METHODS FOR DETERMINING COMPLIANCE AND PERFORMANCE

Part C Compliance Policies and Procedures

Data Desk Audit

Self-Assessment Family Surveys Procedural Safeguards

1. Infants and toddlers with IFSPs receive the early intervention services on their IFSPs in a timely manner.

X X X X

2. Eligible infants and toddlers with IFSPs have an evaluation and assessment and an initial IFSP meeting conducted within Part C’s 45-day timeline.

X X X

3. Children exiting Part C receive timely transition planning to support the child’s transition to preschool and other appropriate community services by their third birthday including:

A. IFSPs with transition steps and services

B. Notification to LEA, if child potentially eligible for Part B: and

C. Transition conference, if child potentially eligible for Part B.

X

X

X

X

X

X

X

X

X

X

X

X

21

Jan

Aug

Dec

Nov Mar

Feb

Oct Apr

July

Sept

June

May

Monthly Activities

Resource Specialist Reports/Meetings

IEP Count Data Report

Evaluations Data Report

Exit Data Report

Personnel Data Report

On-site school visits (Sept - March

School Visits (5 in February/March)

Examine B13, B11, B12 data to determine noncompliance

Issue findings

Enter CAPs in tracking log, review CAP progress, verify correction,

and determine TA needs

Select schools for onsite visits in next school year

Report APR performance to the public

Set agenda for APR indicator drafts meeting in Fall

Provide APR data to indicator teams

Schedule training and share resources/tools

Hold initial meeting of APR indicator teams

Review SPP/APR progress on improvement activities

School Visits (5 in October/November)

Examine B13, B11, B12 data to determine noncompliance

Issue findings

Submit 618 Child Count, LRE, and Assessment DataSubmit 618 Personnel, Exiting, Discipline, Disp.Res.

Submit SPP/APR to OSEP

Enter CAPs in tracking log, review CAP progress, verify correction, and determine TA needs

Organizing the Identification Process - The Wheel

Steps 2 & 3 – Investigating Issues

• Step 2 – Determine the Extent/Level of the Issue

• Step 3 – Determine the Cause of the Issue

22

2323

Factors to Consider• What is the level of

compliance/performance?

Look at percentages:All (100%), mostly compliant (95%+),moderately (85-94%), somewhat (76-

84%), limited compliance (75% or less)

Look at number of instances of compliance in proportion to the N (total): 4 out of 5 -vs.- 40 out of 50

2424

• Where/with whom is the problem happening?

One or more providers/teachers One or more local

programs/schools Statewide

Factors to Consider

2525

Step 2 – Determine the Level/Extent of the Issue

2626

Factors to Consider

• Historical/Trend Data Open CAP on the same issue Previous completed CAPS on the same

issue Repeat offense – not really fixing the problem

although findings are corrected or not issued Trend data – do data show clear progress

or slippage in this area? More applicable when looking at smaller time

periods (quarters)

2727

Factors to Consider

• Other Considerations Demonstrated Ability to Correct

(previously identified noncompliance corrected within timelines)

Exceptional Circumstances Number of findings of noncompliance

(multiple noncompliance issues vs. one)

28

Step 3 – Determine Cause of Issue

• Root Cause Analysis Need for Improvement Need for Correction

• Policies• Procedures• Practices

• Keep in mind what the resolution might be - based on the cause

2929

Root Cause Analysis

Infrastructure (funding, facilities)

Data Personnel

(shortages, turnover)

TA/Training

Supervision

Provider/ Teacher

Practices

Policies

Procedures

What is/are the contributing factor(s)/root cause(s) of the issue?

3030

Use decisions on the level/extent and the root cause of issues, including whether there is noncompliance, in order to determine: At what level resolution needs to happen Who needs to be responsible What actions should be required What data will be used to verify correction

Using Analyses from Steps 2 and 3

Example

3232

Scenario #2• The Lead Agency monitored the Mothers

& Babies Program during an onsite visit. The findings report identified noncompliance based on the following:• 20 out of 50 children did not receive timely

services• 2 out of 40 children did not have an IFSP

meeting in 45 days• 5 out of 45 children did not receive written

prior notice at the appropriate junctures

3333

Scenario #2• What actions might the Lead Agency

require of 2 the Mothers & Babies Program to ensure correction the noncompliance related to each of the following:• timely services (20 of 50)?• IFSP meeting within 45 days (2 of 40)?• written prior notice (5 of 45)?

3434

Step 4 - Assign Accountability for the Issue and its Resolution

• The resolution should be based on the issue and the analyses conducted (extent/level and root cause)

• For improvement issues (not a compliance issue) Improvement plan

• For noncompliance Finding must be issued Corrective action

3535

For Improvement Issues• Use decisions made about the level/extent

and the root cause of the issue to: Determine areas in need of improvement Explore relationship to SPP indicators Determine resources needed (staff, TA

providers, best practice) Develop improvement strategies

36

Written Notification of Finding• Generally, OSEP expects written notification

to be issued less than three months from discovery and should include: SEA’s/LA’s conclusion that LEA/program is not in

compliance Citation of relevant regulatory or statutory

requirement Description of quantitative and/or qualitative data

supporting SEA/LA’s conclusion, AND Statement requiring correction as soon as

possible, but in no case later than one year from notification

37

Counting and Reporting Findings• SEA/LAs may choose how they will count

and report their monitoring findings: Group individual instances in a program

involving the same legal requirement together as one finding, AND/OR

Report each of the individual instances of noncompliance as a separate finding

• Exception: each finding identified through a State complaint or a due process hearing must be counted as a separate finding

38

Required Actions to Correct Noncompliance• Corrective action

What actions need to be taken to correct the noncompliance (based on analyses)? Submit data to demonstrate correction Corrective Action Plan

How will we know they worked?

• Identify which data will be used to verify correction of the noncompliance

39

Steps 5 & 6 - Verify Correction and Follow-up

• Step 5 – Verify Resolution of the Issue OSEP Definition of Verifying Correction

• Step 6 – Follow up on Resolution Continuous Improvement

40

Improvement Issues (not compliance issues)

Several tools are available to assist states in:

• Analyzing indicator data and other performance variables at SEA/LA and school levels

• Designing effective improvement strategies• Evaluating improvement activities

www.rrfcnetwork.org

41

Step 5 – Ensure and Verify Resolution of the Issue

• For improvement issues (not compliance issues), this may be over a period of several years

• For compliance issues, this is clearly defined by OSEP

42

Compliance Issues (defined by OSEP)

• Verifying resolution of compliance issues is clearly defined by OSEP. Two main documents explain and clarify the process states are to use to correct and verify correction of noncompliance: OSEP Memo 09-02, October 17, 2008 F.A.Q. on Identification and Correction of

Noncompliance, September 3, 2008

43

From OSEP on Timeline for Verifying Correction

• The timeline for when correction must be verified (as soon as possible but in no case later than one year) begins on the date on which the SEA/LA notifies the LEA/program, in writing, of its finding of noncompliance

44

From OSEP on Timely Correction

• For an SEA/LA to report that noncompliance has been corrected it must first: Account for ALL noncompliance identified by

SEA/LA Determine:

• in which LEAs/programs noncompliance occurred • the level of noncompliance in each, AND• the root cause(s) of noncompliance

If needed, require change in the LEA’s/program’s• Policies• Procedures, AND/OR• Practices

45

• In order to demonstrate that previously identified noncompliance has been corrected, an SEA/LA must: Prong 1 - Account for the correction of all

child-specific instances of noncompliance AND Prong 2 - Determine whether each LEA or

Program with identified noncompliance is correctly implementing the specific regulatory requirements (achieved 100% compliance)

From OSEP on Timely Correction

46

• Both prongs apply to correction of all findings of noncompliance, and noncompliance reported in APRs, whether there is a high level of compliance (but below 100%) or a low level of compliance

• States cannot use a threshold of less than 100% to conclude that the LEA/program has corrected noncompliance

From OSEP on Timely Correction

• For child-specific noncompliance that is not a timeline requirement, SEA/LA must ensure that LEA/program corrected noncompliance in each individual case, unless: The requirement no longer applies

OR The child is no longer within the jurisdiction of

LEA/program

47

From OSEP on Correction of Child-Specific Noncompliance

48

• For child-specific noncompliance with a timeline requirement, the SEA/LA must ensure that the service/evaluation/etc. was provided, although late, unless: The requirement no longer applies OR The child is no longer within the jurisdiction of

LEA/program

From OSEP on Correction of Child-Specific Noncompliance

49

• To ensure correction of child-specific noncompliance regardless of whether or not it is a timeline requirement… The SEA/LA could review or require a local

agency to review all or a sample of the records of affected children to verify correction

From OSEP on Correction of Child-Specific Noncompliance

50

• For an SEA/LA to report LEAs/programs are implementing the specific regulatory requirements, in addition to the correction explained for child specific noncompliance, the SEA/LA must: Based on its review of updated data, and within

one year of notifying the program of noncompliance, determine if the LEA/program is in compliance

From OSEP on Timely Correction

51

• For an SEA/LA to report LEAs/programs are implementing the specific regulatory requirements: Must be based on NEW (updated) data Correction = 100%

• Hints: May happen very quickly Period of time (at 100%) for verification should

depend on the level of noncompliance and the cause of the noncompliance

How correction will be verified should be determined before finding is made

Verification of Correction of Noncompliance

52

FFY 2009Monitoring

Indicator C8/B12August 15th 2009:

1000 children:

50

100

750

80

20

12 children(94%)

2 children(71%)

Monitoring Results FFY 2009: By November 12 - C8/B12 - 93%, Three Findings

7

9

200

7

3

Noncompliance Corrected & Verified

By November 11, 2010

1. Child Specific 5

210

0

New Children Transitioning from C to B

By February 15th, 2010

2 children(33%)

2. Program

1. Child Specific2. Program

1. Child Specific2. Program

226 16

Quiz

A

B

C

D

E

0 children(100%)

0 children(100%)

53

From OSEP on Verifying Correction

• If 100% is not obtained when reviewing updated data to verify correction: a new finding is NOT issued the original finding remains “open”

• The child-specific noncompliance identified from this review of updated data must be corrected AND the state must review further updated data until the LEA/program achieves 100%

54

Notification of Verifying Correction

After correction has been verified:

• Notify (in writing) the accountable party (LEA/program/etc.) that correction has been verified and the finding of noncompliance is closed out. Notification may include: Corrective actions taken to correct

noncompliance Data used to verify correction

Correction of each instance Updated data demonstrating 100% compliance

Whether the noncompliance was corrected within 12 months of issuing the finding

55

Documentation of Correction and Verification of Correction

What documentation could you use as evidence of how you verify correction?

• Notification of findings• Corrective action plans• Notification of verification of correction• Procedures for verification of correction

How data for verification is collected and verified If samples are used, how they are representative

56

Step 6 - Follow Up on Resolution of the Issue

• Incentives for correction/improvement

• Sanctions/enforcement actions for uncorrected noncompliance Easier to use if set up in advance

Scenario–Correction & Reporting

No Yes

Yes

February 15, 2010; (repeat on May 15, 2010 and August 15, 2010 if needed) Review the 3 most recent months of data** to evaluate the effectiveness of corrective actions and improvement strategies; and/or, to verify timely correction (100%) of child specific noncompliance including documentation that the LEAs/EIS programs are correctly implementing requirements.

Is it less than one year from when written notification of findings was issued?

Correction of Noncompliance November 15, 2010 (no later than one year from date of written notification of findings issued in FFY 2009): Review and analyze current data to verify timely correction (100%) for any remaining finding(s) of noncompliance that were issued on November 15, 2009.

Scenario A – Cont’d

Has correction of noncompliance (100%) been verified including: correction of individual child-specific noncompliance; and correct implementation of specific regulatory requirement (updated data)?

No

Conduct additional program-specific enforcement activities with the LEA/EIS program to ensure correction (100%) as soon as possible.

Document verification of correction of noncompliance and notify LEA(s)/EIS program(s).

** A state may chose to review more or less data to verify correction depending on the level and extent of the noncompliance.

APR Reporting February 1, 2010: Report FFY 2008 APR indicator data in the FFY 2008 APR due Feb 1, 2010.

February 1, 2012: Report on the correction of FFY 2009 findings of noncompliance in individual indicators and in C9 or B15 (based on FFY 2008 data) in the FFY 2010 APR due February 1, 2012.

February 1. 2011: Report findings of noncompliance based on FFY 2008 data in each indicator for FFY 2009 APR due February 1, 2011.

Western Regional Resource Center APR Clinic 2010

Western Regional Resource Center

APR Clinic 2010

November 1-3, 2010 • San Francisco, California