Using Data to Measure Outcomes

Using Data to Using Data to Measure OutcomesMeasure Outcomes

Student Tracking and Using Data Effectively

Geoff Zimmerman

2004 NTPN – Minneapolis, MN

Impact?

What impact are you having on the students and schools that you are serving?

Basic Question:

Goals for Today

1. Big Picture: Impact/Outcomes

2. Nuts & Bolts: Data Collection

3. Tech Prep in Ohio: Measuring Success

4. Demonstration of Database Application

5. The Basics – Designing Your Own

Are you Data-Driven?

In business and industry, data are king. Information about customers, inventory, sales, rates of return and employee turnover is crucial. Good data determines success or failure.

= Educational Accountability

Why Collect Data?

Measure progress

Make program decisions

Perkins driven – shows accountability

Provides hard data to state executive and legislators when making funding decisions

Accountability: Guaranteeing Results

Flexibility: Local Control for Local Challenges

Research-Based Reforms: Proven Methods with Proven Results

Parental Options: Choices for Parents, Hope for Kids

Four Reform PrinciplesFour Reform Principles

No Child Left Behind

Perkins III (1) Definition Section 202(a)(3) of Perkins III states that a "tech-prep program" means a

program of study that: combines at a minimum 2 years of secondary education (as determined

under State law) with a minimum of 2 years of postsecondary education in a nonduplicative, sequential course of study;

integrates academic, and vocational and technical, instruction, and utilize work-based and worksite learning where appropriate and available;

provides technical preparation in a career field such as engineering technology, applied science, a mechanical, industrial, or practical art or trade, agriculture, health occupations, business, or applied economics;

builds student competence in mathematics, science, reading, writing, communications, economics, and workplace skills through applied, contextual academics, and integrated instruction, in a coherent sequence of courses;

leads to an associate or a baccalaureate degree or a postsecondary certificate in a specific career field; and

leads to placement in appropriate employment or to further education. An allowable tech prep program must meet the terms of this definition.

Are your programs making a difference?

Beginning Questions How many students are enrolled in a Tech Prep program? What is the college transition rate for Tech Prep Students? How

many are persisting in college? What are the demographic characteristics of Tech Prep

students? To what extent are Tech Prep students prepared for college?

Measuring Outcomes

Why do you exist? What conditions need to change for this

target group and why?

Describe your program – what does it do and what does it accomplish?

Source: United Way of Greater St. Louis Guide for Measuring and Reporting Outcomes

Measuring Outcomes

What is your program designed to do? Who is the target of the change? Who

benefits from your services? What information do you need to compile

about participants to ensure delivery? What measurable changes will occur?

(cont.)


Describing what you do

Inputs – resources used to implement the program.

Activities – actions that define what is done, when, where, how, how often, and for whom.

Outputs – the direct products of the program activities. Measure of the amount and volume of work performed to implement the program.


Describing your intended results

Outcomes – the results the program aspires to accomplish; they specify immediate, intermediate, and long-term changes among students served by the program

Indicators – measurable data about benefits for and changes in the target group that is collected at the individual level to show an outcome has been achieved

Impact – what the program aspires to accomplish over a long period of time


Outputs vs. Outcomes

Output = what you do

Outcome = what changes because of what you do

-refining selections as a result of college fair or visitas opposed to x number of students attending the college fair

-completing and submitting college applications following advising sessionas opposed to advising x number of students

-submitting a FAFSA on timeas opposed to giving the FAFSA form to x number of students

-persisting in college as opposed to enrolling in college

Logic Model

Inputs Activities Outputs

Outcomes

Impact

Logic Model (alternative)

Activities

Outcomes

Long-Term Impact

Activities

Outcomes

Activities

Outcomes

Uses of Outcome Information

“A key element in effective use of outcome information is to continually ask why differences occur and then assess whether actions can be taken to improve results.” (p.15)

Hatry, H., Morley, E., Rossman, S., and Wholey, J. (2003). How Federal Programs Use Outcome Information: Opportunities for Federal Managers. IBM Endowment for the Business of Government. Online: http://www.businessofgovernment.org/pdfs/HatryReport.pdf

In other words…

In other words outcome information Tells us what is happening So we can ask why it is happening And fix it if it needs to be fixed Celebrate it and tell everyone if it works

Uses of Outcome Information

How are you using it? Triggering corrective action Identifying and encouraging “best practices” Motivating Planning and Budgeting Reporting to Key leaders

*Hatry, et.al, How Federal Programs Use Outcome Information: Opportunities for Federal Managers

Quality is not just a chart, or a standard,or a specification—it’s a state of mind, a commitment, a responsibility, a spirit.It’s a way of doing, being, living.

—Don Galer

Data Collection

“Insanity: the belief that one can get different results by doing the same thing.”

-Albert Einstein

How to Collect?

•Student enrollment form

•Student list report of file from schools

•Surveys – students, parents, teachers, etc.

•Focus groups, interviews

•Program site review process

U.S. Census Bureau, State Departments of Education, National Center for Education Statistics

Primary Data Sources

Secondary Data Sources

What to Collect

Determine what data to collect Demographics Participation in activities College aspirations Academic preparation Persistence in college Remediation in college Feelings about the program Perceptions of education and work

2 year A.S. 1-2 year Tech 4 year degree don't know

How much additional education do you think you will need?

0

10

20

30

40

50

60F

req

uen

cy

How much additional education do you think you will need?

Survey Results

Source: Greater Cincinnati Tech Prep Consortium, May 2004

Very Helpful Somewhat Helpful Not Very Helpful Not at all helpful

How much has this program helped you in feeling your time in school is more worthwhile?

0

10

20

30

40

50

60F

req

uen

cy

How much has this program helped you in feeling your time in school is more worthwhile?

Survey Results


Very Helpful Somewhat helpful Not Very Helpful Not at all Helpful

How much has this program helped you in improving your grades?

0

10

20

30

40

50F

req

uen

cy

How much has this program helped you in improving your grades?

Survey Results


Very Helpful Somewhat Helpful Not Very Helpful Not at all Helpful

How much has this program helped you in knowing what college associate degree programs are available in your

program area?

0

10

20

30

40

50F

req

uen

cy

How much has this program helped you in knowing what college associate degree programs are available in your

program area?

Survey Results


Very Helpful Somewhat Helpful Not Very Helpful Not at all Helpful?

How much has this program helped you in learning more about your specific technology area?

0

10

20

30

40

50

60F

req

uen

cy

How much has this program helped you in learning more about your specific technology area?

Survey Results


Performance Measure Question Sample Responses

Articulated credits Did you have the opportunity to earn articulated credits while in

high school?

-We did not earn any.-I had 12 but the four-year college would only use them as electives.-I had AP credits in English and math, but none in Tech Prep.

Rigorous academic curriculum Did your academic curriculum prepare you for college?

-Some courses were not challenging. -I did an independent study in math as my class was behind me. -Locking you into the pathway allows for no deviation based on your ability.

Market share How can we attract more students into Tech Prep programs?

- Use Tech Prep students as program ambassadors to inform other students and parents about program oportunitieis-Start earlier, in the 7th and 8th grades.

Focus Group Results

Source: Greater Cincinnati Tech Prep Consortium, September 2003

Percent of population 25 years and over 100.0 100.0 100.0

Less than 5th grade 0.9 0.0 1.5

5th to 8th grade 6.3 6.3 6.3

9th to 12th grade, no diploma 21.1 21.3 20.9

High school graduate (incl. equivalency) 42.8 40.8 44.4

Some college credit, less than 1 year 3.5 3.9 3.1

1 or more years of college, no degree 10.9 8.9 12.5

Associate degree 3.4 3.1 3.7

Bachelor's degree 7.7 10.5 5.5

Master's degree 1.5 3.4 0.0

Professional degree 1.8 1.3 2.1

Doctorate degree 0.3 0.6 0.0

Percent high school graduate or higher 71.8 72.4 71.2

Percent bachelor's degree or higher 11.3 15.7 7.6

Educational Attainment, Bethel, Ohio

QT-P20. Educational Attainment by Sex: 2000Data Set: Census 2000 Summary File 3 (SF 3) - Sample Data http://factfinder.census.gov

Tech Prep in Ohio

Ohio College Tech Prep combines college preparatory academics and advanced career-technical education into a seamless program from high school to college to high-skill, high-wage careers.

Innovative Local Delivery Models

• CTP programs on college campuses• CTP programs in industry setting• Industry mentorships• Dual Enrollment and Transcripted college credit options

• College faculty team-teaching with secondary instructors

College TP goals and Benchmarks

Ohio Tech Prep Goals (1998)1. Expand CTP enrollment to meet

Ohio’s critical workforce and economic needs

2. Provide a CTP system that reflects the requirements for success in college and in high-wage, high-skill technical careers.

3. Maximize student options through the integration of high school, associate degree and baccalaureate degree pathways to achieve a seamless, non-duplicative system.

4. Engage the active support and involvement of all College Tech Prep stakeholders.

Performance BenchmarksExpand enrollment: 15% of all 11th and 12th graders

Enter college remediation free: 10% higher rate than traditional college population

Transition to college: 66%

Under-represented groups: 25%➲

➲

➲

➲

3. Building the Secondary Pipeline

3109

1458

9532

11419

6740

4995

607211

11530

12882

0 0 57 212

513 835 1288

1673

35305176

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

93-94 94-95 95-96 96-97 97-98 98-99 99-00 '00-01 '01-02 '02-03

High School

College

Annual Enrollment

Student Tracking and Analysis though HEI

Ohio Board of Regents Higher Education Information (HEI)

Student Tracking System

How is this done? Ohio has a comprehensive postsecondary

data collection system in place - Higher Education Information System (HEI)

Available to all public two & four year institutions.

Limited access to private colleges

How HEI Works

Tracks students by SSN Follows student mobility in higher

education. Tracks by campus type

State Community College & Community Colleges Technical Colleges University Branch or Main Campuses

Reports on remediation levels and persistence

HEI Enrollment Data SubmissionsTech Prep Consortium Tracking (TC) File

Revised May 1, 2002File Description:The Tech Prep Consortium Tracking (TC) File contains one record for each student enrolled in a Tech Prep consortium.

Submission Schedule:The TC file can be submitted anytime throughout the year.

Field Names Field Attributes and Procedures Data Format

Consortium Code Enter the consortium code from the consortium code verification table.

4 Alphabetic Characters, Columns 1-4

Student Identifier Enter the federally assigned Social Security Number (SSN) whenever possible. If the SSN is unavailable, enter another identifier which uniquely relates to this student. If the Student Identifier is not a SSN, it can be entered in the TC file, but the student’s enrollments cannot be tracked.

9 Alphanumeric Characters,

Columns 5-13

Institution Assigned Identifier Switch

Enter “Y” if the Student Identifier is not a federally assigned Social Security Number (SSN). Otherwise enter “N”.

1 Alphabetic Character, Column 14

Data Fields:

Field Names Field Attributes and Procedures Data Format

Program Code Enter the code of the program of study in which the student was enrolled from the Program Code verification table.

6 Numeric Characters, Columns 15-20

Career Center Code

Enter the code of the Career Center or Tech Center from which the student enrolled in Tech Prep from the IRN verification table. If there is no Career Center or Tech Center associated with this student, or to report the association of a compact or collaborative, enter 000000.


High School Code Enter the code of the high school from which the student graduated or will graduate from, from the IRN verification table.


Site of Delivery Enter the code of the site of delivery from which the student received instruction from the IRN verification table. If the site of delivery was a hospital or business, enter 000000. If the site of delivery was a college or university in Ohio, enter a code from the IPEDS/Institutional code verification table.


Graduation Year Enter the year this student graduated, or will graduate from high school.


Delete Switch Enter “Y” if the record is to be deleted from the database. Otherwise, enter “N”.

1 Alphabetic Character, Column 43

Data Fields (continued):

HEI Enrollment Data SubmissionsTech Prep Consortium Tracking (TC) File

Link to HEI – 16 digit code Consortium identifier Student identifier (ss#) Program Code Career Center Code (IRN) Delivery Site Code (IRN) High School Code (IRN) Graduation Year

(CNCI999990153Y11T0110514820065930065932004N)

Internal Data Tracking System

Student Enrollment Forms (paper filing) Microsoft Excel, Lotus Microsoft Access College databases for transition enrollment,

remediation

What Data are Gathered

Consulted and sought input from consortia directors on four major topics. Tech Prep Completion Remediation Transition and Persistence Underrepresented Populations

Improved Remediation Rates

Tech Prep vs. Non-Tech Prep Remediation Rates(18-19 year olds)

45 %

0

400

800

1200

1600

2000

Tech Prep Students

Non-Remedial

RemedialStudents

57.4 %

0

5000

10000

15000

20000

25000

Non-Tech Prep Students

Population are 18-19 year old freshman enrolled at community / technical colleges or university branch campuses.1604 students total 21337 students total

Remediation

Number and Percent Taking Remedial Coursework in AY 2002 (data as of 8/26/03)

Student type Enrollees # math % math # eng % eng # math or eng

% math or eng

# math and eng

% math and eng

Non tech prep 14,295 7,458 52% 5,123 36% 8,894 62% 3,687 26%

Tech prep 2,251 876 39% 644 29% 1,102 49% 418 19%

Number and Percent Taking Remedial Coursework in AY 2002 (data as of 8/26/03)

Student type Enrollees # math % math # eng % eng # math or eng

% math or eng

# math and eng

% math and eng

Non tech prep 41,701 9,118 22% 6,199 15% 12,028 27% 3,289 8%

Tech prep 1,546 371 24% 243 17% 513 33% 121 8%

Two – Year Campuses 2003-04

Four - Year Campuses 2003-04

(cont.)

PersistenceFirst-time, Full-time, degree-seeking Undergraduate Students age 19 and under in Autumn Term. Number and percent persisting to Autumn of next year, by type (tech prep vs. non tech prep). Persisting includes enrollment at an Ohio public institution, or private institution if full-time and receiving a Student Choice Grant, or proprietary institution if receiving a Workforce Development Grant. Data as of 8/27/2003.

* First Time Full Time Degree Seeking Undergraduates

Year Campus Type Tech Prep * FTFTDSUG # persisting % persisting

2000 University Branch N 5,709 4,223 74.0%

2000 University Branch Y 108 83 76.9%

2000 University Main Campus N 31,348 27,109 86.5%

2000 University Main Campus Y 206 156 75.7%

2001 Community College N 2,960 2,000 67.6%

2001 Community College Y 135 96 71.1%

2001 State Comm. College N 2,774 1,741 62.8%

2001 State Comm. College Y 122 65 53.3%

2001 Technical College N 1,654 1,049 63.4%

2001 Technical College Y 101 64 63.4%

2001 University Branch N 5,648 4,258 75.4%

2001 University Branch Y 120 86 71.7%

2001 University Main Campus N 32,160 27,720 86.2%

2001 University Main Campus Y 412 337 81.8%

* First Time Full Time Degree Seeking Undergraduates

New Data to be Collected . .

Graduate’s Age upon Graduation

Non tech prep students <24 years

Tech prep students <24 years

Non tech prep students 24+ years

Tech prep students 24+ years

Discipline Subject # of Graduates

4th Year Salary

# of graduates

4th year Salary

# or graduates

4th year Salary

# of graduates

4th year Salary

Education

(3)

Sports & Recreation

Teaching

Engineering

(4)

Engineering Technology

Associate Degree Graduates: Fourth Year Following Graduation by DisciplineAnnualized Full-Time* Salaries (in thousands)

• Post-program Placement

Demo

Consortium office database for tracking students, programs, articulation agreements, and contacts

Contact information:

Geoff ZimmermanGreater Cincinnati Tech Prep ConsortiumUniversity of [email protected]

Documents

Using Data to Measure Outcomes