THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY

THE QUANTITATIVE/QUALITATIVE IMPACT

STUDYBENEFITS AND CHALLENGES

Dr. Gloria Pryor JamesDr. Brenda R. Shaw

Mr. Samuel T. Rhoades, J. D., President EmeritusNational Association of HBCU Title III Administrators, Inc.

Technical Assistance WorkshopJune 23, 2011

WHAT IS AN IMPACT STUDY?

The Program Impact Study is one of the most important processes that identifies over time the impact of the Institution’s programmatic outputs and inputs and the impact that these components had on the target population and the University, as a whole.

SOME BENEFITS OF THE QUANTITATIVE/QUALITATIVE IMPACT STUDY

Specific outcomes can be compared to resources expended; and successful programs can be highlighted;

SOME BENEFITS (CONTINUED)

Programs that have not accomplished their intended outcomes can be identified;

The extent to which the planned outcomes were met can be documented;


The adequacy of the program’s outcomes as related to the problem that the program was developed and implemented to address can be documented;


Strengths and weaknesses of each activity can be identified that affected the Effectiveness and Adequacy of the programmatic inputs ($$$, human resources, space, etc.) in meeting the activity’s planned outcomes.


The institution has the ability to determine the impact of the Title III funds awarded to various discrete activities and programs.


• The process determines not only the success or failure of the program, but also provides explanations for the actual outcomes.

PRIMARY BENEFIT

A COMPREHENSIVE picture of the overall impact of the programs being evaluated is the major outcome.

WHY CONDUCT EVALUATIONS AND IMPACT STUDIES

As a result of the MANDATE of funding agencies and institutions to demonstrate programmatic accountability, one of the services requested most by institutions of higher education is program ASSESSMENT/EVALUATION.

ABOUT NORTH CAROLINA CENTRAL UNIVERSITY

•NCCU, chartered in 1909, opened its doors to students in 1910 as the National Religious Training School and Chautauqua.


• North Carolina Central University is a part of the University of North Carolina System, made up of 16 individual campuses (17 when we count the School of the Arts) and headed by a single president and governed by the University of North Carolina Board of Governors.


• North Carolina Central University is a comprehensive university offering bachelor’s degrees in more than 100 disciplines, master’s degrees in more than 40 programs and three professional degrees in law.


• Enrollment at North Carolina Central University increased from 5,753 in Fall 2001 to 8, 619 for Fall 2006, an increase over the six year period of 33%. We currently enroll 8,500 students (2010-2011).

• Major emphasis for the current planning period

(2007-2012) is “Enhancing Retention, Graduation, and Placement Rates through Increased Student Learning.”


Our goals: • to provide programs, activities, and services

that will assist students in becoming successful during their tenure at the University so that they persist and graduate and are successful in the career markets of the future;

• to put in place those systems designed to make the matriculation more seamless and easier to negotiate;


• to provide the training and upgrading of the skills of our faculty to work with those students whom we enroll that have been underserved and are from diverse backgrounds; and

• to promote a more efficient, effective, and accountable environment as relates to student learning and student success.

INCEPTION OF THE IMPACT STUDY: INTERNAL MOTIVATION

• Impetus for conducting an Impact Study – Chancellor Charlie Nelms;

• A directive he gave me as Title III Director, which I passed on to the Activity Coordinators in our planning meeting;

• His absolute need to have concrete documentation of the impact that our Title III dollars had had over the years.

CONTRACT OUT OR CONDUCT INTERNALLY

North Carolina Central University (NCCU) contracted with Associates for Institutional Development, Inc. (AID, Inc.) to conduct an Impact Study of select Title III activities encompassing the grant years 2004-2009 to document the extent to which each activity had accomplished the outcomes for which the activity was funded.

THE KEY IS COLLABORATIONTHE INSTITUTION

MUST:•PROVIDE ACCESS

TO ALL RECORDS

PERTINENT TO EACH ACTIVITY

THE RESEARCH TEAM MUST:

WORK CLOSELY WITH THE TITLE

III STAFF TO COLLECT THE

DATA

ASSESSMENT GOALS OF THE IMPACT STUDY

A data collection process MUST be implemented that includes:1. Onsite visits; 2. Interviews with key staff; 3. Participant self-assessments;


4. Interviews with faculty, students, and external stakeholders;

5. The administration of surveys and questionnaires, as needed.


To guarantee objectivity, the process relies upon multiple sources of data encompassing multiple points of view. Finally, to be adaptable, the assessment’s design must meet the assessment needs of the institution.

INFORMATION NEEDED FOR THE STUDY

• PROGRAM PLANS (RATIONALE, TARGET AUDIENCE, BASELINE DATA;

• MEASURABLE OBJECTIVES;• MEASURABLE AND TIME-SPECIFIC

PERFORMANCE INDICATORS; • IMPLEMENTATION STRATEGIES IN

CHRONOLOGICAL ORDER


• DOCUMENTATION ON ACTUAL PROGRAMMATIC OUTCOMES (DESCRIPTIVE AND STATISTICAL)

• EXTERNAL EVALUATION REPORTS

• PROGRESS REPORTS (MONTHLY OR QUARTERLY)


• PROGRAMMATIC OUTCOMES

(QUANTIFIABLE ANDDESCRIPTIVE)


• ANNUAL REPORTS • RESULTS OF SURVEYS AND

QUESTIONNAIRES• FUNDS AWARDED EACH

ACTIVITY DURING THE YEARS INCLUDED IN THE STUDY


• ACTIVITY BUDGET BURN RATES

• ANEDOTAL INFORMATION THAT WILL ADD VALUE TO THE OUTCOMES

CRITICAL CHALLENGES TO CONDUCTING AN IMPACT STUDY

• OBJECTIVES THAT ARE NOT MEASURABLE• TOO MANY TASKS MASQUERADING AS

OBJECTIVES• PERFORMANCE INDICATORS THAT ARE

NOT MEASURABLE AND NOT TIME-SPECIFIC


• THE ABSENCE OF BASELINE DATA FOR THE FIRST YEAR OF IMPLEMENTATION AND EACH YEAR OF FUNDING (WHERE APPROPRIATE)

• LACK OF ANNUAL EVALUATION REPORTS OR SOME OTHER TYPE OF ASSESSMENT TO VERIFY THE ATTAINMENT OF THE OBJECTIVES EACH YEAR OF FUNDING


• ACTIVITY DIRECTORS NOT HAVING A CLEAR VISION OF THE PRIMARY ANTICIPATED/PLANNED OUTCOMES OF THE ACTIVITY AND HOW THE ACTIVITY WILL STRENGTHEN THE INSTITUTION

IMPACT STUDY PROCESS USED AT NCCU

• Identified the over-arching goal or planned outcome for each program;• Identified the actual outcome based on data;

IMPACT STUDY PROCESS USED AT NCCU

• Analyzed the gap between the desired outcome and the actual outcome by utilizing the “Effectiveness Estimate (EE)” technique:

“EFFECTIVENESS ESTIMATE (EE)” TECHNIQUE:

• DOCUMENTS: THE EFFECTIVENESS OF THE

PROGRAM IN AMELIORATING OR STRENGTHENING THE ENTITY FOR WHICH IT WAS FUNDED

ELEMENTS OF THE EFFECTIVENESS ESTIMATE

• (R) Actual Results

• (C) Results Without Program

• (P) Planned Impact

THE EFFECTIVENESS ESTIMATE

EFFECTIVENESS ESTIMATE = (R) Actual Results - (C) Results

Without ProgramDivided By

(P) Planned Impact - (C) Results Without Program

ADEQUACY RATIO (AR)

• QUANTIFIES: PROGRAM ACCOMPLISHMENT

BY CONCEPTUALIZING THE “PROPORTION OF THE PROBLEM” ELIMINATED BY THE PROGRAM

ELEMENTS OF THE ADEQUACY RATIO

• (R) Actual Results

• (C) Results Without Program

ADEQUACY RATIO

ADEQUACY RATIO = (R) Actual Results

1 - (C) Results Without Program

Divided By

THE IMPACT STUDY REPORT INCLUDES:

1. Institutional enhancements as a result of implementing the program;

2. Enhancements to student and faculty performance, where applicable;


3. Affect on administrative personnel effectiveness, where applicable;

4. Overall academic and/or administrative program effectiveness enhancements;

5. Problems encountered that adversely

affected the implementation of the program during the specified time- period;


• Recommendations to strengthen the continued implementation of each program regardless of funding status;

THE IMPACT STUDY REPORT INCLUDES

• Results of the Effectiveness Estimate analysis to determine the level of program effectiveness;

THE IMPACT STUDY REPORT INCLUDES

•Results of the Adequacy Ratio analysis documenting the extent to which the program addressed the problem it was designed and implemented to address and the extent to which extramural funding had on program effectiveness.

CONCLUSIONInformation garnered through the impact study process, in combination with other sources of data, will provide an opportunity for the institutional leadership to incorporate the findings into programmatic and budgetary decision-making.

CONTACTS

• Dr. Gloria P. James: [email protected]

• Dr. Brenda R. Shaw: [email protected]

• Mr. Samuel T. Rhoades: [email protected]

mailto:[email protected]




Documents

THE QUANTITATIVE/ QUALITATIVE IMPACT STUDY