Emerging Issues of Data Collection in TAACCCT Grant Implementation

Presenters:Pradeep KotamrajuBureau Chief, Career & Technical EducationDivision of Community CollegesIowa Department of Education

Stephanie OppelStatewide DirectorIowa-Advanced Manufacturing ConsortiumDes Moines Area Community College

National Council for Workforce Education2014 Annual ConferenceOctober 30, 2014

TAACCCT in Iowa

• U.S. Department of Labor’s Employment and Training Administration awarded over $2 billion to individual and consortium grantees across the nation over four rounds of grant awards.

• Approximately $35.5 million to Iowa across four rounds of grants

• Expand and improve education focusing on career pathways, stacked and latticed credentials, enhanced industry engagement supporting economic growth.

• TAACCCT focus on TAA eligible workers, veterans, unemployed and under employed individuals.

• Bridges2Healthcare: Multi-state Consortium - $12,695,959 ($3,284,296 total awarded to Iowa colleges - NICC and HCC)

• Engineering Technology: EICC - $2,500,000

• Iowa-Advanced Manufacturing: Statewide Consortium - $12,951,165

• Information Technology Credentials to Careers: IWCC - $2,500,000

• Iowa’s Information Technology, Healthcare, Utilities and Manufacturing Network: Statewide Consortium - $15,000,000

In combination, these initiatives are projected to

impact more than 15,000 individuals, arming them with the skills they need to engage in today’s workforce.

TAACCCT in Iowa

• Priority 1: Build stacked and latticed curriculum and career pathways in Signature Programs

• Priority 2: Build a steady pipeline of skilled workers for Iowa’s advanced manufacturing in-demand occupations

• Priority 3: Improve the collaboration and alignment between community college programs, the workforce system and targeted industry employers to keep and create high quality jobs in Iowa.

Priorities of the Iowa Advanced Manufacturing Consortium

• Alignment with relevant industry credentials• Strengthen credit for prior learning• Stacked and latticed career pathways

• Model for intensive advising• Statewide outreach campaign• Enhance technology enabled learning

• Strengthened relationships with industry• Strengthened relationships with workforce

Areas of Focus

• Practical – Grantee collecting to help student achieve program success, aid in implementation; carries into other reporting and evaluation

• Compliance – Grantee collecting for reporting outcomes and implementation progress to the U.S. DOL on a quarterly and annual basis

• Implementation Evaluation – Third Party Evaluation team preparing progress and final reports on the impact of the grant

• Outcomes Evaluation – Third Party Evaluation team preparing outcomes analysis reports on the impact of the grant

Grantee and Third Party Evaluation team work collaboratively to inform all aspects of data collection

Elements of Data Collection

• Practical Data• Critical roles – Pathway Navigators, Faculty, Advisors• Grant impact – Offerings, pathways, institutional practices all aimed

at increased student retention, completion and entrance into the workforce

• Compliance Data• Establishment of consortium policies in line with DOL requirements• Continuous education and training for grant personnel• Creation of tools for collecting data• Establishment of MOUs to gain additional data sets

Grantee and Third Party Evaluation team work collaboratively to inform all aspects of data collection

The Grantee’s Role in Data Collection

In any TAACCCT project, a student follows a pathway from referral and college entrance, through the program study and to the point of exit and employment. There are different aspects of data collection as students navigate the pathway; correspondingly, these aspects have training requirements for staff and other stakeholders. Aspects of collection along the pathway are:

• Workforce Development Services Eligibility• Community College Entry• Enhanced Retention Support Services• Academic Program Progression• Graduation Exit and Employment Entry

Navigating Data Collection on the Student Pathway

• For TAA and other dislocated workers, workforce development staff are the first point of contact for finding training programs

• Many potential participants are referred by workforce development agencies and career services to community college programs, as has been the case in for many TAA-eligible workers.

• Community colleges work to educate workforce development staff about available programs.

• Workforce development staff collect relevant data regarding prior employment experience, education and training prior to the employment dislocation, and any relevant information that can be of potential use during community college application process.

• Students are referred to pathway navigators who continue to help students determine which program they would be interested in and suited for at the college

Touchpoint: Workforce Development Service Eligibility

• Once the TAA-eligible worker or other individual chooses to enter community college, there is specific information that needs gathering.

• This information could include things like prior learning experience, more in-depth information regarding the applicant’s previous education and training, any assessments or any pre-qualifications tests required for program entry.

• Much of this information is entered on an intake form, either hand-written or electronically. Often the institution makes provision to enter that data into its standardized student record databases.

• Connecting information on the intake form to standardized student record databases often requires planning

• Goal here is to gather enough information about the applicant’s current personal life and prior education and workforce training experience to understand what student support services the applicant might require.

Touchpoint: Community College Entry

• A key element of TAACCCT programs is the provision of student support services that target the retention of students through enhanced supports.

• Given the variety of student support services offered and used, multiple forms of data collection are necessary, including perceptual data gathering techniques (e.g., focus groups, interviews) and service delivery tracking (e.g., student interaction portfolios, checklists).

• The collected information should be linked to the extent possible to intake forms and to information in the institutional and workforce databases in order to understand the connections between and among background circumstances, program elements, and outcomes.

• Need to emphasize the importance of gathering consistent data at the most micro level possible (e.g., individual level whenever possible, program level if not individual), and understanding how linkages must be made between the different sources and methods of data collection.

Touchpoint: Enhanced Retention Support Services

• While other data collection sources and methods are used within this stage, the primary source for data collection is the institutional records that reside within the wider information technology system.

• During this stage, much of the information required to gauge academic progression is generally available in the institutional record database.

• The student records data can be supplemented by other methods of data collection, such as surveys. It is essential that the methods by which data from these institutional systems are extracted following standard protocols.

Touchpoint: Academic Program Progression

• Like the workforce development services eligibility stage, when it comes to data collection, this stage requires explicit cooperation between two different state agencies: workforce development and the postsecondary education authorities.

• Requires the transfer of information between the two agencies to gather information about the employment placement and employment retention wages of graduates who have completed a specific educational program.

• Protocols need to be established to facilitate the data collection and transfer. These protocols could be formal such as memoranda of understanding, or they could be less formal, such as a letter of agreement.

• This stage focuses on the specific data needs of the two agencies, the method of transfer, and ensuring data security.

Touchpoint: Graduation Exit and Employment Entry

• Rapid Response to TAA and other closures and layoffs

• Workforce and Colleges connect with workers jointly

• Provide program information

• Workforce referrals and local relationships

• IWD/CC Partnership

• Colleges educate workforce staff on programming – local and state level

• Program recommendation and referral

• Self-assessment tool

• NCRC

• Program and support services eligibility

Practical data gathered through this process enables workforce and college to deliver better service to students for

better outcomes

I-AM: Workforce Development Service Eligibility

• Entrance Interviews, Testing & Advising• Pathway Navigator• Required entry exams & orientation

• Participant Intake Form• Standard across colleges for all grant

participants• Focus on DOL Required reporting elements• Developed with input from 3rd party evaluation

team• Participant Database• Web based, secure, InfoPath database• Simple interface, user guide, webinars and one

on one training of users• Embedded reports

I-AM: Community College Entry

Compliance data and practical data are gathered as the student enters college. Pathway Navigators play a critical role in gathering both sets of data to ensure

accurate reporting and help ensure student success.

Practical Data & Navigators

• Advisor, coach, cheerleader, tutor, recruiter and MORE!

• Facilitates intake and helps link student to support services

• Intrusive advising model

• Tools for student tracking

I-AM: Enhanced Retention Support Services

During this aspect of data collection, I-AM staff focus on the individual student, working with students to maximize opportunity and support to increase

retention, completion and career exploration.

• I-AM Database• Guidelines and timelines for

updating• Relies on student information

systems• College and consortium access• Follows student from intake to

exit

I-AM: Academic Program Progression

I-AM Consortium staff developed a step by step user guide for frontline data entry staff to help ensure consistency of data. Webinars and one-on-one

trainings take place continuously. Data monitoring at local colleges provides I-AM Consortium staff an opportunity to compare student records system data

to I-AM database data.

• I-AM Database• Exit and completion information• Challenges with student

information systems

• Wage Data from IWD• MOU for data sharing• Secure data transfer• Reporting timeline issues• IDOE facilitation

I-AM: Graduation, Exit and Employment Entry

• Outcome data for DOL and to focus activity and resources• College calendar v. grant calendar• Ensuring accuracy at colleges• Availability of data from wage records

• Financial reporting for DOL and to help manage spending of colleges• Required quarterly reporting• Optional monthly reporting

• Programmatic reporting for DOL and to help manage grant activities at each college• Required quarterly reporting using form• Quarterly and annual meetings with college teams

Grantee Compliance Reporting

• Gathering data• Availability of data from wage records• College calendar v. grant calendar

• Financial reporting for DOL and to help manage spending of colleges• Required quarterly reporting• Optional monthly reporting

• Programmatic reporting for DOL and to help manage grant activities at each college• Required quarterly reporting using form• Quarterly and annual meetings with college teams

Compliance Reporting

• Implementation Evaluation• Milestone Evaluation

• Data Source: Quarterly reports submitted by Project Leads• Purpose: Track implementation progress & status

• Evaluation of Activities• Data Source: Surveys of key stakeholders• Purpose: Assess program benefits/strengths & challenges; formative &

summative information

• Outcome Evaluation• Meeting DOL requirement for rigorous analysis of grant impact on

participants – Quasi-experimental design• Establishing comparative cohort• Accessing data records across sources

Third Party Evaluators and Data Collection

Implementation Evaluation: Milestones

Initial Reporting

• Quarterly Report• General overview of

progress• Covered only strategies • True progress unknown• Evaluation of progress was

cumbersome due to reporting structure

Changes in Reporting as a Result of the Evaluation

• Quarterly Report• Very specific descriptions of

progress• Covers each milestone

within each strategy• True progress is known• Evaluation of progress is

easier/faster

Implementation Evaluation: Activities

Surveys

• Student (upon program completion)

• Student Exit (6 months after completion)

• Project Leads• Welding Faculty• Other Faculty• Marketing• Employer/Partner• Committee• College Leadership

Challenges

• Large number of stakeholder groups that require unique surveys

• Difficulty in reaching survey respondents due to differences in how community colleges communicate with their respective stakeholders

Outcome Evaluation: QUASI-EXPERIMENTAL DESIGN

Focus: Students enrolled in credit welding programs at Iowa community colleges between Fall, 2013 and Spring, 2014.

Why welding? Largest signature program in I-AM.13 (now to be 15) of the 15 participating Iowa community colleges have credit welding programs funded by I-AM grant.

Outcome Evaluation: Quasi-Experimental Design

Focus: Students enrolled in credit welding programs at Iowa community colleges between Fall 2013 and Spring, 2014.

Why welding? Largest signature program in I-AM.13 (now to be 15) of the 15 participating Iowa community colleges have credit welding programs funded by I-AM

Data Sources:

Outcome Evaluation: MIS & IWD Data Timeline

Challenge of accessing data:• Identification of stakeholders• Grantee, Evaluator, State Agencies

• Identification of protocols for records release• State agency policies and legal requirements• Systems for distribution and retention of records

• Gaining CONSENSUS• Aggregate data easily available• Individual data difficult to obtain

• Development of a Memorandum of Understanding (MOU)

Grantee, Evaluation team, and state agencies all play a role in

Outcome Evaluation: Access to Data

Takeaways…. • Grantee and evaluation team must work collaboratively to

inform all aspects of data collection• Cohesive data and evaluation plan

• Gaining access to data• Availability of data from institutional records• Identifying key data stakeholders – state agencies• Begin MOU work early

• Training data collectors• Face to face, webinars, troubleshooting• Monitoring

• Communication with stakeholders

Emerging Issues of Data Collection in TAACCCT Grant Implementation