This document is 100% funded by the MoSTEMWINS $19.7 million grant for the U.S. Department of Labor, Employment and Training Administration (TAACCCT). The product was created by the grantee and does not necessarily reflect the official position of the U.S. Department of Labor. The Department of Labor makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to such information, including any information on linked sites and including but not limited to accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership.

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/.

Statewide Data Integration – Supplemental Round 4 Grant

Project Closeout ReportMoSTEMWINS – MoSCORES Scorecard Project

October 2018

Background

In 2014, the Missouri Community College Association, the consortium of Missouri’s Community Colleges and State Technical College received a TAACCCT grant. The Missouri Economic Research and Information Center was subcontracted to complete 3 main deliverables:

1. Develop an Online Employment Results Scorecard 2. Provide Job Demand Analysis and Reporting3. Supplemental Project: Noncredit Data Integration and Enhanced Outcome Reporting

In addition, Missouri submitted a proposal for additional funding to develop a system to report outcomes of noncredit students. The project was one of three chosen for funding.

Scope

Missouri had already developed a couple of tools that would be integrated in to the new project. The Wage Explorer, a tool connecting wage and employment outcomes by program of study by region, and Eligible Training Provider List (ETPL) are database-driven. Through this grant, the capacities would be expanded to include noncredit programs, the reporting of outcomes by individual school with contextual statistics, and a secure, sustainable environment that is cost-effective to maintain moving forward. This information, integrated with the statewide course inventory mapping and grant participant performance information, positions Missouri’s educational and workforce systems in the twenty-first century in order to better align our educational pathways to the career occupational pathways.

Collaboration

Prior to this project, Missouri received grant awards through the Workforce Data Quality Initiative (WDQI). The WDQI projects required several state agencies to work together and share data to successfully complete the project. Therefore, a multi-agency data sharing agreement was signed among several departments with protocols for data usage and sharing for research purposes. Since the agreement was already in place among the agencies, the MoSTEM data sharing project had a head start for the work of the TAACCCT grant.

Core Work Group -- met regularly over the grant period to oversee the project. In addition to the Missouri Community College Association, the main work group is comprised of the Missouri Department of Economic Development, specifically the Missouri Economic Research and Information Center (MERIC) and the Division of Workforce Development, the Department of Higher Education, and the Information Technology Service Division (ITSD).

Portions of the project required assistance and input from other agencies, including the Department of Elementary and Secondary Education and Vocational Rehabilitation.

Data Advisory Task Force -- The Data Advisory Task Force helped to guide the project. Members

include staff from community colleges representing institutional research, workforce, and the Eligible Training Provider List. Thirteen meetings were held during the project to gather input and make decisions as a consortium.

In addition to the Data Advisory Task Force, feedback was sought from all institutions, including public 4 year and private institutions, to ensure that the interests of different types of schools were addressed and that they had an opportunity to comment and provide input during the process.

Key Grant Products

Through this grant opportunity, Missouri accomplished many things on the way to creating a usable system with valuable consumer information for counselors, students, parents, and the public. All aspects of the system were built to be sustainable after the grant ends, such as automating processes and uploads.

MoSCORES – A system comprehensive system with program information and outcomes (employment, wage, credential rates) for noncredit as well as traditional credit programs.

o Reporting Tab – Only available with an approved institution login, schools may use this tool for administrative reports.

Noncredit Supplemental Data File – Collection file for noncredit student records. Program Upload File – A data file for program information was developed. This file replaces a

manual data entry system used by ETPL, and collects additional data points not previously captured by the Department of Higher Education.

Missouri Connections – An adult focused version of Missouri Connections, a career exploration tool, was developed to assist individuals researching career options.

Quick Answer Reports – Four customizable, printable reports answer frequently asked questions. Users select the education level to view performance outcomes of program completers.

o Top Paying Programs o Most Popular Programs o Programs by School o All Program Work Outcomes

User Resource Guides/Manuals – A series of technical guides and manuals detail each data file and the upload process.

o EMSAS Program Upload – Technical guide to the Noncredit Supplemental Data File and Program Upload File along with the upload process.

o MoSCORES Technical Guide – Manual mapping the upload files to the online tool. Each field is matched to the data file, and the calculations for each measure are detailed.

o Missouri Connections – Guide to the resources and tools available in the career exploration tool.

o ETPL Marketing Materials – Marketing materials have been developed to educate the public on the

system.

o What is MoSCORES? – Introduces MoSCORES and the information found in the system.o How to Use MoSCORES? – Tutorial on how to navigate MoSCORES.o Exploring Work Outcomes – Instructions on using the 4 quick answer reports.o Career Exploration – Enhancements to Missouri Connections include adult-focused

career exploration tools. Labor Market Information – Information was compiled and published on occupations targeted

by the grant each month.

Closeout Activities

Final invoices have been submitted for this portion of the project.

Work with the web tools and longitudinal database is complete for the purposes of the MoSTEMWINS grant. The project is now in maintenance mode with ITSD, for data updates and any issues that may arise.

Screen Shots

MoSCORES, Missouri Connections and the Quick Answer Reports can be accessed through the Missouri Division of Workforce Development’s web site (https://jobs.mo.gov/jobseeker/training-and-education).

MoSCORES

Search - In the General Search section, the user may choose from a freeform text box or the dropdown boxes to choose a degree/credential type, field of study or school.

Additional filters allow searches by location or specific course descriptions.

In the Search by Location section, the user may search within a number of miles within certain zip codes, by county or Workforce Innovation Region.

In the Filter By section, the user can select occupational titles or check the filters describing the attributes of the program they are seeking.

The school, degree level, program length and WIOA approval indicator are listed for each program.

The Search Results link to additional tabs of information on the program.

The user can click the checkbox in the select column to compare program information.

Basic information on the program is in the first part of the table. School name, program title and duration describe the program.

After clicking the link into the program, certain information appears at the top of each page, including the school name and address, program, credential, and program length. A button for a printable report with program details allows the user to take information from the system.

Program Details - The Program Details tab begins with information detailing the program. Information such as the training site address, description, URLs for the course catalog and admissions, and school contact information are found in the Program Details.

Scroll down the page for more information from the school, like admissions requirements and comments. The workforce regions that have approved this program for their region are also listed.

In the Financial Information section, information about the total program cost is broken down into categories. Users are encouraged to contact the institution for the final program cost.

This information is provided by the school in the program upload file for the institution.

School Details - The School Details tab offers information about the school, including the main campus address, institution web address, and accreditation. For contextual purposes, the institution’s transfer rate and percentage of students receive PELL grants is also listed to give insight on the student population served by the institution.

Demographics - Demographics of the program completers are on this tab. The cohort years used to calculation the measures are listed under the heading.

Program details appear on the left, while a comparison of similar programs statewide is on the right. Any categories with 10 or less results are combined in the Other/Unknown category to prevent the identification of individuals.

Statistics are provided for Age, Gender and Race/Ethnicity to provide information on the student population of the program.

Performance - The Performance tab contains measures with outcomes of students that completed programs. The years of student completion are listed with each measure. Earnings and Employment Rate are published one and three years after program completion. The colored bar represents median results for the program, while the number to the right of the chart is the statewide median of similar programs. Below the Earnings chart is a breakdown of median earnings in urban and rural areas, providing additional context to the wage data.

The credential rate of credit

programs uses the standard higher education measure of first-time, full-time students for certain academic years listed with the measure. The measure tracks

those that transfer in and out of programs at the same institution and complete within the standard or extended time for the program. For example, the second and third year would be displayed for an associate’s degree. For noncredit programs, any noncredit student is included in the calculation, and a similar methodology is used to determine the standard and extended cohorts.

The total number of students enrolled in the program each year for 3 years is noted in the colored bars. Top Industries match program graduate to the NAICS of the employer’s unemployment insurance record, showing the industries that have hired completers of this program. Potential Careers is a crosswalk from the CIP to O*NET. The career titles are useful to graduates as they begin searching for a job.

Comparison - The Comparison tab provides a second opportunity to compare programs. The selected program may be compared to different programs from the same institution or to similar programs at other institutions. Dropdowns allow additional filters. The comparison table is the same as previously shown in the Search Results.

Reporting - The Reporting tab is only visible to authorized institution staff with an approved login. The reports allow the institutions with the ability to create reports based on filters and selections. Aggregate results (not student level) are returned through this feature.

Screen Shot of Quick Answer Reports

The quick answer reports provide answers to commonly asked questions about programs. Four reports allow users to explore Top Paying Programs, the Most Popular Programs, All Program Work Outcomes (aggregates statewide results by program), or Program Work Outcomes by School (provides results for programs offered at the institution). Reports are customizable, allowing users to make selections based on personal preferences. All reports may be downloaded or printed. The example is for the Top Paying Programs.

Each report contains relevant program results, including:

Program Name and CIP Award Level Graduation Years Number of Graduates Employment Rates - 1 and 3 years after completion Median Wages – 1 and 3 years after completion Top Industries employing program completers

Screen Shots of Career Exploration Tool

Missouri Connections was enhanced to include an option for adults exploring careers. MERIC data was incorporated into the exploration tools to show statewide and regional demand for occupations as well as Career Pathways.

Users may create profiles to save searches and preferences. Printable pages allow users to print the information on careers they select for further reference.

Screen Shots of Job Demand Analysis and Reporting

Informational materials were developed for the 3 areas of focus for the grant - Transportation, Logistics and Advanced Manufacturing; Information Technology; and Health Care and Life Sciences. Each publication was updated every 6 months on a rotating basis, with a new report released monthly.

The Career Pathway Summary helped readers understand STEM occupations and the basics of Career Pathways. Examples of different Career Pathways were mapped along with identifying skills and certifications typically required of people working in that pathway. Occupations were chosen for profiles, providing a detailed information on the work performed by people working in the occupation. A chart of job ads for the occupation over time shows employer demand over several years.

The RealTime Pathway Summaries provides insight from job ads placed by employers for the occupations targeted by the grant. Information includes the number of job ads; entry, average and experienced wages; basic skills, specialized

skills, and certificates requested in job ads; and the names of employers placing the

most job ads. The RealTime Pathway Summaries are produced for the state as well as each workforce development region.

Logos

Print Materials