Measuring Business Impact in Workforce Development

Lisa Soricone Commonwealth Corporation

David Cruise Regional Employment Board of Hampden County

Rosemary Hernandez Regional Employment Board of Hampden County

March 9, 2011

Page 1

Presentation Overview

• Definitions• Our Experience• Steps Involved in Measuring Business Impact• MBI in Action:

- Manufacturing - Early Childhood Education

• Q & A

Page 2

What do we mean by Business Impact?

• How does the business or employer benefit from a workforce development intervention?

• Rooted in Kirkpatrick’s model for evaluation of training1: Reaction2: Learning3: Behavior or application of learning4: Results or business impact

• Jack Phillips’ additions– Return on Investment– Intangibles

• Continuum: from good stories to ROI

Page 3

Where do we get our insights?

• 10 years experience with sector strategies --- workforce development with dual benefit (workers and employers)– BEST– Bay State Works– Workforce Competitiveness Trust Fund (incl. case studies)– Extended Care Career Ladder Initiative

• Research literature and other efforts in measuring business impact or employer impact– Kirkpatrick, Philips – Aspen Institute ‘s Business Value Assessment– Skills for Work project in the U.K.

Page 4

Challenges to MBI

• History of focus on individual outcomes• WD professionals’ discomfort engaging employers• Different needs/measures for different industries• Lack of time• Many employers don’t manage with this kind of data• Measuring the value of training is challenging work

Page 5

How can Business Impact be measured successfully?

5 key steps in the process:

1. Problem definition2. Identifying business impact measures3. Gathering data4. Analyzing and interpreting data5. Using the data

Page 6

Step 1: Problem DefinitionWhat is the problem to be addressed

through the training intervention?

• Ideally, part of the project design phase• Best pursued jointly by employer and WD staff• Requires some understanding of employer context/trends• Often involves multiple individuals from the employer

organization(s)

Page 7

Problem Definition: Things to Remember

• Consider business impact as early on in a project as possible.• Set up a meeting with management at each employer.• Prepare a set of questions to learn about your employer(s).• You needn’t be an industry expert, just be curious!

Working with multiple employers:• Look for common goals• Assess employer interest in engaging in the process• Consider the proportion of workers who’ve been trained at each employer

 

Page 8

Step 2: Identifying Business Impact measures

Once the problem to be solved is identified ,how will you know you’ve solved it?

• Define “success” in the employer context. Where can you see it?• Talk to employer staff at different levels:

• Leadership, Training/HR, Operating staff, Supervisors , Participants

• Use data that the employer is already collecting and using.• Where data/measures do not exist, focus on

meaningful and feasible measures.• Keep it simple.• Keep it close to the training.

Page 9

Sample Business Impact Measures

Workforce-related Measures Performance/Productivity Measures

• Vacancy rates• Recruitment costs• Retention of new hires• Orientation time• Staff diversity

• Ability to perform new tasks• Communication• Scrap /error rate• Machine set-up/run times• Safety• Application of writing skills• Customer/patient satisfaction• Supervisor/peer satisfaction

3/9/11 Page 10

Intangible Measures

Some things don’t convert to numbers but still matter:

•Improved work climate•Increased organizational commitment•Improved teamwork•Improved cooperation/reduced conflict•Reduced employee stress•Improved communication•Improved labor/management relations

Page 11

Step 3: Gathering DataOnce you have defined your measures, how do you get data to see

if desired change has happened?

• What comparisons can you make?Pre- / post- trainingComparison group

• Consider timing: When do you need data? When are data available?

• Clarify roles and responsibilities around data collection

• Create tools (e.g. , survey) for gathering new information

Page 12

Gathering Data: Things to Remember

• Establish a plan and timelines for data collection and reporting.For each measure:

How will data be gathered? ( e.g., survey) When will data be collected? Who will be responsible?

• Engage with employer representatives regularly to get data. Don’t be afraid to send reminders!

• Do a test run if new data is being collected or a survey is being used.

Page 13

Step 4: Analyzing and Interpreting Data

For each measure, review the data: what do they tell you?

• What kind of changes took place? o Did participant skills increase? o Are people working differently? o Did the employer experience a cost savings?

• To what extent did the changes that were expected to result actually occur?

• How significant is the change(s) observed? • Could this change be due to other factors/events?• To what extent do changes meet the goals of the project relative to

business impact?

Page 14

Analyzing and Interpreting Data: Things to Remember

In analyzing your data, focus on one measure at a time. First, pay attention to what the data say (“What happened?”)

before you jump to determining what results mean for the business impact of your project.

Involve stakeholders in the process of analyzing data at an appropriate time.

If you didn’t find the changes you’d hoped for, discuss findings within your project; there may be a lesson in there somewhere.

Page 15

Step 5: Using and Sharing Data

What can you do with all this new information?

• Business Impact data can be used for: Accountability and reporting to stakeholders Continuous improvement Feedback to trainers and project staff

• When sharing your data, place them in the context of the overall goals the project sought to achieve and also what else is happening in the environment.

Page 16

Using and Sharing Data: Things to Consider

• Consider your audience: - What do they really want to know?- What context will they need to understand your findings? - What level of detail will be most appropriate?

• What’s the best mechanism for communicating? - written report- presentation

3/9/11 Page 17

Value of MBI

• Demonstrates the value to employers of investing in their employees Improved employee skills Cost savings and/or quality improvements Address skill shortages or deficiencies in the workforce pipeline Improved relationships with education/training providers and workforce

development agencies

• Shows how public sector investment in workforce development improves competitiveness of companies

• Builds both a culture of and capacity for examination of value of such investments

6/25/2010 Page 18Page 18

Measuring Business Impact in Manufacturing and Early Childhood Ed

• What does ‘business impact’ mean in the employer context?• How did folks determine BI measures?• What was the process used for gathering data?• How were results shared and/or used within the project?

3/9/11 Page 19

Case Study: The Regional Precision Manufacturing Partnership of Pioneer Valley

• Sector-Based Workforce Development Intervention• Small and Medium Sized Precision Manufacturing

Enterprises (SME’S)• The Partnership

1. Precision Manufacturing Companies (8)2. Educational Institutions (6)

• Community Colleges (2)• Technical High Schools (4)

3. Workforce Investment Boards (2)4. One Stop Career Centers (3)

Page 20

The Regional Precision Manufacturing Partnership of Pioneer Valley

• Step 1: Identify the Shared Business Need– Goal 1- Provide training in Machine Tool Technology for 40

Unemployed/Underemployed Individuals.– Goal 2- Provide Skills Enhancement Training for 60 Incumbent Employees.

• Step 2: Negotiate Expected Results – Employee Productivity and Efficiency – Employee Stability (Retention, Turnover) – Employee Attitude, Motivation and Work Habits – Communication between Co-Workers/Supervisors – Company’s Book of Business and Profit Margins

Page 21

The Regional Precision Manufacturing Partnership of Pioneer Valley

• Step 3: Develop the Survey Instrument1. Business Impact Objective2. BI Rating3. Degree of Impact (I)- Difference Variation (N)4. Source of Rating

• Step 4: Conduct Business Impact Survey – Interims– Final

• Step 5: Analyze Data and Prepare Findings • Step 6:Prepare Business Impact Report

1. Funding Source2. Employers 3. Partnership

Page 22

Case Study: Developing Early Childhood Educators

Employer Partners Training Provider Community Colleges Institutions of Higher Education

Springfield Early Childhood &

Education Partnership

Preschool Enrichment Team, Inc.

Holyoke Community College

American International College

Square One Community Education Project

Springfield Technical Community College

Bay Path College

Valley Opportunity Council

Cambridge College

Elms College

Springfield College –School of Human Services

Westfield State College

Case Study: Developing Early Childhood Educators

Baseline of Employer Cluster Center Based: School Age: Family Child Care:

3607 1043 97

Total Children Serviced 4747

FTE PT Per Diem746 89 8482% 9% 9%

Total self reported employer workforce: 919

Associate’s Degree Bachelor’s Degree Master’s Degree

153 198 3

16% 21% 1%

Department of Early Education & Care (DEEC) Credentialed: 496 (53%)

Adult Basic Education / English as a Second Language needs: 56 (6%)

Developing Early Childhood Educators

Identifying the Problem

• Reduce staff turnover, • Increase the numbers of teachers engaging in credentialing

and/or higher education opportunities, • Increase the ability to train more staff in clusters, and • Increase the ability to meet staff qualifications for funding

opportunities, and other opportunities that increase quality and teacher compensation

Developing Early Childhood Educators

Gathering Data

• Setting the expectations from the beginning • One on one conversations• Group gatherings • Direct mail/ email

• Reminder phone calls/ follow through

Developing Early Childhood Educators

Building Connections

• The voices of the employer will continue to shape and measure the level of this workforce intervention.

• Workforce development is aligned with the needs of the industry and the State requirements on the education of the ECE workforce.

• Carefully Monitor the workforce intervention to provide results.

Contact Info

• Lisa Soricone lsoricone@commcorp.org• David Cruise dcruise@rebhc.org• Rosemary Hernandez rhernandez@rebhc.org

Commcorp website: www.commcorp.org

Hampden County REB sector initiative:

http://www.rebhc.org/pages/industry_initiatives.html

6/25/2010 Page 28