Insights from customer needs and satisfaction at UCF’s FEEDS

programResults from the Voice of the Customer component of a Six Sigma methodology

project.

Roberto Champney1 Alain Gaumier2 Naomi Morris1

1University of Central Florida 2Optimum Consulting, Inc.

Objectives

• Project Background and Description

• Six-Sigma methodology overview

• Voice of the Customer (VOC)

• Results and Insights From VOC

• Recommendations

Project Background

• FEEDS- Florida Engineering Education Delivery System.

• Created in 1982 by the Florida State Legislature

• The system has evolved from a tape system in 1982 to a CD and web-based education option in its current form.

• The University of Central Florida (UCF) is one of 12 universities in the state that participates in the program, and holds the largest enrollment, 38% (2002-2003).

Project Background

• ASQ (American Society For Quality) Community Good Works Project.

– Improve communities through the use of quality tools and technologies.

– Provides skills transfer to non-traditional quality practitioners.

• Project Goals1. Document the fall 2003 version of the FEEDS

system.

2. Identify Priority Process Improvements.

DefineDefine

ImproveImprove

ControlControl

MeasureMeasure

AnalyzeAnalyze

Six-Sigma methodology overview

DMAIC

Classical View of Performance Practical Meaning

of “99% Good”:20,000 lost articles of

mail per hour15 minutes of unsafe

drinking water each day5,000 incorrect surgical

operations per week2 short or long landings

at most major airports each day

200,000 wrong drug prescriptions each year

No electricity for almost 7 hours each month

3 Sigma

4 Sigma

6 Sigma

93.32%

99.38%

99.99999%

HistoricalStandard

CurrentStandard

NewStandard

CapabilityLong-Term

YieldStandard

Voice of the Customer (VOC)

• Technically there are numerous ways to enhance the system, yet there has been little attempt to focus this enhancement to represent the needs of its users. – Identify customers and their needs.

– Collect and analyze reactive data.

– Consider proactive approaches.

– Convert collected data into customer needs.

– Sort out the most important attributes “ctq”.

– Obtain specifications for the critical-to-quality characteristics.

• Conducted a student focus group.

• Conducted several interviews with FEEDS students to:– Capture major needs categories and issues.– Further understand FEEDS users.

• Developed a survey concerning the satisfaction and needs of FEEDS student users.

• End of Semester FEEDS Student Online Survey.– Set adequate performance levels.– Correlations to identify key drivers.– Align opportunities for improvement with FEEDS UCF priorities.

Voice of the Customer (VOC)

The Surveys

• Two surveys over a two semester period

• Interview comments were used to create the survey questions:– 30 and 39 questions

– Some demographics.

• gender, study program, program level.

– Three types of questions (5-point likert scale and open ended).

• Satisfaction with current attributes of the system.

• Usefulness and Importance of particular attributes of the system (hypothetical and current).

• Comments on particular issues with the system.

The Survey (Sample Questions)

• Overall how satisfied are you with the FEEDS system?   

• Overall how effective are FEEDS courses compared to live courses?

• How satisfied are you with the quality of the video stream (vision and audio) on your FEEDS courses?

– Audio

– Image

• How satisfied are you with the communication with FEEDS professors?

The Survey (Validating Sample)

Percent (%)

FALL '03 SPRING '04

Aerospace Eng 3.83 4.73

Environmental Eng 3.24 4.05

Industrial Eng 17.70 20.95

Mechanical Eng 9.14 4.05

Eng Technology 18.29 21.62

Civil Eng 8.85 8.78

Computer Eng 10.32 12.84

Computer Science 7.96 6.08

Electrical Eng 12.09 13.51

Other 8.55 3.38

Total 100.00 100.00

•Response Rate: 338 (1st survey) and (149 2nd Survey)

•Mass email, and site banner.

•Percentages represent actual FEEDS student body distribution

Insights From VOC

Survey Questions

Satisfaction Effectiveness

Fall '03 Spng '04 Fall '03 Spng '04

FEEDS Effectiveness 0.690 0.539

FEEDS Satisfaction 0.690 0.539

Communication With Professor / Satisfaction 0.649 0.444 0.522 0.320

Image Quality Satisfaction 0.511 0.381 0.365 0.366

Audio Quality Satisfaction 0.439 0.434 0.330 0.353

Response Time From Professor -0.251 -0.166 -0.126 -0.080

Response Time From Professor / Satisfaction 0.334 0.187

Notes Before Class satisfaction 0.206 0.012

How Consistent is Delivery of Course Material 0.258 0.247

Correlations are significant at the 0.01 level (2-tailed).

Insights From VOC Overall Satisfaction Comments69% and 54% of respondents gave comments

• First Survey

•30 % positive comments

•41 % negative comments

• Second Survey

• 39% positive comments

• 19% negative comments

Note: one respondent may have contributed to more than one category of comments.

- Other11%

No Comment29%

Negative Remarks

4%

Positive Remarks

6%

- Professor Issues12%

+ Convenience24%

- A/V negative14%

No Comment42%

Convenience21%Positive

Remarks3%

Other3%

A/V positive2%

A/V negative4%

Professor Issues -4%

WebCT +3%

Professor Issues +3%FEEDS Site -

3%

NegativeRemarks

3%

Stream Availability +3%

Class Materials +3%

Assignment Confirmation -

3%

Insights From VOC

• On Spring ’04 UCF FEEDS upgraded their stream format.

Fall '03 Spring '04

Mean SD Mean SD

FEEDS Effectiveness* 3.80 1.11 4.01 1.03

FEEDS Satisfaction 3.97 1.05 4.14 0.85

Image Quality 3.45 1.26 3.61 1.15

Sound Quality 3.63 1.17 3.80 1.13

How Consistent is Delivery of Course Material 3.39 1.16

Notes before class satisfaction 3.49 1.18

Communication with professor, satisfaction 3.65 1.19 3.65 1.18

Response time from professor, satisfaction 3.58 1.32

Insights From VOC

       

Difference

  Mean  

Statistic Std. Error

SPRING '04 Video Audio -0.70 0.11

Video Image -0.86 0.11

Assignment Confirmation -0.98 0.14

Course Material Consistency -0.07 0.06

Notes Before Class -1.04 0.13

  Availability of Stream -0.09 0.11

• Prioritize future efforts using differences between ‘received’ and ‘desired’

• See SERVQUAL survey method.

Delivery of Service Quality," Journal of Marketing, April 1988, pp. 35-48. Parasuraman, Zeithaml and Berry, "SERVQUAL: A Multiple-Item Scale for Measuring Customer Perceptions of Service Quality," Journal of Retailing, Spring 1988, pp. 12-40

Insights From VOC• How often are you able to attend a live class?

– 29% to 39% always (4 & 5 scores)

• For what reasons are you not able to ALWAYS attend a live class?

  Count

Personal Schedule 81

Distance 54

Convenience 44

Other 40

Class Schedule 23

Work 20

Count

19 4 0 7 30

15 3 1 7 26

15 4 8 3 30

32 6 7 16 61

81 17 16 33 147

1-25%

26-50%

51-75%

76-100%

CourseworkCompleted

Total

SPRING '041-25% 26-50% 51-75% 76-100%

Coursework via FEEDS

Total

Insights From VOC

• Please indicate all the mediums with which you communicate with FEEDS professors? (Check the two most frequent)

Percent

Email 91.2

Person 49.3

WebCt 39.9

Phone 31.8

Other 2.0

Insights From VOC• How long does it takes on average to get a

response from your professor?• How satisfied are you with how long it takes to

get a response from your professor? – 55 % satisfied (4&5 score)

Insights From VOC

Where do you regularly watch the FEEDS stream or CD? (Check the two most frequent)

– Home 126– Work 47– Library 17– Computer Lab 18– Other 8 (Beach, Starbucks, Outdoors)

Notes access preference Percent

Download 67.5

Email 29.4

Buy Bookstore 2.6

Other 0.5

In the event that you could have access to the class notes before the class session, what would be the best way to gain access to them?

Insights From VOC• How often have you experienced missing material

from a FEEDS stream?

• In the event that you have experienced missing material from a FEEDS stream, what part was missing?

Yes52%

No48%

1-Never to 5-Always

-3-15%

-5-1%

-4-8%

-1-44%

-2-32%

Start16.3

Middle8.8

End31.3

None43.6

VOC Instrument Recommendations

• ~5 interviews will identify most of the problems.

• Don’t get carried away with questions, you’ll have to make sense of them later.

• Everything is important or useful. – Be careful with these types of questions. – Ex. Teacher assistant meetings:

• Usefulness score: 4.07• Attend score: 2.79

VOC Instrument Recommendations

• Open ended comments are most valuable, but very time consuming.– Use at least two persons for categorization.

• Provide enough space for comments.– 255 characters is not enough

• Use online surveys, don’t send attachment surveys

Insight Summary• Identify Student Strata

– 100% remote students.• Miss student interaction.• Miss out on after class talk.• Value recording of ‘pre-class’ and ‘break’ periods

– Convenience students• Register for FEEDS but attend live classes. • Use FEEDS as backup or study aid.

– Gender differences• Females more than males value the interaction with other

students and professors (per comments).

– Study Program differences• Teacher Assistant meetings, and problem solving groups.

Insight Summary

• User satisfaction and system effectiveness are correlated.– Effectiveness is driving satisfaction.

• Communication with professors is critical to satisfaction and effectiveness.

• ~1/3 of students regularly attend live classes• Email and in-person communication is used

mostly for communication between FEEDS students and professors.

Insight Summary

• Provide access to class notes before class, when possible.

• Consider scanning professor hand notes if possible.

• Chalk/White Board is not appropriate for this medium.

• Live students should see the same image as Remote students.– They’ll become the ‘directors’ of the class production.

Insight Summary

• Standardize equipment and training. – Encourage (enforce) professor training.

• Consider standardizing method of delivery and procedures (i.e. all use WebCT, all email assignments).– Use one location for files, content, or materials

access. – Need commitment from professors.

Insight Summary

• Computer demos are particularly difficult to follow (unable to read buttons and options).

• Most of students’ problems with the ‘stream’ are with accessing it due to technical difficulties.

• Provide content online, consider avoiding mailings (they consume resources).

Insight Summary

• Other online services or university sections confused as part of FEEDS (i.e. WebCT).

• Apply VOC methodology with other stakeholders– Professors, Administrators, etc.

Questions ?