AC 2010-373: COMPUTATIONAL THINKING: WHAT SHOULD OUR STUDENTSKNOW AND BE ABLE TO DO?

Dianne Raubenheimer, North Carolina State UniversityDr. C. Dianne Raubenheimer received her PhD from the University of Louisville and is Directorof Assessment in the College or Engineering and Adjunct Assistant Professor in the Departmentof Adult and Higher Education at NC State University. Within the College of Engineering sheserves as the coordinator of ABET and other accreditation processes, acts as an assessment &evaluation resource/consultant to faculty in different programs, develops and implementsassessment plans, and serves as the primary educational assessment data analyst on the Dean’sstaff. A particular interest is in helping faculty to develop and implement classroom-basedassessment and action research plans to establish the effectiveness of instruction and to use thedata to improve teaching and student learning. She is currently working with several engineeringfaculty, researching the impact of in-class use of technology on teaching and student learning.Dianne has also worked as an education consultant for a number of organizations and is currentlyserving as external evaluator on several grants.

Eric Wiebe, North Carolina State UniversityDr. Eric Wiebe is an Associate Professor in the Department of Mathematics, Science, andTechnology Education at NC State University. He received his Doctorate in Psychology and hasfocused much of his research on issues related to the use of technology in the instructionalenvironment. He has also worked on the integration of scientific visualization concepts andtechniques into both secondary and post-secondary education. Dr. Wiebe has been a member ofASEE since 1989.

Chia-Lin Ho, North Carolina State UniversityChia-Lin Ho joined the Computing across Curricula Team in early 2008 as a researchassistant.She is a graduate student in the Industrial/Organizational Psychology Doctoral Program.She received a B.S. in Psychology and a Bachelor of Business Administration at the NationalCheng-Chi University in Taiwan in 2002 and her Masters in I/O Psychology at the University ofNorth Carolina at Charlotte in 2005. Her research interests include measurement and evaluationissues, individual differences, leadership, cross-cultural studies, work motivation, and theapplication of technology on human resources management.

© American Society for Engineering Education, 2010

Computational thinking: What should our students know and be

able to do?

Abstract

A NSF funded project on our campus has two overarching goals: (1) to create a computational

thinking thread in engineering programs that spans from the freshman to senior years and bridges

the divide between freshman year computing and computing in upper-level classes, and (2) to

enable students to take computing competencies to the next level, where they are able to perform

high-level computing tasks within the context of a discipline.

To achieve the goals of the project, faculty fellows from different engineering departments

participate in a series of seminars relating to computational thinking that are held over the

duration of a semester. The faculty fellows also undertake action research projects in their

classrooms as they redesign course curricula to integrate computational thinking skills. In order

to build relevant curricula and to offer appropriate faculty development sessions, it was first

necessary to identify what computational skills and competencies different engineering industries

expect of graduates as they enter the workforce and in their first years on the job. This

presentation will share the results of the data collection and analysis effort centered on

identifying these criteria.

The starting point for identifying industry needs was a workshop held with a panel of industry

representatives. Based on the results of the industry workshop, a model of computational

capabilities emerged that articulated different levels of computational ability in a problem

solving context. Using this framework, a Delphi process was employed to survey a group of

practicing engineers employed by companies hiring graduates from our university, to gain

consensus about desired computational skills and competencies. In the first round of the Delphi,

six open-ended questions were posed. Preliminary versions of the questions were piloted at the

industry panel workshop and refined based on the framework for use in the Delphi study. The

open-ended responses were coded by three researchers independently and then collectively

developed into a consensus coding scheme. The resulting coded responses clustered into five

main themes, with interrelationships established between primary themes and subthemes. Using

these themes, the second round Delphi survey was developed requesting the same employers to

rate the importance of computational skills on a 5-point scale from 'not important' to 'very

important'.

In addition to presenting results of both rounds of the Delphi study and the model of

computational abilities, we will present the pre- and post-course student survey results conducted

in classes where curriculum changes have been piloted based on this model of computational

thinking. The purpose of the student surveys was to find out if changes in the courses reflected

the computational skills highlighted by industry representatives as essential skills for graduates

to possess.

Finally, we will present recommendations for larger scale curricular changes in the different

engineering disciplines based on the findings from this study.

1.0 Introduction

The engineering workplace has been impacted by rapidly developing computational technologies

that are radically reshaping the nature of the workplace.1 This and other immense changes in

global political and economic dynamics means the 21st century engineer will look very different

than their 20th

century counterparts.2 While these changes can be seen as a real threat to the

engineering job market, engineers who have learned how to harness computational capabilities

for advanced analysis and problem-solving will continue to be in great demand for decades to

come. However, while broad, general skills such as computational capabilities are recognized as

crucial to future careers, there is a dearth of understanding as to how to characterize these

abilities and how to integrate them into STEM curricula.3 To make sure that future engineering

graduates are properly prepared for the 21st century workplace, our multidisciplinary National

Science Foundation project, CPATH: Computing Across Curricula, has a twofold goal to (1)

characterize and develop a computational thinking thread that spans beyond the freshman year’s

computing course to all levels of the engineering curricula, and (2) increase students’

computational competency by applying appropriate computing approaches during/in the problem

solving process.

Developing computationally capable engineers requires the understanding of both what

capabilities matriculating students bring with them4 and what engineering employers expect from

their new hires. In addition to being able to characterize students relative to their computational

capabilities at these endpoints, engineering faculty and instructors need to be able to be able to

employ a framework of computational capabilities that both provides a vehicle for measuring

capabilities of students at the beginning and end of their courses, but also helps guide their

strategies around pedagogical innovations intended to impact these capabilities. With the purpose

of specifying the trajectory of computational capabilities through an undergraduate engineering

program, the project gathered data from numerous sources, 1) to understand the endpoint

trajectory of graduating engineers by engaging representatives of companies which historically

have hired our graduates, and 2) to develop a theoretical framework of computational capabilities

and instructional model that can be used to characterize student abilities and guide instructional

design.

To this end, the research reported here both provides background to the overall approach to the

research project, results to date, and new data aimed at answering the following research

questions:

1. What computational capabilities do future managers of engineering graduates expect new

hires to possess?

2. Can the instructional model of computational capabilities, developed during the research

process, be used to characterize the experiences, abilities, and the perceptions of

engineering students along dimensions of interest to future employers?

2.0 Approach

Figure 1 depicts the overall approach to data collection and analysis to address the two primary

goals of this project: (1) to create a computational thinking thread in engineering programs that

spans from the freshman to senior years and bridges the divide between freshman year

computing and computing in upper-level classes, and (2) to enable students to take computing

competencies to the next level, where they are able to perform high-level computing tasks within

the context of a discipline.

Both the experience of engineering education faculty on the project and ongoing reporting in the

literature5, 6

indicated that our strategy should involve the examination of both commonalities

and differences in computational literacy goals across a broad range of engineering disciplines.

The Delphi process7, a method used to gain consensus on a topic among a group of experts, was

selected as the most robust approach to inductively identify the specific needs of engineering-

oriented industries that have been ill-defined in the literature. A literature review and a Delphi

process were paired with the implementation of a “Computing across Curricula” (CAC)

community which sponsored engineering faculty fellows to participate in a seminar series and

conduct action research projects in their classrooms. The CAC community was a crucial test bed

for developing and testing a Computational Capabilities Theoretical Framework.

It was our intention that the emerging theoretical framework and the research results from this

project be used for further research, curriculum decision making and classroom change. This is

reflected in the schematic diagram below, where research results have informed both classroom

interventions, as well as the design of subsequent stages of the research process.

Figure 1: Schematic showing the overall approach to data collection and analysis.

2.1 Initial Industry Panel and Computational Capabilities Instructional Model

With the goal of defining an initial set of computational capabilities around which the Delphi

questions could be developed, an industry panel was convened8. The industry panel was also

asked to identify individuals who would participate in the Delphi study. The industry panel

included thirteen participants and represented companies in the computing, energy, textile and

healthcare industries. Participants included senior executives, as well as first-line engineering

managers, and represented five different engineering disciplines. A small brainstorming activity

was facilitated in each subgroup using Affinity Diagrams to answer three open-ended questions

drafted by our project team, with the potential for those questions to be used later in the Delphi

process. The three questions posed to the industry panel were:

≠ What proficiencies and fluencies are required for new hires in your company?

≠ What proficiencies and fluencies do you expect your workers to develop during their first

years on the job?

≠ What new proficiencies and fluencies do you see emerging in the next couple of years in

your field?

From the combination of results from both subgroups, some common themes emerged as shown

in Table 1. These results and feedback from the workshop were utilized to construct the first

Delphi.

Table 1: – Common Themes from the Industry Panel Workshop

New hires After first year on job Next few years

Specific applications (domain

knowledge)

Technological tools Architecture &

technology skills

Problem solving skills (critical

thinking)

Systems knowledge Soft skills (global

issues)

Communication skills Self motivated innovation Accountability

Knowledge of a programming

language

Understanding business

needs (value proposition)

Data exploration

Database management skills Data reporting

In parallel with the industry panel work was a comprehensive literature review pertaining to

computer competency, proficiency, and fluency at the university level. The results of the

literature review revealed broad and inconsistent interpretations of the terms competency,

proficiency, and fluency, with very little material that spoke to the specific needs of the

engineering profession. While numerous articles were collected as part of the literature review,

two documents ended up being central to the model development: an influential National

Research Council report 9 and more recent work done by Dougherty and colleagues.

10

Using the outcomes of the industry panel workshop and literature review, a first draft of a

Computational Capabilities Instructional Model was completed (Figure 2). This model is part

one of the Computational Capabilities Theoretical Framework that emerged.

The instructional model we developed looks at computational capabilities needed in a problem-

solving context, central to both professional engineering practice and, appropriately, engineering

education.11

Basic, relatively stable intellectual capabilities are recognized as essential for

problem solving, including the general cognitive abilities necessary for learning and applying

declarative and procedural knowledge as well as for engaging in the problem-solving process.

Technical skills refer to the abilities pertaining to manipulating a specific software tool or

programming in a particular language to solve the problem. Two types of specific knowledge

also need to be applied to the problem. Conceptual knowledge is higher-level knowledge (i.e.,

understanding at a more abstract level) of computing systems and languages in general.

Application domain knowledge is within the engineering discipline where the problem resides

(e.g., polymer synthesis, circuit design, mechanical coupling design).

Figure 2 – The Computational Capabilities Instructional Model

Using this model, three levels of Computational Capability were defined (Table 2), and this

forms Part Two of the Computational Capabilities Instructional Model. The goal was to establish

terminology to describe curricula that identify what general capabilities should be assumed as

students leave secondary education and matriculate into an undergraduate engineering program,

and that can be used to describe competencies as these same students enter the workforce.

Table 2: Levels of Computational Capability

Level Description Competency

The individual has technical skill mastery of certain computational tools

and/or programming languages. Limits in conceptual knowledge means

that they are limited to solving well-defined tasks with specified tools.

When faced with a more open-ended or complex problems, limits in

conceptual knowledge will mean they will probably not be able to solve

the problem.

Proficiency

The individual has some conceptual knowledge of both computing

systems and their application domain. When presented with a problem,

they are able to select the appropriate tools(s), seek the necessary

information, and present a solution. The regularly used technical skills

are committed to memory, and external information resources are not

needed in these cases. More complex problems and problems with

multiple possible solution paths for which they have to evaluate the

quality of the different solution paths will create difficulties for the

individual. Overall intellectual capability may be a limiting factor.

Fluency

The individual has extensive knowledge of the technical tools and

conceptual aspects of both computer systems and the application domain

of their profession. Within their professional area, they are able design

and evaluate multiple solution paths to complex problems. They are well

versed in general knowledge in the problem space and do not need to

refer to external resources for common problems. New computing tools

are readily evaluated and integrated into their existing tool set. Limits to

problem-solving usually result from moving outside their professional

application domain or the bounds of general intellectual capabilities.

During their four years in an engineering program, students continue to develop a competency

level of both general capabilities useful in many areas of their education, as well as specific

capabilities for their chosen discipline. The instructional model assumes that many of the

computational capabilities that a student develops and applies will be in a problem-solving

context. It also assumes that as students move through their undergraduate program and into

workplace settings, that these problem-solving scenarios will become increasingly ill-defined

and complex12, 13

, requiring the proficiency level of computational capability. It is important to

note that the assumption (based on feedback from the industry panel) is that few students will

develop capabilities at the fluency level prior to embarking on a professional engineering career.

2.2 Delphi Round 1

The first round of a Delphi study involved posing a set of open-ended questions to a group of

experts. 14

The detail of the Delphi process we used is described in last year’s conference paper 15

.

In summary, our experts were engineering line managers who directly supervise new engineering

hires in a company, including graduates from our university. Using the initial industry workshop

and the model presented above as the framework, we generated six open-ended questions

regarding the competencies, proficiencies and fluencies that industry expects of (a) new

engineering hires and (b) after a few years on the job. We requested the Delphi participants to

answer these questions, providing as much detail as possible.

These six Delphi questions were:

1. What computing competencies are required for new technical hires at your company?

2. What computing proficiencies do you expect your technical employees to develop during

their first few years on the job?

3. What new computing skills and processes do you see emerging in the next couple of

years in your field?

4. Once fluent, what types of problems do you expect your technical employees (with 3-5

years of experience) to solve using computing tools?

5. Once fluent, what types of projects do you expect your technical employees (with 3-5

years of experience) to design using computing tools?

6. What computing capabilities do you expect technical employees to use to be successful in

a global work environment?

All survey responses were content analyzed by three researchers who read all responses,

generated a coding list, coded all responses, checked for inter-rater agreement, sought consensus

if there was any disagreement about coding statements, generated a set of themes and then

explored the relationships among the themes. Thematic clustering of all responses, developed

through consensus, was used to summarize these relationships.

2.3 Delphi Round 2

For the second round of the Delphi, we focused on computing competencies required for new

technical hires (Question 1 in Delphi 1) and those that were expected to be developed during the

first few years on the job (Question 2 in Delphi 1). Themes emerging more than once from any

question in the Delphi Round 1 were used to develop a Likert scale survey, which was sent back

for completion by the respondents from the first Delphi. [We also recruited additional

respondents in this round who represented non-computing industries, specifically textiles

engineering].

The survey contained three sections, where the first section included three personal

identification questions (i.e., respondent name, job title, and company name), the second section

had 12 questions pertaining to the computing capabilities of new hires, and the third part had 14

questions pertaining to computing capabilities expected during the first years on the job. Each

response was assigned a value (1 = not important, 2 = slightly important, 3= average importance,

4 = important, 5 = very important) and the mean rating given by all respondents was calculated

for each question. Responses with a mean value higher than 4.0 and with a standard deviation

less the 1.0 indicate a high level of consensus among participants about the importance of that

particular item, while responses with lower means and higher standard deviations reflect lower

levels of consensus 14

.

Results were further analyzed by type of engineering industry, with computer science, electrical

engineering, computer engineering, information technology and engineering computer services

responses being aggregated separately to the responses from other industries that included

chemical engineering, textile engineering, mechanical engineering, civil engineering, chemical

and biomolecular engineering, and industrial and systems engineering. This decision to analyze

the computer science-related and other engineering responses separately came out of the first

round of the Delphi analysis, with the conclusion that these engineering areas had unique

computational capability needs.

2.4 Student Surveys

During the seminar series, the faculty fellows were exposed to the Computational Competencies

Instructional Model (Figure 2 and Table 2) and the Delphi results. The industry needs along with

other information about computing innovations were delivered to inspire these faculty members

to consider appropriate classroom pedagogical changes that would enhance the computational

capabilities of their students. Each faculty fellow that has participated in the seminar series has

undertaken some instructional change in at least one course that s/he is teaching in order to

enhance the computational capabilities of their students, with each engaging in a parallel action

research project. [The detailed results from each action research project undertaken, though

beyond the scope of this paper, will be reported in the future.] In courses where instructional

changes were implemented, students were recruited to take a pre- and a post-test survey, to

examine whether changes in the course reflected the computational skills highlighted by industry

representatives as essential skills for graduates to possess.

Construction of the initial student survey was based on the most common themes that emerged

from the first round of the Delphi and the Computational Competencies Instructional Model, and

contained five sections. The survey was piloted in summer 2009 and changes made based on

student feedback. The final survey was deployed in fall 2009, and again in spring 2010, although

the latter results are not yet available. In the first section of the pre-survey, students rated the

frequency of use of particular computing processes/tools, thinking about the semester prior to the

current one, while in the second section they rated their level of skill in using those

processes/tools in the previous semester. In the third section, students rated the perceived value

of those processes/tools to their learning in the previous semester. The fourth section contains

attitudinal questions about learning and about computer tools. The fifth section asks

demographic questions. The post-test contains the same items, but in this instance, students were

asked to answer the questions as they pertained to the “enhanced computational capabilities”

course in which they were enrolled. The mean ratings of the pre- and post-survey were calculated

and compared using a paired t-test.

3.0 Results

The Delphi Round 1 research (previously reported in more detail15

) resulted in the identification

of five thematic cluster areas that were used to both inform the Delphi Round 2 research and the

implementation of the student surveys. These three areas of data collection and analysis are

reported in more detail below.

3.1 Delphi 1 Results

A total of 19 participants in the first round of the Delphi represented six different engineering

disciplines, with their work experience ranging from 1 to 20 years and with the position from

first-line engineers to senior managers.

There were 30 themes identified across the Delphi 1 survey questions, and each theme was given

a code. Five overlapping clusters emerged from our examination of the relationship among

themes15.

These five meta-themes were the result of clustering the larger set of more granular

codes, listed in Appendix 1, which emerged from the Delphi Round 1 coding. These larger

thematic clusters were:

≠ Computer Science (including Microsoft Office Tools, Basic Knowledge of Architecture,

Knowledge of Programming, Basic Operating Systems),

≠ Data Analysis (including Data Analysis Skills, Database Fundamentals, Database

Management),

≠ Design Modeling and Simulation (including Ability to Use Simulation Packages, Process

Modeling and Design, Proficiency in Simulations),

≠ Core Individual Work Skills (including Web Searching, Problem Solving),

≠ Meta-Project Level (including Project Management Applications, Communication

Tools).

This thematic analysis revealed there was a core set of computing skills common to all

engineering disciplines. For instance, across the Computer Science cluster, there were Core

Individual Work Skills (e.g., Microsoft Office Applications) common to all engineering

disciplines and there were also themes related to engineering problem-solving common to all

disciplines. While the specific tools varied, all disciplines needed to manage engineering data as

part of their problem-solving processes (e.g., Database Management, Database Fundamentals).

Meta-Project Level themes were also common across disciplines and relate to connecting

engineering design work to other aspects of the companies’ work. Many of these “soft skills”,

while not directly related to engineering problem-solving, have long been recognized by

engineering educators as key capabilities valued by the engineering profession.

As noted above, for computer science and computer and electrical engineering, much of the day-

to-day computational work stayed within the Computer Science cluster. However, other

engineering areas weighed heavily on themes that appeared in the Design Modeling and

Simulation cluster. Data Analysis held themes that were related to both Computer Science

capabilities and somewhat more indirectly related to the design process, such as managing

engineering data used in the decision-making process. As noted above, design data management,

whether it was related to computer science/computer and electrical engineering or to engineering

areas making use of modeling and simulation tools, made use of database management tools that

also resided in the Data Analysis theme.

3.2 Delphi 2 Results

In order to answer the first research question relating to the computational capability

expectations of future employers of engineering graduates, a total of 22 respondents in the

second round of the Delphi represented nine different engineering disciplines from different

organizations.

Of 12 questions pertaining to computing capabilities of new hires, there was no response with a

mean value higher than 4.0 and with a standard deviation less the 1.0, indicating that no

consensus was achieved among participants about the relative importance of different computing

capabilities of new hires (see ranking of the capabilities in Table 3). In contrast, of 14 computing

capabilities expected to develop during the first years on the job, there was consensus by all

industries on four capabilities (as indicated by a mean about 4.0 and standard deviation less than

1.0), Ability to Learn and Adaptability, Microsoft Office Tools, Industry-Specific Tools, and

Data Analysis Skills. These were considered as important skills to develop in those first years

(see ranking of the computing capabilities in Table 4).

To analyze responses by type of engineering industry, the responses from the computer science

and computer/electrical engineering industries were clustered and indicated that there was

consensus on one capability, Basic Knowledge of Programming, is an important computing

ability for new hires (see Table 3). There was consensus that five capabilities, Proficiency in

Programming Languages, Ability to Learn & Adaptability, Basic Knowledge of Programming,

Integrated View of Systems/Applications, and Web Programming & Language, were to be

developed during the first few years on the job (see Table 4).

The results of the responses from all other engineering industries analyzed together indicated

consensus that two capabilities, Microsoft Office Tools and Web Search, are important skills for

new hires (see Table 3). There was also consensus that five capabilities, Microsoft Office Tools,

Ability to Learn & Adaptability, Industry-Specific Tools, Data Analysis Skills, and

Communication Tools, were expected during the first years on the job (see Table 4).

Table 3: Computational capabilities expected for new hires

All (N= 22) CSC Only (N= 8) Non-CSC (N=14)

Computing Capability Rank M SD Rank M SD Rank M SD

Web Search 1 4.2 1.2 3 3.6 1.8 2 4.6 0.7

Microsoft Office Tools 2 4.1 1.2 5 3.1 1.3 1 4.6 0.6

Data Analysis Skills 3 3.9 1.1 4 3.5 1.1 3 4.1 1.1

Basic Knowledge of

Programming 4 3.6 1.3 1 4.6 0.5 7 3.1 1.3

Proficiency in Programming

Languages 5 3.4 1.3 6 3.1 1.3 7 2.8 1.2

Communication

Tools/Organization 6 3.3 1.4 8 2.6 1.4 4 3.7 1.2

Queries Debugging/Testing 7 3.2 1.4 2 4.0 1.3 7 2.8 1.3

Web Programming & Language 8 3.1 1.2 4 3.5 0.8 7 2.8 1.3

Database Fundamentals 9 3.0 1.1 7 2.9 1.3 6 3.1 1.1

Process Modeling & Design 10 2.9 1.5 9 2.0 1.2 5 3.4 1.5

Software Systems Design 11 2.8 1.4 3 3.6 1.2 8 2.3 1.3

Basic Knowledge of

Architectures 12 2.6 1.3 4 3.5 1.2 9 2.1 1.0

Table 4: Computing capabilities expected to develop during the first few years on the job

All (N= 22) CSC Only (N= 8) Non-CSC (N=14)

Computing Capability Rank M SD Rank M SD Rank M SD

Ability to Learn & Adaptability 1 4.7 0.6 1 4.8 0.5 2 4.6 0.6

Microsoft Office Tools 2 4.3 0.9 7 3.5 1.1 1 4.7 0.5

Data Analysis Skills 3 4.2 0.8 6 3.6 0.9 3 4.5 0.5

Industry-specific Tools 3 4.2 0.9 6 3.6 0.9 3 4.5 0.7

Communication

Tools/organization 4 3.9 1.1 8 3.1 1.3 4 4.3 0.8

Basic Knowledge of

Programming 5 3.6 1.4 3 4.5 0.8 8 3.1 1.4

Proficiency in Programming

Languages 6 3.5 1.5 2 4.8 0.5 11 2.7 1.4

Process Modeling & Design 7 3.3 1.6 11 2.6 1.5 5 3.7 1.6

Project Management

Applications 8 3.3 1.2 9 2.9 1.1 6 3.5 1.2

Web Programming & Language 8 3.3 1.5 5 4.0 0.9 10 2.9 1.6

Integrated View of

Systems/Applications 9 3.2 1.4 4 4.1 0.8 11 2.7 1.4

Basic Knowledge of

Architectures 10 2.9 1.6 4 4.1 1.1 12 2.2 1.4

Proficiency in Simulations 11 2.8 1.3 12 2.1 1.3 7 3.2 1.2

Database Management 12 2.7 1.1 10 2.4 0.9 9 2.9 1.1

3.3 Student Survey Results

To answer the second research question in this project (whether the previously developed

instructional model of computational capabilities can be used to characterize the experiences,

abilities, and the perceptions of engineering students along dimensions of interest to future

employers), a sample of 92 students completed a pre- and post-course survey. The survey was

constructed by selecting the six themes that appeared most frequently in the industry responses to

the Delphi Round 1. Most students surveyed were between the ages of 19 to 21 years of age

(78.9%), male (82.4%), and white (84.9%; 5.8% were African-American). Students (35.2%

sophomore; 40.7% junior; 24.2% senior) were recruited from five courses (including a graphical

communications, chemical engineering, mechanical engineering and textile engineering course),

with a mean GPA of 3.27 (SD = 0.05).

The results indicated that, on average, students applied two computing capabilities, Web Search

and Communication Tools/Organization, more than once a week, during classes in the previous

semester (Spring 2009). However, for the semester in which they were enrolled in an enhanced

computational capabilities course (Fall 2009), students reported significantly less usage of

computing abilities related to web searches, communication tools, and basic knowledge of

programming (see Table 5).

Table 5: The frequency students applied the computing capabilities during the class

Pre-test Post-test

Computing Capability M SD M SD

Database Fundamentals 1.84 0.99 1.74 1.01

Process Modeling & Design 2.01 1.27 1.96 1.21

Basic Knowledge of Programminga 1.95 1.23 1.36 0.64

Data Analysis Skills 2.85 1.14 2.63 1.25

Communication Tools/Organizationa 4.11 1.23 3.39 1.47

Web Searcha 4.69 0.73 3.39 1.45

Note. Items are rated on a 5-point scale, ranging from 1 (Never) to 5 (Daily). a indicates pre- and post-test means are significantly different from each other at p < .05.

In terms of self-reported skill level, students initially rated themselves being competent in

Communication Tools/Organization and being proficient in conducting Web Searches in the

semester prior to taking the enhanced computational capabilities course. However, in the

semester of taking the enhanced computational capabilities course the self-rating of the Web

Search skills significantly decreased while the rating of abilities related to Database

Fundamentals, Basic Knowledge of Programming, and Data Analysis Skills increased

significantly (see Table 6).

Table 6: The self-reported skill level in applying the computing capabilities during the class

Pre-test Post-test

Computing Capability M SD M SD

Database Fundamentalsa 2.11 1.07 2.34 1.11

Process Modeling & Design 2.43 1.31 2.59 1.19

Basic Knowledge of Programminga 1.82 1.01 2.07 1.15

Data Analysis Skillsa 2.51 1.01 2.95 1.09

Communication Tools/Organization 3.75 1.02 3.64 1.12

Web Searcha 4.43 0.73 4.10 1.08

Note. Items are rated on a 5-point scale, ranging from 1 (Novice) to 5 (Expert).

a indicates pre- and post-test means are significantly different from each other at p < .05.

As regards the perceived value of different computing capabilities in facilitating their college

education, students gave high value to skills related to Data Analysis Skills, Communication

Tools/Organization, and Web Search at the beginning of the semester. However, at the end of the

semester the perceived value of these six computing capabilities in facilitating the student's

college education significantly decreased for all six dimensions (see Table 7).

Table 7: The perceived value of the computing capabilities in facilitating students' college

education

Pre-test Post-test

Computing Capability M SD M SD

Database Fundamentals a 2.46 0.89 2.08 0.99

Process Modeling & Designa 2.88 0.98 2.24 1.11

Basic Knowledge of Programming a 2.25 0.84 1.79 0.86

Data Analysis Skills a 3.19 0.79 2.76 1.04

Communication Tools/

Organization a

3.54 0.76 3.08 1.02

Web Search a 3.67 0.56 3.10 1.06

Note. Items are rated on a 4-point scale, ranging from 1 (Not at all) to 4 (Very valuable). a The pre- and post-test means of all computing capabilities are significantly different from each

other at p < .05.

The authors also examined the impacts of the course intervention on (a) students' self-efficacy

about learning in the discipline of engineering/computer science (9 questions), and (b) on the

self-efficacy of using computers (7 questions). The results, aggregating across questions in each

scale, are shown in Table 8. No change in either of the scales was found after implementing the

course interventions.

Table 8: Self-reported engineering/computing self-efficacy and computer self-efficacy

Pre-test Post-test

Scale M SD M SD

(a) Engineering/computer science self-

efficacy

32.38 4.73 32.73 4.36

(b) Computer use self-efficacy 27.67 4.23 27.46 3.84

Note. Items are rated on a 5-point scale, ranging from 1 (Strongly Disagree) to 5 (Strongly

Agree). There were no significant differences in pre- and post-test results.

4.0 Discussion

Overall, the iterative process outlined in Figure 1 has provided a robust approach (a) for

developing a Theoretical Framework, (b) for eliciting feedback from industry professionals

about specific industry needs, (c) for using results to inform subsequent research, (d) for making

decisions about curriculum and instructional practices, and (e) for assessing whether curricular

and pedagogical innovations incorporated into engineering courses, in response to these findings,

have impacted the computational capabilities of students.

We will now respond to the two research questions initially posed:

1. What computational capabilities do future managers of engineering graduates expect new

hires to possess?

Round one of the Delphi effectively gathered the range of computational capabilities perceived

by industry representatives to be important for their specific disciplinary area. The first two

questions, relating to expected capabilities of new hires, and to expected capabilities needed after

a few years on the job, were particularly relevant 1) for understanding the endpoint trajectory of

our graduates, and 2) for developing a framework of computational capabilities to be used in

characterizing student abilities and guiding instructional design. The broad themes that emerged

were particularly important for faculty fellows as they considered the content changes and the

instructional modifications necessary for developing the computational capabilities needed for

the future employment of students.

Similarly, the Delphi Round 2 highlighted areas of consensus and non-consensus across

industries and within similar industries. It seems that the Delphi process employed in this project

has effectively captured the broad computational capability expectations of future employers of

engineering graduates. It is not surprising that there was little consensus across the spectrum of

computer science/computer engineering and the range of other engineering disciplines, because

each specific area has unique requirements. However, consensus on the expectation of some

computing capabilities was reached relating to new hires and for employees having worked for

few years, for both the computer science/computer engineering industries and for the non-

computer science industries. For faculty, the industry consensus around specific skills relevant

to their discipline makes it clear what skills students need to acquire before graduation. The

consensus items provide a set of priorities for student learning, and thus for course design and

instructional emphasis.

The only “general job skill” not tied to computational capabilities that was included on the

survey had to do with “Ability to Learn and Adaptability,” in part because so many respondents

on Round 1 of the Delphi brought up this qualification. Not surprisingly, there was also

consensus across all industries in Round 2 on the importance of this soft skill being developed in

the first few years of employment. This capability was also a consensus theme for both industry

groups, when the data were analyzed by industry category.

There was no overall consensus on any capabilities for new hires. Any other overall consensus

capabilities across all industries were related to the first years on the job, and these were the use

of Microsoft Office Tools, Data Analysis Skills and Industry-specific Tools. However, when the

computer science/computer engineering industries were analyzed separately, these were not their

major consensus capabilities, indicating that they have very specific computer science related

expectations of capabilities that are more important. These are Basic Knowledge of

Programming, Proficiency in Programming Languages, Web Programming and Languages, and

Integrated View of Systems/Applications.

Overall, there was an expectation, perhaps not surprisingly, that Industry-specific Tools would

be learned on the job, but that new recruits would bring with them to their new job the general

abilities to use MS Office and Web Search tools. There was overall consensus of the importance

of Data Analysis Skills to be enhanced during the first years on the job, but this was not a

consensus capability as a prior skill to be learned at university (although it did rate fairly highly).

Perhaps Data Analysis Skills are considered to be more industry specific, although there was

consensus for non-computer science jobs around the importance of this capability. Perhaps this is

less valued in some computer science areas, and so did not show up as a consensus capability for

the computer science industries alone.

Not surprisingly, basic programming was important for new hires into computer science jobs

with higher level proficiencies being developed on the job. Interestingly, there was not any

consensus about basic programming for new hires in non-computer science industries.

2. Can the instructional model of computational capabilities, developed during the research

process, be used to characterize the experiences, abilities, and the perceptions of engineering

students along dimensions of interest to future employers?

With respect to the second research question, the variations among respondents and the

differences between the six computing capabilities framed in the student survey seem to indicate

that the Computational Capabilities Instructional Model can be used to characterize some of the

experiences, abilities, and the perceptions of engineering students. Overall, it appears that the

survey was useful in tracing the frequency of use of different capabilities over different

semesters and for tracking the perceived skill level between semesters. However, the results

about the perceived value of the different capabilities are somewhat difficult to interpret, with a

decrease in all dimensions by the end of the semester in which the course was taught. The

engineering/computing self efficacy questions did not show any differences between semesters.

In all cases, the survey results showed a decrease in the frequency of use of different computing

capabilities between the previous semester, and at the end of the semester in which the enhanced

course was implemented. Three of these were statistically significantly lower levels of use,

including Basic Knowledge of Programming, Communication Tools/Organization, Web Search

Tools. The decrease in these specific capabilities is not too surprising, specifically the latter two,

because many instructors may assume students already have these capabilities and so not

specifically targeted them. And programming would be less of a focus in the non-computer

science courses. However, it is somewhat surprising that there was no increased frequency of use

for any of the capabilities. It may be important to discuss with faculty implementing enhanced

courses, whether they are actually teaching towards these capabilities, or whether additional

capabilities need to be added to the survey.

Students self-rated their skill levels for Database Fundamentals and Basic Knowledge of

Programming in the advanced beginner range, indicating they are still developing competencies

in these areas. Process Modeling and Design & Data Analysis Skills were rated in the competent

range, while they rated themselves in the proficient range for Communication

Tools/Organization and Web Searches. However, the relatively small difference in scores

between the pre- and post-tests, for all six computing capabilities, indicates that the shift towards

increased competency, proficiency or even fluency does not occur in a single semester. Indeed,

there may even be a self-reported backward movement in their development, as seen in the case

of Web Search capability, where this may not have been used much in the semester concerned.

The movement towards greater computational ability likely occurs across a range of courses

during the student's program, where the different computational capabilities are used and

expanded upon at successive levels in the curriculum. Thus, it would be valuable to track

students' self-reported capabilities over time.

The survey results show alignment between industry expectations and student perceptions about

the importance of Microsoft Office/Communication tools and Web Searching tools. Students

seem to come into their classes with both high ratings concerning how often they used

communication, web, and programming tools and what their perceived abilities were in the

previous semester. On one hand, they probably used some of these tools less than they did in the

previous semester, but their use of them perhaps also demonstrated that they didn’t know as

much as they thought they did.

The lower perceived values of the six computing capabilities and the higher self-reported skill

level of the computing capabilities in the semester of taking the enhanced computational

capabilities course may demonstrate that the value of learning a particular computing skill/tool

decreases as the corresponding skill level increases. Of note, in the area of more specialized

engineering tools (database, programming, data analysis), the courses seemed to be successful in

providing enhanced (self-reported) skills. So, another explanation for the lower frequency of use

may be that while the frequency of use was less, the level of application was deeper and more

intensive, leading to greater ratings of skill levels for those items.

It also seems that the classroom pedagogical changes may impact students' self-efficacy about

learning in the discipline of engineering/computer science and computer use self-efficacy in the

long term, rather than over one semester. Again, tracking of students over many semesters

would help to determine if there are shifts in efficacy over time.

5.0 Recommendations and Future Work

The two rounds of the Delphi process have provided a benchmark against which innovations in

enhancing computational capabilities can be developed across engineering disciplines. Future

Delphi studies may target specific engineering disciplines or sub-areas within disciplines, so that

specific engineering curricula or curriculum concentrations/tracks may be better served by

understanding the more detailed nuances of particular industries.

The first implementation of a student survey instrument has shown promise of revealing the

impact of pedagogical changes in the classroom on student's skill levels of some of the

computational capabilities indicated as important to industry representatives. Clearly more than

one semester with a relatively small student sample will be needed to see if the innovations

explored by our CAC Fellows are effective. Also, tracking of students over successive semesters

is needed to trace the development of computational capabilities throughout their curriculum.

The results of the first implementation of the student survey, the findings from the second round

of the Delphi, and discussions about the student survey results with faculty implementing the

enhanced courses should all be used to further refine the student survey instrument prior to a

larger scale roll-out. However, even our initial data has indicated that our instructional model of

computational capabilities has provided a useful framework for generating formative and

summative data.

The instructional model within the larger theoretical framework of computational capabilities,

plus the benchmarks established by the Delphi process, provides a mechanism for beginning to

flesh a progression of increased computational skills and knowledge over the undergraduate

years. As more classroom experiments are conducted and more feedback from faculty elicited,

progressions, that not only differentiate between years, but also disciplinary areas, will provide

rich, detailed data around which strategic instructional decisions can be made. A crucial feature

of these progressions is the need to continuously emphasize, throughout each year and all core

classes, the instructors to focus on using and further developing these computational capabilities.

As industry needs continue to evolve, ongoing data collection from students and faculty, as well

as from future Delphi rounds, will provide feedback to enhance the development of curriculum

progression strategies and models.

Acknowledgements

This work is supported by NSF (CISE # 0722192) as part of CISE Pathways to

Revitalized Undergraduate Computing Education (CPATH) program. The project team would

also like to extend its sincere thanks to our partners in industry who served on our panels and our

CAC fellows who are implementing their innovations in their classrooms.

Bibliography

1. Levy, F., & Murnane, R. J. (2004). The new division of labor: How computers are creating the next job market.

Princeton, NJ: Princeton University Press.

2. Vest, C. M. (2008). Context and Challenge for Twenty-First Century Engineering Education. Journal of

Engineering Education, 97(3), 235-236.

3. NRC, National Research Council. (2008). Research on Future Skill Demands: A Workshop Summary.

Washington, DC: Center for Education, Division of Behavioral and Social Sciences and Education.

4. Gonzales, R. F., & Renshaw, S. (2005). Connected Computing - A Hierarchy of Pre-Engineering Computing

Skill Competencies. Paper presented at the Annual Meeting of the ASEE.

5. Davison, L., & Porritt, N. (1999). Using computers to teach. Proceedings of the Institution of Civil Engineers-

Civil Engineering, 132(1), 24-30.

6. Kassim, H. O., & Cadbury, R. G. (1996). The place of the computer in chemical engineering education.

Computers & Chemical Engineering, 20, S1341-S1346.

7. Linstone, H. A., & Turoff, M. (1975). The Delphi method: Techniques and applications. Downloaded 1/4/2008

from http://is.njit.edu/pubs/delphibook/

8. Authors. (2008). Paper presented at the ASEE Annual Conference, Pittsburgh, PA.

9. NRC-CITL, National Research Council - Committee on Information Technology Literacy. (1999). Being fluent

with information technology. Washington, DC: National Academies Press.

10. Dougherty, J. P., Dececchi, T., Clear, T., Richards, B., Cooper, S., & Wilusz, T. (2003). Information Technology

Fluency in Practice. ACM SIGCSE Bulletin, 35(2), 153-171.

11. Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., & Saleem, J. (2007). Engineering Design

Processes: A Comparison of Students and Expert Practitioners. Journal of Engineering Education, 96(4), 359-

379.

12. Stevens, R., O'Connor, K., Garrison, L., Jocuns, A., & Amos, D. M. (2008). Becoming an Engineer: Toward a

Three Dimensional View of Engineering Learning. Journal of Engineering Education, 97(3), 355-368.

13. Strobel, J., & Cardella, M. (2008). Compound Problem Solving: Workplace Lessons for Engineering Education.

Paper presented at the ASEE Annual Meeting, Pittsburgh, PA.

14. McKitrick, S. (2007). Assessing ‘ineffable’ general education outcomes using the Delphi approach. Paper

presented at the NCSU Assessment Symposium, Cary, NC.

15. Authors. (2009). Paper presented at the ASEE Annual Conference.

Appendix 1: Finalized Theme Categories from the 1st Delphi Survey

Theme Code

1. Analyze & Evaluate Existing Process AEEP

2. Ability to Learn & Adaptability ALA

3. Ability to Use Simulation Packages AUSP

4. Basic Knowledge of Architectures BKA

5. Basic Knowledge of Programming BKP

6. Basic Operation System BOS

7. Communication Tools/organization COT

8. Data Analysis Skills DAS

9. Driver Concept DC

10. Database Fundamentals DF

11. Database Management DM

12. Forecasting F

13. Financial/Interdisciplinary Knowledge FIK

14. General: Teamwork, Problem solving (not computing

competencies) G

15. Industry-specific Tools IT

16. Integrated View of Systems/Applications IVSA

17. Knowledge of Architectures KA

18. Microsoft Office Tools MOT

19. None/Not relevant N

20. Project Management Applications PMA

21. Process Modeling & Design PMD

22. Problem-solving & Problem-shooting PP

23. Proficiency in Programming Languages PPL

24. Proficiency in Simulations PS

25. Queries Debugging/Testing QDT

26. Security Control SC

27. Software Systems Design SSD

28. Virtualization V

29. Web Programming & Language WPL

30. Web Search WS