Running Head: COMPUTATIONAL THINKING 1

Computational Thinking in Engineering Students: Expectations from Industry

Manuscript to be submitted to theJournal of Engineering Education

Revision History:EW 6/15/10

COMPUTATIONAL THINKING 2

Abstract

A NSF funded project on our campus has two overarching goals: (1) to create a computationalthinking thread in engineering programs that spans from the freshman to senior years and bridgesthe divide between freshman year computing and computing in upper-level classes, and (2) toenable students to take computing competency to the next level, where they are able to performhigh-level computing tasks within the context of a discipline. To achieve the goals of the project,faculty fellows from different engineering departments participate in a series of seminars relatingto computational thinking that are held over the duration of a semester. The faculty fellows alsoundertake action research projects in their classrooms as they redesign course curricula tointegrate computational thinking skills. In order to build relevant curricula and to offerappropriate faculty development sessions, it was first necessary to identify what computationalskills and competencies different engineering industries expect of graduates as they enter theworkforce and in their first years on the job. This presentation will share the results of the datacollection and analysis effort centered on identifying these criteria.

The starting point for identifying industry needs was a workshop that was held with a panel ofindustry representatives. Based on the results of the industry workshop, a model ofcomputational capabilities emerged that articulated different levels of computational ability in aproblem solving context. Using this framework, a Delphi process was employed to survey agroup of practicing engineers associated with companies that hired graduates from our universityto gain consensus about desired computational skills and competencies. In the first round of theDelphi, six open-ended questions were posed. Preliminary versions of the questions were trialedat the industry panel workshop and refined based on the framework for use in the Delphi study.The open-ended results were coded by three researchers independently and then collectivelydeveloped into a consensus coding scheme. The resulting coded responses clustered into fivemain themes, with interrelationships established between primary themes and subthemes. Usingthese themes, the second round Delphi survey was developed requesting the same employers torate the importance of computational skills on a 5-point scale from 'not important' to 'veryimportant'.

In addition to presenting results of both rounds of the Delphi study and the model ofcomputational abilities, we will present the pre- and post-course student survey results conductedin classes where curriculum changes have been piloted based on this model of computationalthinking. Finally, we will present recommendations for larger scale curricular changes in thedifferent engineering disciplines based on the findings from this study.

COMPUTATIONAL THINKING 3

1.0 Introduction

The engineering workplace has been impacted by rapidly developing computational technologiesthat are radically reshaping the nature of the workplace.1 This and other immense changes inglobal political and economic dynamics means the 21st century engineer will look very differentthan their 20th century counterparts.2 While these changes can be seen as a real threat to theengineering job market, engineers who have learned how to harness computational capabilitiesfor advanced analysis and problem-solving will continue to be in great demand for decades tocome. However, while broad, general skills such as computational capabilities are recognized ascrucial to future careers, there is a dearth of understanding as to how to characterize theseabilities and how to integrate them into STEM curricula.3 To make sure that future engineeringgraduates are properly prepared for the 21st century workplace, our multidisciplinary NationalScience Foundation project, CPATH: Computing Across Curricula, has a twofold goal to (1)characterize and develop a computational thinking thread that spans beyond the freshman year’scomputing course to all levels of the engineering curricula, and (2) increase students’computational competency by applying appropriate computing approaches during/in the problemsolving process.

Developing computationally capable engineers requires the understanding of both whatcapabilities matriculating students bring with them4 and what engineering employers expect fromtheir new hires. In addition to being able to characterize students relative to their computationalcapabilities at these endpoints, engineering faculty and instructors need to be able to be able toemploy a framework of computational capabilities that both provides a vehicle for measuringcapabilities of students at the beginning and end of their courses, but also helps guide theirstrategies around pedagogical innovations intended to impact these capabilities (Figure 1). Withthe purpose of specifying the trajectory of computational capabilities through an undergraduateengineering program, the project gathered data from numerous sources to 1) understand theendpoint trajectory of graduating engineers by engaging representatives of companies whichhistorically have hired our graduates and 2) develop a framework of computational capabilitiesthat can be used to characterize student abilities and guide instructional design. To this end, theresearch reported here both provides background to the overall approach to the research project,results to date, and new data aimed at answering the following research questions:1. Has the Delphi process employed in the project effectively captured the computationalcapability expectations of future employers of engineering graduates?2. Can the previously developed instructional model of computational capabilities be used tocharacterize the experiences, abilities, and the perceptions of engineering students alongdimensions of interest to future employers?

2.0 Approach

Figure 1 depicts the overall approach to data collection and analysis to address the two primarygoals articulated above. Both the experience of engineering education faculty on the project andongoing reporting in the literature5,6 indicated that our strategy should involve the examination ofboth commonalities and differences in computational literacy goals across a broad range ofengineering disciplines. The Delphi process7, a method used to gain consensus on a topic amonga group of experts, was the most robust approach to inductively identify the specific needs of

COMPUTATIONAL THINKING 4

engineering-oriented industries that have been ill-defined in the literature. The literature reviewand Delphi process was paired with the implementation of a “Computing Across Curricula”(CAC) community which sponsored engineering faculty fellows to participate in a seminar seriesand action research projects in their classrooms. The CAC community was a crucial testbed fordeveloping and testing the computational capabilities framework.It was our intention that the theoretical framework and the research results from this project beused for curriculum decision making and classroom change. This is reflected in the schematicdiagram below, where research results have informed both classroom interventions, as well asthe design of subsequent stages of the research process.

Figure 1: Schematic showing the overall approach to data collection and analysis.

2.1 Initial Industry Panel and Computational Capabilities Instructional Model

With the goal of defining an initial framework of computational capabilities around which theinitial Delphi questions could be developed, an industry panel was convened8. The industry panelwas also used to identify individuals who would participate in the Delphi study. The industrypanel included thirteen participants and represented companies in the computing, energy, textileand healthcare industries. Participants included senior executives, as well as first-lineengineering managers, and represented five different engineering disciplines. A small

COMPUTATIONAL THINKING 5

brainstorming activity was facilitated in each subgroup using Affinity Diagrams to answer threepotential open-ended questions to be used later in the Delphi process drafted by our project team.The three questions were:

• What proficiencies and fluencies are required for new hires in your company?• What proficiencies and fluencies do you expect your workers to develop during their first

years on the job?• What new proficiencies and fluencies do you see emerging in the next couple of years in

your field?

From the exercise and combination of the results from both subgroups, some common themesemerged as shown in Table 1. The results and feedback from the workshop were utilized torefine the first Delphi survey and also led to the development of a Model of ComputationalCapabilities.

Table 1: – Common Themes from the Industry Panel Workshop

New hires After first year on job Next few yearsSpecific applications (domainknowledge)

Technological tools Architecture &technology skills

Problem solving skills (criticalthinking)

Systems knowledge Soft skills (globalissues)

Communication skills Self motivated innovation AccountabilityKnowledge of a programminglanguage

Understanding businessneeds (value proposition)

Data exploration

Database management skills Data reporting

In parallel with the industry panel work was a comprehensive literature review pertaining tocomputer competency, proficiency, and fluency at the university level. The results of theliterature review revealed broad and inconsistent interpretations of the terms competency,proficiency, and fluency with very little material that spoke to the specific needs of theengineering profession. Using the outcomes of the industry panel workshop and literaturereview, a first draft of a Computational Capabilities Instructional Model was completed (Figure2). While numerous articles were collected as part of the literature review, two documents endedup being central to the model development: an influential National Research Council report 9 andmore recent work done by Dougherty and colleagues.10

The model looks at computational capabilities needed in a problem-solving context—central toboth professional engineering practice and, appropriately, engineering education.11 Basic,relatively stable intellectual capabilities are recognized as essential for problem solving,including the general cognitive abilities necessary for learning and applying declarative andprocedural knowledge as well as engaging in the problem-solving process. Technical skills referto the abilities pertaining to manipulating a specific software tool or programming in a particular

COMPUTATIONAL THINKING 6

language to solve the problem. Two types of specific knowledge also need to be applied to theproblem. Conceptual knowledge is higher-level knowledge (i.e., understanding at a more abstractlevel) of computing systems and languages in general. Application domain knowledge is withinthe engineering discipline where the problem resides (e.g., polymer synthesis, circuit design,mechanical coupling design).

Figure 2 – The Computational Capabilities Instructional Model

Using this model, three levels of capability were defined (Table 2). The goal was to define acurriculum that identifies what general capabilities should be assumed as students leavesecondary education and matriculate into an undergraduate engineering program. During theirfour years in an engineering program, students will continue to develop a Competency level ofboth general capabilities useful in many areas of their education and specific capabilities to theirchosen discipline. The model assumes that many of the computational capabilities that a studentdevelops and applies will be in a problem-solving context. It is also assumed that as studentsmove through their undergraduate program and into workplace settings, these problem-solvingscenarios will become increasingly ill-defined and complex12, 13, requiring the Proficiency levelof computational capability. It is important to note that the assumption (based on feedback fromthe industry panel) is that few students will develop capabilities at the Fluency level prior toembarking on a professional engineering career. The levels are:

COMPUTATIONAL THINKING 7

Table 2: Levels of Computational Capability

Level DescriptionCompetency The individual has technical skill mastery of certain computational tools

and/or programming languages. Limits in conceptual knowledge meansthat they are limited to solving well-defined tasks with specified tools.When faced with a more open-ended or complex problems, limits inconceptual knowledge will mean they will probably not be able to solvethe problem.

Proficiency The individual has some conceptual knowledge of both computingsystems and their application domain. When presented with a problem,they are able select the appropriate tools(s), seek the necessaryinformation, and present a solution. The regularly used technical skillsare committed to memory and external information resources are notneeded in these cases. More complex problems and problems withmultiple possible solution paths for which they have to evaluate thequality of the different solution paths will create difficulties for theindividual. Overall intellectual capability may be a limiting factor.

Fluency The individual has extensive knowledge of the technical tools andconceptual aspects of both computer systems and the application domainof their profession. Within their professional area, they are able designand evaluate multiple solution paths to complex problems. They are wellversed in general knowledge in the problem space and do not need torefer to external resources for common problems. New computing toolsare readily evaluated and integrated into their existing tool set. Limits toproblem-solving usually result from moving outside their professionalapplication domain or the bounds of general intellectual capabilities.

2.2 Delphi Round 1

The first round of a Delphi study involves posing a set of open ended questions to a group ofexperts. 14 A detailed description of the Delphi process we used is described in last year’sconference paper 15. In summary, our experts were engineering line managers who directlysupervise new engineering hires in a company, including graduates from our university. Usingthe initial industry workshop and the model presented above as the framework, we generated sixopen-ended questions regarding the competencies, proficiencies and fluencies that industryexpects of (a) new engineering hires and (b) after a few years on the job. We requested theDelphi participants to answer these questions, providing as much detail as possible.

The survey responses were content analyzed by three researchers who read all responses,generated a coding list, coded all responses, checked for inter-rater agreement, sought consensusif there was any disagreement about coding statements, generated a set of themes and thenexplored the relationships among themes (see last year’s conference paper 15 for more detail).A cluster diagram was created to summarize these relationships (Figure 3).

COMPUTATIONAL THINKING 8

2.3 Delphi Round 2

For the second round of the Delphi, the themes generated in the Delphi Round 1 were used todevelop a Likert scale survey, which was completed by the respondents from the first Delphi.The survey contained three sections, where the first section included three personal identificationquestions (i.e., respondent name, job title, and company name), the second section had 12questions pertaining to the computing capabilities of new hires, and the third part had 14questions pertaining to computing capabilities expected during the first years on the job. Eachresponse was assigned a value (1 = not important, 2 = slightly important, 3= average importance,4 = important, 5 = very important) and the mean rating given by all respondents was calculatedfor each question. Responses with a mean value higher than 4.0 and with a standard deviationless the 1.0 indicate a high level of consensus among participants about the importance of thatparticular item, and therefore represent the results of the second round of the Delphi process.Responses that do not meet these criteria are excluded from the results because the purpose ofthe Delphi is the establishment of consensus.14

Results were further analyzed by type of engineering industry, with computer science, electricaland computer engineering, and information technology and engineering computer servicesresponses being aggregated separately to the responses from other industries that includedchemical engineering, textile engineering, mechanical engineering civil engineering, chemicaland biomolecular engineering, and industrial and systems engineering. This decision to analyzethe computer science and engineering responses separately came out the first round of the Delphianalysis and the conclusion that these engineering areas had unique computational capabilityneeds.

2.4 Student Surveys

During the seminar series, the faculty fellows were exposed to the theoretical framework (Figure1 and Table 2) and the Delphi results. The industry needs along with other information aboutcomputing innovations were delivered to inspire these faculty members to consider appropriateclassroom pedagogical changes that would enhance the computational capabilities of theirstudents. In courses where instructional changes have been implemented, students were recruitedto take a pre- and a post-test survey. The survey construction was based on the themes thatemerged from the first round of the Delphi and contained five sections. In the first section of thepre-survey, students rate the frequency of use of particular computing processes/tools, thinkingabout the semester prior to the current one, while in the second section they rate their level ofskill in using those processes/tools in the previous semester. In the third section, students ratethe perceived value of those processes/tools to their learning in the previous semester. The fourthsection contains attitudinal questions about learning and about computer tools. The fifth sectionasks demographic questions. The post-test contains the same items, but in this instance, studentsare asked to answer the questions as they pertain to the “enhanced computational capabilities”course in which they are enrolled. The means ratings of the pre- and post-survey were calculatedand compared using a matched samples t-test.

Each faculty fellow that has participated in the seminar series has undertaken some instructionalchange in at least one course that s/he is teaching in order to enhance the computational

COMPUTATIONAL THINKING 9

capabilities of their students, with each engaging in a parallel action research project. Thedetailed results from each action research project undertaken, though beyond the scope of thispaper, will be reported in the future.

3.0 Results

A total of 19 participants in the first round of the Delphi represented six different engineeringdisciplines, with work experience ranging from 1 to 20 years and with the position from first-lineengineers to senior managers. Figure 3 shows the five overlapping clusters of themes thatemerged from our analysis of open ended responses to the Delphi Round 1 survey.15 They were:

• Computer Science• Data Analysis• Design Modeling and Simulation• Core Individual Work Skills• Meta-Project Level

Each theme contains a set of sub-themes which are explained in Appendix 1.

Figure 3: Cluster diagram of emerging themes

COMPUTATIONAL THINKING 10

We found that there was a core set of computing skills common to all engineering disciplines.For instance, across the Computer Science cluster, there were Core Individual Work Skills (e.g.,Microsoft Office Applications) common to all engineering disciplines and themes related toengineering problem-solving. While the specific tools varied, all disciplines needed to manageengineering data as part of their problem-solving processes (e.g., Database Management,Database Fundamentals). Meta-Project Level themes were also common across disciplines andrelate to connecting engineering design work to other aspects of the companies’ work. Many ofthese “soft skills”, while not directly related to engineering problem-solving, have long beenrecognized by engineering educators as key capabilities valued by the engineering profession.

As noted above, for computer science and computer and electrical engineering, much of the day-to-day computational work stayed within the Computer Science cluster. However, otherengineering areas weighed heavily on themes that appeared in both the Design Modeling andSimulation and Data Analysis. Data Analysis held themes that were related to both ComputerScience capabilities and somewhat more indirectly related to the design process, such asmanaging engineering data used in the decision-making process.

3.1 Delphi 2

In order to answer the first research question: whether the Delphi process employed haseffectively captured the computational capability expectations of future employers ofengineering graduates, a total of 22 respondents in the second round of the Delphi representednine different engineering disciplines from different organizations. Of 12 questions pertaining tocomputing capabilities of new hires, there was no response with a mean value higher than 4.0and with a standard deviation less the 1.0, indicating no consensus among participants about theimportance of the computing capability was achieved (see Table 3). However, of 14 computingcapabilities expected to develop during the first years on the job, there was consensus (asindicated by a mean about 4.0 and standard deviation less than 1.0) that Ability to Learn andAdaptability, Microsoft Office Tools, Industry-Specific Tools, and Data Analysis Skills wereconsidered as important skills to develop (see Table 4).

To analyze responses by type of engineering industry, the responses from the computer scienceand engineering industries indicated that Basic Knowledge of Programming was an importantcomputing ability to new hires (see Table 3) while Proficiency in Programming Languages,Ability to Learn & Adaptability, Basic Knowledge of Programming, Integrated View ofSystems/Applications, and Web Programming & Language were expected to develop during thefirst few years on the job (see table 4).

The results of the responses from all other industries indicated that Microsoft Office Tools andWeb Search were considered as important skills to new hires (see Table 3) while MicrosoftOffice Tools, Ability to Learn & Adaptability, Industry-Specific Tools, Data Analysis Skills, andCommunication were capabilities expected to develop during the first years on the job (see table4).

COMPUTATIONAL THINKING 11

Table 3: Computational capabilities expected for new hires

All (N= 22) CSC Only (N= 8) Non-CSC (N=14)

Computing Capability Rank M SD Rank M SD Rank M SD

Basic Knowledge of Architectures 12 2.59 1.26 4 3.50 1.20 9 2.07 1.00

Basic Knowledge of Programming 4 3.64 1.29 1 4.63 0.52 7 3.07 1.27

Communication Tools/Organization 6 3.32 1.36 8 2.63 1.41 4 3.71 1.20

Data Analysis Skills 3 3.86 1.13 4 3.50 1.07 3 4.07 1.14

Database Fundamentals 9 3.00 1.11 7 2.88 1.25 6 3.07 1.07

Microsoft Office Tools 2 4.09 1.19 5 3.13 1.25 1 4.64 0.63

Process Modeling & Design 10 2.86 1.52 9 2.00 1.20 5 3.36 1.50Proficiency in ProgrammingLanguages 5 3.41 1.30 6 3.06 1.25 7 2.79 1.19

Queries Debugging/Testing 7 3.23 1.41 2 4.00 1.31 7 2.79 1.31

Software Systems Design 11 2.77 1.41 3 3.63 1.19 8 2.29 1.33

Web Programming & Language 8 3.05 1.17 4 3.50 0.76 7 2.79 1.31

Web Search 1 4.23 1.23 3 3.63 1.77 2 4.57 0.65

Table 4: Computing capabilities expected to develop during the first few years on the job

All (N= 22) CSC Only (N= 8) Non-CSC (N=14)Computing Capability Rank M SD Rank M SD Rank M SDAbility to Learn & Adaptability 1 4.68 0.57 1 4.75 0.46 2 4.64 0.63Basic Knowledge ofArchitectures 10 2.91 1.60 4 4.13 1.13 12 2.21 1.42Basic Knowledge ofProgramming 5 3.59 1.37 3 4.50 0.76 8 3.07 1.38CommunicationTools/organization 4 3.86 1.13 8 3.13 1.25 4 4.29 0.83Data Analysis Skills 3 4.18 0.80 6 3.63 0.92 3 4.50 0.52Database Management 12 2.73 1.08 10 2.38 0.92 9 2.93 1.14Industry-specific Tools 3 4.18 0.85 6 3.63 0.92 3 4.50 0.65Integrated View ofSystems/Applications 9 3.23 1.41 4 4.13 0.83 11 2.71 1.44Microsoft Office Tools 2 4.27 0.94 7 3.50 1.07 1 4.71 0.47Process Modeling & Design 7 3.32 1.62 11 2.63 1.51 5 3.71 1.59Proficiency in ProgrammingLanguages 6 3.45 1.50 2 4.75 0.46 11 2.71 1.38Proficiency in Simulations 11 2.82 1.30 12 2.13 1.25 7 3.21 1.19Project ManagementApplications 8 3.27 1.20 9 2.88 1.13 6 3.50 1.22Web Programming & Language 8 3.27 1.45 5 4.00 0.93 10 2.86 1.56

COMPUTATIONAL THINKING 12

3.2 Student Surveys

To answer the second research question in this project (whether the previously developedinstructional model of computational capabilities can be used to characterize the experiences,abilities, and the perceptions of engineering students along dimensions of interest to futureemployers), a matched sample of 92 students completed the pre- and post-test survey. Most werebetween the ages of 19 to 21 years of age (78.9%), male (82.4%), and white (84.9%; 5.8% wereAfrican-American). Students (35.2% sophomore; 40.7% junior; 24.2% senior) were recruitedfrom five courses (i.e., GC 120, CHE 205, MAE 206, MAE 308, and TE 303), with the mean ofGPA of 3.27 (SD = 0.05).

The results indicated that, on average, students reported that they applied or were exposed to twocomputing capabilities, Web Search and Communication Tools/Organization, more than once aweek during the class in the previous semester (spring 2009).. However, in the next semester(fall 2009) students reported significantly less usage of computing abilities related to websearches, communication tools, and basic knowledge of programming in the classes surveyed(see Table 5).

Table 5: The frequency students applied the computing capabilities during the class

Pre-test

(Expected)Post-test(Actual)

Computing Capability M SD M SDDatabase Fundamentals 1.84 0.99 1.74 1.01Process Modeling & Design 2.01 1.27 1.96 1.21Basic Knowledge of Programminga 1.95 1.23 1.36 0.64Data Analysis Skills 2.85 1.14 2.63 1.25Communication Tools/Organizationa 4.11 1.23 3.39 1.47Web Searcha 4.69 0.73 3.39 1.45

Note. Items are rated on a 5-point scale, ranging from 1 (Never) to 5 (Daily). a indicates pre- andpost-test means are significantly different from each other at p < .05.

In terms of self-reported skill level of the computing capabilities, students initially ratedthemselves being competent in Communication Tools/Organization and being proficient inconducting Web Searches in the semester prior to taking the enhanced computational capabilitiescourse. However, in the semester of taking the enhanced computational capabilities course theself-rating of the web search skill significantly decreased while the rating of the abilities relatedto database fundamentals, basic knowledge of programming, and data analysis skills increasedsignificantly (see Table 6).

COMPUTATIONAL THINKING 13

Table 6: The self-reported skill level in applying the computing capabilities during the class

Pre-test Post-testComputing Capability M SD M SD

Database Fundamentalsa 2.11 1.07 2.34 1.11Process Modeling & Design 2.43 1.31 2.59 1.19Basic Knowledge of Programminga 1.82 1.01 2.07 1.15Data Analysis Skillsa 2.51 1.01 2.95 1.09Communication Tools/Organization 3.75 1.02 3.64 1.12Web Searcha 4.43 0.73 4.10 1.08

Note. Items are rated on a 5-point scale, ranging from 1 (Novice) to 5 (Expert). a indicates pre-and post-test means are significantly different from each other at p < .05.

As regards the perceived value of different computing capabilities in facilitating their collegeeducation, students gave high value to skills related to Data Analysis Skills, CommunicationTools/Organization, and Web Search at the beginning of the semester. However, at the end of thesemester the perceived value of these six computing capabilities in facilitating the student'scollege education significantly decreased for all six dimensions (see Table 7).

Table 7: The perceived value of the computing capabilities in facilitating students' collegeeducation Pre-test Post-test

Computing Capability M SD M SDDatabase Fundamentals 2.46 0.89 2.08 0.99Process Modeling & Design 2.88 0.98 2.24 1.11Basic Knowledge of Programming 2.25 0.84 1.79 0.86Data Analysis Skills 3.19 0.79 2.76 1.04Communication Tools/Organization 3.54 0.76 3.08 1.02Web Search 3.67 0.56 3.10 1.06

Note. Items are rated on a 4-point scale, ranging from 1 (Not at all) to 4 (Very valuable). Thepre- and post-test means of all computing capabilities are significantly different from each otherat p < .05.

The authors also examined the impacts of the course intervention on (a) students' self-efficacyabout learning in the discipline of engineering/computer science (9 questions), and (b) on theself-efficacy of using computers (7 questions). The results showed no change on any of thequestions after the course interventions and so all the responses for each of (a) and (b) wereaggregated in Table 8 below

COMPUTATIONAL THINKING 14

Table 8: Self-reported engineering/computing self-efficacy and computer self-efficacy Pre-test Post-test

Scale M SD M SD(a) Engineering/computer science self-efficacy

32.38 4.73 32.73 4.36

(b) Computer use self-efficacy 27.67 4.23 27.46 3.84Note. Items are rated on a 5-point scale, ranging from 1 (Strongly Disagree) to 5 (StronglyAgree). There were no significant differences in pre- and post-test results.

4.0 Discussion

We will now respond to the two research questions initially posed:

1. Has the Delphi process employed in the project effectively captured the computationalcapability expectations of future employers of engineering graduates?

With respect to the second research question, the variations among respondents and thedifferences between the six computing capabilities framed in the student survey seem to indicatethat the Computational Capabilities Instructional Model can be used to characterize theexperiences, abilities, and the perceptions of engineering students. Students self-rate their skilllevels for Database Fundamentals and Basic Knowledge of Programming in the advancedbeginner range, indicating they are still developing competency in these areas. Process Modeling& Design and Data Analysis Skills are rated in the competent range, while they rate themselvesin the proficient range for Communication Tools/Organization and Web Searches in theproficient range. However, the relatively small difference in scores between the pre- and post-tests, for all six computing capabilities, indicates that the shift towards increased competency,proficiency or even fluency does not occur in a single semester. Indeed, there may even be aself-reported backward movement in their development, as seen in the case of Web Searchcapability. The movement towards greater computational ability likely occurs across a range ofcourses during the student's program, where the different computational capabilities are used andexpanded upon at successive levels in the curriculum.

Consensus on the expectation of some computing capabilities had been reached both for newhires and for employees having worked for few years, for either the computer science andengineering industries or the non-computer science industries. It seems that the Delphi processemployed in this project has effectively captured the computational capability expectations offuture employers of engineering graduates. Additionally, the different computational capabilityexpectations in different types of engineering industry demonstrated that these engineering areashad unique computational capability needs.

The only “general job skill” not tied to computational capabilities that was included on thesurvey had to do with “Ability to Learn and Adaptability,” in part because so many respondentson Round 1 of the Delphi brought this qualification up. Not surprisingly, there was alsoconsensus in Round 2 on the importance this soft skill. Perhaps, not surprisingly, there as anexpectation that Industry-specific tools would be learned on the job and that they would bringwith them to their new job general abilities to use MS Office and Web search tools. There was

COMPUTATIONAL THINKING 15

also consensus of the importance of data analysis tools the first years on the job, but this was nota consensus as a prior skill learned in school. Perhaps these were considered to be industry-specific. However, there was consensus again for non-computer science jobs around dataanalysis to be learned on the job. Perhaps this is less valued in some computer science areas. Notsurprisingly, basic programming was important for new hires into computer science jobs withhigher level proficiencies being developed on the job. Interestingly, there was not consensusabout basic programming for new hires in non-computer science industries.

2. Can the previously developed instructional model of computational capabilities be used tocharacterize the experiences, abilities, and the perceptions of engineering students alongdimensions of interest to future employers?

With respect to the second research question, the variations among respondents and thedifferences between the six computing capabilities framed in the student survey seem to indicatethat the Computational Capabilities Instructional Model can be used to characterize theexperiences, abilities, and the perceptions of engineering students.

The survey results shows alignment between industry expectations and student perceptions aboutthe importance of Microsoft Office/Communication tools and web searching tools. Studentsseem to come into their classes with both high ratings concerning how often they usedcommunication, web, and programming tools and what their perceived abilities were in theprevious semester. On one hand, they used some of these tools less than they did in the previoussemester, but their use of them perhaps demonstrated that they didn’t know as much as theythought they did.

The lower perceived values of the six computing capabilities and the higher self-reported skilllevel of the computing capabilities in the semester of taking the enhanced computationalcapabilities course may demonstrate that the value of learning a particular computing skill/tooldecreases as the corresponding skill level increases. Of note, in the area of more specializedengineering tools (database, programming, data analysis), the courses seemed to be successful inproviding enhanced (self-reported) skills. It also seems that the classroom pedagogical changesmay have impacted students' self-efficacy about learning in the discipline ofengineering/computer science and computer use self-efficacy in the long term, rather than onesemester. The decidedly higher self-efficacy in engineering computing from computer use ingeneral is one that would need to be probed with interviews with students.

5.0 Recommendations and Future Work

The Delphi process has provided a robust benchmark against which pedagogical innovations inenhancing computational capabilities can be developed. Clearly more than one semester with arelatively small student sample will be needed to see if the innovations explored by our CACFellow are effective. However, our model of computational capabilities has provided a usefulframework for providing formative and summative data on the efficacy of our experimentation.

The model of computational capabilities plus the benchmarks established by the Delphi providesa mechanism for beginning to flesh a progression of increased skills and knowledge over the

COMPUTATIONAL THINKING 16

undergraduate years. As more classroom experiments are conducted and more feedback fromfaculty elicited, progressions that not only differentiate between years, but also disciplinary areaswill provide rich, detailed data around which strategic instructional decisions can be made. Acrucial feature to these progressions is the emphasis for the need of continuously, throughouteach year and all core classes, for faculty to be focused on using and developing thesecomputational capabilities. Ongoing data collection from students and future Delphi rounds willfeed back into the progression models as industry needs continue to evolve.

COMPUTATIONAL THINKING 17

Acknowledgements

This work is supported by NSF (CISE # 0722192) as part of CISE Pathways toRevitalized Undergraduate Computing Education program. The project team would also like toextend its sincere thanks to our partners in industry who served on our panels and our CACfellows who are implementing their innovations in their classrooms.

Bibliography

1. Levy, F., & Murnane, R. J. (2004). The new division of labor: How computers are creating the next job market.Princeton, NJ: Princeton University Press.

2. Vest, C. M. (2008). Context and Challenge for Twenty-First Century Engineering Education. Journal ofEngineering Education, 97(3), 235-236.

3. NRC, National Research Council. (2008). Research on Future Skill Demands: A Workshop Summary.Washington, DC: Center for Education, Division of Behavioral and Social Sciences and Education.

4. Gonzales, R. F., & Renshaw, S. (2005). Connected Computing - A Hierarchy of Pre-Engineering ComputingSkill Competencies. Paper presented at the Annual Meeting of the ASEE.

5. Davison, L., & Porritt, N. (1999). Using computers to teach. Proceedings of the Institution of Civil Engineers-Civil Engineering, 132(1), 24-30.

6. Kassim, H. O., & Cadbury, R. G. (1996). The place of the computer in chemical engineering education.Computers & Chemical Engineering, 20, S1341-S1346.

7. Linstone, H. A., & Turoff, M. (1975). The Delphi method: Techniques and applications. Downloaded 1/4/2008from http://is.njit.edu/pubs/delphibook/

8. Authors. (2008). Paper presented at the ASEE Annual Conference, Pittsburgh, PA.9. NRC-CITL, National Research Council - Committee on Information Technology Literacy. (1999). Being fluent

with information technology. Washington, DC: National Academies Press.10. Dougherty, J. P., Dececchi, T., Clear, T., Richards, B., Cooper, S., & Wilusz, T. (2003). Information Technology

Fluency in Practice. ACM SIGCSE Bulletin, 35(2), 153-171.11. Atman, C. J., Adams, R. S., Cardella, M. E., Turns, J., Mosborg, S., & Saleem, J. (2007). Engineering Design

Processes: A Comparison of Students and Expert Practitioners. Journal of Engineering Education, 96(4), 359-379.

12. Stevens, R., O'Connor, K., Garrison, L., Jocuns, A., & Amos, D. M. (2008). Becoming an Engineer: Toward aThree Dimensional View of Engineering Learning. Journal of Engineering Education, 97(3), 355-368.

13. Strobel, J., & Cardella, M. (2008). Compound Problem Solving: Workplace Lessons for Engineering Education.Paper presented at the ASEE Annual Meeting, Pittsburgh, PA.

14. McKitrick, S. (2007). Assessing ‘ineffable’ general education outcomes using the Delphi approach. Paperpresented at the NCSU Assessment Symposium, Cary, NC.

15. Authors. (2009). Paper presented at the ASEE Annual Conference.

COMPUTATIONAL THINKING 18

Appendix 1: Finalized Theme Categories from the 1st Delphi Survey

Theme CodeAnalyze & Evaluate Existing Process AEEPAbility to Learn & Adaptability ALAAbility to Use Simulation Packages AUSPBasic Knowledge of Architectures BKABasic Knowledge of Programming BKPBasic Operation System BOSCommunication Tools/organization COTData Analysis Skills DASDriver Concept DCDatabase Fundamentals DFDatabase Management DMForecasting FFinancial/Interdisciplinary Knowledge FIKGeneral: Teamwork, Problem solving (not computingcompetencies) G

Industry-specific Tools ITIntegrated View of Systems/Applications IVSAKnowledge of Architectures KAMicrosoft Office Tools MOTNone/Not relevant NProject Management Applications PMAProcess Modeling & Design PMDProblem-solving & Problem-shooting PPProficiency in Programming Languages PPLProficiency in Simulations PSQueries Debugging/Testing QDTSecurity Control SCSoftware Systems Design SSDVirtualization VWeb Programming & Language WPLWeb Search WS