Programming Assistance Tools

Computers & Education 82 (2015) 162e178Contents lists available at ScienceDirectComputers & Education

journal homepage: www.elsevier .com/locate/compeduProgramming assistance tools to support the learning of ITprogramming in South African secondary schools

Melisa Koorsse*, Charmain Cilliers 1, Andre Calitz 2

Nelson Mandela Metropolitan University, P.O. Box 77000, Port Elizabeth, 6031, South Africaa r t i c l e i n f o

Article history:Received 25 August 2014Received in revised form17 November 2014Accepted 26 November 2014Available online 3 December 2014

Keywords:Programming and programming languagesInteractive learning environmentsSecondary educationEvaluation of CAL systems* Corresponding author. Tel.: 27 41 504 3925.E-mail addresses: [email protected] (M

1 Tel.: 27 41 504 2639.2 Tel.: 27 41 504 2235.

http://dx.doi.org/10.1016/j.compedu.2014.11.0200360-1315/ 2014 Elsevier Ltd. All rights reserved.a b s t r a c t

Novice programmers struggle to understand introductory programming concepts and this difficulty,associated with learning to program, contributes mainly to the lack of interest in the field of ComputerScience at tertiary level. Programming assistance tools have been used to assist novice programmersextensively at education institutions. A programming assistance tool (PAT) is a software program that canbe used by novice programmers to learn how to program and/or improve their understanding of pro-gramming concepts.This research proposes that novice programmers, specifically Information Technology (IT) scholars inSouth African secondary schools, could be supported by PATs. The main objective of this research was todetermine whether the use of a PAT impacted IT scholars' understanding of programming concepts andtheir motivation towards programming in general. Criteria for the selection of PATs were derived fromthe programming difficulties identified in literature and from surveys to IT teachers and scholars. Theselection criteria were grouped into programming knowledge and programming skills categories.Existing PATs were evaluated using the selection criteria and three PATs, namely, RoboMind, Scratch andB#, were selected for evaluation in this research study. A convenience sample of schools participated inthe study. The three PATs provided different approaches while being able to support the Delphi pro-gramming language used in schools that participated in the study.The findings of this research indicated that, although scholars perceived the PATs to be useful in theexplanation of certain of the programming concepts, there was no conclusive evidence that IT scholarswho used a PAT had a significantly better understanding of programming concepts and motivation to-wards programming than scholars who did not use a PAT. Participant feedback was used to identify thestrengths and shortcomings of the three PATs and to provide recommendations for the development offuture PATs specifically designed to support IT scholars.

2014 Elsevier Ltd. All rights reserved.1. Introduction

Information and Communication Technology (ICT) related professions have been identified as top professions due to the increase in thenumber of companies developing web, mobile and cloud computing applications (Strieber, 2011). Despite a great demand for ICT qualifi-cations, countries around the world, including the U.S. (Hug, Guenther, & Wenk, 2013; Wilson, Sudol, Stephenson, & Stehlik, 2010), Eu-ropean countries (Porta, Maillet,&Gil, 2010) and South Africa (Alexander& Twinomurinzi, 2012; Calitz, 2010), have experienced a decline inthe number of enrolments in Computer Science (CS) programs at tertiary level education institutions. This decline in enrolments hasresulted in a decrease in the number of software developers available to industry. Globally countries have identified the importance of ICTprofessions for technological and economic growth of the country (Wilson et al., 2010). The importance of promoting ICT related skills andknowledge at school level has also been recognised (Wilson et al., 2010).. Koorsse), [email protected] (C. Cilliers), [email protected] (A. Calitz).

Delta:1_given nameDelta:1_surnameDelta:1_given namemailto:[email protected]:[email protected]:[email protected]://crossmark.crossref.org/dialog/?doi=10.1016/j.compedu.2014.11.020&domain=pdfwww.sciencedirect.com/science/journal/03601315http://www.elsevier.com/locate/compeduhttp://dx.doi.org/10.1016/j.compedu.2014.11.020http://dx.doi.org/10.1016/j.compedu.2014.11.020http://dx.doi.org/10.1016/j.compedu.2014.11.020

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 163In South Africa, ICT related skills and knowledge including software development are included in the subject matter of the InformationTechnology (IT) subject in South African secondary schools for Grade 10e12 scholars. IT in South African secondary schools is perceived asbeing a difficult subject (Havenga &Mentz, 2009). This reputation impacts on scholars' decisions to take IT at school. Many scholars that doattempt the subject change to another, easier subject before Grade 12 (the final year) or remain in the subject class but lack the motivationand interest in the programming content to achieve their full potential. The lack of interest and motivation in the subject and negativeimpression created amongst scholars seems to affect negatively the number of scholars who decide to pursue further education in acomputing discipline.

IT scholars are faced with the same difficulties of learning to program as that experienced by novice programmers in general (Havenga&Mentz, 2009). These difficulties are aggravated by the secondary school learning environment (Koorsse, Calitz & Cilliers, 2010). Short(approximately 45 min), daily lessons make it difficult both to teach and practise complex programming concepts and skills during limitedschool hours. There is consequently inadequate time available for IT teachers to address the specific programming difficulties of individual ITscholars as well as accurately assess scholar understanding of concepts during class time.

IT scholars further lack appropriate support in the form of resources to supplement the prescribed IT subject textbook. Programmingassistance tools (PATs) are one such supporting resource (Pears et al., 2007). PATs are designed specifically for novice programmers in orderto enhance comprehension of algorithms and computer programs, assist with code debugging and/or assess programming knowledge andskills (Kelleher & Pausch, 2005). PATs support self-study (Al-Imamy, Alizadeh, & Nour, 2006) and promote interaction (Baldwin & Kuljis,2000). Research studies have evaluated the use of a PAT as part of the teaching environment such as Scratch (Malan & Leitner, 2007)and a web-based tool (Al-Imamy et al. 2006). However, this study aims to evaluate the impact of PATs provided as part of self-regulatedlearning. It is also not clear from literature if there are specific PATs that could be used to promote the understanding programming con-cepts and skills that are required specifically for IT scholars. None of the PATs identified in literature have been evaluated in terms of theprogramming knowledge and skills supported or the PATs can be used to support the learning of IT programming in South African (SA)secondary schools.

This research study was important in identifying the programming knowledge and skills required by novice programmers in general(Section 2), and specifically IT scholars in SA schools (Section 2.2). This resulted in a list of selection criteria that indicated the programmingskills and knowledge that should be supported by a PAT (Section 2.2). Literature identifies many PATs that have been developed for use bynovice programming subject (Section 3) with different programming environments. However, using the selection criteria, this researchstudy was able to select PATs that met the requirements of IT programmers (Section 3). The research study then evaluated whether or notPATs could have an impact on IT scholar understanding of programming concepts and thus whether or not PATs could support the teachingand learning of IT programming in SA schools (Section 6). Three PATs were selected for evaluation using the selection criteria identified. Thethree PATs were RoboMind (Section 3.2.1), Scratch (Section 3.2.2) and B# (Section 3.2.3). The following research questions were addressedby the study to determine the impact of these PATs:

RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?RQ2: What impact do the PATs have on IT scholar perceived difficulty of programming?RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specific programming concepts?

The analysis of the results related to the research questions provided feedback on the impact, if any, of the different PATs on the un-derstanding of programming concepts from a scholar's perspective and in terms of formal assessment (Section 5). Certain of the pro-gramming concepts such as loops were better supported by PATs and supported scholar understanding of these concepts (Section 5).However, overall there was no evidence to support that scholars using the PATs had a significantly better understanding of programmingconcepts than scholars who did not. Qualitative feedback indicated the importance of the PAT interface, in particular that the interfaceshould not be too simple. The results and findings of the research study provide further criteria and design guidelines for PATs to bedeveloped to support the learning of IT programming in SA schools, and programming in general (Section 6).

2. PAT selection criteria

A dependence relationship exists between the different types of knowledge required by a novice programmer (syntax, computers,programming principles and programming concepts) and code comprehension and generation (Fig. 1). Novice programmers need to un-derstand that code generation involves three sequential steps:

1. Studying a given problem statement or set of requirements and deciding on the best programming strategy to use. This skill requires anunderstanding of the programming principles that can be applied as well as knowledge of how a computer executes code in order toformulate a solution.

2. Producing an algorithm to solve the problem. The algorithm will often be formulated using pseudo code. Knowledge of programmingprinciples and concepts is required in order to select appropriate data types and structures or apply concepts such as looping or se-lection, for example. Knowledge of computer execution of code would assist programmers to efficiently and effectively order thesequence of algorithm instructions.

3. Translating the algorithm into the code of the programming language being used requires knowledge of the programming language syntax.The program code is tested and changed until the program meets the original set of requirements and thus solves the problem.

The development of programming strategies in terms of applying programming knowledge correctly when formulating a solution to aprogramming problem is important. Different knowledge and skills that are important in the programming process (Fig. 1). A particular PATmay not address all these areas, instead supporting only a few areas. Surveys were administered to IT scholars and teachers to identify thedeficiencies in the skill and knowledge areas required by a programmer (Section 2.2). By identifying skill and knowledge areas that needattention for novice programming (Section 2.1) and particularly in IT schools (Section 2.2), selection criteria for PATs were formulated

Fig. 1. Knowledge and skills required by a programmer.

Table 1Criteria for selecting PATs identified from introductory programming literature.

Category Criteria item

Programming knowledge Assist with the understanding of code executionAssist with the learning of programming language syntaxAssists with developing knowledge of programming principles and conceptsProgramming environment suitable for novice programmersAssists with the application of programming knowledge (programming strategies)

Programming skills Provides simple error message to assist with debuggingPromotes problem solvingDevelops code comprehension

M. Koorsse et al. / Computers & Education 82 (2015) 162e178164(Table 2) so that appropriate PATs could be identified and selected for use by IT scholars (Section 3). These PATs should then be able toaddress the deficiencies identified.

2.1. Novice programming knowledge and skills required

Specific programming knowledge and programming skills are required by novice programmers in order to program successfully. Aspectswhich influence programming ability are the application of knowledge and strategies and the relationship between skills such as codecomprehension and code generation (de Raadt, 2008; Robins, Rountree, & Rountree, 2003). Programming knowledge and strategies aredistinct but related (de Raadt, 2008). Programming knowledge relates to the body of information describing three areas:

1. Programming concepts and principles (for example: how a for-loop works or the purpose of a variable),2. Knowledge of computers (for example: how an event is generated) and3. Programming language knowledge or syntax.

If a programmer is not aware of concepts such as loops, data structures or objects, it would be difficult for the programmer to plan asuitable solution. Knowledge of the syntax is required to implement a solution in a particular programming language (Pears et al., 2007).Table 2PAT criteria derived from literature (Lit) and IT teacher (TF) and scholar feedback (LF).

Category Criteria for PAT Lit TF LF

Programming knowledge Assists with the learning of programming language syntax (Delphi in this research study) C CAssists with developing knowledge of programming principles and concepts C CConstructivist to promote self-study C CAssists with the application of programming knowledge C CAssists with the understanding of code execution C C

Programming skills Promotes problem solving and planning C C CProvides simple error messages to assist with debugging C C CDevelops code comprehension C CFeedback to guide solution creation C CFeedback regarding errors C

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 165Regardless of how well the solution is designed, if the syntax is incorrect, the program will not run successfully. Knowledge in all three ofthese areas is required for a novice programmer to be successful in programming.

Programming strategy refers to the way in which programming knowledge is applied to solve a particular problem (de Raadt, 2008;Robins et al., 2003). Programming strategies are required for the first step of code generation (Fig. 1). Novice programmers need assis-tance to develop both their programming knowledge and programming strategies. However, teachers generally spend more time teachingprogramming language knowledge than programming strategy (Al-Imamy et al., 2006; Pears et al., 2007). Novice programmers oftencombine steps 1 and 2 of the code generation process (Fig. 1) as they attempt to solve the algorithm in a particular programming language(Garner, 2007; Rongas, Kaarna, & Kalvianen, 2004). Lack of planning or lack of algorithm design leads to a programmer having difficulty insolving a given problem (Shuhidan, Hamilton, & D'Souza, 2009).

Novice programmers also have an inability to transfer their knowledge of problem solutions to similar problems because they lack theability to comprehend program code and have a fragile knowledge of programming principles. Their knowledge is not deep enough tounderstand how the principles can be applied to new or larger problems that are more complex. Related studies have indicated that noviceprogrammers lack problem solving ability (Heines & Schedlbauer, 2007; Lahtinen, Ala-Mutka, & Jarvinen, 2005; Lister et al., 2004; Robinset al., 2003). However, novice programmers may be capable of problem solving but unable to express solutions in a programming languageor in a manner that a computer would understand. This is supported by the finding that novice programmers have a non-viable mentalmodel of programming concepts which results in misconceptions and difficulties when trying to solve programming problems (Ma,Ferguson, Roper, Ross, & Wood, 2008).

In addition to programming knowledge, novice programmers also require the skill to use a programming language and work in aprogramming development environment. The programming language and environment used when novice programmers are learning toprogram can also impact their understanding of programming concepts. Certain programming languages are too complex to explain or useto teach some of the programming concepts taught to novice programmers (Kolling, 1999). A professional programming developmentenvironment may overwhelm the students by presenting them with functionality and interfaces not needed by novice programmers(Kolling, 1999; Levy, Ben-Ari, & Uronen, 2001; Pendergast, 2006). Furthermore, the compiler messages provided by professional pro-gramming development environments are directed at professional programmers (Rongas et al., 2004).

Program compilers ensure that programs are syntactically and semantically correct prior to execution. If an error is found in the code, acompiler message is displayed to the programmer describing the error and providing the location of the error in the code. Most compilermessages are too complicated and low-level and do not explain sufficiently well to a novice programmer which syntactical or semanticerrors have occurred. Novice programmers thus struggle to develop their skill to debug code.

Vague error messages such as the identification of incompatible data types do not provide programmers with information on how theerror can be corrected. In professional programming development environments, novice programmers struggle to understand compilererror messages intended for professional programmers with a better understanding of the programming environment and more pro-gramming expertise.

Acquiring the programming knowledge and skills identified above should be supported by PATs provided to novice programmers.Programming knowledge criteria would assist IT learner understanding of programming concepts and principles and programming skillscriteria would assist IT learner ability to perform programming related skills such as tracing code execution. PATs should assist scholars todevelop programming knowledge in computer execution of programming code, programming language syntax and programming princi-ples and concepts (Table 1). Addressing these areas together with methods of improving code comprehension should implicitly assist usersto improve problem solving ability. PATs should also support the development of programming strategies in terms of applying programmingknowledge.

PATs selected for use with scholars should promote self-study (Table 1) for the scholar to improve knowledge in spite of the difficultiesrelated to the teaching approach, programming language and development environment. PATs appropriate for use by IT scholars shouldpromote debugging by identifying errors in code and provide simple error messages that novice programmers can understand (Table 1).Table 1 lists PAT selection criteria to address general novice programming difficulties. A survey to IT teachers and scholars reflected theperceptions of the difficulties faced by IT scholars learning to program in South African secondary schools. The survey findings were used toconfirm the criteria for PAT selection from literature and included criteria that could specifically be used to select PATs to support thelearning of IT programming.

2.2. PAT selection criteria to address IT programming difficulties

Surveys were administered to participating IT teachers and Grade 11 IT scholars at four South African secondary schools in order toidentify the selection criteria for PATs to address programming difficulties experienced by IT scholars. The four schools participating were asample of convenience and only IT teachers at these schools were included in the study. IT scholars participating in this study were taughtthe Delphi programming language. Both questionnaire instruments were piloted to establish reliability and validity before administering toparticipants. A combination of quantitative and qualitative data analysis techniques were used.

The IT teacher questionnaire was designed to gather information regarding the methods used by teachers to teach programmingconcepts to their scholars and the teacher's perspective on difficulties experienced by scholars. The aim of the questionnaire to participatingIT scholars was to identify factors contributing to the difficulty of IT programming from an IT scholar perspective.

Feedback from IT teacher and scholar surveys confirmed the findings from literature and identified additional criteria to be applied whenselecting an appropriate PAT to support the learning of IT programming. The feedback from the surveys to IT teachers resulted in acomprehensive set of criteria as follows:

Promotes self-study in a constructivist way: IT teachers identified that the class lessons were short resulting in less time in class topractise programming with the assistance of the IT teacher. Scholars should thus be able to use a PAT without the assistance of theteacher to develop their knowledge and explore different programming concepts. This finding is confirmed by related studies (Areias &Mendes, 2007).

M. Koorsse et al. / Computers & Education 82 (2015) 162e178166 Develops knowledge of programming principles and concepts: Certain programming principles and concepts such as objects, data-structures and planning algorithms were identified as difficult for IT scholars to understand. PATs that provide an explanation ofprogramming concepts and principles can improve scholar understanding of concepts during self-study.

Assists with the application of programming knowledge: It was found that scholars struggle to apply programming knowledge and the PATshould provide support in the application of programming knowledge to different problems and exercises. An example would besupport in the form of scaffolding that assists the scholar to plan a coded or non-coded solution, as identified in related work (Wood,Bruner, & Ross, 1976). When the scaffolding is removed, the scholar should be able to perform the task independently, as identified inwork by Kunkle (2010).

Assists with learning syntax knowledge: IT teachers indicated that struggling scholars have difficulty with programming syntax.Improving syntax knowledge is important for novice programmers to produce a successful code solution. In this research study, PATswere required to assist IT scholars with the learning of the Delphi programming language syntax, particularly as scholars may not haveaccess to Delphi beyond the classroom.

Provides feedback regarding errors: PATs could benefit IT scholars who struggle to identify syntax errors, by providing feedback regardingerrors in the code or solution. Error feedback would assist scholars in the development of debugging skills.

Provides simple error messages: If feedback is provided, simple error messages should be used that are easy to understand and help toidentify the error in order to improve an IT scholar's debugging skills.

Promotes problem solving and planning: IT teachers identified problem solving and planning of solutions as skills that IT scholars havedifficulty understanding and applying. The PAT should help to promote these skills.

Provides feedback to guide solution creation: IT scholars need assistance to improve their ability to create algorithms. The PAT shouldassist scholars to convert a planned solution to a code solution that can be executed.

PAT selection criteria were also formulated from IT scholar survey feedback:

Assists with understanding of code execution: Assistance with code execution could allow scholars to learn how to trace a program andevaluate the state of variables at different points during the execution of the code. These criteria would assist scholars to improvedebugging skills as well as algorithm and code comprehension.

Provides simple error messages: Results indicated that scholars' understanding of compiler error messages rated lower than their use ofcompiler error messages. Simple error messages would allow scholars who use error messages to understand the error duringdebugging. Debugging is a skill scholars identified as difficult.

Develops code comprehension: Improving a scholar's ability to read and understand code could help scholars to evaluate their own codesolutions and assist with debugging. Code comprehension would also assist scholar understanding of common algorithms and how toadapt algorithms to different problems by transferring knowledge from solutions to similar problems.

Promotes problem solving and planning: Results indicated that scholars do not create solutions independent of code when planning asolution. The PAT should thus assist scholars to develop a program solution using diagrams or pseudo-code, including the provision offeedback during code execution. PATs should guide scholars to develop a suitable solution to the problem and to identify any problemsin the logic of the program solution.

Feedback to guide solution creation: The purpose of this criterion is to assist scholars to convert a pseudo-code program solution into acode solution implemented in a programming language, thus improving their ability to create code algorithms.

A comprehensive set of criteria (Table 2) incorporating characteristics evident in related studies as well as feedback from IT teachers andscholars was formulated. The selection criteria were then used to evaluate and select suitable PATs specifically for use in this study, whereparticipating IT scholars in South African secondary schools would independently use and evaluate one of the selected PATs.

3. Programming Assistance Tools (PATs)

PATs are specifically designed for use by novice programmers to support their understanding of programming knowledge and assist withthe development of programming skills. The programming environment provided by a PAT is simpler to use than professional programmingenvironments, even supporting the learning of programming using visualisation techniques. The visualisation of programming conceptssupports novice programmers' understanding of abstract concepts, using interactive microworlds to make programming more interestingand applicable (Pears et al., 2007).

An advantage of using a PAT is that it can assist a novice programmer to develop an understanding of programming concepts as wellas provide automated assessment of programming tasks and correction of simple errors (Rongas et al., 2004). Other advantages are thatPATs can promote interactivity as well as support self-study, providing novice programmers with different pedagogical learningmethods.

PATs can employ visualisation techniques to enhance algorithm comprehension, improve debugging ability and guide exploration tounderstand concepts (Pears et al., 2007). The structure or execution of code can be visualised or a programming concept can be animated.There are PATs that have environments that allow novice programmers to construct algorithms and visualisations graphically, usingflowcharts or drag-and-drop techniques (Kelleher & Pausch, 2005) and thus prevent novice users from making syntax errors. Structuredediting, where elements are specified from a menu when typing the program, can also be used (Guzdial, 2004) where placeholders indicateto users where additional code should be specified, such as variable names or values. Structured editing assists users by providing validconstructs, thus allowing users to develop an executable solution, although logical errors may exist.

Microworlds can be used to make the understanding of programming more concrete as users can view the result of programming so-lutions as actions by characters or objects in a visual and/or animated environment (Kelleher& Pausch, 2005). General-purpose, professionalprogramming languages can be abstract and difficult to understand. Microworld environments motivate users to want to learn as theenvironment is perceived as having a fun element (Kelleher & Pausch, 2005).

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 167PATs use animation and/or visualisation in different ways. A solution can be created by typing code while the resulting program can beanimated, such as a robot performing actions in a 3D world. Alternatively, graphical objects can be used to create the program solution eventhough the running program may not have any animations nor make use of a graphical user interface.

A PAT that uses visualisation techniques may not assist novice programmers. In some cases novice programmers that use a PAT do notperform better than those not using it (Levy et al., 2001). The visualisation techniques used by a PAT may not be effective when the scholarsare not able to accurately and successfully map programming concepts to elements in the animation. PATs may also use animations thatrequire supplementary explanations from the teacher in order to be successful. However, scholars using a PAT with animations reportincreased satisfaction and motivation compared to scholars who do not use a PAT (Levy et al., 2001).

The majority of PATs evaluated for selection in this research study were identified from a study of related work. The PAT selection criteriawere used to evaluate 10 PATs, from which three were identified for evaluation in this research study (Section 3.1.1). The programmingconcepts and criteriawhich each of the three selected PATs address are identified (Section 3.1.2) and the three PATs are discussed (Section 3.2).

3.1. Selection and evaluation of PATs

The research study had to identify PATs suitable for use by IT programming scholars in South African secondary schools (Section 3.1.1). Inparticular, three PATs had to be identified for evaluation at three participating schools as part of the experimental procedure (Section 4).Since there is a possibility that PATs may not satisfy all of the derived selection criteria (Section 2.2), those PATs that were selected had tosatisfy the majority of the selection criteria (Section 3.1.2).

3.1.1. PATs for ITThe PATs selected for review and evaluationwere identified based on availability. PATs identified had to be freely available for download

in order to make the PAT available to IT scholars. The PATs reviewed were a representative sample of tools developed to support noviceprogrammers.

PATs were evaluated using the derived selection criteria (Section 2.2). Only PATs that met most of the criteria (Table 3) were furtherevaluated before selecting the final three PATs for evaluation in the schools. The following PATs were evaluated and omitted as they did notaddress the majority of the selection criteria: MatrixPro, KarelRobot, NoteTab, NotePad2, Phrogram, JHave, SmallTalk, EToys and GameMaker.PATs selected for inclusion in the evaluation phase (Table 3) were: RoboMind, BlueJ, Greenfoot, Scratch, B#, Jeliot, Ville, PlanAni, Alice andjGRASP.

Scratch and Alice allow users to create programs by dragging and dropping visual blocks. The blocks can only be joined in specific waysthus only logical errors can occur. Error messages and feedback for errors are thus not applicable (n/a) for these two PATs (Table 3). Similarly,PlanAni only allows users to run programs that have already been created and are running properly. The purpose of PlanAni is to animate aprogram for users to understand the use of data structures as well as how looping, for example, works.

The five highest ranking PATs (Koorsse, 2012), which were evaluated using the selection criteria were: B#, PlanAni, Scratch, Alice andRoboMind. However, for this particular study only three PATs were required as three schools agreed to participate.

3.1.2. Match to PAT criteriaThe three PATs selected as the intervention tools in the study, based on the evaluation of the PATs (Section 3.1.1) were RoboMind (Section

3.2.1), Scratch (Section 3.2.2) and B# (Section 3.2.3). B# is a PAT developed at the Nelson Mandela Metropolitan University (NMMU).Scratch was recommended by the South African Department of Education to be used in the new IT subject curriculum implemented from

2012 for Grade 10 (Department of Basic Education, 2011). Scratch was thus selected as one of the three PATs. Alice and Scratch are similarprogramming environments e allowing users to create a program by joining blocks to build a script that executes an animated program. Inorder to evaluate different PATs, Alice was not included.

PlanAni satisfies many of the selection criteria and supports the Pascal programming language on which the Delphi programminglanguage is based. However, PlanAni does not allow users to create their own programs and only six example programs are provided forusers to work with. PlanAni was thus not included for this study.

The original RoboMind had a number of shortcomings in terms of the desired features for PATs. However, the source code was accessibleand the original RoboMind was adapted to support the Delphi programming language. Adapted RoboMind and B# can assist users toimprove their knowledge of programming language syntax (Table 3) as program solutions are implemented or generated in the Delphiprogramming language. In Scratch, drag and drop building blocks are used to created program solutions.Table 3Evaluation of PATs using selection criteria.

Criteria B# PlanAni Scratch Alice RoboMind Jeliot Ville BlueJ Greenfoot jGRASP

Programming knowledgeAssists with learning of the Delphi programming language syntax C C CAssists with developing knowledge of programming principles and concepts C C C C C C C C C CConstructivist to promote self-study C C C C C C C C C CAssists with the application of programming knowledge C C CAssists with the understanding of code execution C C C C C C C C C CProgramming skillsPromotes problem solving and planning C C CProvides simple error messages to assist with debugging C n/a n/a n/a CDevelops code comprehension C C C C C CFeedback to guide solution creation C C CFeedback regarding errors C n/a n/a n/a C C C C C C% of criteria met: 90 60 60 60 60 60 60 40 40 40

M. Koorsse et al. / Computers & Education 82 (2015) 162e178168All three PATs are: constructivist to promote self-study (Table 3) by scholars to improve their understanding of programming concepts;assist with the development of knowledge of programming principles and concepts; and allow users to step through the program codesolution line by line during execution to interpret the code and see the effects of each line on the program output. Stepping through the codecan assist user understanding of code execution. None of the three PATs demonstrates to users how programming knowledge can be appliedto solve programming problems.

Scratch makes use of the drag-and-drop interface which ensures that users can only use the correct statements and syntax. Scratch thusindirectly supports error handling but error messages are not explicitly provided to scholars to support debugging. The criteria to providesimple error messages or feedback regarding errors are thus not applicable to Scratch due to the interface employed. RoboMind and B# usesimple error messages to inform users of syntax errors in the code using language and terms that are simple for novice programmers tounderstand. Scratch uses building blocks with statements that indicate to users where conditions or variables must be inserted or if otherstatements must be included within a loop or control structure in order to guide the creation of solution. B# allows users to build a solutionusing a flowchart diagram. Users are able to visualise the execution of the solution using the flowchart.

RoboMind is the only one of the three PATs that does not explicitly support problem solving and planning, the development of codecomprehension and the provision of feedback to guide solution creation. The focus of RoboMind is to provide users with an animated, visualenvironment in which the program solution is executed.

The programming concepts that IT scholars are expected to know (Table 4) are outlined in the Curriculum and Assessment PolicyStatement (CAPS) for Information Technology published by the South African Department of Basic Education (Department of BasicEducation, 2011). The three PATs support different criteria and programming concepts. Procedures are supported by RoboMind andScratch, although only RoboMind supports the passing of parameters in the procedure call. RoboMind does not support text output (Table 4),however, the program output is visualised when the robot moves through the environment.

3.2. PATs selected as intervention tools

An evaluation of the impact of the three PATs selected as intervention tools first required an understanding and description of theprogramming environments presented by each, including the mapping between programming concepts in the PAT and in the Delphiprogramming language. Schools who participated in this study teach the Delphi programming language as prescribed by the Department ofBasic Education (2011). RoboMind (Section 3.2.1) is an open-source PAT which was adapted so that the syntax and semantics more closelymatched the Delphi programming language. The building blocks used in the Scratch environment (Section 3.2.2) are mapped to corre-sponding programming constructs in the Delphi programming language. Finally, the similarities and differences between B# and Delphi arepresented (Section 3.2.3).

3.2.1. RoboMindRoboMind is a PAT that can be used as an introduction to automation and programming without any assumed prior knowledge of

programming (RoboMind, 2009). RoboMind allows users to create and execute a text-based program, similar in syntax to that of the Javaprogramming language, to instruct a robot to interact with objects in a map-based world. A standard set of commands controls the robot'smovement and actions in the world. Basic programming concepts such as looping, nested if-statements and user-defined procedures can beimplemented. The RoboMind source code is freely available thus RoboMind could and was adapted to support syntax similar to Delphi, theprogramming language used by participating schools. The programming concepts supported by the original version of RoboMind arepresented (Section 3.2.1.1). Adaptations to RoboMind are discussed (Section 3.2.1.2) to prepare the PAT for evaluations in the participatingschools.

3.2.1.1. Original RoboMind. RoboMind implements basic programming concepts such as looping and nested if-statements. Procedures withparameters are also acceptable. No variables can be declared or used, except for the procedure parameters. The language syntax used issimilar to Java although there is some variation such as the lack of semi-colons.Table 4IT curriculum programming concepts (Department of Basic Education, 2011) supported.

Programming concept RoboMind Scratch B#

Procedures C CFunctionsOne-dimensional arrays Crepeat-loops C (adapted to support) C CObjects and classes CTwo-dimensional arrayswhile-loops C C CString handling Cfor-loops C C CCorrect use of parameters Cif-statements C C CSQL statementsFile handlingAccessing a databasecase-statements COutput C CVariables C(adapted to support) C CInput C CTotal: 18 7 11 8

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 169The user types out the code instructions to navigate the robot through the world (A in Fig. 2) and then selects Run to execute the in-structions. The tool will notify the user of any errors in the code. If the code is error-free, the user can watch the robot navigate through itsenvironment based on the code instructions provided (B in Fig. 2). The current line of code being executed is indicated by a pointer and amessage at the bottom of the screen (C in Fig. 2). Users can stop or pause the execution of the program (D in Fig. 2) as well as change theexecution speed. The RoboMind environment is freely available for individual, educational and commercial use (RoboMind, 2009). Thedevelopment environment is open source allowing RoboMind to be adapted.

RoboMind also assists novice programmers to develop knowledge of programming principles and concepts and code execution as usersare able to follow step-by-step execution of the program code as the robot moves through the map world. RoboMind assists novice pro-grammers to improve programming skills by providing simple error messages to assist with debugging. RoboMind provides support forbasic programming concepts including if-statements, looping (determinate and indeterminate), creating user-defined procedures andcalling built-in procedures and functions to control the robot, which includes the use of parameters. The RoboMind microworld and pro-gram development environment has been designed to be easy to use and the robot animation provides a tangible output of the textualrepresentation of the program solution.

3.2.1.2. Adapted RoboMind. RoboMind was adapted to make it suitable for use by IT scholars in South African secondary schools. The ad-aptations were necessary as the original version of RoboMind did not support the same syntax as the Delphi programming language. Table 5provides a comparison between the syntax of programming concepts implemented in Delphi, original RoboMind and adapted RoboMind.The original RoboMind supports if-statements and while-loops, although the syntax differs from Delphi. The original RoboMind supports adeterminate looping structure, repeat(x), which repeats statements a specified (x) number of times. The for-loop is the determinate loopingstructure supported by Delphi, whereby the starting and ending values of the looping iterator, are specified.

Table 5 indicates the adaptations to RoboMind and how they compare to the Delphi programming language. Note that the for-loopimplemented in Adapted RoboMind only requires the number of times to repeat the loop; the start value is set to 1 and is not user-specifiedas in Delphi.

Limited support for the declaration and use of variables has also been implemented in the adapted version of RoboMind. Shortcomings ofthe implementation of variables are that no data types are specified when variables are declared.

3.2.2. ScratchSimilar programming construct blocks are grouped together in colour-coded categories (D in Fig. 3), for example, motion, sound and

variables. Selecting a category lists all the associated blocks (C in Fig. 3), thus allowing users to drag required blocks to the work area (AFig. 2. RoboMind interface.

Table 5Comparison of Programming Concept Statements for Delphi, original RoboMind and Adapted RoboMind.

Concept Delphi Original RoboMind Adapted RoboMind

for-loop (determinate) for i: 1 to 10 do repeat(10) for1to(10)while-loop (indeterminate) while (condition) do while (condition) while (condition) dorepeat-loop (indeterminate) repeat

until (condition);e repeat

until (condition);if-statements if (condition) then

elseif (condition)else

if (condition) thenelse

Code block delimiters begin..end; {..} begin..end;

M. Koorsse et al. / Computers & Education 82 (2015) 162e178170in Fig. 3) to build a script. The program is executed in the window on the top right (B in Fig. 3). The script blocks are highlighted as theprogram executes to indicate flow of code execution. In addition, users can set an option to execute blocks one step at a time (single-step), making it easier for users to follow the execution of code and associate code blocks with actions of the graphical objects andcharacters.

The advantages of using Scratch are that it is visually appealing and promotes active learning. Research by Malan and Leitner (2007)showed a decrease in the number of students dropping or failing an introduction to computer science course when Scratch was used tointroduce programming concepts to the scholars. A drawback of using Scratch is that users may struggle to move to a traditional pro-gramming environment without an intermediate software tool to provide a link between the programming concepts introduced in Scratchand the methods of implementing these concepts in a programming language (Resnick et al., 2009) where syntax becomes relevant.

Scratch does not enforce syntax rules due to the use of the drag-and-drop interface. Users, therefore, do not need to be concerned withthe overhead of programming language syntax. More focus can be given to the planning of programming solutions. Scratch is not open-source software, thus no adaptations could be made to Scratch. There are several differences between the implementation of program-ming concepts in Scratch and Delphi.

Multiple scripts can be created for different events in Scratch or when a name is broadcast. Broadcasting a specified name is theequivalent of a procedure call in Delphi. The terminology used in Scratch differs from the Delphi language and syntax. In addition, procedureparameters are not supported in Scratch. Global variables are used instead of passing parameter values to the procedure. Scratch supportsthe implementation of a list, which is similar to a dynamic array in Delphi, but not a one-dimensional array. The length of the list is notspecified when the list is created. Scratch supports the insertion of values at specific positions in the list which would be similar to assigningvalues to a static array at specific index positions.Fig. 3. Creating a script in Scratch by dragging building blocks.

Fig. 4. Scratch: Implementation of arrays and looping.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 171The Delphi repeat-loop equivalent in Scratch is the repeat until block (Fig. 4). The loop terminates when the conditional statement is true,which is equivalent to the implementation of the repeat-loop in Delphi. However, in Delphi the repeat-loop is post-conditional while inScratch it is pre-conditional. The forever if block in Scratch (Fig. 4) is equivalent to the while-loop in Delphi. The loop continues while theconditional statement is true and the loop is pre-conditional, which is equivalent to the implementation of the while-loop in Delphi. Thewording on the block, however, does not correspond to the syntax of the while-loop in Delphi.

The Scratch equivalent of the Delphi for-loop is the repeat block (Fig. 4). However, only the number of times the code in the blockmust berepeated is specified. The repeat block is thus equivalent to a Delphi for-loop inwhich the counter variable iterates from the value one (1) tothe integer value specified. No counter variable is defined in the Scratch repeat block. In addition, the wording used on the block is notconsistent with the Delphi for-loop syntax.

Scratch supports Delphi string operations such as joining words, determining the length of a string and returning specific characterswithin a string. Thewording used on the blocks differs from the syntax that is used in Delphi. Scratch also supports if-statements. A simple ifblock as well as an if-else block are available for use in program solutions. Nested if-statements are also supported in Scratch.

Variables are supported although no data types are specified. Values are assigned to variables using the set block. Scratch is also able toaccept string input from the user and execute scripts triggered by events such as mouse-clicks or key presses, as in Delphi. Scratch supportsdifferent types of program output. Objects or characters, referred to as sprites, can move, change colours, draw pictures and play music.String output, which is the most common form of output in Delphi, can also be displayed.

3.2.3. B#B# (Greyling, Cilliers & Calitz, 2006) is an iconic programming environment designed to simplify programming tasks by assisting novice

programmers with problem-solving strategies and the design of algorithms. B# uses a visual flowchart approach that supports program-ming concepts such as assigning values to variables, conditions, loops, inputs and outputs.

Fig. 5 is a snapshot of the B# user interface currently in the execution state. The iconic flowchart representation of the program running isdepicted (A in Fig. 5). The associated code generated by the system is also displayed (B in Fig. 5) allowing users to compare the flowchartcreated with the code generated. The code generated by B# is Object Pascal (Delphi) code; however, no graphical user interface componentsare used. Input and output is received using readln and displayed usingwrite/writeln, respectively. The resulting program is a command lineconsole program (D in Fig. 5) that accepts text input and displays text output.

Functionality is provided to control execution of the program, such as stepping through the code one line at a time or stopping theexecution at a particular point (E in Fig. 5). The current values assigned to variables are visible at all times during the execution of the code (Cin Fig. 5). The only programming concepts supported by B# that differ with regard to syntax from Delphi are input and output. B# interactswith users using a console application for the main execution (Fig. 5), however, when tracing through the program Graphical User Interface(GUI) forms are used to accept user input. The Delphi programming environment can be used to create console applications as well asprogramswith a GUI. The subject content outline for programming in the IT subject requires IT scholars to create programs that interact withusers (input and output) using a GUI. IT scholars programming in Delphi are not required to andmay not be explicitly taught to use a consoleline application interface.

Fig. 5. Stepping through code created using a flowchart in B#.

M. Koorsse et al. / Computers & Education 82 (2015) 162e1781724. Experimental procedure

The purpose of the study was to determine the impact of a PAT on IT scholar understanding of programming concepts and perception ofthe difficulty of learning to program. The research questions are presented again as follows:

RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?RQ2: What impact do the PATs have on IT scholar perceived difficulty of programming?RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specific programming concepts?

In order to determine the impact of the PATs, an evaluation between the use of a PAT and not using a PAT, was required. Experimentalresearch propositions are ideal to test the proposed PATs where the desired outcome would be improved understanding of programmingconcepts. An experimental approach evaluates the results obtained from a control group in comparison to the results obtained by treatmentgroup: a group receiving a treatment or some form of intervention. In this study the treatment was the use of a PAT. In true experimentaldesigns, the two groups should be equivalent where participants are randomly assigned to control and treatment groups (Gribbons &Herman, 1997; Kenny, 1975). However, in this study it was difficult to randomly assign scholars across different schools to control andtreatment groups. The different teachers and teaching environments were also a factor.

A variation of the non-equivalent groups design of the quasi-experimental research procedure (Gribbons & Herman, 1997; Kenny, 1975)was thus followed where existing IT classes within the schools were used. Each school's participating treatment group received the samePAT. Scholars within the same class could not be separated as control and treatment group scholars, thus the study spanned two academicyears (Table 6). Scholars in Grade 10 and Grade 11 of the first year of the study constituted the control group, while scholars in Grade 10 andGrade 11 of the second year of the study and who indicated that they used the PAT, constituted the treatment group. Scholars who receivedthe PAT but did not use it (self-reported) also formed part of the control group of the study. Grade 10 and Grade 11 scholars were evaluatedseparately for certain of the data collected, due to the difference in programming knowledge and experience.

Table 6Quasi-experimental research design for this study.

First year Second year

Treatment Pretest PAT PosttestControl Pretest Posttest

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 173The aim of the study was for IT scholars to have access to the PAT for an extended period during the school academic year, during whichtime different programming concepts were taught to the scholars at school. The IT teacher did not provide any assistancewith regards to theuse of the PAT. The PAT was evaluated as a tool that IT scholars could use in their own time to improve their understanding of programmingknowledge and skills. The study was completed in three different South African secondary schools offering IT as a subject from Grades 10 to12 (one of the four schools that originally consented had only four participants in the treatment group). Each school received only one of thethree PATs. The IT teacher at each school had at least 10 years' experience teaching programming at secondary school level. The IT teacherdid not teach any programming content using the PAT.

4.1. Instruments and data collection

Three instruments were used to collect data in order to evaluate the impact of the PAT received on IT scholar understanding of pro-gramming concepts. The Perceived Difficulty of Programming Questionnaire (PDPQ) was administered as a pre- and post-test to Grade 11scholars in the control and treatment groups. The PDPQ consisted of items evaluated using a 7-point semantic differential scale. Participantswere required to evaluate their perceived difficulty of IT programming concepts as well as their ability to solve programming problems,create programming solutions and find errors in code (RQ2). This questionnairewas only administered to Grade 11 scholars (Fig. 6). Grade 10scholars could not evaluate their experience of programming at the beginning of the study as they had not yet had any programming in-struction. Grade 11 control and treatment group participants completed this questionnaire.

Multiple-choice class tests were administered as class tests to the control and treatment group scholars (Fig. 6) after certain respectiveprogramming topics had been completed in class. Grade 10 scholars were assessed on their understanding of if-statements, for-, while- andrepeat-loops. Grade 11 scholars were evaluated on their understanding of procedures, parameters and one-dimensional arrays. Multiple-choice class test results were used to assess the programming knowledge of IT scholars in order to evaluate the impact the use of thePATs have on IT scholar knowledge of programming concepts (RQ1).

A PAT Evaluation Questionnaire was also administered to treatment group scholars (Fig. 6) to evaluate the PAT received. Treatment groupparticipants were required to rate the usefulness of the PAT in terms of understanding programming concepts supported. Feedback from thePAT Evaluation Questionnaire was used to determine the IT scholar perceived usefulness of the PAT received with regards to the under-standing of specific programming concepts (RQ3).

4.2. Data analysis

Quantitative data analysis methods, namely descriptive and inferential methods (Cohen, Manion, & Morrison, 2007), were used toanalyse the questionnaires and class tests administered to IT scholars who participated in this research study. Descriptive data analysisFig. 6. Quasi-experimental research strategy with questionnaire instruments.

Table 7Number of participants in control and treatment groups.

PATControl Treatment

Grade 10 Grade 11 Total Grade 10 Grade 11 Total

RoboMind 25 27 52 24 4 28Scratch 24 15 39 5 3 8B# 17 12 29 18 11 29Total 73 59 132 60 24 84

M. Koorsse et al. / Computers & Education 82 (2015) 162e178174methods such as the calculation of the sample mean (M), as well as frequency distribution of responses, were used to report semanticdifferential scale results. Descriptive statistics were also used to describe the relationship between different variables such as control ortreatment group and grade with respect to questionnaire responses in the presentation of results (Section 5).

The questionnaires administered contained items that used a 5- or 7-point semantic differential scale. All semantic differential scale datareferred to in this work was analysed as interval ratio data as only the extreme values had descriptions (e.g. Strongly disagree andStrongly agree) on the questionnaires.

Inferential statistics were used to determine whether the differences in the results of the control and treatment groups were statisticallyand/or practically significant. Statistical significance was determined at the 95% confidence interval. Analysis of covariance (ANCOVA)methods were used to examine the differences between the control and treatment groups where pre- and post-test datawas available. Datacollected from the PDPQwas evaluated using this analysis. Analysis of variance (ANOVA)methods were used to evaluate the multiple choiceclass results as only one dependent variable was available in order to determine significant differences between control and treatmentgroups. One sample t-tests were used to determine the difference between mean scores and the neutral value of rating scale data obtainedfrom the PAT Evaluation Questionnaire results.

5. Results

Table 7 indicates the number of scholars in the Grade 10 and Grade 11 control and treatment groups at each of the three schools,evaluating one of the PATs. Grade 10 and Grade 11 scholars were evaluated separately as Grade 10 scholars had no prior programmingexperience at the start of the control or treatment year and Grade 11 scholars had at least one year of programming experience (from Grade10).

The results for each of the three PATs evaluated e RoboMind (Table 8), B# (Table 9) and Scratch (Table 10) e are presented. Each tablepresents the results of different programming concepts addressed by the particular PAT, obtained using the three different data collectioninstruments (Section 4). In each of the tables, M1 refers to the control group mean, while M2 refers to the treatment group mean.

The results obtained from the different questionnaire instruments completed by participating scholars after evaluating a PAT at each ofthe schools were analysed to address each of the three research questions.

5.1. RQ1: What impact does the use of the PATs have on IT scholar knowledge of programming concepts?

Overall, the results indicated that the evaluated PATs supported scholar understanding of looping concepts. Not all of the results werestatistically significant. The multiple choice class test results (Table 8) indicated that RoboMind treatment group test means were higherthan the control group means for all of the programming concepts evaluated. One-way ANOVA analysis showed no statistically significantdifferences between the class test means. The results for RoboMind indicated statistically significant mean score differences between thecontrol and treatment group for repeat- and while-loops in terms of scholar perceptions of difficulty. The treatment group mean class testmark for these two concepts was higher than the control group mean, although not statistically significant. The results from all three in-struments indicated RoboMind's support for the understanding of if-statements. For each of these concepts, RoboMind supports the correctsyntax and implementation (Section 3.2.1).Table 8RoboMind support of programming concepts results (*p < 0.05).

Concept PDPQ Multiple choice PAT evaluation

M1 M2 ANCOVA M1 M2 ANOVA M2 T-stat M2 T-stat

Procedures 2.56n 27

2.5n 4

F(1; 16) 0.18p 0.674

88.15n 27

90.00n 4

F(1; 29) 0.07p 0.794

2.50n 6

t(5) 0.75p 0.49

2.00n 8

t(7) 1.53p 0.170

Parameters 2.52n 27

3.52n 4

F(1; 22) 1.87p 0.186

Not evaluated Not evaluated

repeat 2.22n 27

1.00n 4

F(1; 27) 6.49p 0.017*

70.00n 22

74.81n 27

F(1; 47) 0.63p 0.433

2.43n 7

t(6) -0.93p 0.386

2.71n 17

t(16) -0.96p 0.040*

while 2.04n 27

1.00n 4

F(1; 27) 5.54p 0.026*

77.92n 22

84.13n 27

F(1; 47) 1.38p 0.247

2.00n 6

t(5) 1.94p 0.386

2.65n 17

t(16) -1.14p 0.269

for 1.63n 27

1.00n 4

F(1; 27) 3.49p 0.073

79.09n 22

82.15n 27

F(1; 47) 0.40p 0.532

2.38n 8

t(7) -1.05p 0.329

2.71n 17

t(16) -1.00p 0.332

if 1.44n 27

1.00n 4

F(1; 27) 2.76p 0.108

73.82n 22

79.56n 27

F(1; 47) 1.60p 0.213

2.96n 25

t(24) -0.17p 0.870

3.00n 21

t(20) 0.00p 1.00

Variables 1.19n 27

1.00n 4

F(1; 27) 1.65p 0.210

Not evaluated 2.88n 24

t(23) -0.55p 0.588

2.70n 20

t(19) -1.14p 0.267

Table 9B# support of programming concepts results (*p < 0.05; **p < 0.01).


M1 M2 ANCOVA M1 M2 ANOVA M2 T-stat M2 T-stat

repeat 3.57n 7

2.5n 8

F(1; 12) 0.58p 0.462

44.00n 20

53.33n 18

F(1; 36) 0.87p 0.357

2.20n 10

t(9) -2.45p 0.037*

2.00n 9

t(8) -2.45p 0.040*

while 2.88n 8

2.13n 8

F(1; 13) 0.75p 0.401

46.49n 20

62.70n 18

F(1; 36) 2.77p 0.105

2.00n 10

t(9) -3.00p 0.015*

1.78n 9

t(8) -3.35p 0.010*

for 2.50n 8

2.00n 8

F(1; 13) 0.05p 0.821

60.89n 19

62.22n 18

F(1; 35) 0.02p 0.879

Error in questionnaire

if 1.88n 8

1.75n 8

F(1; 13) 0.01p 0.912

70.33n 12

No marks e 2.00n 15

t(14) -4.18p 0.001**

1.78n 9

t(8) -3.35p 0.010*

case 2.50n 8

2.50n 8

F(1; 13) 0.36p 0.557


t(11) -2.87p 0.015*

1.67n 9

t(8) -4.62p 0.002**

Variables 2.00n 8

1.50n 8

F(1; 12) 0.41p 0.535

2.60n 15

t(14) -1.47p 0.164

2.00n 9

t(8) -2.00p 0.081

Input 1.75n 8

1.38n 8

F(1; 13) 1.06p 0.321

2.40n 15

t(14) -2.07p 0.057

2.11n 9

t(8) -1.57p 0.154

Output 2.13n 8

1.38n 8

F(1; 13) 1.74p 0.210

2.33n 15

t(14) -2.32p 0.036*

2.11n 9

t(8) -1.57p 0.154

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 175Scratch provides support for the looping concepts, particularly for-loops, as indicated by Grade 10 and Grade 11 treatment group scholarevaluations (Table 10) of the difficulty of programming concepts (PDPQ) as well as the neutral mean results from the evaluation of the PAT(PAT Evaluation Questionnaire). The results were not statistically significant. In Scratch, the for-loop characteristic of finite repetitions isdemonstrated using a repeat block even though the syntax does not match (Section 3.2.2). The multiple choice class test results for theScratch treatment group did not provide any indication that scholar understanding of the programming concepts improved.

B# treatment group scholars have higher mean ratings than control group scholars for the looping concepts evaluated using themultiplechoice test results (Table 9). However, none of the mean differences were statistically significant and treatment group scholars disagree thatB# assisted with the understanding of the looping concepts. Overall, there was no indication from the results that B# assisted with scholarunderstanding of programming knowledge.

5.2. RQ2: What impact do the PATs have on IT scholar perceived difficulty of programming?

Scholars were required to rate (on a 7-point semantic differential scale) their perception of the difficulty of each of the programmingconcepts evaluated when using the Perceived Difficulty of Programming Questionnaire (PDPQ). A rating of one indicated that the scholarsperceived the concept as extremely easy to understand, while a rating of seven was categorised as extremely difficult. A comparison of themean scores of the control and treatment group post-test results for the PDPQ indicate that scholars in the RoboMind treatment groupperceived all the programming concepts to be easier than the control group did (Table 8), except for the use of parameters. The differencesbetween the control and treatment group means for the repeat- (F(1; 27) 6.49, p 0.017) and while-loop concepts (F(1; 27) 5.54,p 0.026) were statistically significant.

The mean ratings (PDPQ) for input, if-statements, string handling and procedures indicated that the Scratch Grade 11 treatment grouprespondents perceived these concepts to be more difficult than the Scratch Grade 11 control group respondents (Table 10). A two-wayTable 10Scratch support of programming concepts results.


M1 M2 ANCOVA M1 M2 ANOVA M2 T-stat

String handling 1.80n 15

2.00n 3

F(1; 15) 0.75p 0.400


t(3) -1.41p 0.252

1-d arrays 1.87n 15

1.33n 3

F(1; 15) 0.82p 0.380

97.67n 15

86.67n 2

PATs received after test 1.50n 2

t(1) -3.00p 0.205

Procedures 2.50n 15

3.30n 3

F(1; 12) 1.60p 0.230

92.00n 15

75.00n 2

F(1; 24) 1.78p 0.202

2.00n 2

t(1) -0.50p 0.500

repeat 2.27n 15

1.67n 3

F(1; 14) 0.41p 0.534

70.00n 20

46.67n 9

F(1; 24) 1.35p 0.257

3.00n 4

t(3) 0.00p 1.00

while 1.93n 15

1.33n 3

F(1; 15) 0.57p 0.462

71.43n 20

53.97n 9

F(1; 24) 0.61p 0.422

2.50n 4

t(3) -0.77p 0.495

for 1.73n 15

1.33n 3

F(1; 15) 0.81p 0.382

62.84n 19

59.63n 8

F(1; 24) 0.00p 0.999

3.00n 5

t(3) 0.00p 1.00

if 1.27n 15

1.68n 3

F(1; 15) 1.07p 0.317

72.00n 20

68.00n 8

F(1; 24) 0.65p 0.430

3.60n 5

t(4) 1.18p 0.305

Variables 1.33n 15

1.33n 3

F(1; 15) 0.62p 0.444


t(4) 0.00p 1.00

Input 1.27n 15

1.33n 3

F(1; 15) 0.02p 0.889

3.20n 5

t(4) 0.30p 0.78

Output 1.47n 15

1.33n 3

F(1; 15) 0.01p 0.914

3.00n 5

t(4) 0.00p 1.00

M. Koorsse et al. / Computers & Education 82 (2015) 162e178176ANCOVA analysis did not indicate statistically significant differences between the control and treatment group mean ratings for any of theprogramming concepts supported (Table 10). The concepts with the lowest treatment group mean (M2) ratings (n 3) and with highercorresponding control group (M1) ratings (n 15), were the for-loop (M1 1.73, M2 1.33), while-loop (M1 1.93, M2 1.33) and arrayconcepts (M1 1.87, M2 1.33). Only three treatment group scholars completed the PDPQ, affecting the reliability of the comparison be-tween control and treatment group results. Themultiple choice class testmean scores for the treatment group scholars were also lower thanthe control group scores for all of the programming concepts supported (Table 10), supporting the perceptions of the participants. None ofthe multiple choice class test differences are statistically significant. An evaluation of the difference between the Scratch treatment andcontrol group results indicated that treatment group scholars had more difficulty understanding programming concepts than control groupscholars (PDPQ and multiple choice test results). The IT teacher was replaced after the first term of the treatment year. The results suggestthat the scholars' transition to a different IT teacher and teaching environment influenced the poorer performance (tests) and greaterperceived difficulty of programming concepts by treatment group scholars. Based on the results, Scratch was not able to support scholarswho did not have a stable teaching environment.

B# treatment group scholars perceived (PDPQ results) looping concepts to be more difficult when comparing mean ratings to the otherprogramming concepts evaluated (Table 9). Treatment group scholars, however, perceived the difficulty of most of the programmingconcepts to be easier to understand, compared to control group scholars. The differences were not statistically significant.5.3. RQ3: What impact do the PATs have on IT scholar perceived usefulness with regards to the understanding of specific programmingconcepts?

The usefulness of the PAT in terms of assisting with the understanding of the programming concepts supported was evaluated using asemantic differential scale where a rating of 1 indicates that scholars strongly disagreed and a rating of 5 indicates that scholars stronglyagreed that PAT assisted their understanding of the concept. Themean evaluation ratings for RoboMind (Table 8) for all of the concepts wereless than, or equal to the neutral value, 3. One sample t-tests indicated that none of the programming concept mean values were statisticallysignificantly different from the neutral value. Evaluation results indicated the highest mean ratings for the support provided by RoboMindfor the understanding of if-statements (M2 2.96).

The Grade 10 and Grade 11 treatment group scholars (n 10) provided qualitative feedback about what they liked and disliked aboutRoboMind (PAT Evaluation Questionnaire). Ninety percent of participants indicated that RoboMind helped them to practise basic pro-gramming concepts. Forty percent of scholars indicated that they disliked that the syntax of certain of the programming concepts supportedin RoboMind (Section 3.2.1) was slightly different to coding in Delphi. Forty percent of the responses indicated dislike for the fact thatRoboMind supports a limited number of programming concepts.

The Scratch Grade 10 and 11 treatment group scholar mean evaluation ratings (Table 10) for input, output, variables, if-statements, for-loops and repeat-loops were greater than or equal to the neutral value, 3. One-sample t-test analysis indicated that none of the ratings werestatistically significant from the neutral value. Many scholars did not complete the questionnaire due to administrative difficulties. Thenumber of scholars who evaluated the usefulness of Scratch to understand the programming concepts supported, ranged from two to fivescholars, depending on the concept (Table 10).

The usefulness of B# with regard to the understanding of the programming concepts supported, was evaluated twice (Table 9) by Grade10 and Grade 11 treatment group scholars in the academic year. In both evaluations, the results indicated that scholars did not agree that B#was useful to assist with the understanding of repeat-loops, while-loops, if-statements and case-statements. For all of these concepts, onesample t-test analysis indicated that the mean rating score was statistically significantly different from the neutral value, 3. The mean ratingfor output in the first evaluation was also statistically, significantly different from the neutral value.6. Discussion

The aim of the research study was to evaluate the impact of three PATs, namely an adapted version of RoboMind, Scratch and B#, onSouth African IT scholar understanding of introductory programming concepts. The impact of the PATs on IT scholar understanding ofprogramming concepts was evaluated by addressing three research questions (Section 4). Each of the research questions were evaluated bycomparing the means of control and treatment group multiple-choice class test assessments and perceived difficulty of programming. ITscholars using the PAT (treatment group) also evaluated their perception of the impact the PAT had on their understanding of programmingconcepts supported in the PAT.

The ANOVA/ANCOVA analysis results for all three PATs indicated that there were no statistically significant differences between themeans of control and treatment groups for the multiple choice class tests although the treatment group means were higher than controlgroup means for scholars using the adapted RoboMind and B#. The use of the three PATs did not have a significant impact e positive ornegativee on IT scholar knowledge of programming concepts, addressing RQ1 (Section 5.1). This is in linewith findings by Levy et al. (2001).Quantitative results may show an improvement, however the use of a PAT, specifically with animation, does not necessarily improve scholarperformance in long term use. In a study by Al-Imamy et al. (2006), participants in the treatment group performed significantly better inexamination assessment. However, the use of the web-based tool was included by instructors in the teaching environment.

This study also aimed to determine the impact of the PATs on IT scholar perceived difficulty of specific programming concepts (Section5.2). The Perceived Difficulty of Programming Questionnaire (PDPQ) results indicated that the adapted RoboMind assisted with the un-derstanding of repeat- and while-loops, as perceived by IT scholars. The difference between control and treatment group means was sta-tistically significant for these two concepts only. The adapted RoboMind thus had a positive impact on the IT scholar perceived difficulty ofrepeat- and while-loops. IT scholar feedback from open questions identified the syntax of RoboMind to be an issue which may have affectedthe understanding of the other concepts. The syntax of for-loops and variables in RoboMind differed from the Delphi programming languagesyntax (Section 3.2.1). Future improvements to the adapted RoboMind to match the syntax more closely or exactly to the Delphi pro-gramming language could result in more programming concepts being perceived as easier to understand by IT scholars.

M. Koorsse et al. / Computers & Education 82 (2015) 162e178 177There were no statistically significant PDPQ results for B# (Table 9) or Scratch (Table 10). Although qualitative feedback from scholarsindicated that Scratch was easy to use, scholars disliked that it was too simple and without purpose. Scholars may not have been motivatedto use Scratch which may have, in turn, impacted the effectiveness of Scratch to assist scholar understanding of programming concepts.Scholars evaluating B# indicated that they liked the automatic code generation and that the graphical flowchart representation explainedprogramming concepts, particularly loops and if-statements. However, scholars disliked that the generated code differed from Delphi. Theonly difference in syntax was with regards to input and output (Section 3.2.3).

An evaluation of participating treatment group IT scholar perception of the usefulness of the PATs with respect to improving theirunderstanding of programming concepts (Section 5.3) provided mixed results. Feedback for the usefulness of adapted RoboMind andScratch did not provide statistically significant results. Scholars indicated, however, that RoboMindwasmost useful for the understanding ofif-statements. Scholars using B# did not agree that B# was useful to improve their understanding of programming concepts (results werestatistically significant). This is a contradiction to qualitative scholar feedback received stating that the flowchart representation assistedtheir understanding. The impact of a differing syntax on scholar understandingmay have had a negative influence on their perception of thePAT. The overall feedback indicates that the syntax of the programming environment was important for participants. The results of studiesby de Raadt (2008), Malan and Leitner (2007) and Rongas et al. (2004) indicate that explicit use of a PAT and focus on specific programmingknowledge and skills may be more successful than self-regulated use of a PAT that is not part of the classroom teaching approach.

Another result of the PAT as part of self-regulated learning and not enforced as part of the classroom teaching environment is that not allparticipants continued to use the PAT in this study. This was a similar finding in the study by Levy et al (2001) evaluating Jeliot, whereweaker scholars were overwhelmed and stronger scholars did not think the PAT was needed.

7. Conclusions and future work

Quantitative analysis indicated that the PATs had no impact on IT scholar understanding of programming concepts in general. Qualitativefeedback from scholars identified several aspects that could be addressed by future PATs developed for use by IT scholars in South Africanschools. The syntax of the programming language used must match the Delphi programming language as closely as possible and the PATshould support the implementation of as many programming concepts as possible. The programming environment should be easy to use,however, not too simple as users will not be challenged or motivated to use the PAT.

The research was limited by participating school environments such as IT teacher workload and small IT classes. The results of theselimitations were incomplete questionnaire and class test data. In spite of these limitations, the research findings and contributions haveformed the basis for future work. Extending the study by increasing the sample size of participating schools and the inclusion of usabilitystudies would provide additional data making the results more generalisable. Inclusion of the PAT as part of the teaching environment mayallow scholars to be more confident using the PAT. Further adaptations to RoboMind as well as the development of a new PAT specificallydesigned for IT scholars based on the finding of the research, are also possible. The contributions of this study such as the identification ofprogramming knowledge and skills, selection criteria for PATs and the evaluation results could be generalised and made applicable tosupport novice programmers in general.

References

Al-Imamy, S., Alizadeh, J., & Nour, M. A. (2006). On the development of a programming teaching tool: the effect of teaching by templates on the learning process. Journal ofInformation Technology Education, 5, 271e283.

Alexander, P. M., & Twinomurinzi, H. (2012). Changing career choice factors as the economic environment changes. In Proceedings of SAICSIT'12, October 1-3 2012, PretoriaSouth Africa (pp. 295e305).

Areias, C., & Mendes, A. (2007). A tool to help students to develop programming skills. In International Conference on Computer Systems and Technologies (CompSysTech07).Baldwin, L. P., & Kuljis, J. (2000). Visualisation techniques for learning and teaching programming. Journal of Computing and Information Technology, 4, 285e291.Calitz, A. (2010). A model for the alignment of ICT education with business ICT skills requirements. DBA, Nelson Mandela Metropolitan University.Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education (6th ed.). Routledge.Department of Basic Education. (2011). Curriculum and assessment policy statement. Grades 10e12. Information Technology. Available at: http://www.education.gov.za/

LinkClick.aspx?fileticketjIbU9d%2B63O4%3D&tabid420&mid1216 Accessed 13.08.13.Garner, S. (2007). A program design tool to help novices learn programming. In ICT: Providing choices for scholars and learning (pp. 321e324). Ascilite Singapore 2007.Greyling, J., Cilliers, C., & Calitz, A. (2006). B#: The development and assessment of an iconic programming tool for novice programmers. In 7th International Conference on

Information Technology Based Higher Education and Training (ITHET06) (pp. 367e375).Gribbons, B., & Herman, J. (1997). True and quasi-experimental designs. Practical Assessment, Research & Evaluation, 5(14). Available at: http://PAREonline.net/getvn.asp?

v5&n14 Accessed 13.08.13.Guzdial, M. (2004). CS education research, chap. Programming Environments for Novices.Havenga, M., & Mentz, E. (2009). The school subject information technology: a South African perspective. In Proceedings of SACLA'09 (pp. 77e83). Mpekweni Beach Resort,

South Africa.Heines, J., & Schedlbauer, M. (2007). Teaching object-oriented concepts through GUI programming. In Eleventh Workshop on Pedagogies and Tools for the Teaching and

Learning of Object Oriented Concepts, held at the 21st European Conf. on Object-oriented Programming. Berlin, Germany.Hug, S., Guenther, R., & Wenk, M. (2013). Cultivating a K12 computer science community: a case study. In Proceedings of SIGCSE13, March 6-9, 2013, Denver, Colorado, USA.Kelleher, C., & Pausch, R. (2005). Lowering the barriers to programming: a taxonomy of programming environments and languages for novice programmers. ACM Computing

Surveys, 37(2), 88e137.Kenny, D. A. (1975). A quasi-experimental approach to assessing treatment effects in the nonequivalent control group design. Psychological Bulletin, 82(3), 345e362.Koorsse, M. (2012). An evaluation of programming assistance tools to support the learning of IT programming: A case study in South African secondary schools. Doctoral thesis,

Nelson Mandela Metropolitan University.Koorsse, M., Calitz, A. P., & Cilliers, C. B. (2010). Programming in SA Schools: the Inside Story. In Proceedings of SACLA. South Africa: Pretoria.Kolling, M. (1999). The problem of teaching object-oriented programming, Part 1: Languages. Journal of Object-Oriented Programming, 11(8), 8e15.Kunkle, W. M. (2010). The impact of different teaching approaches and languages on student learning of introductory programming concepts. Thesis. Drexel University. Doctor of

Philosophy.Lahtinen, E., Ala-Mutka, K., & Jarvinen, H. (2005). A study of the difficulties of novice programmers. In Proceedings of the 10th Annual SIGCSE Conference on innovation and

Technology in Computer Science Education (pp. 14e18). Caparica, Portugal: ACM.Levy, R. B.-B., Ben-Ari, M., & Uronen, P. A. (2001). An extended experiment with Jeliot 2000. In Proc. First International Program Visualization Workshop (pp. 131e140). Pavoo,

Finland: University of Joensuu Press.Lister, R., Adams, E., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., et al. (2004). A multi-national study of reading and tracing skills in novice programmers. InWorking Group

Reports From ITiCSE on innovation and Technology in Computer Science Education (pp. 119e150). Leeds, United Kingdom.

http://refhub.elsevier.com/S0360-1315(14)00273-5/sref1http://refhub.elsevier.com/S0360-1315(14)00273-5/sref1http://refhub.elsevier.com/S0360-1315(14)00273-5/sref1http://refhub.elsevier.com/S0360-1315(14)00273-5/sref2http://refhub.elsevier.com/S0360-1315(14)00273-5/sref2http://refhub.elsevier.com/S0360-1315(14)00273-5/sref2http://refhub.elsevier.com/S0360-1315(14)00273-5/sref3http://refhub.elsevier.com/S0360-1315(14)00273-5/sref6http://refhub.elsevier.com/S0360-1315(14)00273-5/sref6http://refhub.elsevier.com/S0360-1315(14)00273-5/sref7http://refhub.elsevier.com/S0360-1315(14)00273-5/sref8http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://www.education.gov.za/LinkClick.aspx?fileticket=jIbU9d%2B63O4%3D%26tabid=420%26mid=1216http://refhub.elsevier.com/S0360-1315(14)00273-5/sref10http://refhub.elsevier.com/S0360-1315(14)00273-5/sref10http://refhub.elsevier.com/S0360-1315(14)00273-5/sref40http://refhub.elsevier.com/S0360-1315(14)00273-5/sref40http://refhub.elsevier.com/S0360-1315(14)00273-5/sref40http://refhub.elsevier.com/S0360-1315(14)00273-5/sref40http://PAREonline.net/getvn.asp?v=5%26n=14http://PAREonline.net/getvn.asp?v=5%26n=14http://PAREonline.net/getvn.asp?v=5%26n=14http://PAREonline.net/getvn.asp?v=5%26n=14http://PAREonline.net/getvn.asp?v=5%26n=14http://refhub.elsevier.com/S0360-1315(14)00273-5/sref12http://refhub.elsevier.com/S0360-1315(14)00273-5/sref13http://refhub.elsevier.com/S0360-1315(14)00273-5/sref13http://refhub.elsevier.com/S0360-1315(14)00273-5/sref13http://refhub.elsevier.com/S0360-1315(14)00273-5/sref14http://refhub.elsevier.com/S0360-1315(14)00273-5/sref14http://refhub.elsevier.com/S0360-1315(14)00273-5/sref15http://refhub.elsevier.com/S0360-1315(14)00273-5/sref16http://refhub.elsevier.com/S0360-1315(14)00273-5/sref16http://refhub.elsevier.com/S0360-1315(14)00273-5/sref16http://refhub.elsevier.com/S0360-1315(14)00273-5/sref17http://refhub.elsevier.com/S0360-1315(14)00273-5/sref17http://refhub.elsevier.com/S0360-1315(14)00273-5/sref41http://refhub.elsevier.com/S0360-1315(14)00273-5/sref41http://refhub.elsevier.com/S0360-1315(14)00273-5/sref42http://refhub.elsevier.com/S0360-1315(14)00273-5/sref18http://refhub.elsevier.com/S0360-1315(14)00273-5/sref18http://refhub.elsevier.com/S0360-1315(14)00273-5/sref18http://refhub.elsevier.com/S0360-1315(14)00273-5/sref19http://refhub.elsevier.com/S0360-1315(14)00273-5/sref19http://refhub.elsevier.com/S0360-1315(14)00273-5/sref20http://refhub.elsevier.com/S0360-1315(14)00273-5/sref20http://refhub.elsevier.com/S0360-1315(14)00273-5/sref20http://refhub.elsevier.com/S0360-1315(14)00273-5/sref20http://refhub.elsevier.com/S0360-1315(14)00273-5/sref21http://refhub.elsevier.com/S0360-1315(14)00273-5/sref21http://refhub.elsevier.com/S0360-1315(14)00273-5/sref21http://refhub.elsevier.com/S0360-1315(14)00273-5/sref22http://refhub.elsevier.com/S0360-1315(14)00273-5/sref22http://refhub.elsevier.com/S0360-1315(14)00273-5/sref22

M. Koorsse et al. / Computers & Education 82 (2015) 162e178178Ma, L., Ferguson, J., Roper, M., Ross, I., & Wood, M. (2008). Using cognitive conflict and visualisation to improve mental models held by novice programmers. In Proceedings ofthe 39th SIGCSE Technical Symposium on Computer Science Education (pp. 342e346).

Malan, D., & Leitner, H. (2007). Scratch for budding computer Scientists. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (SIGCSE07) (pp.223e227).

Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., et al. (2007). A survey of literature on the teaching of introductory programming. In Working groupreports on ITiCSE on Innovation and technology in computer science education, ITiCSE-WGR 07 (pp. 204e223). New York, NY, USA: ACM.

Pendergast, M. (2006). Teaching introductory programming to IS students: Java problems and pitfalls. Journal of Information Technology Education, 5, 491e515.Porta, M., Maillet, K., & Gil, M. (2010). Dec-cs: the computer science declining phenomenon. In Proceedings of the World Congress on Engineering and computer science 2010

(WCECS2010) (Vol. 2). San Francisco, USA.de Raadt, M. (2008). Teaching programming strategies explicitly to novice programmers. Doctoral dissertation. University of Southern Queensland.Resnick, M., Maloney, J., Monroy-Hernandez, A., Rusk, N., Eastmond, E., Brennan, K., et al. (2009). Scratch: programming for all. In Communications of the ACM, 52 pp. 60e67).Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: a review and discussion. In Computer science education (Vol. 13, pp. 137e172).RoboMind. (2009). RoboMind website. Available at: http://www.robomind.net/en/index.html Accessed 13.03.10.Rongas, T., Kaarna, A., & Kalvianen, H. (2004). Classification of computerized learning tools for introductory programming courses: learning approach. In Proceedings of the

IEEE International Conference on Advanced Learning Technologies (ICALT04) (pp. 678e680).Shuhidan, S., Hamilton, M., & D'Souza, D. (2009). A taxonomic study of novice programming summative assessment. In Proceedings of the Eleventh Australasian Conferenc

Documents

Programming Assistance Tools