Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
Review
Adaptive Behavior2017, Vol. 25(5) 217–234� The Author(s) 2017Reprints and permissions:sagepub.co.uk/journalsPermissions.navDOI: 10.1177/1059712317727590journals.sagepub.com/home/adb
Adaptive feedback in computer-basedlearning environments: a review
Andrew Thomas Bimba1, Norisma Idris1, Ahmed Al-Hunaiyyan2,Rohana Binti Mahmud1 and Nor Liyana Bt Mohd Shuib3
AbstractAdaptive support within a learning environment is useful because most learners have different personal characteristicssuch as prior knowledge, learning progress, and learning preferences. This study reviews various implementation ofadaptive feedback, based on the four adaptation characteristics: means, target, goal, and strategy. This review focuses on20 different implementations of feedback in a computer-based learning environment, ranging from multimedia web-basedintelligent tutoring systems, dialog-based intelligent tutoring systems, web-based intelligent e-learning systems, adaptivehypermedia systems, and adaptive learning environment. The main objective of the review is to compare computer-basedlearning environments according to their implementation of feedback and to identify open research questions in adaptivefeedback implementations. The review resulted in categorizing these feedback implementations based on the students’information used for providing feedback, the aspect of the domain or pedagogical knowledge that is adapted to providefeedback based on the students’ characteristics, the pedagogical reason for providing feedback, and the steps taken toprovide feedback with or without students’ participation. Other information such as the common adaptive feedbackmeans, goals, and implementation techniques are identified. This review reveals a distinct relationship between the char-acteristics of feedback, features of adaptive feedback, and computer-based learning models. Other information such asthe common adaptive feedback means, goals, implementation techniques, and open research questions are identified.
KeywordsAdaptation, learning environment, problem-solving, student modeling, learner model
Associate Editor: Tom Froese
1. Introduction
The process of learning involves mistakes and errors.In these situations, students often review course mate-rials and search the Internet or other sources to assistthem in solving their problems (Ghauth & Abdullah,2010). Seeking solution is usually time consuming anddoes not always insinuate a better learning experi-ence. Having a system which generates effective feed-back that guides students to the solution can improvethe learning process (Munoz-Merino et al., 2011).Feedback is frequently provided in a typical class-room setting; however, most of the information ispoorly received because feedback is presented togroups and so often students do not believe such feed-back is relevant to them (Hattie & Gan, 2011).Currently, the gap between students who excel themost and those who excel less is a challenge thatteachers, school administrators, and government offi-cials face frequently (Luckin & Holmes, 2016).
Adaptive learning environments provide personaliza-tion of the instruction process based on different para-meters such as sequence and difficulty of task, type andtime of feedback, learning pace, and others (Brusilovskyet al., 1999; Stoyanov & Kirchner, 2004). One of the keyfeatures in learning support is the personalization offeedback (Advisors, 2013). Adaptive feedback supportwithin a learning environment is useful because mostlearners have different personal characteristics such as
1Department of Artificial Intelligence, University of Malaya, Kuala
Lumpur, Malaysia2Computer & Information Systems Department, College of Business
Studies, The Public Authority for Applied Education & Training (PAAET),
Kuwait City, Kuwait3Department of Information Systems, University of Malaya, Kuala
Lumpur, Malaysia
Corresponding author:
Norisma Idris, Department of Artificial Intelligence, University of Malaya,
50603 Kuala Lumpur, Malaysia.
Email: [email protected]
prior knowledge, learning progress, and learning prefer-ences. Tailoring feedback according to learner’s charac-teristics and other external parameters is a promisingway to implement adaptation in computer-based learn-ing environment (Narciss et al., 2014). Adaptive feed-back unlike generic feedback is dynamic, as learnerswork through instructions where different learners willreceive different information (Le, 2016). Addressing thisneed, many researchers have proposed variousapproaches to help students in learning (Farid, Ahmad,& Alam, 2015). As a result, they have identified gapsand have been developing various frameworks and edu-cational systems that are able to analyze student learningand provide adaptive feedback.
The main objective of this review is to comparecomputer-based learning environments according totheir implementation of feedback and to identify majoropen research questions in adaptive feedback imple-mentations. Not all the implementations selected haveadaptive feedback as their main design aim. The reasonfor our selection is to provide readers an insight to howadaptive feedback is implemented by comparing awider range of applications.
Previous researchers have conducted reviews ofadaptive feedback systems. Le (2016) analyzed theapproaches used in developing educational systems forprogramming and introduced a classification for adap-tive feedback supported by these systems. Hepplestone,Holden, Irwin, Parkin, and Thorpe (2011) explored var-ious literature supporting the appropriate use of tech-nology for providing feedback to students. Our currentreview follows similar methodologies. However, thisstudy reviews various implementation of feedback,based on the four adaptation characteristics: means,target, goal, and strategy (M. E. Specht, 1998). Basedon our knowledge, there has not been any review ofadaptive feedback implementations according to thefour adaptation characteristics. This classificationscheme provides an overview of the field. It emphasizesthe aspects of the technology, demonstrates openresearch questions, possible research opportunities, andoffers opportunity for researches to identify key charac-teristics, while implementing adaptive feedback systems.The main objective is to compare feedback implementa-tions according to these adaptation characteristics andidentify major open research questions in adaptive feed-back implementations.
The structure of the article is as follows: First, thebackground study on adaptive feedback, explaining thecharacteristics of adaptation and feedback, is discussed inSection 2. In Section 3, we provide the outline of thereview process. The results of the review of adaptive feed-back implementations according to the characteristics ofadaptation and feedback are discussed in Section 4. Wediscuss our findings in Section 5. Future directions andconclusion are presented in Sections 6 and 7, respectively.
2. Background
Brusilovsky (1998) defined systems that model stu-dent’s learning style, prior knowledge, goals, and pre-ference as adaptive, while those systems which useartificial intelligence (AI) techniques to perform therole of an instructor in tutoring and correcting arereferred to as intelligent systems. Learning environ-ments can be either one or a combination of bothadaptive and intelligent elements. According to Chieu(2005), there are five main adaptation techniqueswhich are related to the key components of construc-tive learning environment as follows:
1. Adaptive presentation of learning contents. Thecourse designer should define which learning con-tents are appropriate to a specific learner at anygiven time, for example, simpler situations andexamples for a novice learner than for an expert.
2. Adaptive use of pedagogical devices. The coursedesigner should define which learning activities areappropriate to a specific learner, for instance, sim-pler tasks to a novice learner than to an expert.
3. Adaptive communication support. The coursedesigner should identify which peers are appropriateto help a specific learner, for example, learners withmore advanced mental models help learners withless advanced ones.
4. Adaptive assessment. The course designer shouldidentify which assessment problems and methodsare appropriate to determine the actual performanceof a specific learner, for instance, simpler tests for anovice learner than for an expert.
5. Adaptive problem-solving support. The tutor shouldgive appropriate feedback during the problem-solving process of a specific learner, for example, toshow the learner his or her own difficulties and pro-vide him or her with the way to overcome thosedifficulties.
These adaptation techniques rely on a learner model; anessential component which, among other student rele-vant data, keeps data about the student’s knowledge ofthe subject domain under study.
Adaptive learning involves multiple disciplines suchas Educational Psychology, Cognitive Science, andArtificial Intelligence. This complexity prompted thestructuring of research on adaptivity along the metho-dological questions distinguishing means, target, goal,and strategy (M. E. Specht, 1998):
1. Adaptation means. What information about the lear-ner such as knowledge level, cognitive style, learningstyle, gender, student’s current activity, previousachievements and difficulties, and misconception isknown and used for adaptation?
218 Adaptive Behavior 25(5)
2. Adaptation target. What aspect of the instructionalsystem (pedagogy and domain model) is adaptedbased on the learner model?
3. Adaptation goal. What are the pedagogical reasonsfor the system to adapt to the learner model? Is thesystem aiding inductive or deductive learning; is thesystem adapting to a specific instructional methodbased on the learner model?
4. Adaptation strategy. What are the steps and tech-niques used to adapt the system to the learner model,and how active or reactive are the learners and sys-tem to the adaptation process?
In a learning environment, feedback is seen as theteacher’s (artificial or real) response to the student’saction. There are four main characteristics of feedback:function, timing, schedule, and type (Carter, 1984).Although other researchers (Economides, 2006) sug-gested other characteristics of feedback, we adhere tothe characterization by Carter (1984) because it encom-passes all other characterizations. These characteristicsare briefly explained in Table 1.
For developing an effective adaptive feedback sys-tem, the characteristics of adaptation and feedbackhave to be taken into consideration. The next sectiondiscusses our approach to reviewing implementationsof adaptive feedback based on these characteristics.
3. Materials and method
Scientific journals related to learning, computer tech-nology for education, and artificial intelligence in edu-cation from five main digital libraries were searched,with the aim of reviewing adaptive feedback implemen-tations based on the characteristics of adaptation andfeedback. These libraries include Scopus, Web ofScience, IEEE Xplore, Google Scholar, and ACM. Thelibraries were selected based on their impact evaluation
and wide coverage of peer-reviewed journals in multipleacademic disciplines. In addition to searching thesedatabases, snowballing technique was used to identifysimilar implementations of adaptive feedback systems.Only publications from years 2000 to 2016 were col-lected since most implementations of adaptive feedbackin learning environments were realized during thisperiod. To search for potential articles, keywords suchas feedback, adaptive feedback, intelligent tutoring sys-tem, adaptive learning system, computer-based tutor,pedagogical agents, and computer-assisted learningwere used.
For the searched articles, two criteria were consid-ered: (1) publications from year 2000 to year 2016,which indicated evidence of implementation and scien-tific evaluation of an adaptive feedback system and (2)recent articles with implementations or a clear pro-posed approach. These criteria allow us to considerpublications that have demonstrated practical relevanceand also take into account recently developed adaptivefeedback systems. We also narrowed down our selec-tion based on three views of adaptive feedback systems.First, adaptive learning systems that provide differentinformation to different learners as they work throughinstructions and second, adaptive learning systemswhich generate feedback based on a learner modelwhich distinguishes different learners. Third, wefocused on the proposed adaptive feedback frame-works, with practical implementation strategies.Publications that do not fall within this focus area ormeet the target criteria were excluded.
A total of 1709 articles were found after searchingthrough the five major digital libraries based on key-words as shown in Figure 1. Using EndNote desktopapplication (a software tool for managing articles andcitation), we eliminated the duplicates and selected thearticles that met part of our criteria through relevancesorting. This process resulted in 185 articles excludingthe subject descriptive articles which are mentioned in
Table 1. Characteristics of Feedback.
Characteristicsof feedback
Explanation
Function Feedback can be provided in relation to the instructional goals and objectives. For example, feedback isprovided based on cognitive functions such as promoting information processing, motivational functionssuch as developing and sustaining persistence or provide correct response.
Timing Feedback can be given with respect to timing. It could be in advance, appearing before an action; it couldbe immediate, appearing immediately after an action or delayed, appearing at a longer time after theaction has been made. The feedback is intended to advise, notify, recommend, alert, inform, or motivatethe learner about some concerns.
Scheduling Feedback can also be made available at scheduled instances. For example, when the learner exceeds acertain time threshold, expertise level, after solving certain questions or after every subtopic.
Type There are various feedback types resulting from function, timing, and scheduling. For example,verification feedback, avoidance feedback, correction feedback, informative feedback, cognitive feedback,emotional feedback, scheduled feedback, dynamic feedback, immediate feedback, advanced feedback,delayed feedback, comparative feedback, and isolation feedback.
Bimba et al. 219
the introductory parts. Furthermore, we selected 24 eli-gible articles and added 4 more from snowballingaccording to 20 different implementations of adaptivefeedback. These implementations range from multime-dia web-based ITS, dialog-based ITS, web-based intelli-gent e-learning system, adaptive hypermedia system,theoretical feedback frameworks, and intelligent andadaptive learning environment. The analyzed articlesconsisted of journal articles, conference proceedings,books, and serials. They were examined based on thepublication years, availability, and relevance to theresearch domain.
4. Results
4.1. Classification of adaptive feedbackimplementations
A computer-based learning environment representsknowledge in the form of models. The three key modelsin a computer-based learning environment are the ped-agogical model, domain model, and learner model.Research regarding the design and development ofadaptive learning environments is highly multi-
disciplinary, uniting research from computer scienceand engineering, psychology and psychotherapy, cyber-netics and system dynamics, instructional design, andempirical research on technology enhanced learning(Specht, Kravcik, Klemke, Pesin, & Huttenhain, 2002).While the educational scientists give attention to devel-opment, evaluation, and approval of adaptive instruc-tion algorithms, computer scientists are concernedmore with the development of better algorithms, mod-els (pedagogy, domain, and user), and intelligent adap-tation. The complexity which arises by the union ofthese disciplines initiated the need for structuringimplementations of adaptivity according to the metho-dological questions distinguishing means, target, goal,and strategy (Specht et al., 2002).
Similarly, we adopt this methodology as a classifica-tion scheme to review different implementations ofadaptive feedback in learning environments. Adaptivefeedback implementations can be grouped and ana-lyzed based on adaptation methodology and feedbackcharacteristics. Several adaptive learning systems haveutilized learner’s characteristics to provide adaptivefeedback. In this review, we discuss the following imple-mentations of adaptive feedback.
Figure 1. Review process.
220 Adaptive Behavior 25(5)
Wayang Outpost is a multimedia web-based intelli-gent tutoring system, designed to help students solvemathematics problems. It promotes meaningful andeffective ways of learning (Arroyo et al., 2003; Arroyoet al., 2014). Gerdes’ tutor is an interactive functionalprogramming tutor, which supports stepwise develop-ment in Haskell programming language (Gerdes,Jeuring, & Heeren, 2012). The E-Tutor, which wasdeveloped at Simon Fraser University in Canada, is aweb-based intelligent computer-assisted language learn-ing (iCALL) system for beginner to advanced levelGerman grammar exercises. It consists of Germangrammar concepts and vocabulary tasks, which is usedby students in North American Universities (Heift &Schulze, 2007). AutoTutor is an intelligent tutoring sys-tem which uses natural language and adaptive dialog tohelp students in understanding concepts in Newtonianphysics, critical thinking, and computer literacy(D’Mello & Graesser, 2012). The intelligent TeachingAssistant for programming (ITAP) is a data-driventutoring system that provides personalized help to stu-dents while working on code-writing problems (Rivers& Koedinger, 2015).
DeepTutor is another dialog-based intelligent tutor-ing system that uses scaffolding to improve student’sknowledge during problem-solving (Rus, Niraula, &Banjade, 2015). ACTIVEMATH is a web-based intelli-gent e-learning system that offers access to variousmathematical learning objects, which supports the con-structivist learning approach (Melis, Moormann,Ullrich, Goguadze, & Libbrecht, 2007). Guru, on theother hand, is an intelligent tutoring system which con-sists of exercises in high school biology, supportingconversation with students and virtual instructionalmaterials (Olney et al., 2012). INSPIRE is an adaptiveeducational hypermedia system which provides mean-ingful tasks to students, based on their preferred wayof learning (Papanikolaou, Grigoriadou, Kornilakis, &Magoulas, 2003). FIT Java Tutor is an intelligent andadaptive learning environment which integrates severalpedagogical approaches to assist students in learningJava programming (Gross & Pinkwart, 2015).
ANDES is an intelligent tutoring system whichencourages students to construct new knowledge inintroductory physics (Gertner & VanLehn, 2000;VanLehn et al., 2005). SQL-Tutor is an intelligenttutoring system which teaches database query languageby helping students learn from their mistakes (Mitrovic,2003; Mitrovic & Ohlsson, 1999; Mitrovic, Ohlsson, &Barrow, 2013). COMPASS uses concept maps as alearning tool which allows students to undertake assess-ment activities (Gouli, Gogoulou, Papanikolaou, &Grigoriadou, 2006). Excel Tutor is an intelligent novicetutor which provides feedback through error detectionand correction skills (Mathan & Koedinger, 2005). Thepedagogical motivation for feedback in Excel Tutor isto guide students in error detection and provide an
opportunity to reason about the causes and conse-quences of the errors.
Adaptive feedback frameworks have also been pro-posed by other researchers. Mason and Bruning’s(2001) theoretical framework enables the creation offeedback based on a variety of conditions such as thecomplexity of task, student’s prior knowledge, stu-dent’s achievement, timing of feedback, and learnercontrol. A conceptual framework for designing infor-mative tutoring feedback forms was put forward byNarciss and Huth (2002). The framework is aimed atderiving general principles for designing informativefeedback based on cognitive task and error analysis.
Other mathematics-based intelligent tutoring sys-tems which provide feedback have been proposed.Animalwatch is an ITS which integrates mathematicsand biological sciences for teaching arithmetics to ele-mentary school students (Arroyo, Beck, Woolf, Beal, &Schultz, 2000). It builds empirical models of the stu-dent’s behavior through an analyses of their interactionwith the mathematics tutor. Tsovaltzi and Fiedler(2003) used a natural language dialog module to imple-ment a mathematical tutoring system for teaching naiveset theory. An integrated learning environment for sec-ondary school mathematics was realized by Bokhove,Koolstra, Boon, and Heck (2007). The aim of thelearning environment is to provide students easilyaccessible practice mathematics problems and intelli-gent feedback when interacting with the content mate-rials. A tool called web-based authoring tool for AlgebraRelated domains (WEAR) assist teachers while author-ing exercises, monitors students during problem-sol-ving, and provides appropriate feedback (Virvou &Moundridou, 2000). WEAR combines knowledge ofauthoring algebraic equations which is applicable inother non-mathematical domains and student errordiagnosis (Virvou & Moundridou, 2001). It adapts theinteraction with students to provide individualizedfeedback (Moundridou & Virvou, 2002). In the nextsections, we concentrate on the key factors that distin-guish these systems in their implementation of adaptivefeedback.
4.1.1. Adaptive feedback means. An adaptive learningsystem alters its behavior based on how a learner inter-acts with it. These alterations are decided based on thelearner’s characteristics which are represented in thelearner model (Lo, Chan, & Yeh, 2012). It involves theaccurate tracking of learner’s activity, monitoring theirindividual characteristics, and providing timely adap-tive feedback according to effective pedagogical princi-ples (Narciss et al., 2014). Tailoring feedback accordingto learner’s characteristics and other external para-meters is a promising way to implement adaptation incomputer-based learning environment (Narciss et al.,2014). Adaptive feedback means, poses the question,
Bimba et al. 221
what information about the learner is known and used forproviding adaptive feedback? These information consistof students’ characteristics such as knowledge level,cognitive style, learning style, and gender.
Wayang Outpost provides adaptive feedback in theform of hints. Two types of hints are provided: (1) acomputational and numeric approach to solve a prob-lem and (2) spatial transformations and visual estima-tions to make the problem easier to solve (Arroyo et al.,2003). The choice of the hint provided by WayangOutpost is based on the learner’s cognitive profile. Thelearner profile is built based on an online assessment todetermine the learner’s math proficiency which includesaccuracy and speed of arithmetic computation and spa-tial ability. Wayang Outpost provides hints that capita-lize on the learner’s cognitive strength when one abilityis clearly better than the other. Otherwise, if both skillsare low, computational help is provided, whereas ifboth are high spatial help is provided (Arroyo et al.,2003). Similar to Wayang Outpost, Gerdes’ tutor pro-vides feedback in the form of hints. However, the hintsand feedback provided by Gerdes’ tutor does not utilizethe characteristics of the learner (such as knowledgelevel and learning style), instead it is generated automat-ically from an organized hierarchy according to the syn-tax tree of the model solution (Gerdes et al., 2012).
The student model in E-Tutor captures the path astudent has taken and the underlying source of theerror. It then provides instructional feedback based onthe students’ prior performance (Heift & Schulze,2007). The novice, intermediate, and expert are thethree types of learners assumed in the student model.These student levels are used to determine the specifi-city of the feedback provided. In AutoTutor, feedback isprovided in the form of a dialog. The dialog moves,pumps, hints, prompts, and assertions are selectedbased on the students’ knowledge (D’Mello & Graesser,2012). It constructs a cognitive model of the students’knowledge level based on the analysis of their typed orspoken responses. ITAP automatically generates feed-back in the form of hints. Using the path constructionalgorithm, it generates hints based on the students’solution strategy as determined in the solution space(Rivers & Koedinger, 2015). DeepTutor utilizes the stu-dents’ knowledge level in order to determine the typeand frequency of feedback (Rus et al., 2015). As theknowledge level of the student increases, less amount offeedback is provided.
In ACTIVEMATH, generic computer algebra sys-tems (CASs) are used to diagnose student’s actions inorder to provide hints, flag feedback (correct/incor-rect), and correct solution (Melis et al., 2007). It doesnot use any learner characteristics in deciding the typeof feedback to be provided. Similarly, Guru providesfeedback incrementally based on student’s knowledgelevel (Olney et al., 2012). INSPIRE supports adaptivenavigation support and adaptive presentation of
learning content only (Papanikolaou et al., 2003). Itdoes not use any student characteristics in providingfeedback. Whenever a learner requires help, examplesand hints are provided based on the theory presented(Papanikolaou et al., 2003). The FIT Java Tutor pro-vides feedback based on the students’ structured solu-tion space, which comprises student solution attemptsand sample solutions (Gross & Pinkwart, 2015). Andesprovides immediate feedback at each stage of problem-solving. The system provides immediate feedback basedon the students’ current knowledge or mental state(Gertner & VanLehn, 2000). Feedback in SQL-Tutor isprovided based on the number of student’s unsuccess-ful solution attempts (Mitrovic & Ohlsson, 1999).
Feedback in COMPASS is generally personalizedbased on identified error in a student’s concept maps,knowledge level, preferences, and interactive behavior(Gouli et al., 2006). Based on the theoretical frameworkproposed by Mason and Bruning (2001), an effectivefeedback design should take into consideration the stu-dent’s achievement level and prior knowledge. WhileNarciss and Huth’s (2002) conceptual framework sug-gests that the necessary information required for pro-viding informative feedback are the student’s learningobjectives, prior knowledge, learning strategies, and pro-cedural and meta-cognitive skills. In Excel Tutor, thesystem just considers the error made by students, itdoes not take into consideration any other personalizedcharacteristics of the student (Mathan & Koedinger,2005).
Adaptive feedback in Animalwatch is providedaccording to the student’s cognitive development andgender (Arroyo et al., 2000). In the process of assistingstudents in learning naive set theory, Tsovaltzi andFiedler (2003) developed a taxonomy of hints which isprovided to the students based on their current and pre-vious answers. Unlike the other approaches, Bokhoveet al. (2007) do not take into consideration any charac-teristics of the student while providing feedback.Instead, it provides intelligent feedback based on expertknowledge, common mistakes, and knowledge aboutthe learning domain. However, in WEAR, feedback isprovided based on the student’s knowledge level(Virvou & Moundridou, 2000).
4.1.2. Adaptive feedback target. In a computer-basedlearning environment, the pedagogical model repre-sents the knowledge and expertise of teaching. Specificknowledge represented in the pedagogical modelincludes effective teaching techniques (deductive andinductive); the various instructional methods (lectures,problem-based learning, inquiry learning, etc.); instruc-tional plan that define phases, roles, and sequence ofactivities (Scheuer, Loll, Pinkwart, & McLaren, 2010);feedback types, depending on a learner’s action; andassessment to inform and measure learning (Luckin &
222 Adaptive Behavior 25(5)
Holmes, 2016). The domain model represents knowl-edge of the subject been learned. It mainly consists ofconcepts such as how to add, subtract, multiply num-bers; Newton’s law of motion; how to structure anargument; and different approaches to reading (Luckin& Holmes, 2016). Adaptive feedback target is involvedwith the aspect of the instructional system (pedagogyand domain knowledge) that is adapted to providefeedback based on the learner characteristics. Withinvarious instructional methods, there are certain condi-tions that affect the type of feedback provided.
The hint provided by Wayang Outpost could be invarious forms based on three multimedia learning the-ories which include modality principle, contiguity prin-ciple, and animation principle (Arroyo et al., 2014).The modality principle represents words in form ofspeech, the contiguity principle aligns text to corre-sponding graphics while animation principles producean illusion of characters adhering to the basic laws ofphysics. These principles guide the videos which showhow instructors solve maths problems; synchronizedsound, animation, and contiguous explanations ofmaths problem and worked examples. Adaptive feed-back inWayang Outpost is involved with both the peda-gogical (multimedia learning theories) and domainmodels (maths solution in form of worked examples,speech, graphics, and animation), selecting variouscomponents of these models based on the learner char-acteristics (Arroyo et al., 2014). Gerdes’ tutor providesinteractive feedback to students (Gerdes et al., 2012).These interactions give hints to students on the nextstep to take, list of possible ways to proceed, point-outerrors, and provide complete worked-out examples.These information are all part of the instructional mate-rial, but they are not provided based on the student’scharacteristics. Instead, students are provided with anoption to choose what type of hint they desire.
Unlike Gerdes’ tutor, E-Tutor alters its instructionalfeedback within the domain model based on the stu-dent’s proficiency as determined in the student modelby the percentage of previously correct answers (Heift& Schulze, 2007). AutoTutor provides feedback in theform of dialog. The decision to provide a specific formof feedback, either pumps, hints, an answer or aprompt, depends on the information received by thetutor from the student (Nye, Graesser, & Hu, 2014).ITAP extends the Hint Factory (domain knowledge) byautomatically generating hints that are tailored to anindividual’s solution to a problem (Rivers & Koedinger,2015). DeepTutor provides scaffolding and a sequenceof progressive hints, based on the student’s knowledgelevel as articulated in the student model (Rus, Conley,& Graesser, 2014). In ACTIVEMATH, the domain rea-soner generates and provides hints, flag feedbacks, andcorrect solution based on the diagnosis of student’ssolution steps (Melis, Goguadze, Libbrecht, & Ullrich,
2009). Depending on the type of error made by a stu-dent, the Guru tutor assesses the student’s knowledgeand response with a positive feedback, negative feed-back, neutral feedback, or elaborative feedback. Eventhough INSPIRE provides adaptation for navigationsupport and learning content based on student’s learn-ing style, it does not provide various forms of helpaccording to the student’s learning style.
When there is no information about the quality of astudent’s solution (without representative solution), FITJava Tutor provides feedback in form of self-reflectionprompts. At instances where the quality of a student’ssolution is partially known, feedback F1, F2, or a com-bination of F1 and F2 are provided based on previoussuccesses of theses strategies on similar solution quali-ties (Gross, Mokbel, Paassen, Hammer, & Pinkwart,2014). F1 feedback strategy is when the student’s solu-tion differ partially from the correct solution but imple-ments the same problem-solving strategy, the differenceis highlighted without showing the actual solution. F2feedback strategy is when the student’s solution is con-trast with the actual solution but the correct problem-solving strategy is used, the solutions are contrasted toallow the student to compare and find the possible mis-takes. Andes provides flag feedback accompanied byhints and error messages, which the student can decideto consult when they stuck. These hints are generatedusing the solution graph in the domain model, based onthe state of the student model (Gertner & VanLehn,2000). In SQL-Tutor, feedback messages are providedas right/wrong, error flag, hints, partial solution, andcomplete solution (Mitrovic & Ohlsson, 1999). Theseforms of feedback, which are part of the pedagogicalmodule, are provided to the student based on the num-ber of successful solution attempts.
The process of generating effective feedback inCOMPASS depends on the student’s answer duringproblem-solving (Gouli et al., 2006). In COMPASS,feedback is provided based on an answer categorizationscheme. According to the scheme, a student’s answercan be characterized based on completeness, accuracy,and missing out. However, based on (Mason &Bruning, 2001) theoretical framework, less specificfeedback is provided as the learning tasks and student’sknowledge level increases. Students with higherachievement levels can benefit more from feedbackwhich provides general information. Thus, allowingthem to identify their errors and accurately seek thecorrect solutions (Mason & Bruning, 2001). In Narcissand Huth’s (2002) conceptual framework, the aspectsof the instructional content that affects the type of feed-back provided are the instructional goals, learningtasks, issues, and learning problems. This frameworkrecommends a careful alignment of the type of feed-back provided and the characteristics of the instruc-tional content.
Bimba et al. 223
In Excel Tutor, feedback is generated based on anintelligent novice model. The diagnostic capabilities ofthe model supports the provision of context-specificfeedback to students (Mathan & Koedinger, 2005).Similarly, in Animalwatch, hints are classified based onthe degree of hint symbolism and interactivity (Arroyoet al., 2000). However, these hints are provided ran-domly regardless of the student’s interactions.However, a hinting algorithm is used by Tsovaltzi andFiedler (2003) to evaluate the student’s performanceand to provide relevant hint. Different forms of localfeedback are provided by Bokhove et al. (2007). Thefeedback could be a comment regarding the accuracyof a response or an explanation based on the student’sanswer. But, in WEAR, there is no clear aspect of theinstructional system that is adapted in providing feed-back. However, the instructor and student model inWEAR interact with each other to mimic a one-to-onetutoring setting (Moundridou & Virvou, 2002).
4.1.3. Adaptive feedback goal. Feedback can serve differ-ent purposes based on pedagogical principles or a par-ticular learning theory that is been applied. From anobjectivist perspective, feedback is regarded as a rein-forcement, which is aimed at guiding the learner toprogress from a simpler task to a more complex one.The information processing theory suggests that thegoal of feedback is not only to reinforce correctanswers but also to serve as corrective information toallow learners correct their errors (Hattie & Gan,2011). Socioculturalism considers feedback as a recipro-cal dialog, where meaning is reconstructed by peers.The goal of feedback from this view is the consolida-tion, reorganization, and making knowledge explicitthrough exchange of ideas between peers (Pryor &Crossouard, 2008). Visible learning theory views feed-back in the context of student’s learning (alone, withpeers, or adults), at different levels of expertise (novice,proficient, or expert) and level of understanding (sur-face, deep, and conceptual) (Hattie & Gan, 2011). Inan inductive teaching method such as discovery learn-ing, feedback is provided only based on a student’seffort and not as a direct guide for those efforts (Prince& Felder, 2006). In developing adaptive feedback sys-tems, the designer needs to consider the goal of provid-ing feedback. The adaptive feedback goal identifiespedagogical reasons for providing feedback based onthe learner model, thus differentiating various imple-mentations. These characteristics revile the function offeedback.
In Wayang Outpost, adaptive feedback is providedbased on the theory of cognitive apprenticeship, wherea master teaches skills to an apprentice. The main aimof this theory is to encourage learners to accomplishmore difficult problems than they can accomplish with-out a guide. Thus, adaptive feedback in Wayang
Outpost is aimed at providing motivational support andencouraging engagement in the learning process. Helpis provided in the form of similar work examples, whichenable students to solve similar or harder problemsthan the current problem (Arroyo et al., 2014). Thefeedback and hints in Gerdes’ tutor are provided basedon teacher-specified annotations of solutions (Gerdeset al., 2012). There is no pedagogical principle or learn-ing theory involved. The purpose of providing feedbackin E-Tutor is based on the language teaching pedagogy(Heift & Schulze, 2007). This learning theory ensuresthat the amount of feedback provided does not confusethe student (Heift & McFetridge, 1999). AutoTutor pro-vides feedback with the goal of stimulating active con-struction of knowledge based on the constructivistprinciple (D’Mello & Graesser, 2012).
In ITAP, hints are provided as either point hint orbottom-out hint (Rivers & Koedinger, 2015). The pur-pose of providing such feedback does not depend onany pedagogical principle or learning theory.DeepTutor provides different types of hints which aredynamically sequenced based on a constructivist scaf-folding strategy (Rus et al., 2014). Feedback inActiveMath is based on the moderate constructivistapproach. It is aimed at providing a reasonable amountof guidance which allows learners to choose and reflecton their work (Melis et al., 2007). Even though, Guruprovides imcremental feedback which is aimed at tai-loring conversations based on individual student’sknowledge level, it does not base this feedback on anypedagogical principle or learning theory (Olney et al.,2012).
INSPIRE uses the instructional design theory andlearning style theory to provide individualized instruc-tions. However, the hints provided by INSPIRE onlyindicate right or wrong, but do not depend on any ped-agogical principle or learning theory (Papanikolaouet al., 2003). FIT Java Tutor provides feedback which isaimed at guiding students toward self-reflection basedon the example-based learning theory (Gross et al.,2014). Feedback in Andes is aimed at encouraging con-structive learning (Gertner & VanLehn, 2000). Thus,providing little feedback to students unless they requestfor it. The aim of feedback in SQL-Tutor is to promotethe acquisition of new cognitive skills, through the stateconstraint theory (Mitrovic & Ohlsson, 1999). The con-straint theory suggests that acquiring new cognitiveskills is based on the transfer of knowledge from theevaluative to the generative component. Thus, feedbackin SQL-Tutor is provided when students submit solu-tions for evaluation.
The goal of feedback in COMPASS is to supportreflection by guiding the students to reconstruct theirknowledge (Gouli et al., 2006). Mason and Bruning’s(2001) theoretical framework proposes the incorpora-tion of immediate feedback when the goal of instructionis teaching new concepts or the facilitation of concept
224 Adaptive Behavior 25(5)
acquisition. While delayed feedback should be providedwhen the instructional aim is to develop higher orderskills like abstract reasoning or comprehension (Mason& Bruning, 2001). According to Narciss et al.’s (2014)conceptual framework, feedback can be viewed in thecontext of self-regulated learning, behavioral, and cog-nitive learning theories. Based on these theories, feed-back can either have a goal of tutoring or guiding thelearner, re-enforcement of a concept, or a source ofinformation (Narciss et al., 2014).
The intelligent novice model used in Excel Tutor isaimed at supporting a student in the generative and eva-luative aspects of learning a new skill. It explicitly mod-els these skills and guides students through the processof error detection and correction (Mathan & Koedinger,2005). Tsovaltzi and Fiedler’s (2003) natural languagedialog module provides hints based on the Socratictutoring strategy in order to achieve self-explanation(Tsovaltzi & Fiedler, 2003). While feedback is providedin Animalwatch (Arroyo et al., 2000), Bokhove et al.(2007), and WEAR (Moundridou & Virvou, 2002),there is no clear pedagogical reason for its provision.
4.1.4. Adaptive feedback strategy. Adaptive feedbackstrategies combine several feedback components toassist learners in identifying gaps that exist betweentheir current and desired knowledge state (Narciss,2013). These feedback strategies could be in severalforms which include adaptive bottom-up feedback(where detailed feedback is provided and as proficiencyincreases the feedback changes to general), adaptivetop-down feedback (general feedback is provided first,if there is no improvement then a detailed feedback isprovided), outcome feedback (indicate right or wrong),hints on how to proceed, and location of mistakes(Billings, 2012). Most of the time, a combination ofthese strategies are used to ascertain appropriate feed-back conditions (Narciss, 2013). In some situations, thelearner is given an option to interact with the systemand determine the need for feedback. In implementingthese strategies, several modeling and artificial intelli-gence techniques are used. The adaptive feedback strat-egy looks at the steps taken in providing feedbackbased on changes in student proficiency, how active orreactive are the learners in the feedback process, andthe modeling and artificial intelligence techniques usedin implementing adaptive feedback. The implementa-tion of an adaptive feedback strategy determines thetiming and scheduling of feedback.
The strategy used by Wayang Outpost is through astep-by-step instruction and guidance to a solution.Adaptive feedback is provided only when the learnerrequests for help. There is no explicit process for pro-viding feedback as learner’s proficiency increases(Arroyo et al., 2014). Help is provided when a learnerhas difficulty in one problem, and then a similar
problem is provided, encouraging a transfer of knowl-edge to subsequent problems. The approach used byWayang Outpost to implement this strategy is a data-centric Bayesian Network which produces a probabilitymodel based on student’s previous interaction with thesystem. Help seeking is modeled to see how hint isrelated to skills (Arroyo et al., 2014). The Bayesian net-work has nodes corresponding various hints and skills.Hints in the Gerdes’ tutor is provided in steps (Gerdeset al., 2012). When a student is stuck, they can requestfor help from the tutor. The tutor provides optionsbased on the annotated teacher-specified feedback. If achoice is made, the student can ask for further details ifthe first explanation is not clear. Afterward, the tutorresponds with more details and a bottom-out hint. Toprovide a semantically rich feedback, Gerdes’ tutor usedtechniques such as parsing, rewriting, and programtransformation (Gerdes et al., 2012). In E-Tutor, thefeedback process is iterative. Student’s errors are identi-fied and communicated one at a time. These iterativeprocesses continue until the student gets the correctanswer or decides to submit a solution (Heift, 2010).
Unlike Gerdes’ tutor, E-Tutor does not provide stu-dents with a feedback choice. Instead, feedback is gener-ated based on the correlation between the result of thelinguistic analysis of a student solution and an error-specific feedback message. A parser and head-drivenphase structure grammar (HPSG) are used to determinegrammatically incorrect sentences and associate theerrors detected with feedback messages (Heift, 2016;Heift & Nicholson, 2001). AutoTutor provides feedbackto a student’s initial answer by first providing a shortfeedback, an elaborative feedback, and then an encour-agement (D’Mello & Graesser, 2012). During this pro-cess, the student is actively involved in a conversationwith the tutor. The latent semantic analysis (LSA) algo-rithm in AutoTutor is used to determine the informationwithin a student’s response that matches an expectationin the ideal answer, while a subthreshold expectationselection algorithm determines the prescribed sequentialorder to present expectations (D’Mello & Graesser,2012). When a student makes an error, ITAP provideshints in two levels. The first level (point hint) informsthe student about the type of change required and wherethe change should be. While the second level (bottom-out hint) provides all the information needed to correctthe error (Rivers & Koedinger, 2015). The path con-struction algorithm is used to generate a chain of hintsthat leads to a correct solution state.
DeepTutor provides feedback using the tell-or-elicittactic (Rus et al., 2014). This strategy is based on thescaffolding-modeling-fading theory. DeepTutor imple-ments the theory by eliciting a step, if not compre-hended by the student then it tells. This ensures anactive participation by the student through a two-wayconversation with the tutor. The management of thisdialog is implemented using production rules. In
Bimba et al. 225
ActiveMath, the domain reasoner generates feedbacksuch as flag feedbacks, correct solution, next step hint,correct input, and number of steps to final solution(Melis et al., 2009). These feedback forms are not pre-sented in any sequence. However, students can requestfor hints when needed (Melis et al., 2007). The domainreasoner in ActiveMath is implemented using rule-basedtechniques. The feedback provided by Guru is providedin form of dialog. The response of the tutor on the typeof feedback to be provided is based on an assessment ofthe student’s knowledge. In a case where the studentmakes an error, the incorrect relationship is highlightedand an explanation is provided for the meaning of therelationship; however, if the student has little back-ground knowledge, an extended direct instruction isprovided (Olney, Person, & Graesser, 2011). Guru usesLSA and concept map to align the students’ utteranceswith the domain and students models to determine if aninput is correct or wrong.
In INSPIRE hints and examples are provided to stu-dents to indicate right or wrong or on request.However, these feedbacks are provided without anyconsiderations of a specific type or sequence of presen-tation. The FIT Java Tutor uses a consecutive combina-tion of the F1 and F2 strategy in providing feedback,depending on the learners’ needs and progress. Withthe aim of providing support which is relevant to thestudent’s needs, FIT Java Tutor provides feedback withvarying levels of detail according to the student’s learn-ing progress (Gross et al., 2014). The automated provi-sion of feedback is developed based on clusteredsolution space. ANDES provides help in a sequencebased on three levels. It provides flag feedback in formof a pop up message when the error is likely a slip andnot lack of knowledge, and if it is not recognized as aslip, it is highlighted red. The second level is the what’swrong help, where students can click on a red entry andfind out the reason behind the error. Finally, studentscan request for help when they are not sure of what todo next (VanLehn et al., 2005). During this process, thestudent is actively involved in selecting the sequence ofhints provided. In order to provide immediate feed-back, ANDES uses a context-free parser to detecterrors in student’s input and a solution graph whichcontains relevant solution entries. SQL-Tutor post-pones feedback until the end of problem-solving steps.At the end of problem-solving, the student is presentedwith all the errors, but feedback is given for only oneerror. The feedback is based on the amount of informa-tion they provide. They are in five levels: right/wrong,error flag, hint, partial solution, and complete solution.The levels of feedback are provided based on the stu-dent’s unsuccessful solution attempts (Mitrovic &Ohlsson, 1999). However, the student can request for apartial or complete solution. Violations in a student’ssolution are determined with the aid of constraint-based modeling, relevance, and satisfaction networks.
The steps taken to provide feedback in COMPASSare a gradual provision of various types of feedbacksbased on a four-layered structure and the category of astudent’s answer (Gouli et al., 2006). Feedback inCOMPASS is implemented with the help of conceptmaps used for identifying student’s errors. Mason andBruning’s (2001) theoretical framework suggests a dif-ferent strategy, where the student is provided with aknowledge-of-response feedback and then allowed todecide if they require additional feedback. Mason andBruning (2001) suggest that this strategy will help todevelop the student’s understanding in situations wherethe correct answer was a guess. A proposed guidelinefor selecting and specifying different forms of feedbackis presented by Narciss et al.’s (2014) conceptual frame-work. These guidelines aim at ensuring the studentreceives the appropriate feedback based on the learningtask.
In Excel Tutor, an immediate corrective feedback isprovided to the student at the formulas correction step.However, if the student requests for help in correctingthe error, the system provides a gradual two-step feed-back process. The first step focuses on error detectionand the second step involves error correction (Mathan& Koedinger, 2005). Corrective feedback in Excel Tutoris implemented using production rules associated witherror free and efficient task performance (Mathan &Koedinger, 2005). Whenever a student enters a wronganswer during a tutoring session in Animalwatch, a hintis provided. The first hint provides little amount ofinformation and if the student keeps providing thewrong answer, the system guides them through thewhole process of problem-solving (Arroyo et al., 2000).For implementing feedback in Animalwatch, machinelearning techniques, linear regression, and analysis ofvariance (ANOVA) are used to predicting hint effective-ness. Similarly, Tsovaltzi and Fiedler’s (2003) naturallanguage dialog module supports a gradual provision ofhints based on the number of hints given, the numberof wrong answers, and the category of the student’sanswer. A combination of ontology and productionrules are used to generate feedback in Tsovaltzi andFiedler’s (2003) natural language dialog module. Thereis no identifiable strategy for the provision of feedbackin Bokhove et al. (2007). Student’s errors are only high-lighted in different colors to differentiate correct,incomplete, and wrong answers. Similar to Bokhoveet al. (2007), WEAR (Moundridou & Virvou, 2002) pro-vides individualized feedback to students without anyspecific strategy.
5. Discussion
In this study, 20 different implementations of adaptivefeedback were reviewed and analyzed. These imple-mentations were selected based on their impact and
226 Adaptive Behavior 25(5)
contribution in computer-based adaptive learningresearch. We present our analysis on these implemen-tations based on the classification criteria for adaptivefeedback, highlighting their levels of adaptive feed-back provision. Subsequently, we compared the vari-ous implementations of adaptive feedback based onadaptive feedback means, target, goal, and strategy;domain of implementation; adaptive feedback imple-mentation techniques; and evaluation method.
5.1. Adaptive feedback implementation categories
In Figure 2, the adaptive feedback implementations arepresented based on the classification scheme discussedin Section 4. We categorized the feedback implementa-tion based on their implementation of adaptive feed-back means, target, goal, and strategy. Figure 2 alsoshows how the adaptive feedback characteristics arealigned to the pedagogical domain and student modelsof a computer-based learning environment. The adap-tive feedback target, strategy, and goal are determinedby concepts in the pedagogical model. However, adap-tive feedback target and strategy can be implementedusing concepts represented in the domain model.Finally, the adaptive feedback means is determined byfactors in the student model.
Similarly, there is a relationship between the charac-teristics of feedback and the features of adaptive feed-back. As shown in Figure 2, the implementation of anadaptive feedback strategy determines the timing andscheduling of feedback. The adaptive feedback goal iden-tifies pedagogical reasons for providing feedback basedon the learner model. These characteristics revile thefunction of feedback. Within various instructional meth-ods, there are certain conditions that affect the type offeedback provided. These reveal a distinct relationshipbetween the characteristics of feedback, features ofadaptive feedback, and computer-based learning models(pedagogy, domain, and student models).
5.2. Comparison of adaptive feedbackimplementations
A detailed comparison of the various implementationsof adaptive feedback is presented in Table 2. The mainobjective is to identify the common ways for imple-menting adaptive feedback means, target, goal, andstrategies; common domains for adaptive feedbackimplementations; adaptive feedback implementationtechniques; and common evaluation techniques. Basedon the results of the comparison, the following conclu-sions were obtained:
Figure 2. Classification of adaptive feedback implementations.
Bimba et al. 227
Tab
le2.
Com
par
ison
ofad
aptive
feed
bac
kim
ple
men
tations.
Feed
bac
kim
ple
men
tation
Dom
ain
Adap
tive
feed
bac
km
eans
Adap
tive
feed
bac
kta
rget
Adap
tive
feed
bac
kgo
al(p
edag
ogi
cal
pri
nci
ple
)
Adap
tive
feed
bac
kst
rate
gyIm
ple
men
tation
tech
niq
ue
Eva
luat
ion
tech
niq
ue
Way
ang
Outp
ost
(Arr
oyo
etal
.,2014)
Mat
hem
atic
sSt
uden
t’sco
gnitiv
epro
file
Multim
edia
lear
nin
gth
eori
es(p
edag
ogi
cal
know
ledge
).W
ork
ed-
exam
ple
s,sp
eech
,gr
aphic
s,an
dan
imat
ion
(dom
ain
know
ledge
)
Bas
edon
theo
ryof
cogn
itiv
eap
pre
ntice
ship
Step
-by-
step
inst
ruct
ion.A
ctiv
ele
arner
Dat
a-ce
ntr
icBay
esia
nN
etw
ork
Pre
-tes
tan
dpost
-te
st
Ger
des
’Tu
tor
(Ger
des
,Je
uri
ng,
&H
eere
n,2012)
Pro
gram
min
gN
/AN
/A(h
ints
and
work
edex
ample
sar
epro
vided
equal
ly,w
ithout
consi
der
ing
studen
ts’
char
acte
rist
ics)
N/A
Leve
lsofdet
ail.
Act
ive
lear
ner
Par
sing,
rew
riting,
and
pro
gram
tran
sform
atio
n
Ques
tionnai
re
E-T
uto
r(H
eift
&Sc
hulz
e,2007).
Langu
age
lear
nin
gSt
uden
t’skn
ow
ledge
leve
lIn
stru
ctio
nal
feed
bac
k(d
om
ain
know
ledge
)B
ased
on
langu
age
teac
hin
gped
agogy
Iter
ativ
eer
ror
det
ection.R
eact
ive
lear
ner
Par
ser
and
hea
d-
dri
ven
phas
est
ruct
ure
gram
mar
(HPSG
)
Anal
ysis
oflo
gdat
ain
the
repo
rtm
anag
er
Auto
Tuto
r(D
’Mel
lo&
Gra
esse
r,2012).
Phy
sics
,co
mpute
rlit
erac
y,an
dcr
itic
alth
inki
ng
Studen
t’skn
ow
ledge
leve
lPum
p,hin
ts,an
swer
s,an
dpro
mpts
(dom
ain
know
ledge
)
Bas
edon
const
ruct
ivis
tpri
nci
ple
Sequen
ceof
feed
bac
ks.A
ctiv
ele
arner
Late
nt
sem
antic
anal
ysis
(LSA
)an
dsu
bth
resh
old
expec
tation
sele
ctio
nal
gori
thm
Bys
tander
Turi
ng
test
.Exper
tco
mpar
ison
ITA
P(R
iver
s&
Koed
inge
r,2015)
Pro
gram
min
gSt
uden
t’sso
lution
stra
tegy
Hin
tfa
ctory
(dom
ain
know
ledge
)N
/ALe
vels
ofdet
ails
.R
eact
ive
lear
ner
Pat
hco
nst
ruct
ion
algo
rith
mD
atas
etan
alys
is
Dee
pTu
tor
(Rus,
Nir
aula
,&
Ban
jade,
2015)
Phy
sics
Studen
t’skn
ow
ledge
leve
lSc
affo
ldin
gan
dse
quen
tial
hin
ts(d
om
ain
know
ledge
)B
ased
on
aco
nst
ruct
ivis
tsc
affo
ldin
gst
rate
gy
Leve
lsofdet
ails
.A
ctiv
ele
arner
Pro
duct
ion
rule
sW
ord
-to-w
ord
sim
ilari
tym
easu
reusi
ng
Mic
roso
ftR
esea
rch
Par
aphra
se(M
SRP)
Corp
us
Act
iveM
ath
(Mel
is,
Moorm
ann,U
llric
h,
Gogu
adze
,&
Libbre
cht,
2007)
Mat
hem
atic
sSt
uden
t’sac
tion
Hin
ts,fla
gfe
edbac
ks,an
dco
rrec
tso
lution
(dom
ain
know
ledge
)
Bas
edon
moder
ate
const
ruct
ivis
tap
pro
ach
No
explic
itst
rate
gyfo
rpre
senting
feed
bac
k.A
ctiv
ele
arner
Rule
-bas
edN
/A
Guru
(Oln
eyet
al.,
2012)
Bio
logy
Studen
t’skn
ow
ledge
leve
lPo
sitive
feed
bac
k,neg
ativ
efe
edbac
k,neu
tral
feed
bac
k,or
elab
ora
tive
feed
bac
k(d
om
ain
know
ledge
)
N/A
Leve
lofdet
ails
.A
ctiv
ele
arner
LSA
and
conce
pt
map
sPre
-tes
tan
dpost
-te
st
(con
tinue
d)
228 Adaptive Behavior 25(5)
Tab
le2.C
ontinued
Feed
bac
kim
ple
men
tation
Dom
ain
Adap
tive
feed
bac
km
eans
Adap
tive
feed
bac
kta
rget
Adap
tive
feed
bac
kgo
al(p
edag
ogi
cal
pri
nci
ple
)
Adap
tive
feed
bac
kst
rate
gyIm
ple
men
tation
tech
niq
ue
Eva
luat
ion
tech
niq
ue
INSP
IRE
(Pap
anik
ola
ou,
Gri
gori
adou,
Korn
ilaki
s,&
Mag
oula
s,2003)
Com
pute
rar
chitec
ture
N/A
N/A
N/A
Outc
om
efe
edbac
k.A
ctiv
ele
arner
N/A
Ques
tionnai
rean
ddat
atr
acki
ng
inac
tivi
tylo
gs
FIT
Java
Tuto
r(G
ross
&Pin
kwar
t,2015)
Pro
gram
min
gSt
uden
t’sso
lution
stra
tegy
Self-
refle
ctio
npro
mpts
,F1
and
F2fe
edbac
kst
rate
gies
Exam
ple
-bas
edle
arnin
gth
eory
Leve
lofdet
ail
bas
edon
conse
cutive
com
bin
atio
nofF1
and
F2st
rate
gies
.A
ctiv
ele
arner
Clu
ster
edso
lution
spac
e.R
elat
ional
neu
ralga
s(R
NG
)cl
ust
er
AN
DES
(Van
Lehn
etal
.,2005)
Phy
sics
Studen
t’sm
enta
lst
ate
Pro
vides
flag
feed
bac
kac
com
pan
ied
by
hin
tsan
der
ror
mes
sage
sac
cord
ing
toth
eso
lution
grap
h(d
om
ain
model
)
Bas
edon
const
ruct
ivis
tle
arnin
gth
eory
Sequen
ceofhin
tsw
ith
leve
lsof
det
ail.
Act
ive
lear
ner
Conte
xt-
free
par
ser
and
solu
tion
grap
h
Ques
tionnai
re
SQL-
Tuto
r(M
itro
vic,
Ohls
son,
&Bar
row
,2013)
Pro
gram
min
gSt
uden
t’sunsu
cces
sful
solu
tion
atte
mpts
Rig
ht/
wro
ng,
erro
rfla
g,hin
ts,par
tial
solu
tion,an
dco
mple
teso
lution
(ped
agogi
calm
odule
)
Bas
edon
stat
eco
nst
rain
tth
eory
Leve
lsofdet
ail.
Act
ive
lear
ner
Const
rain
t-bas
edm
odel
ing.
Rel
evan
cean
dsa
tisf
acto
rynet
work
s
Ques
tionnai
re
CO
MPA
SS(G
ouli,
Gogo
ulo
u,
Pap
anik
ola
ou,&
Gri
gori
adou,2006)
Intr
oduc
tory
info
rmat
ics
Studen
t’skn
ow
ledge
leve
l,pre
fere
nce
s,an
din
tera
ctiv
ebeh
avio
r
Com
ple
tenes
sofso
lution,
accu
racy
,an
dm
issi
ng
out
(dom
ain
module
)
Support
for
refle
ctio
nG
radual
pro
visi
on
offe
edbac
kbas
edon
four
laye
rs.
Inac
tive
lear
ner
Conce
pt
map
sEm
pir
ical
studie
s
Mas
on
and
Bru
nin
g(2
001)
Dom
ain
indep
enden
tSt
uden
t’sac
hie
vem
ent
leve
lan
dpri
or
know
ledge
Incr
ease
diff
iculty
of
lear
nin
gta
sk(d
om
ain
module
)
Support
for
new
conce
pt
acquis
itio
nan
dab
stra
ctre
asonin
g
Know
ledge
of
resp
onse
feed
bac
k.A
ctiv
ele
arner
N/A
N/A
Nar
ciss
and
Huth
(200
2)
Dom
ain
indep
enden
tSt
uden
t’sle
arnin
gobje
ctiv
es,pri
or
know
ledge
,le
arnin
gst
rate
gies
,pro
cedura
l,an
dm
eta-
cogn
itiv
esk
ills
Inst
ruct
ional
goal
s,le
arnin
gta
sks,
issu
es,an
dpro
ble
ms
(dom
ain
module
)
Self-
regu
late
dle
arnin
g,beh
avio
ral
and
cogn
itiv
ele
arnin
gth
eori
es
Know
ledge
of
corr
ect
resp
onse
and
elab
ora
tive
feed
bac
kbas
edon
spec
ified
guid
elin
es.
Inac
tive
lear
ner
N/A
N/A
Exce
lTu
tor
(Mat
han
&K
oed
inge
r,2005)
Com
pute
rsc
ience
N/A
Inte
llige
nt
Novi
ceM
odel
whic
hre
pre
sents
the
erro
rsm
ade
by
studen
ts(d
om
ain
module
)
Support
sge
ner
ativ
ean
dev
aluat
ive
lear
nin
g
Imm
edia
teco
rrec
tive
feed
bac
k,gr
adual
2-s
tep
pro
cess
.A
ctiv
eLe
arner
Pro
duct
ion
rule
sA
NO
VA
(con
tinue
d)
Bimba et al. 229
� Student’s knowledge level is mostly used for decid-ing what feedback to provide during problem-solving.
� The forms of feedback in the domain knowledge isthe most common aspect of the instructional systemthat changes based on the student’s characteristics.
� The goal of providing adaptive feedback is mostlydue to the constructivist learning theory.
� Students are usually involved actively in the feed-back process.
� Adaptive feedback is usually presented with a cer-tain level of detail, based on the student’s initialresponse.
� LSA and parsers are the most common techniquesused in implementing adaptive feedback.
� The most common techniques used for evaluatingadaptive feedback implementations are throughquestionnaires, pre-test and post-test, and analysisof log data.
Adaptive feedback seems to be more easily implemen-ted in the programming domain as seen from the fullimplementation of adaptive feedback features in FITJava Tutor and SQL-Tutor. This could be as a result ofthe logical and procedural nature of the programmingdomain.
6. Future direction
Adaptive feedback support is necessary in a computer-based learning environment because of the difference instudents’ characteristics. Based on our review, provid-ing full adaptive feedback is yet to be implemented innon-procedural domains. Further research is requiredto tackle this issue. Subsequently, there is a need for anadaptive feedback framework which can accommodatethe various adaptive feedback criteria presented andsupport multiple adaptive feedback means, target, goal,and strategy. This will allow for a better evaluation ofthe following:
� Is there a right combination of adaptive feedbackmeans, target, goal, and strategy which caters for aparticular student?
� Can multiple student characteristics be used effi-ciently for providing efficient feedback?
An area which requires further investigation is the com-plexity of aligning multiple adaptive feedback charac-teristics to a specific student’s needs.
7. Conclusion
Feedback is an effective tool used in typical classroomsettings during teaching. However, the feedback pro-vided is usually to a group of students with differentT
ab
le2.C
ontinued
Feed
bac
kim
ple
men
tation
Dom
ain
Adap
tive
feed
bac
km
eans
Adap
tive
feed
bac
kta
rget
Adap
tive
feed
bac
kgo
al(p
edag
ogi
cal
pri
nci
ple
)
Adap
tive
feed
bac
kst
rate
gyIm
ple
men
tation
tech
niq
ue
Eva
luat
ion
tech
niq
ue
Anim
alw
atch
(Arr
oyo
,Bec
k,W
oolf,
Bea
l,&
Schultz,
2000)
Mat
hem
atic
sSt
uden
t’sco
gnitiv
edev
elopm
ent
and
gender
N/A
N/A
Incr
easi
ng
amount
ofin
form
atio
n.
Inac
tive
lear
ner
Mac
hin
ele
arnin
gA
NO
VA
Tso
valtzi
and
Fied
ler
(200
3)
Mat
hem
atic
sSt
uden
t’scu
rren
tan
dpre
vious
answ
ers
N/A
Socr
atic
tuto
ring
stra
tegy
Gra
dual
pro
visi
on
ofhin
ts.In
active
lear
ner
Onto
logy
and
pro
duc
tion
rule
sN
/A
Bokh
ove
,K
ools
tra,
Boon,a
nd
Hec
k(2
007)
Mat
hem
atic
sN
/ALo
calfe
edbac
kin
dic
atin
gco
rrec
t,in
com
ple
te,an
dw
rong
answ
ers
N/A
N/A
N/A
Shar
able
conte
nt
obje
ctre
fere
nce
model
WEA
R(V
irvo
u&
Moun
dri
dou,2000)
Mat
hem
atic
sSt
uden
t’skn
ow
ledge
leve
lLo
calfe
edbac
kin
dic
atin
gco
rrec
t,in
com
ple
te,an
dw
rong
answ
ers
N/A
N/A
N/A
Em
pir
ical
study
N/A
:not
appl
icab
le.
230 Adaptive Behavior 25(5)
characteristics. This results in a gap between studentswho excel the most and those who excel less. Providingadaptive feedback that caters for students based ontheir individual characteristics have been implemen-ted in computer-based learning environments witheffective results. This review focuses on 20 differentimplementations of adaptive feedback in computer-based learning environment, ranging from intelligenttutoring system (ITS), multimedia web-based ITS,dialog-based ITS, web-based intelligent e-learningsystem, adaptive hypermedia system, and intelligentand adaptive learning environment. These implemen-tations were carefully selected based on their impactin providing feedback to students. The main objectiveof the review is to compare adaptive feedback systemsaccording to feedback adaptation characteristics andidentify major open research questions in adaptivefeedback implementations.
The review resulted in categorizing these feedbackimplementations based on the students’ informationused for providing feedback (adaptive feedback means),the aspect of the domain or pedagogical knowledge thatis adapted to provide feedback based on the students’characteristics (adaptive feedback target), the pedagogi-cal reason for providing feedback (adaptive feedbackgoal), and the steps taken to provide feedback with orwithout students’ participation (adaptive feedback strat-egy). Other information such as the common adaptivefeedback means, goals, and implementation techniquesare identified. This review reviles a distinct relationshipbetween the characteristics of feedback, features ofadaptive feedback, and computer-based learning models(pedagogy, domain, and student models).
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest withrespect to the research, authorship, and/or publication of thisarticle.
Funding
The author(s) disclosed receipt of the following financial sup-port for the research, authorship, and/or publication of thisarticle: This work was supported by the University of MalayaResearch Grant (RP040B-15AET, 2015).
References
Advisors, E. G. (2013). Learning to adapt: A case for accelerat-
ing adaptive learning in higher education. Education Growth
Advisors. Retrieved from http://tytonpartners.com/library/
accelerating-adaptive-learning-in-higher-educationArroyo, I., Beal, C., Bergman, A., Lindenmuth, M., Marshall,
D., & Woolf, B. P. (2003). Intelligent tutoring for high-
stakes achievement tests. Artificial Intelligence in Educa-
tion: Shaping the Future of Learning Through Intelligent
Technologies, 97, 365.
Arroyo, I., Beck, J. E., Woolf, B. P., Beal, C. R., & Schultz,
K. (2000). Macroadapting animalwatch to gender and cog-nitive differences with respect to hint interactivity and sym-
bolism. In International Conference on Intelligent Tutoring
Systems (Vol. s1839, pp. 574–583).Arroyo, I., Woolf, B. P., Burelson, W., Muldner, K., Rai, D.,
& Tai, M. (2014). A multimedia adaptive tutoring systemfor mathematics that addresses cognition, metacognition
and affect. International Journal of Artificial Intelligence in
Education, 24, 387–426. New York: Springer.Billings, D. (2012). Efficacy of adaptive feedback strategies in
simulation-based training. Military Psychology, 24,114–133.
Bokhove, C., Koolstra, G., Boon, P., & Heck, A. (2007).
Towards an integrated learning environment for mathe-
matics. In 8th International Conference on Technology in
Mathematics Teaching. Czech Republic, 1–4 July 2007, 5pp.
Brusilovsky, P. (1998). Methods and techniques of adaptive
hypermedia. In Adaptive hypertext and hypermedia (pp. 1–
43). Netherlands: Springer.Brusilovsky, P. (1999). Adaptive and intelligent technologies
for web-based education. Ki, 13(4), 19–25.Carter, J. (1984). Instructional learner feedback: A literature
review with implications for software development. Com-
puting Teacher, 12, 53–55. Retrieved from https:
//www.learntechlib.org/p/169781Chieu, V. M. (2005). Constructivist learning: An operational
approach for designing adaptive learning environments sup-
porting cognitive flexibility (Unpublished doctoral disserta-
tion). Universiteit Twente, Enschede, The Netherlands.D’Mello, S., & Graesser, A. (2012). Autotutor and affective
autotutor: Learning by talking with cognitively and emo-
tionally intelligent computers that talk back. ACM Trans-
actions on Interactive Intelligent Systems (Tiis), 2(4),
Article 23.Economides, A. A. (2006). Adaptive feedback characteristics
in cat. International Journal of Instructional Technology
and Distance Learning, 3(8), 15–54.Farid, S., Ahmad, R., & Alam, M. (2015). A hierarchical
model for e-learning implementation challenges using
AHP. Malaysian Journal of Computer Science, 28(3), 166–
188.Gerdes, A., Jeuring, J., & Heeren, B. (2012). An interactive
functional programming tutor. In Proceedings of the 17th
ACM Annual Conference on Innovation and Technology in
Computer Science Education (pp. 250–255). New York,
NY, USA: ACM.Gertner, A. S., & VanLehn, K. (2000). Andes: A coached
problem solving environment for physics. In International
Conference on Intelligent Tutoring Systems (pp. 133–142).
Berlin, Heidelberg: Springer.Ghauth, K. I., & Abdullah, N. A. (2010). An empirical eva-
luation of learner performance in e-learning recommender
systems and an adaptive hypermedia system. Malaysian
Journal of Computer Science, 23, 141–152.Gouli, E., Gogoulou, A., Papanikolaou, K. A., & Grigoriadou,
M. (2006). An adaptive feedback framework to support
reflection, guiding and tutoring. In Advances in web-based
education: Personalized learning environments (pp. 178–202).Hershey, PA, USA: Information Science Publishing.
Bimba et al. 231
Gross, S., Mokbel, B., Paassen, B., Hammer, B., & Pinkwart,
N. (2014). Example-based feedback provision using struc-
tured solution spaces. International Journal of Learning
Technology, 10, 9248–9280.Gross, S., & Pinkwart, N. (2015). Towards an integrative
learning environment for java programming. In 2015 IEEE
15th International Conference on Advanced Learning Tech-
nologies (pp. 24–28).Hualien, China: IEEE.Hattie, J., & Gan, M. (2011). Instruction based on feedback.
In Handbook of research on learning and instruction (pp.
249–271). New York: Routledge.Heift, T. (2010). Developing an intelligent language tutor.
CALICO Journal, 27, 443–459.Heift, T. (2016). Web delivery of adaptive and interactive lan-
guage tutoring: Revisited. International Journal of Artificial
Intelligence in Education, 26, 489–503.Heift, T., & McFetridge, P. (1999). Exploiting the student
model to emphasize language teaching pedagogy in nat-
ural language processing. In Proceedings of a Sympo-
sium on Computer Mediated Language Assessment and
Evaluation in Natural Language Processing (pp. 55–61).
Stroudsburg, PA, USA: Association for Computational
Linguistics.Heift, T., & Nicholson, D. (2001). Web delivery of adaptive
and interactive language tutoring. International Journal of
Artificial Intelligence in Education, 12, 310–325.Heift, T., & Schulze, M. (2007). Errors and intelligence in com-
puter-assisted language learning: Parsers and pedagogues.
New York, USA: Routledge.Hepplestone, S., Holden, G., Irwin, B., Parkin, H., & Thorpe,
L. P. (2011). Using technology to encourage student
engagement with feedback: A literature review. Research in
Learning Technology, 19, 117–127. doi:10.1080/21567069.
2011.586677Le, N.-T. (2016). A classification of adaptive feedback in edu-
cational systems for programming. Systems, 4(2), 22. doi:
10.3390/systems4020022Lo, J. J., Chan, Y. C., & Yeh, S. W. (2012). Designing an
adaptive web-based learning system based on students’
cognitive styles identified online. Computers & Education,
58, 209–222. Retrieved from doi:10.1016/j.compedu.2011.
08.018Luckin, R., & Holmes, W. (2016). Intelligence unleashed: An
argument for AI in education. London: Pearson.Mason, B. J., & Bruning, R. (2001). Providing feedback in com-
puter-based instruction: What the research tells us (Tech.
Rep.). University of Nebraska. Retrieved from https://
www.researchgate.net/publication/247291218_Providing_
Feedback_in_Computer-based_Instruction_What_the_
Research_Tells_UsMathan, S. A., & Koedinger, K. R. (2005). Fostering the
intelligent novice: Learning from errors with metacogni-
tive tutoring. Educational Psychologist, 40, 257–265.Melis, E., Goguadze, G., Libbrecht, P., & Ullrich, C. (2009).
Activemath—A learning platform with semantic web fea-
tures. The Future of Learning, 159–177.Melis, E., Moormann, M., Ullrich, C., Goguadze, G., & Lib-
brecht, P. (2007). How activemath supports moderate con-
structivist mathematics teaching. In 8th International
Conference on Technology in Mathematics Teaching, Hra-
dec Kralove.
Mitrovic, A. (2003). An intelligent SQL tutor on the web.
International Journal of Artificial Intelligence in Education,
13, 173–197.Mitrovic, A., & Ohlsson, S. (1999). Evaluation of a
constraint-based tutor for a database language. Interna-
tional Journal of Artificial Intelligence in Education, 10,
238–256.Mitrovic, A., Ohlsson, S., & Barrow, D. K. (2013). The effect
of positive feedback in a constraint-based intelligent tutor-
ing system. Computers & Education, 60, 264–272.Moundridou, M., & Virvou, M. (2002). WEAR: A web-based
authoring tool for building intelligent tutoring systems. In
Proceedings of the 2nd Helenic Conference on AI, Compa-
nion Volume (pp. 203–214). Thessaloniki, Greece.Munoz-Merino, P. J., Pardo, A., Scheffel, M., Niemann, K.,
Wolpers, M., Leony, D., & Delgado Kloos, C. (2011). An
ontological framework for adaptive feedback to support
students while programming. In Proceedings of the 10th
International Semantic Web Conference (pp. 1–4). Bonn,
Germany.Narciss, S. (2013). Designing and evaluating tutoring feed-
back strategies for digital learning. Digital Education
Review, 23, 7–26.Narciss, S., & Huth, K. (2002). How to design informative
tutoring feedback for multimedia learning. In Instructional
design for multimedia learning (pp. 181–195). Retrieved from
http://www.mendeley.com/catalog/design-informative-
tutoring-feedback-multimedia-learning/Narciss, S., Sosnovsky, S., Schnaubert, L., Andres, E., Eichel-
mann, A., Goguadze, G., & Melis, E. (2014). Exploring
feedback and student characteristics relevant for persona-
lizing feedback strategies. Computers & Education, 71,
56–76. doi:10.1016/j.compedu.2013.09.011Nye, B. D., Graesser, A. C., & Hu, X. (2014). Autotutor and
family: A review of 17 years of natural language tutoring.
International Journal of Artificial Intelligence in Education,
24, 427–469.Olney, A. M., D’Mello, S., Person, N., Cade, W., Hays, P.,
Williams, C., . . . Graesser, A. (2012). Guru: A computer
tutor that models expert human tutors. In Intelligent tutor-
ing systems (pp. 256–261). Berlin, Heidelberg: Springer.Olney, A. M., Person, N. K., & Graesser, A. C. (2011). Guru:
Designing. In Cross-disciplinary advances in applied natural
language processing: Issues and approaches (p. 156) Her-
shey, PA, USA: IGI Global.Papanikolaou, K. A., Grigoriadou, M., Kornilakis, H., &
Magoulas, G. D. (2003). Personalizing the interaction in a
web-based educational hypermedia system: The case of
INSPIRE. User Modeling and User-Adapted Interaction,
13, 213–267.Prince, M. J., & Felder, R. M. (2006). Inductive teaching and
learning methods: Definitions, comparisons, and research
bases. Journal of Engineering Education, 95, 123–138. doi:
10.1002/j.2168-9830.2006.tb00884.xPryor, J., & Crossouard, B. (2008). A socio-cultural theorisa-
tion of formative assessment. Oxford Review of Education,
34(1), 1–20.Rivers, K., & Koedinger, K. R. (2015). Data-driven hint gen-
eration in vast solution spaces: A self-improving python
programming tutor. International Journal of Artificial
Intelligence in Education, 27, 37–64.
232 Adaptive Behavior 25(5)
Rus, V., Conley, M., & Graesser, A. (2014). The dendrogrammodel of instruction: On instructional strategies and theirimplementation in DeepTutor. Design Recommendations
for Intelligent Tutoring Systems, 311–325.Rus, V., Niraula, N. B., & Banjade, R. (2015). Deeptutor: An
effective, online intelligent tutoring system that promotesdeep learning. In AAAI (pp. 4294–4295). Palo Alto, Cali-fornia: AAAI Press.
Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. M.(2010). Computer-supported argumentation: A review ofthe state of the art. International Journal of Computer-Sup-
ported Collaborative Learning, 5, 43–102. doi:10.1007/s11412-009-9080-x
Specht, M., Kravcik, M., Klemke, R., Pesin, L., & Hutten-hain, R. (2002). Adaptive learning environment for teach-ing and learning in winds. In International conference on
adaptive hypermedia and adaptive web-based systems (pp.
572–575). Berlin, Heidelberg: Springer-Verlag.Specht, M. E. (1998). Adaptive methods in computer-based
teaching/learning systems. New York, NY: ACM.Stoyanov, S., & Kirchner, P. (2004). Expert concept mapping
method for defining the characteristics of adaptive
E-learning: ALFANET project case. Educational Technol-
ogy Research & Development, 52, 41–54.Tsovaltzi, D., & Fiedler, A. (2003). Enhancement and use of a
mathematical ontology in a tutorial dialog system. In Pro-
ceedings of the IJCAI Workshop on Knowledge Representa-
tion and Automated Reasoning for E-learning Systems (pp.
23–35). Acapulco, Mexico.VanLehn, K., Lynch, C., Schulze, K., Shapiro, J. A., Shelby,
R., Taylor, L., . . . Wintersgill, M. (2005). The Andes phy-
sics tutoring system: Lessons learned. International Journal
of Artificial Intelligence in Education, 15, 147–204.Virvou, M., & Moundridou, M. (2000). Modelling the
instructor in a web-based authoring tool for algebrarelated
ITSs. In International Conference on Intelligent Tutoring
Systems (pp. 635–644). Berlin, Heidelberg: Springer-
Verlag.Virvou, M., & Moundridou, M. (2001). Student and instruc-
tor models: Two kinds of user model and their interaction
in an its authoring tool. In International Conference on
User Modeling (pp. 158–167). Berlin, Heidelberg: Springer-
Verlag.
About the Authors
Andrew Thomas Bimba received a BEng in Electrical and Electronics Engineering in 2006 anda Master’s degree in Computer Science (Artificial Intelligence) in 2014. He is currently a PhDstudent in Computer Science at University of Malaya. His research interest includes cognitiveknowledge base, natural language processing, artificial intelligence in education, machine learn-ing, and computer–human interaction.
Norisma Idris received a BSc degree, Master’s degree, and PhD in Computer Science at theUniversity of Malaya. Her area of interest includes artificial intelligence in education (summari-zation, summary sentence decomposition, heuristic rules, understanding and categorization,essay grading system) and natural language (Malay text processing, text normalization, stem-ming algorithm). She is currently the head of Artificial Intelligence Department at the Universityof Malaya.
Ahmed Al-Hunaiyyan received a BS in BA in 1983 at Kuwait University, an MS degree in MISfrom Aurora University, Illinois, USA, in 1988, and a PhD in Computer Science atHertfordshire University, UK, in 2000. He has lecturing and training experience in multimediaapplications and authoring, database systems and application, management information systems,programming languages, just to mention a few. His research interest includes web-based tutors,multimedia applications, e-learning, human–computer interaction, and knowledge base. He iscurrently an assistant professor in Computer and Information Systems Department, PublicAuthority for Applied Education and Training, Kuwait.
Rohana Binti Mahmud is a senior lecturer at the Department of Artificial Intelligence,University of Malaya. She received a BSc degree at Waikato University, New Zealand, an MScdegree at Universiti Sains Malaysia, and a PhD at University of Manchester, UK. Her area ofexpertise includes natural language (Discourse Structure, Lexical Relation, Malay LanguageText Processing), expert system (Knowledge Base System, Multi-agent Expert System, ExpertTutoring System), and education (AI in Education, Soft skills, Higher Order Thinking Skills).
Bimba et al. 233
Nor Liyana Bt Mohd Shuib is a lecturer at Department of Information System in the Facultyof Computer Science & Information Technology, University of Malaya. She graduated withBCS (Information System) in 2005 from Universiti Teknologi Malaysia and MIT (InformationTechnology) from Universiti Kebangsaan Malaysia, in 2007. She obtained her PhD in 2013 fromUniversiti of Malaya, Malaysia. Her research interests include data mining, education technol-ogy, and information retrieval tools.
234 Adaptive Behavior 25(5)