Student-teacher communication directed to computer-based learning environments

ELSEVIER

DISPLAYS Displays 17 (1997) 167-178

Student-teacher communication directed to computer-based learning environments

R. Marin, P.J. Sanz, 0. Coltell, J.M. Iiiesta, F. Barber, D. Corella Computer Science Department, Jaume I University, Campus de Penyrta Roja, 12071 Castellon, Spain

Abstract

The main objective of this paper is to provide an optimal way for students to control a computer-based teacher and interact with it. From previous experience, we took a special interest in improving the navigation through tutorial systems, and also in the evaluation techniques, designing and developing a new tutorial based on an open architecture. After implementing the tutorial prototype, it was evaluated by a selected group of users in a controlled laboratory situation in order to gather data about the characteristics and usability of this prototype. In general, users had a good overall opinion of the evaluated tutorial. The key idea behind our experience is the introduction of tutorials in all practical classes as a complement to the instructor in the near future. 0 1997 Elsevier Science B.V.

Keywords: Multimedia educational applications; Human-computer interface; Computer-based learning environment

1. Introduction

A possible classification of the different approaches used in computer-assisted teaching relies on distinguishing between the instructionist and constructionist approaches [l]. The latter considers the role of the computer in an educational environment, not as an information source but as a design tool that permits the student the possibility of building up a product by himself or herself that then can be made public [2]. On the other hand, the instructionist approach, derived from conductism, was the pioneer of the computer applications in the teaching world [3]. The paradigm of a conductist program presents a sequence of data and questions, records the answers from the user, provides a number of error messages, and offers new objectives as the users reach a certain level of knowledge. The present work is related to this approach, in which most of the research work is currently being produced.

In Fig. 1 the main lines of research within this approach are displayed from left to right, in increasing order of com- putational complexity. The ‘precision teaching’ consists of decomposing complex tasks into simpler sub-tasks and training these sub-tasks until automation is reached [3]. The ‘computer-aided instruction’ (CAI) procedures are mainly focused on the adaptive capability of the student. One of the most known CA1 techniques is ‘case-based teaching’ [4], which derives from the artificial intelligence ‘case-based reasoning’ topic. Currently, the most advanced systems are the ‘intelligent tutorial systems’ (ITS). ITS are the systems that offer the highest capability to be adapted to

0141-9382/97/$17.00 0 1997 Elsevier Science B.V. All rights reserved PII SOl41-9382(96)01033-5

the knowledge level of each student. In order to attain this, they need a cognitive model of the student and an expert system (ES) that can provide the suitable explanations to the student at the right moment.

Nevertheless, the setting up of the ITS is costly due to the tool complexity and the need to build up an ES on the corresponding domain of knowledge. A certain lapse of time for the system to adapt to the student and reach its optimal performance is also needed.

A possibility that makes the design of the user-tutorial interface easier is to build a tutorial system without the most complex modules of ITS (ES, and cognitive model of the student). With this approach, the feedback mechanism between the user and the system, needed in the design stage, is clearly simplified. The non-intelligent system also has the advantage that all its parameters can easily be controlled because there is no variability among students, favouring the studies (statistical, etc.) for assessing the prototype by users and making the design stage more efficient.

In relation to the possibilities that multimedia provides, we can state that the main incentive of this new medium for information transmission is its capability to establish a satisfactory man-machine communication [.5,6]. Navigation of the various option menus offered to the user is made easier by means of hypertext, sound, animations, etc. But new design and structural needs arise [7,8]. The key is again creativity and imagination from the researchers involved in this new teaching approach.

Our main objective in this paper has been to make a prototype of a tutorial taking into account the above

168 R. Marin et d/Displays 17 (1997) 167-178

COMPUAllONAL COMPLEXITY

Fig. I. Schematic representation of the common approach used by the computer teaching methodology (instructlonist).

considerations and to introduce it in a practical computer science lecture. Similar works in this line include, for example, the AT&T teaching theatre at the University of Maryland [9].

Once the prototype has been implemented and the modular functionality and integration tests have been concluded, the next objective is the evaluation of the prototype by a selected group of users in a controlled laboratory situation in order to gather data about the characteristics and usability of this prototype.

This project has been integrated as complementary to the computation practicum related to an introductory language programming subject, which is part of a first course in computer science engineering.

2. A case study

2.1. Problem description

The program intends to be an educational prototype with the aim of teaching advanced topics in data structures applied to the Pascal programming language, such as binary tree expressions, recursion, and dynamic memory alloca- tion. Bearing this in mind a guided example has been developed based on a calculator construction simulated by a binary tree.

2.1.1. The need for a multimedia tutorial The greatest problem for students of first courses in tech-

nical studies (and, of course, in computer science studies) is the imbalance of skills among the students. Some come from the high school with a suitable level in mathematics but have not yet seen a computer; others come from profes- sional schools and have already been programming for three years; and others are older people who have been working with computers for many years and who have decided to obtain a university degree. The problem is worsened by the large number of students in each group, which makes personalized attention impossible for the lecturer.

The other problem is student overbooking at the computer classrooms. This situation is disappointing for students who prefer to learn by themselves and work away from the

computer classroom, thus losing out on the lecturer guides and explanations.

To solve these problems, we took the decision of providing complementary tutorials, looking for a personalized teacher for each student, thus enabling one to set the pace of the learning process. This automated teacher should have enough information for taking the place of the lecturer in some laboratory courses. The aim is to enable the student to learn by interacting with the program, solving questions, getting feedback, playing, etc., trying a new way of studying and learning through the program.

To reach these goals we have to face the problem of human psychology. In order to design an educative program a study of student psychology is required to avoid the need for manuals, allowing intuition to guide the student. This forces us to pose a man-machine interface that simplifies somehow the messages that the student can send to the teacher during the learning process.

A number of studies have been carried out in recent years on interfaces for educative programs, the ‘button theory’ [7] being one of the most significant ones. A simplification of the set of buttons in that theory has been tried in this work and the results have proved to be satisfactory.

2.1.2. Resources We have used a multimedia theatre comprising 10 PC

Pentium computers with sound cards and enough disk space and RAM for the validation experience; the program has been developed in a full multimedia-equipped PC. Those are sufficient resources but we need to ensure the suitable running of the tutorial in smaller platforms and sufficient portability for running either under Windows 3.x or Windows 95. Thus, Microsoft routines fully integrated in Windows 95 should be avoided (Video for Windows, Movie Player, etc.). In conclusion, the tutorial will be designed for being portable enough to run under any Windows version higher than 3.0, and for being able to run suitably in inter- mediate series PC computers.

2.2. Objectives

The goals of this project can be divided into seven main points.

R. Marin et d/Displays 17 (1997) 167-178 169

2.2.1. User control versus system control The highly qualified student is expected just to try to

clarify some aspects about the subject that were not suffi- ciently clear from the explanations of the lecturer. This user looks for an organized and flexible way to reach examples and exercises for getting the concrete desired knowledge. In this case, the student wants to control the pace of the learning process (user control).

On the other hand, the lower-level student wants a helping hand to guide him step by step across the subject. In this case, a rigid tool that controls the interaction and set the pace is needed (system control).

Our aim is to design a tutorial able to deal with both approaches: user and system control. As we shall see below, this is attained, including some ‘quick access buttons’ to different key parts of the tutorial, and structuring it through directed graphs.

2.2.2. Velocir?, control for the explanations Lecturer explanations addressed to a hypothetical average

student bore the high-level ones and can frustrate the lower- level ones. Concerning this aspect, the velocity at which concept explanations are provided is a key point. Thus, in the tutorial, depending on the proposed level the subject will be explained faster or slower.

2.2.3. History record generation An additional advantage of the automated teacher is the

possibility of saving data from each student. Some interest- ing items are: identity (name, studies, course number, etc.), times spent doing certain tasks, non-visited parts of the tutorial, passed questions, mistaken answers, etc.

2.2.4. Automatic evaluation The above-mentioned history record allows the automatic

assessment of the student during the tutorial running. We can know the times spent for each tutorial section, so we can infer the concepts to which more attention has been paid, the total running time, etc. Moreover, the user is asked to pass certain tests, and the qualifications are archived in the record.

2.2.5. Connection with the programming development tool This tutorial tries to clarify certain aspects of the Pascal

language such as binary trees for assessing arithmetic expressions, recursion, etc. Our aim is to integrate the tutorial and the programming tool, in such a way that the student can learn the concepts and practise them with the programming environment, like a kind of teacher-assisted practice.

2.2.6. Open architecture We have tried to program keeping in mind the possibility

of a possible ‘intelligent tutorial’ [lo] to be designed in the future, in which AI techniques can be applied in order to solve the concrete problems of any student so that it can be

adapted to each one in run-time. The main features of the student model are the knowledge level, response time, and learning ability.

2.2.7. Robust implementation As stated above, we aim to run the tutorial on different

Windows versions and resource availability, so a robust programming scheme able to run in the worst case has been carried out. Thus, if the host does not have a sound card this does not mean an obstacle in running the program; thus sound is not absolutely necessary but is convenient because it makes the tutorial more attractive from a psycho- logical point of view. The same is considered for a Windows 3.x system where multimedia DLLs are not provided, but the animations must also run. In most cases these considerations have motivated a low-level programming approach for our multimedia routines. An example of this is the AJI animation file format (Jaume I Animation), which includes the frame listing and other aspects such as delays between them, image formats, etc.

2.3. Methodology and tools

In this section the theatrical concepts and the practical steps (programming tools) considered for developing the prototype are described.

A number of works have been developed in some uni- versities involving tutorials featuring the design of man- machine interfaces that model the possible student-teacher relations [ 11,7,8,6,10]. The North-Western University’s Institute for Learning Sciences (ILS) [4] has been experi- menting with a set of buttons that would enable users of educational software systems to ask questions, make requests or communicate something about their mental state, and thus to be in control of their educational process. These efforts have been translated into the so-called ‘button theory’, introduced above, which has been developed in order to allow students using computer-based learning environments to have as much control as possible over what they see, hear, and learn. Besides, this computerized approximation to the traditional teacher has an advantage related to the different speed that each student needs to assimilate the same concept. With a well-structured tutorial the students have more control about their own learning process and the role of the lecturer is less influential in the teaching domain.

In the model, the messages directed to the program are organized into three groups: feelings (the student may wish to talk about himself), questions (about the domain or task of instruction), and control (questions about the process of teaching itself). More specifically we have:

Feelings: This category concerns the student’s internal state. The students may feel bored or excited by the domain, may find their belief challenged, or may be frustrated by their inability to understand the material.

170 R. Marin et aUDisplays 17 (1997) 167-178

Control I/

Fig. 2. Categories on the ‘button theory’.

Each student will also have different reactions to different teaching styles and different matters. Questions: This group involves the efforts of the student in comprehending the subject of the task. The user may be seeking to understand the reasons of a particular theory or may want to know what to do next. Control: This category involves messages about the progress of the lesson. The student may wish to speed up or slow down the rate of instruction, or may wish to have material presented in greater detail. These buttons are intended to deal with the style of teaching and not with the domain.

Graphically, these three groups are related to the domains of discourse showed in Fig. 2.

Within each of the above categories, the three most important communicative acts have been selected, assigning to each one a button on the ‘button pad’. The nine buttons considered can be seen in Fig. 3. Providing the student with such a button set shifts the control of the learning environment from the computer to the user. But this implies that questions that might arise and the different types of answers should have been already considered. In fact, this model represents a complete set of situations that can be found in a communicative interaction with a tutorial, but in practice each designer will choose more or fewer items in each category depending on the level of complexity that will put into the implementation.

This model represents only one reasonable solution that may or not be suitable. The reality is that the ‘button pad’ has been used in several computer-based learning environments and it provides a good balance between ease of use

Feelings Questions Control

Tu had! why? who1 nxt?

J30* HOW? Big Picture

@ No way Hllh? HismY

Fig. 3. Button pad on the ‘button theory’

and expressivity. In fact, the effectiveness of a set of buttons depends on two points: first, whether it allows students to express themselves; and second, whether the number of possible messages permits the programmer to implement all the possible states that can be generated by the user.

Taking these considerations into account, we are going to describe the integration of the ‘button theory’ in our prototype. The first goal was to select the range of buttons that would be included in our tutorial.

From a satisfactory previous experience, by means of a tutorial based on multimedia about programming topics, our users claimed more control and flexibility. They needed a consulting program to rapidly solve some aspects of the subject which were unclear. Thus, the buttons selected were:

1. Index 2. Go to begin 3. Skip this 4. Back up 5. Exit 6. What next?

The ‘Index’ button has been added to speed up a simple query to the system. That permits us to go directly to all the tasks included in the tutorial. The ‘Exit’ button has the same purpose and it ensures that one can exit from all the possible states of interaction that the tutorial supports. This button has been discarded only in the evaluation tasks, thus forcing the user to finish the test and to achieve the final result. The other new button is ‘Go to begin’, which permits less advanced students to begin a new interaction with this computer-based learning environment. This button is the alter- native to the consulting mode that offers the tutorial.

In order to simplify this prototype design, we have post- poned the implementation of the feeling and question buttons. This has meant a great simplification that will be taken into account in the next version.

2.3.1. Authoring systems versus standard programming languages

One of the main problems in attacking the tutorial design was choosing the programming tool. Two main directions may be chosen for multimedia design: first, using high-level tools (authoring systems) for advanced presentations (Core1 Move, TX-Authoring, Multimedia ToolBook, etc.) that offer multimedia routines as part of the software and provide a visual programming environment using icons; and second, using a standard programming language (Borland C + + , Visual Basic, Delphi, etc.), that offers all possibilities but is harder to program, since almost all multimedia routines have to be designed and implemented by the developer.

The first approach was tried in a previous experience, and a number of conclusions can be drawn from that work: the main advantage of these tools is the simplicity of creating images, animations, sounds, etc. Moreover, since the code is presented using icons in a visual form, the directed graph required to implement the prototype that uses the button

R. Marin et d/Displays 17 (1997) 167-178 171

theory is more easily understood. But when the design of a version able to record data from the students, use AI techniques, or integrate the tutorial and the programming environment, etc. is posed, one realizes that these kinds of high-level tools do not have those capabilities.

Having stated these facts, we conclude that a lot of problems arise when trying to design tutorials using authoring tools. These problems led us to change our point of view and try to develop the tutorial using open programming tools like high-level languages such as Delphi, an evolution of Pascal that offers advanced tools for building user-friendly programs. This approach permits definition of personalized multimedia formats, avoiding dependence on external routines of the operating system. Moreover, regarding sound and similar aspects, all platform exceptions can now be managed without problems.

2.4. Prototype description

In the following, we describe the specific aspects related to the developed tutorial. We can see in Fig. 4 the introduction to the tutorial by means of a nice lecturer.

2.4. I. ‘Button pad ’ Fig. 5 shows some of the buttons utilized in the prototype.

At the top the buttons reserved for the administrator are found, i.e. buttons that make reference to the management of the student’s DB. These buttons are only active when the administrator’s password is introduced.

At the bottom the global buttons are found, i.e. buttons that appear on all the possible screens of the prototype. The first one, ‘BEGIN’, will take us to the beginning of the tutorial contents so the student may follow the steps given by the tutorial. This type of interaction is reserved for the less advanced students, who need guidance over the entire learning process.

The next button, ‘INDEX’, refers to the second way of accessing the topics of the tutorial. As we will see later, this button gives access to an index screen that allows one to go directly to all the key points of the tutorial (Fig. 6). This button represents another way of interacting with the program, and it is intended for advanced students who already have some knowledge of data structures and only want to consult some particular item. In this second way of interacting the student has control over the learning process.

The last button of global type is represented at the bottom right as ‘EXIT’. With this button the program saves all the data obtained from the user and exits to the operating system.

The rest of the buttons that appear in the program are not

Fig. 4. Watching arithmetic expressions and binary trees.

172 R. Marin et d/Displays I7 (I 997) 167- I78

Fig. 5. Prototype presentation screen.

global: they are context-dependent and will vary depending on where the user is.

2.4.2. Interactive program The aim of the program is not to provide a book about the

matter of data structures for the student, but a way of learning these concepts while interacting with the program. We think the best learning approach is based on ‘making something’ in place of ‘viewing something’.

One requirement of this type of learning consists in show- ing the minimum amount of text on the screen. We must be schematic and take advantage of visual intuition for representing concepts.

Looking at Fig. 4 we can see that the student has the opportunity to try different arithmetic expressions and let the computer calculate and show the corresponding prefix and postfix expressions on the screen. The prototype also shows the associated binary tree, helping the student in the comprehension of the concepts.

2.4.3. Flexibility Flexibility is another of the requirements of our proto-

type; it allows quick queries to specific points by students with knowledge in the area of the tutorial.

This flexibility is accomplished in different ways, the most significant being the ‘INDEX’ button. This button shows the direct access screen (Fig. 7), which allows one to go directly to those specific topics. This screen also plays the role of summarizing the tutorial for less experienced students.

2.4.4. Connectivity with the environment Another of the main objectives of our prototype is the

possibility of connecting the tutorial with the programming environment (in our case Borland Pascal 7.0 for Windows) so the student can program by following the instructions in the tutorial.

2.4.5. Sound Undoubtedly, one of the main tools that multimedia

offers for making educational programs is sound. With sound we can avoid the overload of text on the screen, which only proves confusing to the student as he receives too much information at once. The possibility of sound allows one to include oral explanations, applause when a question is correctly answered, ‘happy’ music for the introduction and also ‘soft’ music over all the tutorial to make the user feel comfortable.

R. Marin et al/Displays 17 (1997) 167-178 173

Fig. 6. Index screen.

The unique problem with sound is that there are many computers that do not have it, so we should not design a program based completely on this medium. The program has to be equally pedagogical also when the user does not have sound capability in the computer. Moreover, for this reason, the information transferred will be always the same independent of the resources available.

2.5. Prototype evaluation methodology

Once the prototype is completed and the modular func-

Pleasantness degree

60

50

40

%30

20

10

0 EAYENO

Fig. 7. Opinion about the degree of enjoyment of the tutorial.

tionality and integration tests are concluded, the next stage is directed to a training period over a selected group of users in a controlled laboratory situation [ 12,3]. With these aims, two tests were included in the prototype. First is an internal test, which the user must pass before the session is concluded. This test aims to evaluate user performance with regard to the several concepts included in the tutorial and its assimilation ought to the interface used. The second test is an external questionnaire designed to evaluate the user’s subjective opinions [ 121.

2.5.1. Participants Thirty-nine first-year students of the official course in

Computer Sciences at the Jaume I University, took part in this study. The mean age was 20.4 years (range 18-26). The group was composed of 16 women and 23 men.

2.5.2. Details of the two evaluation types and procedure

2.5.2.1. Objective internal test. The internal test was divided into two distinct parts: (A) questions about concepts, and (B) a small program (named ghost). Part A contains 10 questions with a single solution to each one, selected from 10 possibilities. Each successful answer was scored 1 point, and otherwise 0 points. Scores could range from 0 to 10

174 R. Marin et al/Displays 17 (1997) 167-I 78

points, with higher scores representing better learning. Part B is a small program which aims to be visible in one display only and related to the solution of a specific case. The user must find the solution following the program execution step by step. The final score is 0 or 10 points, respectively. Summarizing, the total score for each user ranges from 0 to 10 points (both parts averaged).

With the aim of having more control parameters in the training sessions, the tutorial controls the periods of time employed in each of the parts by the user.

All.these data, together with the identification, studies and user’s course, are registered in a file supported by the server.

2.5.2.2. Users’ opinions. To evaluate the effectiveness of the prototype in relation to the human-computer interface aspects, a questionnaire was developed collecting the users’ subjective opinions [3]. The questionnaire included a general part about multimedia, traditional teaching, importance of the lecture in the learning process, didactic value of the tutorials, and about tutorial preferences; a more specific part about some specific aspects of the tutorial: presentation, intuitiveness, flexibility, content, text, appropriate length, animation, sound, pleasantness (lo-item scale); and possible suggestions. The questionnaire had 20 questions (two open questions and 18 closed questions). Closed questions had four forms of rating associated with them: four binary questions (yes, no), six ternary questions (Yes, no, I don’t know), seven questions belonging to a S-point Likert scale, and a more complex lo-item scale with 5-point Likert scale (1: strongly disagree, 2: disagree; . . .; 5: strongly agree).

The execution of the questionnaire was divided into two parts: the first to be passed before running the tutorial, and the second, more extensive, after the tutorial was finished.

In the first questionnaire part, four questions were included to capture the user’s opinion about general multimedia topics and the role of the lecturer in traditional teaching.

In the second part, with 16 questions left, some topics were re-evaluated.

2.5.2.3. Statistical analysis. Standard descriptive statistics were used to asses the questionnaire characteristics. Selected responses to the questionnaire were subjected to recommended tests of reliability and validity. Internal consistency or reliability is the extent to which items within a scale are correlated with each other. It can be examined by several methods: Cronbach’s alpha is a widely used method. The validity of a measure is conceptually difficult to prove without a standard. One method is to examine construct validity. We assessed construct validity by the principal components model of factor analysis based on the correlation matrix and using the varimax computer algorithm for orthogonal rotation 1141.

Correlation and linear regression analysis were used to

investigate the association between variables of interest. Finally, in order to identify opinion groups, a cluster analysis with the k-means hierarchical method was applied [15,16].

Statistical analysis of the results from the survey was carried out using the SPSS software version 6.2.1.

3. Results

In this section the results obtained for the two different tests are displayed: the survey (subjective assessment) and the internal test (objective assessment).

3.1. Student evaluates the prototype

The questionnaire data collected from the participants in this study were used for the analysis reported below. Thirty- three variables were created from the questionnaire answers. The statistical study consisted on the following phases: (1) descriptive statistics: means, standard deviations, and frequencies; (2) scale validation: reliability analysis and factor analysis; (3) association between variables; (4) cluster analysis.

3.1.1. Descriptive statistics From the set of 33 variables, the results obtained for the

following subset should be noted (key and description):

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

CALENSE: evaluation of traditional teaching before running the tutorial. TRADICIO: evaluation of traditional teaching after running the tutorial. DIDACTICA: didactic value of the tutorials. FLEXITUT: flexibility of the tutorials. GUSTATUT: interest in the tutorial as an introduction to a practical class. IMPORTAN: importance of the lecturer. OPITUTOR: global opinion about the tutorial. EGUSTAPR: opinion about its presentation. EINTUIT: opinion about its intuitiveness. EFLEXI: opinion about its flexibility. EEQUILI: opinion about the balance between content and presentation. ECARGADO: opinion about amount of text. EANIMA: opinion about the animations. ESONIDO: opinion about the effect of sound in the learning process. ECONTENI: opinion about the content level. EDURAC: opinion about the duration/content ratio. EAMENO: opinion about the pleasantness of the tutorial.

From five options (very bad, bad, fair, good, very good), a large majority of students considered traditional teaching to be fair (64.1%) or good (28.2%) before trying the multimedia tool. Their opinion was almost the same after running the tutorial (TRADICIO: fair, 65.8%; good, 26.3%). On the

R. Marin er aUDisplays I7 (I 997) 167- 178 175

Tutorial presentation

EGUSTAPR

Fig. 8. Opinion about the tutorial presentation.

contrary, most of the students (DIDACTICA: yes, 89.7%) considered tutorials as a didactic way to learn. This is con- sistent with a similar percentage (GUSTATUT: yes, 86.8%) supporting the use of the tutorial as an introduction to a practical class. On the other hand, most of students considered the role of the lecturer very important in the learning process (IMPORTAN: important, 5 1.3%; very important, 38.5%).

When students were asked to give their global opinion about the tutorial, a good opinion was expressed (OPITUTOR: very bad, 0%; bad, 5%; neutral, 15%; good, 61.5%; very good, 17.9%). Concerning the particular aspects of the tutorial (lo-item scale), favourable answers could be noted for almost all items. The most highly evaluated aspects of the tutorial were pleasantness (Fig. 7), presentation (Fig. 8) and the balance between content and presentation (Fig. 9) Nevertheless, the amount of text (ECARGADO) was criti- cized, as we shall see in more detail below.

3.1.2. Scale validation

3.1.2.1. Reliability analysis (internal consistency). In Table 1 we can see the mean and standard deviation for the variables included in the IO-item scale. It is observed that these means are high for all variables. Considering the scale as a whole the average score is 36.50 (SD 3.98) over 50. Cronbach’s alpha coefficient is 0.6952, indicating an

Forrnkontenta balance

q 5. strongly agree

q 1. strongly disagree

EEQULI

Fig. 9. Opinion about the balance between form and content of the tutorial.

acceptable consistency for the scale. Taking into account this result a new variable was created (ESCALADI) as an accurate measure of the evaluation of the tutorial by the users. This variable is the sum of the values for the 10 items. Higher values indicate a better opinion.

3.1.2.2. Factorial analysis. A principal components analysis was performed in order to identify different aspects from the data. Factors with eigenvalues higher than 1 were considered. Four factors were extracted that explain up to 70% of the variability. The fist one (eigenvalue 2.93), explains 29.4%; the second one (1.49) explains 14.9%; the third (1.35) explains 13.6%; and the fourth (1.18) explains 11.8%. In Table 2 we present the values punctuations for each factor related to the original variables (after applying a VARIMAX rotation). It is observed that factor 1 is related to EGUSTAPR (0.87), EFLEXI (0.77), and EAMENO (0.61). Thus, this factor

Table I Ten-item scale validation: reliability analysis

Reliability analysis: scale (alpha) Variables Mean SD

1 EUGSTAPR 4.1923 0.6337 2 ETNTUIT 3.8846 0.5883 3 EFLEXI 3.4615 0.9047 4 EEQUTLI 3.9615 0.4455 5 ECARGADG 2.3846 0.6373 6 EANIMA 3.6538 0.8918 7 ESONIDCI 3.3846 1.2354 8 ECONTENT 3.6358 0.6895 9 EDURAC 3.6538 0.8918 10 EAMENO 4.2692 0.6038 No. of cases = 26.0

Cases

26.0 26.0 26.0 26.0 26.0 26.0 26.0 26.0 26.0 26.0

Mean Variance SD No. of variables

Statistics 36.5000 3.9823 for scale Reliability coefficients IO items

15.8600 10

Alpha (Cronbach) = 0,68 I8 Standardized item alpha = 0.6932 - 0.7


Table 2 Factor analysis: rotated factor matrix (VARIMAX)

Rotated factor matrix

Variables Factor 1

EAMENO 0.61617 EANIMA 0.38035 ECARGADO 0.00137 ECONTENT - 0.00962 EDURAC 0.17695 EEQUILI 0.131 18 EFLEXI 0.777 15 EGUSTAPR 0.87175 EINTUIT 0.37137 ESONIDO - 0.00109

Factor 2 Factor 3

0.34099 0.02385 0.42886 - 0.10153

- 0.11206 0.15940 0.43814 0.77737 0.83406 0.02 182

- 0.06971 0.93786 0.08084 0.05977

- 0.05847 0.07357 0.45056 0.26398 0.73026 0.13555

Factor 4

0.10865 0.49392 0.78851 0.24219

- 0.02437 - 0.07792 - 0.06240 - 0.03642 - 0.58825 - 0.15732

can be interpreted as a measurement of the ‘initial impact’ of the tutorial. Factor 2 is related to EDURAC (0.83) and ESONIDO (0.73), so it can be interpreted as the ‘medium impact’. Factor three is related to EEQUILI (0.93) and ECONTENI (0.77) being interpreted as ‘deep impact’. The last one only presents important correlations with ECARGADO (0.78). This variable had the lowest consistency in the scale and has its own dimension. Facts that point the convenience of rewrite the enunciate of the associated question.

3.1.3. Correlation analysis and regression analysis for the study of the correlation between variables

We have applied a correlation analysis with the objective of studying the correlation between the variables that con- form to the scale, obtaining a simple correlation matrix between variables. In this matrix the high correlation among the variables shown in Table 3 stands out, r being the correlation coefficient and p the probability that gives statistical significance.

We can see in that table that users have a good opinion about the intuitiveness of the tutorial; they have also a good opinion about the balance between content and presentation and about the flexibility. We can also appreciate that users who found the content level adequate also found the balance between content and presentation adequate. Finally, those who found the tutorial presentation adequate also found the animation degree adequate.

We have applied a regression analysis, taking the variable that represents the global opinion about the tutorial, OPITUTOR, as dependent variable. In Table 4 we show the independent variables of the regression equation that have higher values in their respective correlation coefficients

Table 3 Association between variables

Correlation coefficients (I-) and statistical sigmticance @) Variables r P

EINTUIT and EEQUILI 0.5613 < 0.0001 ECONTENIDO and EEQUILI 0.5346 < 0.001 EGUSTAPR and EANIMA 0.5743 < 0.0001 EINTUIT and EFLEXI 0.4856 < 0.01

(r), regression slope (b), and with statistical significance (JJ < 0.05). It should be noted that the variable ECARGADO is the only one that gives a negative regression coefficient. This reinforced the results obtained above in the ‘Factorial analysis’ section.

3.1.4. Cluster analysis in order to identiJji opinion groups Finally, with the aim of finding opinion groups in the

tutorial evaluation, a cluster analysis with the k-means hierarchical method has been applied. The variables selected for the cluster analysis were ESCALADI (mean score of the lo-item scale) and OPITUTOR. Three groups of users with statistically significant differences between them could be observed (very satisfied, 65%; satisfied with specific aspects, 12%; and poorly satisfied, (20%). Note that, in general, the degree of satisfaction is high.

3.2. Prototype evaluates the student

Eight variables were created from the internal questionnaire answers. The statistical study consists of the descriptive statistics: means, standard deviations, and frequencies.

From the set of eight variables the results obtained for the following subset should be noted (key and description):

SCORE-TEST: points obtained in the tutorial topic question answering. SCORE-PCS: points obtained in the tutorial program case solving. TMP-TEST: time spent in the tutorial topic question answering.

Table 4 Association between global opinion and concrete features opinion

Linear regression analysis -

Dependent Independent r B P variable variable

OPITUTOR EAMENO 0.38 0.33 0.0181 ECONTENI 0.50 0.55 0.001 EEQUILI 0.58 0.59 0.0002 EINTUIT 0.5 1 0.50 0.0012

R. Marin et aUDisplays I7 (1997) 167-I 78 111

l TMP-PCS: time spent in the tutorial program case solving.

l TMP-TOTAL: total time spent in the whole tutorial.

Note that the mean score obtained in the questions and program solving was lower (about 2.5 over 10) than the predicted points based on the user opinion results. On the other hand, the mean time spent in problem solving was 12 minutes, ranging from 3.5 to 22.2 minutes. Students who spent a shorter time in problem solving obtained a lower qualification. This new technology opens new horizons for university teaching. In this context, there are a number of research groups in our university working on the development of tools and tutorials with specific objectives related to different disciplines.

Generally, results obtained have been unexpectedly poor in relation to the efforts applied to the tutorial contents presentation and development.

4. Discussion

Most participants are not satisfied with traditional teaching methodology, and consider that the use of such tutorials as the ones tested in this research represent a very practical learning approach. The questionnaire designed to obtain students’ evaluation of the tutorial has been direct and rapidly applied to all of them. Questions, in general, have not become ambiguous, and validation tests of the questionnaire have been satisfactory. Therefore, collecting users’ opinions was a good method of evaluation, as other authors have already indicated [ 131.

Evaluation provides useful information that contributes to the development of the tutorial. In our case, users evaluated all the characteristics of the prototype very positively. Global opinion about the tutorial (resulting from the lo-item scale) was good. The most highly valued aspects in the tutorial were its pleasantness, the presentation and the intuitiveness. In addition, the most important associations in the various aspects evaluated have been those regarding the relationship between intuition and balance, and between presentation and animation.

Although the lo-item scale showed good internal consistency, factor analysis revealed the existence of three sub-scales or dimensions in the users’ opinion, so that the students’ general opinion in relation to the tutorial is deter- mined firstly by its presentation and pleasantness, secondly by its duration and sound, and, finally, students take into account such aspects as its content and balance between form and the content itself.

On the other hand, the scores which students obtained in the internal test of the tutorial have been unexpectedly poor. This may be very likely due to the fact that this internal test of the tutorial was not to be considered for each student’s final mark in the course, since we are still at an experimental stage of the tutorial.

5. Conclusions

Different conclusions can be drawn from the results. Introducing tutorials as a complementary tool for supporting computer practical courses has been well considered by the users. They have also given high qualifications to several aspects of the prototype, such as flexibility, presentation, and balance between content and presentation, to mention a few.

The development of multimedia-based tutorials requires new hardware and software resources but this is not a solution by itself because it needs creativity, interdisciplinary skills and clear objectives. Results show that it is worth continuing to work on this line of research.

Using high-level programming tools that permit open programming seems to be a better choice than using authoring systems that are easier to utilize but constrained if further developments are wanted.

The evaluation of the prototype by the group of users resulted in a satisfactory and necessary experience that offers to the authors an important set of data in relation to several aspects of the prototype. In general, users have a good global opinion of the evaluated tutorial.

From the satisfactory results obtained in the tutorial presented, we are now working on a new version of our prototype taking into account the implementation offeeling and question buttons.

The key idea behind our experience is the introduction of tutorials in all practical classes as a complement to the instructor in the near future.

Finally, the indications are that this new technology opens new horizons for university teaching. In this context, there are a number of research groups in our university working on the development of tools and tutorials with specific objectives related to different disciplines.

Acknowledgements

This work has been partially funded by the PIE project: ‘Desarrollo y aplicacion de un entorno informatico basado en Multimedia, para el aprendizaje de conceptos basicos de informitica’ .

References

[l] A. Cabrera, Informatica educativa: La revolution construccionista, Inform. Autom., 28(I) (1995) 24-3 I.

[2] L. Hare1 and S. Papett, Constructionism, Ablex, Norwood, NJ, 1991. [3] B.F. Skinner. The Technology of Teaching, Prentice-Hail, Englewood

Cliffs. NJ, 1968. [4] R.C. Shank, Case based teaching: Four experiences in educational

software design, Technical Report No. 7, Institute for the Learning Sciences, Evanstone, IL, 1991,

[5] R.E. Eberts and J.F. Brock, Computer-based instruction, in M. Helan- der, ed., Handbook of Human-Computer Interaction, North-Holland, Amsterdam, 1988.


[6] J. Vacherand-Revel and C. Bessiere, Playing graphics in the design and use of multimedia courseware, in A.D.N. Edwards and S. Holland, eds., Multimedia Interface Design in Education, NATO ASI Series, Springer-Verlag, Berlin, 1992, pp. 57-82.

[7] M. Jona, B. Bell and L. Bimbaum, Button theory: A taxonomy of student-teacher communication for interface design in computer- based learning environments, Proc. 13th Cognitive Science Conf., Chicago, IL, 1991, pp. 765-769.

[8] R.C. Schank, AI, Multimedia, and Education, IJCAI 93, Chambery, France, 1993, pp. 1667-1672.

[9] B. Schneiderman, M. Alavi, K. Norman and Y. Borkowski, Windows of opportunity in electronic classrooms, Technical Report CS-TR- 3542, University of Maryland, Baltimore, MD, 1995.

[IO] E. Wenger, Artificial Intelligence and Tutoring Systems, Morgan Kaufman, Los Altos, CA, 1987.

[ 111 0. Coltell, P.J. Sam, P. Valero and F. Ester, Tecnicas de Multimedia para el Autoaprendizaje de Conceptos Basicos de Informatica, Actas de ‘I Jomadas de Informatica’, Puerto de la CNZ, Spain, 17-21 July 1995, pp. 693-694.

[ 121 F. Aparicio Perez, Tratamiento Informtitico de Encuestas, Ra-Ma, Madrid, 1995.

[13] J. Preece, Y. Rogers, H. Sharp, D. Benion, S. Holland and T. Carey, Human-Computer Interaction, Addison-Wesley, Reading, MA, 1994.

[14] M. Jambu, Exploratory and Multivariate Data Analysis, Academic Press, San Diego, 1989.

[I51 A. Gifi, Nonlinear Multivariate Analysis. Wiley, Chichester. 1990. [ 161 M.J. Greenacre, Theory and Applications of Correspondence Analy-

sis, Wiley, Chichester, 1989.

Documents

Student-teacher communication directed to computer-based learning environments