A Computer-Assisted Collaborative Approach for …...Take Motorola Corp. for instance, the company budgets 120 billion dollars annually for conducting employee-training programs. American

Journal of Research and Practice in Information Technology, Vol. 41, No. 4, November 2009 319

A Computer-Assisted Collaborative Approach forDeveloping Enterprise e-Training Courses on the InternetPo-Han Wu Department of Engineering Science, National Cheng Kung University, Tainan city 70005, [email protected]

Gwo-Jen Hwang* and Hui-Chun ChuDepartment of Information and Learning Technology, National University of Tainan33, Sec. 2, Shulin St., Tainan city 70005, Taiwan{gjhwang, carolchu}@mail.nutn.edu.tw

Chin-Chung Tsai Graduate Institute of Technological and Vocational Education,National Taiwan University of Science and Technology, Taipei, 106, [email protected]

Yueh-Min HuangDepartment of Engineering Science, National Cheng Kung University, Tainan city 70005, [email protected]

Most e-Training programs of enterprises were mainly developed by senior employees who haveplenty of knowledge and experience; therefore, it becomes a challenging issue to efficiently andeffectively translate the knowledge and experience of the engineers to computerized subjectmaterials, especially for those who are not experienced teachers. In addition, to develop an e-Training course, personal ignorance or incorrect concepts might significantly affect the quality ofthe course if only a single employee is asked to develop the subject materials. To cope with thisproblem, a multi-expert e-Training course design model is proposed, and a computer-assisted e-Training course development system has been implemented accordingly. Moreover, a practicalapplication has shown that the innovative approach not only can improve the quality of the e-Training courses, but also help experienced employees to well organize their domain knowledge.

Keywords: information technology-enhanced learning, computer-supported collaborative work,e-Training, Delphi method

ACM Classification: H.4 (Information Systems Applications), J.7 (Computers in Other Systems)

Manuscript received: 15 April 2008Communicating Editor: Mu-Yen Chen

Copyright© 2009, Australian Computer Society Inc. General permission to republish, but not for profit, all or part of thismaterial is granted, provided that the JRPIT copyright notice is given and that reference is made to the publication, to itsdate of issue, and to the fact that reprinting privileges were granted by permission of the Australian Computer Society Inc.

1. INTRODUCTIONIn recent years, technologies have advance at an amazingly fast pace; therefore, almost all hightechnical knowledge and skills need to be updated or replaced with very high frequency, e.g., theengineering knowledge of the semiconductor and the electronics industries. One of the most

* Corresponding author. Tel: 886-915396558; Fax: 886-6-3017001Email address: [email protected] (G.J. Hwang).

JRPIT 41.4.QXP:Layout 1 13/11/09 9:02 AM Page 319

A Computer-Assisted Collaborative Approach for Developing Enterprise E-Training Courses on the Internet

Journal of Research and Practice in Information Technology, Vol. 41, No. 4, November 2009320

important management strategies of modern enterprises is to keep on improving the knowledge andskills of employees via frequent training and practice. Take Motorola Corp. for instance, thecompany budgets 120 billion dollars annually for conducting employee-training programs.American IDC (International Data Corp.) predicted that the e-Learning market all over the worldwill triple in the next two years, and will share 40% of the entire training market.

The advantages of electronic learning (e-Learning) and electronic training (e-Training) havebeen documented by many researchers (Hwang, 1998, 2002; Adamsa and Ghaly, 2007; Panayiotisand Demetrios, 2007), including the feasibility of anytime and anywhere education, the availabilityof various learning styles, the reduction of education cost, and the reusability of well-constructedand well-managed subject material modules. Most of the e-Training training programs thatenterprises scheduled were designed and planned within their own organizations. Senior andexperienced employees of the organizations usually play the role of training instructors and coursedesigners. That is, the responsibility of sharing experiences, skills and knowledge heavily relies onthose experienced employees.

As most of the employees in an enterprise are not trained for tutoring, it is usually time-consumingfor them to design course structures, training materials and training programs for those up-to-datetechnologies, and hence the development of training courses are often seriously delayed. Moreover,most of these e-Training courses were designed by a single expert (an experienced employee), whomight ignore some important concepts or provide incorrect contents without consultation from otherexperts. To cope with these problems, a computer-assisted collaborative approach is proposed in thispaper. The novel approach employs the Delphi method to assist multiple experts to cooperativelydesign engineering course structures. An application on the development of a LCM (Liquid CrystalDisplay Module) training course is given to depict the superiority of the novel approach.

2. RELEVANT RESEARCHHigh technical knowledge and skills training is not only a frequent activity but also a heavy burdento enterprises owing to the rapid advance of new technologies. As training courses in an enterpriseneed to be replaced or updated frequently, researchers have attempted to find more efficient andeffective ways to develop and manage learning materials, which encourages the development of e-Training courseware. In the past decade, research in information technology-enhanced learning ortraining environments has received increasing attention (Hu et al, 2003; Benbunan-Fich andArbaugh, 2006). For example, a variety of techniques and tools for developing intelligent tutoringsystems have also been proposed, including the use of neural networks technique to model studentbehaviours in the context of intelligent tutoring systems (Harp, Samad, and Villano, 1995), planningmethods, consistency enforcement, objects and structured menu tools to construct intelligentsimulation-based tutors for procedural skills (Rowe and Galvin, 1998), and some adaptivetechnologies for detecting on-line status of students to establish an interactive intelligent tutoringsystem (Hwang, 1998; Giraffa, Mora, and Viccari, 1999). It can be observed that such e-Learning ore-Training systems have been widely applied to schools and industries in the past decades (Ozdemirand Alpaslan, 2000; Hwang, 2002; Adamsa and Ghaly, 2007; Panayiotis and Demetrios, 2007).

The e-Training approach has shown its superiority. For example, the skills and knowledge ofexperienced experts can be retained and transferred to new employees, and then the learners are ableto receive training courses without being limited by space and time. However, the rapid advance ofexpert technologies also reveals several problems of applying it, and one of the most challengingissues is to efficiently design quality learning materials as well as keep the skills and knowledge ofsenior experts up-to-date and to train the new experts (Tseng et al, 2007).




Researchers have indicated that the concept map model could be an effective approach todeveloping e-Training courses (Tseng et al, 2007). In much of the current pedagogical andpsychological literature, “Conception” is defined as common attributes of same category andobjects or events which are giving the same names (Ausubel, 968; Ausubel, Novak, and Hanesian,1978). Novak and Gowin (1984) stated that “meaningful learning involves the assimilation of newconcepts and propositions into existing cognitive structures”. During tutoring, students learn newconcepts and new relationships among previously learned concepts, and this knowledge can berepresented as a concept map (McAleese, 1998).

In the past decades, the concept map has subsequently been used as a tool to increase meaningfullearning in the sciences and other subjects as well as to represent expert knowledge of individualsand teams in education, government and business. (Novak, 1990a, 1990b, 1995, 1998, 2002).Moreover, it has also been employed in the development of various systems or tools to support largevolumes of multimedia materials generated in a variety of contexts, such as knowledge acquisition(Gaines and Shaw, 1992), large-scale project support (Gaines and Norrie, 1994) and diagnosis ofstudent learning problems (Hwang, 2003; Hwang et al, 2003). Some researchers have attempted todevelop a general visual language technology supporting customizable interactive concept maps(Gaines and Shaw, 1993) and semantic networks (Gaines, 1991). Such concept relationships maybe used as stand-alone documents or embedded as interactive pictures in active documents (Gainesand Shaw, 1993); that is, an open architecture, and user interaction where the concept relationshipsmay be programmed to initiate any available operation on the host system. Moreover, each set ofconcept relationships can be linked to other sets for retrieval purposes, and may be used to retrieve,play and edit multimedia materials via remote access.

Representing knowledge as concept relationships is helpful to people in restructuring priorknowledge as well as organizing new experiences; that is, it is an effective way to representknowledge and learning process (Okebukola, 1984; Roth and Roychoudhury, 1992, 1993, 1994).Salisbury (1998) indicated that learning information, including facts, names, labels, or pairedassociations, is often a prerequisite to efficiently performing a more complex, higher level skill(Salisbury, 1998). Such a relationship has been applied to the development of a special concept mapmodel, called the concept effect model (Hwang, 2003). Consider two concepts or skills, Ci and Cj.If Ci is a prerequisite for the efficient performance of the more complex and higher-level conceptCj, then a concept-effect relationship Ci → Cj is said to exist. Notably, a concept may have multipleprerequisite concepts, and a given concept can also be a prerequisite concept of multiple concepts.For example, to learn the concept “multiplication”, one might first need to learn “addition”, whilelearning “division” might require first learning “multiplication” and “subtraction” (Hwang et al,2008). Such relationships not only can precisely represent the learning sequence of concepts, butalso can be easily transferred into web-based structures. In the following sections, a cooperativeenvironment for developing quality e-Training courses based on the concept-effective model ispresented. To assist multiple experts to cooperatively design e-Training course structure, the Delphimethod is employed. The method was originally developed in the early 1950s by RANDCorporation, in Santa Monica, California. It is conducted by rounds interspersed with group opinionand information feedback in the form of relevant statistical data (Murry and Hammons, 1995). Inthe first round, a questionnaire is created, sent to panel members to complete and return, and theresponses are analyzed. In the next round, a new questionnaire is developed from the previousresponses, and then sent to the panel members again. The aim of the series of questionnaires is toachieve a consensus of opinion by allowing members to re-consider and re-rate their opinionsregarding the items in the questionnaire.




3. COOPERATIVE APPROACH FOR E-TRAINING COURSE DESIGNOne of the application domains gaining increasing usage and visibility is computer-supportedcooperative work (CSCW). The goal of CSCW is to provide systems that allow users to effectivelycollaborate on common goals using networked computer systems (Hwang, 2002). Collaborators canexchange messages, share data, and collaborate even when they cannot physically meet. Manyissues concerning the study and development of CSCW systems, including physical applications,data management, system architecture, group awareness mechanisms etc., have attracted theattention of researchers from various fields (Kosoresow and Kaiser, 1998; Prakash, 1999;Fernandez-Breis and Martinez-Bejar, 2002). Moreover, researchers indicated that people workingcollaboratively must establish and maintain awareness of one another’s intentions, actions andresults; therefore, to better support collaborative activity, strategies for design, notification, orfeedback systems are necessary (Carroll et al, 2003).

In this study, a Delphi-based approach is proposed to assist multiple experts to cooperativelydesign enterprise e-Training courses on the Internet, as shown in Figure 1. In the concept selectionphase, the experts are asked to determine the concepts to be considered in the course, based on theDelphi method. In the second phase, the Delphi approach is employed to assist the experts todetermine the relationships among the selected concepts. A concept map is then generated based onthe acquired relationships, which are then transferred into the linking structure of web pages.

Delphi has been defined by Delbecq, Van de Ven, and Gustafson (1975) as “a method forsystematic solicitation and collection for judgments on a particular topic through a set of carefullydesigned sequential questionnaires interspersed with summarized information and feedback ofopinions derived from earlier responses”(Delbecq et al, 1975, p.10). It has been considered areliable qualitative research method with potential for use in problem solving, decision making, andgroup consensus, reaching in a wide variety of areas (Cochran, 1983). A graphical representation ofthe method is shown at Figure 2. Delphi is generally characterized by three important features(Murry and Hammons, 1995):

1. Anonymous group interaction and responses.2. Multiple iteration or rounds of questionnaires or other means of data collection with researcher-

controlled statistical group responses and feedback.3. Presentation of statistical group responses.

Figure 1: The Structure of the Delphi-based course design approach




Once a working problem is defined, the Delphi procedure begins with identifying and selecting thedomain experts who will participate in the Delphi panel. When a pre-determined number of expertsagree to participate, the researcher uses multiple iterations or rounds of questionnaires to collect data.

The first-round questionnaire uses an open-ended format to elicit individual judgments oropinions from each member of the panel about the particular issue or problem under study. Inessence, round one is an anonymous brainstorming session. After all of the round-onequestionnaires are returned, the researcher reviews, edits, and compiles the panel’s responses, thenprepares the round two questionnaires. In the second-round questionnaire, the researcher requeststhe panel experts to consider, to rank and/or rate, to edit, and to comment upon the responsesdeveloped during round one. Typically the ranking or rating (or both) is represented by a Likertscale. When all the round two questionnaires are returned, the researcher tabulates results, and thencalculates frequency distributions, means, and standard deviations for each questionnaire item.During the third and any subsequent questionnaire rounds, the panel experts are given feedbackabout the previous round. The given information includes panel comments, the composite results,and individual rankings or ratings (or both) for each questionnaire item. Panel members are againasked to rank and/or rate, edit, and comment upon each item. The goal of the third round and anyother subsequent round of questionnaires is to achieve consensus or stability of panel memberresponses. Once consensus or stability (or both, depending on individual case) is gained, the Delphiprocedure is completed (Murry and Hammons, 1995).

The typical first-round questionnaire uses an open-ended format to elicit individual judgments oropinions from each member of the panel about the particular issue or problem under study. In this phase,each expert is asked to provide the names of elements to be discussed in this subject. In this study, theelements represent the concepts need to be taken into account in the e-Training course. The Delphiquestionnaire is shown in Table 1, where E1, E2…E16 are items to be discussed. Table 2 shows theanalysis results of the ratings given by seven experts for the e-Training Course Design in the first round.

Figure 2: Flow Chart of Delphi




To analyze the consistency and the stability of the ratings given by the experts, a second-roundquestionnaire is conducted after the summary of the first-round rating is shown to each expert. Inthe second-round questionnaire, the panel of experts are requested to consider, to rank and/or rate,to edit, and to comment upon the responses developed during the first round. When all the secondround’s questionnaires are returned, the results are tabulated, and then the frequency distributions,means, and standard deviations for each questionnaire item is calculated. Table 3 shows thesummary of second-round ratings given by the experts.

Degree of relevance to Highly Inclined to be More or less Inclined to be Highly the concepts in the inclined to be irrelevant relevant relevant inclined to be

e-training LCM course irrelevant relevantdomain

E1E2

•

•

E16

Table 1: Illustrative Example of Delphi questionnaire for the union element set

Degree of relevance to Rating Quartile Ratingthe concepts in the Ratings given by each expert Median deviation Mean

e-training LCM course (Md) (Q)domain

E1 5 5 5 5 4 5 5 5 0 4.86

E2 4 4 3 4 5 5 3 4 1 4.00

• • • • • • • • • • •

• • • • • • • • • • •

E16 2 2 3 3 3 2 1 2 0.5 2.29

Table 2: Summary of the first-round ratings from seven experts

Degree of relevance to Rating Quartile Ratingthe concepts in the Ratings given by each expert Median deviation Mean

e-training LCM course (Md) (Q)domain

E1 5 4 5 5 4 5 5 5 0.5 4.71

E2 3 4 4 4 5 5 4 4 0.5 4.14

• • • • • • • • • • •

• • • • • • • • • • •

E16 2 2 2 3 3 2 1 2 0.5 2.14

Table 3: Summary of the second-round ratings from seven experts




To calculate the mean and standard deviation, the four-point rating scales are converted tonumeric values. If an expert has failed to respond to certain criterion statements, the correspondingdata will not be included in the computation of means or standard deviations. Usually, the threshold(minimum requirement) of having agreement on any particular item at the second round or later isset to 75 percent. The stability or convergence is defined as few or no further shifting of panelresponses from round to round (Merry and Hammons, 1995). Based on this principle, the rules foremploying the Delphi technology are given in Table 4, where Rating_Mean(qi) represents the meanof the ratings for questionnaire item qi, Rating_Variant(qi) represents the ratio of experts whochange their ratings for qi and Q is the quartile range.

The Delphi questionnaire is ended if one of the following situations occurs:

1. All of the questionnaire items are either accepted or rejected.2. Some undetermined questionnaire items still exist; nevertheless, over 75% of the questionnaire

items have their Rating Variant values being less than 15%.

Round t for Delphi questionnaire Round t+1 for Delphi questionnaire Round t+2 for Delphi questionnaire

IF Rating_Mean(qi) ≥ 3.5 and Q ≤ 0.5

Rating_Mean(qi) ≥ 3.5 and Rating_Variant(qi)<15%Then qi is accepted, and no further discussion concerning qi is needed.

If Rating_Mean(qi) ≥ 3.5 and Q ≤ 0.5

Rating_Mean(qi) <3.5 Rating_Mean(qi) ≥ 3.5 or and Rating_Variant(qi) ≤15%Rating_Variant(qi) >15% Then qi is accepted, and no further

discussion concerning qi is needed.

IF Rating_Mean(qi) <3.5 and Q ≤ 0.5and Rating_Variant(qi)≤15%Rating_Mean(qi) <3.5

Then qi is rejected, and no furtherdiscussion concerning qi is needed.

Table 4: Rules for analyzing the ratings from multiple experts with Delphi approach

Degree of relevance to Rating Quartile Ratingthe concepts in the Questionnaire Ratings given by Median deviation Mean

e-training LCM course Round each expert (Md) (Q)domain

E1Round 1 5 5 5 5 4 5 5 4.86 0

14.29%Round 2 5 4 5 5 4 5 5 4.71 0.5

E2Round 1 4 4 3 4 5 5 3 4 1

42.86%Round 2 3 4 4 4 5 5 4 4.1 0.5

• • • • • • • • • • •

• • • • • • • • • • •

E16Round 1 2 2 3 3 3 2 1 2.29 0.5

14.29%Round 2 2 2 2 3 3 2 1 2.14 0.5

Table 5: Analysis of the ratings in the two rounds of ratings given by the seven experts




Table 5 shows the analysis results of the ratings given by the seven experts for the two roundsof questionnaires. As the rating means of E16, E5 and E3 are all less than 3.5, and their Rating Variantvalues are all less than or equal to 15%, and hence questionnaire items E16, E5 and E3 concepts areremoved; that is, those elements (concepts) are considered to be irrelevant to the course domain. Asthe rating mean of E6 concept is less then 3.5, and their Rating Variant value is greater than or equalto 15%, and hence a third-round questionnaire is also needed.

For E2 the rating mean is grater than 3.5, and the Rating Variant value is greater than 15%, andhence a third-round questionnaire is needed. Moreover, for E1 concept, the rating mean in the firstround is less than 3.5 and that in the second round is greater than 3.5, and hence a third-roundquestionnaire is also needed. The Delphi questionnaire is repeatedly conducted until no furtherround of questionnaire is needed.

4. SYSTEM DEVELOPMENTBased on our novel approach, a knowledge acquisition system, MEDET (Multi-Expert Develop -ment for e-Training), has been implemented on Windows XP Professional. Figure 3 shows theDelphi questionnaire of the multi-expert development for e-Training, which consists of an anony -mous brainstorming session and ranking and/or rate, to edit, and to comment upon the responses.

Figure 4 shows that the MEDET system translates the derived concept effect relationships intoa concept relation diagram, which assists experts to understand course architecture, and follow-upcontext to edit in the course. As the number of concepts for a course might be large, MEDETprovides the zoom-in and zoom-out functions, which not only allow the experts to review everydetail of the concept relationships, but also allow them to understand the whole course structure.

Figure 3: MEDET interface for Delphi Questionnaire




With the assistance of the graphical course structure, the experts can then upload or edit the coursecontent in a logical sequence.

Figure 5 shows the first step of editing the course content, that is, enter the title and descriptionof the course and the name of the author. As shown in the example, the title of the course is “LCMdesign theorem and TFT LCD process”.

The second step is to review the concepts in the course. Figure 6 shows the concepts that havebeen defined by applying the Delphi approach for the course titled “LCM design theorem and TFTLCD process”.

The third step is to review the relationships among the concepts. As shown in Figure 7, theprerequisite relationships among the concepts that are relevant to “LCM design theorem and TFTLCD process” are depicted.

Based on the course structure, the author is able to provide the subject content for each concept.The subject materials can be imported from other web page development tools, such as Front Pageand Dream Weaver, or developed by using the MEDET editor. The subject contents are presentedin HTML format. Figure 8 shows an illustrative example of editing subject contents.

After completing the subject contents, the final learning materials can be saved and browsed, asshown in Figure 9.

Figure 4: MEDET interface for depicting the analysis results of concept relation




Figure 5: First step – Enter title, description and author name of the course

Figure 6: Second step – Review the existing concepts of the course




Figure 7: Third step – Review the prerequisite relationships among concepts

Figure 8: Fourth step – Edit subject contents




5. APPLICATION AND EVALUATIONLCM (Liquid Crystal Display Module) is a LCD Module process, including OLB Bonding, COGBonding, PCB Bonding, Assembly, Aging, C-inspection and D-inspection. The main function ofOLB/COG/PCB Bonding process is bonding the TAB/COF/COG IC and PCBA to the panel forsingle input and transmission. The assembly process consists of combining the panel with metalparts and plastic parts to be a module. The aging process involves making a high temperatureoperation test for the assembly module to sieve out any abnormal products early in the process. C-inspection and D-inspection are the final testing for function and cosmetic characteristic to preventany abnormal product from being shipped to the customer.

The courses for freshman training include the basic training (1 month) and practical training (2months). The evaluation of training result is based on homework and an examination after thetraining, and the final estimate is based on the freshman report and the project report.

5.1 Delphi Approach for Concept SelectionThe main purpose of this phase is to confirm the concepts to be included in the course. To analyzethe consistency and the stability of the ratings given by the twelve experts, a summary of the first-round rating is given to each expert. In the second-round questionnaire, MEDET requests the panelexperts to consider, to rank and/or rate, to edit, and to comment upon the responses developedduring the first round. Table 6 shows the analysis results of the ratings given by the experts in thefirst round.

Figure 9: Web page of the course content




To analyze the stability of the ratings given by the experts, a second round questionnaire isconducted. After completing the second round questionnaire, the quartile deviation (smaller orequal to 0.5) in statistics is used to decide whether the team’s opinions have reached an identicalpoint or not, as shown in Table 7. After several iterations, the concepts to be included in the courseare then determined if the ending condition is reached.

Course ConceptRating given by each expert

M MD SD Q1 2 3 4 5 6 7 8 9 10 11 12

1. TFT LCD industrial structure 3 5 3 5 4 4 5 3 4 3 4 5 4.06 4 .34 1*

2. TFT LCD theorem 4 5 3 5 4 4 5 4 4 3 4 5 4.25 4 .28 .5

3. TFT LCD structure 2 5 5 5 4 4 5 4 4 4 4 5 4.31 4.5 .35 .5

4. TFT LCD process introduction 3 5 5 5 4 4 5 4 5 4 5 4.53 5 .19 .5

5. TFT-LCD application 4 5 2 5 3 4 4 3 3 4 3.78 4 .37 .5

6. TFT-LCD normal defect introduce(TFT LCD process include CF/TFT/Array process) 3 5 2 4 4 3 4 5 3 5 4 5 4 4 .375 1*

7. LCM department introduction 2 5 3 5 3 5 5 5 5 5 4 5 4.13 4.5 .5 1*

8. LCM production flow 5 5 5 5 5 5 5 5 5 5 5 5 4.75 5 .28 0

9. LCM noun brief introduction 4 5 4 5 5 5 5 5 5 5 5 5 4.69 5 .17 .5

10. LCM material characteristic 4 5 3 5 5 5 5 5 5 5 5 5 4.62 5 .24 .5

11. LCM equipment movements 3 5 4 4 4 5 5 5 5 5 5 5 4.4 5 .3 .5

12. LCM design theorem 3 5 3 4 4 5 5 5 5 5 4 4 4.2 4 .32 1*

13. Inspection theorem 4 5 4 4 4 5 5 5 5 5 3 5 4.31 4.5 .29 .5

14. LCM Process: OLB Bonding Process 5 5 4 5 5 5 5 5 5 4 4 5 4.63 5 .12 .5

15. LCM Process: COG Bonding Process 5 5 4 5 5 5 5 5 5 5 4 5 4.69 5 .11 .5

16. LCM Process: PCBA Bonding Process 5 5 4 5 5 5 5 5 5 5 4 5 4.69 5 .11 .5

17. LCM Process: PCBI (Inspection) Process 5 5 4 5 5 5 5 5 5 4 4 5 4.56 5 .19 .5

18. LCM Process: Dispenser Process 5 5 4 5 5 5 5 5 5 4 4 5 4.56 5 .19 .5

19. LCM is made to Cheng: Backlight Assembly is made to Cheng 3 5 4 5 5 5 5 5 5 5 4 4 4.5 5 .19 .5

20. LCM Process: Assembly Process 5 5 4 5 5 5 5 5 5 4 4 5 4.63 5 .12 .5

.

.

.

51. ESD protection introduction 3 5 5 4 5 4 4 5 4 5 4 5 4.25 4 .22 .5

Note: * indicates the average value is smaller than 3.5, or the quartile deviation is smaller than 0.5.

Table 6: Summary of the ratings in the first-round for the LCM course




5.2 Delphi Approach for Concept Relationship ConstructionThe objective in this phase is to acquire the relations among the concepts from multiple experts. Theresearch team categorizes the relations among the manufacturing process of an LCM training coursebased on the concepts collected in the first phase, and as the properties of a Delphi questionnaire inthe second phase.

First round questionnaire in the second part is produced by a non-structural questionnaire. Basedon appropriate modification of the opinions that have been collected and categorized from theexperts, first questionnaire properties are produced. The degree of identification is shown in Table 8.Originally, there are 386 candidate relationships.

After several iterations, the Delphi research team has reached common consistency when all ofthe quartile deviations are smaller or equal to 0.5, and then the second part questionnaire processcan be stopped. It represents that the research team has completed the collection of the concepts and

QuestionnaireRating given by each expert

Stability1 2 3 4 5 6 7 8 9 10 11 12 13

Q1 0 0 0 0 0 1 0 0 1 0 0 0 0 0.1538462*

Q2 0 0 1 0 0 1 0 0 1 1 0 0 0 0.3076923*

Q3 0 0 1 0 0 1 0 0 1 1 0 0 0 0.3076923*

Q4 0 0 1 0 0 1 0 0 1 0 0 0 1 0.3076923*

Q5 0 0 0 0 0 0 0 0 1 1 1 0 0 0.2307692*

Q6 0 0 0 0 1 1 1 0 0 1 0 0 0 0.3076923*

Q7 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Q8 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q9 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q10 0 0 1 0 0 0 0 0 1 0 0 0 0 0.1538462*

Q11 0 0 1 0 0 1 0 0 0 0 0 0 0 0.1538462*

Q12 0 0 0 0 0 1 0 0 0 1 0 0 0 0.1538462*

Q13 0 0 0 0 1 1 0 0 0 1 0 0 0 0.2307692*

Q14 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q15 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q16 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q17 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q18 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

Q19 0 0 0 0 0 1 0 0 0 0 0 1 0 0.1538462*

Q20 0 0 0 0 0 1 0 0 0 0 0 0 0 7.692308E-02

.

.

.

Total stability 0.627451**

Note: * indicates the change is smaller than 0.15; ** indicates the stability is greater than 70%.

Table 7: Analysis of the rating stability for the first and the second round questionnaires




Concept Relation Rating given by each expert

M MD SD Q1 2 3 4 5 6 7 8 9 10 11 12

• TFT LCD industrial structure studies and influence the following concepts ?

TFT-LCD application 4 2 2 3 3 4 2 3 3 3 3 4 3* 3 .25 .5

• TFT LCD theorem studies and influence the following concepts?

TFT LCD structure 4 3 3 3 3 3 4 4 4 3 3 3 3.33* 3 .11 .5

TFT-LCD application 4 3 3 3 3 3 4 2 4 3 3 2 3.08* 3 .2 0

TFT-LCD normal defect introduce(TFT LCD process include CF/TFT/Array process) 4 3 3 3 3 3 4 3 4 3 3 3 3.25* 3 .09 0

LCM Process: RW Process 3 3 3 3 3 3 4 3 3 3 3 3 3.08* 3 .03 0

LCM Defect judgment 4 3 3 3 3 3 4 4 3 3 4 3 3.33* 3 .11 0

LCM abnormal analysis 4 3 3 3 3 3 4 4 4 3 3 3 3.33* 3 .11 .5

• TFT LCD structure studies and influence the following concepts?

TFT LCD theorem 4 3 3 3 3 3 4 4 4 3 3 3 3.33* 3 .11 .5


TFT-LCD normal defect introduce(TFT LCD process include CF/TFT/Array process) 4 3 3 3 3 3 4 4 4 3 3 3 3.33* 3 .11 .5

LCM Process: RW Process 4 3 3 3 3 3 4 4 3 3 3 3 3.25* 3 .09 0

LCM Defect judgment 4 3 3 3 3 3 4 4 3 3 4 3 3.33* 3 .11 .5

LCM abnormal analysis 4 3 3 3 3 3 4 4 4 3 3 3 3.33* 3 .11 .5

• TFT LCD process introduction studies and influence the following concepts ?

TFT LCD theorem 4 3 2 3 3 3 4 4 4 3 3 3 3.25* 3 .17 .5

TFT LCD structure 4 3 2 3 3 3 4 4 4 3 3 3 3.25* 3 .17 .5


LCM Defect judgment 4 3 3 3 3 5 4 4 4 3 4 4 3.66 4 .19 .5

LCM abnormal analysis 4 3 3 3 3 5 4 4 4 3 3 4 3.58 3.5 .2 .5

• TFT-LCD application studies and influence the following concepts?

TFT LCD industrial structure 3 3 3 3 3 3 3 3 3 3 3 3 3* 3 0 0

TFT LCD theorem 3 2 2 2 2 2 4 3 3 3 2 2 2.5* 2 .2 .5...

LCM abnormal analysis 3 3 4 5 3 3 4 3 4 5 4 5 2.83* 3 .06 0

Note: * indicates the average value is smaller than 3.5, or the quartile deviation is smaller than 0.5.

Table 8: Assessment results of degree of identification in the first rounds of questionnaire




Prerequisite Concepts Concept

TFT LCD process introduction LCM Defect judgment

TFT-LCD normal defect introduce(TFT LCD LCM Defect judgment process include CF/TFT/Array process) LCM abnormal analysis

LCM Process: OLB Bonding Process

LCM Process: COG Bonding Process

LCM Process: PCBA Bonding Process

LCM Process: PCBI (Inspection) Process

LCM Process: Dispenser Process

LCM material characteristic LCM is made to Cheng: Backlight Assembly is made to Cheng

LCM Process: Assembly Process

LCM Process: Aging Process

LCM Process: C- (Inspection ) Process

LCM Process: D- (Inspection ) Process

LCM Process: Packing Process

LCM equipment movements LCM abnormal analysis

Inspection theorem

LCM Process: RW Process

LCM key component theorem: COG

LCM key component theorem:: TCP

LCM key component theorem: COF

LCM key component theorem: PCBA

LCM design theorem LCM key component theorem: ACF

LCM key component theorem: Backlight

LCM key component theorem: polarizer

LCM key component theorem: plastic parts

LCM key component theorem: metal parts

LCM Defect judgment

LCM abnormal analysis

Inspection theoremLCM Defect judgment


LCM Process: OLB Bonding Process LCM Defect judgment

LCM Process: PCBA Bonding Process LCM Defect judgment

LCM is made to Cheng: Backlight Assemblyis made to Cheng

LCM Defect judgment

LCM Process: Assembly Process LCM Defect judgment

LCM Process: RW Process LCM Defect judgment

LCM key component theorem: COG LCM abnormal analysis




the relevant relationships. For the LCM course, 92 prerequisite relationships that represent theframework of the web course are obtained, as shown in Table 9.

5.3 Questionnaire and AnalysisTo evaluate the effectiveness of the novel approach, a questionnaire is designed to collect theopinions from the experts. Table 10 shows the results of the questionnaire.

From Table 10, the experts believed that the integrity and correctness of the developed coursecan be achieved efficiently by applying the novel approach. Moreover, all of the experts believe that

Table 9: Concept relationships of the LCM course

LCM key component theorem: TCP LCM Defect judgment

LLCM key component theorem: COF LCM Defect judgment

LCM key component theorem: PCBA LCM Defect judgment

LCM key component theorem: ACF LCM Defect judgment

LCM Process: Assembly Process

LCM key component theorem: BacklightLCM Process: RW Process

LCM Defect judgment


. .

. .

. .

LCM design theorem

ESD protection introductionLCM key component theorem: PCBA

LCM key component theorem: ACF

LCM key component theorem: Backlight

Rating Extremely Disagree No Agree ExtremelyQuestion disagree opinion Agree

The novel approach can increase the integrity of the e-Training courses. 0 0 8% 72% 20%

The novel approach can increase the correctness of the e-Training courses. 0 0 8% 56% 36%

The novel approach can increase the efficiency in developing the e-Training courses. 0 0 16% 48% 36%

The feedbacks of questionnaire survey provide me good references to follow up the coming questionnaires. 0 0 0 65% 35%

The feedbacks of questionnaire survey make me more understand the application domain. 0 0 0 40% 60%

I would like to participate in such course design tasks in the future. 0 0 0 80% 20%

Table 10: The results of the questionnaire survey by experts




the feedback of the questionnaire survey is helpful to them in making decisions as well asunderstanding the application domain.

It is surprising to find that 100% of the experts would like to participate in the course designtasks in the future, which is a very different result from most of the past experiences reported forusing the Delphi approach (Martino, 1994; Murry and Hammons, 1995). After interviewing severalexperts, we have found that their willingness to participate in future course design tasks is due tothe benefits of obtaining more domain knowledge; that is, those experts believe that the coursedesign process with Delphi approach is helpful to them in obtaining more structural knowledge ofthe application domain.

Furthermore, an investigation has been conducted to evaluate the quality of the developedcourseware. Thirty-two junior engineers participated in the experiment. Initially, the systemdeveloper gave the participants a brief orientation about the purpose of the experiment and theoperations of the system. The participants were asked to fill in a questionnaire after using the system.The choice of each question is based on a scale of 1 to 5. Table 11 depicts the statistical results. Itcan be seen that the participants had a positive attitude toward the item “The subject materials areeasy to understand” (Mean = 4.25), indicating that the subject contents were understandable by mostof the participants. When considering the item “The courseware is well organized”, 94% of theparticipants showed positive inclination (Mean = 4.63), which shows the effectiveness of theinnovative approach in assisting the domain experts to define the relationships among the concepts.

In addition, based on the statistical results of “The courseware contains the required concepts”(Mean = 4.69) as well as “the courseware is helpful” (Mean = 4.72), we conclude that the innovativeapproach is able to help the domain experts in selecting the appropriate concepts for the subject unit,such that the participants believed that the courseware was helpful to them in improving theirlearning performance. Finally, both the question items “Are you willing to be trained in such a wayfor other courses” and “Will you recommend this courseware to other engineers?” have a high meanof 4.81, which implies that the courseware was well approved by the participants. To sum up, weconclude that the innovative approach is successful.

Negative Inclined Average Inclined Positiveto Negative to Positive

Question items 1 2 3 4 5 Mean

1. Are the subject materials easy to 0 2 3 12 15 4.25 understand? (0%) (6%) (9%) (38%) (47%)

2. Is the courseware well 0 0 2 8 22 4.63organized? (0%) (0%) (6%) (25%) (69%)

3. Does the courseware contain the 0 0 2 6 24 4.69required concepts? (0%) (0%) (6%) (19%) (75%)

4. Is the courseware helpful to you? 0 0 2 5 25 4.72(0%) (0%) (6%) (16%) (78%)

5. Are you willing to be trained in 0 0 2 2 28 4.81such a way for other courses? (0%) (0%) (6%) (6%) (88%)

6. Will you recommend this 0 0 2 2 28 4.81courseware to other engineers? (0%) (0%) (6%) (6%) (88%)

Table 11: Statistical results of the question items




6. CONCLUSIONNowadays, the notation of business management has changed from the emphasis of lower cost andhigh efficiency to the achievement of intellectual properties and innovation. Thus, how to maximizethe benefits of intellectual properties via the assistance of new technologies has become animportant and challenging issue. The development of e-Training courses is one of the effective waysto preserve and promote the intellectual properties of an enterprise. In this paper, we propose aMultiple-Expert approach to cooperatively developing e-Training courses. In the innovativeapproach, the Delphi method is employed to elicit and integrate the course design knowledge frommultiple experts. An e-Training course design system, MEDET, has been implemented accordingly.Table 12 compares several features of traditional course design systems and MEDET, from whichit can be seen that MEDET provides several advantages in developing enterprise e-Training courses.

With this innovative approach, the e-Training course is designed by the cooperation of multipleexperts, which implies that more detailed and more accurate course structures can be derived. Inaddition, owing to the summarization and feedback feature of the Delphi approach, the experts havethe opportunity to obtain various opinions from others, which are helpful to them in reviewing andreorganizing their knowledge. Furthermore, we also found that our approach is helpful to the non-experienced course designer in developing high quality teaching materials. In the near future, weplan to apply the innovative approach to the development of several e-Training courses andcombine the e-Training course design system with a tutoring and assessment system, such that thosee-Training courses can be conducted with the least delay.

ACKNOWLEDGEMENTThis study is supported in part by the National Science Council of the Republic of China undercontract numbers NSC 95-2524-S-024 -002 and NSC 95-2520-S-024 -003.

REFERENCESADAMSA, M. and GHALY, A.E. (2007): Maximizing sustainability of the Costa Rican coffee industry. Journal of Cleaner

Production 15(17): 1716–1729.AUSUBEL, D.P. (1968): Educational Psychology: A Cognitive View. New York, Holt, Rinehart & Winston press.AUSUBEL, D.P., NOVAK, J.D. and HANESIAN, H. (1978): Educational Psychology: A Cognitive View (2nd ed). New York,

Holt, Rinehart & Winston press.BENBUNAN-FICH, R. and ARBAUGH, J.B. (2006): Separating the effects of knowledge construction and group collaboration

in learning outcomes of web-based courses. Information & Management 43: 778–793.COCHRAN, S.W. (1983): The Delphi method: formulation and refining group judgments. Journal of Human Sciences 2(2):

111–17.DELBECQ, A.L., VAN DE VEN, A.H. and GUSTAFSON, D.H. (1975): Group Techniques for Program Planning: A Guide

to Nominal Group and Delphi Processes. Scott-Foresman, Green Briar Press.GAINES, B.R. (1991): An interactive visual language for term subsumption visual languages. IJCAI’91: Proceedings of the

Twelfth International Joint Conference on Artificial Intelligence, 817-823, San Mateo, California, Morgan Kaufmann Press.

Traditional course design system MEDET

Knowledge sources Single expert Multiple expert

Facilitating the cooperation of domain experts No Yes

Integrity of the developed courses Average Higher

Correctness of the developed courses Average Higher

Helpful feedbacks to the experts No Yes

Table 12: Comparisons of traditional course design and MEDET system




GAINES, B.R. and NORRIE, D.H. (1994): Mediator: information and knowledge management for the virtual factory. SIGMANAAAI-94 Workshop: Reasoning about the Shop Floor, 30–39, Menlo Park, California.

GAINES, B.R. and SHAW, M.L G. (1992): Integrated knowledge acquisition architectures. Journal for Intelligent InformationSystems 1(1): 9–34.

GAINES, B.R. and SHAW, M.L.G. (March, 1993): Supporting the creativity cycle through visual languages. AAAI SpringSymposium: AI and Creativity, 155–162, Menlo Park, California..

GIRAFFA, L.M.M., MORA, M.C. and VICCARI, R.M. (September, 1999): Modeling an interactive ITS using a MASapproach: from design to pedagogical evaluation. Third International Conference on Computational Intelligence andMultimedia Applications, New Delhi, India, 153–158.

HARP, S.A., SAMAD, T. and VILLANO, M. (1995): Modeling student knowledge with self-organizing feature maps. IEEETransactions on System, Man, Cybernetics, 25(5): 727–737.

HU, P.J.H., CLARK, T.H.K. and MA, W.W. (2003): Examining technology acceptance by school teachers: a longitudinalstudy. Information & Management 41: 227–241.

HWANG, G.J., HSIAO, J.L. and TSENG, J.C.R. (2003): A computer-assisted approach for diagnosing student learningproblems in science courses. Journal of Information Science and Engineering 19(2): 229–248.

HWANG, G.J. (1998): A tutoring strategy supporting system for distance learning on computer networks. IEEE Transactionson Education, 41(4): 343.

HWANG, G.J. (2002): On the development of a cooperative tutoring environment on computer networks. IEEE Transactionson System, man and Cybernetic, Part C, 32(3): 272–278.

HWANG, G.J. (2003): A concept map model for developing intelligent tutoring systems. Computers & Education 40(3):217–235.

HWANG, G.J., TSENG, J.C.R. and HWANG, G.H. (2008): Diagnosing student learning problems based on historicalassessment records. Innovations in Education and Teaching International 45(1): 77–89.

KOSORESOW, A.P. and KAISER, G.E. (1998): Using agents to enable collaborative work. IEEE Internet Computing, 2(4):85–87.

McALEESE, R. (1998): The knowledge arena as an extension to the concept map: reflections in action. Interactive LearningEnvironments 6(3): 251–272.

MARTINO, J.P. (1994): Technological Forecasting for Decision Making (3rd ed.). New York, McGraw-Hill InternationalEditions.

MURRY, J.W. and HAMMONS, J.O. (1995): Delphi, a versatile methodology for conducting qualitative research. The Reviewof Higher Education 18(4): 423–436.

NOVAK, J.D. (1990a): Concept maps and vee diagrams: Two metacognitive tools to facilitate meaningful learning.Instructional Science 19(1): 29–52.

NOVAK, J.D. (1990b): Concept mapping: A useful tool for science education. Journal of Research in Science Teaching 27(10):937–949.

NOVAK, J.D. (1995): Concept mapping: A strategy for organizing knowledge. In Learning science in the schools: researchreforming practice. GLYNN, S.M. and DUIT, R. (eds). Mahwah, New Jersey, 347-364, Lawrence Erlbaum Associates.

NOVAK, J.D. (1998): Learning, Creating and Using Knowledge: Concept Maps as Facilitative Tools in Schools andCorporations. Mahwah, New Jersey, Lawrence Erlbaum Associates.

NOVAK, J.D. (2002): Meaningful learning: the essential factor for conceptual change in limited or appropriate propositionalhierarchies (LIPHs) leading to empowerment of learners. Science Education 86(4): 548–571.

NOVAK, J.D. and GOWIN, D.B. (1984): Learning How to Learn. Cambridge, New York, Cambridge University Press.OKEBUKOLA, P.A. (1984): In search of a more effective interaction pattern in biology laboratories. Journal of Biological

Education, 18(4): 305–308.OZDEMIR, B. and ALPASLAN, F.N. (2000): An intelligent tutoring system for student guidance in Web-based courses. In

Proceedings of the Fourth International Conference on Knowledge-Based Intelligent Engineering System and AlliedTechnologies, Brighton, UK, 2: 835–839.

PANAYIOTIS, Z. and DEMETRIOS, S. (2007): The ten competence observatory: an enabling technology for commondescription of competences. ICALT 2007L: proceedings of the Seventh IEEE International Conference on AdvancedLearning Technologies, Nigata, Japan, 765–769.

PRAKASH, A., SHIM, H.S. and LEE, J.H. (1999): Data management issues and trade-offs in CSCW systems. IEEETransactions on Knowledge and Data Engineering, 11(1): 213–227.

ROTH, W.M., and ROYCHOUDHURY, A. (1992): The social construction of scientific concepts or the concept map asconscription device and tool for social thinking in high school science. Science Education, 76(5): 531–557.

ROTH, W.M. and ROYCHOUDHURY, A. (1993): The concept map as a tool for the collaborative construction of knowledge:A microanalysis of high school physics students. Journal of Research in Science Teaching 30(5): 503–534.

ROTH, W.M. and ROYCHOUDHURY, A. (1994): Science discourse through collaborative concept mapping: NewPerspectives for the teacher. International Journal of Science Education 16: 437–455.

ROWE, N.C. and GALVIN, T.P. (1998): An authoring system for intelligent procedural-skill tutors. IEEE Intelligent Systems,13(3): 61–69.




SALISBURY, D.F. (1998): Effect drill and practice strategies. In Instructional Designs for Microcomputer Courseware.JONASSEN, D.H. (ed), Hillsdale, New Jersey, 103–124, Lawrence Erlbaum Associates.

FERNANDEZ-BREIS, J. T. and MARTINEZ-BEJAR, R. (2002): A cooperative framework for integrating ontologies.International Journal of Human-Computer Studies 56: 665–720.

TSENG, J.C.R., TSAI, W.L., HWANG, G.J. and WU, P.H. (2007): An efficient and effective approach to developingengineering e-Training courses. Journal of Distance Education Technology 5(1): 31–47.

BIOGRAPHICAL NOTESPo-Han Wu is a PhD student of the Department of Engineering Science at theNational Cheng Kung University of Tainan in Taiwan, Republic of China. Hereceived his Masters degree in the Department of Information and LearningTechnology at National University of Tainan in Taiwan. His research interestsinclude e-Learning, information technology-applied instructions, ubiquitouslearning, cooperative learning.

Gwo-Jen Hwang is currently a professor of the Department of Informationand Learning Technology, as well as Dean of the College of Science andEngineering, at the National University of Tainan in Taiwan, Republic ofChina. Dr. Hwang received his PhD degree from the Department of ComputerScience and Information Engineering at National Chiao Tung University inTaiwan. His research interests include mobile and ubiquitous learning,computer-assisted testing, expert systems and knowledge engineering. Dr.Hwang has published 260 papers, including more than 90 papers inprofessional journals. Owing to his good reputation in academic research, hereceived the 2007 outstanding research award from the National ScienceCouncil of Taiwan.

Hui-Chun Chu is a PhD candidate of the Department of Information andLearning Technology at National University of Tainan in Taiwan, Republic ofChina. She received her Masters degree in Information ManagementDepartment from National Chi-Nan University, Taiwan. Her research interestsinclude mobile and ubiquitous learning, information technology-appliedinstructions, computer-assisted, knowledge engineering and expert systems.

Chin-Chung Tsai received a Master of Education degree from HarvardUniversity and a Master of Science degree from the Teachers College,Columbia University. He completed his doctoral study also at the TeachersCollege, Columbia University, 1996. From 1996 to 2006, he joined the facultyof Institute of Education and Center for Teacher Education at the NationalChiao Tung University, Taiwan. He is currently a Chair Professor at theGraduate School of Technological and Vocational Education, NationalTaiwan University of Science and Technology, Taipei, Taiwan. His researchinterests deal largely with constructivism, epistemological beliefs andInternet-based instruction related to science education. He is currently theAssociate Editor of International Journal of Science and MathematicsEducation (published by Springer). In the last five years, he has publishedmore than forty papers in English-based international journals.

Po-Han Wu

Gwo-Jen Hwang

Hui-Chun Chu

Chin-Chung Tsai




Yueh-Min Huang is a Distinguished Professor and Chairman of theDepartment of Engineering Science, National Cheng-Kung University,Taiwan. His research interests include multimedia communications, wirelessnetworks, artificial intelligence, and e-Learning. He received his MS and PhDdegrees in Electrical Engineering from the University of Arizona in 1988 and1991 respectively. He has published about 200 refereed professional researchpapers. Dr. Huang has received many research awards, such as the Best PaperAward of 2007 IEA/AIE Conference, Best Paper Award of the ComputerSociety of the Republic of China in 2003, the Awards of Acer Long-Term Prizein 1996, 1998, and 1999, Excellent Research Awards of NationalMicrocomputer and Communication Contests in 2006. Dr Huang is on theeditorial board of the Journal of Wireless Communications and MobileComputing, Journal of Security and Communication Networks andInternational Journal of Communication Systems. Huang is a member of theIEEE as well as IEEE Communication, Computer, and Circuits and SystemsSocieties

Yueh-Min Huang


Documents

A Computer-Assisted Collaborative Approach for …...Take Motorola Corp. for instance, the company budgets 120 billion dollars annually for conducting employee-training programs. American