COGNITIVE TASK ANALYSIS OF TOUR MANAGERIGUIDE

PLYING KUCHING-LUBOK ANTUIBATANG A1 TOUR ROUTE

DOUGLAS JAMES

A thesis submitted In partial fulfillment of the requirements for the degree of

Master of Science

Faculty of Cognitive Science and Human Development

UNIVERSITI MALAYSIA SARAWAK

DECEMBER 1999

DECLARATION

I declare herewith that no part of the work referred to in this thesis has been submitted in support of an application for another degree of qualification of this or any other university or institution of higher learning.

This dissertation is dedicated to my grandfather: The late Udin Ak Braoh.

ACKNOWLEDGEMENTS

This research was made possible with the endless support, help and guidance of many. My grateful thanks and appreciation goes to:

Mr. Philip Nuli Anding, my humble and dedicated supervisor, for guiding me with endless effort through all aspects of the study.

Prof. Madya Dr. Muhamed Awang, my lecturer, for the great effort in imparting his expertise in the field of cognitive task analysis and the method of verbal protocol analysis. All lecturers and staff of IAB-Unimas for their professional effort and help in making this course possible and memorable.

The Managing Director, Mr. Philip Yong and Operation Manager, Mr. Emong Tinsang and all staff of the BorneoAdventure Travel Agency Kuching, for enabling me to carry out this study, with special thanks to: Mr. Danis Luart Lang Mr. James Jinyong Mr. Charles Collin Langgie Mr. Robert Samy Pn. Annatasia Lim Mr. Ralp Luku

A wonderful acknowledgement: To my mother, Bata Ak. Udin for her endless support and inspiration; To my wife, Mariana Dom, and our sons, Nobel Awang and Leeroy Shah, for their sacrifice, love and encouragement.

TITLE PAGE

DECLARATION

DEDICATION

ACKNOWLEDGEMENTS

CONTENTS

LIST OF TABLES

LIST OF FIGURES

LIST OF ABBREVIATIONS

ABSTRACT

ABSTRAK

CONTENTS

Page

CHAPTER ONE: THE PROBLEM AND ITS SETTING

1.1 Introduction 1.2 Background of the Problem 1.3 Statement of the Problem 1.4 Significant of the Study 1.5 Purpose of the Study 1.6 Objectives 1.7 Scope and Delimitation of the Study 1.8 Outline of the Study

CHAPTER TWO: LITERATURE REVIEW

2.1 Introduction 2.2 Knowledge 2.3 Information Processing Theory

2.3.1 Sensory Information Storage 2.3.2 Short-term Memory 2.3.3 Long-term Memory

2.4 Cognitive Task Analysis 2.5 Verbal Protocol Analysis 2.6 Protocol-Based Coding Scheme 2.7 Mental Model 2.8 Cognitive Task Analysis of Tour Guide or Tour Manager in Tourism 2.9 Summary

iii

vii

viii

CHAPTER THREE: RESEARCH METHODOLOGY

3.1 Overview of the Chapter 3.2 Operational Definitions 3.3 Research Methodology and Framework 3.4 Subjects 3.5 Data Collection 3.6 Data Processing and Analysis

3.6.1 Analyzing Conceptual Operator for Tour Guide or Tour Manager 3.6.2 Analyzing Knowledge States for Tour Guide or Tour Manager

3.7 Methodological Assumptions 3.8 Limitations 3.9 Summary

CHAPTER FOUR: FINDlNGS O F THE STUDY

4.1 Introduction 4.2 Analyzing the Knowledge States

4.2.1 Knowledge States (a) Hospitality Management in Tourism (HMT) (b) Socio-cultural nnd Art (SCA) O Economy (d) Eco-tourism (e) Education

4.3 Result of Analysis on Knowledge States 4.4 Analyzing the Conceptual Operators

4.4.1 Conceptual Operators (a) Data Explanation (b) Verlflcation O Anticipation (d) Evaluation (e) Recognition (f) Estimation (g) Postulation

4.5 Results of Analysis on Conceptual Operators 4.6 Verbal Protocol Coding Scheme for Tour Guide or Tour Manager 4.7 Simplified Mental Model for Tour Guide or Tour Manager. 4.8 Summary

CHAPTER FIVE: SUMMARY, CONCLUSION AND RECOMMENDATIONS

5.1 Summary of the Study 5.2 Conclusions of the Study 5.3 Recommendations for Further Research

Reference

Appendix A Analysis on Knowledge States of Tour Guide or Tour Manager Appendix B Analysis on Conceptual Operators of Tour Guide or Tour Manager Appendix C Letters of Authority Relating to the Research

LIST O F TABLES

TABLE

2.1 Taxonomy of general and specific task analysis methods (Gordon, 1994)

2.2 Commonly used combinations of task analysis data collection and data representation methods (Gordon,1994)

3.1 Verbal Protocol Transcript Fragment

3.2 Example of Verbal Data for Analyzing Conceptual Operators

3.3 Example of Verbal Data for Analyzing Knowledge States

4.1 Frequency of Knowledge States encountered

4.2 Frequency of Conceptual Operators encountered

Page

vii

LIST O F FIGURES

Figure

2.1

2,2

2.3

3.1

The stages of memory

Basic elements of coding scheme

Detailed description of conceptual operations (Hassebrock and Prietula, 1992)

Conceptual Framework of the Study

Knowledge States of Tour Guide or Tour Manager

Specific Knowledge States of Hospitality Management in Tourism

Knowledge States of Socio-cultural and Art (SCA)

Knowledge States of Economy

Knowledge States of Eco-tourism

Knowledge States of Education

Knowledge states of tour guide or tour manager

Conceptual Operators of Tour Guide or Tour Manager

Verbal Protocol Coding Scheme for Tour Guide or Tour Manager

Verbal Protocol Coding Scheme for Tour Guide or Tour Manager

Simplified Mental Model for Tour Guide or Tour Manager

Page

LIST O F ABBREVIATIONS

CTA

A1

ECO

HMT

SC A

ECT

EDU

ANTHRO

CEO

ACRIC

F & B

K

SK

SSK

SSSK

SSSSK

BM

Ng.

LA

STB

MOCAT

Cognitive Task Analysis

Artificial Intelligent

Economy

Hospitality Management In Tourism

Socio-cultural and Art

Eco-tourism

Education

Anthropology

Geography

Agriculture

Food and Beverage

Knowledge

Specific Knowledge

Sub-specific Knowledge

Sub Sub-specific Knowledge

Sub-Sub-Sub-specific Knowledge

Bahasa Malaysia

Nanga

Lubok Antu

Sarawak Tourism Board

Ministry of Culture and Tourism

ABSTRACT

Cognitive task analysis delineates how experts use learned skills to perform complex task. It specifies the knowledge which skilled performers have encoded into schema and mental models, as well as how they operationalize this knowledge. This study develops a framework adapted from Hassebrock and Prietula (1992) in conducting cognitive task analysis on the verbal protocols report of the tour guide or tour manager while carrying the task of tour guiding. The study incorporates five tour guides or tour managers plying one of the popular tourist routes and destinations in Sarawak. Verbal protocol analysis was done based on two representations, knowledge states and conceptual operators. The results indicates that there are seven conceptual operators being used by the subjects to generate and operationalize the five identified knowledge states while carrying out the tour guiding task. The analysis also indicates that the most frequently used knowledge states in guiding a tour are knowledge of Socio-cultural and Art and Economy. While Explanation and Recognition are the most frequently needed operators in executing the task guiding a tour.

ABSTRAK

Analisa dugas lcognitifmengupas cara mana pakar-pakar menggunakan kemahiran yang diperolehi untuk melaksanakan tugas yang komplek. Analisa tugas kognitif ini turut menjelaskan pengekodan pengetahuan kepada skema dun model mental serta bagaimana pengetahuan ini digerakkan oleh seseorang pelaksana tugas yang berkernahiran. Kajian ini menyingkap kerangka kerja yang disesuaikan daripada kajian Hassebrock dun Prietula (1992), dalam melaksanakan analisa tugas kognitif ke atas laporan protokol lisan pemandu pelancongan atau pengurus pelancongan sewaktu menjalankan tugas membawa pelancong. Lima orang pemandu pelancong terlibat dalam kajian ini melalui laluan dan destinasi pelancongan yang terkenal di Sarawak. Analisa protokol lisan dibuat berdasarkan dua perwakilan, iaitu keadaan pengetahuan dun operasi konsep. I-Iasil kajian ini memperlihatkan bahawa adu tujuh operasi konsep digunakan oleh subjek untuk menjana dan menggerakkan limn keadaan pengetahuan dalam melaksanakan tugas sebagai pemandu pelancongan. Analysis kujian ini juga menunjukkan hahawa Socio- budaya dan Kesenian, dan Ekonomi merupakan keadann pengetuhuan ynng paling kerap digzlnakan dalam melaksanakan tugas memandu pelancongan, sementara Penerangan dan Mengenalpasti adalah dua operator Iconseptual yang sering digunakan dalam menggendalikan tugas memandu pelancongan.

CHAPTER ONE

THE PROBLEM AND ITS SETTING

1.1 Introduction The travel and tourism industry is of major importance for the world economy today. It is one of the largest single employers and in many countries it is the largest services exporting sector, making a significant contribution to the balance of payments. More than 255 million people or 11 . l% of total jobs world-wide are occupied in the tourism industry and according to World Tourism Organization, in 1995 almost 570 million people traveled abroad for tourism reasons and spent 373 billion US dollars, which accounts for 1.5% of the world GNP (Papatheodorou, 1999).

In Malaysia the importance of tourism development started to gain strength in the 1980s at the federal level. By mid 1985, the Sarawak State Ministry of Environment and Tourism was established in line with the Federal policy to accord priority to tourism development. The goal is to make tourism a major revenue earner in the country. Prior to 1985, the state government involvement in the tourism industry is channeled through the State Tourist Development Committee formed in 1976 to assume the planning, coordination and promotion of tourism development. In recognition of the growing importance of the tourism industry in Sarawak's economy and its overall development programs, the state government elevated tourism into a full ministry status and the Ministry of Tourism becomes a reality in 1995 (Rakan Sarawak, 1998). The promotional arm of Ministry of 'Tourism is started with the launching of the Sarawak Tourism Board in the same year to take over the lead in tourism industry and to work closely with the Ministry of Tourism in promoting and developing tourism in Sarawak.

As a result of the hard work and endless effort of the Sarawak Tourism Board (STB) there was an increase of 24% in the number of foreign visitors to Sarawak by 1997, bringing the figure to more than one million people from Asia, Europe, Africa and the United States (Rakan Sarawak, 1998). Nonetheless the Ministry of Tourism is still emphasizing on improving its services, through STB, especially on the professional and the quality of the tours manager or guide. Thus tours manager or guide plays an important role at the grass-root level in the development of tourism as the job of the tour manager involves primarily rendering service and disseminating information of interest to his or her clients.

Basically, the job of the tour manager involves planning and conducting tours. Planning and conducting the tour requires some form of information processing on the part of the tour manager. Tourists expect value-for-money as they demand for better services and tourism related products (Poynter, 1993). This requires knowledge and complex cognitive ability in processing and disseminating information to the tourists. Thus analyzing the concurrent and retrospective verbalization that the tour manager gives at the time he or she plans or conducts the tours or trips will give insight on the thinking processes.

1.2 Background of the Problem Guide or tour managers perform a range of duties. Their roles include conducting and directing all aspects of the tour, tour company representation, and quality control specialist (Poynter, 1993). This is critical and crucial for the Tour Con~pany because the duties and jobs are becoming highly information driven. In Malaysia, all the tourist guides have to pass National Tourist Guide Examination set by the Ministry of Culture, Art and Tourism (MOCAT, 1999) before they could get the license to guide a tour. The skilled performance of the tourist guide or

manager could be explored by employing cognitive task analysis (CTA). This method could be used effectively in the teaching and learning process. Therefore one needs to appreciate and know the proper way of carrying out CTA.

1.3 Statement of the Problem Guides and tour managers are entrusted with a heavy responsibility. He is the key person working for the tour operator (Page and Getz, 1997). His role is to conduct and direct the tour, and be a representative for the tour operator or travel agency. He has to handle himself with professional decorum. The tour guide or tour manager also acts in an arbitrator role. When there is confusion regarding the quality of a service or product to be provided, it is normally expected that the tour manager will make decisions on behalf of the tour operator and arbitrates any differences so that tour members receive the quality of service contracted (Poynter, 1993). Therefore a tour guide requires more than just having the appropriate knowledge and also that this knowledge must be organized properly to enable its optimal retrieval. Thus, cognitive task analysis carried out on the verbal protocol of a tour guide, while performing the task, would enable a better understanding in developing a simple mental models of a tour guide or tour manager.

1.4 Significant of the Study A good tour guide or manager is expected to be able to recall and exhibit a broad and in-depth knowledge about aspects associated with the operation, coordination and conducting of organized tour group trips. This ability of the guide could enable him or her to provide efficient, friendly and satisfactory service to his or her clients. The skill performance of this guide could be reflected in the analyzed concurrent and retrospective verbal protocols.

1.5 Purpose of the Study The tour guide or tour manager arbitrates any differences, manages and coordinates the tour trip right from the beginning to the end of the trip. This study would help to understand how to analyze verbal protocols of a tour guide or tour manager as he or she performs his or her task efficiently and effectively. Verbal protocol coding scheme for the tour guide would be developed while analyzing the verbal protocols. Consequently, this verbal protocol-coding scheme would then be adapted and used to represent a simple mental model of the tour guide or tour manager. Eventually, the knowledge obtained from analyzing the verbal protocols would set as an important platform for further research in area of cognitive task analysis of tour operators.

Foremost, this study strives to analyze verbal protocols, then develop a coding scheme for the verbal protocol of tour guide or tour manager. Eventually, an appropriate mental model is developed to represent how the tour guide processes the available information so that he or she could carry out his or her task properly.

1.6 Objectives Redding (1995) stated that the goal of cognitive task analysis (CTA) is to delineate the mental processes and skills needed to perform a task at high proficiency levels. It can be used also to identify the changes in cognitive structures and process as proficiency develops over time. Therefore the use of verbal protocol analysis as one of the general methods for carrying out CTA, can be used to deduce on tasks, goals, knowledge, conceptual operators, lines of reasoning and strategies used in carrying out unobservable working processes.

Generally this study would explore and develop a simple mental model of a tour guide or tour manager. The analysis of the concurrent and retrospective verbalizations will specifically be used to:

(a) Identify the knowledge used by the tour guide or tour manager in carrying out the tour guiding task.

(b) Identify the conceptual operators used to operationalize this knowledge. (c) Develop a simple verbal protocol coding scheme based upon the tour guide or tour

manager's knowledge and the conceptual operators needed to process those knowledge.

1.7 Scope and Delimitation of the Study This study will only focus on how the tour guide or tour manager executes his or her duty efficiently and effectively. The verbal protocols as concurrent protocols are taken at the moment the tour guide or tour manager carries out the task of rendering service, processing and disseminating information of interest to his or her clients. Thus the assumption is the obtained verbal protocol is the only overall approach the tour guide uses in carrying out the task. If this were true, then the collected data would give a complete representation and a perfect insight on the thinking process of a tour guide or tour manager. However, this is difficult to achieve because not all tasks and actions of the tour guide in executing his task can be recorded verbally. Therefore the verbal protocols would represent thc verbal communication and thinking aloud that the tour guide gives while carrying out the task, and also his account of the event he gives before or after the task was carried out.

The coding schemes of the verbal protocols would be based on two types of protocol representations: (a) the knowledge state, and (b) conceptual operator. Therefore, the verbal protocol analysis would only focus on these two representations to understand the way tour guide or tour manager discharges his or her task.

1.8 Outline of the Study Chapter two will focus on reviewing literature related to the concepts and frameworks being used in this study.

The research nlethodology being used in this study will be described in chapter three, followed by the depiction of appropriate conceptual model with respect to the different stages of the study. This section will elaborate the procedures of data collection, and the processing and analyzing of the tour guide verbatim.

Chapter four discusses and gives the result of the analysis of the verbal protocols. Here, the mental model of tour guide or tour manager will be developed which is based on the conceptual operators and knowledge states being identified and the coded scheme of the verbal protocol.

Summary of the study, conclusion and suggested recommendations will be done in the final chapter, chapter five.

Chapter Two

LITERATURE REVIEW

2.1 Introduction The efficient and effective performance of task depends on the information or understanding acquired through experience. Likewise, conceptual and skilled knowledge are essential in discharging a task. Cognitive task analysis represents a transition from emphasis on knowledge structures to emphasis on the dynamics aspects of mental models (Seamster et al, 1993). Thus this study would use verbal protocol analysis to identify the knowledge states and operators being used by the tour guide or tour manager in executing his or her tasks.

2.2 Knowledge There are at least two senses in which a person can be said to know (McNamara, 1994). One kind of knowledge can be verbalized, visualized, declared in some manner, and for these reasons has been called declarative knowledge. A second type of knowledge consists of skills, cognitive operations, knowledge of how to do things, and has been called procedural knowledge. Anderson (1983) as cited in Philipchalk and McConnell (1994) assumes that there is a single, unified memory storage unit that has two major parts call declarative knowledge and procedural knowledge. The declarative knowledge stores factual information, most of which can be readily described in words. Procedural knowledge, however is that part of memory which stores motor skills, strategies for manipulating facts, and knowledge of various rules for manipulating objects or events. These types of knowledge are essentially important for the performing of task. Cognitive task analysis is to delineate the mental processes and skills needed to perform a task at high proficiency levels. The protocol analysis would be used to identify the knowledge and cognitive operation used by the tour guide or manager in executing his or her tasks.

Stevensen (1993) mentions that, vast amounts of general knowledge are stored in long-term memory, knowledge of concepts and knowledge of specific things and events. Systems of knowledge representation are designed to explain how such knowledge is stored so that it can quickly and easily be retrieved when needed. Two different kinds of knowledge are represented in such system, usually called knowing that and knowing how. Knowing that is declarative knowledge and refers to knowledge of facts, such as knowledge of concepts and knowledge of individuals and events in the world. Knowing how is procedural knowledge and refers to knowledge of how to do things, such as multiplying two numbers or driving a car.

2.3 Information Processing Theory This theory assumes that there is a set of structures for storing information, and a set of processes for transferring information from one structure to another. We also are not mere recorders of data, but that we are actively transforming and processing stimulus inputs, much like a high-speed computer. This approach proposes that information processing occurs at three different levels (Philipchalk and McConnell, 1994). First, sensory inputs from the environment are translated, or transduced, into patterns of neural energy by the sensory receptors. The incoming information is held briefly in sensory registers one for each sensory modality, Initial processing occurs at this level, and consists primarily of feature detection and pattern recognition. This is the sensory information stage of input processing. Second, information that catches your attention is then encoded as it moves to the next stage, which is called short-term memory (STM) (see figure 2.1). At this point, input would be categorized after comparing it

with information filed away in long-term memory storage. And third, if the input is important, it would lilcely be rehearsed and then transferred to long-term memory (LTM).

SENSORY Unattended INFORMATION information

Environment STORAGE is lost

Attended information added to

TERM

LONG-TERM MEMORY

Figure 2. I : The stages of memory (Philipchalk and ILlcConnell, 1994)

2.3.1 Sensory Information Storage This is the first stage of memory storage. The sensory receptors hold a more-or-less exact copy of an input for a fraction of a second before that copy fades or is erased by the next input. Under normal conditions as he or she loolts from one object to another in his or her visual world, his or her visual system would hold on to each stimulus pattern for but a fraction of a second before yet another visual input replaces the previous pattern. As mentioned by Lindsay and Norman (1977) in Philipchalk and McConnell (1994), this storage time typically lasts from 0.1 to 0.5 seconds under normal viewing conditions. However the storage lasts longer if no new stimulus erases the input. George Sperling (1963) in Philipchallc and McConne11 (1994) refers to visual Sensory Information Storage (SIS) as iconic memory. For the amount of information stored in visual SIS is much like that in an icon, which is a detailed image of a person or event. In similar fashion, auditory SIS is sometimes called echoic memory, because it seems to be an exact echo of an auditory input. However, by the time this sensory information reaches our cortex and we become aware of what we are loolcing at, much of the rich detail is lost.

2.3.2 Short-term Memory Most people must rehearse a telephone number continuously until he or she dials it. This fact indicates that the length of time an item typically remains in short-term memory. Short-term memory (STM) refers to as temporary information storage between the SIS and long-term memory (LTM). STM has a limited capacity of about seven items. That is the STM ordinarily cannot store more than about seven items simultaneously as mentioned by Miller (1956) cited in Philipchalk and McConnell (1994). At the moment that the brain inserts an item into STM, that item is strong and clear and easy to recall, if he or she does so immediately. But shortly thereafter, the brain tucks away a second item, then a third item, and a fourth. Although only a few seconds have passed, the brain will now have much more trouble recalling what the first item was.

By the time the brain has passed five or six new items down on top of the first, that original item has lost most of it strength and has faded away. The new items appear to interfere with or erase the earlier items, just as each new visual pattern he or she looks at wipes clean the stimulus he or she was looking at just a moment before. While he or she is holding an item in STM, he or she can recall it more or less at will. And he or she can keep the item around for several minutes by rehearsing it. Rehearsal refers to the act of silently saying things over and over again. When a person rehearse an item, that person actually keeps putting it back into temporary storage again and again. By rehearsing the item in STM several times, he or she can keep it available for a period of several minutes. However, once the item drops out of this temporary storage, it is likely to be gone forever, unless some more or less permanent record of the stimulus input has been made in long-term memory (Philipchalk and McConnell, 1994).

2.3.3 Long-term Memory If a person rehearses an item for a long enough period of time, we might well not forget it all. This fact indicates how items get shifted from short-term memory to long-term memory. LTM is one's store of permanent memories. Inputs important enough to survive his or her short-term memory are transferred to long-term storage. Salient or critical features of an input are stored according to various categories. LTM seems to be much more complex. It stores many different aspects of ones' experiences. First, rather than merely recording total inputs as they are received, LTM abstracts certain critical features. It then files these meaningful features in an appropriate memory stores. When a person draws on his or her long-term memory, it then reconstructs the event from the associations and auditory representations it has in it files. Ebbinghaus (1 885, 1964) as cited in Philipchalk and McConnell (1994) suggested that people typically does not remember complex events. Rather, he or she recalls a few high points and then reconstruct the experience piece by little piece.

Thus, important inputs move from STM into LTM, which is the permanent memory bank of the brain. Recall from LTM involves remembering significant facts and then reconstructing a memory by filling in plausible details.

2.4 Cognitive Task Analysis Redding (1995) mentioned three aspects relating to cognitive task analysis (CTA). First, CTA uses interviewing, modeling techniques and experimental procedures to determine cognitive processes underlying skilled job performance. This involves determining how a task is learned, how to expedite learning, optimal job knowledge, and the required job performance skills.

Second, CTA provides data on the cognitive skills important to the job. Identifies effective problem-solving and decision making techniques for the job, superior methods for organizing job knowledge, and efficient ways of allocating attention to the various job tasks.

Third, CTA analyzes decision making and mental models, learning and skill development, the interrelationships among job concepts and task elements, and group performance as well as individual differences. Most CTA include the identification of key job components, identification of the knowledge and skills required for similar job components, and the identification of the conditions which best facilitate progression to the next level of expertise.

Simon and Kaplan (1989) mentioned that cognitive science has a wide and varied pool of methods of cognitive research, ranging from computer simulation to naturalistic observation, from recording the electrical impulses of the brain to performing linguistic analyses, from using nonmonotonic logic to collecting reaction times or verbal protocols. The following methods are appropriate for performing a CTA: a) Structured interviews b) Verbal protocol analysis c) Critical decision method d) Conceptual graph analysis e) Means and Gott's cognitive task analysis

Gordon (1994) generalized these methods to develop mental models as part of the CTA. The task analysis methods can be divided into three categories as follows: a) General methods for data collection b) General methods for data representation c) Specific task analysis methods

The first two categories are generic methodologies and the third is specific, commonly used methods, see Table 2.1 and Table 2.2 below.

Table 2.1: Taxonomy of general and specific task analysis methods (Gordon, 1994).

General Methods for Data Collection

Document and equipment analysis Unstructured interview Structured interview

Group interviews Sorting and rating

Questionnaires Verbal Protocol Analysis Observation

Table 2.2: Commonly used combinations of task analysis data collection and data representation methods (Gordon, 1994)

Specific Task Analysis

Methods Position Analysis Questionnaire (PAQ) Delphi and Focus groups

Controls and Displays Analysis Hierarchical Task Analysis Extended Task Analysis Procedure (ETAP) The GOMS model Critical Incident Technique

Functional Analysis System Technique (FAST) Cognitive Task Analysis (Mean & Gott) Conceptual Graph Analysis Associative scaling algorithms Activity Sampling Functional Flow Diagrams

Data Representation Methods

General Methods for Data Representation

List and outline

Matrix (cross-tabulation table)

Structural network

Hierarchical network Flow Chart

Timeline chart

Data Collection

Document and Equipment Analysis Unstructured Interview

List and outline

*

*

Matrix

*

Structural Network

*

*

Hierarchical Network

*

*

Flow Chart

*

*

Timeline Chart

From the above Table 2.2, notice that if verbal protocol was the method for data collection, thenthe mode for data representation could be in the form of either list and outline, hierarchicalnetwork, and or the flow chart.

2.5 Verbal Protocol AnalysisProtocol analysis, the use of verbal reports as data, has become an increasingly importanttechnique for studying human intelligence. Its importance stems from the fact that it is one ofthe few methods in cognitive science that gathers data with sufficient temporal density to testmodels that account for behavior nearly second by second (but not millisecond by millisecond),(Simon and Kaplan, 1989). Protocol analysis treats verbal report as a source of data to beaccounted for with an experimenter-generated theory, perhaps in the form of computersimulation. They further mentioned that there are many ways of getting subjects to generateverbal reports, including questioning them, asking them to report on their mental processes, andasking them to talk or think aloud. Verbal reports of the talk or think-aloud variety tend to bethe least intrusive and to provide the most accurate records of the nature and sequence ofsubjects’ mental processes.

According to Singleton (1989) an operator carrying out a task can be asked to talk about whathe is doing as he does it. Clearly this is useful for tasks which involve extensive scanning ofinformation sources and deliberations about what action to take. The visible action mayapparently be trivial - adjusting one or two of very many available controls - or may even benon-existent. Because much of the activity is in searching for information and processing itleading to a decision, the observed output is obviously a poor indicator of total activity and hisverbal commentary is much better guide to his task performance. Such a commentary can berecorded or notes can be taken directly by the analyst.

Hassebrock and Prietula (1992) mentioned that protocol analysis methodology is derived from acognitive psychological or information processing framework for studying human behavior. Inthis theoretical approach, psychologists have stressed the importance of understanding thecognitive processes (e.g. attention, perception-and memory operations) and knowledge (content,representation and organization) that underlie complex problem solutions, decisions orjudgements. Thus, the analysis of verbal protocols is one methodology designed to reveal theknowledge and cognitive processes used by a person while performing a task or behavior.

Redding (1995) stressed that protocol analysis is particularly useful for investigating problem-solving methods and mental models. Most applications of protocol analysis view problem as asearch through a problem space accumulating knowledge about the problem situation, andapplying operators to the knowledge to reach each new point in the space until a solution isreached.

Medin and Ross (1992) explained that verbal protocols have provided a great deal of insightinto the problem-solving processes and can often be followed up by more traditional dependentmeasures. Although there are a number of possible ways to collect data, the most common is toask the subjects to think aloud as they go about solving the problems. That is they are asked tosay whatever they are thinking about, but not to embellish or explain it for the experimenter’sbenefit. The idea behind this measure is straightforward that is, people’s thoughts providefurther information about the course of their problem solving. Although thinking-aloudprotocols may seem intrusive, the use of such protocols usually appears to have no effect on thesolution of the problem other than slowing down the performance.

9

Payne et al. (1998) revealed that protocol analysis is one approach to gathering detailed process-tracing data on decision making. To use this approach, the subject is asked to think out aloud ashe or she is actually performing the task of interest, such as choosing among severalalternatives. Such a verbal record is called a protocol. Protocols differ from introspection orretrospective reports about decision processes because the subject is asked to verbalize thoughtsas they occur in the course of making a decision. The protocol data are then analyzed to attemptto gain insights into the subject’s decision processes. The major advantage of protocol collectionand analysis is that a great deal of data on internal events is made available.

However, Tolbert and Bittner (1991) as cited in Gordon (1994) mentioned that there are threetypes of verbal protocols, such as concurrent, retrospective and prospective. In a concurrentprotocol, the worker or expert is asked to perform one or more task and verbalize both theknowledge and strategies responsible for his or her task activities. Sometimes they are givenspecific focal points such as identification of stimuli or triggers that are important or responsiblefor their responses, decision strategies, or why they made a certain response.

Sometimes concurrent protocols are difficult to collect, especially if they occur quickly or aretoo cognitively demanding (Benjafield, 1992). In such a case, retrospective protocol can becollected, in which the subject is asked about cognitive processes that occurred at an earlierpoint in time (Benjafield, 1992). He further mentioned that concurrent verbalization relies onshort-term memory, whereas retrospective verbalization relies on long-term memory.

In the case of a prospective verbal protocol, a scenario is given for the subject to work on or thathe is to imagine performing the task. The advantage of this method is that the subject would notbe exposed to a situation that is difficult, dangerous, or the task is too costly or time-consumingto perform. The main drawback, however, is that the verbalized data may quantitatively bedifferent since there is lack of real tasking cues or triggers to instantiate the mind of the subject(Peskin, 1998).

Although protocol analysis often allows the researcher to gain important insights into decisionmaking, there are some disadvantages. Collecting protocol data in quantity is extremely time-consuming, so small samples of subjects have typically been used (Payne et al., 1998). Inaddition, protocol data may not be entirely reflective of subjects’ decision processes. Theprotocols may reflect subjects’ biases or may be censored by subjects while they are beingreported. In addition, subjects may be unable to verbalize retrospectively some internalprocesses (Nisbett and Wilson, 1977) as cited in Payne et al. (1998). Finally, protocols may notprovide insights into all of the processing performed, and there may not be output correspondingto all internal states. Subjects may select aspects of processing to verbalize based upon whatthey believe is important and may not verbalize those data most valuable to the researcherPayne et al. (1998).

Although such problems with selectivity in verbal reporting may exist, several researchers haveargued and have provided convincing evidence that decision makers do have self-insight,(Ericsson and Simon, 1993). They further argued that many studies showing no effects of takingprotocols on decision processes. In studies of decision making, however, the results have beenmore mixed. Although Smead et al. (198 1) and Biehal and Chakravarti (1993) as cited in Payneet al. (1998) reported no significant differences between protocol and no-protocol conditions,Biehal and Chakravarti (1993) found differences in the extent of alternative-based processingand problem framing due to verbal protocols, Therefore, although verbal protocols can provide

10

invaluable data on choice processes, one must be very careful to control for any effects of takingprotocols.

However Gordon (1994) expressed that there are certain advantages of verbal protocol as amethod for data collection, as follows:(a) It is more likely to elicit task-relevant information than having the person respond to

interview questions without any task cues.(b) It is relatively easy and inexpensive to implement.(c) It tends to provide information of two types, declarative conceptual knowledge and explicit

rule knowledge for how to perform subtasks, both of which are important for complete taskanalysis.

(d) It has face validity.

According to Simon and Kaplan (1989) verbal protocols generally provide explicit informationabout the knowledge and information heeded in solving a problem rather than about theprocesses used. Thus it is usually necessary to infer the processes from the verbal reports ofinformation heeded instead of attempting to code processes directly. The talk aloud proceduresbeing used to study human information processes while performing the task would entail therecorded protocols, These protocols are then segmented and encoded and the basis ofsegmentation would depend on the type of encoding and analysis. Encoding refers to theprocess of converting the content of segments into a formal language or as Philipchalk andMcConnell (1994) put it, converting into a form that can be entered into memory. The nature oftask and the hypotheses involved would determine what to encode.

In a research done by Hassebrock and Prietula (1992), “A protocol-based coding scheme for theanalysis of medical reasoning”, the audio recording of a case was transcribed for analysis andinterpretation. The transcript includes patient data statement follows by a new paragraph for thephysician’s statements while reading the data card aloud and another new paragraph containingthe subsequent think-aloud statements. The transcript was segmented by identifying eachsegment with a knowledge state, which was in turn associated to one of the possible basicconceptual operations in the coding scheme of the verbal protocol. The following excerpt is anexample of transcript fragment and coding used Hassebrock and Prietula (1992):

Excerpt 1Example of and coding

21. Ah-ascending aorta is slightly enlarged,(1) Ah, well that means post-stenotic dilation or increased ah, blood flow throughit. (2)It would be consistent with ah, ah, transposition of the great vessels with sub-aorticstances, (3) uh, tetralogy of Fallot would be ah, or a tetrad variant.1.1 Hypothesis-generation: triggers 3 (post-stenotic dilation; increased blood flowthrough aorta)21.2 Hypothesis generation: association j (TGV,+& sUb,#)21.3 Hypothesis-evaluation: confirmation 3 (TF; Tetrad variant)

Z’he fzrst numbered paragraph typed in italics represents the physicianreading the card aloud (thus indicating the information available. Theverbalized thinking-aloud responses follow in a new paragraph with theseparate segments numbered and the relevant knowledge states identiJied. Thecoding interpretation of the think-aloud response follows with the cue-number

1 1

(to the left of the decimal point) andsegment-number (to the right o f the decimal point) noted

For each segment, and to the left of the arrow, there is listed the basic conceptual operation and the more specific operators used, ranging from the most general (leftmost) to the most specific form. The knowledge state or states in each segment are shown parentheses to the right of the arrow. (Hassebrock and Prietula, 1992, p. 633).

In this study, Hassebrock and Prietula (1992) developed a coding scheme to provide an explicit framework for documentation of behavioral events and a guide for structuring the analysis. This coding scheme depends largely on the nature of the task and the theoretical constructs of interest or a theory of reasoning in the domain.

2.6 Protocol-Based Coding Scheme This section will focus on the coding scheme based on Hassebrock and Prietula (1992), that they adopted from Newell and Simon ( 1 972). This scheme, being used to code topic representations, suggested that protocols consist of two basic types of semantic elements. Firstly, a given topic segment consists of knowledge states which are units of information or knowledge retrieved from memory used by an individual. The second type is, these knowledge states are associated with conceptual operations which are inferential processes that produce new (or modify existing) knowledge states. Thus the goal of the analysis is to trace the sequence and patterning of knowledge states and conceptual operations used in response to a problem-solving task. However Hessebrock and Prietula included a third element in the coding scheme, that was, a means of representing physician's use of more global diagnostic strategies and problem solving representations known as lines of reasoning. A line of reasoning can be viewed as a problem- specific representation of, in this case, physician's medical problem-solving knowledge. Figure 2.2 summarizes the three elements incorporated in the coding scheme, knowledge states, conceptual operations, and lines of reasoning.

/ / Data examination

Data exploration

Conceptual Data explanation Coding

operations Hypothesis generation scheme Hypothesis evaluation

Discrepancy processing Meta-reasoning

Lines of Summarization

reasoning

Figure 2.2: Basic elements of coding scheme (Hassebrock and Prietula, 1992)

Figure 2.2 depicts the three components of the coding scheme. First the knowledge states that identifies units of medical knowledge used by the physician in diagnostic reasoning and the

occasion at which they were brought into consideration. Data interpretations and diagnostic hypothesis are the two general categories of knowledge states being represented in this analysis.

Another representation in the scheme identifies the conceptual operations being used to generate or instantiate states of medical knowledge. Newel1 and Simon (1972) as cited in Hassebrock and Prietula (1992) referred this to as an inferred cognitive process which modifies (adds, eliminates) existing or currently active knowledge states and produces new, active knowledge states (Hassebrock and Prietula, 1992). In response to a specific data cue, a given segment of verbal protocol will constitute (a) one or more knowledge states (interpretations or hypotheses) and (b) a conceptual operation that produces the associated knowledge state or states as shown in figure 2.2.

Hassebrock and Prietula (1992) identified eight basic types of conceptual operations to characterize distinct segments of a physician's problem-solving behavior. Each will then be further analyzed in order to provide a more detailed representation of reasoning behavior. The conceptual operations are analogous to goal-directed medical problem-solving behaviors found in physicians' diagnostic reasoning.

For instance, data examination, the physician's goal in this problem solving episode is to select and examine cues and make initial interpretations of the findings in the patient data of a case. Whereas in data explanation, the goal is to interpret the significance or role of a given data cue by explaining the underlying pathophysiological basis of a disease. Often, this type of data explanation is not used to generate a particular hypothesis, but is used to account for the natural history, which could have led to the development of data finding. Further example is, hypothesis-generation where the goal is to generate diagnostic hypotheses. Specific operators provide different means of generating hypotheses in response to a particular data cue, pattern of cues or from other diagnostic hypotheses. Whereas hypothesis-evaluation is to evaluate a diagnostic hypothesis by interpreting its ability to account for the presence/absence of data cues, data interpretations, or its relationship to other diagnostic hypotheses. Hypothesis evaluations usually assume either a confirmatory or discomfirmatory value. See Figure 2.3 for more detail description of conceptual operations in the analysis of medical reasoning by Hassebrock and Prietula (1 992).

The third type of protocol representation in the scheme is the lines of reasoning. Hassebrock and Prietula (1992) described it as a reasoning artifact representing a complex set of related pathways of thought bound together and defined by linked pathophysiologies and other medical knowledge evolving in the context of a particular problem instance. In their study, the identification of a line of reasoning is an attempt to capture the salient aspects of reasoning that provides coherence and comprehension in the face of information complexity and uncertainty. Many forms of conceptual and mental models have been proposed for explaining cognitive activities, including comprehension and problem solving. However, Hassebrock and Prietula (1992) have developed the line of reasoning construct within the context of a specific domain of knowledge so that the cognitive properties of a mental model are dependent upon information constraints associated with the task of medical diagnosis.