Human-Computer Interaction -- INTERACT'03 M. Rauterberg et al. (Eds.) Published by IOS Press, (c) IFIP, 2003, pp. 264-271

The Process of Developing a Mobile Device for Communication in a Safety-Critical Domain

Jesper Kjeldskov Department of Information Systems

University of Melbourne Parkville

Victoria 3010 Melbourne, Australia

jesper@cs.auc.dk

Jan Stage Department of Management Science

and Information Systems University of Auckland

Private Bag 92019 Auckland, New Zealand

jans@cs.auc.dk Abstract: This paper describes the process of developing an interactive mobile system for use in a safety-critical domain. The system was developed to inquire into the extent of which coordination between people carrying out a collaborative work task can be improved by means of computer mediated communication on a mobile device. The application area for the device was controlling and monitoring processes in the safety-critical domain of a large container vessel when it arrives at or leaves a harbor. The development process involved ethnographic studies of work activities in this domain, analysis of communication between actors involved in the work activities, design and implementation of an experimental prototype, and two usability evaluations of the prototype. We emphasize the analysis and design activities and outline the challenges involved in the development of the mobile device. Also, we outline how our experiences are relevant in a more general context. Keywords: Mobile device, development process, analysis and design, usability evaluation, safety-critical domain, computer mediated communication

1 Introduction Mobile and wearable computer devices are being developed for use in a broad variety of application areas. Recent research has devoted much attention to the extent to which distributed mobile users can benefit from mobile computer systems. This includes research on situations where actors are concerned with computerized information and processes of critical importance remote from their current location. Examples count distributed process control and error diagnosing in wastewater treatment plants (Nielsen and Søndergaard, 2000) and the use of mobile multimedia communication for telemedicine and early diagnosing in emergency ambulance services (van den Anker and Lichtveld, 2000). Safety-critical domains are potentially interesting application areas of mobile computers. A safety-critical computer system can be defined as a “computer electronic or electromechanical system

whose failure may cause injury or death to human beings” (Palanque et al., 1998). In relation to human-computer interaction in safety-critical domains, the key problem is not software failure as such but rather failure to support correct user interaction caused by poor interface design. Examples of this count the British Midland 1989 air crash, in interface design in the cockpit contributed to pilots erroneously shutting down the only operational (Ladkin, 1998) or the Three Mile Island nuclear power plant accident in 1979. Based on these and similar experiences it has been suggested that instead of forcing additional rules and regulations on the operators of complex computerized industrial installations, better designed user interfaces could support the avoidance of hazardous situations (Fields et al., 1999; Leveson, 1995; Mackay, 1999). This also applies to the use of mobile computer systems is such domains (Nielsen and Søndergaard, 2000; van den Anker and Lichtveld, 2000).

Analysis, design, and prototyping methods are general means for supporting development of user interfaces and software systems. Generally very little has been published on the processes of developing mobile device applications. Interesting examples count Sharples et al. (2002), Mikkonen et al. (2002) and Iacucci et al. (2000) who outline a number of differences from the development of traditional software. In relation to safety critical mobile device applications Knight (2002) touches upon challenges in system development introduced by mobile computers and Bertelsen and Nielsen (2000) presents a design tool that could be applied for supporting aspects of these. Furthermore, Navarre et al. (2003) present a design framework applicable to stationary as well as mobile safety critical interactive systems.

Experience from processes of developing mobile systems for safety-critical domains can support other similar development processes. Experience always originates from a particular situation and therefore cannot replace general methods. Nevertheless, when general methods have not yet been developed, systematically collected experience can be a valuable substitute. Furthermore, such experiences are often needed to support the development of improved analysis and design methods for a specific domain of application.

The following sections presents experiences from the process of developing a mobile device application for supporting communication in a safety-critical domain. Section 2 describes the basic requirements for the device and provides an outline of the overall development process. Section 3 presents the application domain analysis conducted, involving ethnographic studies of work activities in the safety-critical domain of focus. In section 4, we describe the problem domain analysis focusing on communication between actors involved in the work activities observed, resulting in an information-architecture for the mobile device. Section 5 presents the design and implementation of an experimental prototype. Section 6 describes two usability evaluations of the prototype. As focus of this paper is on the activities conducted during the development process, the challenges involved, and the results achieved, the actual design produced will only be presented in order to illustrate aspects of these. Finally, section 7 concludes the presentation and discusses how the experiences presented are relevant in a more general context.

2 Development Process The point of departure of the development process was an essential communication problem

experienced in the safety-critical domain of operating a large container vessel during arrival at and departure from a harbor. This operation requires communication between actors that are physically distributed on the vessel. Currently, this communication is based on spoken natural language being transmitted through handheld VHF radios.

The fundamental design idea was to replace the radios with a mobile device that facilitates exchange of predefined text messages. This is similar to the way in which SMS and e-mail applications on mobile phones and PDAs have been employed to replace direct spoken communication with network-based exchange of text messages. When communicating by means of textual messages on a mobile device, communication is no longer subject to the ephemeral nature of spoken utterances. Moreover, it can be conducted asynchronously, it is not influenced by surrounding noise, and information can be integrated with other computer-based data.

In order to explore this fundamental idea, we developed a prototype of such a mobile device. The activities of this development process are illustrated in figure 1.

Application Domain Analysis

Problem Domain Analysis

Usability Evaluation

Design and Implementation

Figure 1: The activities of the development process

The first activity was analysis of the application domain. The basic approach was to conduct ethnographic studies of work activities in the safety-critical domain. The second activity was the problem domain analysis that focused on the communication between actors involved in the work activities and resulted in an information architecture for the mobile device. The third activity was to design and implement an experimental prototype. The fourth activity was usability evaluations of the prototype.

3 Application Domain Analysis The development of the mobile communication device was carried out in collaboration with Maersk-Sealand. It was based on ethnographic field studies, with contextual interviews and video analysis.

3.1 Field Study Maersk-Sealand operates some of the world’s largest container vessels of sizes equivalent to 3½ soccer fields. A sketch of a ship from this class is shown in figure 2.

Figure 2: Sketch of Sally Maersk compared

to the size of soccer fields

The operation of such a vessel is a safety-critical domain. Especially when maneuvering inside a harbor, erroneous actions may result in the vessel running aground, into the quay, or colliding with other ships. In either case, such collisions would cause serious material damage, potentially severe injuries on personnel, and possible loss of human life.

When the ship is ready for departure, the first step in leaving the quay is to let go of the mooring lines that are holding it in position. A configuration of aft mooring lines is shown in figure 3.

Figure 3: The aft mooring lines of Sally Maersk

When the crew lets a line go, it will remain in the water for a period of time during which no means of propulsion is available due to the risk of lines getting sucked in and wrapped around a propeller or thruster. During this time, the vessel can only maneuvered by means of the remaining lines.

Due to the huge size of the vessel, the work tasks involved when letting go the lines are distributed

among a number of actors located at strategic positions, as annotated at the top of figure 2. On the bridge, the captain and other personnel control the rudder, propeller and thrusters. Fore and aft, the first and second officers control the winches for heaving in the lines.

To ensure the safety of the operation, individual work tasks are carefully coordinated and carried out under strict command of the captain in charge. At present the coordination is primarily based on oral communication via VHF radios.

In order to carry out the operation of departure in a safe manner, the captain needs an overview and total control over the propulsion, direction and mooring of the ship. At present much of this only exists as a mental model in the head of the captain based on his perception of the ongoing communication between bridge and deck. Therefore, considerable cognitive resources are spent on establishing and maintaining common ground (Clark and Schaefer 1989) among the cooperating actors.

The field studies were documented in qualitative descriptions of the application domain (Andersen, 2000; Nielsen 2000). In addition, the processes of departing from and arriving at harbor were recorded on video capturing overall views of the captain, harbor pilot and officers on the bridge as well as close-up views of the interaction with key instruments. The audio channel captured inter-personal communication on the bridge and VHF radio communication with the crews fore and aft.

3.2 Video Analysis In order to facilitate systematic analysis, a person with detailed insight into the application domain transcribed a selection of the video recordings. The transcriptions amount to approximately 200 pages.

Through analysis of recordings, transcriptions, and follow-up interviews, we uncovered the following communication problems:

• Sound quality is often poor • Utterances are not persistent • Communication is time consuming • There is a language barrier • The bridge is a bottleneck • Information is not integrated with other systems

In the maritime domain, spoken communication is prevalent and often carries the majority of vital information. To handle this, a well-established set of formalized procedures for communication exists. However, as the size of vessels and the use of technology increase so does the complexity of systems and the cognitive overhead and amount of

communication required for operating the systems. Thus a strong motivation exists for supporting and improving communication.

4 Problem Domain Analysis The problem domain analysis was directed towards design of a mobile device. It was based on an object-oriented analysis method (Mathiassen et al. 2000) and has similarities to the work presented by Navarre et al. (2003).

4.1 Objects and Classes The object-oriented analysis facilitated identification of relevant objects and their classes. The basis for this was interviews, video recordings, and transcripts. From the interviews and video recordings we identified relevant actors. From the transcripts we identified the key conversation objects as in the following extract from the transcript: 1 <Captain> you can let go the bow line 2 <1st officer> let go bow line 3 <Captain> and you can take the stern spring 4 <2nd officer> letting go stern spring 5 <1st officer> bow line let go 6 <Captain> bow line let go 7 <2nd officer> and stern spring let go 8 <Captain> stern spring let go 9 <Captain> just let go the stern line also 10 <2nd officer> let go line aft 11 <1st officer> and we have the bow line home 12 <Captain> ok 13 <2nd officer> and all let go aft 14 <Captain> all let go aft

This conversation is about three interweaved Let go tasks. The tasks are conducted by two different Teams, represented by the 1st and 2nd officer. They involve two locations: the bow and the stern. The classes and relations are illustrated in figure 4.

Figure 4: Class diagram of the problem domain

Communication consisting of a number of interweaved tracks of conversations can be difficult to overview when sorted by sequence of utterance.

The transcription extract above illustrates this, as it is actually an integration of three conversational tracks taking place in parallel.

Grouping text in accordance to object rather than sequence enables the creation of a more comprehensible representation of communication threads. The objects of this representation are simply expressed in the class diagram.

4.2 Object Behaviour The object-oriented analysis also involves description of behaviour for the classes listed in the class diagram. The most interesting classes were those related to the different tasks. State charts for these classes were used to model sequences of communications as the ones illustrated by the transcripts above and sequences of sub-tasks being carried out sequentially and in parallel.

Figure 5: State chart diagram for the Let go task

5 Design and Implementation A prototype was designed and implemented on the basis of the results from the analysis activities. This prototype should support the task of letting go the lines.

The prototype was targeted at a Compaq iPAQ 3630 handheld computer. Apart from a touch screen, this device facilitated interaction by means of a five-way key located below the display, suitable for one-handed interaction. Due to the potentially harsh conditions of use, in which pen-based interaction might be problematic, we decided to design all interaction for this key.

The configuration of the system consisted of three iPAQs connected through a wireless network. One device was intended for the captain on the bridge while the other two were for the 1st and 2nd officers on the fore and aft deck respectively.

5.1 Overall Design The overall design was based on three key ideas. The first was to replace verbal communication with exchange of predefined text messages. Asynchronous text-based messaging is a flexible, ubiquitous and persistent communication channel that requires only low cognitive overhead (Churchill and Bly, 1999; Popolov et al, 2000). Thus we expected shifting to text-based communication on mobile devices could eliminate or reduce some of the problems that were listed in section 3.2.

The second idea was to provide an updated representation of the process that is monitored and controlled on different levels of abstraction. In order to avoid accidents in safety-critical domains, it is critical to understand the state of the system being operated. Rasmussen (1983; 1986) suggests that computer interfaces should be designed to improve operators’ reasoning about the domain of operation and thus support human interaction contrary of total system automation as also discussed by Norman (1990). For this purpose, Lind (1994; 1999) suggests a formalism for representing complex systems or environments as they were intentionally designed.

The third idea was to use the structure of conversations as the means for interacting with the user. On an overall level, a conversation can be categorized by aspect and tense (Andersen, 2000), so it is either:

• Imminent (future tense) • Executing (present tense) • Ended (past tense)

While executing (present) conversations are still open for negotiation. Ended (past) conversations

imply some kind of mutual agreement having been made among the communicating parties. Imminent (future) conversations are characterized by potentially being initiated when and if appropriate in relation to preceding conversations (ended and executing). Knowing the state of executing conversations, this can be represented visually for fast access: has a request been met? Has an agreement been made? Etc. Also, possible future utterances may be deduced and prioritized over others in the form of predefined standard-phrases as seen on some SMS-enabled mobile phones. Thus demands for user-interaction may be reduced.

5.2 User Interface The user interface is divided into four sections:

• Pictogram of ship and mooring (past and present)

• List of completed communication threads (past) • List of ongoing communication threads (present) • List of unexecuted commands (future)

Past and present

Past

Figure 6: The interface on the

The user interface for the bridgefigure 6. At the bottom of the screenunexecuted commands and confirmaof the list corresponds to the standardoverall operation, and possible utterawhen appropriate in relation to the (figure 5). The state chart diagramdesign these sequences. By default,next step of the operation is highlight

The most important element of thlist of ongoing tasks. When a commit appears on the list of ongocommunication representing uncompto it, a counter displays the time passfor confirmation (figure 7a). Whenconfirmed the timer is substituted byfollowed by a description of the curr“Singling up...”). A counter next to

Present

Future

bridge is illustrated in there is a list of tions. The order sequence of the nces only appear state of the task s were used to the most likely ed. e interface is the and is executed, ing threads of leted tasks. Next ed while waiting a command is the text “[ok]”

ent activity (e.g. this displays the

time passed since confirmation (figure 7b). When a task is reported completed, a short statement (e.g. “1 and 1 fore”) substitutes the description of activity and the captain is prompted for confirmation (figure 7c). When the completion of a task is confirmed), this is indicated by the text “[ok]” (figure 7d).

(a)

(b)

(c)

(d)

Figure 7: Stages of a task

When the captain confirms the completion of a task, the corresponding thread of communication is removed from the list of ongoing tasks and added at the bottom of the history list. When the history list is full, it automatically scrolls the oldest commands and statements out of immediate sight.

For quick reference and supporting different level of abstractions in the interaction with the system, a simple pictogram at the top of the screen graphically represents the lines attached to the quay and textually shows the current status of fore and aft mooring.

On deck, the interface is very similar to that on the bridge thus providing the first and second officers with a view on the present status of the mooring and a list of all past and ongoing communication among the distributed actors. In the list of ongoing tasks, however, officers on deck are requested to confirm commands executed by the captain such as “Let go bow spring”. Correspondingly, the list of pre-defined commands only contains those appropriate at the specific location.

5.3 Implementation The system was implemented using Microsoft Embedded Visual Basic. For the first prototype we connected the iPAQ computers through an IEEE 802.11 11Mb wireless TCP/IP network. Given the specific use domain this will probably not be a viable solution for the final application due to limited range, potential radio interference and consequential risk of loosing network packages. Furthermore TCP/IP has limited capabilities for parallel data and voice transmissions. Instead the use of other network technologies like e.g. Tetra Net (voice and data) should be investigated.

The application running on the captain’s device worked as a server containing a formalized representation of the communication pattern of the task. The devices on deck logged on to this server and identified their physical location. During

operation, function calls and unique command identifiers were exchanged over the wireless network. All network communication was broadcast but processed and represented differently on each device in accordance to their physical location (bridge, fore or aft) and the desired language (defined in an external text-file on each device).

6 Usability Evaluation Evaluating the usability of the prototype was a challenge. Firstly, the use of the device is closely related to specific work activities in a specialized physical context, which can be difficult to recreate realistically in a usability laboratory. Conducting evaluations in the field would, on the other hand, limit means of control and complicate data collection. Furthermore, field evaluations of an early stage prototype in a safety-critical domain could cause a hazardous situation. Hence we decided to carry out two different evaluations in controlled settings.

6.1 Heuristic Inspection In our first evaluation, we applied an established method for expert inspection developed by Nielsen and Molich (1990). The aim of this approach is to test the basic design of an interface using few resources and without involving users.

Figure 8: Heuristic inspection

Three trained usability experts were given a 15-minute joint oral introduction to the application domain of the prototype. Aided by a standard heuristic for usability design (Dix 1998, p. 413) each person spent one hour checking for usability problems while using the prototype. Following the inspections, the team was given one hour of discussion during which a final list of problems should be produced.

The inspection setup consisted of two Compaq iPAQs and a PocketPC simulator on a laptop PC connected through a wireless network. The iPAQs displayed the interfaces for officers fore and aft respectively while the laptop displayed the interface for the captain on the bridge. Two A4 handouts depicted standard patterns of mooring and explained

10 basic concepts and notions of the maritime context for quick reference. All three evaluators were able to use the prototype on their own. In total, 27 usability problems were identified, primarily concerning the graphical interface design.

6.2 Use in a Simulated Context Our second evaluation focused on the performance of the prototype in the hands of prospective users in a more realistic work setting. For this study we used a state-of-the-art ship simulator facilitating a fully equipped bridge and a high-fidelity interactive scenario of the operation of a large vessel. Three teams of two trained maritime officers were given the overall task of departing from harbor using the prototype for communication between bridge and deck. One test subject acted as captain on the simulated bridge while the other acted as 1st officer on the fore deck in a neighboring room. For simplicity, commands targeted at the 2nd officer on the aft deck were fed directly into the simulation and feedback was given by the simulation operator.

The simulator was set up to imitate the operation of a large vessel in challenging weather and traffic conditions corresponding to a real world situation observed during our field studies (Nielsen, 2000).

Figure 9: Video documentation of use in a ship simulator

During the evaluation, the test subjects were asked to think-aloud, explaining their actions and their use of the prototype. Two evaluators located on the bridge and deck respectively observed the test subjects and asked questions for clarification. Total views of the bridge, deck, simulator control room as well as close-up views of the mobile devices were captured by four video cameras and merged into one high-quality video signal providing a synchronized view of the whole setup (figure 9). Following the evaluation, a group interview of 10-15 minutes was carried out.

Observing the use of the prototype by prospective users performing a real work task in the simulator provided rich data on the usability of the design. The study revealed 22 usability problems experienced by more than one user.

6.3 Comparison The two evaluations provided substantial input for redesign. Some of the problems listed in section 3.2 were reduced, but other problems emerged.

The quantitative results produced by the two evaluations were very different. When merging the results, a total of 43 problems were identified of which 63% were accounted for by the expert evaluators. However, real users never experienced 78% of the problems revealed in the inspection, thus questioning if they are problems at all. This indicates that although expert inspections may reveal a large number of usability problems, their importance and scope can be difficult to assess and compare without access to richer data.

To the contrary, all problems identified in the simulator were much more detailed and grounded. Balancing resources, however, the simulator study required 40 man-hours while the inspection required only 10.

7 Conclusion The sections above describe how a mobile device supporting a collaborative work activity in a specific safety-critical domain was developed. The development process involved ethnographic field studies, application domain analysis, problem domain analysis, design and implementation, and usability evaluation.

The safety-critical domain considered was found to be characterized by established procedures and highly formalized communication. This facilitated the development of useful models of the problem domain and the development of a novel design supporting existing work practice and overcoming a number of present limitations in communication.

Thus the process applied to the development of the presented prototype proved very useful in terms of assisting the identification and modelling of relevant characteristics and structures as well as providing input directly applicable in interface design. While we believe that the described process has general value for the development of similar systems, this should, however, be evaluated by applying it to other cases of developing mobile device applications for specialized domains.

Acknowledgements The authors thank the evaluation team and test subjects, Svendborg International Maritime Academy and CHMI. The evaluations were planned, conducted and analyzed in cooperation with Mikael B. Skov

References Andersen, P. B. (2000), Communication and work on

maritime bridges. CHMI Research report CHMI-1-2000 http://www.cs.auc.dk/~pba/ElasticSystems

Bertelsen O. and Nielsen C. (2000) Augmented Reality as a Design Tool for Mobile Devices, in Proceedings of DIS 2000, ACM.

van den Anker, F. W. G. and Lichtveld, R. A. (2000), Early Evaluation of New Technologies: The Case of Mobile Multimedia Communications For Emergency Medicine, in Vincent C. and de Mal B (eds.) Safety in Medicine, Qxford: Elsewier Science.

Churchill E. F. and Bly S. (1999), It’s all in the words: Supporting work activities with lightweight tools, in Proceedings of ACM Siggroup’99, pp. 40-49.

Clark, H.H. and Schaefer, E. F. (1989), Contributing to discourse, Cognitive Science, 1989(13), 259-294.

Dix, A. et al. (1998), Human-Computer Interaction – Second Edition, London, Prentice Hall Europe.

Fields, R. Paterno et al. (1999), Comparing Design and Options for Allocating Communication Media in Cooperative Safety-Critical Contexts: A Method and a Case Study, ACM TOCHI 6(4), 370-398.

Iacucci G., Kuutti K. and Ranta M. (2000) On the Move with a Magic Thing: Role Playing in Concept Design of Mobile Services and Devices, in Proceedings of DIS 2000, ACM.

Knight J. C. (2002) Safety Critical Systems: Challenges and Directions, in Proceedings of ICSE’02, Orlando, Florida, ACM.

Ladkin, P. B. (1998), Computer-Related Incidents with Commercial Aircrafts. University of Bielefeld, http://www.rvs.uni-bielefeld.de/publications /Incidents/

Leveson, N. G. (1995), Safeware: System Safety and Computers, Addison-Wesley Longman Publ. Co., Inc., Reading, MA.

Lind, M. (1994), Modeling Goals and Functions in Complex Industrial Plants, Applied Artificial Intelligence, 8(2), 259-283.

Lind, M. (1999), Plant Modeling for Human Supervisory Control, Transactions of the Institution of Measurement and Control, 21(4/5), 171-180.

Mackay, W. E. (1999). Is Paper Safer? The role of Paper Flight Strips in Air Traffic Control, ACM TOCHI 6(4), 311-340.

Mathiassen, L., Munk-Madsen, A., Nielsen, P. A., and Stage, J. (2000), Object-Oriented Analysis and Design. Aalborg, Marko.

Mikkonen M., Vayrynen S., Ikonen V. and Heikkila O. (2002) User and Concept Studies as Tools in Developing Mobile Communication Services, Personal and Ubiquitous Computing, 2002(6), 113-124, Springer-Verlag.

Navarre D., Palanque P. and Bastide R. (2003) A Tool-Supported Design Framework for Safety Critical Interactive Systems. Interacting with computers, Elsevier, to appear 2003.

Nielsen, C. and Søndergaard, A. (2000), Designing for mobility - an integration approach supporting multiple technologies, in Proceedings of NordiCHI 2000, 23-25 October 2000, Royal Institute of Technology, Stockholm, Sweden.

Nielsen, J. and Molich, R. (1990), Heuristic evaluation of user interfaces, in Proceedings of CHI’90, Seattle, USA, 249 - 256, ACM Press.

Nielsen, M. (2000), Letting go the lines: Departure from Felixstowe harbor. CHMI Research Report CHMI-4-2000 http://www.cs.auc.dk/~pba/ElasticSystems.

Norman, D. (1990), The 'Problem' With Automation: Inappropriate Feedback And Interaction Not Over-automation, in Broadbent D.E. et al. (eds.) Human Factors In Hazardous Situations, Oxford, Clarendon Press, 137-145.

Palanque, P. Paterno, F. and Wright, P. (1998), Designing User Interfaces for Safety-critical Systems, in Proceedings of CHI 98, Los Angeles, USA, ACM Press.

Popolov, D. Callaghan, M. and Luker, P. (2000), Conversation Space: Visualising Multi-threaded Conversation, in Proceedings of AVI2000, Palermo, Italy, ACM, 246-249.

Rasmussen, J. (1983), Skills, Rules and Knowledge: Signals, Signs and Symbols and Other Distinctions in Human Performance Models, IEEE Transactions on Systems, man and Cybernetics 13(3).

Rasmussen, J. (1986), Information Processing and Human-Machine Interaction. New York, North-Holland.

Sharples M, Corlett D. and Westmancott O. (2002) The Design and Implementation of a Mobile Learning Resource, Personal and Ubiquitous Computing 2002(6), 220-234, Springer-Verlag.