Work-centered design

Work-Centered Design: A Case Study of a Mixed-Initiative Scheduler

Keith A. Butler1, Jiajie Zhang2, Chris Esposito3, Ali Bahrami3, Ron Hebron3, & David Kieras4

1Microsoft One Microsoft Way, Redmond, WA, 98052

2University of Texas at Houston

7000 Fannin, Houston, TX 77030

3Boeing Math & Computing Technology 2810 160th Av SE, Bellevue, WA 98008

4University of Michigan

1101 Beal Avenue, Ann Arbor, MI 48109

[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

ABSTRACT We present the case study of a complex, mixed-initiative scheduling system to illustrate Work-Centered Design (WCD), a new approach for the design of information systems. WCD is based on theory of distributed cognition and extends established user-centered methods with abstract task modeling, using innovative techniques for work ontology and top-level algorithms to capture the logic of a human-computer interaction paradigm. WCD addresses a long-standing need for more effective methods of function allocation. The illustrating case study succeeded on a large, difficult problem for aircraft scheduling where prior expensive attempts failed. The new system, called Solver, reduces scheduling labor from 9 person-days a week to about 1 person-hour. These results were obtained from the first user test, demonstrating notable effectiveness of WCD. Further, the value of Solver’s higher quality schedules is far-reaching. WCD extends HCI methods to fill an important need for technical problem-solving systems.

Author Keywords Work-Centered, Design, Methodology, Work Ontology, Mixed-initiative, Scheduling, User Interface, Requirements Analysis, Top-level Algorithm, Business Process Reengineering

ACM Classification Keywords H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

INTRODUCTION Landauer pointed out that successful applications must not only be usable, their functionality must also be useful - actually help people accomplish their work in valuable ways [16]. Many applications have failed for lack of useful functionality, even though their usability was well developed (see Goransson, et al. [10] for a classic set of examples). In fact, if the functionality of an application is not useful, its usability is irrelevant. Conversely, if functionality is chosen effectively, then even poor usability might be acceptable to users.

Good applications need both useful functionality and usability, but the two are not independent. If a task requires a function and the machine wasn’t programmed for it, then by implication it will be left to the user to perform. Without deliberate design the collection of functions left to users may form a workflow that is inherently awkward. Thus the choice of application functionality will not only determine its usefulness, but also sets the limit on how usable it can be. The design of functionality is actually a key step in arriving at both usefulness and usability [14]. Logically it must precede the detailed design of the user interface.

We will present a design experience illustrating how functionality can be the initial focus of analysis and design, and then lead seamlessly to a highly usable interface. Since the overall focus is on supporting information work that spans both the humans and computers, we term the approach Work-Centered Design (WCD).

SUMMARY OF WORK-CENTERED DESIGN METHOD WCD begins with on-site knowledge acquisition and documenting the existing context of work using established methods [11] and process modeling [17]. Problem areas are identified in process reviews with customers and also with discrete-event simulation [12]. The work domain of problem areas is then analyzed with ontology modeling.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee. CHI 2007, April 28–May 3, 2007, San Jose, California, USA. Copyright 2007 ACM 978-1-59593-593-9/07/0004...$5.00.

CHI 2007 Proceedings • Design Methods April 28-May 3, 2007 • San Jose, CA, USA

747

Ontology of Information Work In recent years interest in ontology has grown from an obscure topic in philosophy to an important run-time component for advanced information systems [8]. The novel purpose of work ontology here is to support analysis and design with an abstract, declarative characterization of the work entity (product) in terms of goals, objects, operations, and constraints. Work ontology allows us to describe essential requirements of work in a declarative model that is independent of any context, processes, technology, or even cognitive strategy. It tells us the inherent complexity of the work, and it supports identification of overhead actions that are non-essential for the essential work. Work ontology provides fundamental requirements for the Top-Level Algorithm (TLA) which is developed next.

Top-Level Algorithms The purpose of a TLA is to model, at a logical level, how people and their computers will interact to produce the entity of work, as defined by the ontology. A distinguishing feature of information systems is that their TLA is only realized as the logical set of user procedures working together with the set of logical machine procedures. The logic of TLA is critically important to the success of an information system for technical problem-solving. In conventional application development the machine procedures that are implemented as software are typically well-specified. User procedures, however, are often left unspecified until the user interface (UI) is detailed. Consequently, the logic of the human-machine work-system may never be well understood and its usefulness may never be realized.

A TLA consists of an interacting set of abstract machine procedures and abstract user procedures that will logically accomplish the work goals as required by the work ontology and constrained by the work context. The fundamental requirement, however, is to produce the work entity, which is defined independently from technology. This allows candidate TLAs to be evaluated for how well they will perform in terms of effectiveness, feasibility, usefulness and usability, with confidence that each candidate still meets the fundamental requirement to produce the work. Much of this evaluation can be done abstractly, before functions are implemented or the detailed UI is designed.

The TLA is abstract and does not make any commitments to the design or implementation, such as technology or user interface. Rather, TLA specifies an allocation of required functions across user and machine. A given function allocation can be evaluated for its usefulness and value, and compared with other candidate TLAs. When satisfactory results are obtained the detailed design and implementation of functions and UI can begin.

The TLA is a key design artifact in WCD. The TLA is an abstract but specific definition of how the user and the

computer will interact to accomplish the work - it is a solution to the classic problem of function allocation. In addition, the TLA specifies the top level of the user's procedures, which then drives the UI design process.

Ontology Informs UI Design Another use of work ontology is to provide important requirements to exploit the representation effect for UI design [24]. Psychology research on human problem-solving shows that different representations of a common abstract problem can generate dramatically different efficiencies, task difficulties, and behavioral outcomes. Representational analysis [22] is a technique to identify an effective information display format for a given problem to enable direct interaction. WCD takes user procedures from the TLA and the reference model provided by work ontology as inputs to perform representational analysis. This analysis can generate and evaluate usable, interactive representations of the work problem for the UI.

CASE STUDY OF THE SOLVER The key claim of WCD is that by focusing early attention on the nature of the work, how to distribute work in TLAs and how represent work in UIs, it will be easier to develop an application that is both useful and usable. The design experience for Solver that follows illustrates this claim.

The Work Domain: Flight-line Maintenance Management The general work domain of our study is the management of aircraft maintenance. Much of the touch labor is done where aircraft are parked, so historically it is referred to as flight-line maintenance, although its facilities now include specialized hangars where certain maintenance must be performed. The squadrons responsible for flight-line maintenance perform the majority of all labor on aircraft.

Our knowledge acquisition team conducted extensive interviews and validated process models with about two dozen personnel in three squadrons at three different air bases. Process models showed surprising commonality at high levels across the three sites, which in retrospect should be expected across military units. The processes of a flight-line maintenance squadron organize the work of about 220 people and involve another 6-8 sister organizations.

Our analysis of work processes showed that planning and scheduling of aircraft was a key problem area. Within a maintenance squadron the planning and scheduling function (P&S) plays a central role to produce monthly and weekly schedules. These schedules determine when the majority of all maintenance work will take place, and which aircraft will fly which missions. They also have a strategy for exception handling. In terms of information flow the schedule is a key source of data for any attempt to introduce computing to maintenance management.

The Complex Design Problem P&S is highly complex, technical information work. There is inherent tension between flying aircraft for as much time


748

as possible and the need to maintain them in good mechanical condition. P&S must strike a good balance.

A periodic list of tasks and inspections for maintaining each aircraft is published by its manufacturer. Adhering to the list periodically takes aircraft out of service, but overall it prolongs their life. Each aircraft has a log book that includes its maintenance and flying history. Safety rules require audits of all log books for adherence to maintenance requirements and accurate record-keeping. Squadron policy defines optimal maintenance intervals and acceptable grace periods. The policy also defines a hierarchy of mission and maintenance priorities.

The tension between use and maintenance is also present in scheduling for other types of equipment, but aircraft scheduling is especially complex. It requires a broad technical understanding of maintenance tasks, the resources they require, and how the tasks are related to aircraft systems. A squadron can have 25-45 aircraft, each with dozens of impending maintenance tasks, which require hours of expensive resources with limited capacities. In addition to scheduled work there are often new, unexpected repair requirements when systems fail on aircraft. All repairs must be done in a timely manner, but those that are safety-related get treated with urgency, which can disrupt the most carefully planned schedules. Any change can cause subtle but important ripple effects on the rest of the maintenance schedule and also on the schedule for flying missions, which are interdependent.

Maintenance scheduling is complicated because flying schedules take aircraft away from home for extended periods, where some key maintenance is difficult to schedule. A different organization, the Operations squadron, develops the schedule for flying missions. Operations usually has precedence over Maintenance, but its scheduling proceeds rather independently.

The Equipment Utilization & Maintenance Schedule (EU&MS) is the document in which all the factors are presented and resolved in detail. The EU&MS accounts for how each aircraft will be flown or maintained on a day-by-day basis over either a month or a week. It must be produced on-time week-in and week-out to satisfy all flying and maintenance requirements. It is so important that the EU&MS constitutes a literal performance contract between Operations commanders and Maintenance commanders.

A good EU&MS must be the result of effective trade-offs among all the scheduling factors, but the current situation is very difficult. P&S personnel must integrate data from a different “stovepipe” system for many of the factors. The data have different formats and the schedulers must compensate with manual effort. P&S schedulers are very dedicated and proud of the quality they achieve. The situation is very labor intensive, however, and requires 2-3 technical managers working hard for 3 days every week to produce a good quality EU&MS in time for the following week.

Figure 1: Ontology of Schedule for Aircraft Use & Maintenance

The level of effort is not the only issue. Current systems are seriously inflexible: When a change must be made late in the EU&MS process, schedulers don’t have time to re-optimize. They must look for the smallest possible change if they are to get it revised in time for approval. Each group of schedulers told us about earlier, major efforts to automate P&S that failed due to inflexibility. Scheduling work is done under time pressure and the stakes are high. Lost flying hours or broken schedules can easily cost millions of dollars, and disruptions in maintenance efficiency can easily double or triple its expense. The job performance evaluations of Maintenance commanders are based on how well they execute their part of the EU&MS.

We observed a variety of different displays and formats for the EU&MS at different squadrons. Each version was used to create and document EU&MS in a different way, yet they appeared to represent the same logical content. Before considering how information technology should be applied to P&S we needed to know how to define what a schedule is in a more fundamental way that was common over these variations. Ontology modeling provided this definition.

Ontology of Aircraft Scheduling Figures 1 and 2 provide a diagrammatic overview of the work ontology which served as our definition of the essential nature of the information work for scheduling the maintenance and use of aircraft. The models are shown as class diagrams for brevity here. The detailed version, modeled in relational algebra [4], runs nine pages.

As shown in figure 1 our ontology is based on the principle that both mission schedules and maintenance schedules can be represented as a set of quadruples: (Activity, Entity, Resources, Date-Hour). A schedule has Activity that is intended to change the state of an Entity. For example, a repair activity changes the state of a grounded aircraft from not mission-capable to mission-capable. Or, a training mission changes the state of an aircrew from unqualified to checked and certified. Also shown in figure 1, an Activity requires Resources in order to effect the state change on the


749

Entity. For examples, repairing an aircraft requires Resources for labor, parts, tools, etc., or a flying mission requires at least one aircraft among its Resources.

Relationships among the objects also impose important constraints in our Schedule model. In order to schedule an Activity, the Entity and all required Resources must be available. Consequently, developing a schedule necessarily involves finding a Date-Hour when the Entity and the Resources can be synchronized at the needed location for the required duration. For example, scheduling an oil change involves finding a Date-Hour when the aircraft can be in the required facility when the required parts and the required labor will be available in the same facility- all for the duration required to complete oil change. Similarly, scheduling a mission always involves finding a Date-Hour when Aircraft can be in the location for the Mission.

The Schedule Ontology provides a meta-model (template) that can be applied to maintenance schedules and to mission schedules. The left half of figure 2 shows how a Mission Schedule is a set of quadruples: 1) A Mission is activity to change the state of some Entity; 2) The Entity depends on the type of the Mission (Cargo, Targets, etc.); 3) Resources always include at least one Aircraft, and may include other Resource types, depending on Mission requirements; 4) A Date-Hour when Entity and Resources are all available within the needed time window for sufficient duration.

The Mission Schedule keeps its relationships from the meta-model, but has three sub-classes of Mission:

Transport, Combat, or Training. The sub-classes of Entity correspond to the purposes of Mission sub-classes: A Transport mission is supposed to change the location of a Cargo entity; A Combat mission is supposed to change the state of a Target entity (e.g., from functional to non-functional); A Training mission is supposed to change the state of an Aircrew entity (e.g., increasing their skill level).

The Resource sub-classes in the Mission Schedule always include at least one Aircraft. There are many other Resource sub-classes that are too numerous to list in the diagram that depend on the Mission type. For example, a Training mission may require a Practice Range. It is important to note that Aircrew has dual roles: the Entity for Training Missions, but also a type of Resource.

Maintenance Schedules can also be analyzed by the same meta-model, where the Activity is Maintenance, as shown in the right half of figure 2. It keeps its relationships from the meta-model, but also has many sub-classes, including Scheduled, Preflight, Post-Flight, and Repair (a.k.a. Unscheduled Maintenance).

In the Maintenance Schedule an Aircraft must be an entity because the unique history of flying and maintenance determines every aircraft’s maintenance requirements. The Resource has many sub-classes, including Labor, Part, Tool, and Facility. The type of maintenance determines the Resources that are needed.

Figure 2: Integrating Mission Scheduling and Maintenance Scheduling. The left half is the meta-model for Missions and the right half is for Maintenance.


750

Figure 3: Concept of a Unified Scheduler

Product Concept for Solver The meta-model revealed how scheduling work that was previously done by separate groups asynchronously could be done in much greater collaboration. Mission schedules and Maintenance schedules share a key object in common, the Aircraft. In the Mission Schedule at least one Aircraft is always required as a Mission-Resource. In the Maintenance Schedule the Aircraft is always the Entity on which Maintenance activity is performed. The logical unification allows a search operation on the combined schedule space, which enables a new system concept, as shown symbolically in figure 3.

In figure 3 there are two Activities, Flying and Maintenance, which both require an Entity and a Resource. Aircraft#1 is required as a Resource for Flying and as an Entity for Maintenance. The availability of Resources and Entities are both shown on a common calendar, along with their respective Need-Windows. The algorithm searches within the Need-Window for the first available date for required Entity and Resource, displayed in green. Unavailable dates are shown in red for the sake of contrast in figure 3. The algorithm continues until it finds the first common available date for all required Resources and Entity.

In this fictional scenario the Flying Activity requires a Cargo Entity and Aircraft Resource for one day, and the Maintenance Activity requires a Labor Resource and Aircraft#1 Entity for one day. Aircraft#1 is available on Thursday (Th) to support Flying, but the Cargo is not. However, both the required Labor Resource and Aircraft#1 are available on Wednesday (as indicated by a pair of green bars) so the quadruple can be set and added to the schedule for Maintenance. Similarly, both Aircraft#1 and Cargo are available on Friday (F) so that quadruple can be added to the schedule for Flying. We named the concept The Solver.

Analysis & Design of Solver’s Top-Level Algorithm The measurable design objectives were to: Reduce the amount of labor for scheduling; Optimize schedules better than the current manual system; Support rapid re-optimization when changing a schedule. The design of the TLA was an iterative process that involved complex decisions about function allocation, the capability of available technology, and models of how users would respond to interface concepts. The eventual TLA reflected trade-offs among these and other factors.

As shown in figure 4 the two sets of procedures for Solver’s TLA were designed with care because they are mutually constraining at several levels. In addition, the user procedures must include the means to control the machine, and the machine procedures must provide the user interface.

The machine procedures and the user procedures were designed in two distinct tracks using concurrent engineering methods so the implications of design decisions in one could be understood in the other [2]. The machine procedures were modeled initially as math equations and then as software prototypes to evaluate technical feasibility. User procedures were modeled with GOMS. GUI prototypes were used to identify more detail for the user procedures.

Machine Optimization Procedures in the TLA A thorough review of optimization technologies determined the best fit was integer programming, which is based on linear programming of cost functions and is available in commercial software libraries [18]. A mathematical model for linear optimization was developed using the schedule ontology as the direct reference model. For the optimization criteria we created a mathematical encoding of the existing organizational policy. Fast, efficient optimization of a large number of integer variables and constraints was an important consideration because the encoded policy has more than 5,000 variables and constraints. The encoding prefers optimal solutions. However, integer programs must run to completion, which in turn requires some way to handle maintenance tasks that get pushed outside of their time window. Solver penalizes these so they will not be preferred over others with fewer or smaller deviations. If there are significant deviations from policy they are detected and reported to the user interface so that the human scheduler can take the appropriate action. The result is that the Solver is a mixed-initiative scheduler. The logical relationship between the human scheduler and the integer program for this mixed-initiative scheduler is explained in the next section.

Complex interactions made ordinary optimization unlikely. If the maintenance requirements were expressed as hard constraints without any flexibility then the optimization


751

Figure 4: Solver’s TLA


752

program could easily determine that there was no feasible solution, and not reveal which constraints led to the failure. Conversely, too few constraints could easily produce poor schedules. For example, if the program’s cost functions gave missions top priority and some mission needed all the aircrafts, then the program would assign them all, even if delaying an oil change meant damaging an engine.

Even if it were possible to encode all the relevant policy knowledge, integer programs are not good at dealing with these kinds of boundary conditions. Experienced human schedulers are expert at policy and dealing with boundaries. So, the characteristics of the problem and the available technology led us to integrate human judgment with machine optimization

Model-based Development of User Procedures in TLA User procedures had to support a strategy to propose partial solutions and a strategy to review solutions for completeness and policy satisfaction. In order to develop the procedures and evaluate their adequacy we used GOMS to model them explicitly. Kieras [14] proposed High-Level GOMS Models (HL-GOMS) as an approach to the design of functionality, i.e., the problem of how the functions of a system should be chosen to maximize both the usefulness and usability of the system.

In HL-GOMS the methods contain only high-level operators, which describe parts of the user’s activity that are independent of the specific interface, and do not specify specific operations in the interface. Thus, the user is modeled as interacting with system functionality, not the system’s user interface. We iterated on the choice of system functions and the corresponding high-level methods until a satisfactory result was obtained. The HL-GOMS model made clear how the more powerful system functionality would affect the overall top level of the user's task: namely, the system would create the initial schedule itself, the user would then check it for validity, and correct any problems manually.

Gaps in the HL-GOMS model quickly exposed and made explicit two major design issues. The first arose from the complexity that users would experience if they had to manually assign aircraft to missions that spanned more than one day, then track the consequences for other assignments. The need to have this function automated focused attention on a study of its technical feasibility. Scheduling code was prototyped and it became pivotal for the rest of the design. With the proper UI it would allow users to make or edit some assignments, then invoke the optimizer which would accept the user’s entry as constraints to finish the solution. This became the paradigm for mixed-initiative interaction.

The second major issue was the type of intermediate results that the user would have to review. The answer

depended on the limitations of the optimizer’s technology. A characteristic of integer programs is that they must run to completion in order to provide much information, regardless of how much scheduling policy they violate. A fully automated solution could frequently produce unacceptable schedules. For example, if the cost functions give missions top priority and a mission needs all the aircraft in a squadron, then an integer program would assign all the aircraft, despite delaying an oil change to the point where an engine could be damaged. Even if it were possible to encode all the relevant knowledge, integer programs are not good at dealing with these kinds of boundary conditions. So, the characteristics of the technology meant that intermediate results would probably be solutions with some maintenance out of bounds. The results of the HL-GOMS analysis are summarized as the Solver’s TLA in figure 4.

In Solver’s TLA the general role of machine procedures (right column) is to accept constraints from the user and use them to perform optimization, while the general role of the user procedures (left column) is to apply policy and judgment to exceptions. The TLA begins with the user selecting a month and the machine retrieving projected data for that month. The projection takes into account all intervening schedules. The user reviews the display for completeness and accuracy. If there will not be enough mission-capable aircraft on hand for the flying requirements, then the Production Supervision must be notified, who will initiate release from commitments. If sufficient quantity will be on hand the user enters any last-minutes changes to maintenance status or missions.

Machine procedures accept user-entered data and check for conflicts in limited maintenance resources. If there are conflicts Solver attempts to move tasks around within their time windows and displays results as shown in figure 5. If tasks must be moved outside their windows they are displayed in red. The user must try to double-up tasks, trade them, or request exemptions to start early.

The user can assign specific aircraft to specific missions if they have unusual characteristics. The Solver then accepts all new data as constraints and assigns specific aircraft to any remaining multi-day missions to meet their needed number of aircraft, while checking that sufficient quantities will be available for daily missions.

The user reviews the Solver’s attempt to satisfy all the constraints and requirements. Any missions or maintenance tasks that cannot be satisfied within policy are colored red. Maintenance may be moved to satisfy flying requirements. The user must apply judgment about policy for unsatisfied requirements, and escalate the problem to seek either an exemption or a release from commitments.


753

Figure 5: Screen Image of Solver’s GUI

Once the TLA stabilized we began to design machine procedures, user procedures, and the UI to support the user procedures. The ontology served as a reference model for the development of the software and also the user interface. We applied cognitive performance modeling against user interface concepts to evaluate users’ procedures. Finally, summative user testing is used to confirm the effectiveness of the work-system to execute the TLA and meet the design objectives.

Representation Analysis of the Interactive Display We treated the UI for Solver (figure 5) as an interactive problem display. There are several distinct requirements that drove its design. It must induce users to perform their part of the TLA (figure 4). Second, a mixed-initiative HCI paradigm has some UI requirements that are distinct from other paradigms. It requires that the user have visibility of the entity of work as it progresses through all of its states. The user must enter data that serve as constraints in machine optimization. A related requirement was that users must be able to manipulate the work in its intermediate state with minimal overhead.

The meta-models for missions and maintenance provided reference models that supported an analysis of the technique for displaying variables. It focused on the concordance between the variable types that were represented and the corresponding representations that were used to display them. With direct interaction interfaces, users can directly, completely, and efficiently perform their procedures of the TLA.

The first step of representational analysis [24] identifies the scale type of each dimension (variable) in the ontology: ratio, interval, ordinal, or nominal. The second step considers alternative representations (e.g., length,

distance, etc.) of a given dimension (e.g., duration of a repair). The critical issue for dimensional representations is to make sure that the scale type of the dimension to be represented (e.g., status of aircraft) matches the scale type of the dimension doing the representation (e.g., color).

The third step is to generate alternative representations of relations (i.e., constraints) in the ontology. These relational information displays can be categorized by a representational taxonomy, which uses three levels of structures (dimensionality, scale types, and dimensional representations) to categorize various displays such as Coxcomb, Polar Plot, Bar chart, Pie chart, Line graph, Table, Matrix, etc.

The fourth step maps between displays and tasks. Although there are no general principles for designing the best display efficient for all types of tasks, there exists a general principle that can identify correct or incorrect mappings between displays and tasks: the information perceivable from a display should exactly match the information required for the task, no more and no less. In other words, the tasks assigned to a display should be the tasks afforded by the external representations of the display and the displays assigned to a task should be the displays whose external representations support the task.

One advantage of representational analysis is it does not require empirical confirmation for each design decision. Instead, it is built on a representational taxonomy which is structured according to established empirical findings. A second advantage of representational analysis is that it offers a systematic method that can generate and evaluate alternative representations of a given dimension or relation in a systematic and organized way.

Figure 5 (above) serves to illustrate the representational analysis of the Solver display. Date, which is an interval dimension (variable), was represented by horizontal table cell positions which are on the top of the list for representing interval, ordinal, and nominal scales in the representational taxonomy. Likewise, tail numbers (nominal scale) were represented by vertical cell positions (the true tail numbers have been disguised for this paper). Aircraft status (a nominal scale) was represented by color (e.g., green for Fully Mission Capable) which is on the top of the list for nominal dimensions. The relation among Date, Tail Number, and Aircraft Status is represented by the table cells with colors (status) constructed by the horizontal (Date) and the vertical (Tail Number) positions. Note that the tabular representation in Figure 5 is only one of many representations for the Solver problem and it is a good representation for basic tasks such as finding out which aircraft is available for which assignment for which date or checking the status of an aircraft for a specific period of time. For other tasks, alternative representations might be more appropriate than the tabular display. For example, to find out which


754

aircraft is closest for a specific scheduled maintenance or to find out the engine hour distributions of the aircraft in a squadron, a bar chart is a better representation than the tabular display in Figure 5. This illustrates the utility of representational analysis for generating alternative representations for the components in a work ontology and mapping appropriate representations to specific tasks.

Evaluation with User Testing The alpha version of Solver was subjected to summative user testing to verify that the Solver’s software and qualified users could execute the TLA to successfully create a typical monthly schedule, deal with boundary cases, handle exceptions to policy, and revise it for aircraft groundings.

Six maintenance officers each developed a schedule using Solver to satisfy actual mission and maintenance requirements that were based on typical historical data. After viewing a 6-minute training video all 6 officers performed all the sub-tasks to produce a complete schedule and revise it without errors. The unassisted task completion rate was 98%, the assisted task completion rate was 100%. Mean time to develop a complete schedule that satisfied all mission and maintenance requirements was 8.03 minutes, with a 95% confidence interval of 4.6 to 11.4 minutes. The previous system required a team of 3 people working for 3 days.

DISCUSSION Solver is being prepared for wide-scale deployment. Our WCD method succeeded on this complex application where far more expensive attempts failed from inadequate flexibility for the way human schedulers need to interact with computing to handle exceptions. The demonstrated labor reduction alone would justify Solver’s cost. The results of business process simulations also show significant downstream time and labor savings from a digital schedule. Even greater value, though, is expected from better quality schedules. A conservative estimate of a 4-5% increase in fleet-wide scheduling efficiency will be worth hundreds of millions of dollars annually.

Comparison to Other High-Level Design Methods Function allocation was addressed earlier by static guidelines [18]. Static guidelines can be too general, and difficult to interpret for dynamic interactions. WCD models interactions in a way that is most similar to Dowell & Long, who proposed that interactive applications should be viewed as joint human-machine work systems. Dowell also developed an impressive mathematical definition of the work domain of aircraft controllers [7]. Ontology can handle work problems that are difficult to model with math.

WCD and several other approaches, such as cognitive systems engineering [19], cognitive work analysis [22], contextual design [11], and scenario-based design [3],

address the same basic problem of system design at a higher level than basic interface usability. Space limitations preclude an extended discussion, but the similarities and differences between WCD and these other approaches can be briefly summarized.

Our use of ontology to model scheduling was informed by, but distinct from ontology for factory scheduling by Smith & Becker [21]. It was inspired by Constantine & Lockwood [5], Vicente [22], Rasmussen [19], and Jacobson, et al. [15], who all advocate high-level work models to analyze application requirements. Ontology offers a new technique that is declarative, as opposed to procedural models. Work ontology is suitably abstract, and more rigorous and formal than class diagrams in UML. Solver’s ontology used relational algebra [Codd 4].

The focus on the TLA as the primary goal of initial design is the other major source of similarities and differences with earlier approaches. First, the TLA is a kind of task analysis, but one in which the machine's task and the human's task are both represented. As a design artifact it most resembles the job process chart form of partitioned operational sequence diagrams [Kirwan & Ainsworth]. These also show the sequence of activities done jointly by interacting human and machine. WCD shares a concern for the work context considered as a whole with Contextual Design [10], but the goal of designing the TLA is concurrent engineering of both machine procedures and user procedures, as opposed to focusing on the user side of the interaction.

WCD is a different paradigm from usability engineering [1] where the primary artifact is a UI design and user procedures are not known until it has many details. This treats user procedures as a response to the UI. But WCD makes user procedures a part of the primary design and the UI becomes the means to achieve the user procedures. Similarly, WCD differs from the approach advocated in cognitive work analysis [20], where users are supposed to "complete the design" by creating their procedures for using the UI.

WCD's emphasis on joint design of machine procedures and user procedures offers a direct connection to the software engineering concept of use-cases [13], which despite its obvious relation to usability, has yet to be integrated with usable design.

TLAs offer another solution to the dilemma of the task-artifact cycle [3], in which the designed artifact alters the user's task. Carroll et al. [3] presented scenario-based design as a solution for showing how the user's task would be adapted to suit the artifact and vice-versa. The TLA shows this same information, but in more general, abstract, and rigorous specification of how the user and the artifact interact to produce the work entity.


755

Limitations of WCD & Research Directions WCD may be controversial. It focuses on user procedures before the UI is considered, and reflects a very deterministic design philosophy. Also, issues such as scalability and generality are common with new design methods, and it will take more experience to address them.

More research and experience is needed to model a work domain with ontology. We evaluated our scheduling ontology largely from the design insight it provided, which was not known until we were already confident the design was powerful. Guidelines for work ontology modeling are needed to reduce the risk of depending on this technique.

Better tools are needed to model TLAs at the level of logical functions. Although Solver’s TLA was captured in a table (figure 4) it was not a good way to create or evaluate it. HL-GOMS models seem much better for developing TLAs. Our longer term goal is to integrate TLA modeling with algorithm proving and cognitive performance modeling. We do recommend WCD for similar design problems as Solver, where technical problem-solving is the domain, human users are required, and the stakes are too high to leave function allocation until late in the project.

ACKNOWLEDGEMENT This research was supported by USAF/AFMC Air Force Research Lab contract F33615-03-2-6315, Work-Centered Support Systems for Autonomic Logistics, POC: Capt. Mike Blake (937) 255-8049. We thank Dr. Bob Eggleston for insightful questions and valuable discussion.

REFERENCES 1. Butler, K.A. (1996) Usability Engineering Turns 10.

Interactions, 1, pp. 59-75. 2. Butler, K.A.; Esposito, C.; & Hebron, R. (1999)

Connecting the Design of Software to the Design of Work. Communications of the ACM, 42(1), January, pp. 38-46.

3. Carroll, J.M. Rosson, M.B. (1992) Getting around the task-artifact cycle: how to make claims and design by scenario. ACM Transactions on Information Systems. 10, 181 - 212.

4. Codd, E.F. (1972) Relational Completeness of Database Sublanguages. In: R. Rustin (Ed.) Data Base Systems. Courant Computer Science Series, vol. 6, Prentice-Hall.

5. Constantine, L. & Lockwood, L.A. Software for Use. ACM Press, 1999.

6. Dowell, J., & Long, J. (1989). Towards a conception for an engineering discipline of human factors. Ergonomics, 32, 1513-1535.

7. Dowell, J. (1998) Formulating the Cognitive Design Problem of Air Traffic Management. International

Journal of Human-Computer Studies. 49(5), pp. 743-766.

8. Gruber, T. (1993). A translation approach to portable ontologies. Knowledge Acquisition, 5(2), 199-220.

9. Gray, W.D. & Fu, W.T. (2004). Soft constraints in interactive behavior: The case of ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head. Cognitive Science, 28(3): 359–382.

10. Goransson, B.; Lind, M.; Pettersson, E.; Sandblad, B.; & Schwalbe, P. (1987) The interface is often not the problem. In: Proceedings of CHI+GI 1987. New York: ACM.

11. Holtzblatt, K. & Beyer, H. Contextual Design. Morgan-Kaufmann, 1998.

12. Kelton, WD.; Sadowski, RP.; & Sturrock, DT. (1998) Simulation with Arena. Morgan-Kaufmann.

13. Kirwan, B. & Ainsworth, L.K. (1992) A Guide to Task Analysis. London: Taylor & Francis.

14. Kieras, D. E. (2004). Task analysis and the design of functionality. In A. Tucker (Ed.) The Computer Science and Engineering Handbook. Boca Raton, CRC Inc. 46-1 through 46-25.

15. Jacobson, I., Christerson, M., Jonsson, P., & Overgaard, G. (1992). Object-oriented software engineering - a use case driven approach. Reading, MA: Addison-Wesley.

16. Landauer, T. (1995) The Trouble with Computers: Usefulness, Usability, and Productivity. Cambridge, MA: MIT Press.

17. Mayer, R.J.; Cullinane, T.P.; deWitte, P.S.; Knappenberger, W.B.; Perakath, B.; & Wells, M.S. Information Integration for Concurrent Engineering (IICE) IDEF3 Process Description Capture Method Report. Armstrong Laboratory Report AL-TR-1992-057, May, 1992.

18. Parasuraman, R. Sheridan, T.B. Wickens, C.D. (2000) A model for types and levels of human interaction with automation. IEEE Trans. Systems, Man, and Cybernetics, Part A, vol. 30 (May) issue 3.

19. Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive Systems Engineering. New York: Wiley & Sons.

20. Schrijver, A. (1986) Theory of Linear and Integer Programming. New York: Wiley & Sons.

21. Smith, S. F., & Becker, M. A. (1997). An ontology for constructing scheduling systems: Carnegie Mellon University Robotics Institute.

22. Vicente, K. J. (1999). Cognitive Work Analysis. Mahwah, NJ: Erlbaum.

23. Zhang, J. & Norman, D. (1994) Representations in distributed cognitive tasks. Cognitive Science, 18, pp. 87-122.

24. Zhang J. (1996). A representational analysis of relational information displays. International Journal of Human-Computer Studies, 45, 59-7


756

Documents

Work-centered design