© 1999 Macmillan Magazines Ltd

Neville A. Stanton and Mark S. Young

Engineers make things that are useful to peo-ple. In collaboration with designers, ergono-mists make things that are usable by people.The concept of usability means making arte-facts easy, efficient and comfortable to use(anything from a corkscrew1 to a controlroom in a nuclear power station2). Mostpeople have experience of poorly designedobjects. At best they cause frustration andannoyance (for example when a videorecorder fails to record your favourite pro-gramme). At worst they can lead to injury oreven death (as in the release of radioactivematerial from a nuclear reactor).

‘User friendly’, ‘easy to use’ and‘ergonomically designed’ are commonlyfound claims for domestic appliances anddevices, particularly in the advertising busi-ness. For instance, an advertisement for awell-known music system boasts that it is‘Technologically advanced, yet remarkablyeasy to use’. But only a few people — includ-ing some, but not all, of those behind suchadvertisements — know what is involved inan ergonomic assessment.

Ergonomists do more than simply designchairs (a common misconception). Ratherthey are involved in various aspects of thedesign and evaluation of a host of products.They work on the specification of userrequirements, the development of designguidelines, the evaluation of device proto-types, and the implementation and analysisof user trials. Traditionally, ergonomics hashad most influence on the physical charac-teristics of design. For example, in conceiv-ing of a new radio-cassette machine, ergono-mists might advise on the buttons: their sizeand placing, and the reach and force requiredto activate them. The ergonomist draws on asubstantial body of research findings andhard data, such as human anthropometrics— tables of data that contain informationabout the range of human physical attributesand characteristics3.

Nowadays, ergonomists also advise oninteractive devices, such as so-called WIMP(Windows, Icons, Menus and Pointers)interfaces. Advice includes the form that theinterface should take, relating the groupingof functions and device elements to the goalsof the user, predicting user satisfaction, andestimating task performance time and theerror potential of the device. These analysesare used to select from a choice of alternativedesigns and to improve the chosen option.But analyses of human cognitive functioningdo not rest upon hard data to the same ex-

tent as anthropometrics4. Questioning thesetechniques is our purpose here.

Despite the proliferation of ergonomicmethods in research5, teaching6 and industry7,there is little substantive evidence that thesemethods actually work8. Part of the reason isthat researchers tend to stick to one or two thatthey know, trust and, in many cases, have devel-oped themselves. The main point here, howev-er, is that methods for predicting the usability ofdevices have become so entrenched that theirvalidity is simply assumed and seldom tested.These methods are used in good faith, in thebelief that the results are reliable and valid. Butthat assumption is questionable, and we findthat some approaches do not do all that isclaimed for them.

Reliability and validityThe objective way to see whether ergonomicmethods work is to assess their reliability andvalidity. Methods can be reliable withoutbeing valid, but they cannot be valid withoutbeing reliable. We can liken reliability andvalidity to the accuracy of a rifle marksmanusing a gunsight. Reliability refers to thegrouping of the shots (the error of the marks-man), whereas validity refers to the closenessof each shot to the centre of the target (theerror of the gunsight). If ergonomic methods

are shown to be adequately reliable and valid,they may be used with increased confidence.

In a series of studies, we took ten of themore popular ergonomic methods, and sub-jected them to tests of reliability and validity(details of the methods are found in ref. 1,and ergonomic textbooks). The ergonomicmethods each predicted different aspects ofusability, from objective performance mea-sures (such as reaction time and errors)through to subjective criteria (such as usersatisfaction). The methods evaluatedincluded classical approaches (such as inter-views and observation), as well as some of themore modern techniques (such as SHERPA,the Systematic Human Error Reduction andPrediction Approach). Only a few peoplewere involved in the study: eight were trainedin each of the methods and 30 were observedusing a radio-cassette machine, the devicewe chose to use. But these numbers are realis-tic in terms of the kind of studies carried outby ergonomists. Our analyses producedbetween 500 and 1,650 data points for eachmethod, depending upon the method used,which were classified as hits, misses, falsealarms and correct rejections.

The results, which appear in Table 1,show huge variability in both reliability andvalidity of the various methods. The validity

NATURE | VOL 399 | 20 MAY 1999 | www.nature.com 197

What price ergonomics?Ergonomists have a say in the design of almost everything in the modern world, but there is little evidence thattheir methods actually work. Here is an evaluation of those methods and of the worth of ergonomics in design.

commentary

DAV

ID N

EW

TO

N

© 1999 Macmillan Magazines Ltd

statistics were calculated using either Pear-son or Phi (a correlation coefficient fordichotomous data). For the latter PhiMax isalso given, as this shows the maximum valuethat could be achieved, given the marginaldistributions9. Interestingly, two of themethods return small but negative correla-tions, meaning that the predicted behaviourwas the opposite of that observed.

So is it worth using these methods at all?Thus far in the argument, the answer has toremain ‘perhaps’. The reason is that ergono-mists not only have an agenda of making life inour technological world easier, safer and moreenjoyable, but also that of helping to maxi-mize manufacturer revenue and minimizecosts. Designing a product in accordance withusability principles can increase profits; and,given the sums of money involved in productdevelopment, even methods with relativelylow validity could offer returns.

Methods that can predict good or baddesigns should save time and money in prod-uct development. They might increase salesand profits, and reduce need for training andafter-sales support. But those benefits haveto be set against the costs of setting up anergonomics laboratory, conducting themethods and evaluating iterative proto-types. So does the relative inaccuracy of themethods reduce these benefits beyond thepoint where they are worth the effort andcost? That is, does it cost more to use amethod than is gained from the end result?

Cost–benefit analysisClaims as to the cost-effectiveness ofergonomic methods normally centre on theresults of cost–benefit analysis. Here theapproach is to calculate the cost of applyingthe method (in terms of person-hours,materials and so on) and subtract this fromthe estimated savings generated by theimprovement in design. The net figure isproposed as the benefit brought about byusing ergonomic methods.

The figures can look good. According toone textbook example10, the cost of a usabili-ty assessment of the interface for new com-puter software (US$171,445) compares verywell to the potential savings of the new

design ($592,635) in terms of the reducedcosts of training and after-sales support inthe first year. On the face of it, then, in thisinstance ergonomic methods provide aprofit of more than $400,000.

The estimate of likely economic benefitsdoes not however attempt to take the accuracyof the ergonomic methods into account. Theapplication of ergonomic methods (if inaccu-rate) could fail to predict true problems withdevice design or, worse still, identify problemsthat do not exist. We argue that the degree towhich the financial benefits will be realizedwill depend upon the accuracy of the methodsused. Using the reliability and validity data weacquired, the net benefit of each method canbe calculated from its utility U (that is, itspractical usefulness couched in financialterms): U 4 (validity 2 prospective increasein value) 1 costs. A recalculation of the text-book cost–benefit analysis would find a sub-stantial reduction in the benefit by includingthe validity data. It is therefore essential thatthese data are considered when addressingutility, as they show how much variability canbe accounted for in ergonomic methods.

The practical context here is that, in prod-uct design, a manufacturer is typically facedwith the prospect of choosing between ten ormore designs. Given this degree of choice,how are they to arrive at the best decision?They could flip a coin or rely upon intuition.Alternatively, they could use the methods topredict differences in user performance toselect the best design. The difference betweena good and poor design could be worth mil-lions of dollars to a manufacturer over thelifetime of a product.

For the purpose of testing the formulawith the computer software example10, wecan estimate that choosing a good designover an average design will increase prof-itability by upwards of $500,000 per year (arounding down of the over-precise figureprovided by the example). Similarly, we use$100,000 as the cost of making prototypesand assessing the device. The same costs areused in both analyses to show the effect ofvalidity on utility. In our analysis, themethod with the lowest validity value wasuse of repertory grids; that with the highest

was the keystroke level model (KLM). Thecontrast of utilities brought about by thesetwo methods, with respective validity valuesof 0.078 and 0.769, can be estimated as:U(repertory grids) 4 (0.078 2 $500,000) 1$100,000 4 1$61,000; U(KLM) 4 (0.7692 $500,000) 1 $100,000 4 $284,500. Sosome methods may return negative values inthe first year, whereas others show a healthyreturn on investment.

A range of factors can affect the selectionof ergonomic methods to be applied in eval-uating specific products, including time andresources available, efficacy of the methods,criteria by which the product is to be evaluat-ed, and access to a ‘test’ sample of end-users.Utility analysis can help make some of thesejudgements more explicit and more objec-tive. It can help in the selection of methods,show where effort may be best expended,and make the design process more effective.Experience from the field of personnel selec-tion shows that for utility analysis to be effec-tive it must be easy to use and the derivationmust be transparent — people using the for-mula must be able to see how it works andcarry out the calculations without having aPhD in mathematics.

We can provide the reliability and validitydata, the costs of conducting each method,and the appropriate procedure for conduct-ing a cost–benefit analysis. Product develop-ers will have their own budgets for sales andtraining, and these will differ depending onwhether the product is a vending machine ora video recorder. So it is up to the productdevelopment teams concerned to make theirown calculations and decide whetherergonomic methods are actually worth using.

Our aim here has not been to condemnergonomic methods or ergonomics as a dis-cipline. Rather, our take-home message isthat, if ergonomists are really to improve thedesign of artefacts, their methods have to beshown to be both useful and usable.Neville A. Stanton and Mark S. Young are in theEngineering Psychology Research Group,Department of Psychology, University ofSouthampton, Southampton SO17 1BJ, UK. e-mail: n.a.stanton@soton.ac.uk1. Stanton, N. A. Human Factors in Consumer Products (Taylor &

Francis, London, 1998).

2. Stanton, N. A. Human Factors in Nuclear Safety (Taylor &

Francis, London, 1996).

3. Pheasant, S. T. Bodyspace — Anthropometry, Ergonomics and

Design (Taylor & Francis, London, 1986).

4. Dowell, J. & Long, J. T. Ergonomics 41, 126–139 (1998).

5. Salvendy, G. Handbook of Human Factors and Ergonomics, 2nd

edn (Wiley, New York, 1997).

6. Wilson, J. & Corlett, N. Evaluation of Human Work, 2nd edn

(Taylor & Francis, London, 1995).

7. Jordan, P. W., Thomas, B., Weerdmeester, B. A. & McClelland, I. L.

Usability Evaluation in Industry (Taylor & Francis, London, 1996).

8. Stanton, N. A. & Young, M. Appl. Ergonomics 29, 41–54 (1998).

9. Guilford, J. P. Psychometric Methods (McGraw-Hill, New York, 1957).

10.Bias, R. G. & Meyhew, D. J. Cost-Justifying Usability (Academic,

Boston, 1994).

Acknowledgements. The research reported here was supported by

the EPSRC under the LINK transport initiative. We thank S. F.

Blinkhorn for advice on the content and form of this article.

commentary

198 NATURE | VOL 399 | 20 MAY 1999 | www.nature.com

Table 1 Reliability and validity of ergonomic methods

Ergonomic method Reliability Validity PhiMax

CChheecckklliissttss 0.307 0.206* 0.659

HHeeuurriissttiiccss 0.471 0.087* 0.464

HHiieerraarrcchhiiccaall ttaasskk aannaallyyssiiss 0.226 0.188* 0.937

IInntteerrvviieewwss 0.449 10.112* 0.422

KKeeyyssttrrookkee lleevveell mmooddeell 0.916 0.769* N/A

LLiinnkk aannaallyyssiiss 0.830 0.356* 0.572

OObbsseerrvvaattiioonn —— eerrrroorrss 0.890 10.141* 0.343

OObbsseerrvvaattiioonn —— ttiimmee 0.890 0.729* N/A

QQuueessttiioonnnnaaiirreess 0.578 0.615* N/A

RReeppeerrttoorryy ggrriiddss 0.562 0.078 0.681

SSHHEERRPPAA 0.392 0.238* 0.922

Reliability was calculated by Pearson's product moment correlation coefficient. Validity was calculated by Phi (thosewith associated PhiMax values) and Pearson's product moment correlation coefficient. Asterisk indicates that validityvalue was statistically significant.