Université Toulouse I 1 CADUI'96 - 5-7 June 1996 - FUNDP Namur Automatic Ergonomic Evaluation : What are the Limits ? Christelle FARENC *, Véronique LIBERATI

Université Toulouse I 1 CADUI'96 - 5-7 June 1996 - FUNDP Namur LIS

Automatic Ergonomic Evaluation :Automatic Ergonomic Evaluation : What are the Limits ? What are the Limits ?

Christelle FARENC *, Véronique LIBERATI **, Marie-France BARTHET *

* LIS - University of Toulouse (France)** Post Office Technical Research Unit (France)


IntroductionIntroduction

• Automatic evaluation tool– for non- expert

– based on ergonomic rules

• Principles– Evaluation : comparaison between observed values and

reference values

– Automatic : evaluation without the help of the user

• How far can evaluation computerization go ?Study from Ergoval


• Evaluation of any UI– in the Windows environment,

– independently of the developement tool used,

– at different stages of the development cycle,

• Ergonomic rules– lexical, syntaxic, semantic and pragmatic levels

– Vanderdonckt, Scapin, MICE/D

Characteristics of ErgovalCharacteristics of Ergoval


Steps of the studySteps of the studyErgonomic rules used

Rules inherently respected

Rules that can be automated

Rules that can not be automated

Rules that can not be automated. Information not automatically retrievable

Rules that can be automated. Information automatically retrievable

Information related to items included in the application

Information related to items not included in the application

Information of the semantic type

Information of the pragmatic type

Information of the « semantics of the text » type

Information of the « semantics of the object » type

With source files

Whatever the implemented methods are

Rules need information

{{

{{nature of

informat°


Minimal Automatic EvaluationMinimal Automatic EvaluationErgonomic rules used












With source files


Rules needs information

{{

{{nature of

informat°


Minimal Automatic EvaluationMinimal Automatic Evaluation

• Recovery of information in source files• Categories

– rules directly obeyed by construction,

– rules requiring automatically recoverable information,

– rules requiring not automatically recoverable information

• Focus of the rules– Static UI presentation

– UI or system behavior


ExamplesExamples• Rules directly obeyed by construction

– /presentation : "Labels for push buttons must be centred"– /behaviour : "In a menu bar, ..., the cursor must run automatically from the last

option to the first."

• Rules requiring automatically recoverable information– /presentation : "All boxes and windows must have a title"– /behaviour : "All boxes and windows must be movable"

• Rules requiring not automatically recoverable informat°– /presentation : "For any input field, if there are any acceptable values, such

values must be displayed"– /behaviour : "If the system's response time is of between two and five seconds,

a wait pointer must be displayed"


Results Results

Rules /presentation

Rules /behavior Total

Rules inherently respected (1) 28 64 93 (22.9%)

Rules that can be automated with source files (2)

82 2 84 (20.6%)

Rules that can not be automated with source files (3)

161 69 230 (56.5%)

Total 271 135 406


Ergonomic rules used












With source files


Rules need information

{{

{{nature of

informat°

Maximum Automatic EvaluationMaximum Automatic Evaluation


• Recovery of information whatever the technical resources used

• Focus of the informations– on elements in the application

– on elements not in the application

• Categories– rules requiring information automatically retrievable,

– rules requiring information not automatically retrievable.

Maximum Automatic EvaluationMaximum Automatic Evaluation


ExamplesExamples

• Rules requiring information automatically retrievable– on elements in the application

• "if there are acceptable values within the system, then they must be displayed"

• Rules requiring information not automatically retrievable– on elements in the application and other not

• "if there are codes in an input field literal, then these codes must be known to the user"


ResultsResults

Rules Total

Elements in the application 140 62 164

Elements not in the application 0 66 66

161 69 230Total (60.87 %) (39.13 %)

Information automaticallyretrievable

Information notautomaticallyretrievable


Qualitative analysis of informationsQualitative analysis of informationsErgonomic rules used












With source files


Rules needs information

{{

{{nature of

informat°


• Information on elements not in the application– semantic,

– pragmatic.

• Information on elements in the application– semantic of displayed text,

– semantic of graphical objects.

Qualitative analysis of the information not Qualitative analysis of the information not automatically recoverableautomatically recoverable


ExamplesExamples

• Information on elements not in the application– semantic : "If a literal or title contains an abbreviation, such an

abbreviation must comply with abbreviation norms."– pragmatic : "If a literal or title contains a code, the meaning of

this code must be known to the user"

• Information on elements in the application– semantic of displayed text : "If a text message signals an error,

it must contain an explanation of the cause of that error"– semantic of graphical objects : "A list box literal must be

presented above the object that it designates"


ResultsResultsInformation about elements in the application

Information about elements not in the application

Total

Semantic 21

Pragmatic 45

66

Total

Semantic of displayed text 39

Semantic of graphical objects 23

62


ResultsResultsInformation about elements in the application

Information about elements not in the application

Total

Semantic 21

Pragmatic 45

66

Total

Semantic of displayed text 39

Semantic of graphical objects 23

62


ConclusionConclusion

MIN44 %

MAX78 %0 100%

ERGOVALknowledge base

?


ConclusionConclusion

• Advantages of automatic evaluation– systematic verification

– help to the designer

– a useful preliminary to tests with users

• Ideal tool of evaluation– executing some rules automatically

– cooperating with a human operator to execute others


Thank you for your attention!

Documents

Université Toulouse I 1 CADUI'96 - 5-7 June 1996 - FUNDP Namur Automatic Ergonomic Evaluation : What are the Limits ? Christelle FARENC *, Véronique LIBERATI