Measuring Information Architecture Quality:

Prove It (or Not)!Panel

Gary Marchionini

University of North Carolina Chapel Hill

CHI 2001

Seattle, Washington

April 4, 2001

My Position: It Depends

• Some parts of an IA can be measured, the whole IA can only be evaluated qualitatively (i.e., considering context and different human interpretations)

• Consider:– Granularity– Tasks– People

Granularity• Some fine grains of architecture can be measured

quantitatively:– Checklist compliances (e.g., ADA, consistent layout,

reading levels, vocabulary control, button placement) – Average path length for common tasks– Post hoc aggregate performance (e.g., hits, links)

• Other fine grains can only be classified, not ordered (e.g., aesthetics)

• Overall architecture is a quality that emerges from fine grains. It cannot be directly measured.

User Tasks: Some Measurable, Some Not

• Retrieve/Buy/Print/Verify– Few discrete acts– Clear understanding of progress and stopping

conditions

• Explore/Browse/Read/Learn– Many continuous acts– Emergent understandings

People

• Performance and Satisfaction measures are useful by degree of:– Population range

• It is easier to measure homogeneous groups

– Population expectations• It is easier to measure consistent tasks and settings

– People-technology changes• It is very difficult to measure dynamics of

population literacies and technical change

Bottom Line

• We can expect to measure IAs on some criteria, for some tasks, for some population, for some point in time.

• We can expect people to make judgments about overall quality of IA. These judgments will be rooted in personal preferences and current culture as much as measurable criteria.