Defining and Evaluating the Usability of CMS - Saurabh Kudesia

1 © Nokia Siemens Networks 2011

Defining and Evaluating Usability Criteria for Content Management Systems

Saurabh KudesiaCustomer Documentation ManagerNokia Siemens Networks, BangaloreAugust 2011

Presented at STC India Usability Conference 2011, Bangalore, India


About the Speaker

• Customer Documentation Manager, Nokia Siemens Networks, Bangalore• Co-founder, former editor-in-chief, KnowGenesis International Journal for Technical

Communication (IJTC) • Former associate editor of Directives, a newsletter published by the (STC) Management SIG• Reviewer: ISO/IEC 25612 standards for International Standards Organization (ISO) • Bachelor of Electronics and a Certified Scrum Master (CSM)• Alumnus of Symbiosis Institute (Pune) and IIM-B• Part of Author-it DevHub Community


In this Session

Generic Content Management Framework (CMF) and Usability Perspectives

Usability Testing Tactics for CMS

Usability Measurement Methodology


Content Management is Universal

The information viewpoint applies at different levels

Micro Macro


Content Management Perspectives

Content

Business Perspective

Organizational Perspective

Content Perspective

Process Perspective

Technical Perspective

Distributes Business Value

Balances Organizational Forces

Combines Content-related disciplines

Collects/Manages/Publishes/Distributes Information

Technical Infrastructure


Content Management Framework (CMF) Content/Process View

Print

Electronic Web

Syndication

Collection System Management System Publication System

Impacts publication designs

Impacts component designs

Impacts access structure designs


Access

Quality Assurance

R&D

Support Teams

Auditors

Customers

Interface

Relational Links

Formats/Standards

Format Conversion

Access Attributes

Version Tracking

Content Management Framework (CMF) Technical View

Tools

Automatic Notifications

Feedback/Issue Reporting

Advanced Search

Workflow

Analysis

Design

Development

Implementation

Evaluation

Auditing

Issue Categorization/Tracking

Audit Data

Usage/Performance Statistics

Security

Configuration and Administration


Usability Mea Culpa (or “That’s Not Architectural”)

“Making a system’s user interface clear and easy to use is primarily a matter of getting the details of a user’s interaction correct…but these details are not architectural”- Bass, Clements, and Kazman [Software Architecture in Practice, first edition]


The ‘Planes’ of User Experience

Surface

Skeleton

Scope

Structure

Strategy Strategy is where it all begins: What do we want to get our of the system? What do our users want?

Scope transforms strategy into requirements: What features will the system need to include?

Structure gives shape to scope: How will the pieces of the system fit together and behave?

Skeleton makes structure concrete: What components will enable people to use this site?

Surface brings everything together visually: What will the finished product look like?


The ‘Planes’ of User Experience

Discrete ‘Planes’ of User Experience

Requiring work on each plane to finish before work on the next can startleads to unsatisfactory results for the designers and the users.

Concurrent ‘Planes’ of User Experience

A better approach is to have work on each plane finish before work on the next can be finish.

Time

Effo

rt

TimeEf

fort

Strategy Scope Structure Skeleton Surface Strategy Scope Structure Skeleton Surface


Usability Tactics

• Maintain a model of the task• Maintain a model of the system• Maintain a model of the user

• Localize changes to user interface code• Use supported software architecture patterns like Model-View-Controller

(MVC), Presentation-Abstraction-Control (PAC)

Runtime Tactics Design time Tactics

Runtime Tactics

Separate User Interface

Support User Initiative

Support System Initiative

CancelUndo

Aggregate

User ModelSystem Model

Task Model

Localize changes to User Interface code

Design time Tactics


Quality Attributes Stimuli

Availability

Unexpected EventNon occurrence of expected event

Modifiability

FunctionalityPlatform (Portability)Quality attributeCapacity (Scalability)

Performance

PeriodicStochasticSporadic

Security

Display ServicesModify ServicesModify InformationModify AccessReduce availability to system services


Usability General Scenarios

Artifact

Source

Stimulus Response

Environment Response Measure

Source

End Users

Stimulus

Learn system featuresUse system efficientlyMinimize error impactAdapt systemFeel comfortable

Artifact

System

Environment

Run timeConfigure time

Response

Support “learn system features”Support “use system efficientlyMinimize error impactAdapt systemFeel comfortable

Response Measure

Task TimeNumber of errorsNumber of problems solvedUser satisfactionGain of user knowledgeProportion of Successful operationAmount of time/data lost


Usability Measurement Methodology

Identification of Dimensional scale, Sub constructs and item pool

Selecting the Measurement model

Measurement and Validation of Scale based on measurement model

Result analysis and estimation of usability


Usability Measurement is Fuzzy


Dimensional Scales, Sub Constructs and item pool

1. Visual Efficiency and Consistency

Consistent Interface Design labelingConsistent Interface ResponseScreen design inline with visual design principlesUnderstandable Interface language

2. Error Handling and Functional Efficiency

Easy to understand error messagesExplicit error messagesNo problems with CMS functions

3. Interface-Task Performance

Simple structure to accomplish a taskUI screens don’t include too many elements

4. Interface Familiarity

Interface resemble user’s prior experienceInterface behaves similar to what user is accustomed to

5. Interface Operation Performance

Prompt CMS responses to users’ promptsInterface design changes are localizedLess user interaction required to complete a task

6. Efficient/Flexible Access to Content

Easy interface design for accessing contentNo specialized training required to perform functionsShortcuts to access frequently used featuresMultiple options to perform the same actionCMS loads fastCMS has a flexible use

7. Efficiency of Navigation

Easy access to critical navigational buttons Navigation remains unaffected with increase in content volume

Simplicity Familiarity Consistency Visual Feedback Responsiveness Fault Tolerance Scalability


Measurement Models

Model 2: Group-Factor Model

Model 3: Essentially Unidimensional

Model 1: Strictly Uni-dimensional

X1

X2

X3

:

Xk

C

Ek

E1

E2

E3

:

Model 4: Multidimensional Structural Model


Fit Indices of Models

Goodness-of-fitness Lack-of-fitness

Models NNFI CFI RMSEA SRMR

Model 1 0.85 0.86 0.134 0.121

Model 2 0.92 0.93 0.075 0.073

Model 3 0.92 0.92 0.078 0.075

Model 4 0.92 0.93 0.075 0.062

Sub 2

Sub 6

Sub 3

Upper 1

Upper 2

Sub 1

Sub 4

Sub 7

Sub 5

0.57

0.17

0.68

0.14

0.56

0.25

0.46

0.66

0.91

0.57

0.93

0.66

0.86

0.74

1.00

1.00

-0.02

Sub 1 Sub 2 Sub 3 Sub 4 Sub 5 Sub 6 Sub 7

Sub 1 1.00

Sub 2 0.59 1.00

Sub 3 0.01 -0.01 1.00

Sub 4 0.37 0.52 -0.01 1.00

Sub 5 0.01 -0.01 0.64 -0.01 1.00

Sub 6 0.61 0.84 -0.01 0.53 -0.01 1.00

Sub 7 0.43 0.60 -0.01 0.38 -0.01 0.62 1.00

Upper 1 (ζ) 0.66 0.91 -0.02 0.57 -0.01 0.93 0.66

Upper 2 (ζ) -0.01 -0.02 0.86 -0.01 0.74 -0.02 -0.01

Covariance Matrix


Important Guidelines

• Don’t mix content presentation and architectural design; both are perceived as separate constructs by participants

• Usability scale is multidimensional and should be treated accordingly in benchmarking usability of CMS

• Consider context dependencies of User’s perceptions and functional use of CMS into usability calculation

• Generalizability and transferability of ideas across contexts, extensive collaboration is essential for the multidimensionality of usability scales


For further discussion:

Phone: +91-9620208216Email: [email protected]