Web Standards and (the future of)

User Interface Design

Stéphane Sire

(stephane.sire@epfl.ch)

École Polytechnique Fédérale de Lausanne

CGC/MEDIA

Seminario

Martedì 22 aprile 2008, h. 15:00

Aula Seminari, seminterrato

Dipartimento di Scienze dell'Informazione

Bologna

Plan

Web Standards and (the future of) User Interface Design

Part 1

Part 2

Part 3

Rich User Interaction

SVG and alike

Iterative Design and supportive technologies

Part 1

(the future) of User Interfaces

Rich User Interaction

4

Evolution of User Graphical Interaction Styles

Desktop Web Mobile

Command Line

Direct (1983)Manipulation

Point & Click(hypertext)

HTML Forms

WAP

Rich User Interaction

WIMP

Rich Desktop Application (RDA)

Rich Internet Application (RIA)

iPhone and alike (touch based)

5

Characteristics of Rich User Interaction

• Better Graphics• Extensive use of Animations • Creative layout models• Borrowings from other interaction styles• Better feedback times (AJAX)

A new vocabulary for design

6

Better Graphics (1)

• Layers– Painter's model of rendering– Compositing operators

• (Porter & Duff)

– Clipping and masking

• Applications– Layout based on layers (see later)– Non rectangular windows

• Objects with holes

– Layer's effects Source: Digistrip (CENA)

7

Better Graphics (2)

• Color models– Alpha channel (RGBA)– Gradient colors– Texture filling

• Applications– Light and shadows effects– Glossy effect– Reflection (cow.neondragon.net/stuff/reflection/)

– etc

8

• Shapes & lines– Stroke styles

• Joints• extremities

– Bézier paths• Very efficient algorithm to draw it (De Casteljeau

algorithm)

• Applications– Rounded shapes

Better graphics (3)

9

Better Graphics (4)

• Pixel Effects– Anti-aliasing

– Filters (blur, bump mapping, etc.)

Source www.corzo.com

Source www.treebuilder.de

10

Extensive use of animations

User's guidance

System state change

Visual state transition

Source www.joehewitt.com/iui/samples/music.html Source www.lemonde.fr(choose Journal Electronique)

Source www.panic.com/goods/

11

Animation Tricks

• "Exageration"

• Pace• Application: Kinetic Scrolling (video on youTube)

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

Author's made movie from www.laredoute.fr Author's made movie from www.youarethemnodel.com

12

New Layout Models (1)

• Layered information

Source www.laredoute.fr

Source www.natoora.com

Source maps.google.com

13

New Layout Models (2)

• Contraction/dilatation– Accordion

• Continuous surfaces– Linear (ribbon) : www.gucci.com

– Two dimensions : www.goruneasy.com

– Unconventional : www.etsy.com (choose Time Machine)

Source www.goruneasy.com Source www.yomiuri.co.jp

14

New Layout Models (3)

• Fixed layout + Animation– Scrolling Ribbon

• more information with constant space within the page

• Layers + Animation– Drawer windows– Docks

Source www.tf1.fr

15

Borrowings from other styles

• Direct manipulation– Drag & drop but still marginal

• Post-WIMP techniques– Bifocal menus

• Video-games– Rotative menus

Source www.amazon.com

Source www.wat.tv

16

Better Feedback Times with AJAX

• No more page reload and full page refresh– Increase reactivity– Requires special tricks to get user's attention

• Applications– Auto-suggestion input field (see Google Autosuggest)

– Dynamical queries• allows some forms without no "Submit" button

• or allows to pre-filter results before submitting

Sourcewww.trivop.com

Sourcewww.darty.com/nav/achat/telephonie/telephonie_mobile/telephone_portable/guide.html

17

Rich User Interaction (summary)

• Graphical design– To increase intuitivity– To create emotions– Emotions increase perceived usability

• See Don Norman last book "Emotional Design"

• Requires multi-disciplinary teams – User experience (interaction) designer– Visual designer, Motion designer– Developers– etc

(video on youTube)

Part 2

User Interface Design

Iterative Design and supportive technologies

19

Product Development Process

Reprinted from "Sketching User Experiences" (Buxton, 2007)

20

Iterative Design

21

Close to Agile Software Development

• Agile Manigesto (source agilemanifesto.org)

– Individuals and interactions over processes and tools

– Working software over comprehensive documentation

– Customer collaboration over contract negotiation – Responding to change over following a plan

• See also Extreme Programming• Web version

– the permanent Beta

22

Examples

Source: Nielsen

Source: Hiser

Source: Alex Poole

23

Example from

• Air Traffic Control Strip– DMAN project

24

Classical development cycle

Users

SketchesStoryboardWireframesIA diagrams

Pro

gram

me

rs

Interaction,Visual,Motion,…Designers

ExecutableCode

BrowserPluginRuntime Env.

Feedback loop

How to increase the number of iterations ?

25

Model Driven Development Cycle

Users

SketchesStoryboardWireframesIA diagrams Programmers

Interaction,Visual,Motion,…Designers

Graphical modelsAnimation modelsOther models…

Browser, PluginRuntime Env.

Executable code

26

What is a model ?

• Example from

27

How to represent a model ?

• eXtensible Markup Language (XML)

<g id="Root"><g id="Back">

<path id="Background"><g id="Zones"><g id="CatchZones">

<text id="Callsign"><text id="RWY"><text id="Gate"><path id="Catch">

<g id="DMAN"><text id="STA">

28

How to Edit a Model ?

• Specialized editors – Example for graphics: Adobe illustrator

SVG File

29

A short Example

Interaction Designer

Visual Designer

Programmer:LET's DO IT !

Graphical ModelFile(s)

Example from Adobe dev web site

30

Adding More Models….

Interaction Designer

Visual Designer

Graphical Model

Behavior Modelling with State Machines

Finite State Machine Models

31

Example from

Visual Design

Paper Mockup

Prototyping

Integration

Airnet Project

32

Model Driven Summary

Models

Models(XML)

CodeCode

• Models– Specialized editors– Easy to move from

one platform to another

• Code– Programming skills– Hard to maintain– Less portable

Part 3

Web Standards

for User Interface Design

SVG and alike

34

W3C standards (1)

• Mainly "Models" – XML based languages– From "Draft" to "Recommendation" status– Models both for output and input modalities– HTML, XHTML, XForms, SMIL, MathML, VoiceXML,

SCXML, InkML, EMMA, …, and SVG

• Some "Code" too– Application Programming Interfaces (APIs)– DOM (3 levels)

• API for manipulating models loaded into memory as trees• Many more APIs (events, load and save, progression, etc.)

– XMLHttpRequest• Ajax programming

35

Web Standards and MDA

Markup Only

Code Behind

Markup Behind Code Only

XML

XML

ImperativeProg. Lang.

Imperative Prog. Lang.

Definition of Interactive ComponentsInstantiation

of Intera

ctive C

omp

one

nts

Today Web applications

Future of applications

Today Desktop applications

36

W3C and Rich User Interaction

(*) simplified with specialized toolkits (scriptaculous, jQuery, etc)

Characteristics Standard solution

Better graphics SVG or CSS3 (with HTML)

Animation SMIL/SVG animation module

DOM Manipulation with Javascript(*)

Creative layout SVG or CSS (with HTML)

DOM Manipulation with Javascript(*)

Better feedback time XMLHttpRequest

Cross domain XHR

Borrowing from other styles All of the above

37

SVG History

• Scalable Vector Graphic• Started in 1998• 1st draft: February 1999• SVG 1.1 recommendation 2003

– http://www.w3.org/TR/SVG11/

• SVG 1.2 Tiny candidate recommendation 2006– http://www.w3.org/TR/SVGMobile12/– SVG 1.2 Full to come later

• Working Group with main software editors– Sun, Adobe, Apple, Microsoft, Corel, Ilog, HP, Canon,

Autodesk, etc.

38

SVG vs. Flash

Property SWF SVG

XML - Std W3C No Yes

Metadata Yes, propietary Yes, RDF compatible

Format Binary Text

Size < SVG > SWF : but can be compressed with gzip

by about 80% (*)

Runtime Plugin (200K) Native (Opera, Safari, Firefox, …)

Source No Yes

Audio/Video Yes Yes (SVG 1.2 Tiny)

(*) see http://www.w3.org/TR/SVG/minimize.html

39

Authoring Tools

• Graphical Editors– Sodipodi, Illustrator, Corel Draw, OmniGraffle Pro, …

• Specialized Editors– Ikivo Animator : animations

• Automatic generation – Client-side Javascript generation

• Example from http://www.destatis.de/

– Client or server size XSLT generation• demo from www.treebuilder.de/default.asp?file=250484.xml

• Automatic conversion from/to other formats– From MathML, AutoCAD, Visio, etc.– To PDF, PNG, etc.

• Manual authoring :)

40

Players

• Adobe to stop its plugin support in 2009…– Adobe to switch to Flash/AIR ?

• … but becoming native in Web browsers– Opera, Safari, Firefox, …– open source rendering libraries (Cairo)

• Not directly available in IE…– Microsoft to switch to XAML/WVG with silverlight plugin

• Standalone players– Batik (Java) - contains a useful utility to generate SVG fonts

• SVG Tiny 1.2 on mobile phones– www.bitflash.com/mobile_primer.html– www.ikivo.com/02player_svg.html

– And more…• See list of implementations www.svgi.org/

41

Drawing Basic Shapes

• Shapes– Rectangles <rect> – Cercles <circle>– Ellipses <ellipse> – Lines <line>– Polylines <polyline>– Polygons <polygon> – Text <text>

• Attributes– x, y, width, height, rx, ry– fill, stroke, stroke-

linecap, stroke-linejoin, stroke-miterlimit, stroke-width, …

– opacity, stroke-opacity, fill-opacity

– ...

<rect x="200" y="300" width="90" height="40"

stroke-linejoin="bevel" stroke-width="3"/>

fill="lime" stroke="blue"

42

Drawing Arbitrary Shapes

• Path <path d="path data" >– outline of a shape which can be filled, stroked,

used as a clipping path – defined by connected lignes, arcs and curves.

• Base attribute– "path data" : complex vocabulary – "," and EOL are allowed in "path data", spaces are

optional between a command and a number (to compress data)

<path d="M 100 100 L 300 100 L 200 300 z"

fill="red" stroke="blue" stroke-width="1" />

43

Path data vocabulary

• Each command is named by a letter (M: moveto L: lineto, etc.) followed by a coordinate, either

– Absolute if capital (e.g. "M")– Relative to the previous point if lowercase (e.g. "m")

• M and m:– Move the point to the given coordinate– Create a new sub-path starting at the origin– Syntax: M|m (x y) (x y)*– The second group (x y)* is interpreted as lineto commands– Ex: M100 100 200 200

• L and l:– Draw a line from the current point to the given coordinate and draw lines

between subsequent given points– Syntax: L|l (x y) (x y)*

• H and h (V and v):– Draw a horizontal line (vertical) from the current point to the current point

incremented with the given number– Syntax: H|h (x) et V|v (y)

• Z and z:– Close current subpath with the last point defined with M|m

M 100,100

L 200, 200

x

Cur point

44

More path data vocabulary

Command Args (can be repeated)+ Description

A, a rx ry x-axis-rotation direction sweep x y

Elliptical arc

C, c x1 y1 x2 y2 x y Cubic Bézier curve from current point to x y with control points x1 y2 and x2 y2

S, s x2 y2 x y Cubic Bézier curve with control point as the reflection of the reflection of second control point of a C, c command

Q, q x1 y1 x y Quadratic Bézier curve from current point to x y with control point x1 y1

T, t x y Quadratic Bézier curve with control point as the reflection of previous control point of a Q, q command

See details on http://www.w3.org/TR/SVG/paths.html

45

Defining reusable graphical components

• Definition of a component– Group element: <g id="name">– Container element for grouping graphics elements– Can be nested– Its attributes are inherited by its descendants

• Instantiation of a component– <use xlink:href="#name"/>

<g id="shape" stroke="blue">

stroke-width="0.1cm" fill="none"/>

stroke-width="0.1cm"/>

<use xlink:href="#shape" x="70" y="60"/><use xlink:href="#shape" x="140" y="20"/></g> <rect x="10" y="50" width="50" height="1cm" <path width="1cm" d="M 0 50 35 25 70 50 Z"

46

Declaring graphical components without drawing them

• Everything that appears in a <defs> section will not be drawn– It will be available to be reused in other parts– This is a way to group resources (at the beginning

of the file)

<defs> <g id="shape" stroke="blue">

stroke-width="0.1cm" fill="none"/>

stroke-width="0.1cm"/>

</defs>

<use xlink:href="#shape" x="70" y="60"/><use xlink:href="#shape" x="140" y="20"/> </g> <rect x="10" y="50" width="50" height="1cm" <path width="1cm" d="M 0 50 35 25 70 50 Z"

47

Positioning graphics

• Most elements can be positioned with x and y within their parent container

• Groups <g> are also used to apply transformations– <g transform="translate(x,y)">– Other transformations include rotate, skewX|Y, scale and matrix– This is equivalent to associating a transformation matrix with each

group <g> (and to redefine a new coordinate system for the descendants)

• Transformation are cumulative in nested groups– Current transformation matrix at each <g>– Obtained as the product of all transformation matrices up to the

parent <g>

<g transform="rotate(20, 200, 300)">

fill="lime" stroke="blue" stroke-width="3"/>

</g><rect x="200" y="300" width="90" height="40"

48

Painting and Coloring

• Colors applies with different attributes to– Strokes (attribute "stroke")

– Shapes interior (attribute "fill")

• Colors are rendered by a paint server– The attribute URI reference selects a paint server

– Solid colors (e.g. "orange" or "#50A619")

– Gradients or Patterns (e.g. "url(#myGradient)")

<defs>

<rect x="200" y="300" width="90" height="40"

fill="url(#Gradient01)" stroke="blue" stroke-width="3"/> </defs> </linearGradient> <stop offset="90%" stop-color="#F3F" /> <stop offset="20%" stop-color="#39F" /> <linearGradient id="Gradient01">

49

Masking and Clipping

• Masking– Attribute

• mask="url(#mask)"

– Draw #mask in an offscreen buffer

– Convert it to gray level image

– Composite with the masked layer

Layer1

Layer2Masked by Layer3

Result

Layer3MaskNot visible

<g mask="url(#Mask)"> <use xlink:href="#logo" transform="scale(1 -1"/></g>

50

Animating Graphics (2)

• Each graphical attribute can be animated– animate element– Animation description based on SMIL2

(Synchronized Multimedia Integration Language level 2), W3C: http://www.w3.org/TR/smil20/

• Animation describing– Attribute value variation– Time variation

• Including time effect (slow-in, slow-out, etc.)

<rect x="10" y="10" width="200" height="20"> <animate attributeName="width" attributeType="XML" from="200" to="20" begin="0s" dur="5s" fill="freeze" /> </rect>

51

Animating Graphics (2)

• Other types of animation– Animation along a path (motion tweening)

• animateMotion

– Special instruction for color animation• animateColor

• More advanced animation can be built with more programming (e.g. Morphing)– Exemple: Dragicevic, Artistic Resizing (UIST, 2005)

Click image to see movie from Web (if still online…)

52

Metadata

• Images can be describe with – <desc>

• Can contain any RDF data

– <title>• Standard element to give a title

• This can be useful for – Accessibility (screen readers, …)– Indexing image file (do not forget it's text files)

<g> <title> Company sales by region </title> <desc> This is a bar chart which shows company sales by region. </desc> <!-- Bar chart defined as vector data --> </g>

53

Extra features

• Pixel filers • Markers• Glyphs• Text along a path• Adding Interactivity with Javascript Programming

– SVG APIs • SVG DOM API: for scripted interactivity• Network communication• …

• Much more … explore by yourself – Mozilla SVG home: http://developer.mozilla.org/en/docs/SVG– Lot of fascinating examples: http://www.treebuilder.de/– SVG-Wiki: http://wiki.svg.org/– W3C SVG Home: http://www.w3.org/Graphics/SVG/– User Interfaces examples developped by IntuiLab with its own toolkit: http://www.intuilab.com/gallery/

54

Conclusion: RIA Standards and Technologies

X11

1987 1992 1994 2002 2004

ShockwaveHTML HTML 4.0

Flash/SWF

X-HTMLSVG

Javascript DOM

XMLHttpRequest Ajax

CSS

IFrame

1990 1996 1998 2000 2006

OpenGL

PDF

Direct 3D OpenGL2.0

Mosaic NetscapeIE

FirefoxKonfabulator AIR

Silverlight

XAMLFlex/MXMLXUL

WVG

Credits

Web sources indicated directly on the slides, otherwise (many thanks to them):

Illustrations

p7 CENA Digistrip www.tls.cena.fr/divisions/PII/toccata/composants/digistrips.html

p24 IntuiLab DMAN strips (courtesy of Céline Schlienger) – www.intuilab.com

p32 IntuiLab Airnet (courtesy of Stéphane Vales) – www.intuilab.com

Demos

p10 SVG Slot Machine – www.treebuilder.de

p30 Adobe Airbus A321 Panel demo – www.adobe.com/svg/demos/main.html

Book

Sketching User Experience, Bill Buxton, Morgan Kaufman by Elsevier inc. 2007 – www.billbuxton.com

Article

Combining SVG and models of interaction to build highly interactive user interfaces, S.

Chatty, A. Lemort, S. Sire, J-L. Vinot, SVGOpen 2005 – www.svgopen.org/2005/papers/CombiningSVGModelsBuildInteractiveUserInterfaces/

56

Extra Slides

57

Complete SVG File Example

<?xml version="1.0" encoding="utf-8"?><!DOCTYPE svg [ <!-- entities etc. here --> ]><svg version="1.1" baseProfile="full" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="8cm" height="6cm"> <defs> <!-- resources to be reused --> <linearGradient id="Gradient01"> <stop offset="20%" stop-color="#39F" /> <stop offset="90%" stop-color="#F3F" /> </linearGradient> <rect id="shape" width="1cm" height="1cm" stroke="blue" stroke-width="0.1cm"/> </defs> <!-- content --> <rect x=".1cm" y=".1cm" width="7.9cm" height="5.9cm" fill="none" stroke="black" stroke-width="1px" /> <use x="1cm" y="1cm" xlink:href="#shape" fill="#BBB"/> <use x="4cm" y="1cm" xlink:href="#shape" fill="url(#Gradient01)"/> <use x="1cm" y="4cm" xlink:href="#shape" fill="url(#Gradient01)"/> <use x="4cm" y="4cm" xlink:href="#shape" fill="blue"/> </svg>

58

Web Standardization

• W3C Agenda for Rich User Interaction– Markup languages

• HTML 5 ? XHTML 2 ?• SVG Tiny … SVG 1.2

– Audio and Video

– Styling with CSS3– Application Programming Interfaces (APIs)

• XMLHttpRequest (AJAX)• Cross-domain XHR

– No integrated programming environment !

• W3C is not alone…– Microsoft: XAML/WVG

• Execute in browser plugin (Silverlight)

– Adobe: FLEX/SWF• Execute on desktop runtime environement (Apollo) and browser

plugin (Flash)

59

Client Side Javascript Toolkits

• Fix "incompatibilities" between browsers– Adaptation layer

• More efficient DOM manipulation• Animation support

– Prototype + scriptaculous– Mochikit– Jquery– Yahoo UI– Dojo Toolkit– etc.

• Major sites starting to adopt them– E.g. Prototype + Scriptaculous on Apple