Upload
jaylyn-goodchild
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
User Engagement in the eSciences
Dr Marina Jirotka
Workshop on Requirements Capture for Humanities ICT projects
The Classics Centre Oxford 12th October 2006
Overview
Importance of User Engagement
Different Forms of User Participation
The Problems of Analysis
Critical Issues from Related Research (CSCW)
Embedding User Requirements into System Design
Challenges for Future Research
Importance of User Engagement
Delivered but never used
Paid for but not delivered
Used but extensively re- worked or abandoned
Used but after changesUsed as delivered
(Davis 1993)
53% of projects investigated failed
31% partially successful
planned time overrun 122%
average budget overrun 89%
Standish Group (1995-99)
53% of projects investigated failed
31% partially successful
planned time overrun 122%
average budget overrun 89%
Standish Group (1995-99)
Only 2% of software used as delivered
USA Government Accounting Office
Only 2% of software used as delivered
USA Government Accounting Office
Study of 500 IT managers in US & UK
76% experienced complete project failure
Sequent Computer Systems Inc
(1997)
Study of 500 IT managers in US & UK
76% experienced complete project failure
Sequent Computer Systems Inc
(1997)
Reasons for Project Failure
incomplete requirements 13.1%
lack of user involvement 12.4%
unrealistic user expectations 9.9%
requirements kept changing 8.7%
product no longer needed 7.5%
inadequate resources 10.6%
lack of management support 9.3%
inadequate planning 8.1%
Standish Report 1995-99
incomplete requirements 13.1%
lack of user involvement 12.4%
unrealistic user expectations 9.9%
requirements kept changing 8.7%
product no longer needed 7.5%
inadequate resources 10.6%
lack of management support 9.3%
inadequate planning 8.1%
Standish Report 1995-99
Study of 500 IT managers (76% complete project failures)
changing user requirements
Sequent Computer Systems Inc (1997)
Study of 500 IT managers (76% complete project failures)
changing user requirements
Sequent Computer Systems Inc (1997)
Mapping Out Different Forms of User Participation
None
Users as Sources of Information/Opinion
Participatory Design
Users in the Workplace
Contextual Design
Co-Design
Users as Sources of Information/Opinion - Elicitation (1)
Traditional techniques - surveys, questionnaires, interviews
Often quantitative in nature
Advantages
• Large population can be surveyed
• Costs lowered, pre-coding and computerisation speed up analysis
• Can get information quite quickly
Disadvantages
• Response rate of questionnaires low (less than 50%)
• Answers may be incomplete, illegible and incomprehensible
• Need clear idea of what questions will elicit answers to research problems and skill in interviewing
• Relationship between what we say we do and what we actually do is problematic
Users as Sources of Information/Opinion - Elicitation (2) Group techniques - brainstorming, focus groups, SSM, RAD/JAD workshops,
creative workshops
Often more in depth qualitative interviews with small number of people
Advantages
• Can get stakeholders together to discuss issues
• Guided discussion - more interactional
• Share different viewpoints and generate data for analyst and each other
• Possible to converge more easily on agreement of issues and prioritisation
Disadvantages
• Skill in managing and moderating
• Eliciting all views not only loudest and most confident
• Difficult to compare results in quantitative sense
Participatory Design
Scandinavian approach (Greenbaum and Kyng 1991)
Original goal of increasing worker involvement in technical change and innovation
To improve IT system design by encouraging shared practice between users and designers
Advantages
• Can use ‘low tech’ mock ups to elicit requirements
• Promotes opportunities to build up shared understandings
• Envision practices for how new system will be used in practice - scenarios
Disadvantages
• Often focus on early design and prototype not into development and deployment
• Not just involving users but how and when that is important
• Tacit knowledge
Scenarios
Scenarios are a powerful antidote to the complexity of systems and analysis. Telling stories about systems helps to ensure that people – stakeholders – share a sufficiently wide view to avoid missing vital aspects of problems. Scenarios vary from brief stories to richly structured analyses, but are almost Always based on the idea of a sequence of actions carried out by intelligent agents. People are good at reasoning from even quite terse stories, for example, detecting inconsistencies, omissions, and threats with little effort. These innate human capabilities give scenarios their power. Scenarios are applicable to systems of all types, and my be used at any stage of the development lifecycle for different purposes.
Ian Alexander, 2004p3, Scenarios, Stories, Use Cases, I. Alexander & N. Maiden (eds.) Wiley
Scenarios
• A linear sequence of steps taken by independently acting agents playing roles
• Analysis is about: decomposition, abstraction,moves toward the technical
• Problems• Engineers’ enthusiasm for early analysis and technical
language• How to maintain the engagement of users throughout the
development process?• Scenarios are simple way to help bridge this gap
Scenarios - Suzanne Robertson
Elicitation, Analysis, Design, Testing, Agile methods
Normal, Alternative, Exception and What if
Get rough outline of business case
• Identify right users and stakeholders
Sketch normal case scenario
• Ask questions and refine scenarios
Identify alternative case scenarios
Identify exception case scenarios
Identify what if scenarios
Iterate
Derive requirements
Users in the Workplace
Context - observation, ethnographic techniques, workplace studies
Rich qualitative fieldwork to gain understanding of users work in organisational settings
Advantages
• Can make visible ‘real world’ naturalistic workplace activities vs operationalised accounts
• Use of video
• Focus on contingencies of workplace
• Focus on details of everyday activities and use of artefacts in interaction
• Resource throughout design and development process- transportability of experience
• Resource for: informing requirements, design and generation of scenarios, prototype evaluation and deployment
Disadvantages
• Data not immediately amenable to formal representation for design
• Cannot predict future work practice from analysis of how things are now
• Need some skill in analysis and communication
Contextual Design
Understanding context indispensable to design (Beyer and Holzblatt 1998)
Contextual enquiry - talking to people as they do their work
Interpretations and modelling with cross functional teams
Consolidation of information gained through previous steps
Visioning about work practices and development of storyboards
User environment design - using storyboards to develop software floor plan that drives user interface design
Co-Design Cooperative design (Trigg et al 1999) Co-realisation (Hartswood et al 2002 )
Focus on user led innovation and design-in-use
Co-development of technologies
• Re-specification of IT design and development
• Tightly coupled ‘lightweight’ design,, construction and evaluation techniques
Attending to evaluation of technologies
Appreciation of active user participation
Adapting to particular organisational setting
Commitment to long term engagement - putting users in control
Shift focus of technical work of design and development into users workplace
Role of broker / facilitator
Problems for Analysis
Many different analytic techniques
Task analysis, walkthroughs, naturalistic analysis
Providing a warrant for analysis
Contingencies of project - time , resources, money
Recording - practical issues for audio-visual recording
Iterative development of analysis involving users, designer in data sessions, design sessions etc
The The Five Foci of Interface Development
Mainframe used by operator
after Grudin 1990
Mainframe used by
programmer
Workstation used by skilled
user
Personal computer used by individual
Networked computers used
by groups
Groupware and CSCW
TIM
E synchronous
asynchronous
Co-present Distributed
Groupware - systems and applications to support groups CSCW - Computer Supported Co-operative Work (the
study, the field)
PLACE
face-to-face
Post It sticker
telephone
letter
Example collaborative applications
TIM
E
synchronous
asynchronous
Co-present Distributed
PLACE
GDSS
-
shared drawing toolsmedia spaces
VREelectronic mailbulletin boards
shared writing toolsworkflow systems
VRE
Issues from CSCW Research
Synchronous or asynchronous
Distributed or co-located
Level of awareness of others required e.g. does the collaboration require face to face contact or is voice contact sufficient
Artefacts used e.g. phone, whiteboard, email, video conferencing
How information disseminated across setting
Issues or barriers to collaboration
Public and private activities - what resources publicly available
Tacit practices - overseeing, overhearing, practices of writing and reading etc
Embedding User Requirements into System Design
Conventional system design - requirements to design to development
Flexible cross functional teams
Iterative prototype development
Evaluate prototypes to feed into design and development (eDiaMoND)
Technological interventions - cultural probes - Gaver 1999 (IBVRE)
Involving users and designers in data sessions, design meetings, preliminary reports and presentations
Managing user expectations
Plan for conflict and conflict resolution
Formal requirements inspections
Evaluation for Requirements -eDiaMoND Evaluation of Screening
Workstation Prototype Feed into requirements for next
iteration Based on workplace analysis QuickTime™ and a
DV - PAL decompressorare needed to see this picture.
QuickTime™ and aDV - PAL decompressor
are needed to see this picture.
Challenges
Securing user participation
Benefit to users
Time constraints
Incentives
Embed resources and commitment at project proposal stage
Commitment from project management
Ethical legal and institutional dynamics - global
Analysis still problematic
How fits into organisational practices
**System development culture **
References
The Standish Group (1995) The Chaos Report. www.projectsmart.co.uk/docs/chaos_report.pdf
Beyer, H. and Holtzblatt, K. (1998) Contextual Design. Defining Customer-Centred Systems. Morgan Kaufmann.
August, J. (1991) Joint Application Design. The Group Session Approach to System Design. Yourdon Press.
Greenbaum, J. and Kyng, M. (1991) (Eds) Design at Work: Cooperative Design of Computer Systems. Lawrence Erlbaum Associates.
Heath, C.C. and Luff, P. (2000) Technology in Action. Cambridge University Press.
Checkland, P. (1981). Systems Thinking, Systems, Practice. John Wiley.
Hartswood, M., Procter, R., Slack, R., Voss, A., Buscher, M., Rouncefield, M., Rouchy, P. (2002) Co-realisation. Towards a Principled Synthesis of Ethnomethodology and Participatory Design. In The Scandinavian Journal of Information Systems.
Grudin, J. (1990) The computer reaches out: The historical continuity of interface design. CHI ‘90, April Seattle.
Working IT out in e-Science: Experiences of Requirements Capture in a HealthGrid project (2005) inn Proceedings of HealthGrid 2005, Oxford UK.
Jirotka, M and Wallen, L. (2000) Analysing the Workplace and User Requirements: Challenges for the Development of Methods for Requirements Engineering. In Workplace Studies: Recovering Work Practice and Informing System Design (Eds) Luff, P., Hindmarsh, J., and Heath, C.C. Cambridge University Press.
Robertson, S. (2004) in (Eds) Alexander, I. and Maiden, N. Scenarios, Stories, Use Cases. Wiley