Upload
martin-bazley
View
107
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Slides used during workshop at GEM conference 2011
Citation preview
Sustaining digital learning provision
GEM conference 2011 Norwich8 Sep 2011
Martin Bazley
Martin BazleyPreviously• Teaching (7 yrs)• Science Museum, London,
Internet Projects (7yrs)• E-Learning Officer, MLA South East
(3yrs)
Martin Bazley• Current• Vice Chair of Digital Learning Network
DLNET• Developing online resources, websites, user
testing, evaluation, training, consultancy…Martin Bazley & Associateswww.martinbazley.com
Slides and notes available online afterwardswww.slideshare.net/martinbazley
www.martinbazley.com
Note to self: check stats tomorrow to see if anyone looked up the website
To sustain your digital learning offer:• (a) find out and
(b) stay in touch with what your users want – gather views and data (online audience research)
• Use iterative review as part of planning and developing your online offer
• Make use of free and low cost tools and services
Session elements:
planning, developing and evaluating online resources
intro to planning digital audience research
free / low cost digital tools and services / techniques for developing online resources
planning, developing and evaluating online resources
intro to planning digital audience research
free / low cost digital tools and services / techniques for developing online resources
not only resources labelled as ‘learning’: teachers and others are using content in different ways – teachers mainly look for images, videos, etc
also learning experiences using digital technology – see New Media handbook
the key point is that review and evaluation are essential to sustaining digital learning offer
Various advice and guidance from:
onlineaudiences.wordpress.com (this link will be repeated later)
www.martinbazley.com (for contact details)
planning, developing and evaluating online resources
intro to planning digital audience research
free / low cost digital tools and services / techniques for developing online resources
Website users• Who uses your website (or resource) ?
• Why would they want to use it?
• How would they find it?
• What do they get out of it?
• What do they dislike about it?
How do you get it right for everyone?
• Answer:• You can’t get it right for everyone.
• You have to make choices, and stick to them:
• Who is it for? • What..• How…
Who for…?Who for…?What for?What for?
How will they use it?How will they use it?
Who for what for ...• Who for? (audience)
Need to be clear from start• mum + 2 children looking for something to do
this weekend• teachers of yr5/6 in local area with whiteboards• men interested in gadgets
Who for what for ...• What ‘real-world’ outcomes?
What will they do as a result of using the site?
• make a donation• plan a visit to a museum• buy a train ticket• think differently about learning disability
Who for what for ...• How will they use it? (user experience)
What do they actually do on the site? • browse and read articles
• working alone or in pairs? (learning resources)• lean forward or sit back?• Browsing, following, searching…
• Also Where, When and Why?
Testing is an iterative process
Testing isn’t something you do once
Make somethingMake something=> test it => test it
=> refine it=> refine it=> test it again=> test it again
Developing a learning resource: iterative review
your content curriculum (find a match)
Ch
eck
Does it match your audience’s specific needs?
If so TEST - and then amend
Learning activities Learning outcomes (find a match)
When to evaluate or test and why
• Before funding approval – project planning
• Post-funding - project development
• Post-project – summative evaluation
Before funding – project planning• *Evaluation of other websites
– Who for? What for? How use it? etc– awareness raising: issues, opportunities– contributes to market research– possible elements, graphic feel etc
• *Concept testing – check idea makes sense with audience– reshape project based on user feedback
Focus group
Research
Post-funding - project development• *Concept testing
– refine project outcomes based on feedback from intended users
• Refine website structure– does it work for users?
• *Evaluate initial look and feel – graphics,navigation etc
Focus group
Focus group
One-to-one tasks
Post-funding - project development 2
• *Full evaluation of a draft working version – usability AND content: do activities work, how
engaging is it, what else could be offered, etc
Observation of actual use of website
by intended users,
using it for intended purpose,
in intended context – workplace, classroom, library, home, etc
Post-funding - project development 3
• Acceptance testing of ‘finished’ website– last minute check, minor corrections only– often offered by web developers
• Summative evaluation– report for funders, etc– learn lessons at project level for next time
Website evaluation and testingNeed to think ahead a bit:
– what are you trying to find out?
– how do you intend to test it?
– why? what will do you do as a result?
The Why?Why? should drive this process
Martin BazleyOnline experience consultant
Evaluating online learning resources in the classroom
Key point:
for a site designed for schools,
the most effective user testing
observations
will be made in a real classroom situation
National Archives Moving Here projectalso Victorians (with V&A) – about to be released
For teachers of 8 – 14 yr olds
History Geography and Citizenship
Features: Interactives, activity sheets, audio and video
clips
Moving Here Schools:For 8 – 14 yr olds studying:
History Geography and Citizenship
Features:
Interactives, activity sheets, audio
and video clips
1. preliminary testing sessions –
conventional user-testing with teachers (at TNA)
2. in-class testing – teachers used the Moving Here Schools site with pupils in their own classrooms This meant sitting at the back of the
classroom observing and taking notes…
Evaluation: 2-phase approach
Site ready in parts – but not too ready:
The environment had a significant impact on how the site was used.
The class dynamic within the different groups contributed to how much the students learned.
The environment and social dynamics
The environment had a significant impact on how the site was used.
The class dynamic within the different groups contributed to how much the students learned.
in-class testing picked up elements not there in conventional user testing.
teachers in preliminary user testing did not spot some problems until actually in the classroom. For example…
interactive activities:
looked big enough when viewed on a screen nearby…
… but text/images too small for some children to see from the back of the class…
…so interactives needed to be viewable full-screen
Only spotted during in-class testing:
…so interactives needed to be viewable full-screen
content: when students tried to read text out loud, teachers realised some text was too difficult or complex
activity sheets:
some sheets did not have spaces for students to put their names - caused confusion when printing 30 at same time…
Manchester Art Gallery art interactive
For teachers of 8 – 11 yr olds, and for pupils
History Art and Citizenship
Features: interactive with built in video, quiz, etc,
plus activity sheets and background info
Martin Bazley
www.ICT4Learning.com
Martin Bazley
www.ICT4Learning.com
Martin Bazley
www.ICT4Learning.com
Martin Bazley
www.ICT4Learning.com
'This classroom user testing is all very well, but...'
How can you see everything in a class of 30 children – don't you miss things?
You see things in a classroom that don't arise in one-to-one testing
They are the real issues
'This classroom user testing is all very well, but...'
How can you see everything in a class of 30 children – don't you miss things?
You see things in a classroom that don't arise in one-to-one testing
They are the real issues
'This classroom user testing is all very well, but...'
Doesn't using a specific class with particular needs skew the results?
» For example, low ability, poor English, equipment not working, behaviour issues, etc - are results as reliable as those in a 'neutral' environment?
» ‘neutral environment’ ? – no such thing - any test will be subjective, and in any case:
» Testing is to make website work well in classroom, - need to see effects of factors like those.
'This classroom user testing is all very well, but...'
'This classroom user testing is all very well, but...'
Doesn't using a specific class with particular needs skew the results?
» For example, low ability, poor English, equipment not working, behaviour issues, etc - are results as reliable as those in a 'neutral' environment?
» ‘neutral environment’ ? – no such thing - any test will be subjective, and in any case:
» Testing is to make website work well in classroom, - need to see effects of factors like those.
'This classroom user testing is all very well, but...'
'This classroom user testing is all very well, but...'
Can't my Web developer do the testing for us? » best not to use external developer to
do user testing - conflict of interest » also likely to focus more on the
technical aspects of the site than on effect on the teacher and pupils.
» observe classes yourself but use an independent evaluator for key decision points
'This classroom user testing is all very well, but...'
'This classroom user testing is all very well, but...'
Can't my Web developer do the testing for us? » best not to use external developer to
do user testing - conflict of interest » also likely to focus more on the
technical aspects of the site than on effect on the teacher and pupils.
» visit a classroom yourself but use an independent evaluator for key decision points
'This classroom user testing is all very well, but...'
'This classroom user testing is all very well, but...'
I don't have the time or budget to do this!
» need cost no more than conventional user testing. one person could attend a one-hour class session in a school, giving the teacher the same small token payment
» This programme had evaluation built into project: 6.7% of total Schools site budget.
» Allow 5 -10% of total project budget for user testing
=> videos
'This classroom user testing is all very well, but...'
Video clips • Moving Here key ideas not lesson
plans etc http://www.vimeo.com/18888798
• http://www.vimeo.com/18892401 Lesson starter
• Time saver http://www.vimeo.com/18867252 S
User test early
Testing one user early on in the project…
…is better than testing 50 near the end
Two usability testing techniques
“Get it” testing- do they understand the purpose, how it
works, etc
Key task testing- ask the user to do something, watch how
well they do
Ideally, do a bit of each, in that order
User testing – who should do it?• The worst person to conduct (or interpret)
user testing of your own site is…– you! you!
• The 2nd person is…– Your web developer! Your web developer!
• Beware of hearing what you want to hear…
• Useful to have an external viewpoint• First 5mins in a genuine setting tells you
80% of what’s wrong with the site
planning, developing and evaluating online resources
intro to planning digital audience research
free / low cost digital tools and services / techniques for developing online resources
Plan, evaluate, do audience research
o How to plan and conduct research into online audiences:http://sca.jiscinvolve.org/audience-publications/
o Focus on why you are measuring what you plan to measure
o Web stats, online questionnaires, focus groups (concept testing), user testing – various ways of gathering data
o Build costs into the project for regular evaluation – avoid expensive remedial action later in project
How can we get a sense of who our online visitors are and what they do with our online content?
How do we gather data to help us improve what we do?
How do we measure success from the user's point of view, and against our own objectives and constraints?
For example, how justify investment (or lack of it) in digital, including social networks etc?
Tools available
• Qualitative – focus groups, “free text” questions in surveys, interviews
• Quantitative – web statistics, “multiple choice” questions in surveys, visitor tracking
• Observational – user testing, ethnographic
examples
• SurveyMonkey online survey
• Google Analytics website statistics
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Define audience research goal
Analyse data
Collect data Use results to guide changes
Plan methodology
Martin Bazley0780 3580 737
www.martinbazley.com
More information / advice / ideas
planning, developing and evaluating online resources
intro to planning digital audience research
free / low cost digital tools and services / techniques for developing online resources
Freebies and cheapies
o WordPress and other ‘free’ CMSs for creating websites, blogging, or prototyping
o Online survey tools, online event booking
o YouTube, Vimeo, podcasting
o Social media: Facebook, Twitter
o Google maps, Google site search, Google Analytics
o TripAdvisor, etc
Making websites - CMSs
http://wordpress.com - free website creation service (pay extra for features like own domain name etc). All hosting and upgrading etc is done for you
Making websites - CMSs
http://wordpress.org - free website creation service – like .com except you have to install it on your server and you are responsible for updating, hosting costs, etc
Making websites - CMSs
www.contentcurator.net - free open-source CMS specially developed for cultural and heritage sector. Powerful and easy to use e.g. in-place editing
Making websites - CMSs
www.cmsmadesimple.org - free website creation service – you install it on your server and you are responsible for updating, hosting costs, etc - very similar to wordpress.org
Prototyping with WordPress
http://ruskinelementsofdrawing.wordpress.com
http://educationonline.ashmolean.org/ruskin
Video – maximise impact
http://vimeo.com
http://ruskinelementsofdrawing.wordpress.com
Social media – maximise impact?
Has potential to increase engagement but be aware:– long term commitment essential so financial
implications in resourcing– Can also work against you. DLNet vs. Dulwich Picture Gallery blog
Online audiences - links
Advice and links on online audience data gathering, evaluation, New Media handbook etc
http://onlineaudiences.wordpress.com/
More info / advice / ideas
Martin Bazley
0780 3580 727
www.martinbazley.com