View
40
Download
0
Category
Tags:
Preview:
DESCRIPTION
A Suggested Methodological Framework for Evaluating and Selecting an Open Source LMS. Dr Philip Uys Manager, Educational Design and Educational Technology, Centre for Enhancing Learning and Teaching Matt Morton-Allen - PowerPoint PPT Presentation
Citation preview
A Suggested Methodological Framework for Evaluating and Selecting an Open Source LMS
Dr Philip Uys <puys@csu.edu.au>Manager, Educational Design and Educational Technology,
Centre for Enhancing Learning and Teaching
Matt Morton-Allen <mmorton-allen@csu.edu.au>Teaching, Learning and Community Source Liaison Officer
Division of Information Technology
Introduction• Set out in Feb 2006 to enhance the virtual
learning environment• Became the Online Learning Environment (OLE)
Programme• Originally focused on individual tools but morphed to
framework emphasis• Started with 12 possible solutions• Mix of open source, commercial and in-house options• Ended with two open source• Selected Sakai
Fast Track Approach• Initially used “fast track” approach• Attempted to avoid lengthy investigation of
requirements• Focus on reusing previously supplied high
level business requirements• Success hinged on ability to easily identify low
risk solution
Fast Track Approach (cont.)• Reality was that too little information meant
too many options• Too many options meant too high a risk• High risk combined badly with lack of process
transparency• Also did little to bring together cross-silo
issues between requirements
A Different Approach• Once “fast track” abandoned needed
alternative• Extensive experience in the group not
sufficient to address OSS complexities• Short environment scan showed two possible
frameworks:• Business Readiness Rating• Open Source Maturity Model
Business Readiness Ratinghttp://www.openbrr.org/
• Geared at helping evaluate OSS• Identifies 12 criteria each with their own tests• Suggests only portion of these be applied• Assigns a weight to each test within a criteria
giving end score• Has online records of submissions made by
others
Business Readiness Rating• When looked at closely BRR has some flaws• The measures of the tests within criteria are high
level and generic• Leads to need to revise for local use• Or be faced with possibility of having undiminished
pool of options• Either means you need to move beyond BRR for a
final decision• In retrospect could have been useful early on as a
filter
Open Source Maturity Modelhttp://www.navicasoft.com/pages/osmm.htm
• Another model that could be considered – but limited• The OSMM assesses the maturity level of all key product
elements: • Software • Support • Documentation • Training • Product integration • Professional Services
A Different Approach• When neither BRR or OSMM seemed to fit began to
consider afresh• Agreed on the need for a framework that will be:
• Flexible – willingness to adapt throughout• Aligned – consistent with strategy• Comprehensive – extensive and in-depth investigation• Transparent – rigorous debate
• Devised the FACT framework for our own needs
The FACT Framework1. Identify requirements2. Weigh the requirements3. Identify possible solutions4. Identify “killer” requirements5. Apply “killer” requirements6. Determine short list7. Identify overarching concerns8. Apply overarching concerns
1. Identify Requirements• Utilised collaborative process to create
extensive (> 40) requirements list• Sources included strategy documents, feature lists
from commercial and OSS products, team member experience
• Split into high medium and low priority• Identified levels of compliance with each
requirement or “criteria”
2. Weigh the Requirements• Next we gave a weighting to each requirement• Again followed a highly collaborative process• Required several iterations to get consensus• Split 1000 points over 40 requirements• Revised several weightings when unable to
differentiate possible solutions• Always done in collaborative and transparent way
3. Identify Possible Solutions• Compiled list of possible solutions• Derived from a number of sources including
team expertise, industry reports, peer institutions etc
• Resulted in list of 12 options• It was hoped this might be trimmed
4. Identify Killer Criteria• Realised evaluating over 12 products against
40 requirements would take a long time• Decided some requirements were “show
stoppers” and thus “killed” the option• Collaboratively decided which of the
requirements had a criteria level that was unacceptable
5. Apply “Killer” Criteria• Applied killer criteria to each of possible
solutions• Once a killer had been reached further
analysis was stopped• Not all “killers” consider equal - some options
needed more than one to be removed • Reduced the list of 13 options down to 5:
• Blackboard, Angel, In-house, Moodle & Sakai
5. Apply “Killer” Criteria (cont.)• Removed options for a number of reasons:
• Mergers• Insufficient local support• Small user base
• Interestingly cost did not rule out any options at this point
6. Determine Short List• From the list of 5 we then removed:
• Blackboard – concerns over lack of competition, little leverage to control costs
• In-house – advantage of reinventing wheel questionable, OSS seemed to have same benefits without starting from scratch, lack of agility
• Angel – user base too small, too high risk, detriments of commercial without benefits of size
• Leaving us with Moodle and Sakai
7. Develop OACs• After many months of effort the quantitative
analysis gave near identical scores:• Sakai – 2428, Moodle – 2402
• If quantitative comparisons had come up empty what about qualitative ones?
• Developed “overarching concerns” or OACs:• Completely qualitative• Focus on general ideology not current features• Designed to ensure alignment between culture of
solution and the University
7. Develop OACs (cont.)• Ended up with 10 OACs covering range of
issues:• Was the community decision making centralised
or decentralised?• Was the product enterprise oriented?• Was the product stronger in secondary or tertiary
sectors?• Was the community more technically or more
pedagogically focused?
8. Apply the OACs• Once compiled the OACs were applied to the
short list of Moodle and Sakai• Four major stakeholder groups asked to
decide on Sakai or Moodle for each OAC• End result looked like …
8. Apply the OACs (cont.)
8. Apply the OACs (cont.)• 3 of the 4 team members agreed but
consensus could not be reached after intense debate
• A final Steering Committee vote selected Sakai unanimously
…
Observations• The deeper the analysis the more possible
solutions you can remove• Shallow analysis using models such as BRR can
be useful in early stages• Quantitative comparisons are less meaningful
when you can change any aspect of the software – there’s lots of grey areas
Observations (cont.)• The introduction of qualitative measures is
unavoidable and should be accepted throughout
• Qualitative comparison can only be accepted in an environment of transparent rigour
• Qualitative measure can only follow quantitative comparison – it lacks conviction in isolation
Observations (cont.)• Removing cost from the equation helped
compare OSS and commercial• Assuming cost is near equal over a period of time
removes bias and misconception (i.e. no “free lunch”)
• Evaluations require consideration of local needs and politics – highly strategic decisions cannot be based on off shelf comparisons
Observations (cont.)• Requiring consensus was time consuming but
gave strength to the results:• Forced rigorous debate• Ensured transparency throughout
• You cannot rush decisions this large – taking the time allowed a considered decision
Observations (cont.)• The framework wouldn’t have worked outside
the context of the project management methodology
• A framework needs to be contextualised within the organisational culture and strategies
A Different ApproachThe FACT framework
• Flexible – willingness to adapt throughout• Aligned – consistent with strategy• Comprehensive – extensive and in-depth
investigation• Transparent – rigorous debate.
Thank You!
Dr Philip Uys <puys@csu.edu.au>Manager, Educational Design and Educational Technology,
Centre for Enhancing Learning and Teaching http://www.csu.edu.au/division/celt/exec_staff/philip.uys
Matt Morton-Allen <mmorton-allen@csu.edu.au>Teaching, Learning and Community Source Liaison Officer
For more information http://www.csu.edu.au/division/landt/interact/
Recommended