Students' experience of component versus integrated virtual learning environments

  • Published on

  • View

  • Download


Students experience of component versusintegrated virtual learning environmentsM. Weller, C. Pegler & R. MasonInstitute of Educational Technology, The Open University, Milton Keynes, UKAbstract The use of virtual learning environments (VLEs) has become increasingly common in mosthigher education (HE) institutions. Recent developments have proposed the interoperabilityof software systems and content, to create component VLEs in contrast with the integrated,monolithic ones that are currently prevalent. This paper examines the student experience oftwo VLEs, one integrated approach and the other component. In general, students preferredthe component system, although this may have been influenced by other factors such asperformance. Although the study is limited to one cohort of student it makes a number ofsuggestions relevant to anyone deploying a VLE. These are that the component approach is aviable one from a student perspective, the broader context in which the VLE operates isimportant in student perception and that poor system performance may have unpredictableconsequences for the learning experience.Keywords e-learning, interview, satisfaction, virtual learning environmentsIntroductionVirtual learning environments (VLEs) have become apervasive technology in much of higher and furthereducation, with 86% of respondents from UK HE in-stitutions reporting the presence of a VLE in theirinstitution (Brown & Jenkins 2003) and 70% of UKfurther education (FE) colleges using a proprietaryVLE (Becta 2004). As with many new technologies,there are many definitions for the term VLE, and it isoften used synonymously with the term learningmanagement system (LMS). The differences betweenmany definitions are often subtle and serve the parti-cular aims of the definer. Definitions can be in terms offunctionality, for instance states Theprincipal components of a VLE package include cur-riculum mapping (breaking curriculum into sectionsthat can be assigned and assessed), student tracking,online support for both teacher and student, electroniccommunication (e-mail, threaded discussions, chat,Web publishing), and Internet links to outside curri-culum resources. For the purpose of this paper theJoint Information Systems Committee (JISC) defini-tion (JISC 2000) which states the term refers to thecomponents in which learners and tutors participate inon-line interactions of various kinds, including on-line learning is sufficient (interactions here can referto interactions with content as well as those withstudents and tutors). Although this paper focuses onthe VLE, much of what is covered can be broadened tothe wider technical environment, commonly termedthe Managed Learning Environment, which, accordingto the JISC definition (JISC 2000), includes the wholerange of information systems and processes of theCollege (including its VLE if it has one) that con-tribute directly or indirectly to learning and learningmanagement.There are two approaches to the design of VLEs.The predominant one is what can be termed a mono-lithic or integrated approach. This provides all of theCorrespondence: M. Weller, Institute of Educational Technology,The Open University, Milton Keynes MK7 6AA, UK.Email: 7 April 2005& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259 253Original articlecommon online learning tools within one softwarepackage. As well as providing an off-the-shelf solutionto most of the e-learning needs of an institution thereare also benefits to having all components within onecommon system. The key benefits are administrativein that they provide a single integration point withwhich other institutional systems can interface, andpedagogical, in that they allow student activity to betracked across components, so a holistic view of theironline behaviour can easily be viewed. For many in-stitutions being able to purchase a single system thathas most of the online tools they will require enablesthem to quickly establish an online presence.Recently it has been suggested that an alternativeapproach is both feasible and desirable. This design canbe termed a component, or hybrid architecture. As wellas offering the benefits of a single interface to the variousonline components for the user, it also offers the possi-bility to adopt a best of breed approach, by combiningcomponents from different providers (including free oropen source software) into a single system.The viability of such component VLEs has beenraised by recent developments which seek to specify ageneric, standards-based approach to VLEs, often fo-cused around open-source systems. These include theSAKAI initiative in the US and the JISC service-or-iented architecture in the UK. The SAKAI projectaims to deliver the following (, all as open source:The products of this project will include an EnterpriseServices-based Portal, a complete Course ManagementSystem with sophisticated assessment tools, a ResearchSupport Collaboration System, a Workflow Engine, anda Technology Portability Profile as a clear standard forwriting future tools that can extend this core set ofeducational applications.The JISC framework (Wilson et al. 2004) outlines thebenefits and approach for adopting a service-orientedarchitecture, which can be seen as a means of viewingthe integration of systems:When we embark on this kind of analysis, identifyingthe parts of the MLE at a more granular level thanmonolithic systems, then we eventually end up with aframework of service descriptions. We are no longerinterested so much in replicating data between largesystems, but instead focus on what kinds of services areneeded in the overall architecture to provide certainkinds of behaviour from applications.This can all be viewed in the broader context of amove towards interoperability. Over recent years anumber of standards have been developed in educa-tional technology, notably by the IMS Project andADL in the US. The IMS web site ( describes key specifications in simpleterms by placing them in categories describing theircore function: Specifications used to describe, discover and ex-change content. Specifications for Content interaction and tracking. Specifications for Application system interoperability.The motivation behind all of these standards is thedesire for interoperability. This can be in terms ofcontent (being able to share e-learning material be-tween institutions and thus reduce the cost of devel-oping it), student data (being able to easily transferstudent records to provide a more flexible learningpath for students) or systems (being able to describegeneric interfaces for systems).This move towards looser integration of systemsand less proprietary based solutions is not confined tothe educational sector. In broader terms the develop-ment of web services seeks to achieve much the samegoal. The idea behind web services is that a piece ofcode is available to remote machines over the Internet.The code in effect acts as a service, providing func-tionality or data to a remote machine without having todownload the code. It is delivered over the web, henceweb services.XML underpins all of this and is at the heart of webservices, being the method by which data is packagedand passed between systems. Web services are lan-guage-agnostic, thus one service might use Java todevelop its service, and another use Visual Basic, butthey could still communicate. This is significant be-cause it means services can be integrated across dif-ferent systems without the need for costly reengineer-ing of both systems (although the implementation ofweb services will have an associated cost).Web services are often promoted as a solution toenterprise architecture integration. Their advantage inthis area is that they do not require expensive in-tegration of databases or recoding of existing systems.They are simpler, based on open standards, moreflexible and cheaper than other forms of integration. If254 M. Weller et al.& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259one views the VLE as essentially an integration pro-blem, then web services offer a viable means of rea-lising this across most applications, without the needfor extensive alteration to existing systems.There are inherent benefits in each of these twodesign approaches but both can be seen as striving forthe advantages of the other approach also. For in-stance, many integrated commercial VLEs such asBlackboard and WebCT offer means of integratingnew software components into their existing provisionthrough the use of a standard application programmeinterface (API). Conversely the SAKAI and JISCproposals seek to provide a common interface both interms of data and look and feel to a disparate set oftools. Both approaches are keen to support the use ofstandards. This raises the issue of whether there ismuch difference between the systems from a usersperspective. While they may operate differently andnecessitate different technical skill sets, from an enduser perspective they may feel similar, which makesthe institutional decision to adopt either approachlargely a strategic one.It is this perspective that this paper seeks to addressby examining the students experience of two differentVLEs, which correspond to the two design ap-proaches, while studying the same postgraduatecourse. The course in question is Learning in theConnected Economy (H806), which was one of threepilot courses for the UK eUniversities (UKeU) in-itiative which aimed to deliver online courses fromUK-based Higher Educations Institutions to a globalaudience. The UKeU developed a new VLE (in col-laboration with Sun Microsystems) through which allcourses were delivered.The course was developed by the UK Open Uni-versity (OU), but presented through the UKeU. TheUKeU was responsible for marketing and hosting thecourse, but students were supported by OU tutors andthe award was through the OU. The course consists offour modules, each divided into approximately 1weeks worth of study material. Each week is thencomprised of a number of learning objects. The courselasts for 8 months and provides approximately 450student study hours.In June 2004 it was announced that the UKeUwould cease operation by mid-July ( As thecourse still had 3 months of presentation remaining, itwas necessary to migrate students from the UKeUVLE to the OUs platform. This provided a rare op-portunity to evaluate the student experience of VLEsbased on different design principles.The UKeU VLE was developed in collaborationwith Sun Microsystems. It was based around a com-ponent architecture, but designed from a top-downperspective as an integrated system. It incorporatedsome third party technologies, including forums,multiple choice software and calendar system. As suchit can be viewed as representative of the monolithic,integrated design approach.In contrast the OU has developed a range of com-ponents to meet specific needs over a number of years,without necessarily integrating these into one system.Such in-house developments include an assignmenthandling system and an authentication system. In ad-dition some commercial systems have been adaptedand incorporated into widescale use in the OU. Themost notable of these is OpenTexts FirstClass, whichnow has 252 000 registered OU users. In order to in-tegrate these the OU has adopted a web services ap-proach, developing a student portal and VLE calledthe eDesktop. The VLE uses a template approach, toprovide course teams with a means of creating anonline presence. This is the standard web presenceused for over 250 of the OUs current courses. Minorcustomisation is supported and varying course re-sources are provided, but the default eDesktop pre-sents a student portal incorporating course-specificcalendars and news, direct access to relevant assess-ment, conferencing and library resources all ofwhich existed as free-standing components before thedevelopment of the eDesktop. The OU VLE cantherefore be seen as representative of the component,best of breed approach.While there has been much interest around the po-tential to build a component VLE, there are relativelyfew examples of this in practice. Both the JISC andSAKAI work is aimed at fostering the development ofsuch systems. The main implementation of such anapproach has been the LeAP project in Tasmania, whichhas successfully employed a service-oriented archi-tecture, based around a set of loosely coupled compo-nents. Given that it is unusual to have a set of studentswho experience two different platforms, and that thenumber of component-based implementations is limited,the following research is necessarily limited in scope.Students experience of VLE 255& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259Data collectionMethodOn completion of H806 students were asked to com-plete a web-based questionnaire, which focused onissues associated with their use and experience of thetwo VLEs and the migration process. Twenty two ofthe 24 students who completed the course respondedto the web questionnaire and twenty of these weresubsequently interviewed by telephone (one declinedto be interviewed and one responded to the ques-tionnaire after the interviews had already been com-pleted) The 20 interviewees were allocated randomlyto the three course team members to conduct struc-tured interviews according to an agreed set of ques-tions, each interview also drew on the intervieweesresponses to the web questionnaire.The interviews lasted between fifteen and fortyminutes each, and were transcribed by the interviewer.They were conducted over a 5-week period after thecompletion of the course.Issues in data collectionThe issue of subjectivity and potential bias is raised bymembers of the course team conducting the inter-views. However, given the requirement for familiaritywith the migration situation and knowledge of both ofthe learning environments involved (one of which wasno longer available for reference), it was felt that theadvantages outweighed potential disadvantages. Thepossibility of bias was compensated somewhat byrandomly allocating students to three separate inter-viewers.It should also be borne in mind that this is a smallnumber of students, at Masters level, and many ofthem were already experienced in e-learning at thestart of the course (52% (11) had studied onlinepreviously). Similarly, the evaluation was limited toonly the two VLEs mentioned, and a much broadersurvey would be necessary to validate any findings.The course represented an unusual opportunity toperform an evaluation whereby the VLE was theonly variable the course and cohort remained con-stant, which would not be the case with any widerresearch.A further issue with the data collection is possiblebias in the response students gave to representatives ofthe OU when drawing comparisons between theirplatform and that of another institution. However,these are mature students and are accustomed to giv-ing honest feedback, and as the researchers were notresponsible for implementing either platform, it wasfelt that this potential bias was negligible.Survey resultsThe results of the web survey are shown below:256 M. Weller et al.& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259Issues arising in interviewsThe immediate view from the questionnaire is that theOU platform was preferred by most students. How-ever, the interviews revealed a more complex picture.The UKeU platform, being a new development, suf-fered from performance issues, in that some pageswould take a long time to load, and there were occa-sional downtimes of up to forty-eight hours. The lastUKeU upgrade was on the 1 June, introducing addi-tional functionality which students would have hadonly a small time to adjust to before the platform waswithdrawn. The OU platform was more stable and thusits performance was more reliable. In a pure e-learningcourse such as Learning in the Connected Economy,VLE performance is an important issue as it representsthe environment in which nearly all study takes place.Therefore the preference of the OU platform waslikely to be influenced by this performance factor,which was unique to the UKeU platform, and not ty-pical of commercial VLEs. In interviews, 12 studentsreported performance problems including slow access,screens freezing, access to particular resources ortools and downtimes. These students responded thatthe performance issue did influence their view of theplatform overall.Additionally, although most students preferred theOU platform overall, they often differentiated betweendifferent functions of the platforms, particularly withregard to computer conferences (or discussiongroups). In this area nine students stated that theypreferred the discussion facility in the UKeU platform,and of the five who stated a clear preference for theOUs discussion system (which uses FirstClass), threeof these used the client software to access the dis-cussion . FirstClass can be accessed via both the weband specialised client software. The speed andfunctionality of the client is usually superior, but someof the integration within a single web browser inter-face is lost. In this respect it was not possible to makea direct comparison between systems. The remainingstudents did not express a preference in this area offunctionality.Another important issue that arose from the inter-views was the significance of the broader contextwithin which learning takes place. The OU has astudent portal, termed StudentHome, which offerspersonalised information regarding the students re-cord, general advice from a range of university sources(e.g., advice on course choice), university and regionalnews, etc. As the UKeU operated as a portal to manydifferent university providers, this level of informationwas absent. Although the course team did not stressthe use of StudentHome (as students had migratedmid-course the focus was on the course area ratherthan university level information), ten students re-ported that they used this area and found it useful.Many students stated that they found its presencereassuring, since it made them feel part of a wideruniversity, with the associated support. As one studentput it although I didnt use it much, it was nice toknow it was there, I felt part of a university. It wascomforting.On the issue of integration eleven students (52%)responded that they felt the OU platform was moreintegrated, five (24%) preferred the UKeU platformand the remainder stated that there was little differencebetween the two in this respect. Integration is, how-ever, a difficult concept to define, and it was clear thatits meaning varied between students. For some it was amatter of design, or aesthetic uniformity, for others itwas more related to functionality, and for others na-vigation was the key issue. Integration within a com-plex system is a combination of all of these and theinterviewers deliberately avoided a tight definition soas to elicit an impressionistic, rather than academic,response. The issue of integration is likely to be in-fluenced by the previous two issues performance willinfluence a persons perspective and the presence ofthe OU student portal can be seen as providing a moreintegrated experience.On the issue of integration, one area of discussionthat arose from some interviews was the extent towhich integration is desirable anyway. Four studentsoffered the comment that they felt integration was notimportant. One student suggested that if youve gotgood tools then that is enough. Another studentclaimed that integration may in some respects bedetrimental, since it does not expose students tothe sort of software they may encounter outside ofthe educational context, the tools which would beavailable to them beyond the course, and there wasvalue in going out and bringing things back from theweb. Doing this makes you more confident becausethere is no sense of the safety of a hermetically sealedproduct.Students experience of VLE 257& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259DiscussionWhile it is necessary to be cautious about generalisingfrom such a small study, there are some lessons andconclusions that can be drawn from this work. Thefirst is probably the most obvious, but is worth stating,namely that in an e-learning context platform perfor-mance and reliability are paramount. The con-sequences of poor, or unreliable performance can bedifficult to predict. They may not always be manifestin standard course feedback parameters, for example,student satisfaction for this course, as a whole andwith regard to content, was very high. Students in thiscourse were required to differentiate between thecourse and the method of its delivery. Some studentscommented that they engaged in dialogue less fre-quently than they may have done otherwise because ofperformance issues associated with the discussion toolin the UKeU platform (pages frequently froze).A related issue, that should act as a caveat to anyconclusions drawn here, is that the OU has gainedconsiderable experience over a number of years with anumber of the components of its VLE, particularlyFirstClass and the assignment handling system. Thus,the degree to which the component approach re-presents a good starting model for any institution isdebatable.For those with an interest in the potential of com-ponent VLEs, based around the combination of anumber of separate components, the results from thisresearch are encouraging. These were sophisticatedstudents, many of whom had prior experience of VLEsand who were acquainted with relevant theoreticaldevelopments, such as personalisation, standards andlearning theories. From the student perspective the OUplatform felt, and behaved, like a VLE. AHerran(2000) has suggested that there are four perspectiveson VLEs, and that different criteria are importantwithin these: For administrators: scalability, value for money,integration with existing systems. For technicians: robustness, user base, technicalsupport, ease of maintenance. For course developers or teachers: customisability,flexibility, integration of legacy materials. For learners: consistency, accessibility, quality ofdesign.This research has only focused on the learner per-spective, and it is possible that either platform wouldhave provided a different response from one of theremaining perspectives. However, Timmis et al (2004)claim that to date, the evaluation, subsequent supportand use of VLEs has focused on staff rather thanlearners. While all perspectives are valid, this is theone, which should probably have the highest priority.The OU VLE is not a fully realised component VLE,in the manner of the Tasmanian LeAP project (LeAP2004) which uses a service oriented approach to createa flexible VLE:The project has guiding principles of interoperabilityand the use of standards for data and infrastructure. Thepreferred application architecture model uses a servicebased infrastructure approach. The reality is that thediversity of products within the educational computingenvironment makes it impossible to adopt a single ap-proach to application architecture. LeAP considers itgood practice to use existing services and create newservices as application development progresses.The LeAP project represents the adoption of thecomponent approach in developing a VLE, whereasthe OU experience is that a similar approach is auseful means of combining pre-existing componentsinto a cohesive architecture. Work is currently underway at the OU to further integrate back-end systemsand to describe interfaces in a generic manner that faci-litates easy decoupling and exchange of components.The results of this research may indicate a con-vergence in learning environments between techno-logical developments and a growing maturity in theuse of online technologies by learners and educators.The move towards interoperable, service-based solu-tions makes the notion of a more fluid VLE, whichoffers different components within the same overalltechnical framework, viable. In parallel, as learnersand educators become more sophisticated with theiruse of online technologies, the need for systems thatare easily reconfigurable to suit the demands of aparticular learner or cohort becomes more apparent.Kraan (2003) of the UK advisory body CETIS hassuggested that the only way in which all subjectcommunities will be catered for properly, may be toforget about monolithic VLEs, and move to collec-tions of specialised tools that do one or two thingsreally well. This idea can be extended beyond subject258 M. Weller et al.& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259communities to different types of learners also, inessence, is a VLE that is suitable for a first time un-dergraduate learner the same as one for an experiencede-learner?The last issue raised by this work, which has im-plications beyond this course, is that of the role of thebroader context in the learning process. While it mayseem obvious to state that a VLE does not exist inisolation, and operates as part of a broader institutionand educational experience, the role of this widercontext is little understood and appreciated. Even be-fore the demise of the UKeU it was being reported thatstudents preferred to study online with well-knownproviders rather than with new purely online ventures(Berry-Helmlinger 2004). The significance of thiswider educational context is sometimes overlooked inthe move towards exchangeable content and data. Thisview typically overestimates the importance of contentand underestimates the role of support. For example,McCrea et al (2000) suggest that more and morecontent will migrate out of the classroom with theview that content is like any other consumable. As onestudent put it on this course, the OU looks and feelslike a serious organisation which for a learner ner-vous about unproven e-learning approaches musthave great psychological value. How this broadercontext is realised online is only just being appre-ciated, with the development of personalised portals,and is likely to be an area of growth in the comingyears as tools which facilitate informal dialogue areadopted.In conclusion then, the OU platform was preferredoverall. This may have been a result of performance orfamiliarity however, and should not be interpreted as acomplete vindication of the component approach.This evaluation does demonstrate however that thestudent perspective of architecturally different systemsis that they were largely comparable with areas ofpreference in both, in our case performance and theportal in the OU system and discussion boards in theUKeU one. The implications of this for institutionsadopting a VLE or renewing their VLE provision arethat the decision should either be determined by stra-tegic, financial considerations or they should decidewhich specific area of functionality is of prime im-portance to them, and choose the solution that bestserves that need.ReferencesAHerran A. (2000) Research and evaluation of online sys-tems for teaching and learning. AusWeb2k-The SixthAustralian World Wide Web Conference, Rihga ColonialClub Resort, Cairns, 1217 June 2000. Retrieved 27January 2005 from (2004) ICT and e-learning in Further Education: em-bedding technology, evolving practice. A summary of thereport to the Learning and Skills Council. Retrieved 27January 2005 from L. (2004) New survey gives an edge tononprofit online universities. Denver Business Journal,May 21, 2004. Retrieved 27 January 2005 from T. & Jenkins M (2003) VLE Surveys: a longitudinalperspective between March 2001 and March 2003 forHigher Education in the United Kingdom. Retrieved 27January 2005 from Joint Information Systems Committee (JISC) (2000)Circular 7/00: MLEs in Further Education: progress re-port. JISC 1 July 2000. Retrieved 27 January 2005 from W. (2003) The teachers teach the techies. 15 April 152003. Retrieved 27 January 2005 from Project Case Study (2004) Implementing Web Ser-vices in an Education Environment. Version Date: 20July 2004. Retrieved 27 January 2005 from F., Gay K. R. & Bacon R. (2000) Riding the bigwaves. A White Paper on the B2B e-Learning Industry,Thomas Weisel Partners.Timmis S., OLeary R., Weedon E., Harrison C. & Martin K.(2004) Different Shoes, Same Footprints? A Cross-Dis-ciplinary Evaluation of Students Online Learning Ex-periences: Preliminary Findings from the SOLE Project.Journal of Interactive Media in Education, [On-line] (13)Available at: S., Olivier B., Jeyes S., Powell A. & Franklin T.(2004) A Technical Framework to Support e-Learning.JISC. Retrieved 27 January 2005 from experience of VLE 259& Blackwell Publishing Ltd 2005 Journal of Computer Assisted Learning 21, pp253259


View more >