rSmart Sakai CLE in an Irish university: a user’s perspective

rSmart Sakai CLE in an Irish university: a user’s perspective

Angélica RísquezCentre for Teaching and Learning

University of Limerick

Background• University of Limerick: ~13,000 students; ~500 lecturers• UL adopted Sakai as the common CLE in September 2005• Constructive pilot phase Semester 2 - 2005/06• Decision to use hosted solution rSmart Sakai CLE• Larger test group Semester 1 of the 2006/7 Academic year (over 30 modules and 600 students) • Integration with Active Directory, single-password access for all campus members. Integration

with the Student Information system. • Full implementation during Semester 2 2006/7 (sites for every module)• To date, more than 80 module sites published, more than 300 modules, project and practice

sites published in total. ~2,600 students• The CLE is used to complement f-t-f teaching, online courses and to coordinate research and

professional networks.

Evaluations

• Readiness survey with faculty and students, May 2006– Survey Question Bank, 5th Sakai Conference (Vancouver)

http://bugs.sakaiproject.org/confluence/display/UDAT/Survey+Question+Bank

• Survey of students using Sakai rSmart CLE, December 2006

• Faculty survey using Sakai rSmart CLE, May 2007

http://bugs.sakaiproject.org/confluence/display/UDAT/%0BSurvey+Question+Bank

http://bugs.sakaiproject.org/confluence/display/UDAT/%0BSurvey+Question+Bank

Student Evaluation• First semester of implementation of Sulis (Sakai at UL)• Approximately 30 active modules, 600 students enrolled• Online survey sent to 400 students approximately• 17% response rate• 61.5% female, 93.9% Irish

37.9

15.2

47

0

5

10

15

20

25

30

35

40

45

50

Only UL campus Internethome/work (no

broadband)

Internethome/work

(broadband)

Student Evaluation (2)Access to the Internet

26.2

44.3

29.5

0.50

5

10

15

20

25

30

35

40

45

Limited Moderate Extensive Online only

Student Evaluation (3)Use of Sulis in your course

1.6

17.2

20.3

14.1

34.4

12.5

0

5

10

15

20

25

30

35

Never Few timessemester

Few timesa month

Once aweek

Few timesa week

Daily

Student Evaluation (4)Frequency of visits

Student Evaluation (5)Overall Experience (1 = Completely disagree; 5 = Completely agree)

I think all lecturers should integrate Sulis into their modules

3.52

3.22

2.44

3.25

3.28

2.92

1 1.5 2 2.5 3 3.5 4 4.5 5

Content delivery (replication of public foldersfunction)

Communication teacher-class

Communication between students

Prompt feedback and opportunities forevaluation

Control of actitivies

Better learning (communication of complexconcepts, critical thinking, etc)

Student Evaluation (6)Sulis has helped …

(1 = Completely disagree; 5 = Completely agree)

32.8

24.6

88.3

16.7 18.3 20

8.5

45

20

40

30

15.3

10.2

70.7

5.2

67.2

3.4

36.2

8.6

Roster Gradebook

Project News Syllabus Discussion Resources Lesson builder

Announcements Post it Web Page Drop box Wiki Test & Quizzes

My Workspace Calendar Chat Email archive

Assigt’s

%

Student Evaluation (7)Tools used

75

86.791 90 92

100

83.3

96.2

83.3

69.2

86.490

10092.5

33.3

92.1100

93.7100

%

Not statistically representative

Student Evaluation (8)I found it useful/very useful (from those that used it)

Roster Gradebook

Project News Syllabus Discussion Resources Lesson builder

Announcements Post it Web Page Drop box Wiki Test & Quizzes

My Workspace Calendar Chat Email archive

Assigt’s

Student Evaluation (9)How did using Sulis impact on your learning experience (open ended)?

(Feedback related to rSmart Sakai CLE interface, n=6)

The interface is confusing and requires too much navigation to reach the desired location. I found it to be over-elaborate, simplification of the menu systems would greatly increase its potential

The distinction between learn and inform is not at all clear. Most of the students in our course have missed something, an announcement or instructions… When you become familiar with the structure and how it's used (different lecturers probably use it in different ways) it is convenient.

The discussion forums need to tell the student either by email or by having a notice on 'My Workspace' page, that a new discussion topic or reply has been posted. I do realise that there is an option that can be checked at the end of posting a discussion, however from my experience it is forgotten 99% of the time.

Indexing poor (if you want to find an article seen previously you have to remember way in)

Very easy to use

The most important factor is that everything related to the module is in one place that is user friendly and student friendly.


(Feedback related to its pedagogical use)

Helps manage study load• It allows me to keep track of coursework related to this module -the syllabus, reading lists, etc.

It's handy to have these available online without opening or printing out Word documents.• Reminders of deadlines for assignments and the like...keeps you on your toes as you see the

deadline coming closer.• Found it useful in keeping up to date on requirements for tutorials from week to week. • Automatic update by email when resources have been posted is helpful.• I find the month and semester overview option in the Calendar very helpful in getting a

broader perspective on up coming deadlines. Also it could possible show the combined calendars of the other Sakai modules being taken, would just be ideal!!

• With announcements, I felt that I was always informed of what stage I was supposed to be at.



Enhances faculty-student-student communication• Overall I think it was effective, it is different from the usual way and it is good to communicate

with people on the website.• It ensures that all relevant information pertaining to the module is made available to every

student. It doesn't discriminate against students who miss lectures (which can often be for valid reasons, e.g. illness). It is accessible from any computer, including off-campus, while Public Folders can be difficult to access.

• I know some students tend to erase their mail without opening it therefore, if it is from Sulis, you know it must be important.

• Better to communicate with class and clearly understand what we have to do in tutorial• Its helpful because I can understand and ask questions that I find difficult and I can have

access to all lectures instantly.• I find the discussion board very helpful as you can view the questions others have asked which

may be similar to your problem or may be something you had not thought of but learn because it was asked by another student



Facilitates collaboration between students• Facilitates discussion with other students and provides a facility for students to bring

attention to resources (articles, books and so on) that are relevant to the subject but which may not be listed in the normal course reading lists. Seems to be less intrusive way of communicating with lecturers, instead of the normal e-mail.

• Our project site provided a central location for storage of files pertaining to our project, that aren't relevant to other project teams in the same module. …rather than emailing or handing out printouts of every document.



Facilitates assignment submission/feedback provision• Has made submission of assignments easier and provided prompt availability of assignment

feedback. Announcements and information regarding modules can easily be accessed.• The submission of assignments is very definite, and I prefer it to dropping pages in a box in the

Dept office, … reports can be viewed on screen and backed up electronically • Its very helpful to be able to go online, and get EXACT instructions for your assignments.

Also, its handy to upload your work-it gives you a bit of leeway with the deadlines!!!• Let's face it most students leave their projects until the last minute. Therefore, every second

counts and it makes it easier, being able to submit five minutes before the deadline. It takes the pressure off of the student.

• Lecturer posted feedback to an assignment I completed. It was better than hearing it from the tutor in front of a group of people

• The online test was useful in that it allowed me to test my knowledge of the subject, and gave me feedback information afterwards so that I would understand where I had made an error.

• I prefer submitting assignments online to handing in printed reports, which is wasteful and can be susceptible to human error (loss). The late submission functionality is also useful.



But it depends on the lecturer’s input and support…• Its good insofar as an assignment and the related journal articles and any additional info are all

accessible in the one location. However, the degree to which this works depends on the lecturer in question.

• The discussion board has potential but I find it unlikely that students not experiencing difficulty, who may be able to offer help, will see the need to open it or check up on others.

• Very limited updates, no comments from lecturers• I feel that lecturers should choose one method only, i.e. either Sulis or Public Folders• It gave a clear indication as to what was required of students. This was most likely due to the

fact that our lecturer was very thorough in what she posted on the site

• I found the assignment tool very useful, but the lecturer really puts a lot of work in to our feedback and that's really probably what makes me like it so much

• It has been a useful tool for the passing on of relevant information, but I think that it could be expanded on



A bit of attention to design is required… • The more recent announcements stack up on earlier ones before they 'disappear'. Afterwards

we discovered we could go into announcements. If you have to check in Announcements to ensure you haven't missed something, the only announcement you need is 'NB: Check announcements'.

• Stick to just one solution please. As well as checking the department notice board, my private email, my UL email and public folders - I had to keep checking the Sulis site!

• If the lecturer uses the web page link option, it would be better if the webpage opened in a separate window



And remain aware of students’ diversity and possible problems… • It is too advanced for the very mature students. I did not find the lecturer helpful at all. We

were never showed how to work it properly. I think you need to know a lot about computers to following the instructions or better tutors to teach it.

• I resubmitted the assignment, after another student had shown me how to do so correctly. However, the deadline had since passed and my assignment was marked as 'late'. …I feel that there could be a risk of students being given lateness penalties as a result of an assignment not being correctly submitted, as opposed to printed reports, where a student will obviously be sure that they have handed the assignment up.

• Downloading notes through Sulis would be time consuming and a waste of paper and ink (printing on only one side from home).

• I find lectures and reading from a physical text more stimulating.

Faculty Evaluation• Second semester of implementation of Sulis (Sakai at UL)• Approximately 80 active modules, ~2,600 students enrolled• Online survey sent to 500 faculty approximately• ~11% response rate• 44.5% of respondents were Sulis users (n=24, ~70% of users)

Faculty Evaluation (2)Size of courses taught in Sulis

Faculty Evaluation (3)Frequency of visits

Faculty Evaluation (4)How do you use Sulis? (open ended)

• My students currently use Sulis to submit their coursework assignments as file attachments to me on a fortnightly basis. I also use the system to distribute information such as interesting resources as well as posting instructions on how to complete projects/assignments. I also use the system to post announcements regarding changes in venue etc. for lectures although these would be announced verbally

• Module information for students e.g. module handbook, syllabus, timetable, web links. Reading material for self-directed learning. Link to electronic reading list. PowerPoint presentations. Quizzes -based on self-directed learning material

• I store and disseminate course documents -reading lists, PowerPoint slides, occasional articles and cases, newspaper reports of points of interest. The discussion forum has been set up to discuss topics of relevance to the course, often using a current news story as the starting point.

Faculty Evaluation (5)How do you use Sulis? (open ended)

• I upload the course outline. I upload slides before every lecture so that they can print off their slides and reduce the amount of writing they have to do during lectures, so that they can listen and participate more. I upload assignment instructions, submission instructions, assignment resources (articles, links to websites). I use the discussion function to answer questions about the assignment, such that the whole class shares the benefits. My students submit their projects through Sulis and they receive their feedback through Sulis. I use the announcements section to alert the whole class to new additions to the site, to due dates, to major events etc. I add the due dates to the calendar. I upload weekly tutorial readings.

• I use course sites to make lecture notes available and to initiate discussions on course-related topics. I also make module-related announcements and students have submitted one of their assignments online. I use project sites to give groups working on projects a space for their group work.

• Mostly focused on providing resources and keeping students up-to-date with things through announcements. Students were asked to submit one essay using Sulis, and grades and comments were returned to them. They made quite a lot of use of the chat room.

Faculty Evaluation (6)How has it changed the way you teach? (open ended)

Some definitive gains…• I can concentrate on the students' learning to a greater degree, as the details of

'housekeeping' can be done through the site. I also don't have to repeat myself in lectures and tutorials as frequently, or repeatedly bring copies of handouts, as I can just refer the students to the site.

• It has greatly diminished the amount of photocopying that I have had to do over the last semester. It has also allowed me to concentrate more on teaching, because general procedural and administrative announcements can be put on Sulis.

• Better communication, deadlines/scheduling, forces better planning, student resources now on Sulis

• It has acknowledged that our students (on a part-time postgraduate programme) are at their work in the region and has helped us to have some professional on-line contact with them on a weekly basis

• My email contact time has been reduced a lot • Some discussion that would normally be in tutorials has been online.

Faculty Evaluation (7)How has it changed the way you teach? (open ended)

Versus some “status quo” and undiscovered potential …

• Very little. It was mostly used as a back-up, with handouts given in lectures copied to the Calendar and Resources, and as a way of making students aware of available resources.

• It is a convenient way for me to handle my notes. I don't think the students have used it much.

• It's quite fun, but I haven't really found that it does much that can't be accomplished just as well via email. This may be because I am still to master everything that it has to offer.

• I'm in favour of the online framework, but have not used the full range of functionality. That which I did use is okay, but I was doing largely the same via module web sites already.

Faculty Evaluation (8)Overall, how do you find it? (open ended)

Positive views…

• Fantastic. It allows for a lot of communication and resources sharing. These resources are available to students in an accessible, equitable and organised fashion. It promotes students acquiring feedback. It promotes the prompt receipt of assignments as the program records the exact time of submission. It's a one stop shop.

• An excellent resource we are only learning about it at the moment

• Great, its a wonderful system. I would like it to be more useful for programmes and not just modules so that information on a programme can be centralised

• Once I got used to the unintuitive interface, it works well. I particularly like the project worksites feature, which is great for groups. The chat client is very problematic.

• Excellent, though a bit of a learning curve at first!

Faculty Evaluation (9)Overall, how do you find it? (open ended)

Criticisms… • I find the interface unattractive, and use of the tools counter-intuitive.• Above all the only main criticism that I have is that the system is not hosted on-site at UL. As

my assignments are quite large 5-10 MB per student, often downloading these from the system fails

• Good .... A bit tedious from one semester to next; difficulty in seamless transfer of info to similar modules next time On line assessment facilities could be improved

• Overall, it's good, and the students found it useful, and relatively easy to use once they got used to it. However, it's slow to load, navigation/where things are is not always obvious, as a learning environment it's not very dynamic, and it can be very inflexible and cumbersome, so that, for example, you can't (or at least I couldn't) give a mark using Sulis to a student who had been given an extension, so hadn't submitted through Sulis. Rather than replacing some systems, it tended to just become another system running parallel to existing ones.

Conclusion

• Students are overall satisfied. However students’ satisfaction with the use of Sulis is associated with:– More active, student-centred teaching (i.e. formative feedback, self-

assessment, group collaboration, class interaction) as opposed to reproducing traditional face-to-face delivery (i.e. providing resources and posting announcements)

– Lecturers guidance and support (i.e. not redundant tools, clear guidelines for using the tools, not duplication of communication/resource distribution channels, support if problems happen).

– ICT skills (e.g. mature students may struggle)• Overall agreement there is a learning curve to be surmounted• Most lecturers are duplicating traditional delivery, scope for exploration,

encouragement and support to progress towards a a cultural change• Navigation of CLE needs to be simplified… (see new version!)

Background (2)• A collaborative team with members from CTL and ITD provided training and

support. 95 members of staff voluntarily attended introductory and advanced training sessions during Semester 2 06/07

• Sulis Support site with training and support information, discussion forums, etc.; but mainly one-to-one email/f-t-f support

• Liaison with different groups of faculty to guide, advice and support on the strategic adoption of technology in teaching in existing and new programmes

• TEL Days in September 2006, where a group of faculty and staff from UL presented their experiences with Sakai.

Documents

rSmart Sakai CLE in an Irish university: a user’s perspective