Embed Size (px)
Citation preview
Outlook Web App
Final Report
Authors:
Bruno Picão
João Correia
Jorge Santos
Patrícia Mota
Tiago Cavaleiro
Subject:
Human-Computer Interaction
Oporto, December 2015
2
Abstract
This report was assigned within the Human-Computer Interaction
subject of Faculdade de Engenharia da Universidade do Porto. We
were asked to identify a system that provides a faulty interaction to
its users and improve it according to their needs.
Therefore, we chose to study Microsoft’s Outlook Web App (OWA),
which is a service that allows users to connect to their email
accounts via a web browser, without requiring the installation of
Microsoft Outlook.
Our primary motivation to study this software was the fact that it is
used by several companies in their everyday activities and it raises
some problems to the employees’ work. These issues can represent
major setbacks in the companies’ work flow and productivity.
In addition, we intend to propose improvements in the Outlook Web
App that would be beneficial to organizations that use this email
service and, consequently, could improve their overall efficiency. We
will take advantage of our access to Sheraton Porto Hotel & Spa’s
employees to further develop and sustain the report’s results.
3
Table of contents
1. Introduction ................................................................................ 5
2. Questionnaires ........................................................................... 6
2.1 Email Questionnaire .................................................................. 6
2.1.1 Gmail ...................................................................................... 7
2.1.2 FEUP Webmail ....................................................................... 8
2.1.3 Hotmail ................................................................................... 9
2.1.4 Microsoft Office Outlook ....................................................... 10
2.1.5 Thunderbird .......................................................................... 12
2.2 Outlook Web App Questionnaire ............................................. 13
2.2.1 Users with access to Microsoft Outlook ................................ 15
2.2.2 Users without access to Microsoft Outlook ........................... 16
2.3 Major Findings ......................................................................... 17
3. Heuristic Evaluation ................................................................. 19
4. Usability tests ........................................................................... 21
5. Proposed Solutions .................................................................. 25
6. Conclusion ............................................................................... 29
7. Attachments ............................................................................. 31
4
List of illustrations
Figure 1 -Email questionnaire’s number of replies per email service 6
Figure 2 -Gmail users’ satisfaction degree graphic .......................... 7
Figure 3 -FEUP Webmail users’ satisfaction degree graphic ........... 8
Figure 4 -Hotmail users’ satisfaction degree graphic ....................... 9
Figure 5 -Microsoft Outlook users’ satisfaction degree graphic ...... 11
Figure 6 -Outlook Web App questionnaire's number of replies ...... 14
Figure 7 -Outlook Web App users’ satisfaction degree graphic...... 14
Figure 8 -OWA Heuristic Evaluation .............................................. 19
Figure 9 -Usability tests’ pre-test information ................................. 22
Figure 10 -Usability tests’ task duration (in seconds) per user ....... 22
Figure 11 -Proposed solutions ....................................................... 26
Figure 12 -Mockup of the proposed solution to problems 8 and 9 .. 26
Figure 13 -Mockup of the proposed solution to problem 2 ............. 27
Figure 14 -Mockup of the proposed solution to problem 8 ............. 27
Figure 15 -Mockup of the proposed solution to problems 5 and 8 .. 28
Figure 16 -Mockup of the proposed solution to problems 3 and 5 .. 28
5
1. Introduction
In order to understand the users of the OWA we will, in a first
instance, understand the preferences and satisfaction level of a wide
range of several email services’ users, using an online survey.
Additionally, with the purpose of gathering more information
regarding a more specific sample of users that depend on the OWA
in their work activities, we intend to question a group of Sheraton
Porto Hotel & Spa personnel, using an online questionnaire with the
presence of one of our team members to follow-up each user
completion, answering any possible doubts they might have (and
possibly gaining more insight on their opinion).
Furthermore, we will perform a Heuristic Evaluation aiming an in-
depth knowledge on all the OWA usability problems. To verify
whether these problems affect the typical users and possibly
discover additional issues, we will undertake some Usability Tests.
Finally, we anticipate that we will be able to propose some
improvements based on the data retrieved on the previous research
methods. Regarding the problems we feel it is relevant, we will
construct some prototypes that allow the visualization of the related
improvements.
6
2. Questionnaires
2.1 Email Questionnaire
We inquired a broad sample of e-mail users aiming to obtain
relevant data for statistical analysis of their satisfaction and
preferences, segmented to the email services they use. In this
questionnaire, our sample provided following information:
-Which email service they use;
-Whether they use it for professional purposes or not;
-Their satisfaction degree with the email service they use
professionally;
-Their assessment on the Design, Usability, Interface Interaction,
Content Organization and Personalization;
-What changes would they make in their email.
The analysis of the 647 questionnaires replies showed the following
division between 5 different email services:
EMAIL SERVICE REPLIES
GMAIL 248
FEUP WEBMAIL 230
HOTMAIL 113
MICROSOFT OUTLOOK 43
THUNDERBIRD 13
Figure 1 - Email questionnaire’s number of replies per email service
7
From the total of responses we gathered, only 5% (33) of the users
do not use their email service for professional purposes. This will
reinforce the importance of the data as the service we proposed to
study is primarily used in a corporate environment.
Furthermore, we were able to observe that the majority of users are
satisfied with the email service they use. Overall, 529 (82%) of the
respondents gave positive feedback, assessing their level of
satisfaction as “4” or “5” in a scale from 1 to 5.
2.1.1 Gmail
Out of our sample, the Gmail users are the most satisfied with their
current webmail service.
Figure 2 - Gmail users’ satisfaction degree graphic
According to Figure 2:
- None of the inquired assesses its satisfaction level as “1” out of 5;
0
20
40
60
80
100
120
140
1 2 3 4 5
8
- 1 out of the 248 google mail users we inquired perceives the
service as negative accessing its satisfaction level as “2”;
- 91% (225) of the users assess it as a “4” or “5”.
By comparing all the different characteristics assessed1 we can
conclude that:
- The questioned Google mail users value its Usability – 83% (206)
evaluated it as “Good” or “Very Good” and only 3 consider it “Very
Bad” or “Bad”;
- They also value its Interface Interaction –78% (194) evaluated it as
“Good” or “Very Good” only 4 consider it “Very Bad” and “Bad”;
- The least rated feature was Personalization – 8% (19) evaluated it
as “Bad” or “Very Bad”.
2.1.2 FEUP Webmail
According to our questionnaire analysis, although most FEUP
Webmail users are satisfied users, this service ranked last regarding
the satisfaction degree.
Figure 3 - FEUP Webmail users’ satisfaction degree graphic
1 See attachment 2
0
20
40
60
80
100
120
140
1 2 3 4 5
9
It is observable in the Figure 3 that:
- 73% (169) provide reasonable or positive feedback –evaluated it
as “Good” or “Very Good”;
- 10 out of the 230 users we inquired perceive the service as
negative accessing its satisfaction level as “1” or “2” out of 5.
The comparison of all the different characteristics that we
questioned the users about provided us the following information:
- The questioned FEUP Webmail users value its Usability – 71%
(163) evaluated it as “Good” or “Very Good”;
- The least rated feature was Personalization – 48% (86) evaluated
it as “Bad” or “Very Bad” and only 11 users consider it “Very Good”;
- The Design was also criticized – 50% (114) find it “Reasonable”
and only 8 users consider it “Very Good”.
2.1.3 Hotmail
After analyzing the obtained data, we were able to conclude that the
majority of Hotmail users are satisfied with the service.
Figure 4 - Hotmail users’ satisfaction degree graphic
0
10
20
30
40
50
60
70
80
1 2 3 4 5
10
As it is possible to retrieve from Figure 4:
- 83% (94) of the users we inquired perceives their satisfaction with
the service as positive accessing its satisfaction level in “4” or “5” out
of 5;
- None of the users assesses it as level “1” and only 1 out of 113
measures it as level “2”.
By analyzing all the different characteristics assessed we can
conclude that:
- The best rated feature by the Microsoft Outlook is Usability – 80%
(90) evaluated it as “Good” and “Very Good” and none assess it as
“Very Bad”;
- The questioned Hotmail users value its Interface Interaction – 73%
(83) assess it as “Good” and “Very Good” and none assess it as
“Very Bad”;
- The least rated feature was Personalization – 21% (24) evaluated
it as “Bad” and “Very Bad” and only 13 users consider it “Very
Good”.
2.1.4 Microsoft Office Outlook
After analyzing the obtained data, we were able to conclude that
Microsoft Outlook users’ satisfaction degree is also high:
11
Figure 5 - Microsoft Office Outlook users’ satisfaction degree graphic
- 74% (32) out of the 43 Microsoft Outlook users we inquired
perceives their satisfaction with the service as positive accessing its
satisfaction level in “4” or “5”;
- Only 2 users assess it as level “1” and “2” out of 5.
The comparison of all the different characteristics we questioned
about provided us the following information:
- The best rated feature by the Microsoft Outlook is Usability – 81%
(35) evaluated it as “Good” and “Very Good” and none assess it as
“Very Bad”;
- The questioned Microsoft Outlook users value its Interface
Interaction – 74% (32) evaluated it as “Good” and “Very Good” and
none assess it as “Very Bad”;
- The least rated feature was Personalization – 26% (11) evaluated
it as “Bad” and “Very Bad” and only 13 users consider it “Very
Good”;
- The Design aspect had a negative feedback – 16% (7) of the users
attributed “Very Bad” or “Bad” to this feature.
0
5
10
15
20
25
1 2 3 4 5
12
2.1.5 Thunderbird
Thunderbird was the email service with fewer replies, 13 users
provided their opinion regarding this software. Therefore, the
information we gathered regarding this service will have less
relevance in the following report. The majority (9) user are satisfied
assessing it as level “4” or “5” and none of the users rated “1” out of
5.
13
2.2 Outlook Web App Questionnaire
We questioned a targeted sample of OWA users, namely Sheraton
Porto Hotel & Spa employees, with the purpose of elaborating a
statistical analysis of their satisfaction and preferences:
-Their satisfaction degree with Outlook Web App;
-Whether they have access to Microsoft Office Outlook Software or
not;
-Their assessment on the OWA’s Design, Usability, Interface
Interaction, Content Organization and Personalization;
-Their preference between OWA and Microsoft Office Software (in
case they have access to both at work) regarding the same
characteristics previously mentioned;
-Weather they perceive the differences between them as negative
their work and if they would value more consistency between the
two services;
-What changes would they make in Outlook Web App.
In order to have a more concise view on the OWA’s assessment, it
is important to separate Microsoft Outlook users’ that complement
their work with the webmail version (OWA) from those who only use
the webmail service to perform their daily professional activities.
The following chart provides the division our sample according to
those 2 types of users:
14
Figure 6 - Outlook Web App questionnaire's number of replies
According to our questionnaire results, it is possible to observe a
decrease in the users’ satisfaction degree when comparing with the
other analyzed email services data:
Figure 7 - OWA users’ satisfaction degree graphic
22
5
Outlook Web App +Microsoft Office Outlook
Outlook Web App
0
2
4
6
8
10
12
1 2 3 4 5
15
After observing Figure 7, we determine that:
- Only 3 users assess it as level “5” out of 5;
- 10 out of the 27 Microsoft Outlook and Outlook Web App users we
inquired perceive their satisfaction with the service as medium
accessing its satisfaction level in “3”;
- 5 users evaluated the webmail service as level “1” or “2”.
2.2.1 Users with access to Microsoft Outlook
By comparing all the different characteristics assessed we can
conclude that:
- The least rated feature was Personalization – 8 evaluated it as
“Bad”, 13 as “Reasonable” and none of the users consider it “Very
Good”;
- The Content Organization aspect had a negative feedback – 7
users attributed “Bad” to this feature and none of the users consider
it “Very Good”.
After analyzing all the replies, we found that between 16 and 17 out
of the 22 users that access their email account using both services
prefer Microsoft Office Outlook over Outlook Web App in all of the
analyzed categories. Furthermore, the majority (13) of users
considers that the differences between them affect their work
negatively and 20 users would like to see a greater consistency
between the software and the webmail.
16
2.2.2 Users without access to Microsoft Outlook
The analysis of the data collected for each feature we inquired the
users about, showed that:
- Design is the only category that had a negative appreciation – 1 out
of the 5 users consider it “Bad”, 1 rated it as “Reasonable” and the
other 3 evaluated it as “Good”;
- The remaining characteristics were valued as “Reasonable” by 2
users and “Good” by 3 workers that only use Outlook Web App in
their work activities.
17
2.3 Major Findings
The most valuable data we gathered from our sample during both
questionnaires and the usability tests is the following:
1. Gmail users are the most satisfied of the general email users
sample
- They value its Usability and Interface Interaction;
- The feature they are more unsatisfied about is Personalization.
2. FEUP webmail users are the least satisfied of the general email
users sample
- The characteristics they most dislike are Personalization and
Design;
- They appreciate its Usability.
3. The majority of the Microsoft Outlook and Hotmail users are
satisfied with both services
- They assess Usability and Interface Interaction as valuable
features;
- They do not value its Personalization options.
4. Most Outlook Web App users that also have access to Microsoft
Outlook find that the differences between both have a negative
effect on their professional activities:
- These type of users assess Outlook Web App’s
Personalization and Content Organization as its worst features;
- Users that only access Outlook Web App rated Design as the
least valued characteristic.
18
5. Almost all users believe that more consistency between both
services would be beneficial to their work
6. The majority of users considers that the analyzed characteristics
are better implemented in Microsoft Office Outlook than in
Outlook Web App
19
3. Heuristic Evaluation
Using the previously collected data, we identified and defined the
usability principles that we will use while evaluating the OWA. Then,
separately, our group members browsed the email service aiming to
find usability errors throughout the multiple functionalities.
After, we combined and analyzed all the recognized issues gathered
by the 5 members and listed them as is observable in the following
table:
Problems2 Heuristic Severity
1 - Some users cannot change their signature 3 5
2 - System gives users no information on how to change the password in case they have forgotten it
3 4
3 - Hard to access all functionalities on the Options Menu
2 3
4 - System gives users no possibility to re-enter the system after logging out
2 2
5 - System provides users too much customization options and does not show an easy way to undertake more important email options
2 2
6 - Logout button is too exposed to users, they can logout at any time without intention
2 2
7 - Undo functionality (in some options) does not go to previous screen, but to options menu
3; 4 2
8 - Lack of familiar design features (icons) to specific user actions (ex: New Email)
2 1
9 - System shows users too much information 2 1 Figure 8 - OWA Heuristic Evaluation
2 See attachment 9
20
The Heuristics shown in the previous table were identified according
the 10 Nielsen’s Heuristics3 and the severity of the problems was
evaluated according to the Severity Code4.
3 See attachment 4
4 See attachment 5
21
4. Usability tests
To better understand weather the problems we identified in the
Heuristics Analysis affect OWA typical users and to figure out if they
are able to find some additional issues, we gathered a group of
Sheraton employees that are constantly in contact with this software
for professional purposes.
The group was chosen considering their job title, because their level
of interaction with this tool varies according to it, and availability,
since they were fulfilling their professional obligations at the time.
In the first instance, we briefed the participants regarding the
following activities they were about to execute. Additionally, we
assured that they were comfortable on providing their personal data,
having their test recorded and fully collaborating with us, knowing
that we were going to use these information in a further evaluation of
the Usability Tests. None of them objected and all 6 were
comfortable with the test and with the testing environment.
Moreover, we inquired the group before5 the usability test in order to
have relevant information of each participant. We also provided a
task list6 they had to follow through in order to successfully complete
the test. Finally, we questioned them after7 the test about their
difficulties so we could assess their performance more accurately.
The following table shows the 6 participants’ information that was
collected in the pre-test questionnaire:
5 See attachment 6
6 See attachment 7
7 See attachment 8
22
Raquel Hering - Test 1 Andreia Ferreira - Test 4
21 years old 24 years old
Front-Office Intern Human Resources Admninistrative
1 year using OWA 1 years using OWA
Daily Weekly
Mafalda Correia - Test 2 Mariana Tenreiro - Test 5
28 years old 26 years old
Receptionist Sales Specialist
2 years using OWA 3 years using OWA
Daily Weekly
Manuel Silva - Test 3 Pedro Pinto - Test 6
25 years old 27 years old
Receptionist IT Assistant
3 years using OWA 3 years using OWA
Daily Monthly Figure 9 - Usability tests’ pre-test information
The results show that even though 3 participants easily completed
the test without any problems. Furthermore, we were able to identify
that the other 3 users struggled with simple and routine tasks as well
as with more complex and less used functionalities.
It is important to refer that their level of satisfaction with this tool is
not directly connected with their performance on the test. Despite
this fact, we observed that those who were better succeeded in the
test rated their satisfaction level as “4” out of 5. On the other hand,
the less succeeded assessed it as “2” or “3”.
Task\Test 1 2 3 4 5 6
Login 61 95 35 60 45 14
Send email 40 63 25 60 33 22
Create new folder 64 31 14 32 23 5
Move email to folder 19 20 10 10 12 7
Filter emails 16 21 10 28 11 8
Define "New Email Signature"
66 74 21 - 33 7
Define "Automatic Reply" 48 142 35 - 13 14
Logout and login again 84 65 15 68 24 18 Figure 10 - Usability tests’ task duration (in seconds) per user
23
By analyzing the data above, we arrived to the conclusion that out of
the 8 tasks assigned to the users, “Define “New Email Signature””
and “Define “Automatic Reply”” represented the main difficulties.
According to the Post-test questionnaires:
- 2 users found assessed the “Define “Automatic Reply”” task as “5”
out of 5, regarding their difficulty level;
- 3 of them evaluated the “Define “New Email Signature”” with as “4”
or “5”;
- One of the participants wasn’t able to successfully conclude both of
these activities;
- Another user pressed the “Go back” button several times during
the “Define “New Email Signature”” task and it took a total of 2
minute and 22 seconds (142 seconds) to conclude it.
The tasks in which the participants were better succeeded were:
“Create New Folder”, “Move email to folder” and “Filter email”. Still,
one user struggled with two of them:
- The user found difficult to “Create New Folder”, taking over a
minute (64 seconds) to accomplish it, and assessed this task as “4”
out of the 5 levels of complexity;
- Did not successfully finish the “Filter email” task as she filtered the
emails inside a specific folder instead of doing so in the Email Box.
The “Log In”, “Send Email” and “Log Out and Log In again” tasks
were also crucial in helping us understand the complexity of this
tool. They represent what we consider to be the most important and
simpler functionalities of an e-mail service. However, the majority of
24
participants took longer than what is expected, as it is observable in
Figure 10.
A further development of the tests to possibly discover problems that
we had not found in the previous evaluation was not possible, due to
constraints that are out of our control8. Despite that, their analysis
strengthened the significance of the issues identified during the
Heuristics Evaluation.
Therefore, we can conclude that this software needs some
improvements in order to reduce its complexity level and provide a
faster service. According to our studies, the difficulties that
participants faced can be partially attributed to the fact that this
software is not intuitive as they had to search the items on the email
to find the desired options.
8 Limited time with each employee
25
5. Proposed Solutions
Regarding the problems we identified in the Heuristic Evaluation and
that the users experienced during the Usability Tests, we hereby
propose some solutions, as the following table shows:
Problem Solution
1 - Some users cannot change signature
OWA developers should further develop this feature to prevent this type of problems
2 - System gives users no information on how to change the password in case they have forgotten it
System should display a link in the initial page to change the user’s password
3 - Hard to access all functionalities on the Options Menu
Create an Options Menu with those functionalities: general, email, calendar, contacts, tasks, search, languages and advance
4 - System gives users no possibility to re-enter the system after logging out
When user presses the logout button, the system should redirect to the initial page and not to another page. That way users can re-enter in an easier way
5 - System provides users too much customization options and does not show an easy way to undertake more important email options
Options Menu should provide information in order to access all options in one place, prioritizing the more relevant functionalities
6 - Logout button is too exposed to users, they can logout at any time without intention
System should present a safer way to logout by transferring that option to options menu
7 - Undo functionality (in some options) does not go to previous screen, but to options menu
All undo actions should go to the previous page
8 - Lack of familiar design features (icons) to specific user actions (ex: New Email)
Usage of an action bar, with relevant icons, that is more intuitive to users
26
9 - System shows users too much information
Reorganize and reduce information on the screen to be easier to user
Figure 11 - Proposed solutions
Bearing in mind the users’ preferences obtained from the
questionnaire results, we elaborated the following mock-ups to
provide a view on the graphical design improvements which we
believe that will better the overall user experience, especially the
Sheraton Porto Hotel & Spa employees:
Figure 12 - Mockup showing the proposed solution to problems 8 and 9
27
Figure 13 - Mockup showing the proposed solution to problem 2
Figure 14 - Mockup showing the proposed solution to problem 8
28
Figure 15 - Mockup showing the proposed solution to problems 5 and 8
Figure 16 - Mockup showing the proposed solution to problems 3 and 5
29
6. Conclusion
The overall assessment of our choice to study Outlook Web App as
a software with failures which cause problems and difficulties to its
users daily routine, was extremely positive.
In the first instance, with the questionnaire to several e-mail
services’ users, we acquired a general idea of their opinion and
where OWA fits in that average. With this, we reached the
conclusion that, as whole, people were satisfied with their e-mail
service and that the most satisfied were the Gmail users. Moreover,
the less pleased respondents use FEUP webmail.
Then, supported by the Nielsen’s Heuristics, we detected the most
critical OWA issues and assessed them according to their severity
level. Even though we found some catastrophic and major problems,
that must be changed immediately, most of the identified usability
flaws can be overcome by the users and are fixable with a few
graphical alterations.
In this sense, the usability tests with the cooperation of Sheraton
Porto Hotel & Spa’s workers were key in providing us additional data
to validate the previous findings. These users, whose professional
activities rely on the use of OWA and are highly experienced with
this tool, struggled with those same problems.
After all the studies and taking in consideration the retrieved
information, we created our design proposals by essentially having
in mind a bigger consistency between OWA and Microsoft Outlook
software. Our idea was to evolve OWA into a more familiar, practical
and intuitive software with the aim to better some functionalities and
30
improve the user experience. Additionally, we also considered some
Gmail and Hotmail characteristics that users find valuable while
preparing the mock-ups.
To conclude, our main goal with this project was to create solutions
to barriers that OWA users face in their day-to-day work. We believe
that we reached our goals with this report as we succeeded in
advising some improvements that can positively change the way
Sheraton employees perceive this email service, as well as the
remaining OWA users.
31
7. Attachments
Attachment 1 – Email Questionnaire
32
33
Attachment 3 – Email Questionnaire results
Gmail
0 20 40 60 80 100 120 140 160
Very bad
Bad
Reasonable
Good
Very good
Design
0 20 40 60 80 100 120 140
Very bad
Bad
Reasonable
Good
Very good
Usability
0 20 40 60 80 100 120 140
Very bad
Bad
Reasonable
Good
Very good
Interface interaction
34
0 20 40 60 80 100 120
Very bad
Bad
Reasonable
Good
Very good
Content organization
0 20 40 60 80 100 120
Very bad
Bad
Reasonable
Good
Very good
Personalization
35
FEUP Webmail
0 20 40 60 80 100 120
Very bad
Bad
Reasonable
Good
Very good
Design
0 20 40 60 80 100 120 140
Very bad
Bad
Reasonable
Good
Very good
Usability
0 20 40 60 80 100 120 140
Very bad
Bad
Reasonable
Good
Very good
Interface interaction
36
0 20 40 60 80 100 120
Very bad
Bad
Reasonable
Good
Very good
Content organization
0 20 40 60 80 100
Very bad
Bad
Reasonable
Good
Very good
Personalization
37
Hotmail
0 10 20 30 40 50 60
Very bad
Bad
Reasonable
Good
Very good
Design
0 10 20 30 40 50 60
Very bad
Bad
Reasonable
Good
Very good
Usability
0 10 20 30 40 50 60 70
Very bad
Bad
Reasonable
Good
Very good
Interface interaction
38
0 10 20 30 40 50 60
Very bad
Bad
Reasonable
Good
Very good
Content organization
0 10 20 30 40 50
Very bad
Bad
Reasonable
Good
Very good
Personalization
39
Microsoft Outlook
0 5 10 15 20
Very bad
Bad
Reasonable
Good
Very good
Design
0 5 10 15 20 25 30
Very bad
Bad
Reasonable
Good
Very good
Usability
0 5 10 15 20 25 30
Very bad
Bad
Reasonable
Good
Very good
Interface interaction
40
0 5 10 15 20 25
Very bad
Bad
Reasonable
Good
Very good
Content organization
0 5 10 15 20
Very bad
Bad
Reasonable
Good
Very good
Personalization
41
Thunderbird
0 1 2 3 4 5 6
Very bad
Bad
Reasonable
Good
Very good
Design
0 1 2 3 4 5 6 7
Very bad
Bad
Reasonable
Good
Very good
Usability
0 1 2 3 4 5 6 7 8
Very bad
Bad
Reasonable
Good
Very good
Interface interaction
42
0 1 2 3 4 5 6 7 8
Very bad
Bad
Reasonable
Good
Very good
Content organization
0 1 2 3 4 5 6 7
Very bad
Bad
Reasonable
Good
Very good
Personalization
43
Attachment 3 – Outlook Web App Questionnaire
OWA with access Microsoft Office Outlook
44
OWA without access to Microsoft Office Outlook
45
Attachment 4 – Nielsen’s Heuristics
The 10 Nielsen heuristics
1. Visibility of the system status
The system should always keep users informed about what is going on through appropriate feedback within reasonable time.
2. Match between the system and the real world
The system should speak the users language, with words, phrases, and concepts familiar to the user rather than system‐oriented terms. Follow real‐world conventions, making information appear in a natural and logical order.
3. User control and freedom
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
4. Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
5. Error prevention
Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error‐prone conditions or check for them and present users with a confirmation option before they commit the action.
6. Recognition rather than recall
Minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
46
7. Flexibility and efficiency of use
Accelerators ‐ unseen for the novice user ‐ may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
8. A esthetic and minimalist design
Dialogues should not contain information that is relevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
9. Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
10. Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.
47
Attachment 5 – Severity code
Severity code
Five point rating scale:
1. Cosmetic ‐ will not affect the usability of the system, fix if possible
2. Minor ‐ users can easily work around the problem, fixing this should be given low priority
3. Medium ‐ users stumble over the problem, but quickly adapt to it, fixing this should be given medium priority
4. Major ‐ users have difficulty, but are able to find workarounds, fixing this should be mandatory before the system is launched. If the problem cannot be fixed before launch, ensure that the documentation clearly shows the user a workaround
5. Catastrophic ‐ users are unable to do their work, fixing this is mandatory
48
Attachment 6 – Usability Test’s Pre-test
Questionnaire
49
Attachment 7 – Usability Test’s Work Instructions
Outlook Web App – Teste de usabilidade
1. Inicie sessão no Outlook Web App com a sua conta de utilizador
2. Crie um novo email, com um ficheiro em anexo à sua escolha e
envie para: [email protected]
3. Crie uma nova pasta com o título “IPCO”
4. Mova um email à sua escolha para a pasta “IPCO”
5. Aplique um filtro na caixa de entrada para vizualizar apenas os
emails “Não lidos”
6. Defina uma nova assinatura de Correio Eletrónico e salve-a. Não
é necessário fazer alterações na assinatura, apenas guardar a
assinatura.
7. Coloque um template de Resposta Automática para avisar que
se encontra de férias.
8. Termine sessão e volte a Iniciar
Obrigado pela colaboração!
50
Attachment 8 – Usability Test’s Post-test
Questionnaires
51
Attachment 9 – Identified problems
Problem 1
Problem 2
52
Problem 4
Problem 5, 6, 8 and 9
