Formulation of Usability Risk Assessment Model

Jayaletchumi T. Sambantha Moorthy, Suhaimi bin Ibrahim, Mohd Naz’ri Mahrin Universiti Teknologi of Malaysia (UTM)

Kuala Lumpur, Malaysia. tjaya771129@yahoo.com, suhaimiutmkl@gmail.com, mnazrim@ic.utm.my

Abstract— Usability is considered as one of the significant factors of software product quality and becoming an emergent property of an entire closed and open system. Existence of usability problem has been discovered to cause many quality problems in several studies. Despite various efforts have been taken to overcome these problems, usability problems still exist in software products. This affects continuous interaction of an open system with the user and environment. This paper presents study on formulation of Usability Risk Assessment Model to identify, analyse and prioritize potential usability risk during Software Development Lifecycle (SDLC). Using this model, usability problems in software products are dealt before it occurs to produce more usable software product and continuous interaction of an open system with the user and environment can be ensured.

Keywords-usability; usabiltiy problem, usability risk, risk assessment model.

I. INTRODUCTION

One important aspects of software development process is software quality, as it might result in serious consequences such as financial loss and reputation loss [2]. Usability is an important quality factor and even in the very first model of software quality known as McCall Factors Criteria Metrics (FCM) proposed by McCall has mentioned this fact [3].

Many studies (e.g. [4][5]) have revealed that usability problems causes quality problem. Example of usability problems that leads to quality problem are usability activities poorly integrated in product designs, lack of skills and knowledge in designers and management on usability, incompetence of various usability activities in usability engineering life-cycle and badly chosen usability methods [6]. A usability problem in interface design, operating process or product structure also lowers effectiveness, efficiency and difficulty of use for end users [8]. Indirectly, existence of usability problems could lead to failure of software product [7].

Websites as well faces huge usability problems such as high number of broken links, slower accessibility speed, lack of interactive features and accessibility features which acts as factors for lower usage level [9][10][11][12]. But usability problems are almost ten-times more common on business applications than on websites because business software

contains a lot of complex functionality and there are more things that can impede a good user experience. User-experience could be improved and quality of a software product can be increased by reducing the frequency of usability problem [13].

To reduce usability problems, usability process, standard, techniques and practices was closely integrated with software development process [14]. However, software developers face difficulties when new usability practices are introduced [15]. Some usability approaches are only integrated in requirement and design phase [16]. In fact, their practical implementation is largely missing [17]. Usability practices as well are not part of requirement engineering [18], so developers are often given an incomplete, confusing, and sometimes contradictory requirement. As a result, many development teams are facing difficulties in avoiding and minimizing usability problems.

Besides this, various usability evaluation activities such as inspection, empirical testing, and metrics for usability standards in computing has been integrated into software development process to measure and improve usability of software [19]. However, it only evaluates a completed system and does not intervene at earlier stages of development process [20]. An International standard, ISO 13407 [21] also had proposed a framework for integration of usability in all phases of software development process. This standard uses the approach of User-Centered Design (UCD) that focuses specifically on developing usable system. Even so, current usability engineering practices had failed to reduce usability problems in software products.

Alternatively, by using the approach of risk management, usability problems can be dealt before it occurs. However, there are great ignorance on managing usability risk compared to managing other risks such as technology risk, market risk and money risk [22]. Furthermore, there is little effort in identifying, analyzing and prioritizing potential usability risks at earlier phases of development process.

In relation to this, development of Usability Risk Assessment Model could guide development team on identifying, analysing and prioritizing potential usability risks that could arise during SDLC in order to produce more usable software products. This paper presents formulation of Usability Risk Assessment Model systematically.

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

978-1-4799-0285-9/13/$31.00 ©2013 IEEE 168

This paper is structured with Section I on Introduction, Section II reviews existing studies on usability and software risk assessment models, Section III on Formulation of Usability Risk Assessment Model and Section IV includes conclusion and future work.

II. RELATED WORKS

A. Usability Usability is one of the significant factors in software

product quality [1] and neglecting this factor could lead to failure or success of software product. Usability can significantly improve product acceptability and reliability, increase user satisfaction, and produce financially benefits to companies [23]. Usability is the best factor that balances between technical and human aspects of a software product which is important in defining quality [2].

Usability has no universally accepted definition or models and defined as combination of attributes and sub-attributes in the quality model.

B. Software Risk Assessment Models Many scholars have suggested risk assessment models to

evaluate the risk of software projects, namely:

(i) Software Risk Assessment Model (SRAM) [27] considered nine risks, namely: software complexity, project staff, targeted reliability, requirements, estimation methods, monitoring method, development process, usability of software and tools used for development as risk indicator to predict possible outcome e of software projects.

(ii) Risk is estimated risk using risk exposure and software metrics of risk management in Software Risk Assessment and Estimation Model (SRAEM) [28].

(iii) Risk Identification, Mitigation and Avoidance Model for Handling Software Risk (RIMAM) [29] identifies, manages and avoid certain risk factors such as immature requirement, less reusability, delivery deadline, staff experience, preservation of Intellectuals, staff Turnover and others.

(iv) Software fault tree approach (SFTA) is used to identify and analyse the risk in the Software Risk Assessment and Evaluation Process using Model Based Approach (SRAEP) [30]

All four assessment model are developed from the perspective of software development and not from the perspective of software product itself [31][32]. These models are explained based on theoretical aspect of Software Risk Management and lacks in practical guidelines. There is also a

lack in standard framework or model for assessing and managing software product risk based on attributes of quality, particularly usability.

This creates extensive need for proposed Usability Risk Assessment Model, developed from the perspective of software product by considering usability as an important attribute of software product in this assessment model

III. FORMULATION

Formulation of Usability Risk Assessment Model involves three phases with six activities. Each phase and its activities are explained as below:

A. Phase 1: Identification of elements and activities in software risk assessment models

The objective of this phase is to identify important elements and activities of software risk assessment models. This phase has one major activity, Activity 1 as explained below.

Activity 1 investigates important elements and activities of software risk assessment models using Systematic Literature Review (SLR) [33]. SLR involves three main steps: Review Planning, Conducting the Review and Reporting the review.

Review planning comprises identification of the need for a review and development of a review protocol, as explain below:

(a) Identification of the need for a review is the first step in Review Planning. The objective of Activity 1 is to identify elements in software risk assessment model which is used as a basis for development of Usability Risk Assessment Model in this study and to identify gaps in current software risk assessment models that can be overcome in the proposed model in this study.

(b) The second step in Review Planning is Development of a review protocol is as explained below:

i. Background of the studyThis study reviews existing literature and modelsrelated to software risk assessment.

ii. Research goal and research questionsThe goal of this review is to identify elements andactivities that constitute in software risk assessmentmodels. The research question will be: “What are themajor elements and activities that need to beconsidered in developing Usability Risk AssessmentModel?”

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

169

iii. Identifying the KeywordsIn this step, search keywords related to our study isdetermined. Search keyword that will be used in thisreview are such as software risk assessment model,software project risk assessment model and softwaredevelopment risk assessment model.

iv. Identifying the resourcesResources are data sources from where related paperscan be obtained. The primary studies were searchedfrom five main online search databases: Institute ofElectrical and Electronics Engineers (IEEE),Association for Computing Machinery (ACM),Springer, Science Direct and Wiley Online. Besidessearching directly from the online databases, manualsearch of references from primary journals are alsoconducted to ensure completeness in the search.

v. Identifying the inclusion/exclusion criteriaIn this step, criteria for including and excluding relatedpapers are determined. As for this research, inclusioncriteria will be to consider all expert reviewed paper(proceeding and journals) and excluding criteria will beto exclude any related books and chapters.

vi. Identifying the data extraction strategyOnce related papers obtained from the search, data willbe extracted from the paper and a form is design torecord the data. Table 3-1 shows the design of the formwhich will be used to record the extracted data.

Once Development of Review Protocol is completed, the second part in SLR, Conducting the Review will be implemented as five steps:

i. Identification of researchThis step deals with generating search strings andqueries by decomposing the search keyword in partsand drawing synonyms, abbreviations and alternativespellings. The search strings that will be used in thissearch are:

(“Risk assessment model” AND (“softwaredevelopment” OR “software project”));

(“Risk Evaluation model” AND (“softwaredevelopment” OR “software project”));

(“Risk Estimation model” AND (“softwaredevelopment” OR “software project”));

(“Risk Measurement model” AND (“softwaredevelopment” OR “software project”)); and

(“Risk Consideration model” AND (“softwaredevelopment” OR “software project”))

Risk evaluation, estimation, measurement and consideration are chosen because it is synonyms with risk assessment which is widely used. These synonyms are included to ensure completeness in our search.

ii. Selection of primary studiesPrimary studies are selected based on search stringsand inclusion/exclusion criteria identified earlier,

iii. Study quality assessmentIn this step, quality of the selected studies is assessedby using a check list. The check list is adopted from[33].

iv. Data extraction & monitoringIn this step, data extraction forms will be used to recorddata from primary studies.

v. Data synthesisExtracted data are compiled and synthesized to addressthe objective of the SLR.

(c) The final part in SLR is Reporting the review. Reporting the review is an important step to communicate the result of SLR effectively. The result of SLR (the list of major elements and activities of software risk assessment models) will be reported in conference papers and PhD thesis.

B. Phase 2: Identification of Potential Usability Risk

The objective of this phase is to identify potential usability risk during SDLC. It involves 2 activities, Activity 2 and Activity 3.

Activity 2 involves identification of usability risk from usability attributes and literature. In 2012, Aman Kumar, Arvind and Hardeep [34] suggested that factors affecting the quality of software can be identified from attributes defined in software quality models. This study considers attributes of usability as factors in producing usable software products. These usability attributes are subjected to risks that a software product might have troubles in that area. Scenario which affects the ability to achieve these attributes is considered as potential usability risk during software development process.

Since this model focuses on developing usable software products and emphasizes on usability aspect in software in each phases of development process, the attributes and sub attributes defined by Alonso-Rios et al. [25], and later enhanced by Dubey et al. [26] will be used in this study.

Based on usability attributes, Effectiveness, Efficiency, Comprehensibility, Satisfaction and Safety, a list of potential usability risks is derived, as shown in Table 1

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

170

[24] Besides this, usability risks identified from literature are analyzed to identify more possible usability risks. Table 1: Potential Usability Risk derived from usability

attributes Attributes Potential Usability Risk

Effectiveness Low percentage of task accomplishment

Incorrect task execution

Incomplete functionalities to perform a task

Lack of cultural diversity in user interface

Inability to adapt to changing user preferences and environment

High ratio of failure/errors

• Human error

• Execution error

Lack of user control

Efficiency Incorrect or inaccurate result produced

Lack of utilization of command

Longer execution time of a task

Satisfaction Lack of software stability

Lack of trust on software

Lack of aesthetic features and good UI design

Comprehensibility

Lack of clarity in system’s properties and functionalities

Lack of skills for user and developer

Inadequate training for user

Longer time to learn the software

Incomplete and Inadequate documentation/ user manuals

Insufficient support system (help)

Safety

Loss of information/data

High vulnerability to threat

High prone to system failure/corruption

Environment prone to hazards

Change in environment

Activity 3 in this phase involves in designing and distributing questionnaires to the industry to identify usability risk from the perspective of industry. As a guide in designing questionnaire, seven stage processes are followed [36]:

i. Identify the objectives.

The objective of this questionnaire is to seek information in identifying potential usability risks that would impact software development process.

ii. Identify and characterize the target audience.Target audience for this survey are software developersin Malaysian Public Sector.

iii. Design the sampling plan.In this step, sample size of the target audience isestimated. Once the size of population is obtained, asoftware called G*Power is used to estimate the samplesize. Since the respondents for these surveys arecarefully chosen from the area of softwaredevelopment, non-probability samples will be used inthese surveys. The difference between non-probabilityand probability sampling is that non-probabilitysampling does not involve random selection andprobability sampling does.

iv. Design and write the questionnaire.In this step, questionnaire will be designed anddistributed to public sector to acquire data on potentialusability risks. This questionnaire is divided to twosections. The first section is contains questions relatedto demography and experience of respondents. Thesecond section includes thirty eight questions related tousability factors which could contribute to usabilityrisk.

v. Pilot test the questionnaire.Pilot test is carried out by distributing 30questionnaires to software developers. Their feedbackwhether the questions are easy to understand andwhether it is doable is gathered. The questionnaire isthen modified for distinguishing reliability andconstructs validity. Also, experts’ opinions areconsidered to implement in this survey as the contentvalidity.

vi. Distribute the questionnaire.After revising questionnaire, the questionnaire will bedistributed to the respondents in public sector throughonline survey, hardcopy and softcopy through email.

vii. Analyse the results and write a report.Results from the survey will be organized into tablesand analysed using SPSS statistical software.

Collection of usability risks identified from Activity 2 and Activity 3 will be considered as the potential usability risk that could arise during SDLC.

C. Phase 3: Analysis of Potential Usability Risk

The objectives of this phase are to analyze, classify and prioritize all identified usability risk using risk analysis, risk classification and risk prioritization

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

171

techniques. This phase contains three main activities, Activity 4, Activity 5 and Activity 6.

Activity 4 involves conduction of one-to-one meeting and discussion with experts to determine likelihood and impact of each usability risk towards phases in SDLC. Classification of usability risk based on SDLC phases is also determined in this session. Experts are selected carefully based on their experiences in software development and risk management. Experts with more than 5 years of experience are chosen. Initially, each expert will be given a brief explanation on the goal of the session and how the outcome of the session contributes to the development Usability Risk Assessment Model. Then, an interview session with the expert will be held using predetermine questions on determining likelihood, impact and classifications of usability risk based on SDLC phases.

Activity 5 involves analysis of result combining output from Phase 2 and Activity 4. The exposure level for each potential usability risk is calculated based on likelihood and impact of each usability risks. Each usability risk will be classified according to phases in software development lifecycle and correlation analysis will be done to determine relationship between potential usability risk and phases in software development lifecycle.

Activity 6 involves prioritization of each potential usability risk by sorting respective risk exposure level in descending order to know the impact of usability risk in each phase in software development lifecycle. Higher the risk exposure, more priority and attention should be given to that particular usability risk because it has more impact in creating less usable software product. Prioritization is important because it gives insight of critical and noncritical usability risk based on SDLC phases, among usability risks and in overall software product development.

The phases and activities in the formulation of Usability Risk Assessment Model are summarized in Figure 1.

IV. CONCLUSION AND FUTURE WORK

Usability is an important factor in developing a usable software product. Ignorance on usability standards and inability to overcome challenges in integrating usability practices has led to existence of many usability problems in software product.

Usability Risk Assessment Model uses the approach of risk management in dealing with usability problems before even it occurs. By identifying, analyzing and prioritizing usability risk during SDLC, usability risk can be eliminated during development process and this produces a more usable software product.

With this model, open system can be developed with more usability features and could enhance the communication between users.

In future, this model will be developed and evaluated using a tool. A case study will be conducted in Malaysian Public Sector for this purpose.

ACKNOWLEDGMENT This research is funded by the RU-Tier1 Research grant

of Universiti Teknologi Malaysia (UTM) in collaboration with

Phase 1: Identification of Element in Software Risk Assessment Models

Activity 1: Investigate existing software risk assessment models using Systematic Literature Review (SLR)

Elements in software

risk assessment models

Figure 1: Usability Risk Assessment Model Formulation Phases

Usability Risk Assessment Model

Activity 3: Design and distribute questionnaires to the industry

Activity 2: Identification of usability risk from usability attributes and literature

Phase 2: Identification of Potential Usability Risk

List of Potential Usability Risk

Activity 5: Result Analysis

Activity 4: Interview conduction with experts to determine • likelihood and impact of each usability risk• Classification of usability risk based on SDLC

phases

Phase 3: Analysis of Potential Usability Risk

Activity 6: Prioritizing risk exposure level by sorting it in descending order

Incorporation of usability risk to SDLC phases

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

172

the Malaysian Ministry of Education (MOHE) under the Vot no. 03H74. The authors would like to thank the Research Management Centre of UTM and MOHE for their support and cooperation including students and other individuals who are either directly or indirectly involved in this project.

REFERENCES [1] M. Xenos, “Usability perspective in software quality,” in Usability

Engineering Workshop, The 8th Panhellenic Conference on Informatics with International Participation, Southern Cyprus, November 2001.

[2] O. K. Hussain, T. S. Dillon, F. K. Hussain, and E. J. Chang, “Understanding Risk and Its Importance in Business Activities,” in Risk Assessment and Management in the Networked Economy, Springer Berlin Heidelberg, 2013.

[3] J. A. McCall, P. K. Richards, and G. F. Walters, “Factors in software quality,” General Electric, National Technical Information Service, 1977.

[4] S. U. Farooq and S. M. K. Quadri, “Quality Practices in Open Source Software Development Affecting Quality Dimensions,” Trends in Information Management, 7(2), 2012.

[5] L. Hua, and Y. Gong, “Usability Evaluation of a Voluntary Patient Safety Reporting System: Understanding the Difference between Predicted and Observed Time Values by Retrospective Think-Aloud Protocols,” in Human-Computer Interaction. Applications and Services, Springer Berlin Heidelberg, pp. 94-100, 2013.

[6] T. Jokela, “Performance rather than capability problems. insights from assessments of usability engineering processes,” in Product Focused Software Process Improvement, pp. 115-127, Springer Berlin Heidelberg, 2005.

[7] A. Seffah, M. Donyaee, R. B. Kline, and H. K. Padda, “Usability measurement and metrics: A consolidated model,” Software Quality Journal, 14(2), pp. 159-178, 2006.

[8] L. Liang, X. Deng, and Y. Wang, “Usability Measurement Using a Fuzzy Simulation Approach,” in International Conference on Computer Modeling and Simulation, 2009. ICCMS'09, IEEE, pp. 88-92, February 2009.

[9] W. A. M. Isa, M. R. Suhami, N. I. Safie, and S. S. Semsudin, “Assessing the usability and accessibility of Malaysia e-government website,” American Journal of Economics and Business Administration, 3(1), pp. 40-46, 2011.

[10] S. Lee and J. E. Cho, “Usability evaluation of Korean e-government portal,” in Universal Access in Human-Computer Interaction, Applications and Services, Springer Berlin Heidelberg, pp. 64-72, 2007.

[11] M. Abdul Aziz, W. A. R. Wan Mohd Isa, and N. Nordin, “Assessing the accessibility and usability of Malaysia Higher Education Website,” in 2010 International Conference on User Science and Engineering (i-USEr), IEEE, pp. 203-208, December 2010.

[12] E. N. Asiimwe and N. Lim, “Usability of government websites in Uganda,” Electronic Journal of e-Government, 8(1), pp. 1-12, 2010.

[13] J. Sauro, “How common are usability problems,” http://www.measuringusability.com/problem-frequency.php, September 2010.

[14] G. Lindgaard, “Usability testing and system evaluation: A guide for designing useful computer systems,” London: Chapman & Hall, pp. 221-246, 1994.

[15] P. Carlshamre, and M. Rantzer, “Business: Dissemination of Usability: Failure of a Success Story,” Interactions, 8(1), pp. 31-41, 2001.

[16] A. Seffah, R. Djouab, and H. Antunes, “Comparing and reconciling usability-centered and use case-driven requirements engineering processes,” In Australian Computer Science Communications, IEEE Computer Society, Vol. 23, No. 5, pp. 132-139, January 2001.

[17] Q. S. Durrani and S. A. Qureshi, “Usability Engineering Practices in SDLC,” 2012.

[18] H J. Heiskari, M. Kauppinen, M. Runonen, and T. Mannisto, “Bridging the gap between usability and requirements engineering,” 17th IEEE International in Requirements Engineering Conference, 2009. RE'09, pp. 303-308, August 2009.

[19] S. R. Humayoun, “Incorporating usability evaluation in software development environments, ” KI-Künstliche Intelligenz, 26(2), pp. 197-200, 2012.

[20] C. Lallemand, “Toward a closer integration of usability in software development: a study of usability inputs in a model-driven engineering process,” in Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems, ACM, pp. 299-302, June 2011.

[21] International Organization for Standardization, “ISO 13407: Human-centred design processes for interactive systems,” 1999.

[22] A. B. Platt, “The usability risk,” in Proceedings of the 18th IEEE Symposium on Reliable Distributed Systems, IEEE, pp. 396-400, Proceedings of the First Asia-Pacific Conference on Quality Software, IEEE, 297-305, 1999.

[23] R. B. Grady, and D. L. Caswell, “Software metrics: establishing a company-wide program,” 1987.

[24] J. T. Sambantha Moorthy, S. Ibrahim, and M. Naz’ri Mahrin, “Identifying Potential Usability Risk During Software Development Process,” Open International Journal of Informatics, 1, 29-45, 2013.

[25] D. Alonso-Ríos, A. Vázquez-García, E. Mosqueira-Rey, and V. Moret-Bonillo, “Usability: a critical analysis and a taxonomy,” International Journal of human-computer interaction, 26(1), 53-74., 2009.

[26] S. K. Dubey, A. Gulati, and A. Rana, “Integrated Model for Software Usability,” International Journal on Computer Science and Engineering, 4(3), 2012.

[27] S. W. Foo, and A. Muruganantham, “Software risk assessment model,” in Proceedings of the 2000 IEEE International Conference on Management of Innovation and Technology, 2000. ICMIT 2000, IEEE, Vol. 2, pp. 536-544, 2000.

[28] D. Gupta, and M. Sadiq, “Software risk assessment and estimation model,” in International Conference on Computer Science and Information Technology, 2008. ICCSIT'08, IEEE, pp. 963-967, August 2008.

[29] B. Shahzad and A. S. Al-Mudimigh, “Risk Identification, Mitigation and Avoidance Model for Handling Software Risk,” in 2010 Second International Conference on Computational Intelligence, Communication Systems and Networks (CICSyN), IEEE, pp. 191-196, July 2010.

[30] M. Sadiq, M. K. I. Rahmani, M. W. Ahmad and S. Jung, “Software risk assessment and evaluation process (SRAEP) using model based approach,” in 2010 International Conference on Networking and Information Technology (ICNIT), IEEE, pp. 171-177, June 2010

[31] Y. Bazaz, S. Gupta, O. Prakash Rishi and L. Sharma, “Comparative study of risk assessment models corresponding to risk elements,” in 2012 International Conference on Advances in Engineering, Science and Management (ICAESM ), IEEE, pp. 61-66, March 2012.

[32] H. M. Mofleh and A. Zahary, “A Framework For Software Product Risk Management Based On Quality Attributes And Operational Life Cycle (SPRMQ),” Risk, 1, 3, 2010.

[33] B. Kitchenhama, “Guidelines for performing systematic literature reviews in software engineering (version 2.3),” Software Engineering Group, School of Computer Science and Mathematics, Keele University and Department of Computer Science, University of Durham, 2007.

[34] A. K. Sharma, A. Kalia and H. Singh, “An Analysis of Optimum Software Quality Factors,” IOSR Journal of Engineering, 2(4), pp. 663-669. 2012.

[35] M. Kasunic, “Designing an Effective Survey (CMU/SEI-2005-HB 004),” Retrieved December 10, 2012, from the Software Engineering Institute, 2005.

2013 IEEE Conference on Open Systems (ICOS), December 2 - 4, 2013, Sarawak, Malaysia

173