A Software Tool to Assist Scenario Testing of Websitescourses.cecs.anu.edu.au/courses/CS_PROJECTS/10S2/Reports/Siqi Fan.pdf · A Software Tool to Assist Scenario Testing of Websites

SCHOOL OF COMPUTER SCIENCE

ANU COLLEGE OF ENGINEERING & COMPUTER SCIENCE

A Software Tool

to Assist Scenario Testing of

Websites

Student: Siqi Fan (u4761704)

Supervisor: Assoc Professor Chris Johnson

Course Code: COMP8790 – Software Engineering Project

31th October, 2010

A Software Tool to Assist Scenario Testing of Websites

Pag

e1

Pag

e1

Acknowledgement

First and foremost I would like to offer my sincerest gratitude to my supervisor,

Assoc Professor Chris Johnson, who has supported me throughout of my thesis with

his patience and knowledge. In the various research works and laboratories along the

entire project, I have been also aided in setting the research direction by him. Without

Chris Johnson‟s kind support and encouragement this thesis would not have been

possible.

Secondly, I would like to express my thankfulness to all my previous colleagues

in the Office International Publishing Group (OIPG), Microsoft Ireland. It was them

who created the heuristic work environment and offered me the opportunity to receive

the fundamental knowledge and experience of a large scale business oriented website

production.

Finally, I would like to thank my beloved parents, wife and friends for their

understanding and endless love throughout all my studies.


Pag

e2

Pag

e2

Abstract

Effective prioritization on tremendous amount of defects can ease the pain for

large scale business oriented websites‟ production. The severity of defects plays a

critical role in assisting defect prioritization. However, the traditional way of ranking

severity lessens the significance of the original purpose, especially facing a large

amount of defects with limited developing and testing resources. A thorough study has

been done for discovering the way to represent severity in a more distinguishable

form and the way to estimate severity in a more systematic manner. The analysis

result shows that three factors (impact, importance and likelihood) are identified as

having significant referential value in evaluating severity but these factors are

measured according to different perspectives (scenario, quality and usage). Each

perspective has been further studied respectively, especially the two aspects of

website quality and their implicit gap. To consider these views while assessing

severity in a systematic manner, a theoretical framework is proposed to estimate

defect severity by utilizing the structural view of predefined scenarios and deriving

testing tasks from scenario related quality dimensions. The three sub factors‟

classification scheme and the severity representation are defined to assist the

framework to represent severity in a more distinguishable form. A prototype is built to

prove the feasibility to adopt the framework into the practical website scenario testing

tools.


Pag

e3

Pag

e3

Table of Contents

Acknowledgement ........................................................................................................ 1

Abstract ......................................................................................................................... 2

1. Introduction .......................................................................................................... 4

2. Severity and its three sub factors ........................................................................ 7

2.1. Impact on Scenario ..................................................................................................... 8

2.1.1. Case Study .......................................................................................................... 8

2.1.2. Scenario ............................................................................................................ 11

2.2. Importance ................................................................................................................ 13

2.2.1. Quality .............................................................................................................. 13

2.2.2. Case Study ........................................................................................................ 22

2.3. Likelihood ................................................................................................................ 24

2.3.1. Case Study ........................................................................................................ 26

3. Theoretical Framework and Severity Classification Scheme ......................... 26

3.1. Building the Framework........................................................................................... 27

3.2. Severity Classification Scheme ................................................................................ 31

3.2.1. Impact ............................................................................................................... 32

3.2.2. Importance ........................................................................................................ 33

3.2.3. Likelihood ......................................................................................................... 34

3.2.4. Representation of severity ................................................................................ 34

3.3. Case Study ................................................................................................................ 34

4. Prototype ............................................................................................................. 40

5. Results and Discussion ....................................................................................... 43

6. Related Works..................................................................................................... 51

7. Conclusion and Future work ............................................................................. 61

8. Bibliography ........................................................................................................ 62


Pag

e4

Pag

e4

1. Introduction

Nowadays business oriented websites are becoming increasingly complex in three

dimensions, quantity of content, functionality of features, and diversity of language,

while the production of this kind of website is experiencing painfully. The content of

the website needs to be frequently added, removed, and updated for providing

up-to-date information. The features of the website need to be enriched for enhancing

interaction with site users. The languages supported by the website need to be

increased for expanding the global business. The trend can be driven by various

business objectives not mentioned above.

However, this paper is not meant to evaluate or to predict the trend but to focus

on what is happening behind the scene. The production of this type of websites in

industry is becoming to be an endless cycle. The business analysts add and track the

customer requirements aligning to the business strategies. When a new requirement is

added, the authors create the new content or update the existing content. Meanwhile

the product manager and website designers design the new features and refine the

website layout and theme. The developers then develop and maintain the site features

and content publishing tools. The testers launch testing when the new version website

is assembled. They assign severity to each defect caught during testing and report the

defects to the project managers. The project managers then prioritize the defects and

assign the defects to the developers. The developers pick up the defects to investigate

and to fix according to the priority level set by the project managers. When a defect is

assumed to be fixed, the developer will assign the defect back to the testers for

verifying. The testers will close the defect if it is fixed or otherwise reassign back to

the developer. When the scheduled releasing date is approaching, the project

managers will triage the open defects and reprioritize them to avoid delivering critical

defects to the users. Although all seems to be blameless on the surface so far, it is

worth having a closer look at what they are suffering from.


Pag

e5

Pag

e5

During the production, defects are often captured and counted in a significantly

high volume. The previous version Office Online of Microsoft Office.com [19]

reported more than 15,000 defects during 3 years production. Generally the reason is

not just because of the poor production activities, unavoidable human errors, or low

quality production tools but the accumulatively nature (If the defects cannot be fixed

by developers quick enough before the testers discovering new defects, the total

amount of defect count will be accumulated into a very high number.) and the fast

change of the business needs. High volume of defects‟ occurrence cannot help but

leave panic to the developers. Many high priority defects flowing into developers‟ bug

account soon bewilder the developers. “They are all high priority defects and which

one should I choose to fix at first?” Hearing such a voice, the questions need to be

answered that why the priority loses its meaning and how to get the defects truly

prioritized.

To answer the question, it is necessary to go back to see where the priority is

coming from. Priority is set with main consideration in severity in addition to taking

some other project factors into consideration such as impact on business value,

available developing and testing resources, business decisions and so on. Priority is

ranked subjectively to describe the relative order the defect should be investigated and

resolved. While severity is set according to certain guidelines defined in the

organizations after evaluating the defect usually from website quality perspective.

Severity is ranked to describe how serious the impact of the defect is on the site users.

There is no standard to instruct who should set severity and priority. In industry,

severity is usually ranked by testers and priority is set by project managers.

Although the priority is assigned to the defects, the situation occurs often that

many defects cannot be fixed when scheduled release date is approaching. Commonly

the cause is not inefficiency of the development team or the immature technical skill

level but the facts that defect fixing rate is lower than the defect reporting rate and

ineffective prioritization.


Pag

e6

Pag

e6

If there is no way to avoid the situation, there is at least some way to ease the pain.

Effective prioritization can assist developers to tackle the most important defects

within limited resources rather than placing the triage team in a dilemma. The severity

of defects is the key in deciding priority. Although it is not the only factor in

determining priority, severity should provide the greatest reference value to other

factors. In order to make it be fuller in meaning, severity should be represented in a

more distinguishable form and set in a more systematic manner.

Following the intuitive idea, this paper adventures an opinion to decompose

severity into three sub factors impact, importance, and likelihood based on three

circumstances observed. The three factors and cases are considered in detail in section

2. Impact in this context is the consequence brought from the defect upon the process.

The process is one particular users‟ activity on the website, to fetch information (for

example, read news), to receive services (for example, order product), and so on.

Fortunately, using a scenario is probably the best way to describe the process.

Scenario is an ordered set of interactions between the website user and the

information and the functionalities (features) of the website. Scenario becomes the

author‟s research focus. Its importance is that scenario can provide a systematic

overview of the website, which offers great support to judge the impact objectively.

On the other side, importance is to describe the relative order of quality factors. This

relationship is emphasized and applied to single scenario after researching website

quality. It is discovered that website quality has two aspects, the customers' quality

expectation and the website providers' quality expectation. These two aspects and the

gap between them are often blurred. This paper proposes a way to narrow the gap by

defining the deliverables according to quality factors and associating the testing tasks

with the deliverables. It assumes the failure of certain testing affect the corresponding

quality factor directly. Hence, the relative quality impact of the defect can be

objectively measured. Last but not least, likelihood is inspired by risk analysis to

reflect the end user impact scope of defects. Existing website users‟ behavior


Pag

e7

Pag

e7

statistical analysis is feasible to deliver the objective justice. It is suggested to

consider the factor for justifying the severity level. A theoretical framework is

proposed to conduct three sub factors to be considered into scenarios. The framework

includes predefining the testing tasks for each step of the scenario, evaluating the

three sub factors based on the classification scheme before testing, and representing

defect severity in a more distinguishable form. A prototype is implemented and the

result proves the feasibility to adopt the framework into practical use.

The rest of paper is structured as follows. The theoretical framework, the details

of the classification scheme of three sub factors, and the severity representation are

discussed in section 3. The entire framework is adopted and implemented in a

prototype introduced in section 4. The implementation result of the prototype is

discussed in section 5. The related work from other researchers is commented in

section 6. The paper is summarized and the future work is pointed out in section 7.

The scenario cases in this paper are selected from a specific website Microsoft

Office.com [19].

The objective of the present research is to build a theoretical framework for

taking good advantage of the systematicness of scenario to evaluate severity of defect

objectively. It is meant to light up the direction for further investigation on the areas

such as identifying the sub factors of severity, narrowing the gap between customers‟

quality expectation and website providers‟ quality expectation, identifying the

deliverables and the testing tasks, and adopting the framework into practical tools.

2. Severity and its three sub factors

IEEE standard 1044-2009 defines “Severity is the highest failure impact that the

defect could (or did) cause, as determined by (from the perspective of) the

organization responsible for software engineering” [0]. Different organizations have

different understanding of severity. It is easy to understand as different projects and


Pag

e8

Pag

e8

products have different objectives and orientations. In a project building a human life

critical system, severity of defects is to measure the impact on the end users. In a

project building a complicated workflow product, severity of defects is to measure the

impact on the operational system. Whereas in a project that building a large scale

business oriented website, severity of defects is suggested to measure the impact on

the scenarios, the importance of the website quality dimensions, and the usage of the

pages and their elements. This section will guide you to understand the points listed

below:

Why these three factors need to be considered?

How is impact measured?

Why scenario plays a critical role in measuring severity?

What is the relationship between scenario and three sub factors?

Why scenario testing is the potential testing activity to measure severity?

What are the two aspects of website quality?

What is the order relationship of website quality factors?

How to narrow the gap between the two aspects of website quality?

How is importance measured?

How is likelihood measured?

2.1. Impact on Scenario

Impact is an ambiguous term. Before determining what object the impact should

be upon, let us consider a situation.

2.1.1. Case Study

A defect is captured during testing that the “Download” button as pointed by the

black arrow in Figure 1 does not behave properly when user clicking the button. The

image selected by the user cannot be downloaded. Another defect is captured that the

“Download” button as pointed by the black arrow in Figure 2 is out of function as


Pag

e9

Pag

e9

well. The PowerPoint template selected by the user cannot be downloaded. The

traditional way of evaluating severity will rank both defects in the same level that

describe the defect stop the downloading operation. However, if the defects are

considered in a scenario context, the relative order can be distinguished.

Let us briefly describe the scenarios for the both cases. The first scenario is a user

would like to find an image from the website and to use the image in his PowerPoint

slides. He browses the image pages and selects the image he likes. (1) He clicks the

“Copy to Clipboard” button on the page and to manually paste the image in his slides.

(2) He clicks the “Download” button on the page, saves the image on the local disk,

and inserts the image manually into his slides. (3) He clicks the “Add to Basket”

button on the page and decides to search for more similar images. He is not happy

with other images and clicks the “download” link in the pop up dialog illustrated in

Figure 3, saves the image on the local disk, and inserts the image manually into his

slides. The second scenario is a user would like to find a PowerPoint template and to

use the template in his PowerPoint application. (1) He clicks the “Download” button

on the page, saves the template on the local disk, and the template is automatically

loaded in his PowerPoint application. As you can see from the first scenario, there are

three paths to achieve the user‟s goal. In another word, there are two ways to walk

around if the “download” button does not work. However, in the second case, the

broken “download” button will block the scenario that the user will have no way to

achieve his goal. Therefore, should it be reasonable to differentiate the relative order

of two defects that the broken “download” button in the “templates” page should be

investigated and resolved earlier than the one in the “images” page.


Pag

e10

P

age1

0

Figure 1 the "Download" button is reported as a defect in the "images" sub site.

Figure 2 the "Download" button is reported as a defect in the "templates" sub site.


Pag

e11

P

age1

1

Figure 3 Dialog is popped up after click “Add to Basket” button

2.1.2. Scenario

From the case study above, it is noticeable that the sub factor impact is measuring

upon the scenario. Considering impact on scenario gives a big lift on differentiate the

relative order of defects.

IEEE Standard 1044-1993 [1], the ancient version of IEEE Standard 1044-2009,

once mentions “The impact of an anomaly shall be considered at each step of the

anomaly process.” Process is the word describing a formal path in a limited scope for

the large scale business oriented website, a derivative of traditional software or web

application. The large scale business oriented website is a platform of information

exchanging and services providing. The user or the customer has freedom to use the

website (for examples, jumping between links, querying the searching engines,

interacting with the controls and so on) to achieve his purposes or leave alternatively

to visit other websites. Like a customer in a supermarket, he has many ways (for

example, following the promotion advertisements, searching by category signs, asking

for assistance, wandering around and so on) to find the products he want or to leave

for other shops. Focusing on a single path (process) is not enough to justify the impact


Pag

e12

P

age1

2

from a systematic view.

From the other hand, using scenarios is probably the best way to describe how the

user will access the information and the functionalities (features) of the website to

illustrate the entire picture. Scenario is reviewed thoroughly in the section 7 related

works. There are two key points in Ryser‟s and Glinz‟s work [8] need to be

emphasized. One is the predefined scenarios during requirement elicitation and

analysis phase have the critical value for requirement validation and verification.

Another is scenarios can help to choose appropriate testing strategy and testing

techniques to make decisions in allocating resource in testing plan.

In the first point, the customer requirements include not only the functional but

also the non-functional requirements. And the non-functional requirements in natures

are the customer quality expectations which will be discussed in the section 2.2.

Hence, ideally, a testing activity should cover both functional and non-functional

testing. The suitable testing phases [2] are System Testing, User Acceptance Testing,

and Pilot Testing. In this paper, System Testing phase is only considered as the phase

is the earliest stage which is capable of launching both functional and non-functional

testing. The typical testing activity in System Testing is the scenario testing. Literately

speaking, the scenario testing is tests based on scenarios, the hypothetical stories [4].

The test cases are built based on the scenarios. Each step in the test case can be

matched with the user behavior on the website (for example, click a link, click a

button, and many other mouse events and control interactions). Details of scenario

testing will be discussed in section 3.

In the second point, it implicitly emphasizes that scenarios provide a systematic

and structured view so that the right testing strategy can be selected, the proper testing

techniques can be applied, and the reasonable effort can be allocated. Regarding the

effort allocation, it is related to another sub factor likelihood which will be discussed

in section 2.3.

To sum up, this section analyzes a real life example to reveal the relative order for


Pag

e13

P

age1

3

defect handling through considering the defect impact on the scenario. Meanwhile,

scenario testing and the interrelationship between scenario and the other two sub

factors are introduced with further research into scenario.

2.2. Importance

Defects have implicit relationships with quality. In industry, it is a common

understanding that defects with higher severity level have deeper impact on quality.

Although the statement is not wrong, it lacks transparency of the inside view of the

relationship between severity and quality. This section will review the two aspects of

quality, the website quality dimensions, the ordering relationship of website quality

dimensions, and the approach to associate the ordering relationship with severity

measuring.

2.2.1. Quality

Business oriented websites play center role to the business success. High quality

website leads to customer satisfaction which is essential for customer to open a door

to commit to the company or organization. Therefore the quality of the website needs

to be seriously concerned.

After researching on website quality, it is discovered that website quality has two

aspects, the customers' quality expectation and the website providers' quality

expectation, which are not often distinguished and discussed. The customers‟ quality

expectation is subjective to the website design, the content and features on the website,

and the personally experiences of using the website. Loiacono, Watson and Goodhue

identified 13 quality dimensions listed in the Table 1 in their study [14]. The business

analysts are responsible for gathering the customer quality expectation (for example,

via surveys or interviews to conduct marketing research) and integrating the quality

expectation into the requirements.


Pag

e14

P

age1

4

Quality

Category

Quality

Dimension Description

Ease of use Ease of

understanding

Easy to read and understand.

Intuitive

operations

Easy to operate and navigate.

Usefulness in

gathering

information

Information

Quality

The concern that information provided is accurate,

updated, and appropriate.

Tailored

Communications

Communications can be tailored to meet the user‟s

needs.

Usefulness in

carrying out

transactions

Informational

fit-to-task

The extent to which users believe that the Web site

meets their needs.

Response time Time to get a response after a request or an

interaction with a Web site.

Trust Secure communication and observance of

information privacy.

On-line

completeness

Allowing all or most necessary transactions to be

completed on-line (e.g., purchasing over the Web

site).

Relative

Advantage

Equivalent or better than other means of interacting

with the company.

Consistent image The Web site does not create dissonance for the user

by an image in compatible with that projected by the

firm through other media.

Entertainment

Value

Visual appeal The aesthetics Web site.

Innovativeness The creativity and uniqueness of a Web site.

Emotional appeal The emotional effect of using the Website and

intensity of involvement.

Table 1 Quality Dimensions of Website [14]

From the other hand, the website providers' quality expectation is normally

defined by the website producer who assumes the minimum acceptable quality for the

customers according to the requirements. As you can see here, the requirements are

the common understanding between the customer and the producer. To satisfy the


Pag

e15

P

age1

5

customer quality expectation, the producer need to define the testing strategy against

the quality documented in the requirements. However, the quality dimensions are still

too abstract to identify the exact testing tasks. How can those testing activities such as

the unit testing, the functional testing and the integration testing guarantee to fulfill

the customer perception quality? Before answering this question, let us have another

interesting point of view found during research.

Zhang and Dran in their studies identify the importance order of the website

features and interfaces [11]. More information regarding Zhang‟s and Dran‟s studies is

detailed in the section 7 related works. They considered the forms of quality factors as

the groups of website features and interfaces listed in Table 2, namely the quality

feature cluster listed in Table 3, which might not be a bad idea to give a concrete

representation of the customer perception quality. Their studies have two important

conclusions strongly relating to the topic of this paper, (1) the quality feature clusters

are unlikely to have the same importance in different website domains; for instance,

the five most important quality factors in its order are D03, D02, D07, D01 and D10

in a Finance domain website; while the five most important quality factors in its order

are D08, D04, D07, D06 and D13 in an Entertainment domain website; (2) the

importance order of the quality features are likely to change as time goes on; for

instance, when the business requires an online ordering feature to be added into a

product demonstration website, D12 Security/Privacy surely will increase its

importance position in its original order list.

Category

Feature

ID (FID) Features

C1 Information

content

F1-1 Information on Web site stays for a reasonable period of

time before it disappears.

F1-2 Absence of improper materials.

F1-3 Accurate information.

F1-4 Appropriate detail level of information.

F1-5 Up-to-date information.

F1-6 Relevant information.


Pag

e16

P

age1

6

F1-7 Complete coverage of information.

F1-8 Content that supports Web site‟s intended purpose.

F1-9 Controversial materials.

F1-10 Novel (new) information.

C2 Cognitive

outcomes

F2-1 Learned new knowledge and/or skills by using Web site.

C3 Enjoyment F3-1 Use of humor.

F3-2 Multimedia.

F3-3 Fun to explore.

C4 Privacy F4-1 Access requirements (e.g., pay a fee, sign on, enter a

password, and provide some private info before one can

access info).

F4-2 Authorized use of user‟s data for unanticipated purposes.

F4-3 Authorized collection of user data.

F4-4 Assurance that user-entered data is encrypted.

C5 User

empowerment

F5-1 User controls order or sequence of information access.

F5-2 User controls how fast to go through Web site.

F5-3 User controls opportunities for interaction.

F5-4 User controls complexity of mechanisms for accessing

information.

F5-5 User controls difficulty level of information accessed.

C6 Visual

appearance

F6-1 Attractive overall color use.

F6-2 Sharp displays.

F6-3 Visually attractive screen layout.

F6-4 Attractive screen background and pattern.

F6-5 Adequate brightness of screens/pages.

F6-6 Eye-catching images or title on homepage.

C7 Technical

support

F7-1 Indication of system loading/responding time.

F7-2 Support for different platforms and/or browsers.

F7-3 Stability of Web site availability (can always access Web

site).

C8 Navigation F8-1 Indication of user‟s location within Web site.

F8-2 Navigation aids.

F8-3 Directions for navigating Web site.

C9 Organization

of information

content

F9-1 Presence of overview, table of contents, and/or

summaries/ headings.

F9-2 Structure of information presentation is logical.

C10 Credibility F10-1 Reputation of Web site owner.

F10-2 External recognition of Web site (e.g., site has won

awards, number of times it has been visited).

F10-3 Identification of site owners/designers.

C11 Impartiality F11-1 Unbiased information.


Pag

e17

P

age1

7

F11-2 Absence of gender or racial/ethnic biases and stereotypes.

Table 2 Web Site Features [11]

ID Family Definition

D01 Accuracy No errors, correct, exact, precise, right, true

D02 Completeness

/Comprehensiveness

of Information

Large in scope or content, contains variety of information

or sources

D03 Currency /timeliness

/Update

Information is current, up to the moment, real-time, timely

D04 Engaging Cognitive advancement, emotional connections, personal

expressions

D05 Information reliability

/reputation

Information dependable, condition of being held in high

esteem, authoritative, good reputation of information

source

D06 Information

representation

The way information is presented, maybe in different

format/media, customized displays

D07 Navigation Features to make navigation possible, site maps

D08 Visual design Visual appearance

D09 Product and service

concerns

Features concerned with products/services offered/sold

through Website, not about site itself; price and

availability of products/services

D10 Readability

/Comprehension

/Clarity

Ability to comprehend meaning of written or printed

words or symbols, to perceive or receive well

D11 Relevant Information Information that directs to the point, having to do with

matter at hand

D12 Security /Privacy Confidentiality of information, things that give or assure

safety and guarantee

D13 Site accessibility

/responsiveness

Being able to access website, responsiveness of site to

user's request in terms of time

D14 Site technical features Such features as search tools, downloadable

(printer-friendliness), chat rooms.

Table 3 Quality Feature Clusters [11]

The interpretation of the first conclusion is the interface elements on certain

webpage (for example, links, text, images, media, controls and so on) and the

functions (invoked by the elements‟ events) as the essential parts of the website

features are not equally important to the customers. In another word, the defects


Pag

e18

P

age1

8

associated with the interface elements or the functions are not equally important to the

customers.

Similarly, it can be stated that website quality dimensions might not be equally

important. This can be proved by building the linkage between quality factors and

quality dimensions. The general approach proposed in this paper is mapping the

quality feature clusters (introduced in Zhang„s and Dran‟s studies and listed in Table 3)

to the website quality dimensions (introduced in Loiacono‟s, Watson‟s and Goodhue‟s

studies listed in Table 1). For example, the feature cluster “Navigation” is able to

deliver the quality dimension “Intuitive operations” to the website users. The mapping

result is shown in Table 4(a) and 4(b). Table 4(a) and 4(b) show what feature clusters

can directly deliver the quality dimension, although there is no feature cluster

identified for the quality dimensions “Consistent image” and “Innovativeness”. This

linkage is important as the abstract customer quality expectation has a relative

concrete form. Since the relative order of feature clusters is identified by Zhang and

Dran, the relative order of quality dimensions can be determined by the linked feature

clusters.


Pag

e19

P

age1

9

Table 4(a) Mapping Quality Feature Clusters to Quality Dimensions


Pag

e20

P

age2

0

Table 4(b) Mapping Quality Feature Clusters to Quality Dimensions


Pag

e21

P

age2

1

Another key point embedded in the first conclusion is the importance order does

not simply depend on the website domain as the large scale business oriented website

may have many sub domains such as product demonstration, product promotion,

product online shopping, product support, corporation introduction, business

cooperation and so on. And this can further breakdown even to a single page or a

response to an event (For example, AJAX technology [3] allows a part of page to be

updated.). This paper suggests considering the scale as a single step in the context of

scenarios as scenarios describe the use of the website and the customer‟s purpose of

using the website. For example, an online shopping scenario might involve a couple

of steps. The first few steps are about viewing products and adding products into the

basket (for instance, by clicking “add” button). The last few steps are about

processing order (for instance, by clicking “confirm” button) and getting confirmation

message. The tolerance of response time (D13) between earlier and later steps is

different. The customer expects to view product information with a quick response

(for example, click a product image and receive the production information page

within a second). Otherwise the longer loading time will be likely to lose the

customers‟ patience. To the later steps, security (D12) is more expected than response

time (D13). Hence, as long as the step in the scenario changes, the relatively

importance order of the quality dimension will change.

Now, let us come back to the previous question that how to guarantee the testing

activities would fulfill the customer perception quality. There might be no way to

guarantee at all because some quality dimensions such as Consistent Image and

Innovativeness in Table 4(b) are purely subjective to the customers and some quality

dimensions are not well covered although some feature clusters have been mapped to

the dimensions in Table 4(a) and Table 4(b). The gaps exist that further research work

is potentially needed on these areas. However, this paper proposes a naïve approach to

build the linkage between quality dimensions and testing tasks and to narrow the gap.

This approach is aimed to identify the deliverable corresponding to certain quality


Pag

e22

P

age2

2

dimension, to determine whether the deliverable can be tested and measured, to define

the test actions if the deliverable is testable, and to consider the way to test and

whether the testing tasks will be included in the scenario testing. Table 5 illustrates the

result of applying this approach to link the quality dimension “Intuitive operation” to

its testing tasks. The general step is to extend Table 4(a) and Table 4(b) by mapping

the features in Table 2 to the corresponding feature cluster. In Table 2, F8-1, F8-2, and

F8-3 are the features found belong to the feature cluster D07 Navigation. Hence, these

features are treated as deliverable in Table 5. Then each deliverable is analyzed what

associated interface elements or functions are testable. In Table 5, it shows that the

breadcrumb is the testable interface element associated with the feature F8-1 and the

test action is to verify breadcrumb is indicating the correct location on the webpage.

The last four columns are to label whether this testing task can be executed

automatically, or manually, or not sure and whether this testing task need to be

included in the scenario testing. Again, the feature items in Table 2, the testing task,

the testing mode are open to discuss and redefine.

Table 5 Mapping Deliverable and Testing Task onto Quality Dimension

2.2.2. Case Study

Let us have a real case study to consolidate the idea of considering the

importance of quality factors during measuring severity. The defects are captured on

the Visio article page that the side navigation bar links are not translated in Arabic and


Pag

e23

P

age2

3

their links are invalid. The defects are pointed by black arrow in Figure 4 and its

original U.S. English version is shown in Figure 5. The purpose of the scenario is to

provide product information to Arabian website users. Should the engineers launch

content re-localization process first or correct the hyperlinks first? As it is easy to see,

the quality dimension D06 information representation is more important than D07

navigation at this step when the user browse to the article page. The untranslated sub

navigation bar links will disable the sub navigation functionality. Similarly, what if

the article itself is not translated, which content should be re-localized first?

Figure 4 untranslated side navigation bar links on Visio article in Arabic


Pag

e24

P

age2

4

Figure 5 the original version in U.S. English

To sum up, this section explains that the importance of quality should also be

considered when measuring the defects and the considering should be based on the

scenarios.

2.3. Likelihood

Likelihood is inspired from risk analysis and found it usefulness to assist to

prioritize, especially at the testing and developing resource shortage time. There is a

concept here illustrated in Figure 6 needed to introduce first. It is called the visit trip

of the website. The visit trip is counted when a user enters the website (regardless

whether the entrance is the homepage of the website or any other page of the website)

till the user leaves the website (regardless whether the user turns off the browser or

jumps to the other site). A visit trip could be a single page trip which only one page of

the website is visited or a multiple pages trip which at least two pages of the website

is visited. Likelihood in the context of measuring severity is equivalent to the

percentage of the page (on which the deliverables are provided) occurring in the visit


Pag

e25

P

age2

5

trip (page trip rate) multiply the percentage of the deliverable used on that page (hit

rate). Likelihood can be represented in the following formula where P(page) is the

probability of the page occurring in all visit trips (page trip rate) and P(deliverable) is

the probability of the deliverable (the associated interface element or function) is used

on the page (hit rate*):

Likelihood = P(page) × P(deliverable)

or

Likelihood = Page Trip Rate × Hit Rate

* The hit rate of the layout, the strings, and the images are 100%.

Figure 6 single and multiple trips

The information of P(page) and P(deliverable) can be gathered from the website

statistical analysis service provided by third party such as Webtrends [5]. The

statistical analysis is able to tell not only the percentage of usage of certain elements

(for example, links and buttons) on a single page, but also how visitors move through

the website (This information can help to validate the scenarios defined during

requirements and refine the scenarios timely.), where the user tends to drop out. There

is no reason to fix the defects that unlikely to be hit by the user. It is no surprise to

know that the usage rate of the scenarios is changing from time to time (For example,

when a new product is going to be release, the usage of seeking for product

information will increase.). It is even interesting to know that the usage rate of the

elements on a single page is changing from time to time. The typical case will be

discussed in the following section 2.3.1 Case Study. Therefore, the usage of the


Pag

e26

P

age2

6

elements or functions or the usage of the scenarios can differentiate the relatively

order of the defects.

2.3.1. Case Study

A typical case is the link “Calendars” pointed by the black arrow in Figure 7 start

to receive much higher hit rate than usual when the New Year is approaching.

Meanwhile the page trip rate is increasing. This phenomenon may occur when some

special events such as season, festival, social, news, and even fashion are triggered.

Defects caught during the time critical period should be seriously considered to fix

with high priority.

Figure 7 the link "Calendars" on the sub homepage "Template"

3. Theoretical Framework and Severity Classification Scheme

In section 2, the three sub factors of severity are discussed and their related cases


Pag

e27

P

age2

7

are reviewed. During the discussion, a series of extended topics such as scenario,

scenario testing, website quality, deliverable and testing tasks are elicited but they are

disconnected. This section is going to walk you through to connect all of them to

build a comprehensive framework which turn severity to be in a more distinguishable

figure and be set in a more systematic and objective manner. This framework is the

principle of the prototype discussed in section 4.

3.1. Building the Framework

The two aspects of website quality are revealed that the requirements are the

common understanding between the customer and the website provider. The model is

illustrated in Figure 8.

Figure 8 requirements and quality expectations

However, the gap between customer‟s quality expectation and website provider‟s

quality expectation exists because the customer‟s quality expectation is too abstract,

shown in Figure 9.


Pag

e28

P

age2

8

Figure 9 the gap between different roles’ quality expectations

.

Scenarios can be derived from the requirements and are found its usefulness in

differentiating the impact of defects, shown in Figure 10.

Figure 10 Scenarios are derived from Requirements

The relatively importance order of quality dimensions is found by mapping the

quality feature clusters to the quality dimensions, shown in Figure 11. Meanwhile the

quality feature clusters offer a more detailed view of customer‟s quality expectation.


Pag

e29

P

age2

9

Figure 11 Mapping Feature Cluster on Quality Dimensions

Nevertheless, the feature clusters are still too abstract to be measured objectively.

The deliverable is suggested to be identified, which is shown in Figure 12.

Figure 12 Identify deliverables from the feature clusters

Scenario testing is adding testing tasks into scenarios derived from requirements.

Scenario testing facilitates the objective measuring of the importance of quality

dimensions by identifying the testing tasks for the deliverables, which is shown in

Figure 13.


Pag

e30

P

age3

0

Figure 13 identify the testing tasks for deliverables and integrate into scenarios

Scenario testing is a typical testing activity of system testing which can validate

both functional and non-functional requirements to ensure the quality deliverables

produced to satisfy the users. When adopting this framework, it does not mean that

scenario testing is the only testing activity can deliver quality to the users. Scenario

testing is the earliest testing activity that can measure website quality. The functions

of a feature have no flaws, which do not mean the feature has fine quality. The feature

might not be easy to operate. However, to deliver a fine quality feature, the feature

should avoid any function flaw. Therefore, Unit testing, Functional testing, Integration

testing and other testing activities not mentioned at earlier testing phases [2] are the

fundamental upstream activities to ensure the high quality website to be tested and

measured at the scenario testing stage.

Finally, the three sub factors of the defects can be measured objectively after

scenario testing. The overview of the framework is shown in Figure 14.


Pag

e31

P

age3

1

Figure 14 the overview of the framework

This framework gives the capability of setting the three sub factors beforehand

when creating the scenario test cases. As long as the scenario is defined, the impact of

the scenario and the importance of the quality dimension are known before testing.

The likelihood can be gathered from the historical data of the statistical analysis of

usage. When a defect is exposed by certain testing task, the severity will be measured

by these three predefined sub factors. The benefit of having these three factors set

earlier than testing is because it avoids the subjective setting by the testers who

execute the testing and often lack the big picture.

3.2. Severity Classification Scheme

The only thing left is to explain the severity classification scheme. This paper

applies a naïve approach to classify these three sub factors, impact, importance and

likelihood and to convert three values into one value to represent the severity. The


Pag

e32

P

age3

2

entry criteria for testing is all level settings of three sub factors are not 0.

3.2.1. Impact

As discussed in section 2.1, impact is to measure how the defect affects the

scenario. The following list is suggested for ranking impact:

Level 0: Not set

Level 1: The defect impairs at least the third level scenario and can be skipped

without worries in the entire scenario. There is at least one easy work around. The

user can go further with no difference in the scenario. The issue is low and

cosmetic.

Level 2: The defect impairs at least the third level scenario and can be skipped

with notice by the user. There is at least one work around. The user can go further

but it will be better to have the defect fixed. The issue is enhancement.

Level 3: The defect impairs the dependent (secondary) scenario and can be

skipped with worries in the entire scenario. There is at least one work around. The

user can go further with mind in the scenario. The issue is minor.

Level 4: The defect impairs the key (primary) scenario and can be skipped with a

lot of troubles in the entire scenario. There is at least one unexpected work around.

The user can hardly go further in the scenario. The issue is major.

Level 5: The defect blocks the entire scenario. There is no work around available.

The user cannot go further in the scenario. The issue is fatal.

As been noticed from the above the list, there is a scenario level concept involved.

The scenario level relationship is illustrated in Figure 15. To reach the goal of the

scenario, the shortest path is the primary scenario (the first level scenario) denoted in

red in Figure 15. The secondary scenario (the second level scenario) is either the

alternative path (longer than the primary path) or the control interaction (using the

control will not jump pages but get more information for supporting) on the page

which belong to the primary scenario (Both cases are denoted in yellow.). The third


Pag

e33

P

age3

3

level scenario is either the loop path from a page which belong to the secondary

scenario or the control interaction on the page which belong to the secondary scenario.

The level of scenario from design perspective is to provide the comprehensive

information to the user.

Figure 15 Scenario Level

3.2.2. Importance

Importance is to measure how important the quality feature cluster or quality

dimension in the scenario. The following list is suggested for ranking importance:

Level 0: Not set

Level 1: Negligible. The quality factor or the quality feature cluster or the quality

dimension will not affect the overall website quality.

Level 2: Acceptable. The quality factor or the quality feature cluster or the quality

dimension will slightly affect the overall website quality.

Level 3: Remarkable. The quality factor or the quality feature cluster or the

quality dimension will moderately affect the overall website quality.

Level 4: Significant. The quality factor or the quality feature cluster or the quality

dimension will strongly affect the overall website quality.


Pag

e34

P

age3

4

Level 5: Critical. The quality factor or the quality feature cluster or the quality

dimension will dominantly affect the overall website quality.

3.2.3. Likelihood

Likelihood is to measure the product value of page trip rate and hit rate. The

following list is suggested for ranking likelihood:

Level 0: Not set

Level 1: 0% - 20%

Level 2: 20% - 40%

Level 3: 40% - 60%

Level 4: 60% - 80%

Level 5: 80% - 100%

3.2.4. Representation of severity

The severity is measured by normalizing the product of the rank value of three

sub factors, impact, importance and likelihood. The representation of severity is

formulated in the following equation:

𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = Impact + Importance + Likelihood

ImpactMax + ImportanceMax + LikelihoodMax

where ImpactMax is the max level of impact, ImportanceMax is the max level of

importance and LikelihoodMax is the max level of likelihood.

The representation of severity becomes to be a double number which is more

distinguishable, comparing to an integer ranking value in the traditional way. This

formula is not proved and is open to further research for refining.

3.3. Case Study

This section is going to walk you through how to set three sub factors of severity

at the key step in the buying Office product “Home and Student 2010” scenario in


Pag

e35

P

age3

5

Australian English version shown in Figure 16. The blue line guides the primary

scenario and the green lines guide the secondary scenarios in Figure 16. The dashed

boxes are the triggers (elements) in Figure 17. The key step, “Buy Office Home and

Student 2010” page is snapshotted in Figure 18.

Figure 16 snapshot of simplified version of the buying Office product scenario

Figure 17 snapshot of simplified version of the buying Office product scenario with triggers


Pag

e36

P

age3

6

Figure 18 “Buy Office Home and Student 2010” page

Since there is no design document for reference, there are some assumptions

needed. Firstly, let us assume the following quality features and quality dimensions

are considered at this step:

D10 Readability/Comprehension/Clarity (Ease of understanding)

D07 Navigation (Intuitive operations)

D01 Accuracy (Information Quality)

D02 Completeness/Comprehensiveness of Information (Information Quality)

D03 Currency/timeliness/Update (Information Quality)

D04 Visual design (Visual appeal)

D11 Relevant Information (Information fit-to-task)

Then let us breakdown the page into deliverables, shown in Figure 17.


Pag

e37

P

age3

7

Figure 19 the key step breakdown by deliverables

Now let us pick up two quality features to see their associated deliverables and testing

tasks, and to see how to set three sub factors.

For D10 Readability/Comprehension/Clarity (Ease of understanding), it is meant

to make the user understand the meaning of the interactive elements such as links and

controls. Therefore the associated deliverables A (breadcrumb), G (button), H

(clickable string), I (tab control), and J (clickable strings) are identified. The testing

task is verifying all the strings of the interactive elements are readable and meaningful.

This testing task should involve manual effort. Taking the deliverable G for example,

the importance of deliverable G should be set as level 5 because the button is unlikely

to be clicked if the string on the button is not readable and meaningless (for example,

a strange sign). The impact should be set as level 5 because the entire scenario is

going to be blocked (the button “Buy Now” is the trigger for jumping to the next page,


Pag

e38

P

age3

8

shown in Figure 16). Calculating the likelihood should notice that the hit rate should

be counted as 100% but not the percentage of page jumping caused by clicking the

button. If assuming the page trip rate is 47%, likelihood is equal to 47% from the

formula:

Likelihood = 47% × 100% = 47%

Likelihood should be set as level 3. Therefore the severity can be calculated from its

formula:

𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = 5 + 5 + 3

5 + 5 + 5 ≈ 0.867

Taking the deliverable I for another example, the importance of deliverable I

should be set as level 3. That is because the user can still read the article in the tab

control to recovery the meaning if the strings “Product Overview” and “System

Requirements” are not readable and meaningful. The impact should be set as level 3

because clicking the tabs “Product Overview” and “System Requirements” will only

update the article area on the same page which can be treated as the second level

scenario. The consequence caused by such defect might be that the tab “System

Requirements” will not be clicked by the user and the article in the “System

Requirements” tab will not be read by the user. However, this does not bother the user

to click the “Buy Now” button. Let us assume the page trip rate is 47% and the hit

rate is 100% (two strings of the tabs are obviously presented on the page.). Hence,

likelihood is equal to 47% from the formula:

Likelihood = 47% × 100% = 47%

Likelihood should be set as level 3. Therefore the severity can be calculated from its

formula:


5 + 5 + 5 ≈ 0.6

Taking the deliverable H for the last example, the importance of deliverable H

can be set as level 1 that the user might guess the meaning of the link but will not care


Pag

e39

P

age3

9

too much about it in this scenario. The impact on the scenario can be set as level 1

that the link does not affect much on this scenario (It will link to the other scenario).

The page trip rate and the hit rate are calculated like the previous ones. The severity

can be calculated from its formula:


5 + 5 + 5 ≈ 0.267

For D07 Navigation (Intuitive operations), the deliverables must have hyperlinks.

Such deliverables are identified as A, G, H, and J. The testing task is verifying the

links of the deliverables are valid and this testing task can be executed automatically.

Let us have a look at deliverable G again. The impact and the importance settings

should be in level 5 as the button is the key in this scenario. The hit rate here is not

100% as the testing task is to verify whether the link is valid. The real usage of the

link should be measured. The hit rate and the page trip rate are assumed as 50% and

47% respectively in this case. Hence likelihood can be calculated by its formula:

Likelihood = 47% × 50% = 23.5%

Base on the result, likelihood should be set as level 2. Therefore, the severity can be

calculated from its formula:


5 + 5 + 5 ≈ 0.8

So far, we have estimated severity for three deliverables (In this example, G, I

and H are all interface elements.) for feature cluster D10 and one deliverable for

feature cluster D07. Now if assuming all the elements tested above fail the testing and

the defects are reported with severity assigned according to the predefined settings in

a table like Table 7, the personnel who is responsible for assigning priority can easily

make decision to assign high priority to Bug 1and 4 (with relatively higher severity

score), to assign middle priority to Bug 2 and to assign low priority to Bug 3

(Assigning priority is just an example to show the meaning of the severity score. The

meaning of the severity score is to reveal the relative order between defects). Even if a


Pag

e40

P

age4

0

developer receive two high priority Bug 1 and 4, he can still easily make decision to

work on Bug 1 at first as Bug 1 has relatively higher severity score.

Bug ID Description Severity D(F) …

1 Unable to read the string on the button… 0.867 G(D10) …

2 Unable to read the string on the tab control… 0.6 I(D10) …

3 Unable to read the string of the link… 0.267 H(D10) …

4 Broken link of the button… 0.800 G(D07) …

… … … … …

Table 6 bug report is simulated in a table

4. Prototype

The prototype is built to prove the concept that the framework can be feasibly

adopted into a practical website scenario testing tool. Existing commercial tools [15,

16] have mature capabilities such as creating and managing scenario testing cases and

arranging various testing tasks into the steps of the test cases. The prototype is not

intended to copy the mode and to provide an advanced automation solution of any

testing tasks but (1) to assign and to evaluate impact, importance and likelihood to the

elements which will be tested; (2) to represent the severity scores of the defects

caught during testing. The basic requirements of the prototype are listed in the

following:

The system shall allow the user to load the predefined scenario.

The system shall provide basic user interface for configuring the scenario and executing

testing.

The system shall allow the user to configure the scenario to assign the testing tasks to

any step in the scenario according to the quality dimensions

The system shall allow the user to select the elements to be the target of the testing tasks.


Pag

e41

P

age4

1

The system shall allow the user to assign the level value of impact, importance and

likelihood onto the testing elements.

The system shall display the severity score for each defect after testing.

The prototype is built as a Windows Presentation Foundation (WPF) application

[20] coded in C#. The motivation is from the author‟s personal learning purpose. WPF

is a new concept to the author. The difficulties during implementation are to

understand the WPF data binding concept [21], to integrate Ribbon Control [22], and

to use WatiN library [18] which will be mentioned in section 6.

For instance, WPF data binding concept is to facilitate a fast, effortless and

codeless way to tie data instances and user interfaces and to have a clean business

logic and user interfaces separation. The WPF data binding concept is applied in a

couple of places in the prototype. For example, binding the sub factors setting user

interfaces which is shown on Panel F in Figure 21 (combobox [25], a drop down list)

with the severity XML file shown in Figure 27 and the WebPart object shown in

Figure 20. The former binding is to enable the combobox to display the level values in

the severity XML file, which could avoid the XML file parsing codes. The later

binding is to enable the change of the user‟s selection of the combobox to be reflected

in the data object that the impact, importance, and likelihood attribute values will be

updated automatically if the user changes the selection of the comboboxes. In order to

achieve the data binding, it is necessary to understand the binding target, the binding

source, the direction of the data flow and the triggers of the source updates. And the

knowledge is gained from self-learning through [21, 25].

During the implementation, the author had some unsuccessful experience. The

memorable one was to integrate W3C link validation [23] to be the logic of the testing

task for deliverable “Directions for navigating Web site”. It was planned to let W3C

link validation valid all links of a page and to let the system match the user selected

elements and the running results. It was found that the incompatibility of W3C


Pag

e42

P

age4

2

validation service provided could hardly achieve the goal. Due to the time constraint,

W3C validation library was not studied thoroughly. Further research work may

discover the possibility of utilizing the W3C validation services. Alternatively, the

prototype uses a WatiN‟s web browser object to valid the links.

The design of the prototype is quite straightforward and described by the class

diagram shown in Figure 20. The WindowMain class is responsible for user interface

logic. The inherited classes from the Verification class contain the specified logic of

the testing tasks. The interface INotifyPropertyChanged is used for data binding

between the application user interface and the data. The Verification class is an

abstract class that contains the logic of the testing task. In this prototype, it is designed

to override the “run” function in the inherited classes to specify the logic. The

WebPart class is to describe the deliverable elements mentioned before. Different

Verifications (testing tasks) will have the different WebParts (elements). The Result is

to keep the testing result for the element and the TestResult class is to describe all the

testing results and their severity scores. This prototype is designed for meet the needs

of the requirements only and the architecture can be redesigned in a more

sophisticated way.


Pag

e43

P

age4

3

Figure 20 class diagram of the prototype

5. Results and Discussion

In order to satisfy the requirements mentioned in section 4, especially to

demonstrate how to assign testing tasks to a single step of the scenario, how to set

three sub factors to an individual interface element, and how to represent the testing


Pag

e44

P

age4

4

result, a vertical prototype [24] is implemented to provide the basic user interface to

facilitate the scenario configuration functionalities and the severity score illustration.

This section will present the prototype and discuss the implementation results.

The prototype has seven panels ranged from A to G shown in Figure 21 and

Figure 29. Panel A is the command bar (Ribbon Control) which replaces the

traditional drop down menu and toolbars for increasing the discoverability of features

and functions.

Figure 21 prototype user interface layout and the relationships between panels

Panel B is the project panel which contains a tree view control to describe the

relationship between scenarios and projects. The project and scenario information is

stored in the XML file (Project File) shown in Figure 22.


Pag

e45

P

age4

5

Figure 22 the project and scenario information is stored in the XML file

After clicking the “Open” button highlighted in Figure 23, the project and

scenario information will be represented in a tree view control in Panel B. The tree

view control will display the dependent relationship between projects and scenarios.

When a particular scenario is selected by the user, the scenario-and-step relationship

will be represented in a tree view control in the scenario panel (Panel C). The tree

view control in Panel C will only display the dependent relationship between

scenarios and steps at this stage. When a specific step is selected from the tree view

control in Panel C by the user, the target web page of the step (the attribute

“actionValue” of the “Step” tag in the project XML file shown in Figure 22) will be

displayed in a web browser control in Panel D. The selecting and displaying

relationships, from B to C and from C to D, are arrowed as 1 and 2 respectively in

Figure 24.

Figure 23 Open button is highlighted


Pag

e46

P

age4

6

Figure 24 the interaction relationships 1 and 2, from B to C and from C to D

After loading the webpage, Panel E shown in Figure 26 will appear to list all the

quality dimensions, the feature clusters, the deliverables and their associated testing

tasks in a tree view control. The information is stored in a XML file shown in Figure

25. The structure of the XML file just matches the structure of Table 6 discussed in

section 2.2.1. In the prototype, only the testing task “Verify the link is valid” of the

deliverable “Direction for navigating website” and the testing task “Verify the image

is visible with proper content” of the deliverable “Attractive screen background and

pattern” are implemented. However, the testing tasks, the deliverables, and the feature

clusters can be customized and refined according to the real features designed on the

website. When the user select a specific deliverable/testing task, Panel F shown in

Figure 26 as well will list all related elements and allow the user to decide whether the

element needs to be tested and which level the three sub factors should be set. To

assist the user to make decision, the element will be highlighted on the webpage in

Panel D when the user selects an element. The interaction relationships, from E to F

and from F to D, are arrowed as 3 and 4 respectively in Figure 26.


Pag

e47

P

age4

7

Figure 25 quality dimensions, feature clusters, and deliverable structured in XML file

Figure 26 interaction relationships 3 and 4, from E to F and from F to D

The levels of the three sub factors are defined according to the classification

scheme proposed in section 3.2 and the information is kept in a XML file with the

structure shown in Figure 27. This file can be modified as long as the classification

scheme is customized and refined.


Pag

e48

P

age4

8

Figure 27 severity classificaion scheme structured in XML file

After selecting the testing elements and setting the sub factors, the user can

simply click the “Add” button in Panel F. The series of information, the testing task,

the deliverable, the feature clusters, and the quality dimension will be grouped and

added as a child node under the step in the tree view control in Panel C. This allows

the user to see what quality dimensions are measured by what specific testing tasks at

what step. Such information can explain whether the quality of the step is well

covered. Such information can indirectly explain whether the quality of the scenario is

well covered and whether the quality of the website is well covered. The interaction

relationship from F to C is arrowed as 5 in Figure 28.


Pag

e49

P

age4

9

Figure 28 interaction relashionship 5

The testing result is displayed in Figure 29. All defects caught during testing are

listed in the result table. The items in Group 1 bracketed in Figure 29 are the elements

tested by one particular testing task. From Figure 29, it is shown four defects

(elements) are caught by the testing task “Verify the link is valid” of the deliverable

“Direction for navigating website”. The severity scores are calculated based on the

formula discussed in section 3.2.4 and displayed on the next column to three

predefined sub factors for each defect. If the single factor is at its maximum level

(level 5 in the defined classification scheme), the factor is highlighted in red in the

table. The absolute severity score does not have much meaning but provide the

relative order to distinguish defects and give a strong reference for prioritization.

Interestingly, if summing up the severity scores for Group 1, the severity score can be

used to represent the deliverable score. In Figure 29, 2.667 is the severity scores

summed for the deliverable “directions for navigating website”. Similarly, the severity

scores for Group 2 (deliverables and testing tasks) can be aggregated to represent the

step score and the severity scores for Group 3 (steps) can be aggregated to represent

the scenario score. These scores can reveal the relative order at different level


Pag

e50

P

age5

0

(deliverable level, step level, scenario level, and even project level) which could be

the strong reference for prioritization.

Figure 29 testing result and severity score aggregated in different level

There is another interesting found that some severity scores are same (for

example, 0.6 and 0.533). Although the three sub factors‟ settings are different, the

total values of their settings are same. This phenomena gives a hint to go back to

refine the equation of the severity representation. What if three different weights are

added into the equation, should the severity score be more distinguishable:

𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = ω1 × Impact + ω2 × Importance + ω3 × Likelihood

ImpactMax + ImportanceMax + LikelihoodMax


Pag

e51

P

age5

1

However, it is just an immature thought that both the weights and the equation need

further research for discussion.

As mentioned at the beginning of the section, this is a vertical prototype to prove

the concept that (1) Severity can be set in a systematical way; (2) Severity can be

represented in a more distinguishable shape. The implementation result is pretty

positive to demonstrate the feasibility to adopt the framework into the practical

website scenario testing tools.

6. Related Works

The definition of scenario in software engineering discipline was firstly

conceptualized by Jacobson in [6] in early 90‟s. Scenario has been popularized to

apply in requirement elicitation and analysis to capture the system functionality and

behavior from a user-centered perspective. However there was neither any proper

procedure defined to derive the scenarios from the requirements nor any concept

mentioned to apply scenarios in any other phases of software development life cycle.

With the rapidly development of the maturity of software engineering in 90‟s, the

value of using scenario for system design has been explored by many researchers such

as Carroll [13]. Carroll concludes five reasons for using scenario in design. (1)

Scenarios expose not only the functionalities of the system but specific claim about

how the user will access the functionalities and what the user will experience in doing

so. Hence, scenarios assist the designer to coordinate design action and reflection.

Scenarios provide a concrete envisionment of a design solution. (2) Scenarios are

flexible to be revised and be elaborated so that the designer can manage the instable

design situations. (3) Single scenario can describe designs at multiple levels of detail

and with respect to multiple perspectives to help designers move toward various

consequences. (4) Scenarios can be abstracted and categorized to help designers to

recognize, capture and reuse generalizations to mitigate the risk of gaps between

technical knowledge and technical design. (5) Scenarios are efficient and effective


Pag

e52

P

age5

2

media for both stakeholders and designers to avoid the communication risk.

In the meantime, the use of scenario has been gradually appreciated in software

testing. Ryser and Glinz [8] extend Hsia et al. s‟ work [17] to disseminate to bring

scenario use into software testing field for requirement validation and verification.

They have done significant work in this area. Ryser and Glinz state that testing is

usually done in an ad-hoc manner and test cases are usually developed in an

unstructured and unsystematic way. They analysis the existing testing problems

beneath the surface: (1) Testing is done in the last phase of the development only; (2)

Testing methods are not integrated with development methods; (3) Test cases are not

created/generated in a systematic manner. They also discuss the drawbacks of some

proposed solutions such as expensiveness and complicatedness of testing automation,

specialized test languages and formal specifications. Ryser and Glinz suggest reusing

the predefined scenarios created during requirement elicitation and analysis phase to

formalize the natural languages of scenarios; then transform the natural languages into

statecharts and dependency chart; next generate test cases based on the charts in order

to address the issues. They define this method as a scenario-based approach to support

systematic test case development for validation and test, named SCENT (standing for

SCENario-based validation and Test of software). Ryser and Glinz believe that (1)

Scenarios can validate the system under development; (2) The statecharts of scenarios

can uncover ambiguities, contradictions, omissions, impreciseness and vagueness in

natural language descriptions; (3) Scenarios contains the information such as pre- and

post-conditions, data ranges and data values, and non-functional requirements (for

example, performance) which are extremely valuable to testing; (4) The statecharts of

scenarios can help to choose appropriate testing strategy and testing techniques to

make decisions in allocating resource in testing plan.

Ryser and Glinz refine the terminology of scenario, use case and their

relationships: Scenario is an ordered set of interactions between partners, usually

between a system and a set of actors external to the system. It may comprise a


Pag

e53

P

age5

3

concrete sequence of interaction steps (instance scenario) or a set of possible

interaction steps (type scenario); Use case is a sequence of interactions between an

actor (or actors) and a system triggered by a specific actor, which produces a result for

an actor, being a type scenario. They also propose a series of steps for scenario

creation listed in the Table 8 and the ideal practical steps are demonstrated in Figure

30.

Steps Step Description Outcome

1 Find all actors interacting with the system List of actors

2 Find all (relevant system external) events List of events (triggers)

3 Determine results and system in- and output System in- & outputs

4 Determine system boundaries Context diagram

5 Create coarse overview scenarios (instance or type

scenarios on business process or task level), name,

short description

[First high level statecharts are created: Few states,

no fully specified behavior]

List of scenarios

6 Prioritize scenarios according to their importance

and assure that the scenarios cover all system

functionality (completeness)

List of prioritized scenarios.

Links scenarios – actors

7 Transform instance to type scenarios. Create a

step-by-step description of events and actions for

each scenario (task level), structuring scenarios

according to a given template

[Statecharts created in step 5 are modified and

extended to describe fundamental behavior]

Flow of actions in scenarios

8 Create a dependency chart and an overview

diagram

Dependency chart, overview

diagram

9 Have users review and comment on the scenarios

and diagrams

Comments to scenarios

10 Extend the scenarios by refining the description of

the normal flow of actions, break down tasks to

single working steps

[Statecharts are refined along with scenarios]

Description of normal flow of

actions

Hints on test case derivation


Pag

e54

P

age5

4

11 Model alternative flows of actions, specify

exceptions and how to react to exceptions

Alternative flows of

actions, exceptions and their

handling in scenarios

Hints on test case derivation

12 Factor out abstract scenarios Abstract scenarios

13 Include qualities, non-functional and performance

requirements in scenarios

Scenarios with non-functional

requirements

14 Revise the dependency charts and overview

diagram

Revised diagrams

15 Have users check and validate the scenarios

(Formal reviews)

Validated scenarios

16 Structure scenarios according to given template;

create preliminary test cases / write test plan and

specification

Scenario specification paper

Table 7 steps to Create Scenario [8]


Pag

e55

P

age5

5

Figure 30 Generate Scenario Workflow [8] Figure 30 Generate Scenario Workflow [8]


Pag

e56

P

age5

6

Ryser and Glinz also define the textual representation in a template format which

consists of scenario identifier, name, a short description to explain the goal, the actors,

the preconditions and postconditions, trigger, normal flow, alternative flows,

non-functional requirements, version/change history, owner, type, diagrams, open

questions/known problems, and its associated test cases. They also distinguish three

major kinds of dependencies of scenarios, abstraction dependencies (hierarchy

pattern), temporal dependencies (time-variant execution order), and causal

dependencies (conditional pattern). Ryser and Glinz extend Jackson-like notations to

represent the scenario and their dependencies in comprehensive Dependency Chart

and statecharts in order to not only deliver a clear understanding of the dependencies

and relationships between scenarios but also help to derive additional test cases to

enhance the test suites generated from the state model of the system.

Finally, Ryser and Glinz light the path to derive the test cases from the state and

dependency charts by traversing all paths in the statecharts and the dependency chart.

They also suggest prioritizing the scenarios to refine the test suite.

Ryser and Glinz emphasize the dependency of scenarios in [9] and the practical value

in [7]. However, Ryser and Glinz neither define the concept of scenario testing nor

propagate the scenario testing for websites. In addition, their objects have not been

developed further to include the evaluation of testing results that how the failure of

scenario tests impact the entire quality of the system.

Ricca and Tonella categorize the empirical reported web faults to build a fault

model and evaluate the available web testing techniques against the model in [10].

They group the existing web testing techniques into three sets which are functional

testing, structural testing and model-based testing. Ricca and Tonella illustrate the

differences among these three categories of techniques by running a simple sample.

The difference is exhibited in finding various types of defects. Functional testing is

targeted on verifying the functionalities using black box method. Structural testing

and model-based testing is focused on the code coverage in either source code level or


Pag

e57

P

age5

7

page level. Ricca and Tonella select those faults reported for publicly available open

source web applications to construct a fault model. Those five web applications are in

from middle to large scale by measuring the lines of code. The general approach of

building such a fault model is in the following steps:

1) Review the randomly selected bugs and their fixing solutions and comments from

repository

2) Produce a description for each bug

3) Map the bug descriptions to different categories (create a new one if necessary)

4) Revise and merge/split categories.

5) Build a matrix to check what web defects can be detected by any testing technical

category.

By using this approach, Ricca and Tonella convincingly prove that there are still

demands for novel techniques to tackle these types of defects which cannot be

detected by the categorized testing technologies. Ricca and Tonella bring a heuristic

idea to think backward but in a pity that they miss one further step to relate the fault

categories and quality attributes of website.

Zhang and Dran study the website quality from user perceptions by investigating

the importance of website features and interfaces [11]. They did two studies. One is to

apply Kano‟s model to observe customer quality expectations for a specific type of

site. One is to extend the model to perceive the various domains. The Kano model, an

original marketing model defines that the product and service quality from customer

perspective have three levels: basic, performance and exciting. The quality

expectation is measured by the features. Basic quality features are the minimum

acceptable to customer. They are often ignored by customer and complained for not

having. Performance quality features are consciously noticed by customer but

disappointed feeling is raised if missing. Exciting quality features will surprise and

please the customer. The customer will be greatly motivated to increase loyalty if

receiving the features but will have no effect if missing. The Kano model also reveals


Pag

e58

P

age5

8

the time factor to these three quality features. With the passing time, the exciting

quality features will degrade to performance expectations, and the performance

quality features will turn into basic expectations. Zhang and Dran refine a feature list

of CNN website base on existing website and ask a group of frequent users to judge

the type of quality features and quality features transition. They prove: (1) the Kano

model is applicable to reasonably categorize the quality features of website; (2) the

features unlikely have the same importance to the user; (3) some quality features take

different length of transition time.

Zhang and Dran do the second study to address the limitations of the first study.

They extend to weight the importance of each feature across multiple domains and

take some domain-specific features into account. They leave the limited diversity of

participants to their future work. The result of the second study reveals that: (1) the

customer has different expectation of quality factors according to different web

domains. In another words, the importance of quality factors to different domains are

not equal. (2) the most important factors in most domains are shown as being

dominated by the basic and performance features, which is consistent to the Kano

theory. To sum up, Zhang and Dran provide solid empirical evidence to illustrate the

unequal importance of quality factors regarding the various domains from customer

perspective. Furthermore, they build a theoretical framework for evaluating website

quality from the customer satisfaction. However, their framework does not meet the

gap of the complexity of the modern websites having multiple sub domains and site

features. They also leave the blank of measurability of those user quality factors from

website testing point of view.

Loiacono, Watson and Goodhue develop a measure of website quality in order to

predict consumer reuse of the website [14]. They start from combining the Theory of

Reasoned Action (TRA) established by Ajzen and Fishbein in 1980 and Technology

Acceptance Model (TAM) extended base on TRA by Davis in 1989 to bridge the user

beliefs about a website and the behavior of reusing the website. They found TRA miss


Pag

e59

P

age5

9

specifying appropriate beliefs to technology user behaviours and TAM only identify

two very general beliefs, ease of use and usefulness. They believe it exist some other

relevant and distinct categories of beliefs such as entertainment which will also

encourage to reuse of website. Their general approach is to define the possible

dimensions of quality and identify the corresponding aspects firstly by extensively

reviewing the management information system and marketing literatures, examining

popular press publications, soliciting criteria from Web surfers, interviewing Web

designers, and studying a large organization‟s standards for website design. They

define four major categories: (A) ease of use, (B) usefulness in gathering information,

(C) usefulness in carrying out transactions and (D) entertainment value and identify

the aspects of each category. They detail that ease of use including (1) being easy to

read and understand, and (2) being easy to operate and navigate through. Usefulness

in gathering information means a website should provide (3) quality information and

(4) tailored communications. Usefulness in carrying out transactions includes (5)

functional fit-to-task, (6) response time, (7) trust, (8) online completeness, (9) relative

advantage and (10) consistent company image. The entertainment value contains (10)

visually appealing, (11) creative or innovative, and (12) emotionally appealing. Next,

they develop three questions for each aspect to create an initial instrument. Then they

refine the instrument and empirically confirm the measurement validity of the

instrument.

Although Loiacono, Watson and Goodhue theoretically present the natural of the

quality of website, they leave several areas for future researching such as concept of

total quality, cultural issues, the relationship between quality dimensions and website

types (for instance, business-to-business and non-commercial website) and so on. The

quality dimensions are not further discussed regarding evaluating from web testing

point of view that how to quantify the quality, and whether the quality could be

assured through testing , and how the quality dimensions influence the total quality of

website, and so on.


Pag

e60

P

age6

0

Huang and Chen [12] prototype an automated scenario testing tool for functional

scenario testing of web applications. They extend the semantics of UML activity

diagram to capture and refine description of use cases for covering both success

scenarios and failure or exception scenarios. They construct a scheme and breakdown

the web applications into five testable stereotypes: web pages, navigators, forms, links,

and frames. Each concept has at least one element for modelling testing. Then they

develop an algorithm named Distilling Testing Scenario to split the branches of the

activity diagram into a standalone scenario and export the reconstructed activity

diagram from XMI standard into XML file in a format of defined scheme. Then they

propose an algorithm TTSITC (Translating Testing Scenario Into Testing Code) to

transform each XML file (a scenario) into testing code. TTSITC applies XML DOM

(a XML and XSLT parser) to parse the XML file and coordinates HttpUnit (An open

source API for building executable test cases) to generate the testing code. However,

Huang and Chen miss the linkage between the scenario testing results and quality

measurement.

Existing tools in the market such as Microsoft Visual Studio 2010 Testing Edition

(VS10TE) [15] and Telerik WebUI Test Studio 2010 (TWTS) [16] provide essential

features for Website testing such as creating and managing test cases, running test

cases in automatic or semi-automatic mode, capturing and replaying user behavior on

the website, testing UI control, configuring testing environments and managing

testing effort. These features can be combined to fit the purpose to assist website

scenario testing. However, they do not have such a feature to bridge the quality

analysis and testing results. Often, in practical use, only the number of pass and failed

test cases are shown in the testing report. It is not efficient to provide such

information to the tester as one has to go back to review each failure and to determine

the impact of the failure. It is also not effective for the engineer to spend limited effort

on the proper defects which should be prioritized to fix. Contrasting to these existing

tools, the proposed prototype shifts the quality analysis into upstream, test case


Pag

e61

P

age6

1

creation phase. The quality analysis includes extracting quality factors from customer

requirements, prioritize the quality factors according to the scenario, and select the

corresponding scenario testing methods base on the quality factors. Since the quality

factors are ordered for each scenario, the associated testing results can be weighted

and summed to represent the scenario quality, the severity of the defects can be

differentiated. Hence, an overall quality of the website can be described. The testing

result will be provided with a figure to assist the engineer to prioritize the fixing

targets.

WatiN [13] is an open source library for web application testing in .Net, being

similar with WatiR (in Ruby scripting language). WatiN is developed in C# to ease the

website testing using .Net. It supports all major HTML elements testing, AJAX

website testing, major control testing, multiple web browsers testing such as Internet

Explorer (version 6, 7, 8) and Firefox testing. This library can facilitate the

implementation of various testing tasks in the prototype.

7. Conclusion and Future work

This paper analyses three sub factors (impact, importance and likelihood) of

severity and discusses their significant referential value in evaluating severity. This

paper also investigates the different views (scenario, quality and usage) from which

the three factors are measured. Considering the different views of these factors, this

paper proposes a framework to estimate defect severity by utilizing the structural view

of predefined scenarios and deriving testing tasks from scenario related quality

dimensions. This paper also defines the three sub factors‟ classification scheme and

the severity representation to support the framework. A prototype is built to prove the

feasibility to adopt the framework into the practical website scenario testing tools.

This study is relevant because it enables the severity of defects to be set in a

systematic manner and be represented in a more distinguishable form so that severity

can provides the greatest reference value for effective defect prioritization. This study


Pag

e62

P

age6

2

also can contribute to the website design and website testing research regarding

quality delivery and quality assurance. It is also foreseeable that an improved

framework will be widely adopted in the industry.

However, this paper just lights up a new direction and further research is needed

on the following areas: (1) although the quality dimensions, the feature clusters and

the deliverables are identified by the previous research work, extensions to these

contents are necessary in order to enhance the importance of scenario testing; (2) this

framework needs to be empirically proved whether it will improve the effectiveness

of defect prioritization; (3) the severity classification scheme and the representation

equation should be refined, especially likelihood which needs to be verified by

practice; (4) the way to aggregate the severity scores needs to be refined to make it

meaningful.

8. Bibliography

[0] IEEE Standard Classification for Software Anomalies, IEEE Std 1044-2009

[1] IEEE Standard Classification for Software Anomalies, IEEE Std 1044-1993

[2] Managing the Testing Process: Practical tools and Techniques for Managing Hardware and

Software Testing, 3rd

edition, page 4-8, Rex Black, ISBN: 978-0-470-40415-7

[3] Building Rich Web Applications with Ajax, Linda Dailey Paulson, Computer, vol. 38, no.

10, pp. 14-17, Oct. 2005.

[4] An Introduction to Scenario Testing, Cem Kaner, Florida tech, June 2003

[5] Webtrends Website: http://worldwide.webtrends.com/

[6] Object Oriented Software Engineering: A Use Case Driven Approach, I. Jacobson, M.

Christerson, P. Jonsson, G. Övergaard, Amsterdam: Addison-Wesley, 1992.

[7] A Practical Approach to Validating and Testing Software Systems Using Scenarios,

Johannes Ryser, Martin Glinz, Quality Week Europe '99, Brussels, 1999


Pag

e63

P

age6

3

[8] SCENT: A Method Employing Scenarios to Systematically Derive Test Cases for System

Test, J. Ryser, M. Glinz, University of Zurich, Institution of Information, Zurich, 2000/03,

2000

[9] Using Dependency Charts to Improve Scenario-Based Testing, J. Ryser, M. Glinz, the

17th International Conference on Testing Computer Software TCS„2000 in Washington, D.C.

[10] Web Testing: a Roadmap for the Empirical Research, Filippo Ricca and Paolo Tonella,

Proceedings of the 2005 Seventh IEEE International Symposium on Web Site Evolution

(WSE‟05)

[11] User Expectations and Rankings of Quality Factors in Different Web Site Domains, Ping

Zhang, Gisela M. von Dran, International Journal of Electronic Commerce / Winter 2001–

2002, Vol. 6, No. 2, pp. 9–33

[12] A Tool to Support Automated Testing for Web Application Scenario, Cheng-hui Huang,

Huo Yan Chen, 2006 IEEE International Conference on Systems, Man, and Cybernetics,

October 8-11, 2006, Taipei, Taiwan

[13] Carroll, J., "Five reasons for scenario-based design", Proc. of the IEEE Annual Hawaii

International Conference on System Sciences (HICSS). Hawaii, USA, 1999.

[14] WebQual™: A Measure of Web Site Quality, Eleanor T. Loiacono, Richard Watson,

Dale L. Goodhue

[15] Visual Studio Test Professional 2010

http://www.microsoft.com/visualstudio/en-us/products/2010-editions/test-professional

[16] Telerik Website: http://www.telerik.com

[17] Formal Approach to Scenario Analysis, P. Hsia, J. Samuel, J. Gao, D. Kung, Y.

Toyoshima, C. Chen, IEEE Software, vol. 11, # 2, pp. 33-41, 1994.

[18] WatiN Website: http://watin.sourceforge.net

[19] Microsoft Office.com Website: http://www.office.com/

[20] Microsoft Windows Presentation Foundation Website (WPF) Website:

http://msdn.microsoft.com/en-us/library/ms754130.aspx

[21] Microsoft Windows Presentation Foundation Website (WPF) Data Binding Website:

http://www.telerik.com/

http://watin.sourceforge.net/

http://www.office.com/



Pag

e64

P

age6

4


[22] Microsoft WPF Ribbon Control Website:

http://msdn.microsoft.com/en-us/library/cc872782.aspx

[23] W3C validation Website: http://www.w3.org/Status.html

[24] Software Requirements, 2nd

Edition, page 236, Karl E. Wiegers, ISBN: 0-7356-1879-8

[25] WPF 4 Unleashed, 1st Edition, page 282, Adam Nathan, ISBN: 0-672-33119-5


http://msdn.microsoft.com/en-us/library/cc872782.aspx

http://www.w3.org/Status.html

Documents

A Software Tool to Assist Scenario Testing of Websitescourses.cecs.anu.edu.au/courses/CS_PROJECTS/10S2/Reports/Siqi Fan.pdf · A Software Tool to Assist Scenario Testing of Websites