Visual Testing of Browser Screen Captures - Muni

MASARYKOVA UNIVERZITA

FAKULTA INFORMATIKY

}w�� !"#$%&'()+,-./012345<yA|Visual Testing of Browser Screen

Captures

BACHELOR’S THESIS

Lukas Fryc

Brno, Autumn 2010

Declaration

Hereby I declare, that this paper is my original authorial work, which I have worked out bymy own. All sources, references and literature used or excerpted during elaboration of thiswork are properly cited and listed in complete reference to the due source.

Advisor: Mgr. Marek Grac

ii

Abstract

This work intends to fill the gap in the functional testing by providing concept of visualtesting. It provides implementation of tool which have ambitions to significantly contributeto decision about release of enterprise web framework. It deals with difficulties withimplementation of stream parsing of data that are declaratively bound to programmaticmodel. It brings valuable summary need to implement Delta-E algorithm for computingdistances in CIE 1976 color space.

iii

Keywords

Visual Comparison, Visual Testing, Visual Difference, RGB, CIELAB, CIEXYZ, Delta-E, Color Distance, Java, XML, JAXB, StAX, Binding, Stream Parsing, Proxy, Selenium,RichFaces, Open Source, LGPL

iv

Acknowledgement

I am very grateful to my technical advisor, Ing. Pavel Tisnovsky, PhD., who gave me plentyof advice from the area of his expertise - image processing - which was fundamental forthe successful implementation of my ideas to practice. I am pleased to thank my advisor,Mgr. Marek Grac, for introducing my work to the university. I also want to express theendless admiration to my wife, my mother and my mother in law for supplying me withtheir support and care while I was working on this thesis.

v

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Background of Current Functional Testing . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Tools Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Employing Visual Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Current Visual Comparison Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 The Wanted Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Conceptual Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.1 Visual Comparison Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.1.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1.2 Masking Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1.3 Standard Data Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.1.4 Sample Data Retrieving . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 Sample Application and Functional Test . . . . . . . . . . . . . . . . . . . . . . . . 104.1 Introducing Sample Application . . . . . . . . . . . . . . . . . . . . . . . . . . 104.2 Functional Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 Implementing Visual Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.1 Implementing Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.2 Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.2.1 Timing of Capture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2.2 Proxy Selenium API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2.3 Storage Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.2.4 Storing or Generated Images . . . . . . . . . . . . . . . . . . . . . . . . 16

6 Comparison Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.1 Introduction of Color Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.2 Perceptual Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

6.2.1 Conversion from RGB to CIEXYZ . . . . . . . . . . . . . . . . . . . . . 186.2.2 Conversion from CIEXYZ to CIELAB . . . . . . . . . . . . . . . . . . . 19

6.3 Filtering - Masks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.3.1 Selective-Alpha Mask . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6.4 Comparison Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Implementation of Comparison Tool . . . . . . . . . . . . . . . . . . . . . . . . . . 22

7.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227.1.1 Visual Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

7.2 Suite Definition Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1

7.3 Reading Test Suite and Writing Results . . . . . . . . . . . . . . . . . . . . . . . 247.3.1 XML Binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247.3.2 XML Stream Parsing and Writing . . . . . . . . . . . . . . . . . . . . . . 247.3.3 Marshalling and Unmarshalling of XML using StAX . . . . . . . . . . 257.3.4 I/O Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

7.4 Comparison Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257.5 Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267.6 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.7 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277.8 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

8 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288.1 Identifying Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288.2 Result Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

8.2.1 Applying Masks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298.2.2 Functional Test Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . 298.2.3 Suppressing Randomness and Time Factor . . . . . . . . . . . . . . . . 298.2.4 Application Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . 30

8.3 Stabilized results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

9.1 Space for Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31A Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35B Visual Suite Descriptors Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39C Sample and Difference Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

2

Chapter 1

Introduction

In the measures of current software development needs, there is well-known demand foradequate software validation methods. Targeting short release cycles, fully-automatedsolutions of testing and deployment increase on popularity.

The proper way of automation was already brought on level of white-box testing to giveus adequate assurance that systems internally work in specified integrity constraints andinvariants. These constraints are typically covered by unit testing. The communication withother systems is also successfully validated by using of integration testing methods.

However there can be seen lack of proper tools which satisfy the testing in userperspective, mostly described as functional testing. This approach has its place everywherewhere systems communicate through human interface.

Current tools for functional testing are mostly robots that simulate user interaction.Automated test suite then consists of scripts for controlling such robots. Running of thesuite gives system a feeling that it is controlled by human being.

A problem that evokes this access inhere in proper validation of user vision. All therobots and technologies around them are able to retrieve many of the attributes of finalform of user interface though no quality engineer is able to describe all of the attributes thatinfluence final user experience.

It is the matter of tester’s correct presumption to validate appropriate attributes. Thewhole process of automation then might give us wrong feeling, or even fiction, that oursoftware product operates as it was designed.

Particularly the testing of web applications suffers from the incompleteness of testscripts. By using of common tools one can simulate user interaction across the browsersand operating systems though it guarantee nothing else than integrity checks that havebeen described by tester. In the language of browsers testers typically describe the changesin DOM1 and CSS2 caused by triggering of events handled by JavaScript.

By automation of these test scripts and deploying to different underlying platforms testermay get a good picture of the compatibility of browser’s JavaScript, respectively the pictureof validity of its generation on server-side. But no more. This is the fact which needs to beconsidered each time the automated testing of web user interface is employed.

Let’s imagine an access of watching the browser during the functional test run by one

1. Document Object Model2. Cascade Style Sheet

3

1. INTRODUCTION

man. This viewer has precise information what the application should look like and maymark all the differences from previous versions. His knowledge of application then satisfiesstability of application from end-user’s view.

And this is exactly the area where concepts brought by this work serve testers - theyvalidate browser-rendered output. By automation of the visual testing they reduce theneed of tester’s attention and reliably and reproducibly validate much more conditionsthan during regular automated functional testing. They also avoid error-prone andtime-consuming manual testing.

The idea of automated visual testing of browser screen captures has basis in applyingperceptual comparison on quanta of images. The reason why this idea is not more employedin practice comes from the overhead which is connected with maintenance of such test suite.That is why this work comes with several concepts how this overhead may be reduced byemploying of proper tooling to acceptable level and provides with solutions where no suchcan be found.

In the first chapter the reader will familiarize himself/herself with the background ofcurrent practice of functional testing. The second chapter provides with the concept ofvisual testing. Third chapter offers introduction to the sample application and functionaltest suite which necessitates the concepts brought by this work. The fourth and fifth chaptersdescribes nuances of comparison algorithm and implementation of the tool developed tosupport it. Finally in sixth chapter reader may get familiar with the results of employing theimplemented tool, while in seventh chapter finds the conclusion of the implemented accessand explores the space for improvements.

4

Chapter 2

Background of Current Functional Testing

2.1 Tools Overview

Following tools are used for the functional testing of web applications. They will be shortlyintroduced to see the fundamental difference in their deployment. Both of them simulateuser behavior by controlling client program that communicates with browser. They haveAPI for triggering some of user interactions like clicking to button or typing text. They canexecute client-side scripts, thus they can perform AJAX1 communication with server. Theycan obtain extensive informations about current state of page like retrieving whole markupor obtaining DOM attributes and CSS properties. So they provide us with good resources toperform testing of output generated from HTTP server.

HtmlUnit2

HtmlUnit is developed to cover functional testing in very efficient way. It’s core simulatesHTTP client communication with server via regular HTTP requests and because it supportsexecution of JavaScript it can perform asynchronous communication in AJAX method.However because it operates in headless mode, you can never get the output renderedfrom markup. It also only tries to simulate the execution of scripts using its JavaScriptimplementation Rhino, but does not reflect the real browsers’ implementations. HtmlUnitis thus good for black-box testing on basic level when we don’t need verify functionalityfrom user perspective.

Selenium3

On the other hand HtmlUnit’s rival, Selenium, is able to perform the same variety ofcommands in real browser environment. Essentially Selenium is API for controllingbrowser and needs driver implementation to perform actions. The driver is extension tobrowser’s core to allow execute user actions. Some of the drivers are implemented in nativebrowser’s language, others are implemented in directly in JavaScript. Some of the functionsare on selected drivers completely impossible. Although it offers assurance that the actionsperformed in the browser will work when triggered by user in real browser.

1. Asynchronous JavaScript and XML2. http://htmlunit.sourceforge.net/3. http://seleniumhq.org/

5

2. BACKGROUND OF CURRENT FUNCTIONAL TESTING

2.2 Employing Visual Perception

Both of the presented tools suffer from lack of the evaluation of user experience. They doesnot integrate functions for evaluation of human vision by processing output of browser’srendering algorithms.

Fortunately it does not mean that we can’t use these tools in visual testing process. SinceSelenium API contains functions for capturing screen of the browser, we only need to findlast piece of the puzzle - the way of comparison of browser screen captures. By finding suchway we may be close to automation of whole process of visual testing.

2.3 Current Visual Comparison Tools

There are already tools used for recognition of image differences in practice. PerceptualImage Diff [8] has been developed for comparing results of rendering algorithms. This toolshines when it compares differences of OS, machine architecture or factor of randomness inimage rendering. But since we expect no binary differences when capturing screens of onespecific version of browser on one OS and one machine, the employing of this tool can onlyadd the complexity in handling of its output to achieve appropriate level automation.

2.4 The Wanted Tool

The tool which is needed to succeed is focused on processing thousands of images andrejecting the images which are perceptually different in the scope of pixels. When detectingdifference, we need to let space for user’s feedback to decide where can be result fixed -in the test scenario, underlying architecture or tested application. Instead of complexity ofreducing false-positives we emphasize efficiency of result post-processing by human. Toallow proper level of automation we require the interchangeable data format. All of theseaspects are projected to development of the tool introduced by this work.

6

Chapter 3

Conceptual Overview

The concept of visual testing may be described as sequence of processes, each of themgenerate output which creates input for subsequent process. Fundamentally they differ inactor that utilizes the data. Activity diagram on figure 3.1 illustrates who should proceedeach task.

Figure 3.1: Concept of Visual Testing Process

7

3. CONCEPTUAL OVERVIEW

It is evident from diagram that most of the actions still rely on human, but it is necessaryto emphasize that all of initial manual actions (definition of functional test and validation ofpatterns) are one-off activities. Once they are done, whole process starts to depend only onmaintenance steps.

Assuming that both non-manual activities left to Selenium and visual comparisontools can be fully-automated, the only human intervention is needed when found positivedifference in comparison phase. The actual necessity will be presented on real sampleswhen describing test results (8.3).

The consideration of importance of proper tooling for checking and modifying results isleft to reader. It is not in scope of this work to develop such tools. However some ideas inthis direction are elaborated here (9.1).

3.1 Visual Comparison Tool

The idea behind the introduction of the perceptional comparison of images lies in reducingone of most tedious work in process of software validation - manual testing. Thereforethis work provides testers not only with tool, but also exposes its internals in form of theframework which can support tool’s deployment to test environments.

3.1.1 Algorithm

Framework is built around the algorithm computing differences between colors as theyare visually perceived. User of the tool might choose appropriate level of acceptance ofperceptional differences. In cases where default values of perception are not adequate topurpose, they can be changed by user. Its basics will be described in chapter 6.

3.1.2 Masking Differences

As it will be shown (8), the algorithm for visual perception with its settings can’t be enoughsuccessful on real samples. Therefore additional principles of masking differences has beendeveloped and built into framework.

3.1.3 Standard Data Format

Essential requirement of choosing proper format (based on XML) allows users to processtheir suites as simple as possible and reuse existing tools in development of customprocesses around data generated by this tool.

Additionally this work introduces the standard way of loading and storing suite usingJava (the target implementation platform) by providing marshalling and unmarshallingmechanisms.

8

3. CONCEPTUAL OVERVIEW

3.1.4 Sample Data Retrieving

Choosing of proper way how sample data are stored and loaded is essential question inthe environment of the automated testing. Therefore final implementation allows user tocustomize way of retrieving data for comparison and way of writing result data.

9

Chapter 4

Sample Application and Functional Test

For practical testing of the concept and implementation it was necessary to choose rightsoftware product as the source of the images that are compared by presented tool. Thepreconditions exposed on this product was live development necessitating automation ofvisual testing and also existing functional test suite.

As sample product was chosen RichFaces Live Demo [3] - the demonstration applicationfor RichFaces [2] web-framework. It shows the benefits of interactive communication withserver and employs several concepts which can be seen on modern web.

4.1 Introducing Sample Application

RichFaces is visual web development framework built on top of JavaServer Facesspecification 1. It contains lot of visual components that simplifies development of theenterprise web applications. The enterprise area as target is crucial in decision to developand deploy automated visual testing.

The target platform for viewing pages written with help of RichFaces framework isweb browser. This framework supports 5 mainstream browsers namely Mozilla Firefox,Microsoft Internet Explorer, Google Chrome, Apple Safari and Opera. Additionally thesupport goes to history of 2 stable versions for the first two enumerated browsers.

RichFaces comes with support of 12 distinct skins serving as base for building userinterfaces and contains about 46 purely visual components.

When presuming enterprise environment as target platform, it is clear that stability ofrendered components can’t be satisfied by manual release testing. Although most of thecompanies employ exploratory manual testing in similar cases instead of exact validationprinciples.

Evidently the possibility of extending the quality assurance process by automatingof visual testing may bring lot of confidence in framework, may provide valuableinformations during release cycles and in corner case it may provide users of the frameworkwith information what components is visually changed from previous release.

From perspective of tester described principles reduce need for extensive manual testingand eliminate risk of visual problem in components. They reduce error-prone process toacceptable level and give tester time to focus on aspects which can’t be automated so easily.

1. http://www.jcp.org/en/jsr/detail?id=314

10

4. SAMPLE APPLICATION AND FUNCTIONAL TEST

Figure 4.1: RichFaces’s Extended Data Table in Live-Demo

4.2 Functional Test Suite

In this section is described the functional test suite used for testing RichFaces Live Demo.Figure 4.2 shows the conceptual decomposition of particular parts used during tests.

Managing Life-Cycle

As the controller over all of the subtasks needed in the process of testing is used Maven [1]and its plug-ins.

Maven is software project management tool with declarative definition of project objectmodel, its functionality is extensible by definition of plug-ins and standard life-cycle phases.

Building WAR Plugin

First Maven task in process of functional testing is compilation of source code and itspackaging into web archive together with additional libraries and mandatory and optionalweb resources. That way Maven prepares application for deployment onto applicationserver. The plug-in used in for this task, maven-war-plugin, is part of the standard set ofMaven plug-ins.

11


Figure 4.2: The Components of Selenium Functional Test

Starting and Deploying onto Application Server

Once the web archive is ready, it is possible to start application server and deploy it. Both ofthis tasks are satisfied by using of cargo-maven2-plugin2 in pre-integration-testphase.

Starting Selenium Server

Selenium Server is the part of Selenium infrastructure which triggers commands in realbrowser. It consists of Selenium Core (driver) and Selenium Remote Control. Selenium Coreis the extension on top of particular browsers and facilities execution of commands thatsimulate user interaction. Selenium Remote Control instantiates browser, parse commandsincoming from clients and execute them through Selenium Core. Selenium Server is startedthrough selenium-maven-plugin 3 in pre-integration-phase.

2. http://cargo.codehaus.org/Maven2+plugin3. http://mojo.codehaus.org/selenium-maven-plugin/

12


Running Selenium tests

Tests are defined in format specified by TestNG4 framework and executed usingmaven-surefire-plugin. This plug-in is part of standard set of Maven plug-ins.

TestNG tests uses Selenium’s language specific client library to connect to startedSelenium Server. The Selenium Remote Control inside Selenium Server then creates newbrowser session that effectively makes all of the commands from Selenium API available.The commands may be finally executed in order to simulate user interaction.

The standard way of running tests in Maven is their execution in Maven test phase. Butbecause it is necessary to run them after starting of application server and Selenium Server,they are rebound to integration-test phase.

4. http://testng.org/

13

Chapter 5

Implementing Visual Test Suite

Each Selenium test suite may be easily switched to visual suite by using one of the methodsprovided by Selenium API that enables browser to capture screen-shot of printable area ofthe page. Specifically in our scenario it is used method captureEntirePageScreenshot.This method satisfies that entire content of the page is obtained using browser-specificfeatures. It is necessary to note difference of this method to captureScreenshot method- it captures only visible area of the page including browser controls and the area aroundbrowser which is unnecessary when targeting later automated visual inspection.

The support and implementation differs between browsers but two major browsers atthese days support similar feature to capture screen-shots of entire page. Mozilla Firefoxuses third-party add-on Screengrab!1, while Internet Explorer enables capturing by itsActiveX extension called snapsIE2.

5.1 Implementing Switch

One of the options how to switch functional test suite to visual test suite is explicitlyintegrate invocation of captureEntirePageScreenshot to each command causinginteractive action on the page. But it is convenient to avoid need for code changesand use instead declarative manner to instrument code to do that work. For thisreason need to be identified subset of Selenium API commands which usually leadto interactive communication with server. These commands may be supplied withcaptureEntirePageScreenshot invocation in order to captures screen-shots andeffectively watch the progress of changing the page content.

This principle is internally stabilized by capturing screen-shots in proper timing andoptimized by storing only such screen-shots where was identified some change.

Overview of whole process in context of Selenium test might be seen on activity diagram5.1.

1. https://addons.mozilla.org/en-US/firefox/addon/1146/2. http://snapsie.sourceforge.net/

14

5. IMPLEMENTING VISUAL TEST SUITE

Figure 5.1: Implementation of Proxy for Switching to Visual Suite

5.2 Stabilization

Common usage of Selenium API shows that tests need to be written in manner satisfyingstabilization after each interactive call. In practice it includes some kind of waiting forconditions caused by the user interaction. This may include a waiting for load of the newpage (for regular HTTP requests) or a waiting for particular change of page elements (in acase of AJAX requests which cause partial page refresh).

5.2.1 Timing of Capture

The stabilization practice provides good conditions for switching Selenium-based test suitesautomatically and satisfies that they will be suitable for image-generation.

Once the suite is properly stabilized, it is enough to capture screen before each interactivecommand, after waiting to open new location and immediately before the end of each test.

5.2.2 Proxy Selenium API

For the purpose of implicit invocation of captureEntirePageScreenshot after eachinteractive command it is possible to use Dynamic Proxy3 feature of Java Reflection API4.

3. http://download.oracle.com/javase/6/docs/technotes/guides/reflection/proxy.html4. http://java.sun.com/developer/technicalArticles/ALT/Reflection/

15

5. IMPLEMENTING VISUAL TEST SUITE

Dynamic Proxy allows wrap call to the specific interface with own code which effectivelyemploys interception as one of the concepts of Aspect Oriented Programming.

Proxy internally asks if the invoked command is interactive and if so, it captures thescreen-shot (fig. 5.1).

It the context of test code it is necessary to replace implementations of Seleniumwith dynamic proxy internally invoking commands on real implementation (usuallyDefaultSelenium instance). The replacement needs to be done across whole functionaltest suite. The replacement is easy when using proper abstraction of test cases andcentralizes management of Selenium instance of single point.

5.2.3 Storage Efficiency

It is possible to implement optional step that satisfies that only captures which show thechange of the rendered page will be stored. The principle is shown in fig. 5.1. However thisstep requires employing the perceptual comparison algorithm to compare images duringthe run of functional test suite.

Second type of optimization assumes that the test suite contains lot of duplicate screens.Then it is possible to store the images under name reflecting hash of the image which allowsto reduce the required disk space.

In conjunction with usage of dynamic proxy, it is validly solved automatic generationof sequence of pictures as may be seen by user when interactively changing state of theapplication.

The illustration that this principle respects all of the page changes which may be causedby user interaction is provided in fig. 5.1.

However we can also identify actions like scheduled tasks and random actions whichcan’t be predicated and have to be processed explicitly.

5.2.4 Storing or Generated Images

With the proxy, it is possible to capture screen-shots as the result of each interactivecommand that simulates user behavior.

Captures are immediately stored to the permanent storage and identified by uniquenames.

Here was identified two principles that satisfy uniqueness of names:

• name derived from the context of test and

• name given by hash of the screen image.

The first one composes name from fully-qualified name of test class, test method and linenumber. All of these informations are derived from by inspecting stack trace in the momentof capturing of the screen-shot.

In the latter situation, we need to count hash of the image.

16

Chapter 6

Comparison Algorithm

6.1 Introduction of Color Spaces

Solving the problem of difference between two colors mostly employs simple algorithmsfor computing distance of two colors as points in color space which trivially falls back tocalculation of Euclidean distance between them. But the situation completely changes whenasking how the eye-ball perceive the image and how the brain decodes it. Such situationrequires color spaces reflecting non-linear response of the eye perceiving the color.

Most of the traditional color models used for storing images (RGB, HSB, CMYK) hasbeen designed to produce output to computer controlled displaying devices like computermonitors or printers. Thus computing the distances in these color spaces may provideimagination how much are two colors different in output to computer devices. The distanceis then far away of difference perceived by human being [6].

Over the years of development the CIELAB1 color space has been introduced to reflectgood approximation modeling color space how it is perceived by human eye. Its advantageis uniform distribution of color components across the space. Computing distance oftwo points in this three-dimensional space may result in imagination how two colors aredifferent from human eye’s point of view.

CIELAB was introduced to fix disadvantage of its predecessor CIEXYZ2 which hasnon-uniform distribution of color components, but though reflect the non-linear perceptionmodel of human eye.

6.2 Perceptual Comparison

The computing of difference of two colors is trivial operation in CIELAB space, because it isreduced to Euclidean distance computation that results into three components L*, a* and b*,where

• L* = 0 represents black and L* = 100 is diffuse white (specular white may have higherL* value),

• a* is distance between magenta and green (negative values indicate green whilepositive values indicate magenta),

1. CIE 1976 (L*a*b*) Color Space2. CIE 1931 XYZ Color Space

17

6. COMPARISON ALGORITHM

• b* is distance between yellow and blue (negative values indicate blue while positivevalues indicate yellow).

The formula for calculating the difference ([4, p. 73]) is known as Delta-E and can beexpressed as

(∆E)2 = (∆L∗)2 + (∆a∗)2 + (∆b∗)2

where ∆E is color distance in CIELAB color space.However this formula doesn’t reflect two additional factors ([7]), which wasn’t

considered in design of CIELAB:

• perceptual non-uniformness (different amount of visual color shift in different colorareas may result in same numbers) and

• issue with the human eye which is most sensitive to hue differences, then chroma andfinally lightness.

There are solutions more or less suitable for particular cases ([7]), but for the purposes ofthis work was chosen Delta-E as it is known for basic fast calculations.

The problem of getting distance is then directed to computing L*a*b* components fromthe RGB color space. In order to do conversion from RGB to CIELAB color space we need touse intermediate transformation through CIEXYZ.

6.2.1 Conversion from RGB to CIEXYZ

Since the transformation differs for certain RGB spaces, we will make assumption aboutthe color space used in Java language. The implementation of this language states that theinternal implementations are using sRGB and other color spaces are always converted usingICC profiles to default.

The transformation for sRGB color space follows:

C ∈ {R,G,B}

C ′ =C8bit

255

C =

C′

12.92 if C ′ ≤ 0.03928(C′+0.0551.055

)2.4otherwise

where C ′ stands for non-linear sR’G’B’ values and C stands for linear combinationof CIEXYZ components. Since we have (R,G,B) values computed, we can do lineartransformation to CIEXYZ:

18


X

Y

Z

=

0.4124 0.3576 0.1805

0.2126 0.7152 0.0722

0.0193 0.1192 0.9505

· R

G

B

6.2.2 Conversion from CIEXYZ to CIELAB

When converting value from CIEXYZ to CIELAB, it is necessary to reflect illuminant valueof given RGB space (since they can differ between platforms). In the Java, we can count withthe standard illuminant D65 since the color space used for internal image representation issRGB.

The reference illuminant values for D65 ([6, p. 15]) is:

XD65 = 95.047

YD65 = 100

ZD65 = 108.883

The L*a*b* components are then derived using following transformation ([6, p. 14]):

L∗ =

(

116 ·(

YYD65

) 13 − 16

)if Y

YD65> 0.008856(

903.3 · YYD65

)otherwise

a∗ = 500 ·

((X

XD65

) 13

−(

Y

YD65

) 13

)

b∗ = 200 ·

((Y

YD65

) 13

−(

Z

ZD65

) 13

)

6.3 Filtering - Masks

The practice shows that the perceptual algorithm is not enough successful in filteringchanges in screenshots caused by differences or non-determinism in underlying application.

To achieve stability of the visual suite, the advanced mechanisms need to be provided.One of the methods which excels before the others are filtering evident differences using

masks.Mask is the image which by its nature signals the comparison algorithm that the given

pixel should not be processed using perceptual comparison.We identified two methods which achieves desired results on sample test suite:

19


6.3.1 Selective-Alpha Mask

The first of the mask types is based on the possibility to define image area in which shouldbe all differences shaded.

To define such mask we use the ability of PNG images to store one additional Alphachannel together with standard RGB channels.

This channel can contain 256 levels of transparency for 8-bit PNG images. Effectively thealgorithm examines if the alpha channel contains non-zero value (the zero transparency)and in such case applies the masking.

The image can be also positioned in scope of compared image, the following types ofalignment are currently supported:

• horizontal alignment: {top, bottom}

• vertical alignment: {left, right}

It is possible to implement more options for aligning mask in scope of compared imageslike centering or aligning on specific coordinates.

However implementation of this mask is trivial and computationally easy: each pixelasks for each selective-alpha mask if it contains on the given coordinate non-zero value -principle which effectively stands for constant complexity.

Ignore-Bitmap Mask

The second introduced mask type stands for masking specific ”patterns” from comparedimages.

Ignore-Bitmap mask can be employed in cases where the ”pattern” occurs randomly andthus the exact alignment may not be determined to apply Selective-Alpha Mask concept.

However this type of mask needs non-trivial algorithm employing perceptualcomparison.

Conceptually we need determine for each pixel if the mask does not suit to imagestarting in (0, 0) at given pixel. In worst case this brings quadratic complexity. Thedifference between colors in image and mask may not be greater than 1 since we want toapply mask only when they are imperceptibly different.

6.4 Comparison Process

The final implementation of image comparison is the combination of perceptual comparisonand masking.

The process of comparison goes through all of the pixels in image and asks if the pixelis masked. If it isn’t masked, the perceptual comparison algorithm is used. The result ofcomparison of each pixel is the distance of the pixels and the color of pixel applied indifference image. It depends on perception settings chosen for given test if the pixel willbe considered as perceptually same or different.

20


Difference image is constructed during the process of comparison. To allow easyevaluation by human being, it converts all of the pixels perceptually same to gray-scale.Different pixels are emphasized by colors of magenta, blue and red to highlight changes.The color for differences is chosen by the level of distance. Finally the masked pixels aretransformed to green tones to distinguish them on the different image.

21

Chapter 7

Implementation of Comparison Tool

The implementation of the tool for the automated visual testing is written in Javaprogramming language and uses external libraries for implementation generic tasks likeconcurrency, parsing and writing suite descriptors.

The name of the project implementing concepts described in this work is RushEye, thepart of implementation focused on processing image comparison is called RushEye Core.

The tool is distributed in two parts: RushEye API (rusheye-api.jar) andimplementation (rusheye-impl.jar). API provides developers with option to extendbasic functionality of tool without need of coupling their implementation with originalimplementation, while implementation provides the runtime dependency necessary to runvisual test suite in scenario described in this work.

Implementation doesn’t give third-parties any guarantee to stay stable in the time andsubsequent changes can happen to fix functionality issues or improve performance.

Implementation is responsible for parsing suite descriptor (based on XML meta-format)to load configuration of suite. During the loading, appropriate event listeners are fired tostart processing of particular sample images. Each sample are then compared to one or morepatterns using the perceptional setting and using masks as difference filtering option.

During the process of comparison the difference image is built to make manual checkpossible and the numbers as details of comparison process are computed.

Once the result of comparison is available it is passed to subsequent processors storingthe difference image and writing result description and statistics.

Whole process is illustrated on activity diagram on figure A.2.

7.1 Configuration

7.1.1 Visual Suite

Visual Suite is the description of input images and settings used in testing process. Thesettings included in suite can be divided to these groups:

• default configuration,

• global configuration,

• test-specific configuration.

22

7. IMPLEMENTATION OF COMPARISON TOOL

Figure 7.1: Configuration-Chain

Generally, configuration of suite is built from:

• perceptional settings,

• setting of masks,

• component tool configuration,

• test sources properties.

Default Configuration is the configuration of perception which suites most of the usecases (by convention-over-configuration paradigm).

Global Configuration is specific settings appropriate to whole run of suite. Containsconfiguration for component implementation used in process of comparison forcustomizing retrievers, result and statistics writers and storage manager. It is the placewhere can be defined masks or perceptional settings applied for all tests in suite.

Test-Specific Configuration is the only place to define the particular location of sourcesof tests and patterns. Additionally it is possible to redefine perception and filtering settingshere.

Command-Line Configuration The configuration which always takes precedence is takenfrom command-line options of parser. Currently only configuration of suite listeners are

All these configuration are in the given order summarized when the later definitionstakes precedence. Summarized configuration is then provided to comparison algorithm onper-test basis.

23


7.2 Suite Definition Language

The suite definition language is XML compliant, which brings possibility to process, queryand transform both suite definition and results in many programming languages usingexisting tools.

Language Grammar

The definition of the language used by suite and result descriptors is written in XMLSchema1. It provides testers with option to validate their definitions before they will betaken by tool.

Usage of XML Schema also provides tool with standard way of validation of inputsource and prevents lot of vulnerabilities which might be introduced to tool on input-level.Therefore shades the problematic aspect of user input validation.

You can see samples of visual suite descriptor on fig. B.1 and result descriptor on fig. B.2.

7.3 Reading Test Suite and Writing Results

In order to load configuration from XML descriptor, it is necessary to parse XML structureto Java objects. In opposite way, the data written from objects to XML describes the results.

7.3.1 XML Binding

Binding between suite descriptor (respectivelly result descriptor) and Java representationsolves JAXB2. JAXB allows model to marshall XML stream to instances of Java classes. Inopposite way Java classes can be easily unmarshalled to XML stream. JAXB uses the JavaAnnotations3 to define bidirectional object-to-XML mapping.

Once the model is defined and annotated it can be read or written to XML using standardway. Therefore are annotated classes part of RushEye API.

The suite’s class diagram is shown on A.3.

7.3.2 XML Stream Parsing and Writing

To minify memory footprint and thus allow processing wide amount of the input data theprogram reads data from stream using StAX4.

In case of document reading, Streaming API for XML offers to developer event-basedprogrammatic interface, which prevents from necessity to load whole DOM (DocumentObject Model) tree to memory heap (as in case of tree-based APIs).

1. http://www.w3.org/XML/Schema2. Java API for XML Binding; http://jaxb.java.net/3. http://download.oracle.com/javase/1.5.0/docs/guide/language/annotations.html4. Streaming API for XML, http://stax.codehaus.org/

24


StAX also provides way of writing result document to output as stream and thus allowscontinuous writing of the objects on demand.

7.3.3 Marshalling and Unmarshalling of XML using StAX

JAXB provides the way how to read and write data from several kind of sources like files,streams or DOM. The process of the reading objects from stream is called marshalling.

Additionally JAXB supports reading (resp. writing) of events fired (resp. consumed) bystream or event APIs of StAX. By merging these two principles introduced above, we can getmemory efficient stream event marshalling (resp. unmarshalling) using annotation-basedmapping.

7.3.4 I/O Validation

By integrating StAX (specifically Woodstox5 implementation of StAX 2) we are getting onemore advantage - standard way of XML validation on input and output level by registeringvalidation against XML Schema into process.

7.4 Comparison Processors

Besides the parsing and writing the RushEye needs to implement chain of processors whichcan take the configuration from parser, process it and give result to output.

The simple schema how this chain receives data from parser and how it finally writethem to result descriptor are shown on 7.2.

The concept of processing data was already described and shown on activity diagramA.2.

For the overview what components the chain employs you can see A.1.The core of the processing is SuiteListener. It uses internally ImageComparator

and certain kinds of Retrievers in process of image retrieving and comparison.SuiteListener also controls ResultCollector as the instance which is provided

with detailed overview about the progress of comparison.Finally ResultCollector internally uses ResultEvaluator to process result to

intelligible form.ResultCollector fires events for subsequent components in order to write statistics

(ResultStatistics), store the difference images (ResultStorage) and finally writeresult descriptor (ResultWriter).

Note that the latter is in common case responsible for implementing unmarshalling toXML output stream.

5. http://woodstox.codehaus.org/

25


Figure 7.2: Parser

7.5 Parallelization

From the schema 7.2 we can see that procedure achieving generation of comparisonresults using input images and suite description may be internally parallelized without thechanging the meaning of result data.

Although the concept is ideal candidate for parallelization, the current implementationis parallelized only on level of loading images.

This access presumes compressed set of images on the input and can count with basicscenario saving the results in PNG format which is known to use compression.

With the preconditions above and fact that compression itself is non-trivial computationaloperation the tool achieves data-parallelization by using methods provided by Javalanguage: for each image it opens the FutureTask6 and then waits while the image isnot available in it’s object form in memory. The same principle is used for storing images.While writing and reading images, the core of the tool can meanwhile parse or executecomparison algorithm.

It is possible that latter implementations will bring further methods of parallelism andso it is necessary to claim that the order given by input data have not be necessary compliedon output.

6. http://download.oracle.com/javase/6/docs/api/java/util/concurrent/FutureTask.html

26


The only convention which needs to be respected is that each test will finally determineone result conclusion and provide result of comparison with one or more pattern images notnecessarily using all of the masks which can apply. Thus also difference images can differ insome cases.

7.6 Error Handling

The tool states that all of the errors caught in the run of testing process are logged andadequately recorded to result descriptor for further investigation except the validationerrors in suite definition and runtime errors (insufficient memory, network issues, etc.).

Suite descriptor validation is processed on input level which can’t satisfy recovery fromfailure. Thus when parser finds configuration validity problem, whole build is stopped. It isrecommended to use input validation before parsing using the attached schema.

7.7 Testing

For the stability of the tool was developed extensive test suite covering both black-box andwhite-box testing.

Several concepts was employed here like testing life-cycle management and abstractionusing TestNG, developing fake implementations in order to test components independentlyusing Mockito7 or generation of sample suite descriptors using Dom4J8 in order to checkinput validation implementation (since it is achieved by external StAX implementation).

7.8 Licensing

Project is developed under the quality assurance department of JBoss division of Red Hat,Inc.

RushEye project is licensed under LGPL9 license and as it is can be freely redistributed.

7. http://mockito.org/8. http://dom4j.sourceforge.net/9. http://www.gnu.org/licenses/lgpl-2.1.html

27

Chapter 8

Test Results

Limiting the Scope of this Work

There exists variety of situations where can be this tool employed in context of webdevelopment. You can for example use it to compare captures from

1. subsequent releases of product,

2. different versions of one browser,

3. distinct browsers or

4. distinct application skins.

However in the scope of this work we focused on case testing subsequent releases (1)which will be described in detail later. This case is also considered to be part of releaseprocess of RichFaces framework.

The second method, testing different versions of one browser (2), is not considered at allbecause it doesn’t give web developer any important information about tested application.

All of the remaining cases (3, 4) are not investigated since the sample results have shownthat the perception algorithm and masking aren’t together enough successful to reliablyaccept suite how you can see later.

8.1 Identifying Errors

When starting deploying the comparison tool to practice, we first decided to compareimages from two runs of functional test suite on sample application from one release. Itrevealed fundamental problems:

• static animations

• random data

• time-dependent data

• non-deterministic functional tests

28

8. TEST RESULTS

Then we decided to compare two subsequent releases of RichFaces framework andfound new issue:

• release-specific data

Static Animations are caused by animated images sometimes used to provide user withimagination that data are processed in background (progress bar, status icon, etc.).

Random and Time-Dependent Data introduces non-determinisms which can beacceptable in functional testing (since we may reflect their context) but is completelyunacceptable in visual testing. Live Demo uses random data generation for most of the dataiteration components samples (data tables, grids).

Non-Deterministic Tests are written with assumption that output may differ butvalidation needs to be done in context of circumstances. Like in case of random data, visualtesting cannot deal with non-determinisms.

Release-Specific Data can be visual change occurring on all of the pages like new buttonfor navigation, changed header or the release version displayed on the page.

8.2 Result Stabilization

8.2.1 Applying Masks

Some of the static animations or release-specific data can be filtered out by masks.The preferred way is applying selective-alpha mask in cases where coordinates of

changes can be predicted.In other cases the ignore-bitmap mask can be used.

8.2.2 Functional Test Stabilization

The stabilization of tests are long-term process when applying good practices to thefunctional test suite. But describing good practices of writing Selenium tests isn’t in scopeof this work.

8.2.3 Suppressing Randomness and Time Factor

In Live Demo application we successfully employed the solution based on providing ownimplementation of classes causing non-determinism.

We identified classes java.util.Date and java.util.Random causing thedifferences and decided to provide own version to runtime.

29

8. TEST RESULTS

The method for injecting own versions of classes is based on using bootstrap classpath[5]. This method is based on fact that before loading of initial ClassLoader written in Java,it is necessary to use ClassLoader written in native code in order to initiate all dependenciesof pure Java ClassLoader. Because ClassLoader needs to by convention delegate upwhen looking for classes, we can provide the custom classes to native ClassLoadersatisfying that they will be loaded before the first pure-Java ClassLoader and all it’sdependencies be loaded.

The implementation of Date and Random classes then internally implements checkthat code is called in scope of your application. For this purpose stack trace is inspectedfor presence of classes from your application (or simply from some specific package). Thedeterministic algorithms are provided in the scope of your application. For example forint Random$getRandom() method you will get values looking like random but based onlinear transformations of initial seed value.

8.2.4 Application Stabilization

The best way to stabilize results however lies in ability to change the underlyingapplication. But this principle usually raises fundamental question how much we can go tobend our application to testable form. We rejected to use this principle in testing RichFacesFramework.

8.3 Stabilized results

After process of stabilization, the visual test suite containing 394 images was able torecognize 38 differences. The 14 images was identified as they can be filtered using oneof the stabilization methods. The final number is 6% of total number of images needed tocheck manually by user.

30

Chapter 9

Conclusion

The concept of visual testing of browser screen captures has been introduced and reasonsfor its introduction have been described in this work.

Sample of web application which might profit from introducing automation of visualtesting has been provided.

The reader has been familiarized with the application’s functional test suite and gotprinciples how this test suite can be transformed to visual testing suite.

The grammar for description of such suite has been defined and schema for itsrepresentation has been implemented in widely accepted format.

The tool which allows reading of the suite description and retrieving set of imagesand which accepts or rejects the differences between these images has been implemented.Additionally this tool was designed with the extensibility and the reusing of a code in mind.

The important step has shown several weak points which have to be consideredwhen trying to implement similar scenario. Solution for some of the weaknesses has beenprovided.

9.1 Space for Improvements

The scope of this work has not been intended to cover all of the corner cases whereintroduced principles might be used. Only the mechanisms which were necessary to beprojected to proof the concept of visual testing has been implemented and described.

When achieving completeness, the following list of additional thoughts, principles,extensions and improvements which are connected with the current implementation andwhich might be considered to implement in near future to help increase the level ofautomation and advance the user towards the idea of unattended visual testing whichrequires the tester’s attention only in cases where it is really needed.

Automated Evaluation

The analysis of the results is currently matter of proprietary piece of code and manual work.Although some of the steps to allow processing of visual suite results have been done. Byusing of provided schema of visual suite result it is possible to implement transformationsor querying results to obtain data in format analyzable by human.

31

9. CONCLUSION

Algorithm Modularization

By allowing the modularization of algorithm for image comparison it is possible toachieve deployment in areas outside of the scope targeted in this work. This principle mayeventually lead to need for improvements in current code base.

Tooling

Because this work counts with human-factor in analysis of results, the tooling might cometo mind to make it easy.

Especially the tool for reviewing and managing results retrieved from the continuousintegration system is required.

Two forms of such tool can be considered:

• web application as part of continuous integration interface

• desktop application

First of them, a web application, may be very useful for browsing and reviewing results,but also managing results in simple cases.

The latter, a desktop application, may be more suitable for batch operations andextensive changes.

Both tools definitely need to solve intuitive user interface simplifying work on largeamount of screen captures and need to consider off-line caching of downloaded images.

Version Control Awareness

As the test suite changes the source in time, the generated visual suite may start differing.Problem above needs to be solved by sophisticated methods of source code analysis

based on version control system inspection.

Caching Configuration

Current implementation does not allow other than linear method of loading configurationby parsing file. Thus it allows only forwarding method of processing suite description.

For implementations of parallel evaluation of visual suite while running functional testsuite, more sophisticated principles need to be developed.

Such implementation need introducing parser able to seek through suite description orindex it.

Capturing Part of Page

The principle of cropping the captured screen-shot to area containing specific applicationcomponent is not covered in this work.

32

9. CONCLUSION

Deployment in other areas

Although only one particular case where testing tool may successfully serve has beenintroduced here, it is possible to imagine lot of other situations where can be its featuresbeneficial.

33

Bibliography

[1] Apache maven, 2010. [available at http://maven.apache.org/; accessed 30-Dec-2010].

[2] Richfaces, 2010. [available at http://www.jboss.org/richfaces; accessed 30-Dec-2010].

[3] Richfaces live demo, 2010. [available at http://livedemo.exadel.com/richfaces-demo/index.jsp; accessed 30-Dec-2010].

[4] A. K. Jain. Fundamentals of Digital Image Processing. Prentice Hall, Inc., 1989.

[5] T. Neward. Using the bootclasspath: Tweaking the java runtime api. 2001. [availableat http://www.tedneward.com/files/Papers/BootClasspath/BootClasspath.pdf;accessed 05-Jan-2010].

[6] D. Pascale. A Review of RGB Color Spaces ...from xyY to R’G’B’. 5700 Hector Desloges,Montreal (Quebec), Canada HIT 3Z6, 2003.

[7] S. Upton. Delta-e: The color difference. CHROMiX ColorNews, (17), 2005. [available athttp://www2.chromix.com/colorsmarts/smartNote.cxsa?snid=1145; accessed 04-Jan-2010].

[8] H. Yee. Perceptual image diff, 2004. [available at http://pdiff.sourceforge.net/; accessed26-Dec-2010].

34

Appendix A

Models

35

A. MODELS

Figure A.1: Interfaces

36

A. MODELS

Figure A.2: Illustration of Comparison Process37

A. MODELS

Figure A.3: Suite-Classes

38

Appendix B

Visual Suite Descriptors Samples

39

B. VISUAL SUITE DESCRIPTORS SAMPLES

1 <?xml version="1.0" encoding="UTF-8"?>2 <visual-suite xmlns="http://www.jboss.org/test/visual-suite"3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"4 xsi:schemaLocation="http://www.jboss.org/test/visual-suite">5

6 <global-configuration>7 <listener type="org.jboss.lupic.parser.listener.CompareListener">8 <result-listener>org.jboss.lupic.result.ResultStorageListener</result-listener>9 <storage-class>org.jboss.lupic.storage.FileStorage</storage-class>

10 </listener>11 12 <pattern-retriever type="org.jboss.lupic.retriever.FileRetriever" />13 <mask-retriever type="org.jboss.lupic.retriever.FileRetriever" />14 <sample-retriever type="org.jboss.lupic.retriever.sample.FileSampleRetriever" />15 <perception>16 17 <one-pixel-treshold>5</one-pixel-treshold>18 20 <global-difference-treshold>10</global-difference-treshold>21 22 <global-difference-pixel-amount>1%</global-difference-pixel-amount>23 </perception>24 <mask id="xxx" type="ignore-bitmap" source="xxx.png" />25 <mask id="yyy" type="selective-alpha" source="yyy.png" vertical-align="bottom" />26 </global-configuration>27

28 <test name="AjaxSupportTestCase.testEmpty">29 <mask id="xxx" type="ignore-bitmap" source="xxx.png" />30 <pattern name="AjaxSupportTestCase.testEmpty.1"31 source="AjaxSupportTestCase.testEmpty.1.png" />32 <pattern name="AjaxSupportTestCase.testEmpty.2"33 source="AjaxSupportTestCase.testEmpty.2.png" />34 </test>35

36 <test name="BeanValidatorTestCase.testAgeIntegerOnly:168">37 <perception>38 <global-difference-pixel-amount>10px</global-difference-pixel-amount>39 </perception>40 <pattern name="BeanValidatorTestCase.testAgeIntegerOnly:168.1">41 <xxx>BeanValidatorTestCase.testAgeIntegerOnly:168.png</xxx>42 <yyy>10</yyy>43 </pattern>44 </test>45 </visual-suite>

Figure B.1: Example of Suite Descriptor

40

B. VISUAL SUITE DESCRIPTORS SAMPLES

1 <?xml version="1.0" encoding="UTF-8"?>2 <visual-suite-result xmlns="http://www.jboss.org/test/visual-suite-result"3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"4 xsi:schemaLocation="http://www.jboss.org/test/visual-suite-result">5

6 <test name="AjaxSupportTestCase.testEmpty">7 <pattern name="p1" result="DIFFER"8 output="AjaxSupportTestCase.testRelative.1.png">9 <comparison-result>

10 <area width="558" height="1145" />11 <rectangle>12 <min x="0" y="5" />13 <max x="10" y="140" />14 </rectangle>15 <total-pixels>5</total-pixels>16 <masked-pixels>15</masked-pixels>17 <perceptible-different-pixels>345</perceptible-different-pixels>18 <global-different-pixels>53</global-different-pixels>19 <unperceptible-different-pixels>2345</unperceptible-different-pixels>20 <same-pixels>3245</same-pixels>21 </comparison-result>22 </pattern>23 <pattern name="p2" result="SAME" />24 <pattern name="p3" result="NOT_TESTED" />25 </test>26 <test name="BeanValidatorTestCase.testAgeIntegerOnly:168">27 <pattern name="p4" result="DIFFER" />28 <pattern name="p5" result="DIFFER" />29 <pattern name="p6" result="DIFFER" />30 </test>31 <test name="xxx">32 <pattern name="p7" result="PERCEPTUALLY_SAME" />33 <pattern name="p8" result="NOT_TESTED" />34 <pattern name="p9" result="NOT_TESTED" />35 </test>36 <test name="yyy">37 <pattern name="p10" result="ERROR">38 <error-output>Stack trace...</error-output>39 </pattern>40 <pattern name="p11" result="SAME" />41 </test>42 </visual-suite-result>

Figure B.2: Example of Result Descriptor

41

Appendix C

Sample and Difference Images

42

C. SAMPLE AND DIFFERENCE IMAGES

Figure C.1: Sample image captured by Live Demo’s Functional Test Suite

43


Figure C.2: RushEye detected small change in styling of ComboBox component

44


(a) The server date has changed between the times of capturingscreen

(b) The first sample of random data

(c) The second sample of random data

Figure C.3: The differences in time-dependent and random generated data

45


(a) The one pixel significantlydifferent because of staticanimation

(b) Non-deterministic test influenced bynetwork latency

(c) Image was not available in the time of capture

(d) The progress bar givinguser imagination of work inbackground requires staticanimation

(e) The release version haschanged between releases

Figure C.4: The samples of changes which requires stabilization

46


(a) The styling of the Extended Data Table has changed between releases(it can lead into inconsistent design changes in customer’s application)

(b) The font-weight of the Search link has changed

(c) Description text was changed

(d) Application doesn’t remember the active page after refresh

(e) Positioned of the box has changedsignificantly

Figure C.5: Captures of screen showing the positive detection of error on tested page

47

Documents

Visual Testing of Browser Screen Captures - Muni