53
Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools [email protected] ADVANCED AUTOMATED VISUAL TESTING WITH SELENIUM

SeConf2015: Advanced Automated Visual Testing With Selenium

Embed Size (px)

Citation preview

Page 1: SeConf2015: Advanced Automated Visual Testing With Selenium

Adam Carmi (@carmiadam)

Co-Founder and VP R&D at Applitools

[email protected]

ADVANCED AUTOMATED VISUAL TESTING WITH

SELENIUM

Page 2: SeConf2015: Advanced Automated Visual Testing With Selenium

YOU CAN AND SHOULD

AUTOMATE YOUR

VISUAL TESTS!

Page 3: SeConf2015: Advanced Automated Visual Testing With Selenium
Page 4: SeConf2015: Advanced Automated Visual Testing With Selenium

AGENDA

Why automated visual testing?

Tools & Technology

Where does it fit?

Q & A

Page 5: SeConf2015: Advanced Automated Visual Testing With Selenium

WHAT IS VISUAL TESTING?

A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users

Page 6: SeConf2015: Advanced Automated Visual Testing With Selenium

A VISUAL BUG

Page 7: SeConf2015: Advanced Automated Visual Testing With Selenium

AND ANOTHER…

Page 8: SeConf2015: Advanced Automated Visual Testing With Selenium

AMAZON PRIME DAY – JULY 15th, 2015

Page 9: SeConf2015: Advanced Automated Visual Testing With Selenium

WHY SHOULD IT BE AUTOMATED?

THE TEST MATRIX IS TOO BIG TO COVER MANUALLY Web browsers Devices Operating systems Screen resolutions Responsive design L10n 3rd Party upgrades

Page 10: SeConf2015: Advanced Automated Visual Testing With Selenium

WHY SHOULD IT BE AUTOMATED?

Page 11: SeConf2015: Advanced Automated Visual Testing With Selenium

WHY SHOULD IT BE AUTOMATED?

NATIVE / HYBRID MOBILE APPS Harder to roll back changes

Can’t push daily

Updates take battery and data

Higher quality bar

Page 12: SeConf2015: Advanced Automated Visual Testing With Selenium

WHY SHOULD IT BE AUTOMATED?

Many are already doing it…

PhantomCSSFighting Layout Bugs

CSS Critc Wraith

Needle

Grunt PhotoBox

dpdxt

WebdriverCSS

EyesHuxley

FBSnapshotTestCase

GeminiSelenium Visual Diff

VisualCeption

Specter

Snap And Compare

kobold

Page 13: SeConf2015: Advanced Automated Visual Testing With Selenium

AGENDA

Why automated visual testing?

Tools & Technology

Where does it fit?

Q & A

Page 14: SeConf2015: Advanced Automated Visual Testing With Selenium

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

Page 15: SeConf2015: Advanced Automated Visual Testing With Selenium

DEMOhttps://github.com/webdriverio/webdrivercss

Page 16: SeConf2015: Advanced Automated Visual Testing With Selenium

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

Page 17: SeConf2015: Advanced Automated Visual Testing With Selenium

SCREENSHOTS

Full page? Regions? Frames? Device pixel ratio? Rotation? Viewport size? Page preparation? Page stabilization?

Page 18: SeConf2015: Advanced Automated Visual Testing With Selenium

QUICK FEEDBACK TOOLS

MOTIVATION

Get fast feedback on code

changes

Run tests locally in the

background

SETUP

Render screenshots with a

headless browser

PhatomJS, SlimerJS

Configuration file driven tests

CONS

Partial coverage

Chrome, IE?

Limited navigation

Tests stale browser versions

Page 19: SeConf2015: Advanced Automated Visual Testing With Selenium

VISUAL COVERAGE TOOLS

MOTIVATION

Verify that the app / site looks right

in all real execution environments

Test as many UI states as possible

Execute many tests in parallel

SETUP

Render screenshots using real

browsers on a variety of

operating systems and devices

in parallel

WebDriver, Grid

Code driven tests

WebDriver, DSLCONS

Requires test infrastructure

Implement and maintain test

code

Page 20: SeConf2015: Advanced Automated Visual Testing With Selenium

SELENIUM BASED TOOLS

CODE / SCRIPT

Needle (Python WD)

WebDriverCss (JS WebDriverIO)

Gemini (JS DSL)

Selenium Visual Diff (Java WD)

VisualCeption (PHPCodeCeption)

Pix-Diff (JS Protractor)

Shoov (JS WebDriverIO)

Vizregress (.NET WD)

Rspec Page Regression(Capibara)

Applitools Eyes (All WD + Appium)

Fighting Layout Bugs (JavaWebDriver)

Huxley (Record Playback)

CONFIGURATION

Viff

CSS Visual Test

GreenOnion

Wraith-Selenium

Page 21: SeConf2015: Advanced Automated Visual Testing With Selenium

NON-SELENIUM BASED TOOLS

CODE / SCRIPT

PhantomCSS (JS CasperJS)

Specter (JS DSL)

FBSnapshotTestCase (XCTest)

CONFIGURATION

Wraith

Dpdxt

Grunt PhotoBox

Grunt-Vigo

Snap And Compare

BackstopJS

CSSCritic

Kobold

DiffCop

SUCCSS

MANUAL

Eyes Express (Chrome)

Page 22: SeConf2015: Advanced Automated Visual Testing With Selenium

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

Page 23: SeConf2015: Advanced Automated Visual Testing With Selenium
Page 24: SeConf2015: Advanced Automated Visual Testing With Selenium

WHY NOT?

FALSE POSITIVES

Page 25: SeConf2015: Advanced Automated Visual Testing With Selenium

ANTI-ALIASING?

Page 26: SeConf2015: Advanced Automated Visual Testing With Selenium

ANTI ALIASING 1/2

Page 27: SeConf2015: Advanced Automated Visual Testing With Selenium

ANTI ALIASING 2/2

Page 28: SeConf2015: Advanced Automated Visual Testing With Selenium

BRIGHTNESS 1/2

Page 29: SeConf2015: Advanced Automated Visual Testing With Selenium

BRIGHTNESS 2/2

Page 30: SeConf2015: Advanced Automated Visual Testing With Selenium

TEXT POSIONING 1/2

Page 31: SeConf2015: Advanced Automated Visual Testing With Selenium

TEXT POSIONING 2/2

Page 32: SeConf2015: Advanced Automated Visual Testing With Selenium

IMAGE SCALING 1/2

Page 33: SeConf2015: Advanced Automated Visual Testing With Selenium

IMAGE SCALING 2/2

Page 34: SeConf2015: Advanced Automated Visual Testing With Selenium

ANDMORE…

1 pixel offsets in element positioning

Dynamic content

Moving elements

Images of different size

Performance

Page 35: SeConf2015: Advanced Automated Visual Testing With Selenium
Page 36: SeConf2015: Advanced Automated Visual Testing With Selenium

Image Comparison APIs

Page 37: SeConf2015: Advanced Automated Visual Testing With Selenium

ImageMagick A powerful command line tool for image processing.

APIs are available for most programming languages.

Fuzzing is used to eliminate slight color differences

An error ratio is usually used to determine a match

$ compare –metric AE –fuzz 5% img1.png img2.png diff.png

2246

Page 38: SeConf2015: Advanced Automated Visual Testing With Selenium

The Javascript Engines Resemble.js

Pixel by pixel + error ratio + anti-aliasing

http://huddle.github.io/Resemble.js

Blink-DiffPixel by pixel + error ratio + anti-aliasing + perceptual color distance computation

https://github.com/yahoo/blink-diff

Looks-SamePixel by pixel + perceptual color distance computation + ignore Caret

https://github.com/gemini-testing/looks-same

Page 39: SeConf2015: Advanced Automated Visual Testing With Selenium

Applitools Eyes A specialized image processing stack designed to compare computer

generated UI images

Handles anti-aliasing, pixel offsets, color similarity, and image scaling

Dynamic and moving content

Compare images of different sizes

No error ratio configuration required

Validates full UI pages

Fast!

Supports layout matching

Page 40: SeConf2015: Advanced Automated Visual Testing With Selenium

DEMO

Page 41: SeConf2015: Advanced Automated Visual Testing With Selenium

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

Page 42: SeConf2015: Advanced Automated Visual Testing With Selenium

REPORT DIFFERENCESAs files on the file system (combined with source control)

Page 43: SeConf2015: Advanced Automated Visual Testing With Selenium

REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)

Page 44: SeConf2015: Advanced Automated Visual Testing With Selenium

THE WORKFLOW

Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

Page 45: SeConf2015: Advanced Automated Visual Testing With Selenium

UPDATE THE BASELINE

Rename or commit individual image files

GUI (Gemini GUI)

Page 46: SeConf2015: Advanced Automated Visual Testing With Selenium

UPDATE THE BASELINE

Overwrite mode

Automatic maintenance (Applitools Eyes)

Page 47: SeConf2015: Advanced Automated Visual Testing With Selenium

AGENDA

Why automated visual testing?

Tools & Technology

Where does it fit?

Q & A

Page 48: SeConf2015: Advanced Automated Visual Testing With Selenium

WHERE DOES IT FIT?

• Component

s

• Code

review

• Developers

• Designers

• QA

Visual testing of frontend components by

frontend developers

Page 49: SeConf2015: Advanced Automated Visual Testing With Selenium

WHERE DOES IT FIT?

• Pages

• Page

sections

• Developers

• Designers

• QA

• Others

Full or partial validation of application

screens

A collaboration tool

Page 50: SeConf2015: Advanced Automated Visual Testing With Selenium

WHERE DOES IT FIT?

• Staging vs.

Production

• Ops

• QA

Validate your staging deployment using your

production deployment as a baseline

Page 51: SeConf2015: Advanced Automated Visual Testing With Selenium

WHERE DOES IT FIT?

• Monitoring

• Ops

• QA

No missing resources in production

No corruption due to 3rd party data

No corruption due to browser / OS upgrades

Page 52: SeConf2015: Advanced Automated Visual Testing With Selenium

QUESTIONS?

Page 53: SeConf2015: Advanced Automated Visual Testing With Selenium

Adam Carmi (@carmiadam)

Co-Founder and VP R&D at Applitools

[email protected]

THANK YOU