Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
Spot the difference Automating visual
regression testing
@11vlr
@11vlr
Spot the difference Automating visual
regression testing
• Automated testing is more reliable, as
it is performed by tools and/or scripts.
• Automated testing is executed by
software tools, so it is significantly
faster than a manual approach.
WHY AUTOMATE END
TO END UI TESTS?
@11vlr
@11vlr
• Investment is required
• Automated testing is a practical option
when the test cases are run
repeatedly over a long time period.
• Automated testing entail
human observation and cannot
guarantee user-friendliness or positive
customer experience.
COMMON UI
AUTOMATION ISSUES
DOES NOT
IT CAN BE A SLIPPERY SLOPE
@11vlr
ICE-CREAM CONE ANTI-PATTERN
@11vlr
Unit
Service
UI
Manual & Exploratory
Testing
£
£££
Adaption of https://watirmelon.blog/testing-pyramids/
HOUR GLASS ANTI-PATTERN
@11vlr
£
£££
£
£££
UI
Unit
Manual & Exploratory
Testing
Adaption of: https://www.getautoma.com/blog/the-test-hourglass
@11vlr
WHEN YOU HAVE
UNIT BUT NO
INTEGRATION TESTS
@11vlr
TESTINGfrAgile
TESTING PYRAMID
@11vlr
£
£££
UI
Service
Unit
Manual & Exploratory
Testing
Adaption of https://martinfowler.com/bliki/TestPyramid.html
@11vlr
WHY NOT JUST STICK
WITH MANUAL
TESTING THEN?
• Manual testing allows for human
observation, which may be more
useful if the goal is user-friendliness
or improved customer experience.
UI
Service
Unit
Manual & Exploratory
Testing
@11vlr
• Can be time intensive
• Is not accurate at all times due to
human error, hence it is less reliable.
• Is time-consuming, taking up human
resources.
• Is only practical when the test cases
are run once or twice, and frequent
repetition is not required.
MANUAL TESTING
WE DON’T HAVE TIME TO AUTOMATE
@11vlr
CATCH 22
@11vlr
We don’t automate
We don’t have time
BECAUSE BECAUSE
@11vlr
MANY OPTIONS
@11vlr
• Open source; anybody can see the
code and help
• Big community of support
• Can operate on almost any OS
• Multiple language support (Python,
Ruby, .Net, PHP, Java, JavaScript)
• Supports a range of browsers like
Opera, Chrome, IE (6,7,8,9,10,11),
Mozilla Firefox, Safari
SELENIUM
IT’S NOT A SILVER BULLET
@11vlr
@11vlr
• Speed to execute tests
• Render differently
BROWSER
COMPATIBILITY
ISSUES
EXECUTION SPEED
@11vlr
MAINTAINABILITY
@11vlr
@11vlr
HOW VALUABLE ARE THESE TESTS?
@11vlr
WHAT THE….
@11vlr
ALL IS NOT WHAT IT SEEMS
@11vlr
MAKING TESTS LESS
FLAKY
@11vlr
• Page object patterns
MAKING TESTS LESS
FLAKY
http://martinfowler.com/bliki/PageObject.html
@11vlr
• Qa-selectors
MAKING TESTS LESS
FLAKY
@11vlr
Qa- selectors
@11vlr
HOW GOOD ARE YOU AT SPOT THE
DIFFERENCE
@11vlr
HOW MANY DIFFERENCES CAN YOU FIND?
@11vlr
HOW MANY DIFFERENCES DID YOU FIND?
@11vlr
THERE WERE 11 DIFFERENCES
@11vlr
HOW DO YOU KNOW WHEN YOU’VE
FOUND THEM ALL?
@11vlr
CAN WE ADD MORE VALUE TO OUR
TESTS?
@11vlr
UI
Service
Unit
Manual & Exploratory
Testing
WHAT IF WE COULD SIMPLY
@11vlr
equalsAssert
VISUAL TESTING
@11vlr
@11vlr
• Didn’t work straight out of the box
and/or contained lots of bloat
• Steep learning curve
• Small community so lack of support
• Needed to work with our current
automation framework
• Costs
ISSUES WITH OTHER
OFFERINGS
@11vlr
• Been in development 1 year
• Facing many challenges which are
because of the products we offer, the
ways in which we have developed
them and the way in which we test
them
ROLLING OUR OWN
@11vlr
• Browser rendering issues
RELIABILITY
@11vlr
EXECUTION SPEED
@11vlr
HUMAN VS MACHINE
@11vlr
HUMAN VS MACHINE
@11vlr
LETS ZOOM IN A LITTLE
@11vlr
LETS ZOOM IN A LITTLE BIT MORE…
@11vlr
GOT HIM!
@11vlr
HOW QUICK WAS THE MACHINE?
@11vlr
MAINTAINABILITY
Old code
@11vlr
MAINTAINABILITY
Old code
New code using some helpers
@11vlr
MAINTAINABILITY
New code using Specflow
New code using some helpers
@11vlr
EASILY IDENTIFY THE DIFFERENCES
@11vlr
ACCURACY
@11vlr
IMAGE QUALITY DIFFERENCE
@11vlr
CHALLENGES SO FAR
@11vlr
MAINTAINING BASE
IMAGES
Keep a base image for each driver
type
@11vlr
SCREENSHOTS
• Viewport (default - WebDriver
W3C standard)
• Full Screen (stitching)
• Capture images of just
elements
@11vlr
SCREEN RESOLUTIONS
@11vlr
SCREEN RESOLUTIONS
@11vlr
Responsive design
@11vlr
HARDWARE ISSUES
Machine hardware rendering
snapshots slightly differently
@11vlr
STORAGE
Database / Online storage – file
sizes (many images)
@11vlr
DYNAMIC CONTENT
@11vlr
DYNAMIC CONTENT
class=“attendeesLink"
@11vlr
MOVING ELEMENTS
• CSS animations
• Videos
@11vlr
VISUALLY CHECKING DOCUMENTS
@11vlr
PDF DYNAMIC
CONTENT
• A way to blanket content within
documents
UPCOMING CHALLENGES
@11vlr
@11vlr
MORE COMPARISON
OPTIONS
• Text only
• Layout only
ANOTHER TOOL TO ADD TO YOUR BELT
@11vlr
@11vlr
THANK YOU
Git repository: http://bit.ly/VisualTesting
NuGet.org Package:
@11VLR
WWW.VIVRICHARDS.CO.UK