1

MYTHS OF AUTOMATION TESTING

Author

Abhishek Saxena

2

INTRODUCTION

In this presentation we shall examine some of the most common myths of test automation and how these prevent organizations from succeeding in test automation.

3

SETTING REALISTIC EXPECTATIONS

Probably the most difficult and challenging aspect of any test automation endeavour is to understand the limitations of automated testing and setting realistic goals and expectations to avoid disappointments.

4

AUTOMATION TESTING IS BETTER THAN MANUAL TESTING

Automation testing is not actually testing. In fact it is checking of facts. If we have knowledge of system, than we can enforce this understanding in

form of checks on the system and running them automatically.On the other hand testing is investigative approach towards the system under

test, in order to obtain new information/behavior of system by exploring it.One should not entirely rely on either approach as as both methods are

required to higher quality of the application.

5

ACHIEVING 100% COVERAGE BY AUTOMATION TESTING

There is no practical way to achieve 100% test coverage (due to endless possible permutations) using either manual or automation testing approach.

Test coverage can only be increased by automation testing with more data/configurations/OS/browsers etc.

More automation tests don’t guarantee better quality or better confidence. It depends on how good a test is designed.

Organization’s focus should be to cover the most important area of functionality which is crucial to the business rather than 100% test coverage.

QUICK ROI (RETURN-ON-INVESTMENT)

Automation testing is not just scripting test cases, it needs framework to arrange/ customize useful operations such as test case selection, reporting, data driven, etc.

Development of framework is project on it’s own and it requires skilled developers and takes time to build.

Even if a fully functional framework is available, scripting a automated test case takes longer than executing it manually.

If we need a quick feedback on recently developed new feature checking it manually is usually quicker than automating the test.

However, the ROI is returned in the long run when we need to execute the same tests in regular intervals.

6

7

HIGHER RATE OF DEFECT DETECTION BY AUTOMATION TESTING

It is expected from automation testing to find lots of bugs because of allegedly increased test coverage, but in reality this is not the case.

Automated tests are good at catching regression issues and the number of regression issues, in most cases, tends to be far less than new functionality that’s being developed.

Automated scripts only check what they have been programmed to check by the person who wrote the script and it leaves the probability of major flaws remain unnoticed if script writer doesn’t cover the case in the script.

8

AUTOMATION REQUIRES AT UNIT LEVEL ONLY

Unit level automation testing is good at finding programming errors, not the failures as there is a much larger aspect of testing when all the components are tied together at system level.

Scripting automation checks for functionality under development is a daunting task as the new functionality tends to be volatile (subject to many changes) during development.

9

AUTOMATION REQUIRES AT SYSTEM LEVEL - UI ONLY

UI is constantly changing to enhance visual design and usability and automation checks may fail due to UI changes.

Execution speed of automation UI checks are much slower comparative to unit or API level. In case of occurring a bug the root cause analysis takes longer because it is not easily apparent where the bug is.

10

LOSING FAITH AND TRUST IN TEST AUTOMATION

It’s really not a myth, but it’s a side effect when test automation goes wrong. If the team has no visibility or knowledge on what is automated and executing,

they either release with fear of unknown or duplicate their regression testing efforts.

If the automated checks are flaky, slow, give intermittent results then it can confuse the team more than providing a safety net and a confidence booster.

Don’t hesitate of removing automated checks that are always failing or give inconsistent results. Instead aim for a clean and reliable suite of tests that can give correct indications of the health of the application.

11

CONCLUSION

Test Automation is a long term investment. It will take time and expertise in developing and maintaining test automation frameworks and automated scripts.

Test automation is not a one-off effort where you deliver a solution and let it run. It needs constant monitoring and updating.

Rather than aiming to replace manual QAs or expecting the automated checks to find lots of defects, we should instead embrace the advantages it brings to the team

liberating QA’s time for more exploratory testing where chances of revealing defects is maximized, or using automated scripts to create test data that can be used for manual testing.

Understanding the limitations and setting realistic expectations is important in overcoming these myths of test automation.

12

THANK YOU!