Upload
alan-richardson
View
266
Download
1
Embed Size (px)
Citation preview
Effective Software Testing for ModernSoftware Development
Agile Tour London ‐ October 2017
Alan Richardson
www.eviltester.com/agilewww.compendiumdev.co.uk@eviltester
@EvilTester 1
Common Questions About ModernSoftware Development, Can you...
Develop Software without Testing?
Develop Software without Testers?
@EvilTester 2
Have we had such bad experiences ofSoftware Testing?
@EvilTester 3
We tend to argue over names anddefinitions
"Agile", "Devops""Automation Pyramid", "ATDD", "BDD", "TDD", "ContinuousIntegration", "Continuous Delivery""Tester", "Test Automator", "SDET"
@EvilTester 4
We argue over roles, responsibilities andapproaches
Who should test on Agile Projects?Should testers code?Can programmers test?
Can we automate all testing?Should we automate through the GUI?
@EvilTester 5
Most of this seems like the wrong focus
@EvilTester 6
We need a customised and tailored process forour organisation.
We build a System Of DevelopmentWe want the System to work effectively in our environmentPeople are part of that System
We Automate as part of that System
@EvilTester 7
Alan Richardson: Software DeveloperI Test Things ﴾Tester﴿
I help people test Better ﴾Consultant﴿I develop software ﴾Programmer﴿I learn to develop software better ﴾Practitioner﴿etc.
@EvilTester 8
"I know that I am not a category. I amnot a thing ‐ a noun. I seem to be a verb,an evolutionary process ‐ an integralfunction of the universe."R. Buckminster FullerI Seem to Be A Verb, 1970
@EvilTester 9
You can... Develop software withouttesting
If you don't want to know if it is good enough.
@EvilTester 10
You Can... Develop Software withoutTesters
If you call the people doing the testing a differentname.
@EvilTester 11
Those are Verbal Hacks
@EvilTester 12
"What is a mathematician ‐ someonewho does mathematics?"Ian StewartThe Magical Maze, 1997
@EvilTester 13
"...a mathematician is someone who seesopportunities for doing mathematicsthat the rest of us miss."Ian StewartThe Magical Maze, 1997
@EvilTester 14
"...a tester is someone who seesopportunities for doing testing that therest of us miss."Paraphrasing: Ian Stewart, The Magical Maze
@EvilTester 15
Software development used to look easywhen it was a long line of specialistprocesses
@EvilTester 16
Modern software development is hardbecause we have to customiseeverything
@EvilTester 17
Have we carried forward too manyconcepts from Legacy SoftwareDevelopment?
@EvilTester 18
Guiding Principle: Regularly ReleaseSoftware That Works
@EvilTester 19
Hard, but worth itBetter Quality ProductSpeed of delivering requirementsMore interesting workBetter quality staff
@EvilTester 20
What is Modern Software Development?A mess of terms ‐ TDD, BDD, ATDD, Agile, Devops, CI, CDA Plethora of Technologies
commonalities ‐ flow, fast, automated execution and assertion,short cycles, more releases, automated deployment pipeline
@EvilTester 21
What is Not Modern SoftwareDevelopment?
Development without testingTools above Concepts
Development without riskAccumulation of Technical Debt
@EvilTester 22
Modern software Development goes fastbecause it has more safety controls
@EvilTester 23
Risk: A Process might only look andsound like Modern SoftwareDevelopment
@EvilTester 24
Historic Software Development calledthe safety control "Testing"
@EvilTester 25
Historic Software Development created aTesting Process around...
Test Phases that related to Development Phases
Test Strategy ‐ AnalysisTest Planning ‐ Design
Test Cases & Scripts ‐ ProgrammingTest Execution ‐ Build and Deployment
... but that was never "Testing", that was the part of the SoftwareDevelopment process that The Test Team performed
@EvilTester 26
Modern Software Development hasdropped phases
Analysis PhaseDesign PhaseProgramming Phase
Testing PhaseDeployment PhaseSupport Phase
@EvilTester 27
We need to separate the concept of"Testing" from...
A Testing PhaseA Formal Testing ProcessAn Outsourced Activity
An Activity that only Testers doetc.
We need to work out what "Testing" really is.
@EvilTester 28
Software Testing is part of The SoftwareDevelopment Process
Testing Process needs to be created tocontextually fit the Development Process.
@EvilTester 29
Modern Software DevelopmentProcesses Build in Safety Controls Basedon Risk
@EvilTester 30
Risk that:we build something the user doesn't want
prioritised backlog, MVP, agreed acceptance criteria,release working implementation regularly, work in smallchunks ﴾stories﴿, review stories when 'done'
we break something as we build new stuffautomate acceptance criteria assertions, automated unittesting, continuous integration
We create a disciplined process to mitigate risk.
@EvilTester 31
Testing has always been related to Risk
@EvilTester 32
But there is still a risk thatthe acceptance criteria are not complete enough
we don't cover edge cases
we only focus on acceptance criteria and ignore technical riskwe don't test integration and interfaceswe don't step back enough to review and test the wholesystemperformance, emergent behaviour, failover, security, data, etc.
@EvilTester 33
Testing Takes a Risk & Uncertainty Focus
@EvilTester 34
What fundamentally is testing?
@EvilTester 35
TDDTest before writing code
testing is modelling, design, specificationtest fails
Write codeRun Test
Assert on specific conditions we agree in advance
Test test scope ﴾review﴿tests document the scope of developmentAny more tests needed?
Done ‐ Exit
@EvilTester 36
A side‐effect of TDD is a set of code...that we can use to continually assert that the conditions we wrotethe code to meet, are still met.
We sometimes refer to that using the legacy term"Regression Testing"
@EvilTester 37
Do we still need to test if we do TDD?Integration risk mitigation?
What about missing @Test ?What about missing asserts?Just enough to drive code, not always enough to cover all usesof the code
e.g. exceptions, null, bad dataWhat about future unplanned usage?
e.g. add(int, int) tested against 1‐100 , used with Integer.MAX_VALUE
@EvilTester 38
Basic Model of a Testing Process
@EvilTester 39
"Observers are men, animals, ormachines able to learn about theirenvironment and impelled to reducetheir uncertainty about the events whichoccur in it, by dint of learning."Gordon PaskAn approach to Cybernetics, 1961
@EvilTester 40
Despite Observation being an importantpart of the process of Testing, we alsotest to increase our uncertainty.
New RisksNew QuestionsNew Decisions
@EvilTester 41
Automated ATDD, BDDVery often:
this is not Driving Development ﴾DD﴿
this is automated DSL based acceptance condition assertionswe execute against deployed and integrated systemsRequirement Risk Based
We sometimes refer to that using the legacy term"Acceptance Testing"
@EvilTester 42
Do we still need to test if we do ATDD?Very often:
we spend so much time automating ATDD, BDD we don't leaveenough time to test
Automation backlogTest sprints
we spend time maintainingRequirements not complete, examples don't cover full riskprofile
@EvilTester 43
Acceptance TestingTom Demarco, 1978, "Structured Analysis and System Specification"
"The Specification is the Acceptance Test"
...
"Acceptance tests are derived from the Target Document andfrom the Target Document alone"
"Acceptance tests will fall into the following five categories:normal path tests, exception path tests, transient state tests,performance tests, special tests"
@EvilTester 44
Special Tests"It would be naive to think that any system, no matter howcomplex, could be tested completely by the derived test setsas I have explained them."
"...they must be placed explicitly in the Structured Specificationbefore it is considered complete. This will maintain thestandard that all the criteria for acceptance are contained inthe Structured Specification"
Exploratory Testing to Learn
Add what we learn to the Acceptance Criteria and coverage
@EvilTester 45
We still need to test because we makemistakes and we miss things out
Risks
and we might not want to assert on them continuously
@EvilTester 46
Risk: Overlap Between Feeding ModelsTDD, ATDD, BDD:
all use the same models as Testingresult in Automated Execution and Condition Assertion
We want to avoid a lot of overlap.
@EvilTester 47
Automating Vs Testing Vs Developing
@EvilTester 48
Abstraction
"The purpose of abstraction is not to be vague,but to create a new semantic level in which onecan be absolutely precise."
E. W. Djkstra1972 ACM Turing Lecture: 'The Humble Programmer'
@EvilTester 49
No Abstractions ‐ Web Testing
@Testpublic void canCreateAToDoWithNoAbstraction(){ WebDriver driver = new FirefoxDriver(); driver.get( "http://todomvc.com/architecture‐examples/backbone/");
int originalNumberOfTodos = driver.findElements( By.cssSelector("ul#todo‐list li")).size();
WebElement createTodo; createTodo = driver.findElement(By.id("new‐todo")); createTodo.click(); createTodo.sendKeys("new task"); createTodo.sendKeys(Keys.ENTER); assertThat(driver.findElement( By.id("filters")).isDisplayed(), is(true));
int newToDos = driver.findElements( By.cssSelector("ul#todo‐list li")).size(); assertThat(newToDos, greaterThan(originalNumberOfTodos));}
@EvilTester 50
But there were Abstractions
WebDriver ‐ abstraction of browser
FirefoxDriver ‐ abstraction Firefox ﴾e.g. no version﴿WebElement ‐ abstraction of a DOM elementcreateToDo ‐ variable ‐ named for clarityLocator Abstractions via methods findElement findElements
Manipulation Abstractions .sendKeys Locator Strategies By.id ﴾did not have to write CSS Selectors﴿...
Lots of Abstractions ‐ just no Domain Abstractions
@EvilTester 51
Using Abstractions ‐ Same flow
@Testpublic void canCreateAToDoWithAbstraction(){ TodoMVCUser user = new TodoMVCUser(driver, new TodoMVCSite());
user.opensApplication(). and(). createNewToDo("new task");
ApplicationPageFunctional page = new ApplicationPageFunctional(driver, new TodoMVCSite());
assertThat(page.getCountOfTodoDoItems(), is(1)); assertThat(page.isFooterVisible(), is(true));}
@EvilTester 52
What Abstractions were there?Domain level abstractions
TodoMVCUser
User
Intent/Action/Behaviour BasedHigh Level
ApplicationPageFunctional
StructuralModels the rendering of the application page
﴾see also Domain Driven Design by Eric Evans﴿
@EvilTester 53
Automating With Abstractions SupportsTesting
regular automated execution with assertions
creation and maintenance by many rolesexploratory testingmodel based automated executionperformance testing
multithreaded bots
potential re‐use in performance/stress tools
@EvilTester 54
Adaptation
"If the people can not adapt to the methods, thenthe methods must be adapted to the people."
E.F. Schumacher,
Small is Beautiful, 1973
@EvilTester 55
We are creating A System ofDevelopmentEverything that applies to the System Under Development appliesto our Development System
Testing as "opportunities for doing testing that the rest of usmiss."Systems have risksTesting Targets Risk
Re‐think testing in terms of RiskAutomate Strategically to Mitigate Risk
Flexible Abstractions
@EvilTester 56
EndAlan Richardson www.compendiumdev.co.uk
Linkedin ‐ @eviltesterTwitter ‐ @eviltesterInstagram ‐ @eviltesterFacebook ‐ @eviltester
Youtube ‐ EvilTesterVideosPinterest ‐ @eviltesterGithub ‐ @eviltesterSlideshare ‐ @eviltester
@EvilTester 57
BIOAlan is a test consultant who enjoys testing at a technical levelusing techniques from psychotherapy and computer science. In hisspare time Alan is currently programming a multi‐user textadventure game and some buggy JavaScript games in the style ofthe Cascade Cassette 50. Alan is the author of the books "Dear EvilTester", "Java For Testers" and "Automating and Testing a REST API".Alan's main website is compendiumdev.co.uk and he blogs atblog.eviltester.com
@EvilTester 58