Automated Software Testing - Software Automated Software Testing Maurأ­cioAniche m.f.aniche@

  • View
    2

  • Download
    0

Embed Size (px)

Text of Automated Software Testing - Software Automated Software Testing Maurأ­cioAniche m.f.aniche@

  • Automated Software Testing Maurício Aniche

    m.f.aniche@tudelft.nl @mauricioaniche

    Delft University of Technology, 2019

    mailto:m.f.aniche@tudelft.nl

  • Agile is not only about processes…

    • Technical practices are also very important: • Automated testing • Test-Driven Development • Pair programming • Refactoring • Continuous Integration

    • Extreme Programming (XP) has a strong focus on the technical aspects of agility.

  • Agile is not only about processes…

    • Technical practices are also very important: • Automated testing • Test-Driven Development • Pair programming • Refactoring • Continuous Integration

    • Extreme Programming (XP) has a strong focus on the technical aspects of agility.

  • Yes, I can do it!!

    Given a list of numbers, I want to know the smallest and the largest numbers there!

  • def find(nums): smallest = 9999 largest = -9999 for num in nums:

    if(num < smallest): smallest = num

    elif(num > largest): largest = num

    print(smallest) print(largest)

    find([29, 37, 4, 5])

  • Ø 4 Ø 37

    Yes, it works!

  • find([29, 28, 27, 26])

    Ø 26 Ø -9999

    My bad…. :/ If I pass {29, 28, 27, 26}, the program gets crazy!!

  • def find(nums): smallest = 9999 largest = -9999 for num in nums:

    if(num < smallest): smallest = num

    elif(num > largest): largest = num

    print(smallest) print(largest)

    find([29, 37, 4, 5])

  • def find(nums): smallest = 9999 largest = -9999 for num in nums:

    if(num < smallest): smallest = num

    if(num > largest): largest = num

    return min, max min, max = find([29, 37, 4, 5]) print(min) print(max)

  • Have you made such mistakes before?

    Beller, Moritz, Andy Zaidman, and Andrey Karpov. "The last line effect." Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE Press, 2015. Beller, M., Zaidman, A., Karpov, A., & Zwaan, R. A. (2017). The last line effect explained. Empirical Software Engineering, 22(3), 1508-1536.

  • A (bad) fairy tale

  • What’s the solution for that?

    Test your software!!

  • What’s wrong with manual testing?

    • Too slow • Too expensive • Not easy to reproduce • Susceptible to failures • … boring!

  • Let’s automate some tests!

    • Unit testing framework • Pytest!

    • Triple A (Arrange-Act-Assert)

    •Let’s do it!

  • def test_decreasing_order(): min, max = find([29, 28, 27, 26])

    assert min == 26 assert max == 29

    def test_any_random_numbers(): min, max = find([29, 37, 4, 5])

    assert min == 4 assert max == 37

    https://gist.github.com/mauricioaniche/73366e8e00686e3d44cb48f971599716

    https://gist.github.com/mauricioaniche/73366e8e00686e3d44cb48f971599716

  • Source code

    • Implementation here: • https://gist.github.com/mauricioaniche/9735db28d7091dac0f701dac

    a327313a

    https://gist.github.com/mauricioaniche/9735db28d7091dac0f701daca327313a

  • ”But if you write 100 lines of production code, now you’ll write only 50, as the other 50 are testing. Therefore, you are less productive.”

    – says a bad manager.

  • Not true.

    • You spend a lot of time in manual testing. • Now, you will spend it only once: to write the test.

    • Teams with automated test suites spend less time debugging.

    George, B., Williams, L., An Initial Investigation of TDD in Industry. ACM Symposium on Applied Computing. Melbourne, Florida, USA, 2003. Janzen, D., Software Architecture Improvement through Test-Driven Development. Conference on Object Oriented Programming Systems Languages and Applications, ACM, 2005

  • Why is a failing test a good thing?

    • If it fails here, it didn’t fail in production! J • You have the possibility to fix the bug before going to production. • ”If you have a test that never fails, you have a useless test.” –

    pragmatic developer. • Always see your test failing at least once.

    • How do I test the test?

  • The bug cycle

    • You found a bug! • Write a test to make sure it exists • Fix the bug • The bug will never haunt you again.

  • FIRST!

    • Fast • Isolated • Repeatable • Self-validating • Timely

  • Fast

    • If they are fast, you run them all day long. • If they are not, you stop using them. • Make your tests fast!

    • Unit tests tend to be fast • Mock dependencies (Curious? Google for it! Or ask Steve Freeman!)

  • Isolated

    • A test should not depend on another test. • Test dependencies lead to flaky tests + hard to diagnose failures.

    • Isolate your tests as much as you can.

  • Repeatable

    • Tests that produce different results in different executions are hard to diagnose.

    • If you run your test twice, the results should be the same. • Write small, focused, isolated tests to avoid such issue.

  • Self-validating

    • Your tests should be 100% automated. • If you need a ”small manual test” to arrange or to assert the test, it is not

    automated anymore.

    • We want our tests to be automatically executed by machines without any human intervention.

  • Timely

    • You should write tests all day long. • Do not wait until the end to start writing tests. • Your team should understand the importance of such thing.

    • ”peer pressure” helps.

  • A ATest

    B BTest

    C CTest

    That’s what we call Unit Testing.

  • Some definitions

    • ISQTB: “Searches for defects in, and verifies the functioning of software items (e.g., modules, programs, objects, classes, etc) that are separately testable”.

    • Osherove: “A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behavior of that unit of work. [...] A unit of work is a single logical functional use case in the system that can be invoked by some public interface (in most cases). A unit of work can span a single method, a whole class or multiple classes working together to achieve one single logical purpose that can be verified.”

  • Advantages Disadvantages

    • Very fast • Easy to control • Easy to write

    • Less real • Some bugs can’t be

    reproduced at such level

  • When do we need more reality? What can we do to gain reality?

  • BigFeature Test

    We can do System Testing!

  • Advantages Disadvantages

    • Very realistic • Captures the user

    perspective

    • Slow • Hard to write • Flaky

  • Testing pyramid

    Unit tests

    Integration tests

    System tests

    Manual

    M or

    e re

    al ity

    M or

    e co

    m pl

    ex ity

  • How I (Maurício) do the trade-off

    All business rules should be tested here.

    Avoid at all cost. But do it when needed.

    Complex integrations with external services.

    Main/Risky flow of the app tested.

    Unit tests

    Integration tests

    System tests

    Manual

  • ”Whenever you are tempted to type something into a print statement or a debugger expression, write it as a test instead”.

    --- Martin Fowler

  • License

    • You can use and share any of my material (lecture slides, website). • You always have to give credits to the original author. • You agree not to sell it or make profit in any way with this.

    • Material that I refer has its own license. Please check it out.

  • Images in this presentation

    • Female engineer by GDJ: https://openclipart.org/detail/230143/female-engineer-9

    • Lady bug, by Scout: https://openclipart.org/detail/191074/lady-bug • Business man, by GDJ:

    https://openclipart.org/detail/279728/business-presentation • Rocket explosion, by bf5man:

    https://openclipart.org/detail/203747/rocket-explosion

    https://openclipart.org/detail/230143/female-engineer-9 https://openclipart.org/detail/191074/lady-bug https://openclipart.org/detail/279728/business-presentation https://openclipart.org/detail/203747/rocket-explosion