Automated testing 101

Automated Testing 101

By Tabby Chapman

@tabbymarie

http://www.tabbychapman.com

1

http://www.tabbychapman.com/

About Tabby Chapman@tabbymarie

https://github.com/tabbymarie

2

About Me

• Work for the University of California, Irvine

• Computer Scientist

• Developer Cheerleader

• Education Junkie

3

About Me

• WordPress Skillz– Reaching my 10th anniversary developing

with WordPress – Started with 2.0.5, building a custom theme.– Ran my own company for four years– Joined Zeek for almost two years– Currently build highly specialized custom

themes and plugins for UCI.

4

About This Presentation

• Geared towards intermediate QA testers and Developers and also for stakeholders/clients/customers

• Assumptions:– You know what WordPress is.– You know what PHP is.– You have written PHP code several times.

Successfully. Or.– You test all the things.

5

About This Presentation

• We’ll go over– The Developer Workflow(s)– Different Types of Tests– Automated Testing– Demo– Last Words

6

How it Currently Works

7

• Generic Software Development Lifecycle– In the dinosaur days of computers, the

majority of its users were very technical people.

– As computers became more accessible (and smaller), more and more users began to use it

– Developers needed to create a process to think outside of their own processes to accommodate non-technical users


8

• Generic Software Development Lifecycle (cont)– Many various methods were created to

attempt to meet this need– One of these is the Software Development

Lifecycle (which was a similar process used in other disciplines)

– It is a top down approach: generic to specific– Specification, Design, Coding


9

• Generic Software Development Lifecycle (cont)– Waterfall method is created in the ‘70s. – Structured analysis, Structured design,

Structured programming (sounds a lot like specification, design, and coding)

– Most developers tend to use Define, Design, Develop, Deploy, Maintain.

Generic Modern Development

10

Image credit: https://www.owasp.org/images/8/84/SDLC.jpg

Freelance Development Cycle

11

Image credit: http://mohamadsubhan.files.wordpress.com/2009/06/linear.jpg


12

• Full Software Development Lifecycle– Many developers on teams tend to use some

form of the full development lifecycle, as there tends to need to be some form of accountability for teams.

– This includes: feasibility study, requirements definition, software design, implementation and unit testing, integration and system testing, operation and maintenance.

Team Development Cycle

13

Image credit: http://www.softwaretestinghelp.com/unit-testing/

Types of Tests

• Black box• White box• Unit • Integration• Functional • System• End-to-end

• Sanity• Regression• Acceptance• Load• Stress• Performance• Usability

14

Source: http://www.softwaretestinghelp.com/types-of-software-testing/

Testing Players

15

• Quality Assurance (QA) Engineers

• Developers

• Product Owners

• Clients

• End-Users

• Other Stakeholders

Black Box Testing?

16

Black Box Testing

17

• Tests against requirements and functionality. Is not concerned with the internal workings.

• “If I put in 2+4, I expect 6. Does that work?”

White Box Testing?

18

White Box

19

• Tests against requirements and functionality with knowledge of the inner workings.

• “I know that I this system has these conditionals, so I will test all possible outcomes.”

Unit Testing?

20

Unit Testing

21

• Tests specific methods or functions, created by a programmer

• “Does function addTwoIntegers($num1,$num2) work as expected?”

Integration Testing?

22

Integration Testing

23

• Testing of whole application after new work has been integrated as they interact with the client/server

• “Does this site’s individual modules still work as expected when we put it all together?”

Functional Testing?

24

Functional Testing

25

• Black Box testing for specific functional requirements.

• “If I tell it to add five text inputs and send an email, does it add five inputs and send an email as expected?”

Systems Testing

26

• Black Box Testing based on overall requirements.

• “Testing All the Things”. “Does this system meet all of the requirements as expected?”

End-to-end Testing

27

• Similar to Systems, but also tests interaction between various elements, protocols, layers and servers.

• “Testing All the Things and Moar”. “Does this system meet all of the requirements as expected and communicate as expected to its various parts?”

Sanity Testing

28

• A pre-test to make sure the new development is done enough to fully test.

• “Does it at least work a little bit?”

Regression Testing

29

• Testing the whole application after something has been modified to see if something was broken as a result of the modification.

• “Does it STILL work as expected?”

Acceptance Testing

30

• User or Customer does a round of manual tests to make sure that it meets their requirements/needs

• “Does the client think it works as expected?”

Stress Testing

31

• Testing Performance Requirements

• “Can more than one interaction happen at the same time? How many more than one?”

Usability Testing

32

• Testing for user-friendliness. This can be done BEFORE and AFTER development phase.

• “Yeah, it works, but is it easy to use?”

Which testing is important for developers?

33

Which testing is important for developers?



34


For the Developers

35

• Which testing is important for developers?– All of them (know thy tester and thy tester’s

methods)– But especially

• Unit• Functional• Systems• Integration

Which testing is important for testers?

36

Which testing is important for testers?



37


For the Testers

38

• Which testing is important for testers?– All of them (uphold they developer to unit

tests)– But especially

• Black Box• White Box• Integration• Systems• Sanity

Which testing is important for Clients/Stakeholders?

39

Which testing is important for Clients/Stakeholders?



40


For the Clients/Stakeholders

41

• Which testing is important for clients/stakeholders?

• Black Box• Usability• Regression• Acceptance

For the Clients/Stakeholders

42

• Which testing is important for clients/stakeholders?

• Black Box• Usability• Regression• Acceptance

What does Automation of Testing mean?

43

What does Automation of Testing mean?

44

• The traditional/easiest way to test what you develop is to make a change, and then load the app to see if that change was effective. Very Manual

• Automated Testing means to write code to test code and use testing software to test the front-end. Don’t rely on the old manual check.

How does this apply to WordPress?

45

• WordPress utilizes a ‘test suite’ “which have multiple test cases and a library and command line tool that runs the test suite or suites” according to WordPress’ automated testing core handbook page

• For this, they use phpUnit. https://phpunit.de

How does this apply to WordPress?

46

• PHPUnit has a series of commands that you can use to run all tests, specific tests, or specific types of tests.

Developer Workflow

47

• When you are ready to write your plugin:– Think of the outcomes you want for your

plugin– Think of the individual methods you want to

write. Should have one output.– Write a test that checks that the output is what

you expect.– Write your method.– Lather. Rinse. Repeat.

Developer Workflow

48

• Testing Tools

• phpUnit (official wordpress test suite)

• WordPress Developer Branch– https://make.wordpress.org/core/handbook/au

tomated-testing/

• Terminal or Editor with Testing Integrated

https://make.wordpress.org/core/handbook/automated-testing/

https://make.wordpress.org/core/handbook/automated-testing/

Developer Workflow

49

Editors (not limited to…)– TextMate - https://macromates.com/download

• (https://github.com/meritt/phpunit-tmbundle)

– Sublime – http://www.sublimetext.com/• https://packagecontrol.io/packages/PHPUnit

– phpStorm - https://www.jetbrains.com/phpstorm/

• Supports all OS’• Phpunit built in

https://packagecontrol.io/packages/PHPUnit

https://www.jetbrains.com/phpstorm/

Developer Workflow

50

• Put your tests IN your plugin

• /plugins/[plugindir]/tests/test-[pluginslug].php

• Add a phpunit.xml file:

• /plugins/[plugindir]/tests/phpunit.xml

Developer Workflowphpunit.xml

51

<?xml version=“1.0” encoding=“UTF-8” ?>

<phpunit bootstrap=“vendor/autoload.php” >

<testsuites>

<testsuite>

<directory prefix="test-” suffix=".php">

./tests/

</directory>

</testsuite>

</testsuites>

</phpunit>


52



<testsuites>

<testsuite>


./tests/

</directory>

</testsuite>

</testsuites>

</phpunit>


53



<testsuites>

<testsuite>


./tests/

</directory>

</testsuite>

</testsuites>

</phpunit>


54



<testsuites>

<testsuite>


./tests/

</directory>

</testsuite>

</testsuites>

</phpunit>

Tester Workflow

55

• When you are ready to test – Be familiar with the general requirements for

the site as a whole– Read the requirements of the change request– Focus on specifically what is supposed to

have changed.– Write a test while waiting for the change.– Run the test once you get the change from

the developer

Tester Workflow

56

• Selenium IDE, Firefox Plugin– http://www.seleniumhq.org/download/

• Browser Specific– Chrome

https://sites.google.com/a/chromium.org/chromedriver/

– Safari http://selenium-release.storage.googleapis.com/2.46/SafariDriver.safariextz

http://www.seleniumhq.org/download/



http://selenium-release.storage.googleapis.com/2.46/SafariDriver.safariextz

http://selenium-release.storage.googleapis.com/2.46/SafariDriver.safariextz

Tester Workflow

57

• Selenium– Installationhttp

://geekpad.ca/blog/post/automating-browser-testing-with-phpunit-and-selenium• First you must download the Selenium server and

the Internet Explorer Server from http://docs.seleniumhq.org/download/

• Then you must download the Chrome server from http://chromedriver.storage.googleapis.com/index.html

http://geekpad.ca/blog/post/automating-browser-testing-with-phpunit-and-selenium



Tester Workflow

58

• Selenium– Installation (continued)

• You do not need to download a Firefox driver since it’s builtin the main Selenium file.

• Once all files have been downloaded you must add your selenium folder to your existing path see http://www.java.com/en/download/help/path.xml

• Then you can launch the server using java. With something that looks like this– Mac - java -jar selenium-server-standalone-[version].jar

Tester Workflow

59

Why is Selenium cool?

•Records your actions

•Let’s you test expected outcomes

•Let’s you test them in a series

Tester Workflow

60

Example

•The developer was given a task to make sure that all instances of the word “Orange County” is replaced with the word “OC” on the website, for “legal reasons”.

•Demo

Jenkins

61

Jenkins

62

• Jenkins is an award-winning application that monitors executions of repeated jobs, such as building a software project or jobs run by cron. Among those things, current Jenkins focuses on the following two jobs:– Building/testing software projects continuously– Monitoring executions of externally-run jobs

https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins

Jenkins

63

Jenkins

64

Questions?

65

Resources

• http://www.dcs.bbk.ac.uk/~sven/cdm06/cdm3.pdf

• https://unit-tests.trac.wordpress.org/

• http://www.slideshare.net/ptahdunbar/automated-testing-in-wordpress-really

• http://johnregan3.co/2013/08/13/why-you-need-to-get-into-wordpress-unit-testing/

• http://blog.aizatto.com/2010/05/15/beginning-wordpress-automated-testing/

66

http://www.dcs.bbk.ac.uk/~sven/cdm06/cdm3.pdf

http://www.dcs.bbk.ac.uk/~sven/cdm06/cdm3.pdf

https://unit-tests.trac.wordpress.org/

http://www.slideshare.net/ptahdunbar/automated-testing-in-wordpress-really

http://www.slideshare.net/ptahdunbar/automated-testing-in-wordpress-really

http://johnregan3.co/2013/08/13/why-you-need-to-get-into-wordpress-unit-testing/

http://johnregan3.co/2013/08/13/why-you-need-to-get-into-wordpress-unit-testing/

Resources

• https://www.owasp.org/index.php/Testing_Guide_Introduction

• https://make.wordpress.org/core/handbook/automated-testing/

• http://stackoverflow.com/questions/22414134/different-types-of-testing-eg-unit-functional-integration-etc-document

67

Resources

• http://www.seleniumhq.org/• http://geekpad.ca/blog/post/automating-browser-

testing-with-phpunit-and-selenium

• https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins

• https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins

68