CI Fitnesse Selenium

  • View

  • Download

Embed Size (px)


Fitnesse & Selenium

Text of CI Fitnesse Selenium

  • WITH


    Continuous Integration

    By Brian

  • Intro

    Who am I?


    Continuous Integration

    The Tools

    Selenium Overview

    Fitnesse Overview

    Data Dependence

    My Solution


  • What is CI?

    From Wikipedia: Continuous Integration (CI) implements a continuous processes of applying quality

    control - small pieces of effort, applied frequently. Continuous integration aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.

    In English: automatic code integration and validation!

    What do we need? A Build Server to control everything and store reporting A development/deployment process that supports being automated. Scripts to do the work One or more test frameworks to test the code (nUnit, Fitnesse, Selenium) A minimal set of tests that cover a broad scope of functionality

    What do we get? Automatic testing and validation for each code change Immediate and automatic isolation of the cause Up-to-date, centralized reporting You know who broke it, and who needs to fix it. You never will test a broken environment again.

  • The Tools

    Build Server (Hudson) Runs our Tests

    Unit Tests

    Tests the individual functions

    Fitnesse Tests:

    Database (DB Fit)

    API calls

    Web Services

    Selenium Tests:

    UI Issues

    Functionality problems

    Javascript Errors

  • Selenium IDE

    Firefox only plug-in, no other browsers.

    Step based testing, no complex functionality

    Record and Playback

    Fast test prototyping


    Can be run from Hudson using a plug-in.

  • Selenium Core

    Stored on your websites web server

    Executable via a url

    Supports any browser

    Tests stored in a central repository

    Has problems supporting https or changing domains

  • Selenium RC

    Client/server java application needs to be installed

    Supports any Browser/OS combination

    Test can be run locally or remotely

    Controlled and executed through xUnit

    Test written in your favorite programming language/IDE (C#, java, python, ruby)

    Access to full features of that language and any extra libraries


    public void testGoogle()




    selenium.Type("q", "selenium");



    Assert.IsTrue(selenium.IsTextPresent("Selenium web application testing system"));


  • Selenium Grid

    Distributed parallel testing

    More setup, but faster execution

    Run on multiple machines simultaneously

    Generates more load

  • Pros Cons

    Injects itself as javascript into the page object Any browser that supports javascript supports

    selenium The ONLY tool that can work with any


    Extremely fast execution time Direct access to page objects

    Doesnt hijack your computer You can do other things while its running

    Full access to outside libraries If you need something else, you can code it.

    Pick your language Record in IDE and convert to RC.

    Easier for non-coders Fast test prototyping Troubleshooting

    Open source tools can be customized

    Injects itself as javascript into the page object Non Firefox browsers dont always work

    perfectly It cant see anything outside of the page

    Cant click on the menu, navigation buttons, or any extra windows

    Predefined routines reproduce this functionality

    XSS Limitations Changing domains/https

    It fakes a lot It doesnt use the mouse/keyboard

    Doesnt really click on anything Accesses each item directly and calls a function Can bypass front end functionality such as


    Can access hidden items even if you dont want it to

    Some functionality cant be automated But can be gotten around

    Open source tools require a lot of coding

    How Selenium Works

  • Fitnesse Overview

    What is Fitnesse? Fitnesse = FIT + Wiki

    FIT : Framework for Integrated Testing - The engine that translates tables into actions

    Wiki: The GUI that allows the user to edit and run tests

    A website that allows us to create, edit, and run tests.

    The tests must be written as a table

    Each row is a test

  • Fitnesse continued

    Typically used to test an application or database directly, without accessing the GUI.

    Calls a Web service, API, DB Procedure

    Test the Business Logic directly.

    We dont have to try to trick the ui into an incorrect state

    Typically more stable than GUI testing as less functionality changes over time

    Requires the creation of a Fixture which knows how to translate the test into an action.

    The fixtures are many times written by the application developers

    Some standard fixtures exist, such as the RESTFixture for testing REST based services

    Once written, the fixtures can be largely ignored

    Different fixtures allow different functionality.

  • Fitnesse continued

    Typical data driven testing - same test multiple times for all data types

    Boundary conditions

    Error code checking

    Business Logic

    Acceptance Testing-

    Wiki style input allows non-coders to develop and run tests.

    Business and Product personnel can (theoretically) write acceptance tests

    However it requires a knowledge of the AUT

    GUI testing is possible with appropriate fixture

    Cant test for javascript errors without a GUI test

    However selenium is more robust

  • Fitnesse

  • Fitnesse pros and cons

    Tests are editable and executable from a web page. Uses Xpath. Can generate fake bugs (stuff looks broken because of a Fitnesse

    problem) Test maintenance can be time consuming Everything is a text document Same tests many different places

    Global variables allow you to change url, user, etc very easily Many tests can require unique information (id, name) that can

    change in different environments. Run-time variables allow you to create more dynamic tests. Knowing what data should be returned can be troublesome Can generate exact error-causing conditions Useless if your application sucks

  • Data Dependence

    The biggest problem with automation is changing data Time changes, items expire, dates change. Need to test in different environments going to out to production. Many items are referenced by an ID that is unique, or increment. Need to know what are the expected results of your test. If something you assume is incorrect, your tests will fail.

    User not there Items expired Dates changed GUI item changed or vanished

    In unit testing, you can mock data to test with. When testing a real system, your data can fluctuate

    DB Refresh Virtualized environments User activity can change your data set Date oriented things expire Ids increment Using external or live data

  • Data Options

    Set our data manually each run.

    DB Copy/refresh to return to a known state.

    Start from and return to a known state

    CRUD only your data

    Get current data

    Connect to the db to grab data.

    Run UI automation to get data.

    Generate data

    Use automation to generate a data set at run time

  • My Solution

    Data Generation Suite- Can point to any client/environment. Checks to see if my data is there, if it is it deletes it. Then creates all data fresh, and sets state.

    A Smoke Test to be run by the Hudson server. Checks all major functionality. Checks each of the functional areas Doesnt validate what data it gets. (it fluctuates too much)

    A Regression Test that tests full functionality. Data Generation Suite needs to be run first Tests full functionality and validates data is correct.

    Fitnesse tests that check the services Use the data selenium created. Verify boundary conditions Verify response code and format. Verify errors

  • The end result

    All builds automatically are smoke tested for viability. Takes ~15 minutes to regression test a build in each

    environment Can change users, clients, environments, configuration. All tests dont always pass in all environments, have to

    check test results. Can use Test Driven Development methodology Create tests first Code is good when they pass

    Relying heavily on automation Fast turnaround. High confidence level for code changes, new services

  • In progress

    Work on new framework

    Migrate to Selenium2 (mobile browser support)

    Fully implement PageObjects

    Use a VS config file

    Support multiple browsers