Integration Testing

From PKP Wiki
Revision as of 13:15, 28 October 2009 by (Talk | contribs) (we -> you)

Jump to: navigation, search

Before you start: What not to test...

Compared to unit tests, the maintenance of web tests is relatively costly as the GUI usually changes more quickly than back-end services. It is very important that we follow clear guidelines for what should be automated and what shouldn't. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.

Before developing a new web test decide for yourself whether the total cost of recording, automating and maintaining the test in the future will be less than the cost for manual testing.

Of course writing automated tests is a lot more fun than doing manual testing and this should be considered benefit (or cost) as well.

And some tests are simply not obvious (like some very special error or workflow condition not easily visible from the GUI alone). Such tests also should be automated thereby documenting the hidden feature. They should be recorded by the developer at development time and not be the tester at test time as the tester simply may not know about the hidden functionality.

Our general guidelines are as follows...

What you should not test with Selenium:

  • correctness texts and translations
  • presence of images
  • design guidelines compliance
  • browser compatibility of the design (e.g. CSS)
  • anything else that testers and end users will see immediately anyway

What you may automate:

  • highly repetitive tests (or parts of tests)
  • complicated workflows with many branches and conditions
  • testing form fields, especially form field validation
  • correct handling of border and error conditions (invalid request parameters, direct get/post, XSS, CSRF, etc.)
  • correct display of error messages for error conditions
  • browser compatibility of workflows and border/error conditions

But as always: In the end the decision remains with the developer or tester. These are not strict rules, just guidelines.

Installing and Configuring your Web Test Environment

We use Selenium for automated web tests. If you like to get an overview over the different Selenium components then please have a look at the Selenium documentation.


Selenium tests can best be developed using a plug-in for the Firefox browser. This plug-in can record all actions you perform on a web page and replay them. It also contains an editor to manually extend or adapt your test cases.

Once the code Selenium code (they call it Selenese) has been developed, it can be executed in all browsers.

This means that you should have the latest version of Firefox installed.

Selenium IDE

  • Download the latest Selenium IDE Firefox Plugin from the Selenium Download Page. You won't find the word "Firefox" there. Just download "Selenium IDE", that's the plug-in. Don't use the plug-in you might find on the Firefox add-on pages as this is often outdated.
  • Install the plug-in into FireFox.
  • Restart FireFox.

Selenium RC (optional)

Selenium RC is only required if you plan to automate your web tests with PHPUnit. It is the "remote control" of Selenium that exposes an API that has bindings to several programming languages, including PHP. PHPUnit requires Selenium RC to execute tests.

Selenium RC is written in Java. This means you need a current Java Runtime first.

To install Selenium RC please:

  • Download the latest Selenium RC from the Selenium Download Page
  • Extract the zip file into any folder (e.g. C:\Program Files\selenium-rc) you want to run Selenium RC from.

Installing Selenium RC as a Windows Service (optional)

If you plan to write automated Selenium tests regularly then it might be a good idea to install Selenium RC as a service. This allows you to easily start and stop Selenium RC.

First download and install the correct Windows Resource Kit for your platform.

Then open a command line window (Start->Execute...->"cmd").

Register the Selenium service on the command line like this:

instsrv.exe SeleniumRC "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"

If you installed the Windows Resource Kit to another location then you have to change the command path. If you don't have the resource kit's Tools directory on the path then cd there to execute the command.

Now coppy the following text to a file called "selenium.reg":

Windows Registry Editor Version 5.00
"AppDirectory"="C:\\Program Files\\selenium-rc\\selenium-server-1.0.1"
"AppParameters"="-Xrs -jar selenium-server.jar"

If you have installed Selenium RC to another directory or you downloaded another version then you have to adapt the "AppDirectory" key.

Double-click the file "selenium.reg" to add the entries to the registry.

Now open the Windows services configuration program. (You can find it in your "System Configuration" tools under "System Administration".)

Open the new "SeleniumRC" service with a double click. Go to the log-in tab and configure the service user. If you leave the system account then also check the "allow data exchange between service and desktop" option. This will help in debugging automated tests later.

Close the service configuration.

You should now be able to start the service. If you've allowed data exchange between service and desktop, then a command line window should open showing you the successful start of the Selenium RC service.

Developing Web Tests (Tutorial)

Software environment

I used the following software configuration when I created this tutorial:

  • Firefox 3.5.3
  • Selenium IDE 1.0.2
  • Selenium RC 1.0.1
  • Eclipse Galileo with PDT 2.0
  • PHPUnit 3.4
  • PHP 5.2.11

Create "Selenese" for a test case

  • Open the web page you want to test.
  • Open the Selenium IDE Sidebar in Firefox (View -> Sidebar -> Selenium IDE). The red "record" button should already be activated.
  • Now execute your test actions (click links, enter text into forms, etc.) on the web-site. You should see that the Selenium IDE is recording everything you do.

Tests do not make a lot of sense if they only contain actions. They also should contain assertions that test whether the actions resulted in the expected result.

You can insert assertions at any time while recording a test using Firefox's context menu. The most common assertions are at the bottom of the context menu. Other commands are reachable through the context menu entry "Show all available commands".

I cannot explain all these commands here. Please refer to the introductory chapter about Selenium Commands for more information.

  • Once you're done with your test, switch the red "record" button off.

Now you have to decide whether you want to use this test sequence just for manual repetition or whether you want to transform it to an automatic web test.

Save "Selenese" for manual repetition

If you just want to record tests to manually replay them later then you press Ctrl+S in the Selenium Sidebar or you go to "File->Save Test Case".

Save the file and that's it.

Later you can open and replay the test any time you like.

Transform "Senelese" to PHP code

Extracting Selenese from the Selenium IDE

If you want to integrate your web test into our automated test suite then:

  • Go to "Options->Format->PHP – Selenium RC"
  • Copy the test code (only the test method) to the clipboard

Create a test case that extends PkpWebTest

(PkpWebTest has not yet been committed to the CVS so this cannot yet be actually done.)

  • copy the test code into the method
  • correct the following manually:
  • use assertTitleEquals(), assertLocationEquals(), etc. rather than assertEquals(..., $this->getTitle() / getLocation())
  • use waitForLocation() when you use redirects
  • if you used the waitForLocation command in Selenium IDE then replace the generated for loop with a call to waitForLocation()
  • use ...AndWait() rather than waitForPageToLoad()

Now you can execute the test in the same way you execute all Unit Tests.