Before you start: What not to test...
Compared to unit tests, the maintenance of web tests is relatively costly as the GUI usually changes more quickly than back-end services. You should think well about what you should automate and what not. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.
Before developing a new web test decide for yourself whether the total cost of recording, automating and maintaining the test in the future will be less than the cost for manual testing.
Of course writing automated tests is a lot more fun than doing manual testing and this should be considered benefit (or cost) as well.
Some tests are simply not obvious (like some very special error or workflow condition not easily visible from the GUI alone). Such tests also should be automated thereby documenting the hidden feature. They should be recorded by the developer at development time and not by the tester as the tester simply may not know about the hidden functionality.
Usually the general guidelines for web tests are as follows...
What you should not test with Selenium:
- correctness of texts and translations
- presence of images
- design guidelines compliance
- browser compatibility of the design (e.g. CSS)
- anything else that testers and end users will see immediately anyway
What you may automate:
- highly repetitive tests (or parts of tests)
- complicated workflows with many branches and conditions
- testing form fields, especially form field validation
- correct handling of border and error conditions (invalid request parameters, direct get/post, XSS, CSRF, etc.)
- correct display of error messages for error conditions
- browser compatibility of workflows and border/error conditions
In the end the decision remains with the developer or tester. These are not strict rules, just guidelines.
Installing and Configuring your Web Test Environment
We use Selenium for automated web tests. If you like to get an overview over the different Selenium components then please have a look at the Selenium documentation.
Selenium tests can best be developed using a plug-in for the Firefox browser. This plug-in can record all actions you perform on a web page and replay them. It also contains an editor to manually extend or adapt your test cases.
Once the Selenium code (they call it Selenese) has been developed, it can be executed in all browsers.
So first step: get the latest version of Firefox installed.
You can view a tutorial on installing, configuring and using the Selenium IDE to complete PKP Web tests here.
- Download the latest Selenium IDE Firefox Plugin from the Selenium Download Page. You won't find the word "Firefox" there. Just download "Selenium IDE", that's the plug-in. Don't use the plug-in you might find on the Firefox add-on pages as this is often outdated.
- Install the plug-in into FireFox.
- Restart FireFox.
Selenium RC (optional)
Selenium RC is only required if you plan to automate your web tests with PHPUnit or if you want to execute your tests in other browsers than Firefox. It is the "remote control" of Selenium that exposes an API that has bindings to several browsers and programming languages, including PHP. PHPUnit requires Selenium RC to execute tests.
Selenium RC is written in Java. This means you need a current Java Runtime first.
To install Selenium RC, please:
- Download the latest Selenium RC from the Selenium Download Page
- Extract the zip file into any folder (e.g. C:\Program Files\selenium-rc) you want to run Selenium RC from.
Installing Selenium RC as a Windows Service (optional)
If you plan to write automated Selenium tests regularly then it might be a good idea to install Selenium RC as a service. This allows you to easily start and stop Selenium RC.
First download and install the correct Windows Resource Kit for your platform.
Then open a command line window (Start->Execute...->"cmd").
Register the Selenium service on the command line like this:
instsrv.exe SeleniumRC "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"
If you installed the Windows Resource Kit to another location then you have to change the command path. If you don't have the resource kit's Tools directory on the path then cd there to execute the command.
Now coppy the following text to a file called "selenium.reg":
Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SeleniumRC\Parameters] "Application"="java.exe" "AppDirectory"="C:\\Program Files\\selenium-rc\\selenium-server-1.0.1" "AppParameters"="-Xrs -jar selenium-server.jar"
If you installed Selenium RC to another directory or you downloaded another version then you have to adapt the "AppDirectory" key.
Double-click the newly created file "selenium.reg" to add the entries to the registry.
Now open the Windows services configuration program. (You can find it in your "System Configuration" tools under "System Administration".)
Open the new "SeleniumRC" service with a double click.
On the main page you probably want to configure the service to start manually only.
Now go to the log-in tab and configure the service user. If you leave the system account then also check the "allow data exchange between service and desktop" option. This will help in debugging automated tests later.
Close the service configuration.
You should now be able to start the service. If you've allowed data exchange between service and desktop, then a command line window should open showing you the successful start of the Selenium RC service.
Developing Web Tests
I used the following software configuration when I created this tutorial:
- Firefox 3.5.3
- Selenium IDE 1.0.2
- Selenium RC 1.0.1
- Eclipse Galileo with PDT 2.0
- PHPUnit 3.4
- PHP 5.2.11
Create "Selenese" for a test case
- Open the web page you want to test.
- Open the Selenium IDE Sidebar in Firefox ("View -> Sidebar -> Selenium IDE") or if you prefer an independent window then "Tools->Selenium IDE".
- The red "record" button should already be activated.
- Now execute your test actions (click links, enter text into forms, etc.) on the web-site. You should see that the Selenium IDE is recording everything you do.
Tests do not make a lot of sense if they only contain actions. They also should contain assertions that test whether the actions resulted in the expected result.
You can insert assertions at any time while recording a test using Firefox's context menu. The most common assertions are at the bottom of the context menu. Other commands are reachable through the context menu entry "Show all available commands".
I cannot explain all these commands here. Please refer to the introductory chapter about Selenium Commands for more information.
- Once you're done with your test, switch the red "record" button off.
Now you have to decide whether you want to use this test sequence just for manual repetition in Firefox or whether you want to transform it to an automatic web test to be executed in other browsers or during the automated build process.
Save "Selenese" for manual repetition
If you just want to record tests to manually replay them later then you press Ctrl+S in the Selenium Sidebar or you go to "File->Save Test Case".
Save the file and that's it.
Later you can open and replay the test any time you like.
Transform "Senelese" to PHP code
Extracting Selenese from the Selenium IDE
If you want to integrate your web test into our automated test suite or execute your test on other browsers than Firefox then:
- Go to "Options->Format->PHP – Selenium RC" in the Selenium IDE side bar.
- Copy the test code (only the test method) to the clipboard
Create a test case that extends WebTestCase
(WebTestCase has not yet been committed to git so this cannot yet be actually done.)
- write a test class that extends WebTestCase
- copy the test code into the method
- correct the following manually:
- use assertTitleEquals(), assertLocationEquals(), etc. rather than assertEquals(..., $this->getTitle() / getLocation())
- use waitForLocation() when you use redirects
- if you used the waitForLocation command in Selenium IDE then replace the generated for loop with a call to waitForLocation()
- use ...AndWait() rather than waitForPageToLoad()
Now you can execute the test in the same way you execute any other Unit Tests.