Difference between revisions of "Integration Testing"

From PKP Wiki
Jump to: navigation, search
m (we -> you)
m (Small bug fixes)
Line 1: Line 1:
 
= Before you start: What <i>not</i> to test... =
 
= Before you start: What <i>not</i> to test... =
  
Compared to unit tests, the maintenance of web tests is relatively costly as the GUI usually changes more quickly than back-end services. It is very important that we follow clear guidelines for what should be automated and what shouldn't. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.
+
Compared to unit tests, the maintenance of web tests is relatively costly as the GUI usually changes more quickly than back-end services. You should think well about what you should automate and what not. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.
  
 
Before developing a new web test decide for yourself whether the total cost of recording, automating and maintaining the test in the future will be less than the cost for manual testing.
 
Before developing a new web test decide for yourself whether the total cost of recording, automating and maintaining the test in the future will be less than the cost for manual testing.
Line 7: Line 7:
 
Of course writing automated tests is a lot more fun than doing manual testing and this should be considered benefit (or cost) as well.
 
Of course writing automated tests is a lot more fun than doing manual testing and this should be considered benefit (or cost) as well.
  
And some tests are simply not obvious (like some very special error or workflow condition not easily visible from the GUI alone). Such tests also should be automated thereby documenting the hidden feature. They should be recorded by the developer at development time and not be the tester at test time as the tester simply may not know about the hidden functionality.
+
Some tests are simply not obvious (like some very special error or workflow condition not easily visible from the GUI alone). Such tests also should be automated thereby documenting the hidden feature. They should be recorded by the developer at development time and not be the tester as the tester simply may not know about the hidden functionality.
  
Our general guidelines are as follows...
+
Usually the general guidelines for web tests are as follows...
  
 
What you <b>should not</b> test with Selenium:
 
What you <b>should not</b> test with Selenium:
* correctness texts and translations
+
* correctness of texts and translations
 
* presence of images
 
* presence of images
 
* design guidelines compliance
 
* design guidelines compliance
Line 26: Line 26:
 
* browser compatibility of workflows and border/error conditions
 
* browser compatibility of workflows and border/error conditions
  
But as always: In the end the decision remains with the developer or tester. These are not strict rules, just guidelines.
+
In the end the decision remains with the developer or tester. These are not strict rules, just guidelines.
  
 
= Installing and Configuring your Web Test Environment =
 
= Installing and Configuring your Web Test Environment =
Line 36: Line 36:
 
Selenium tests can best be developed using a plug-in for the Firefox browser. This plug-in can record all actions you perform on a web page and replay them. It also contains an editor to manually extend or adapt your test cases.
 
Selenium tests can best be developed using a plug-in for the Firefox browser. This plug-in can record all actions you perform on a web page and replay them. It also contains an editor to manually extend or adapt your test cases.
  
Once the code Selenium code (they call it Selenese) has been developed, it can be executed in all browsers.
+
Once the Selenium code (they call it Selenese) has been developed, it can be executed in all browsers.
  
This means that you should have the latest version of Firefox installed.
+
So first step: get the latest version of Firefox installed.
  
 
== Selenium IDE ==
 
== Selenium IDE ==
Line 48: Line 48:
 
== Selenium RC (optional) ==
 
== Selenium RC (optional) ==
  
Selenium RC is only required if you plan to automate your web tests with PHPUnit. It is the "remote control" of Selenium that exposes an API that has bindings to several programming languages, including PHP. PHPUnit requires Selenium RC to execute tests.
+
Selenium RC is only required if you plan to automate your web tests with PHPUnit or if you want to execute your tests in other browsers than Firefox. It is the "remote control" of Selenium that exposes an API that has bindings to several browsers and programming languages, including PHP. PHPUnit requires Selenium RC to execute tests.
  
 
Selenium RC is written in Java. This means you need a current Java Runtime first.
 
Selenium RC is written in Java. This means you need a current Java Runtime first.
  
To install Selenium RC please:
+
To install Selenium RC, please:
 
* Download the latest Selenium RC from the [http://seleniumhq.org/download/ Selenium Download Page]
 
* Download the latest Selenium RC from the [http://seleniumhq.org/download/ Selenium Download Page]
 
* Extract the zip file into any folder (e.g. C:\Program Files\selenium-rc) you want to run Selenium RC from.
 
* Extract the zip file into any folder (e.g. C:\Program Files\selenium-rc) you want to run Selenium RC from.
Line 80: Line 80:
 
</pre>
 
</pre>
  
If you have installed Selenium RC to another directory or you downloaded another version then you have to adapt the "AppDirectory" key.
+
If you installed Selenium RC to another directory or you downloaded another version then you have to adapt the "AppDirectory" key.
  
Double-click the file "selenium.reg" to add the entries to the registry.
+
Double-click the newly created file "selenium.reg" to add the entries to the registry.
  
 
Now open the Windows services configuration program. (You can find it in your "System Configuration" tools under "System Administration".)
 
Now open the Windows services configuration program. (You can find it in your "System Configuration" tools under "System Administration".)
  
Open the new "SeleniumRC" service with a double click. Go to the log-in tab and configure the service user. If you leave the system account then also check the "allow data exchange between service and desktop" option. This will help in debugging automated tests later.
+
Open the new "SeleniumRC" service with a double click.
 +
 
 +
On the main page you probably want to configure the service to start manually only.
 +
 
 +
Now go to the log-in tab and configure the service user. If you leave the system account then also check the "allow data exchange between service and desktop" option. This will help in debugging automated tests later.
  
 
Close the service configuration.
 
Close the service configuration.
Line 92: Line 96:
 
You should now be able to start the service. If you've allowed data exchange between service and desktop, then a command line window should open showing you the successful start of the Selenium RC service.
 
You should now be able to start the service. If you've allowed data exchange between service and desktop, then a command line window should open showing you the successful start of the Selenium RC service.
  
= Developing Web Tests (Tutorial) =
+
= Developing Web Tests =
  
 
== Software environment ==
 
== Software environment ==
Line 107: Line 111:
  
 
* Open the web page you want to test.
 
* Open the web page you want to test.
* Open the Selenium IDE Sidebar in Firefox (View -> Sidebar -> Selenium IDE). The red "record" button should already be activated.
+
* Open the Selenium IDE Sidebar in Firefox ("View -> Sidebar -> Selenium IDE") or if you prefer an independent window then "Tools->Selenium IDE".
 +
* The red "record" button should already be activated.
 
* Now execute your test actions (click links, enter text into forms, etc.) on the web-site. You should see that the Selenium IDE is recording everything you do.
 
* Now execute your test actions (click links, enter text into forms, etc.) on the web-site. You should see that the Selenium IDE is recording everything you do.
  
Line 118: Line 123:
 
* Once you're done with your test, switch the red "record" button off.
 
* Once you're done with your test, switch the red "record" button off.
  
Now you have to decide whether you want to use this test sequence just for manual repetition or whether you want to transform it to an automatic web test.
+
Now you have to decide whether you want to use this test sequence just for manual repetition in Firefox or whether you want to transform it to an automatic web test to be executed in other browsers or during the automated build process.
  
 
== Save "Selenese" for manual repetition ==
 
== Save "Selenese" for manual repetition ==
Line 132: Line 137:
 
=== Extracting Selenese from the Selenium IDE ===
 
=== Extracting Selenese from the Selenium IDE ===
  
If you want to integrate your web test into our automated test suite then:
+
If you want to integrate your web test into our automated test suite or execute your test on other browsers than Firefox then:
* Go to "Options->Format->PHP – Selenium RC"
+
* Go to "Options->Format->PHP – Selenium RC" in the Selenium IDE side bar.
 
* Copy the test code (only the test method) to the clipboard
 
* Copy the test code (only the test method) to the clipboard
  
Line 140: Line 145:
 
(PkpWebTest has not yet been committed to the CVS so this cannot yet be actually done.)
 
(PkpWebTest has not yet been committed to the CVS so this cannot yet be actually done.)
  
 +
* write a test class that extends PkpWebTest
 
* copy the test code into the method
 
* copy the test code into the method
 
* correct the following manually:
 
* correct the following manually:
Line 147: Line 153:
 
* use ...AndWait() rather than waitForPageToLoad()
 
* use ...AndWait() rather than waitForPageToLoad()
  
Now you can execute the test in the same way you execute all [[Unit Tests]].
+
Now you can execute the test in the same way you execute any other [[Unit Tests]].

Revision as of 12:31, 28 October 2009

Before you start: What not to test...

Compared to unit tests, the maintenance of web tests is relatively costly as the GUI usually changes more quickly than back-end services. You should think well about what you should automate and what not. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.

Before developing a new web test decide for yourself whether the total cost of recording, automating and maintaining the test in the future will be less than the cost for manual testing.

Of course writing automated tests is a lot more fun than doing manual testing and this should be considered benefit (or cost) as well.

Some tests are simply not obvious (like some very special error or workflow condition not easily visible from the GUI alone). Such tests also should be automated thereby documenting the hidden feature. They should be recorded by the developer at development time and not be the tester as the tester simply may not know about the hidden functionality.

Usually the general guidelines for web tests are as follows...

What you should not test with Selenium:

  • correctness of texts and translations
  • presence of images
  • design guidelines compliance
  • browser compatibility of the design (e.g. CSS)
  • anything else that testers and end users will see immediately anyway

What you may automate:

  • highly repetitive tests (or parts of tests)
  • complicated workflows with many branches and conditions
  • testing form fields, especially form field validation
  • correct handling of border and error conditions (invalid request parameters, direct get/post, XSS, CSRF, etc.)
  • correct display of error messages for error conditions
  • browser compatibility of workflows and border/error conditions

In the end the decision remains with the developer or tester. These are not strict rules, just guidelines.

Installing and Configuring your Web Test Environment

We use Selenium for automated web tests. If you like to get an overview over the different Selenium components then please have a look at the Selenium documentation.

Firefox

Selenium tests can best be developed using a plug-in for the Firefox browser. This plug-in can record all actions you perform on a web page and replay them. It also contains an editor to manually extend or adapt your test cases.

Once the Selenium code (they call it Selenese) has been developed, it can be executed in all browsers.

So first step: get the latest version of Firefox installed.

Selenium IDE

  • Download the latest Selenium IDE Firefox Plugin from the Selenium Download Page. You won't find the word "Firefox" there. Just download "Selenium IDE", that's the plug-in. Don't use the plug-in you might find on the Firefox add-on pages as this is often outdated.
  • Install the plug-in into FireFox.
  • Restart FireFox.

Selenium RC (optional)

Selenium RC is only required if you plan to automate your web tests with PHPUnit or if you want to execute your tests in other browsers than Firefox. It is the "remote control" of Selenium that exposes an API that has bindings to several browsers and programming languages, including PHP. PHPUnit requires Selenium RC to execute tests.

Selenium RC is written in Java. This means you need a current Java Runtime first.

To install Selenium RC, please:

  • Download the latest Selenium RC from the Selenium Download Page
  • Extract the zip file into any folder (e.g. C:\Program Files\selenium-rc) you want to run Selenium RC from.

Installing Selenium RC as a Windows Service (optional)

If you plan to write automated Selenium tests regularly then it might be a good idea to install Selenium RC as a service. This allows you to easily start and stop Selenium RC.

First download and install the correct Windows Resource Kit for your platform.

Then open a command line window (Start->Execute...->"cmd").

Register the Selenium service on the command line like this:

instsrv.exe SeleniumRC "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"

If you installed the Windows Resource Kit to another location then you have to change the command path. If you don't have the resource kit's Tools directory on the path then cd there to execute the command.

Now coppy the following text to a file called "selenium.reg":

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SeleniumRC\Parameters]
"Application"="java.exe"
"AppDirectory"="C:\\Program Files\\selenium-rc\\selenium-server-1.0.1"
"AppParameters"="-Xrs -jar selenium-server.jar"

If you installed Selenium RC to another directory or you downloaded another version then you have to adapt the "AppDirectory" key.

Double-click the newly created file "selenium.reg" to add the entries to the registry.

Now open the Windows services configuration program. (You can find it in your "System Configuration" tools under "System Administration".)

Open the new "SeleniumRC" service with a double click.

On the main page you probably want to configure the service to start manually only.

Now go to the log-in tab and configure the service user. If you leave the system account then also check the "allow data exchange between service and desktop" option. This will help in debugging automated tests later.

Close the service configuration.

You should now be able to start the service. If you've allowed data exchange between service and desktop, then a command line window should open showing you the successful start of the Selenium RC service.

Developing Web Tests

Software environment

I used the following software configuration when I created this tutorial:

  • Firefox 3.5.3
  • Selenium IDE 1.0.2
  • Selenium RC 1.0.1
  • Eclipse Galileo with PDT 2.0
  • PHPUnit 3.4
  • PHP 5.2.11

Create "Selenese" for a test case

  • Open the web page you want to test.
  • Open the Selenium IDE Sidebar in Firefox ("View -> Sidebar -> Selenium IDE") or if you prefer an independent window then "Tools->Selenium IDE".
  • The red "record" button should already be activated.
  • Now execute your test actions (click links, enter text into forms, etc.) on the web-site. You should see that the Selenium IDE is recording everything you do.

Tests do not make a lot of sense if they only contain actions. They also should contain assertions that test whether the actions resulted in the expected result.

You can insert assertions at any time while recording a test using Firefox's context menu. The most common assertions are at the bottom of the context menu. Other commands are reachable through the context menu entry "Show all available commands".

I cannot explain all these commands here. Please refer to the introductory chapter about Selenium Commands for more information.

  • Once you're done with your test, switch the red "record" button off.

Now you have to decide whether you want to use this test sequence just for manual repetition in Firefox or whether you want to transform it to an automatic web test to be executed in other browsers or during the automated build process.

Save "Selenese" for manual repetition

If you just want to record tests to manually replay them later then you press Ctrl+S in the Selenium Sidebar or you go to "File->Save Test Case".

Save the file and that's it.

Later you can open and replay the test any time you like.

Transform "Senelese" to PHP code

Extracting Selenese from the Selenium IDE

If you want to integrate your web test into our automated test suite or execute your test on other browsers than Firefox then:

  • Go to "Options->Format->PHP – Selenium RC" in the Selenium IDE side bar.
  • Copy the test code (only the test method) to the clipboard

Create a test case that extends PkpWebTest

(PkpWebTest has not yet been committed to the CVS so this cannot yet be actually done.)

  • write a test class that extends PkpWebTest
  • copy the test code into the method
  • correct the following manually:
  • use assertTitleEquals(), assertLocationEquals(), etc. rather than assertEquals(..., $this->getTitle() / getLocation())
  • use waitForLocation() when you use redirects
  • if you used the waitForLocation command in Selenium IDE then replace the generated for loop with a call to waitForLocation()
  • use ...AndWait() rather than waitForPageToLoad()

Now you can execute the test in the same way you execute any other Unit Tests.