Test Infrastructure for PKP projects

[jerico]

Hi Alec (and others),

I am opening this thread to publicly document the PHPUnit/Selenium prototype that we already discussed over email.

As we agreed, I am going to implement a prototype for the author's submission process, paying special attention to:
- testing file uploads
- testing email responses

I have already developed a test infrastructure for PHPUnit tests which can be seen in my submissions to the Debian Packaging Project which are fully unit tested (see patch attachments in BugZilla's "Debian Packaging" category, especially #4518). Now Selenium web tests need to be integrated with the existing PHPUnit infrastructure.

I'll be using:
- Selenium IDE (Firefox Plugin) as a development environment (but not to run tests!)
- PHPUnit's selenium test integration
- An email abstraction layer together with a mock implementation to test email responses

On first caveat: The Selenium IDE that can be found on Firefox's add-on site is not compatible with FF 3.5. You have to go to http://seleniumhq.org/download/ and download the IDE there.

Florian

[jerico]

One of the most important points that I mentioned to you already is that we'll have to develop guidelines when to do web tests and (above all) when not.

Compared to UnitTests, maintenance of web tests may be rather costly if you don't do them well. In my experience it is very important that you develop clear and easy to remember guidelines for what should be automated with Selenium and what shouldn't. Imagine you'd have to change your tests whenever you changed a translation string or made some small change in your page design.

IMO what we shouldn't automate:
- texts and translations correct
- images present
- design guidelines compliance
- browser compliance (design)
- anything else that testers and end users will immediately seen anyway
- ...

What we should automate:
- complicated workflows with many branches and conditions
- form fields, especially form field validation
- correct handling of border conditions
- correct display of error messages for error conditions
- browser compliance (workflows, border/error conditions)
- ...

[jerico]

Here come the links to the websites that describe workarounds for file uploads in Selenium. There may be some "recommended solution" from the Selenium project by now, though. I think elevating JS browser permissions would be the right way to go. I'll check that.

http://www.brokenbuild.com/blog/2007/06 ... nd-profit/
http://cakebaker.wordpress.com/2006/03/ ... -selenium/

Here is the link to the PHPUnit manual that describes the Selenium integration I'll use:

http://www.phpunit.de/manual/3.3/en/selenium.html

[jerico]

This thread has been discontinued in favour of http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=4815 and http://pkp.sfu.ca/bugzilla/show_bug.cgi?id=4518.