Whether we like it or not, testing takes a huge amount of time. Often a third of each web development project is spent testing, but it's not usually something that gets a lot of attention.
SimpleTest and Selenium testing are now relatively common topics of discussion in the Drupal community, but the actual testing process is less widely discussed. Figuring out what to test for, how to track the results of testing, and techniques for testing Drupal projects are difficult problems to solve.
Whatever testing or project management methodologies are used, improving the testing process can save valuable development hours and increase the chances of finishing projects on schedule. There are lots of challenges particular to testing Drupal projects, and at DrupalCamp New Hampshire in October, I got a chance to share some of this at a session about Improving Your Drupal Testing Process.
Benefits of Testing
Besides the fact that testing a website will find bugs that need to get fixed, testing can have other positive outcomes. Testing is a great opportunity to write technical documentation and documentation for site administrators. Bug reports can lead to documentation of known issues and help unearth unforeseen requirements.
Challenges of Testing Drupal Websites
Testing Content
Most Drupal sites are full of content, and testing a site with real-world content is one of the main tasks we're faced with when testing Drupal projects. Variations in content that can cause issues to appear include:
- Quantity of content
- Length of individual pieces of content
- Interlinking between content
- Different types of media
- Language
- Formatting
If a website includes a content import component, doing this at the beginning of a project is a great way to ensure that real content is tested as the site's functionality is built. For user-generated content, creating a suite of realistic test content that includes a variety of what users might enter is a great investment of time at the beginning of the project.
Testing Configuration
In Drupal, configuration is functionality and the configuration settings that you're delivering need to be tested thoroughly.
Since contributed modules often introduce functionality through the admin UI that is not relevant to the project at hand, documenting which functionality has been tested is important. A good way to clarify this is to use Features as a form of documentation, including certain confiuration settings in 'configuration' features which only developers can update. This can clarify what configuration is intentional and key to the functionality of the website.
Things that are Assumed to Work
When you're writing requirements for a Drupal project, a lot of things get glossed over or left over because it's assumed that Drupal core will handle them, or that they will 'just work'. It's important to remember these during the testing process:
- Core Drupal Features
- Security
- Performance
- Accessibility
- Browser Compatibility
- Integration with APIs
- Import Scripts
- Admin UI
- Admin Workflows
- Mail Servers
Using an Issue Tracker to Improve Testing
My presentation included a series of tips for improving bug reporting by taking advantage of issue tracking features. Over the past few months at Evolving Web, we've been working to improve how we use Redmine to manage projects and track issues, and this plays a big role in the testing process.
Since Redmine is an open source tool, we've written some custom plugins to extend what it does and customize it to fit our needs. You can read more about our Redmine Google Docs Plugin.
Regardless of which issue tracker you use, you can improve your issue tracking process by making sure that testing results in better bug reports:
- Review issues before they're closed or resolved
- Assign issues to the right person
- Link to related issues
- Don't make duplicate issues
- Categorize and prioritize issues
- Integrate specifications with the issue queue
- Use pictures (i.e. screenshots)
- Add lots of links (to the issue, to the config settings for the issue, to related Drupal.org issues etc.)
- Include 'Steps to reproduce', or even a script to reproduce
Another key to making effective use of your issue tracker is making sure that there's an issue for everything. This means that these changes are documented and, hopefully, tested over the course of the development process.
- Configuration changes
- Adding a content type
- Adding a user role
- Installing a module
- Installing a new language
- Importing content
- Writing custom code
Methods of Testing
Different situations call for different methods of testing. Some features, like validation or e-commerce transactions, are critical and very specific. These are well-suited to specific, scripted tests that can be repeated and tested frequently. At the other end of the spectrum, usability and design features are very difficult to test in a scripted fashion. Exploratory testing, in which tests can be adjusted on the fly by the tester, are better suited for these types of features.
Automated Testing
One of the major benefits of scripted tests is that they can be automated. Tests that are automated can be run frequently. Automation can also serve as documentation in the issue queue. Using a tool like Selenium IDE, non-technical testers can record the 'steps to reproduce' when creating a bug report.
What is Selenium IDE?
Selenium IDE is a Firefox add-on that makes it easy to record tests. It records your actions in the browser and then replays them when you run each test. Tests consist of a set of command, which you can edit. You can also 'verify' that an element appears on the page in a particular place. A simple example of a Selenium test for Drupal would be filling in the fields for a particular content type and then verifying that they appear in the correct place on the node page. The Software Testing Club has a great intro to Selenium series if you're looking for help getting started.
Sharing Tests
Automated tests can be hard to maintain, since they need to be updated as the functionality of the site changes. Sharing tests through version control or on a testing server can help reduce efforts in writing automated tests. Selenium tests include a 'base URL', which lets you easily change the base URL of tests so they can be run on different environments.
Managing Test Content
One of the first things you'll notice when running Selenium tests is how quickly you can create a lot of test content. Writing tests that create, verify, and then remove content is a good way to manage this. It will also avoid overlapping namespaces if you're testing items that have to be unique, like adding custom fields to a content type or creating users.
Test Suites
Besides writing individual tests, Selenium allows you to create test suites which group tests together and allows you to run them sequentially. This is really for a set of tests for a particular user role. Rather than including a login step at the beginning of each test, this means you can login once before the tests are run.
Beyond the Basics
Selenium also has a Web Driver which can run tests server-side and allow you to test in other browsers. This is useful if you want to extend what Selenium can do, run your tests automatically, or write tests by hand.
We've come a long way since the early days of tracking issues on a white board, but our quest to turn Redmine into the ideal issue tracker is far from over. We're also still figuring out the best way to integrate unit tests and automated tests into our development workflow. As Drupal projects become more complex, testing becomes more important to each project's success. We hope to share more about the evolution of our testing process in the near future, so stay tuned!