Horizon/Testing/UI

During the Icehouse cycle we want to set up a robust, UI-driven integration testing framework for Horizon.

Currently, the preferred approach is to use the Page Object Pattern, and we're planning to implement this during the Icehouse-2 (foundations and infrastructure needed to run the tests, hopefully the first few pages and tests) and Icehouse-3 timeframe (more and more tests!).

Page Object Pattern (Selected Approach)

Description

The Page Object Pattern offers a level of indirection, and segregates the tests so that the UI model does not affect them.

You build up regions that become reusable components (example of a base page). These properties can then be redefined or overriden (e.g. selectors) in the actual pages (subclasses) (example of a page another example).

The page objects are read-only and define the read-only and clickable elements of a page, which work to shield the tests. For instance, from the test perspective, if "Logout" used to be a link but suddenly becomes a an option in a drop-down menu, there are no changes because it still simply calls the "click_on_logout" action method.

This approach has 2 main aspects:

The classes with the actual tests should be as readable as possible
The other parts of the testing framework should be as much about data as possible, so that if the CSS etc. changes you only need to change that one property. If the flow changes, only the action method should need to change.

There is little that is Selenium-specific in the Pages, except for the properties.

There is little coupling between the tests and the pages. Writing the tests becomes like writing out a list of steps (by using the previously mentioned action methods).

One of the key points, particularly important for this kind of UI driven testing is to isolate the tests from what's behind them.

Python

On the Python side, the recommendation is to use pytest as a test runner, which makes it easy to swap things in and out based on the data available, and the mozwebqa-plugin in the tests themselves.

Next steps

Focus on a page/base.py and page/page.py in order to create a good foundation (there are example available - see example projects below - and they are probably abstract enough to be mostly reusable.)

Implement the navigation first, because it will be on every page - implement it as a region. Regions can then be built manually.

Additionally, let's consider writing unit tests for the page objects themselves, in order to validate the selectors (this can help to catch bugs that would happen on only one kind of browser).

Summary and links

Pros: Tests are readable and robust - they should not need to change when the codebase does

Cons: A new (small) framework to learn and maintain

More information:

Mozilla: https://wiki.mozilla.org/QA/Execution/Web_Testing/Docs/Automation/StyleGuide#Page_Objects (examples in Python)
Mozilla Page Object Templates example: https://github.com/mozilla/mozwebqa-test-templates (Python)
Selenium: https://code.google.com/p/selenium/wiki/PageObjects (examples in Java)

Dependencies of interest:

Example projects that use Page Objects:

Other Implementation ideas (Shelved)

Directly using Selenium

Point Selenium to a (probably devstack-based) OpenStack installation. Advantages: Simplest solution Disadvantages: Must know Python, must know how to write robust UI-driven tests using Selenium

Translator middleware

A translator framework that makes it possible to write tests in a high-level language (similar to BDD) and makes the tests more resilient to changes in the software implementation details. See also: JBehave. Advantages: Simplify the writing of test scenarios by experienced QA/QE/Testing teams who may not know Python, by enabling them to write the scenarios in English. Disadvantages: The load is now shifted onto the development team who must spend the time keeping the translator layer up to date, and enabling new keywords/scenarios. (?)