During the Icehouse cycle we set up the foundations for a robust, UI-driven integration testing framework for Horizon. The tests follow the Page Object Pattern.
Related blueprint: https://blueprints.launchpad.net/horizon/+spec/selenium-integration-testing
- 1 Integration tests in Horizon
- 2 Page Object Pattern (Selected Approach)
- 3 Test areas breakdown
- 4 Other Implementation ideas (Dismissed approaches)
Integration tests in Horizon
The integration tests currently live in the Horizon repository, see https://github.com/openstack/horizon/tree/master/openstack_dashboard/test/integration_tests which also contains instructions on how to run the tests. In the future, the tests themselves may move to the Tempest repository to live together with the other OpenStack integration tests (this will be discussed once we have a larger test suite that we know is stable).
Writing a test
If you're interested in contributing to the tests, thank you!
- Please add your name to the Test Area Breakdown section to avoid people accidentally duplicating work.
- It would be nice if you could also include: "Partially implements blueprint: selenium-integration-testing" in the commit message so that people interested in the integration tests get a notification about your patch (anyone can subscribe to the blueprint).
- In general if you're not sure how to address something, have a look at how Tempest does things. We want to keep consistent with the way integration tests are written in the wider project.
Reviewing the tests
- Gerrit query showing the currently open reviews, that touch the integration tests files
Page Object Pattern (Selected Approach)
The Page Object Pattern offers a level of indirection, and segregates the tests so that the UI model does not affect them.
You build up regions that become reusable components (example of a base page). These properties can then be redefined or overriden (e.g. selectors) in the actual pages (subclasses) (example of a page another example).
The page objects are read-only and define the read-only and clickable elements of a page, which work to shield the tests. For instance, from the test perspective, if "Logout" used to be a link but suddenly becomes a an option in a drop-down menu, there are no changes because it still simply calls the "click_on_logout" action method.
This approach has 2 main aspects:
- The classes with the actual tests should be as readable as possible
- The other parts of the testing framework should be as much about data as possible, so that if the CSS etc. changes you only need to change that one property. If the flow changes, only the action method should need to change.
There is little that is Selenium-specific in the Pages, except for the properties.
There is little coupling between the tests and the pages. Writing the tests becomes like writing out a list of steps (by using the previously mentioned action methods).
One of the key points, particularly important for this kind of UI driven testing is to isolate the tests from what's behind them.
Another interesting aspect to explore: pytest improves on the concept of fixtures, which can be used to create utility functions that can easily be injected into the tests (e.g. maximizing a window). More information on fixtures from the pytest website.
Focus on a page/base.py and page/page.py in order to create a good foundation (there are example available - see example projects below - and they are probably abstract enough to be mostly reusable.)
Implement the navigation first, because it will be on every page - implement it as a region. Regions can then be built manually.
Additionally, let's consider writing unit tests for the page objects themselves, in order to validate the selectors (this can help to catch bugs that would happen on only one kind of browser).
Pros: Tests are readable and robust - they should not need to change when the codebase does
Cons: A new (small) framework to learn and maintain
- Mozilla: https://wiki.mozilla.org/QA/Execution/Web_Testing/Docs/Automation/StyleGuide#Page_Objects (examples in Python)
- Mozilla Page Object Templates example: https://github.com/mozilla/mozwebqa-test-templates (Python)
- Selenium: https://code.google.com/p/selenium/wiki/PageObjects (examples in Java)
Note: the Mozilla guidelines in the first link sound very sensible and I think we should stick to them as well.
Dependencies of interest:
Example projects that use Page Objects:
Test areas breakdown
Multiple people have come forward to help out with writing the tests (which is great, thank you!). To avoid accidentally duplicating work and wasting effort, it makes sense to divide Horizon into different areas. As a starting point, the following list does so by panel: please add your name besides the area you want to take on (feel free to add missing ones or break it down further).
- Logging in - Daniel Korn (dkorn)
As a regular user
- Overview page and usage form -
- Instances (may be broken down further if needed, there is a lot in there)
- Instance Overview - santib
- Luanch Instance - Mohan Seri
- Allocate IP to Project - Ravikumar Venkatesan
- Release Floating IPs - Ravikumar Venkatesan
- Associate Floating IP - Ravikumar Venkatesan
- Edit Instance - Mohan Seri
- Soft Reboot Instances
- Terminate Instances - Mohan Seri
- Create Sanpshot
- Disassociate Floating IP
- Edit Security Groups
- View log
- Pause Instance
- Suspend Instance
- Resize Instance
- Soft Reboot
- Hard Reboot
- Shut Off Instance
- Terminate - Mohan Seri
- Search for an instance in the instances main page.
- Volumes - Mohan Seri
- Create Volume
- Edit Volume
- Extend Volume
- Edit Attachments
- Create Snapshot
- Delete Volume
- Delete Volumes
- Search Volumes
- Images - Daniel Korn (dkorn)
- Access and Security
- Keypair - Ravikumar Venkatesan
- Security Groups - Ravikumar Venkatesan
- Networks - Ravikumar Venkatesan
- Routers - Ravikumar Venkatesan
- Load balancers -
- Firewalls -
- VPN -
- Containers - Ravikumar Venkatesan
- Orchestration - Daniel Korn (dkorn)
- Databases / Database backups -
- Network topology -
As an admin
- Admin overview -
- Resource Usage -
- Hypervisors -
- Aggregates -
- Instances -
- Volumes -
- Images -
- Networks -
- Routers -
- Defaults -
- System Info -
- Flavors -
- Domains -
- Projects - Vlad Okhrimenko
- Users -
- Groups -
- Roles -
- User settings - Daniel Korn (dkorn)
- Change password - Daniel Korn (dkorn)
Other Implementation ideas (Dismissed approaches)
Directly using Selenium
Point Selenium to a (probably devstack-based) OpenStack installation.
Advantages: Simplest solution
Disadvantages: Must know Python, must know how to write robust UI-driven tests using Selenium
A translator framework that makes it possible to write tests in a high-level language (similar to BDD) and makes the tests more resilient to changes in the software implementation details. See also: JBehave.
Advantages: Simplify the writing of test scenarios by experienced QA/QE/Testing teams who may not know Python, by enabling them to write the scenarios in English.
Disadvantages: The load is now shifted onto the development team who must spend the time keeping the translator layer up to date, and enabling new keywords/scenarios. (?)