MagnetoDB/QA/Organization of tests in CI

Organization of Tests in CI

Idea

During projects sprints, the amount of implemented functionality and the appropriate tests increases all the time. Development of functional code units and test case scenarios are two separate and separately managed processed, and they are not always synchronous due to several objective reasons.

In view of that, to combine efforts of both development and QA teams, it was decided to create a CI process that takes into account differences in pace between the two flows (Dev and QA) and, in the same time, provide the project management with a clear picture of development progress and summery status of code quality.

Structure

Automated test case scenarios are divided into 3 folders depending on readiness of the appropriate pieces of code they are to check:

1. /stable contains scenarios that verify already implemented functionality
2. /in_progress keeps scenarios that are in progress (i.e. is being worked on) or the functionality that is a subject of changes (minor additions, refactoring, debugging)
3. /not ready is dedicated for the test scenarios that were created, but the appropriate functionality is not ready yet (not implemented or partially implemented)

Flow

Tests are being run each time in sequential order starting from /stable to /not ready.

Folders /in_progress and /not ready are marked as not voting in this CI flow, so fails of the appropriate test scenarios do not affect the overall result of CI flow. Only the result of test run of /stable folder is taken into account.

It is expected that closer to the end of the project all the tests will migrate from /in_progress and /not ready folder to /stable. Such an approach helps with keeping the whole development process under control and allows detecting and quick amendment of omissions and mistakes in synchronization between the code and the tests.