Rally/Updates
Contents
Weekly updates
April 14, 2014
Hello stackers,
let us list our main contributions to Rally during the past week:
- The Bechmark context manager class, which mostly finishes the work on contexts integration in Rally; this manager allows Rally to use only those contexts that are really needed for the current benchmark scenario, thus greatly reducing the runtime of the preparation step, manages the order of entering those contexts, adds logging for contexts and does a couple of other important things;
- Improvement of the CLI for the Tempest verifier: we've added the functionality of saving verification results into a database and also a range of new commands to handle these results, such as rally verify list, rally verify detailed and others;
- Validation for required parameters in benchmark scenarios has now been implemented through a new validator, added, as usual, as a decorator to the corresponding benchmark scenarios;
This week, our work continues with several new benchmark scenarios (including those for Tempest), new validators and also new functionality in the Rally core, e.g. the ability to use already existing tenants/users for benchmarking, implemented now through contexts.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
April 7, 2014
Hello stackers,
our contributions this week have been pretty diverse and touched different parts of Rally. Let us name a few that seem to be the most interesting ones:
- Nova and Cinder quotas management through a new context class. Another example of how useful and easy to use contexts are in Rally;
- Concurrent users creation and deletion in the user context. This change greatly reduces the runtime of this benchmarking step;
- New benchmark scenarios for Keystone and Glance;
- Nice enhancement of the auto-generated report html page through adding histograms for runtimes there.
There are also other parts of Rally which are in progress now and will hopefully have their patches merged the next week, e.g. the work on Tempest integration or further context classes implementation.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
March 31, 2014
Hello stackers,
this week, there has been much progress both in the functionality and quality of the Rally code. This has become possible due to the numerous contributions of members of our constantly growing community. The most interesting novelties include:
- New scenario runner which is called "serial" and enables running scenarios in the same thread as Rally one by one without any concurrency, which makes it easy to debug the benchmark scenarios code;
- Several contributions to the benchmark scenarios code:
- a new cinder scenario for creating a volume and then listing all volumes;
- the introduction of a special decorator that makes it easy to distingush the benchmark scenario methods inside Scenario classes from the helper ones;
- passing the context objects to scenarios which enables to run scenarios like NovaServers.boot_runcommand_delete_server, which needed SSH access to the servers based on the user information passed through contexts.
- NB! We have changed the names of the commonly used DummyEngine and DummyProvider classes to ExistingCloud and ExistingServers, respectively. This has been done in order to improve the readability of deployment configuration files by using more obvious naming.
This week, we concentrate our efforts on diverse work directions that include performance enhancement of the core parts of Rally (as e.g. the UserGenerator context), introducing new benchmark scenarios and finally improving the overall quality of unit tests.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
March 24, 2014
Hello stackers,
with great pleasure we've been observing the high rate at which the number of patches pending for review grew over the past week in Rally. Indeed, our community constantly becomes larger and larger (including one new core developer this week), and much more active as well. The highlights of our recent efforts are as follows:
- New benchmark scenarios, including those for testing the tenant creation performance in keystone and also for booting a server and then issuing the "servers list" command;
- Introduction of the Rally REST API basics which ultimately will make it possible to use Rally as a Service;
- A wide range of nice local code improvements that bring more consistency and simplicity to it: among others, let's mention:
- a patch that removed quite a lot of unused code in server providers;
- reducing code duplicate in logging wrappers via their unification
- standardizing the scenario runner output by switching from the simple dictionaries usage to a bit more involved ScenarioRunnerResult class which automatically performs the format correctness tests, thus making the code much more reliable.
This week, we are going to invest a lot of time in further code refactoring (in a range of areas), as well as in implementing new benchmark scenarios to make Rally applicable for more and more testing use cases.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
March 17, 2014
Hello stackers,
the past week has resulted in the further refinement of different parts of Rally code, of its CLI as well as of the configuration file formats. Several of these changes have been caused by the current integration of contexts into Rally (let us remind you that the notion of contexts is used to define different environments in which benchmark scenarios can be launched, e.g. environments with temporarity generated OpenStack users and/or with generic cleanup facilities). Some interesting changes include:
- An enormously important and overall refactoring patch that brings different optimizations to the config validation step, to the CLI output/logging and also (NB!) changes the input task configuration file format (take a look at the updated task configuration samples);
- An important step on the way to a complete Rally-Tempest integration is adding the ability to launch Tempest tests without "sudo";
- OpenStack clients helper module refactoring, which is concentrated on reimplementing the "lazy" client handles in a more elegant way and also making them accessible in a direct way, without auxiliary methods like rally.benchmark.utils.create_openstack_clients(), which made the code unreasonably more complicated;
- We've also added several missing unit tests: the ones for the "deployment list" command as well as for the Authenticate benchmark scenario group.
This week, the directions of our efforts are going to be mostly defined by the changes from the refactoring patch mentioned earlier. A lot of stuff has to be rebased, while a couple of deferred important patches (including those implementing support for pre-created users in Rally or introducing the new "stress" scenario running strategy) will be brought to life again.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
March 10, 2014
Hello stackers,
over the past week the direction of our efforts hasn't changed significantly: we are still working hard on further logical organization of the core parts of Rally which will enable the system to be even more extendable than it is now. Some important changes include:
- Further work on integrating the Context classes into Rally. Let us remind you that the notion of contexts is used by us to define different environments in which benchmark scenarios can be launched by Rally, e.g. an environment with temporarity generated OpenStack users and/or a context that enables generic cleanup for the benchmark scenarios. This week, we have added the base Context class with a unified interface and we have also rewritten some already existing context classes according to the base class API (https://review.openstack.org/#/c/78193/);
- Various fixes in the Devstack deploy engine, including the support for connecting to the VM with a user-password combination instead of a key-pair (https://review.openstack.org/#/c/77540/), minor bugfix in the cleanup procedure (https://review.openstack.org/#/c/70727/) and adding support for git branching (https://review.openstack.org/#/c/78225/);
- Many small but important improvements that make the code overall more readable, e.g. using the configuration files in appropriate places (https://review.openstack.org/#/c/78325/), moving a couple of helper methods for the benchmark engine to the correct modules (https://review.openstack.org/#/c/78524/), replacing the incorrect mocking syntax with the decorator-based one (https://review.openstack.org/#/c/78589/) and so on.
This week, we are going to continue the work on the context classes for benchmark scenarios since this is going to be a tool which will make Rally really pluggable. Current tasks include:
- Changing the benchmark scenario input config format;
- Splitting the already existing validation procedures to different context classes in a logical way;
- Implementing the Context class factory (like we did with deploy engines or scenario runners);
and many others.
We are also going to introduce several enhancement both to the task result output (in its HTML form) and to the code (by moving some common code to a special utils module).
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
March 3, 2014
Hello stackers,
the most important changes during the past week have been concentrated on further logical structuring of the core part of Rally, namely the benchmark engine, and include:
- Benchmark scenario arguments validation refactoring: we've moved the whole process from the ScenarioRunner to the BenchmarkEngine class (which is a much more logical place for that) and also added the support for admin-based and user-based validation differentiation (https://review.openstack.org/#/c/76162/);
- Context introduction, which is a very important novelty for Rally: from now on, we are going to use the notion of context to define different environments in which benchmark scenarios can be launched by Rally. The already existing temporary UserGenerator and ResourceCleanuper classes are in fact also context, so a natural first step was to move them to a special context module (https://review.openstack.org/#/c/77322/);
- Much work has been done around small fixes in the code (https://review.openstack.org/#/c/77170/, https://review.openstack.org/#/c/77192/) and in the unit tests (https://review.openstack.org/#/c/75877/, https://review.openstack.org/#/c/76884/), as well as around user experience improvements touching the CLI (https://review.openstack.org/#/c/76226/, https://review.openstack.org/#/c/76221/).
The current direction of our work is the further development of contexts, which will involve introducing new context classes as well.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
February 24, 2014
Hello stackers,
this week, several important contributions have been made to Rally, considering both the overall system stability and the improvements of the user interface. To name a few:
- Vast refactoring of the ScenarioRunner class has enabled to stop sharing OpenStack clients objects between processes in the core of the system, which occasionally caused bugs in Rally (https://review.openstack.org/#/c/74769/);
- Another important refactoring step resulted in the replacement of OpenStack endpoint dictionaries with special objects throughout the system, which has made the code more reliable and extendable (https://review.openstack.org/#/c/74425/);
- Perhaps the prettiest patch of the week was the introduction of a benchmark result visualization tool, implemented with the nvd3 plugin to d3.js (so that the actual charts are drawn to a html file). The graphs look really nice and will be of great use for those who want to share their benchmarking results (https://review.openstack.org/#/c/72970/);
- Several nice improvements in the CLI include the showing of 90- and 95- percentile results in the benchmark summary (https://review.openstack.org/#/c/73522/) and a new show command which allows the user to get the information on images/flavors/networks/etc. available in the current deployment in a very quick way (https://review.openstack.org/#/c/75699/).
The ongoing work includes:
- An extention of the use command which will be applicable soon not only to deployments but also to tasks (https://review.openstack.org/#/c/75936/);
- Further refactoring of the core benchmark engine, including the work around input configuration parameters validation (for a detailed description of what's going to be done, see this special document);
- After finishing some major refactoring procedues, we have also resumed the work around passing pre-created user endpoints to the DummyEngine (https://review.openstack.org/#/c/67720/) and generating the "stress" load on the cloud.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
February 17, 2014
Hello stackers,
the first thing we would like to mention today is an extensive piece of work that has been done recently on our Wiki. We have updated both the main page and the basic tutorials: how to install Rally and how to use it. These tutorials have been simplified quite a lot and have been updated with new features that have been merged in Rally recently.
As for the actual updates in the Rally code, the main ones are as follows:
- The refactoring of the ScenarioRunner class has been successfully continued by reimplementing different benchmark execution strategies (continuous/periodic executions), which resided previously inside the original class, via subclassing. The new ContinuousScenarioRunner and PeriodicScenarioRunner classes enable us to make the code much more readable (no more complicated if...else logic to choose the appropriate execution strategy is present in the code) and extendable, so that it is now very easy to add your own ScenarioRunner (https://review.openstack.org/#/c/70771/);
- We've successfully started the work on Tempest & Rally integration in order for the latter to be able to perform OpenStack deployment verification procedures as well as to possibly add new Tempest-based benchmark scenarios (https://review.openstack.org/#/c/70131/);
- On the way to adding a full support for benchmarking with predefined OpenStack users (instead of using the generated ones), we've refactored the Endpoint entity, making it able to distinguish between administrator/user permisisons. Besides, each deployment gets now stored in the database with a list of endpoints instead of only one endpoint, since we are going to enable the DummyEngine to take several endpoints as its input (https://review.openstack.org/#/c/67154/);
- One of the future features of Rally will be Heat-based benchmark scenarios which will make it possible to test the VMs performance. This week, we have started contributing to this as well (https://review.openstack.org/#/c/72749/);
- Among many other simplier refactoring patches finished this week, we'd like to mention the one that improves the CLI code by structuring it to submodules (https://review.openstack.org/#/c/73059/).
The ongoing work includes:
- Further changes needed for the DummyEngine in order to be able to accept predefined user endpoints (instead of a single admin endpoint) and for the ScenarioRunner to use them in benchmarks (https://review.openstack.org/#/c/67643/, https://review.openstack.org/#/c/67710/, https://review.openstack.org/#/c/67720/);
- New benchmark result visualization tool based on nvd3 plugin to d3.js (so that the actual charts are drawn to a html file). This also includes a new CLI command rally task plot2html <task_uuid> (https://review.openstack.org/#/c/72970/);
- Adding 90- and 95-percentile results to the CLI output for benchmark runtimes, i.e. printing the runtime ranges that encompass 90% and 95% of benchmarks respectively (https://review.openstack.org/#/c/73522/);
- Continuing the work on Rally & Tempest integration, new deployment engine types etc.
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
February 10, 2014
Hello stackers,
the past week has been extremely successful for us regarding the overall Rally code improvement, bugfixing, as well as new features implementation. Rally is actually about to become a totally easy-to-understand and easy-to-use piece of software which can be exploited by everyone interested in it.
The most important contributions to Rally made during the past week are as follows:
- The code refactoring stuff has been quite involved:
- We have issued a drastic rearrangement of the ScenarioRunner class (that is responsible for the actual benchmark method calls using a particular benchmarking strategy) by moving out some code from this class to new context classes. This change also enabled Rally to process all errors occuring on the cloud during benchmarking/cleanup correctly (https://review.openstack.org/#/c/69886/);
- Another significant contribution is the sshutils module refactoring, which involves the API improvement as well as the new ability to process the stdin data (https://review.openstack.org/#/c/68063/);
- Finally, a very nice work has been done on the benchmark scenarios refactoring by moving the hardcoded timeout and cloud poll interval values to rally.conf (https://review.openstack.org/#/c/71272/).
- Very important bugfixes addressing the improper implementation of OpenStack resource deletion (https://review.openstack.org/#/c/66856/) and benchmark timeout handling (https://review.openstack.org/#/c/72103/) have been merged this week as well;
- Our set of available benchmark scenarios has been expanded with benchmark scenarios for Glance: they include a scenario for adding and deleting an image and a scenario for booting several instances from a previously added image (https://review.openstack.org/#/c/60469/).
The current work encompasses the following directions:
- Further scenario runners refactoring: we are now reimplementing the different benchmark execution strategies (continuous/periodic executions) via subclasses of the base ScenarioRunner class thus making the code much more readable and extendable (https://review.openstack.org/#/c/70771/);
- Reimplementing the patches for DummyEngine refactoring (making it able to work with a predefined set of users instead of the generated ones) based on the updated scenario runners (https://review.openstack.org/#/c/67154/, https://review.openstack.org/#/c/67643/, https://review.openstack.org/#/c/67710/, https://review.openstack.org/#/c/67720/);
- Rally & Tempest integration is a new ambitious piece of work we are conducting now. Tempest is going to be used inside Rally as a base for the cloud verification functionality in Rally as well as for new benchmark scenarios that use Tempest (https://review.openstack.org/#/c/70131/).
- We are putting now much effort in finishing the work on new deploy engines: the MultihostEngine (https://review.openstack.org/#/c/57240/), the LxcEngine (https://review.openstack.org/#/c/56222/) and the FuelEngine (https://review.openstack.org/#/c/61963/).
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
February 03, 2014
Hello stackers,
our efforts during the past week were heavily focused on code refactoring and bugfixing. Among the most significant contrubutions are:
- A fix for certain inconsistencies in the code that checked the availability of resources of OpenStack, e.g. whether a particular resource got deleted or not (https://review.openstack.org/#/c/66856/);
- The work on refactoring the scenario runner to make its code clean (https://review.openstack.org/#/c/69846/).
Several novelties have been introduced to Rally:
- After having developed the abstract validators mechanism, we have developed a couple of useful concrete validators as well: the one that checks that the image indicated in the config for, say, the NovaServers.boot_and_delete_server benchmark scenario really exists and can be used (https://review.openstack.org/#/c/68055/) and another validator that does the same for flavors (https://review.openstack.org/#/c/70082/). Both validators have been attached to benchmark scenarios where they are of great use.
- We've implemented the mechanism of measuring the time taken by atomic actions in our benchmark scneario (https://review.openstack.org/#/c/69828/): e.g. now Rally outputs not only the information on how long it took the cloud to boot and delete a single server (in the NovaServers.boot_and_delete_server scenario), but also how much time it took to boot the server and to delete it.
This week there is still a huge amount of work to be done around refactoring the very fundamental code in Rally. Among other things, we now rewrite the ScenarioRunner class which is the tool for launching benchmark scenarios (https://review.openstack.org/#/c/69886/) so that its functionality gets split into several context classes (responsible for temporary users management and resource cleanup after benchmarking), and also implementing different scenario launching strategies via inheritance (https://review.openstack.org/#/c/70771/).
We continue implementing new features in Rally as well. One example is the ongoing work on atomic actions runtime measurement: it is about to be supported by the CLI which will now display this detailed runtime information in a user-friendly way (https://review.openstack.org/#/c/70362/).
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
Stay tuned.
Regards,
The Rally team
