Jump to: navigation, search

Difference between revisions of "Rally/Updates"

(February archives)
(January 22)
 
(47 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
== Weekly updates  ==
+
== Updates ==
  
  
−
==== April 14, 2014 ====
+
==== January 22, 2014 ====
  
−
Hello stackers,
+
Hi stackers!
 +
 
 +
We are happy to announce that we have completely redesigned our [http://rally.readthedocs.org/en/latest/ Rally documentation in ReadTheDocs]. The docs have now received a simpler structure and have become much easier to get through!
  
−
let us list our main contributions to Rally during the past week:
+
One of the nicest new things is the [http://rally.readthedocs.org/en/latest/tutorial.html Rally step-by-step tutorial] that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.
−
* The [https://review.openstack.org/#/c/85732/ '''''Bechmark context manager class'''''], which mostly finishes the work on contexts integration in Rally; this manager allows Rally to use only those contexts that are really needed for the current benchmark scenario, thus greatly reducing the runtime of the preparation step, manages the order of entering those contexts, adds logging for contexts and does a couple of other important things;
 
−
* Improvement of the CLI for the '''''Tempest verifier''''': we've added [https://review.openstack.org/#/c/85584/ '''''the functionality of saving verification results into a database'''''] and also [https://review.openstack.org/#/c/85584/ '''''a range of new commands to handle these results'''''], such as ''rally verify list'', ''rally verify detailed'' and others;
 
−
* [https://review.openstack.org/#/c/84961/ '''''Validation for required parameters in benchmark scenarios'''''] has now been implemented through a new validator, added, as usual, as a decorator to the corresponding benchmark scenarios;
 
  
 +
Since our previous update, there have been many interesting updates in Rally:
 +
* Rally now has a [https://review.openstack.org/103306 Network Context class] that enables easy Neutron network management.
 +
* Input task files now can be written using the [https://review.openstack.org/137716 jinja2-based templates syntax]. Very useful if you want, say, parameterize the image name used throughout your complex input task file.
 +
* Rally scenarios have been [https://review.openstack.org/127192 100%-covered with docstrings]. That means that now the '''rally info find <query>''' command will always output a complete piece information about whatever you ask it.
 +
* New benchmark scenarios include thos for '''Cinder''' ([https://review.openstack.org/142500 list_volumes], [https://review.openstack.org/146580 list_snapshots], [https://review.openstack.org/147489 extend_volume]), '''Nova''' ([https://review.openstack.org/148307 cold_migrate])
 +
* We have [https://review.openstack.org/147412 moved] the directory with samples in our repository to the root level: now it is ''rally/samples'' instead of ''rally/doc/samples'' (and so much quicker to get to).
 +
* Rally is on the way to being '''Python 3 compatible'''. We have added a Gate job that checks Rally in Python 3 and have produced [https://review.openstack.org/144706 lots] [https://review.openstack.org/144971 of] [https://review.openstack.org/145450 patches] [https://review.openstack.org/145492 that] [https://review.openstack.org/141163 fix] [https://review.openstack.org/144185 incompability] [https://review.openstack.org/144143 issues]. Few changes are left to make Rally fully Python 3 compatible.
  
−
This week, our work continues with several new '''''benchmark scenarios''''' (including those for [https://review.openstack.org/#/c/86337/ '''''Tempest''''']), new [https://review.openstack.org/#/c/86655/ '''''validators'''''] and also new functionality in the Rally core, e.g. the ability to use already existing tenants/users for benchmarking, implemented now through contexts.
 
  
 +
Current work includes new benchmark scenarios ([https://review.openstack.org/144320 Mistral], [https://review.openstack.org/137650 Murano]), [https://review.openstack.org/148079 new success criteria (SLA)] and a lot of refactoring in scenario runners, Rally API etc.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
−
Stay tuned.
+
Stay tuned!
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
Line 31: Line 37:
  
  
−
==== April 7, 2014 ====
 
  
−
Hello stackers,
+
==== December 15, 2014 ====
  
−
our contributions this week have been pretty diverse and touched different parts of Rally. Let us name a few that seem to be the most interesting ones:
+
Hi stackers!
−
* [https://review.openstack.org/#/c/83835/ '''''Nova and Cinder quotas management'''''] through a new context class. Another example of how useful and easy to use contexts are in Rally;
 
−
* '''''Concurrent users [https://review.openstack.org/#/c/79968/ creation] and [https://review.openstack.org/#/c/84106/ deletion]''''' in the user context. This change greatly reduces the runtime of this benchmarking step;
 
−
* '''''New benchmark scenarios for [https://review.openstack.org/#/c/81781/ Keystone] and [https://review.openstack.org/#/c/82717/ Glance]''''';
 
−
* Nice enhancement of the auto-generated report html page through adding [https://review.openstack.org/#/c/82912/ '''''histograms for runtimes'''''] there.
 
  
 +
Let us share with you our recent accomplishments in Rally:
 +
* '''CLI improvements:'''
 +
:* The '''''rally info''''' command (which is a kind of built-in Rally reference) has been [https://review.openstack.org/#/c/131005/ enhanced] in such a way that it now prints detailed explanations of main concepts used in Rally whenever you type something like '''''rally info BenchmarkScenarios''''' or '''''rally info SLA'''''. We've also improved the output formatting so that now it is much easier to get through.
 +
:* The '''''rally task list''''' command now [https://review.openstack.org/131005 supports] '''filters'''. You can filter the task list either by deployment (using the ''"... --deployment <deployment_name_or_id>"'' parameter) or by status (''"... --status <status_name>"'')
 +
* '''New benchmark scenarios:'''
 +
:* [https://review.openstack.org/#/c/129282 '''''NovaSecGroup.boot_and_delete_server_with_secgroups''''']: creates a number of Nova security groups with rules, then creates a Neutron network with one subnet, finally boots a VM with created security groups, lists the created resources and performs cleanup.
 +
:* [https://review.openstack.org/#/c/139019/ '''''CinderVolumes.create_nested_snapshots_and_attach_volume''''']: creates a volume, its snapshot, and then (recursively) a volume from that snapshot. The recursion depth can be set by the user through the ''--nested_level'' parameter.
 +
* '''Other improvements''':
 +
:* Work on '''Trove support in Rally''' has been started with the [https://review.openstack.org/139047 integration of its client];
 +
:* New [https://review.openstack.org/139022 '''hacking rules'''] are there to provide better codestyle throughout Rally.
  
−
There are also other parts of Rally which are in progress now and will hopefully have their patches merged the next week, e.g. the work on Tempest integration or further context classes implementation.
 
  
 +
Current work is centered aroud code refactoring (both major, as in the '''benchmark engine''' or the [https://review.openstack.org/139987 '''contexts'''], and minor, as introducing some syntax sugar via '''decorators''' to mark [https://review.openstack.org/138489 deprecated stuff] and [https://review.openstack.org/140033 scenario samples]). We also constantly work on expanding our [https://review.openstack.org/141672 scenario] [https://review.openstack.org/136922 base]. Last but not least, we're about to merge the [https://review.openstack.org/103306 Network Context class] that enables easy Neutron network management.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
−
Stay tuned.
+
Stay tuned!
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
Line 61: Line 72:
  
  
−
==== March 31, 2014 ====
 
  
−
Hello stackers,
+
==== December 1, 2014 ====
  
−
this week, there has been much progress both in the functionality and quality of the Rally code. This has become possible due to the numerous contributions of members of our constantly growing community. The most interesting novelties include:
+
Hi stackers!
−
* '''''New scenario runner''''' which is called [https://review.openstack.org/#/c/83189/ '''''"serial"'''''] and enables running scenarios in the same thread as Rally one by one without any concurrency, which makes it easy to debug the benchmark scenarios code;
 
−
* Several contributions to the '''''benchmark scenarios code''''':
 
−
:* [https://review.openstack.org/#/c/82723/ a new cinder scenario for creating a volume and then listing all volumes];
 
−
:* the introduction of [https://review.openstack.org/#/c/83705/ a special decorator] that makes it easy to distingush the benchmark scenario methods inside Scenario classes from the helper ones;
 
−
:* [https://review.openstack.org/#/c/79752/ passing the '''context''' objects to scenarios] which enables to run scenarios like ''NovaServers.boot_runcommand_delete_server'', which needed SSH access to the servers based on the user information passed through contexts.
 
−
* '''''NB!''''' We have ''changed the names'' of the commonly used '''''DummyEngine''''' and '''''DummyProvider''''' classes to [https://review.openstack.org/#/c/83290 '''''ExistingCloud'''''] and [https://review.openstack.org/#/c/83293/ '''''ExistingServers'''''], respectively. This has been done in order to improve the readability of deployment configuration files by using more obvious naming.
 
  
 +
It's been a while since our last post here, and we've done quite a nice job in Rally during November. Let us share with you new things about Rally:
 +
* '''Autogenerated HTML benchmark reports''' in Rally (which can be created by the '''''"rally task report"''''' command after a benchmark task has completed) have [https://review.openstack.org/#/c/131844/ been] [https://review.openstack.org/#/c/136435/ improved] further within the last month. As of now, the [http://logs.openstack.org/05/131005/28/check/gate-rally-dsvm-rally/f8f3da9/rally-plot/results.html.gz report page] contains an overview table, detailed informations about whether [http://logs.openstack.org/05/131005/28/check/gate-rally-dsvm-rally/f8f3da9/rally-plot/results.html.gz#/Authenticate.validate_cinder SLA (service-level agreement) checks] were successful and also detailed error logs, if any. Rally reports have become a wonderful tool indeed to analyse the benchmarking data as well as to share your results with others!
 +
* Similar improvements have been made for HTML reports generated for the '''Tempest cloud verification''' ('''''"rally verify results --html --output_file <file>"'''''). New [https://review.openstack.org/#/c/135232/ enhanced] report pages have improved styling and refactored JS code.
 +
* We have [https://review.openstack.org/#/c/137502/ changed] the way '''context classes''' in Rally should be declared. Having introduced a new '''''@context''''' decorator, we've made it much easier and also more readable.
 +
* There is a new '''[https://review.openstack.org/127392 "servers" context]''' that allows you to create temporary servers before benchmark scenarios start and use these servers for testing inside these scenarios.
 +
* '''New benchmark scenarios''' in Rally include those for '''[https://review.openstack.org/128631 Nova live migrate]''' and also a '''[https://review.openstack.org/127392 Cinder stress scenario]'''.
 +
* '''Command-line interface improvements''' include an ability to [https://review.openstack.org/131463 refer deployments not only by uuid but also by name]. Please note that the syntax has changed a bit so now you have to supply the ''--deployment'' parameter to commands like ''"rally use deployment"'' (instead of ''--uuid'').
 +
* There has been some '''major refactoring''' of the most critical parts of Rally code: the [https://review.openstack.org/129060 cleanup mechanism] and the [https://review.openstack.org/119297 "users" context code]. We are sure that after refactoring, this code has become both cleaner and less error-prone (as well as very pluggable in the case of cleanups).
  
−
This week, we concentrate our efforts on diverse work directions that include performance enhancement of the core parts of Rally (as e.g. the [https://review.openstack.org/#/c/84106/ '''''UserGenerator''''' context]), introducing [https://review.openstack.org/#/c/82717/ new benchmark scenarios] and finally improving the overall quality of unit tests.
 
  
 +
Current work includes further code refactoring (e.g. in the ''Benchmark engine'' part), further CLI improvements (e.g. for the [https://review.openstack.org/131005 "rally task list" command]) and also new benchmark scenarios (e.g. for [https://review.openstack.org/137661 Murano]). We are also going to introduce a possibility of building Rally images for [https://review.openstack.org/#/c/132556/ Docker].
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
−
Stay tuned.
+
Stay tuned!
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
Line 93: Line 105:
  
  
−
==== March 24, 2014 ====
+
 
 +
==== October 27, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
−
with great pleasure we've been observing the high rate at which the number of patches pending for review grew over the past week in Rally. Indeed, our community constantly becomes larger and larger (including one new core developer this week), and much more active as well. The highlights of our recent efforts are as follows:
+
much time has passed since our last update and we are happy to announce that we are moving towards making our first official Rally release! Our active recent contribution to Rally has enabled us to make a significant progress. Here are the highlights of the novelties in Rally:
−
* '''''New benchmark scenarios''''', including those for testing the [https://review.openstack.org/#/c/81718/ tenant creation performance in keystone] and also for [https://review.openstack.org/#/c/82428/ booting a server and then issuing the "servers list" command];
+
* We have completely '''[https://review.openstack.org/125119 redesigned] the auto-generated benchmark report page''' so that it looks now even nicer than before and is much more easy to navigate. Besides, further improvements of this HTML report page are on their way to being merged soon.
−
* Introduction of the [https://review.openstack.org/#/c/66788/ Rally REST API basics] which ultimately will make it possible to use '''''Rally as a Service''''';
+
* Rally now has an extended support of '''[https://review.openstack.org/103145 plugins]''': in addition to writing custom scenarios, the plugin mechanism now enables to extend Rally with new context classes/scenario runners without actually contributing to the Rally master branch.
−
* A wide range of nice '''''local code improvements''''' that bring more consistency and simplicity to it: among others, let's mention:
+
* The work on extending the support of OpenStack projects in Rally has been conducted for '''[https://review.openstack.org/128874 Heat]''' and '''[https://review.openstack.org/126900 Sahara]'''.
−
:* a patch that [https://review.openstack.org/#/c/81514/ removed quite a lot of unused code in server providers];
+
* '''CLI improvements''': the command-line interface gets more and more user-friendly over time: recently, it has begun to support [https://review.openstack.org/124910 detailed] informations about correct commands usage in case of a failure. The ''"rally info"'' command has also been improved so that it now supports '''[https://review.openstack.org/125238 misspellings handling]'''. Finally, there has been some work on '''[https://review.openstack.org/129306 bash completion]''', which hasn't been completely finished yet.
−
:* [https://review.openstack.org/#/c/81731/ reducing code duplicate in logging wrappers] via their unification
+
* '''Test code improvements''': we have greatly proceeded in out continuous work on unit/functional test coverage improvement. We also have [https://review.openstack.org/126379 moved] all the tests into a special ''tests/'' directory so that the test code is now organized in a more neat way. We also have removed the ''./run_tests.sh'' script for the sake of using the ''tox'' command to launch the test suite.
−
:* [https://review.openstack.org/#/c/81970/ standardizing the scenario runner output] by switching from the simple dictionaries usage to a bit more involved ''ScenarioRunnerResult'' class which automatically performs the ''format correctness'' tests, thus making the code much more reliable.
 
−
 
 
  
−
This week, we are going to invest a lot of time in further code refactoring (in a range of areas), as well as in implementing new benchmark scenarios to make Rally applicable for more and more testing use cases.
 
  
 +
In the nearest future, several interesting refactoring patches are going to come to Rally. To be more specific, there will be a vast change of the benchmark engine and cleanup mechanisms.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 119: Line 130:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
−
 
+
==== September 26, 2014 ====
−
==== March 17, 2014 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
−
the past week has resulted in the further refinement of different parts of Rally code, of its CLI as well as of the configuration file formats. Several of these changes have been caused by the current integration of '''''contexts''''' into Rally (let us remind you that the notion of ''contexts'' is used to define different environments in which benchmark scenarios can be launched, e.g. environments with temporarity generated OpenStack users and/or with generic cleanup facilities). Some interesting changes include:
+
here is a brief overview of what has happened in Rally recently:
−
* An enormously important and overall [https://review.openstack.org/#/c/80151/ refactoring patch] that brings different optimizations to the '''''config validation''''' step, to '''''the CLI output/logging''''' and also ('''NB!''') changes '''''the input task configuration file format''''' (take a look at the updated [https://github.com/stackforge/rally/tree/master/doc/samples/tasks task configuration samples]);
+
* We have [https://review.openstack.org/111977 refactored] the code responsible for '''atomic actions processing''' in Rally benchmarks. Let's remind you that each benchmark scenario in Rally consists of a series of atomic actions, whereas the running time of each atomic action is measured in the same way as that of the whole scenario. After refactoring, we now ensure that Rally '''doesn't skip atomic actions that failed''' and '''distinguishes different runs of two atomic actions with the same name''' (see an example of how a typical results table could look [http://paste.openstack.org/show/114111/ before] and looks [http://paste.openstack.org/show/114112/ after] refactoring).
−
* An important step on the way to a complete '''''Rally-Tempest integration''''' is adding the ability to [https://review.openstack.org/#/c/79664/ launch Tempest tests without ''"sudo"''];
+
* Another direction of refactoring was the [https://review.openstack.org/118243 SLA code]; it has been modified so that now SLA results are '''stored in the DB''' along with other benchmarking data.
−
* [https://review.openstack.org/#/c/79372 OpenStack clients helper module refactoring], which is concentrated on reimplementing the "lazy" client handles in a more elegant way and also making them accessible in a direct way, without auxiliary methods like ''rally.benchmark.utils.create_openstack_clients()'', which made the code unreasonably more complicated;
+
* Check out a nice [https://github.com/stackforge/rally/blob/master/doc/user_stories/nova/boot_server.rst user story] about '''VMs boot performance''' in Nova with Rally. It shows how Rally can be used in practice to catch reals bugs and improve OpenStack performance.
−
* We've also added several missing unit tests: the ones for [https://review.openstack.org/#/c/80772/ the "deployment list" command] as well as for [https://review.openstack.org/#/c/80342/ the Authenticate benchmark scenario group].
+
* We work hard on achieving '''100% test coverage''' in Rally: last week, additional unit tests for [https://review.openstack.org/122127 contexts] and [https://review.openstack.org/122729 scenarios] have been merged.
  
  
−
This week, the directions of our efforts are going to be mostly defined by the changes from the refactoring patch mentioned earlier. A lot of stuff has to be rebased, while a couple of deferred important patches (including those implementing support for '''''pre-created users in Rally''''' or introducing '''''the new "stress" scenario running strategy''''') will be brought to life again.  
+
Our current priorities are further '''refactoring''' steps, including those in critical parts of Rally (e.g. [https://review.openstack.org/119297 temporary user creation] and [https://review.openstack.org/116269 cloud cleanup] code); we also strive towards making Rally bug-free and continuously issue different bugfixing patches.
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 149: Line 158:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
−
==== March 10, 2014 ====
+
==== September 15, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
−
over the past week the direction of our efforts hasn't changed significantly: we are still working hard on further logical organization of the core parts of Rally which will enable the system to be even more extendable than it is now. Some important changes include:
+
there has been much diverse and useful contribution to Rally recently. Let us highlight some interesting updates in our project:
−
* '''''Further work on integrating the Context classes''''' into Rally. Let us remind you that the notion of ''contexts'' is used by us to define different environments in which benchmark scenarios can be launched by Rally, e.g. an environment with temporarity generated OpenStack users and/or a context that enables generic cleanup for the benchmark scenarios. This week, we have added the base ''Context'' class with a unified interface and we have also rewritten some already existing context classes according to the base class API (https://review.openstack.org/#/c/78193/);
+
* Rally is on its way to support of '''benchmarking OpenStack clouds using ordinary user accounts that already exist'''. Rally lacked such functionality (it only supported benchmarking either from an admin account or from a bunch of temporarily created users), which posed a problem since some deployments don't allow temporary users creation. There have been [https://review.openstack.org/#/c/116766/ two] [https://review.openstack.org/#/c/119344/ patches] that prepare the code for this new functionality. It is going to come very soon - stay tuned.
−
* Various fixes in the '''''Devstack deploy engine''''', including the support for connecting to the VM with a user-password combination instead of a key-pair (https://review.openstack.org/#/c/77540/), minor bugfix in the cleanup procedure (https://review.openstack.org/#/c/70727/) and adding support for ''git branching'' (https://review.openstack.org/#/c/78225/);
+
* We constantly improve our '''gate jobs''' (jobs that run a test suite for every patch pending review in Rally). Recently, we have [https://review.openstack.org/#/c/119584/ introduced] a nice aggregated results page for that gate job (see an [http://logs.openstack.org/84/119584/13/gate/gate-rally-dsvm-rally/d30c028/ example]). It now makes it very easy for developers to navigate through the results of tests agains their patches to Rally.
−
* Many small but important improvements that make the code overall more readable, e.g. using the configuration files in appropriate places (https://review.openstack.org/#/c/78325/), moving a couple of helper methods for the ''benchmark engine'' to the correct modules (https://review.openstack.org/#/c/78524/), replacing the incorrect mocking syntax with the decorator-based one (https://review.openstack.org/#/c/78589/) and so on.
+
* We have [https://review.openstack.org/116446 added] a new '''"volumes" context''' which makes it possible to create cinder volumes in the benchmark environment and use them later in the actual benchmark scenarios.
 +
* Much work has been accomplished on the overall code quality improvement. We have both made several code refactoring patches and introduced several new test suites (say, [https://review.openstack.org/118714 functional tests for CLI]).
  
  
−
This week, we are going to continue the work on the '''''context classes''''' for benchmark scenarios since this is going to be a tool which will make Rally really pluggable. Current tasks include:
+
Current work centers around introducing benchmarking with pre-created users as mentioned above; much work is currently devoted to new benchmark scenarios as well. Also please note that there will be soon an important update on [http://rally.readthedocs.org/en/latest/ Rally ReadTheDocs page], which will make it much easier to navigate and get through, especially for newbies.
−
* Changing the benchmark scenario ''input config format'';
 
−
* Splitting the already existing validation procedures to different context classes in a logical way;
 
−
* Implementing the ''Context class factory'' (like we did with ''deploy engines'' or ''scenario runners'');
 
−
and many others.
 
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
We are also going to introduce several enhancement both to the task result output (in its HTML form) and to the code (by moving some common code to a special ''utils'' module).
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
−
 
 
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
 
−
 
 
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 184: Line 186:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
−
==== March 3, 2014 ====
+
==== September 1, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
−
the most important changes during the past week have been concentrated on further logical structuring of the core part of Rally, namely the benchmark engine, and include:
+
over the past week, Rally has been extended with the following features:
−
* '''''Benchmark scenario arguments validation refactoring''''': we've moved the whole process from the ''ScenarioRunner'' to the ''BenchmarkEngine'' class (which is a much more logical place for that) and also added the support for admin-based and user-based validation differentiation (https://review.openstack.org/#/c/76162/);
+
* Rally now offers a pretty simple [https://review.openstack.org/#/c/116403/ '''feature request''' mechanism]. Everyone who is interested in adding new functionality to Rally now can share his ideas in a standartized format ([https://github.com/stackforge/rally/blob/master/doc/feature_request/historical_performance_data.rst see an example]). All you need to do is to write down your feature request in a separate ''rst-file'' in the ''doc/feature_request'' folder and submit it as a patch to Rally (if you are unsure about how to do this, read our [[Rally/Develop#How_to_contribute|"How to contribute" tutorial]]).
−
* '''''Context introduction''''', which is a very important novelty for Rally: from now on, we are going to use the notion of ''context'' to define different environments in which benchmark scenarios can be launched by Rally. The already existing temporary ''UserGenerator'' and ''ResourceCleanuper'' classes are in fact also context, so a natural first step was to move them to a special context module (https://review.openstack.org/#/c/77322/);
+
* We added a new benchmark scenario for [https://review.openstack.org/#/c/115929/ '''Cinder quotas''' creation/deletion].
−
* Much work has been done around '''''small fixes in the code''''' (https://review.openstack.org/#/c/77170/, https://review.openstack.org/#/c/77192/) and in the unit tests (https://review.openstack.org/#/c/75877/, https://review.openstack.org/#/c/76884/), as well as around '''''user experience improvements touching the CLI''''' (https://review.openstack.org/#/c/76226/, https://review.openstack.org/#/c/76221/).
+
* Designate support in Rally has been extended so that it is posisble now to set up [https://review.openstack.org/#/c/116852/ '''Designate quotas'''].
  
  
−
The current direction of our work is the further development of '''''contexts''''', which will involve introducing new context classes as well.
+
There is pretty much work going on this week, including the introduction of [https://review.openstack.org/116766 benchmarking with existing users] (in addition to those created temporarily, which was the only option before), and also improvements in the [https://review.openstack.org/103145 plugin mechanism], so that it will be possible now to write plugins for runners and contexts. We do lots of code refactoring and will do even more in the upcoming weeks.
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 212: Line 213:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
−
 
+
==== August 25, 2014 ====
−
==== February 24, 2014 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
−
this week, several important contributions have been made to Rally, considering both the overall system stability and the improvements of the user interface. To name a few:
+
here is a quick overview of what has been added to Rally recently:
−
* '''''Vast refactoring of the ScenarioRunner class''''' has enabled to stop sharing OpenStack clients objects between processes in the core of the system, which occasionally caused bugs in Rally (https://review.openstack.org/#/c/74769/);
+
* Rally now [https://review.openstack.org/#/c/116405/ supports] '''Designate''', which is a ''DNaaS service'' for OpenStack (providing REST API for domain/record management). Several new [https://github.com/stackforge/rally/tree/master/doc/samples/tasks/scenarios/designate bechmark scenarios for Designate] have been added as well;
−
* Another important refactoring step resulted in the '''''replacement of OpenStack endpoint dictionaries with special objects throughout the system''''', which has made the code more reliable and extendable (https://review.openstack.org/#/c/74425/);
+
* In the CLI, we have added a new command called [https://review.openstack.org/102853 '''rally info''']. It is a essentially a special '''search engine''' embedded into Rally, which, for a given search query, prints '''documentation''' for the corresponding benchmark scenario/deploy engine/... as fetched from the source code. Thus you can learn about different Rally entities without leaving the Command Line Interface. For usage samples, see [https://wiki.openstack.org/wiki/Rally/HowTo#Available_Rally_facilities this link];
−
* Perhaps the prettiest patch of the week was '''''the introduction of a benchmark result visualization tool''''', implemented with the ''nvd3'' plugin to ''d3.js'' (so that the actual charts are drawn to a ''html'' file). The graphs look really nice and will be of great use for those who want to share their benchmarking results  (https://review.openstack.org/#/c/72970/);
+
* We have [https://review.openstack.org/#/c/113253/ extended the SLA output] which indicates whether a benchmark has passed some set of predefined success criteria with customizable messages;
−
* Several '''''nice improvements in the CLI''''' include the showing of 90- and 95- percentile results in the benchmark summary (https://review.openstack.org/#/c/73522/) and a new '''show''' command which allows the user to get the information on ''images/flavors/networks/etc.'' available in the current deployment in a very quick way (https://review.openstack.org/#/c/75699/).
+
* We have performed a range of nice '''optimizations''':
−
 
+
:* [https://review.openstack.org/#/c/114839/ Reduced gate jobs runtime];
 +
:* [https://review.openstack.org/#/c/104962/ Code clarification in the context manager];
 +
:* [https://review.openstack.org/#/c/116014/ Gates tests fixing].
  
−
The ongoing work includes:
+
The main work directions for the next week comprise further improvement in the contexts mechanism (both [https://review.openstack.org/116045 the algorithm of context creation] and [https://review.openstack.org/116446 new context classes]), further optimizations (e.g., [https://review.openstack.org/113536 multithreaded objects deletion]) and also code refactoring (e.g., of that for [https://review.openstack.org/111977 tracking atomic actions in benchmark scenarios])
−
* An extention of the '''use''' command which will be applicable soon not only to deployments but also to tasks (https://review.openstack.org/#/c/75936/);
 
−
* Further refactoring of the core benchmark engine, including the work around input configuration parameters validation (for a detailed description of what's going to be done, see [https://docs.google.com/a/mirantis.com/document/d/1LYUAHkZQD8W7dtlj2I3PDA6x67TiD3AMnSWG6ljsups/edit#heading=h.ae5lk415py0q| this special document]);
 
−
* After finishing some major refactoring procedues, we have also resumed the work around passing pre-created user endpoints to the DummyEngine (https://review.openstack.org/#/c/67720/) and generating the ''"stress"'' load on the cloud.
 
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 244: Line 242:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
−
 
+
==== August 4, 2014 ====
−
==== February 17, 2014 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
−
the first thing we would like to mention today is an extensive piece of work that has been done recently on our Wiki. We have updated both the [[Rally|main page]] and the basic tutorials: [[Rally/installation|how to install Rally]] and [[Rally/HowTo|how to use it]]. These tutorials have been simplified quite a lot and have been updated with new features that have been merged in Rally recently.
+
below you will find the most recent updates in Rally:
 +
* The '''periodic scenario runner''' has been [https://review.openstack.org/#/c/102363/ refactored and renamed], as of now, to '''rpc''' (runs per second). Note that this renaming also affects task configuration files that use that runner type. The runner itself now has been reimplemented with the ''multiprocessing.Process'' class (instead of ''multiprocessing.Pool''), which potentially decreases memory usage on large iterations and reduces errors.
 +
* A nice [https://review.openstack.org/#/c/106031/ optimization to the '''chart generation code'''] ensures that the Rally plots will be rendered fast in browsers even for tasks with a huge number of iterations completed.
 +
* '''New benchmark scenarios''' include those testing the [https://review.openstack.org/#/c/109915 Nova server resize] operation and a set of benchmarks for [https://review.openstack.org/#/c/107962 Sahara group node templates].
 +
* We continue '''extending our gates''' with nice features. A new one are the [https://review.openstack.org/#/c/111347/ SLA checks] that add information about whether benchmark scenarios pass a set of success criteria (see an [http://logs.openstack.org/47/111347/5/gate/gate-rally-dsvm-rally/a14f11b/rally-plot/sla.txt example])
  
−
As for the actual updates in the Rally code, the main ones are as follows:
 
−
* '''''The refactoring of the ScenarioRunner class''''' has been successfully continued by reimplementing different benchmark execution strategies (''continuous''/''periodic'' executions), which resided previously inside the original class, via subclassing. The new '''''ContinuousScenarioRunner''''' and '''''PeriodicScenarioRunner''''' classes enable us to make the code much more readable (no more complicated ''if...else'' logic to choose the appropriate execution strategy is present in the code) and extendable, so that it is now very easy to add your own ScenarioRunner (https://review.openstack.org/#/c/70771/);
 
−
* We've successfully started the work on '''''Tempest & Rally integration''''' in order for the latter to be able to perform OpenStack deployment verification procedures as well as to possibly add new Tempest-based benchmark scenarios (https://review.openstack.org/#/c/70131/);
 
−
* On the way to adding a full support for '''''benchmarking with predefined OpenStack users''''' (instead of using the generated ones), we've refactored the '''''Endpoint''''' entity, making it able to distinguish between administrator/user permisisons. Besides, each deployment gets now stored in the database with a list of endpoints instead of only one endpoint, since we are going to enable the DummyEngine to take several endpoints as its input (https://review.openstack.org/#/c/67154/);
 
−
* One of the future features of Rally will be '''''Heat-based benchmark scenarios''''' which will make it possible to test the VMs performance. This week, we have started contributing to this as well (https://review.openstack.org/#/c/72749/);
 
−
* Among many other simplier refactoring patches finished this week, we'd like to mention the one that '''''improves the CLI code''''' by structuring it to submodules (https://review.openstack.org/#/c/73059/).
 
  
 +
Many interesting improvements are on their path to being merged to Rally. Among them, let us mention the [https://review.openstack.org/#/c/102853 "rally info" command], which prints descriptions for different entities in Rally to the console, further work on the [https://review.openstack.org/110738 Rally documentation] which gets cleaner and cleaner, and also a set of new context classes, including those for [https://review.openstack.org/#/c/104564/ generating images] for VM benchmarks, as well as for creating [https://review.openstack.org/#/c/96300/ Neutron networks].
  
−
The ongoing work includes:
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
−
* Further changes needed for the DummyEngine in order to be able to '''''accept predefined user endpoints''''' (instead of a single admin endpoint) and for the ScenarioRunner to use them in benchmarks (https://review.openstack.org/#/c/67643/, https://review.openstack.org/#/c/67710/, https://review.openstack.org/#/c/67720/);
 
−
* New '''''benchmark result visualization tool''''' based on ''nvd3'' plugin to ''d3.js'' (so that the actual charts are drawn to a ''html'' file). This also includes a new CLI command '''''rally task plot2html <task_uuid>''''' (https://review.openstack.org/#/c/72970/);
 
−
* Adding '''''90- and 95-percentile results to the CLI output''''' for benchmark runtimes, i.e. printing the runtime ranges that encompass 90% and 95% of benchmarks respectively (https://review.openstack.org/#/c/73522/);
 
−
* Continuing the work on '''''Rally & Tempest integration''''', new '''''deployment engine types''''' etc.
 
  
−
 
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
−
 
 
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=icehouse&metric=commits&project_type=all&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 280: Line 269:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
−
 
+
==== July 17, 2014 ====
−
==== February 10, 2014 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
−
the past week has been extremely successful for us regarding the overall Rally code improvement, bugfixing, as well as new features implementation. Rally is actually about to become a totally ''easy-to-understand'' and ''easy-to-use'' piece of software which can be exploited by everyone interested in it.
+
here are some recent updates in Rally:
−
 
 
−
The most important contributions to Rally made during the past week are as follows:
 
−
* The code refactoring stuff has been quite involved:
 
−
:* We have issued '''''a drastic rearrangement of the ScenarioRunner class''''' (that is responsible for the actual benchmark method calls using a particular benchmarking strategy) by moving out some code from this class to new context classes. This change also enabled Rally to process all errors occuring on the cloud during benchmarking/cleanup correctly (https://review.openstack.org/#/c/69886/);
 
−
:* Another significant contribution is the '''''sshutils module refactoring''''', which involves the API improvement as well as the new ability to process the stdin data (https://review.openstack.org/#/c/68063/);
 
−
:* Finally, a very nice work has been done on the '''''benchmark scenarios refactoring''''' by moving the hardcoded timeout and cloud poll interval values to rally.conf (https://review.openstack.org/#/c/71272/).
 
−
* Very important '''''bugfixes''''' addressing the improper '''''implementation of OpenStack resource deletion''''' (https://review.openstack.org/#/c/66856/) and '''''benchmark timeout handling''''' (https://review.openstack.org/#/c/72103/) have been merged this week as well;
 
−
* Our set of available benchmark scenarios has been expanded with '''''benchmark scenarios for Glance''''': they include a scenario for ''adding and deleting an image'' and a scenario for ''booting several instances from a previously added image'' (https://review.openstack.org/#/c/60469/).
 
−
 
 
  
 +
* Results of scenario runners [https://review.openstack.org/#/c/104518 are now stored asynchronously]. This is precondition for such further features like piecemeal storing of results to DB (to reduce RAM usage), or progress displaying.
 +
* Add [https://review.openstack.org/#/c/97556 check if required services are available] before starting the scenario. This is realized by adding services validation to scenarios. Also this patch includes new validation mechanism.
 +
* Ability to [https://review.openstack.org/#/c/103145/3 cancel task by keyboard interrupts] is availiable now;
 +
* Add base to provide [https://review.openstack.org/#/c/102899 interation testing];
 +
* A lot of of patches to improve [https://review.openstack.org/#/q/status:merged+project:stackforge/rally+branch:master+topic:bp/improve-unit-test-coverage-rally,n,z unit tests coverage].
  
−
The current work encompasses the following directions:
 
−
* Further '''''scenario runners refactoring''''': we are now reimplementing the different benchmark execution strategies (''continuous''/''periodic'' executions) via subclasses of the base '''''ScenarioRunner''''' class thus making the code much more readable and extendable (https://review.openstack.org/#/c/70771/);
 
−
* Reimplementing the patches for '''''DummyEngine refactoring''''' (making it able to work with a predefined set of users instead of the generated ones) based on the updated scenario runners (https://review.openstack.org/#/c/67154/, https://review.openstack.org/#/c/67643/, https://review.openstack.org/#/c/67710/, https://review.openstack.org/#/c/67720/);
 
−
* '''''Rally & Tempest integration''''' is a new ambitious piece of work we are conducting now. Tempest is going to be used inside Rally as a base for the cloud verification functionality in Rally as well as for new benchmark scenarios that use Tempest (https://review.openstack.org/#/c/70131/).
 
−
* We are putting now much effort in finishing the work on '''''new deploy engines''''': the '''''MultihostEngine''''' (https://review.openstack.org/#/c/57240/), the '''''LxcEngine''''' (https://review.openstack.org/#/c/56222/) and the '''''FuelEngine''''' (https://review.openstack.org/#/c/61963/).
 
  
 +
This week, our work continues with such main novelties as adding support for [https://review.openstack.org/#/c/100579 Keystone API v3], adding context classes for [https://review.openstack.org/#/c/103306 avoiding vm creation if tenant has no network] and [https://review.openstack.org/#/c/104564 prepare an image that will have installed the required programs], mechanism to provide [https://review.openstack.org/#/c/103377 rally as a service], [https://review.openstack.org/#/c/104962 code refactoring in context manager] to make code more clear.
  
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 319: Line 299:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
−
 
+
==== July 07, 2014 ====
−
==== February 03, 2014 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
−
our efforts during the past week were heavily focused on code refactoring and bugfixing. Among the most significant contrubutions are:
+
here is the update for the last weeks. From all the work we've completed we would like to highlight the following:
−
* A fix for certain inconsistencies in the code that checked the availability of resources of OpenStack, e.g. whether a particular resource got deleted or not (https://review.openstack.org/#/c/66856/);
 
−
* The work on refactoring the scenario runner to make its code clean (https://review.openstack.org/#/c/69846/).
 
−
 
 
−
 
 
−
Several novelties have been introduced to Rally:
 
−
* After having developed the abstract '''''validators''''' mechanism, we have developed a couple of useful concrete validators as well: the one that checks that the image indicated in the config for, say, the ''NovaServers.boot_and_delete_server'' benchmark scenario really exists and can be used (https://review.openstack.org/#/c/68055/) and another validator that does the same for flavors (https://review.openstack.org/#/c/70082/). Both validators have been attached to benchmark scenarios where they are of great use.
 
−
* We've implemented the mechanism of measuring the time taken by '''''atomic actions''''' in our benchmark scneario (https://review.openstack.org/#/c/69828/): e.g. now Rally outputs not only the information on how long it took the cloud to boot and delete a single server (in the ''NovaServers.boot_and_delete_server'' scenario), but also how much time it took to boot the server and to delete it.
 
  
 +
* New benchmark scenarios:
 +
:* Benchmark to [https://review.openstack.org/#/c/89326 validate a '''keystone''' token N times at service endpoint] that allows to check effect of caching related to tokens;
 +
:* [https://review.openstack.org/#/c/93739 Scenario for benchmark '''Neutron''' ports].
 +
* [https://review.openstack.org/#/c/89555 Add nova floating ip management in VM scenario], before this patch the VM runcommand scenario used the fixed ip to ssh connect to the instance, this could only worked if fixed ip range was directly accessible with was the case in very limited deployments.
 +
* [https://review.openstack.org/#/c/95353 Add image context class] that allows adding images to each user for benchmarks, also this patch provides scenario which tests glance image-list command;
 +
* [https://review.openstack.org/#/c/99304 Large update of documentation] that simplifies its index structure and headings, extends the 'Benchmark' page with info from the "Main concepts" and adds some introductory text on Deploy & Verify parts;
 +
* [https://review.openstack.org/#/c/98158 Add service-level agreement checking] that allows to add section in task configuration that contains the criteria of success, e.g "less then 5% of failure rate", or "faster then 5 seconds for one iteration";
 +
* [https://review.openstack.org/#/c/103706 Use oslo.db module and remove rally/openstack/common/db].
  
−
This week there is still a huge amount of work to be done around '''''refactoring the very fundamental code in Rally'''''. Among other things, we now rewrite the ''ScenarioRunner'' class which is the tool for launching benchmark scenarios (https://review.openstack.org/#/c/69886/) so that its functionality gets split into several context classes (responsible for temporary users management and resource cleanup after benchmarking), and also implementing different scenario launching strategies via inheritance (https://review.openstack.org/#/c/70771/).
 
  
−
We continue implementing new features in Rally as well. One example is the ongoing work on '''''atomic actions runtime measurement''''': it is about to be supported by the CLI which will now display this detailed runtime information in a user-friendly way (https://review.openstack.org/#/c/70362/).
+
Current work includes such interesting novelties as [https://blueprints.launchpad.net/rally/+spec/benchmark-context-semantic-validation context semantic validation], [https://review.openstack.org/#/c/102363 periodic runner refactoring], [https://review.openstack.org/#/c/102853 "rally info" command] which prints descriptions for different entities in Rally, [https://review.openstack.org/#/c/97556 checking if required services are available before starting the scenario] and refactoring of validation system, continue work on [https://review.openstack.org/94806 "stress" runner].
  
  
−
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
−
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
−
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 353: Line 332:
  
  
−
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
Line 359: Line 338:
  
 
==== Archives ====
 
==== Archives ====
−
* [[Rally/UpdatesFebruary2014|Weekly updates archives - February 2014]]
+
* [[Rally/UpdatesJune2014|Updates archives - June 2014]]
−
* [[Rally/UpdatesJanuary2014|Weekly updates archives - January 2014]]
+
* [[Rally/UpdatesMay2014|Updates archives - May 2014]]
−
* [[Rally/UpdatesDecember2013|Weekly updates archives - December 2013]]
+
* [[Rally/UpdatesApril2014|Updates archives - April 2014]]
−
* [[Rally/UpdatesNovember2013|Weekly updates archives - November 2013]]
+
* [[Rally/UpdatesMarch2014|Updates archives - March 2014]]
 +
* [[Rally/UpdatesFebruary2014|Updates archives - February 2014]]
 +
* [[Rally/UpdatesJanuary2014|Updates archives - January 2014]]
 +
* [[Rally/UpdatesDecember2013|Updates archives - December 2013]]
 +
* [[Rally/UpdatesNovember2013|Updates archives - November 2013]]

Latest revision as of 12:02, 22 January 2015

Updates

January 22, 2014

Hi stackers!

We are happy to announce that we have completely redesigned our Rally documentation in ReadTheDocs. The docs have now received a simpler structure and have become much easier to get through!

One of the nicest new things is the Rally step-by-step tutorial that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.

Since our previous update, there have been many interesting updates in Rally:

  • Rally now has a Network Context class that enables easy Neutron network management.
  • Input task files now can be written using the jinja2-based templates syntax. Very useful if you want, say, parameterize the image name used throughout your complex input task file.
  • Rally scenarios have been 100%-covered with docstrings. That means that now the rally info find <query> command will always output a complete piece information about whatever you ask it.
  • New benchmark scenarios include thos for Cinder (list_volumes, list_snapshots, extend_volume), Nova (cold_migrate)
  • We have moved the directory with samples in our repository to the root level: now it is rally/samples instead of rally/doc/samples (and so much quicker to get to).
  • Rally is on the way to being Python 3 compatible. We have added a Gate job that checks Rally in Python 3 and have produced lots of patches that fix incompability issues. Few changes are left to make Rally fully Python 3 compatible.


Current work includes new benchmark scenarios (Mistral, Murano), new success criteria (SLA) and a lot of refactoring in scenario runners, Rally API etc.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



December 15, 2014

Hi stackers!

Let us share with you our recent accomplishments in Rally:

  • CLI improvements:
  • The rally info command (which is a kind of built-in Rally reference) has been enhanced in such a way that it now prints detailed explanations of main concepts used in Rally whenever you type something like rally info BenchmarkScenarios or rally info SLA. We've also improved the output formatting so that now it is much easier to get through.
  • The rally task list command now supports filters. You can filter the task list either by deployment (using the "... --deployment <deployment_name_or_id>" parameter) or by status ("... --status <status_name>")
  • New benchmark scenarios:
  • Other improvements:


Current work is centered aroud code refactoring (both major, as in the benchmark engine or the contexts, and minor, as introducing some syntax sugar via decorators to mark deprecated stuff and scenario samples). We also constantly work on expanding our scenario base. Last but not least, we're about to merge the Network Context class that enables easy Neutron network management.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



December 1, 2014

Hi stackers!

It's been a while since our last post here, and we've done quite a nice job in Rally during November. Let us share with you new things about Rally:

  • Autogenerated HTML benchmark reports in Rally (which can be created by the "rally task report" command after a benchmark task has completed) have been improved further within the last month. As of now, the report page contains an overview table, detailed informations about whether SLA (service-level agreement) checks were successful and also detailed error logs, if any. Rally reports have become a wonderful tool indeed to analyse the benchmarking data as well as to share your results with others!
  • Similar improvements have been made for HTML reports generated for the Tempest cloud verification ("rally verify results --html --output_file <file>"). New enhanced report pages have improved styling and refactored JS code.
  • We have changed the way context classes in Rally should be declared. Having introduced a new @context decorator, we've made it much easier and also more readable.
  • There is a new "servers" context that allows you to create temporary servers before benchmark scenarios start and use these servers for testing inside these scenarios.
  • New benchmark scenarios in Rally include those for Nova live migrate and also a Cinder stress scenario.
  • Command-line interface improvements include an ability to refer deployments not only by uuid but also by name. Please note that the syntax has changed a bit so now you have to supply the --deployment parameter to commands like "rally use deployment" (instead of --uuid).
  • There has been some major refactoring of the most critical parts of Rally code: the cleanup mechanism and the "users" context code. We are sure that after refactoring, this code has become both cleaner and less error-prone (as well as very pluggable in the case of cleanups).


Current work includes further code refactoring (e.g. in the Benchmark engine part), further CLI improvements (e.g. for the "rally task list" command) and also new benchmark scenarios (e.g. for Murano). We are also going to introduce a possibility of building Rally images for Docker.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



October 27, 2014

Hello stackers,

much time has passed since our last update and we are happy to announce that we are moving towards making our first official Rally release! Our active recent contribution to Rally has enabled us to make a significant progress. Here are the highlights of the novelties in Rally:

  • We have completely redesigned the auto-generated benchmark report page so that it looks now even nicer than before and is much more easy to navigate. Besides, further improvements of this HTML report page are on their way to being merged soon.
  • Rally now has an extended support of plugins: in addition to writing custom scenarios, the plugin mechanism now enables to extend Rally with new context classes/scenario runners without actually contributing to the Rally master branch.
  • The work on extending the support of OpenStack projects in Rally has been conducted for Heat and Sahara.
  • CLI improvements: the command-line interface gets more and more user-friendly over time: recently, it has begun to support detailed informations about correct commands usage in case of a failure. The "rally info" command has also been improved so that it now supports misspellings handling. Finally, there has been some work on bash completion, which hasn't been completely finished yet.
  • Test code improvements: we have greatly proceeded in out continuous work on unit/functional test coverage improvement. We also have moved all the tests into a special tests/ directory so that the test code is now organized in a more neat way. We also have removed the ./run_tests.sh script for the sake of using the tox command to launch the test suite.


In the nearest future, several interesting refactoring patches are going to come to Rally. To be more specific, there will be a vast change of the benchmark engine and cleanup mechanisms.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 26, 2014

Hello stackers,

here is a brief overview of what has happened in Rally recently:

  • We have refactored the code responsible for atomic actions processing in Rally benchmarks. Let's remind you that each benchmark scenario in Rally consists of a series of atomic actions, whereas the running time of each atomic action is measured in the same way as that of the whole scenario. After refactoring, we now ensure that Rally doesn't skip atomic actions that failed and distinguishes different runs of two atomic actions with the same name (see an example of how a typical results table could look before and looks after refactoring).
  • Another direction of refactoring was the SLA code; it has been modified so that now SLA results are stored in the DB along with other benchmarking data.
  • Check out a nice user story about VMs boot performance in Nova with Rally. It shows how Rally can be used in practice to catch reals bugs and improve OpenStack performance.
  • We work hard on achieving 100% test coverage in Rally: last week, additional unit tests for contexts and scenarios have been merged.


Our current priorities are further refactoring steps, including those in critical parts of Rally (e.g. temporary user creation and cloud cleanup code); we also strive towards making Rally bug-free and continuously issue different bugfixing patches.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 15, 2014

Hello stackers,

there has been much diverse and useful contribution to Rally recently. Let us highlight some interesting updates in our project:

  • Rally is on its way to support of benchmarking OpenStack clouds using ordinary user accounts that already exist. Rally lacked such functionality (it only supported benchmarking either from an admin account or from a bunch of temporarily created users), which posed a problem since some deployments don't allow temporary users creation. There have been two patches that prepare the code for this new functionality. It is going to come very soon - stay tuned.
  • We constantly improve our gate jobs (jobs that run a test suite for every patch pending review in Rally). Recently, we have introduced a nice aggregated results page for that gate job (see an example). It now makes it very easy for developers to navigate through the results of tests agains their patches to Rally.
  • We have added a new "volumes" context which makes it possible to create cinder volumes in the benchmark environment and use them later in the actual benchmark scenarios.
  • Much work has been accomplished on the overall code quality improvement. We have both made several code refactoring patches and introduced several new test suites (say, functional tests for CLI).


Current work centers around introducing benchmarking with pre-created users as mentioned above; much work is currently devoted to new benchmark scenarios as well. Also please note that there will be soon an important update on Rally ReadTheDocs page, which will make it much easier to navigate and get through, especially for newbies.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 1, 2014

Hello stackers,

over the past week, Rally has been extended with the following features:

  • Rally now offers a pretty simple feature request mechanism. Everyone who is interested in adding new functionality to Rally now can share his ideas in a standartized format (see an example). All you need to do is to write down your feature request in a separate rst-file in the doc/feature_request folder and submit it as a patch to Rally (if you are unsure about how to do this, read our "How to contribute" tutorial).
  • We added a new benchmark scenario for Cinder quotas creation/deletion.
  • Designate support in Rally has been extended so that it is posisble now to set up Designate quotas.


There is pretty much work going on this week, including the introduction of benchmarking with existing users (in addition to those created temporarily, which was the only option before), and also improvements in the plugin mechanism, so that it will be possible now to write plugins for runners and contexts. We do lots of code refactoring and will do even more in the upcoming weeks.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


August 25, 2014

Hello stackers,

here is a quick overview of what has been added to Rally recently:

  • Rally now supports Designate, which is a DNaaS service for OpenStack (providing REST API for domain/record management). Several new bechmark scenarios for Designate have been added as well;
  • In the CLI, we have added a new command called rally info. It is a essentially a special search engine embedded into Rally, which, for a given search query, prints documentation for the corresponding benchmark scenario/deploy engine/... as fetched from the source code. Thus you can learn about different Rally entities without leaving the Command Line Interface. For usage samples, see this link;
  • We have extended the SLA output which indicates whether a benchmark has passed some set of predefined success criteria with customizable messages;
  • We have performed a range of nice optimizations:

The main work directions for the next week comprise further improvement in the contexts mechanism (both the algorithm of context creation and new context classes), further optimizations (e.g., multithreaded objects deletion) and also code refactoring (e.g., of that for tracking atomic actions in benchmark scenarios)

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


August 4, 2014

Hello stackers,

below you will find the most recent updates in Rally:

  • The periodic scenario runner has been refactored and renamed, as of now, to rpc (runs per second). Note that this renaming also affects task configuration files that use that runner type. The runner itself now has been reimplemented with the multiprocessing.Process class (instead of multiprocessing.Pool), which potentially decreases memory usage on large iterations and reduces errors.
  • A nice optimization to the chart generation code ensures that the Rally plots will be rendered fast in browsers even for tasks with a huge number of iterations completed.
  • New benchmark scenarios include those testing the Nova server resize operation and a set of benchmarks for Sahara group node templates.
  • We continue extending our gates with nice features. A new one are the SLA checks that add information about whether benchmark scenarios pass a set of success criteria (see an example)


Many interesting improvements are on their path to being merged to Rally. Among them, let us mention the "rally info" command, which prints descriptions for different entities in Rally to the console, further work on the Rally documentation which gets cleaner and cleaner, and also a set of new context classes, including those for generating images for VM benchmarks, as well as for creating Neutron networks.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


July 17, 2014

Hello stackers,

here are some recent updates in Rally:


This week, our work continues with such main novelties as adding support for Keystone API v3, adding context classes for avoiding vm creation if tenant has no network and prepare an image that will have installed the required programs, mechanism to provide rally as a service, code refactoring in context manager to make code more clear.


We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


July 07, 2014

Hello stackers,

here is the update for the last weeks. From all the work we've completed we would like to highlight the following:

  • New benchmark scenarios:


Current work includes such interesting novelties as context semantic validation, periodic runner refactoring, "rally info" command which prints descriptions for different entities in Rally, checking if required services are available before starting the scenario and refactoring of validation system, continue work on "stress" runner.


We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


Archives

Retrieved from "https://wiki.openstack.org/w/index.php?title=Rally/Updates&oldid=72196"