Jump to: navigation, search

Difference between revisions of "Rally/Updates"

(Weekly updates: 03.02.14)
(January 22)
 
(85 intermediate revisions by 3 users not shown)
Line 1: Line 1:
== Weekly updates  ==
+
== Updates ==
  
==== February 03, 2014 ====
 
  
Hello stackers,
+
==== January 22, 2014 ====
 +
 
 +
Hi stackers!
 +
 
 +
We are happy to announce that we have completely redesigned our [http://rally.readthedocs.org/en/latest/ Rally documentation in ReadTheDocs]. The docs have now received a simpler structure and have become much easier to get through!
 +
 
 +
One of the nicest new things is the [http://rally.readthedocs.org/en/latest/tutorial.html Rally step-by-step tutorial] that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.
  
our efforts during the past week were mostly concentrated on code refactoring and bugfixing. Among the most significat contrubutions are:
+
Since our previous update, there have been many interesting updates in Rally:
* A fix for certain inconsistencies in the code that checked the availability of resources of OpenStack, e.g. whether a particular resource got deleted or not (https://review.openstack.org/#/c/66856/);
+
* Rally now has a [https://review.openstack.org/103306 Network Context class] that enables easy Neutron network management.
* The work on refactoring the scenario runner to make its code clean (https://review.openstack.org/#/c/69846/).
+
* Input task files now can be written using the [https://review.openstack.org/137716 jinja2-based templates syntax]. Very useful if you want, say, parameterize the image name used throughout your complex input task file.
Several novelties have been introduced to Rally:
+
* Rally scenarios have been [https://review.openstack.org/127192 100%-covered with docstrings]. That means that now the '''rally info find <query>''' command will always output a complete piece information about whatever you ask it.
*After having developed the abstract '''''validators''''' mechanism, we have developed a couple of useful concrete validators as well: the one that checks that the image indicated in the config for, say, the ''NovaServers.boot_and_delete_server'' benchmark scenario really exists and can be used (https://review.openstack.org/#/c/68055/) and another validator that does the same for flavors (https://review.openstack.org/#/c/70082/). Both validators have been attached to benchmark scenarios where they are of great use.
+
* New benchmark scenarios include thos for '''Cinder''' ([https://review.openstack.org/142500 list_volumes], [https://review.openstack.org/146580 list_snapshots], [https://review.openstack.org/147489 extend_volume]), '''Nova''' ([https://review.openstack.org/148307 cold_migrate])
* We've implemented the mechanism of measuring the time taken by '''''atomic actions''''' in our benchmark scneario (https://review.openstack.org/#/c/69828/): e.g. now Rally outputs not only the information on how long it took the cloud to boot and delete a single server (in the ''NovaServers.boot_and_delete_server'' scenario), but also how much time it took to boot the server and to delete it.
+
* We have [https://review.openstack.org/147412 moved] the directory with samples in our repository to the root level: now it is ''rally/samples'' instead of ''rally/doc/samples'' (and so much quicker to get to).
 +
* Rally is on the way to being '''Python 3 compatible'''. We have added a Gate job that checks Rally in Python 3 and have produced [https://review.openstack.org/144706 lots] [https://review.openstack.org/144971 of] [https://review.openstack.org/145450 patches] [https://review.openstack.org/145492 that] [https://review.openstack.org/141163 fix] [https://review.openstack.org/144185 incompability] [https://review.openstack.org/144143 issues]. Few changes are left to make Rally fully Python 3 compatible.
  
  
 +
Current work includes new benchmark scenarios ([https://review.openstack.org/144320 Mistral], [https://review.openstack.org/137650 Murano]), [https://review.openstack.org/148079 new success criteria (SLA)] and a lot of refactoring in scenario runners, Rally API etc.
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
Stay tuned.
+
Stay tuned!
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
==== January 27, 2014 ====
 
 
Hello stackers,
 
  
we are happy to share our recent updates in Rally:
 
* We've done a very nice job on '''improving the command line interface''' for Rally. This has been done through several separate relatively small enhancements, which have resulted together in an overall much more positive user experience while using Rally. These improvement comprise:
 
:* The '''''rally use deploy''''' command which allows to specify the deployment we are working with only once and thus there is no need to write down the long deployment id string every time we want to launch a benchmarking task(https://review.openstack.org/#/c/68395/);
 
:* The ability to create a deployment from the environment variables, if they are specified (https://review.openstack.org/#/c/68347/);
 
:* More useful deployment-related output from CLI after a deployment has been created (https://review.openstack.org/#/c/68766/);
 
:* The '''''deployment check''''' subcommand which verifies that keystone endpoints are reachable and prints user-friendly information on that (https://review.openstack.org/#/c/68901/).
 
* A new feature, namely the '''validators''', has been added to Rally. The validators are essentially checker methods that can be bound to different benchmark scenarios and are called by Rally before the actual benchmarking starts to check whether the resources needed by that benchmark are available etc. (https://review.openstack.org/#/c/67157/);
 
* There has been a huge work on '''refactoring the benchmark engine code''' to improve its quality and make it more object-oriented (https://review.openstack.org/#/c/68593/). This work is going to be continued during the next weeks;
 
* The '''FUEL client''' for Rally is now ready to use (https://review.openstack.org/#/c/59943/). This work is going to be followed by the '''''FUEL deploy engine''''', which is currently in progress (https://review.openstack.org/#/c/61963/).
 
  
 +
==== December 15, 2014 ====
  
 +
Hi stackers!
  
The basic plan for this week consists of the following tasks:
+
Let us share with you our recent accomplishments in Rally:
* We are about to finish the implementation of '''''benchmark launching with predefined users''''' (instead of the temporary ones generated automatically by Rally). This work laos includes a set of changes in the '''''Dummy deploy engine''''', which now is going to accept as its input not the single set of endpoints with administrator permission, but a list of endpoints that can be all with ordinary user permissions - in that case, these endpoints will be used instead of temporary users during benchmarking (https://review.openstack.org/#/c/67154/, https://review.openstack.org/#/c/67643/, https://review.openstack.org/#/c/67710/, https://review.openstack.org/#/c/67720/);
+
* '''CLI improvements:'''
* The work on the '''''stress execution''''' of benchmarks is going to be completed as well. It will, however, differ a little from the originally planned one: instead of creating a separate execution option in the input config, we will just extend the already existing ''continuous'' and ''periodic'' ones (https://review.openstack.org/#/c/63055/).
+
:* The '''''rally info''''' command (which is a kind of built-in Rally reference) has been [https://review.openstack.org/#/c/131005/ enhanced] in such a way that it now prints detailed explanations of main concepts used in Rally whenever you type something like '''''rally info BenchmarkScenarios''''' or '''''rally info SLA'''''. We've also improved the output formatting so that now it is much easier to get through.
 +
:* The '''''rally task list''''' command now [https://review.openstack.org/131005 supports] '''filters'''. You can filter the task list either by deployment (using the ''"... --deployment <deployment_name_or_id>"'' parameter) or by status (''"... --status <status_name>"'')
 +
* '''New benchmark scenarios:'''
 +
:* [https://review.openstack.org/#/c/129282 '''''NovaSecGroup.boot_and_delete_server_with_secgroups''''']: creates a number of Nova security groups with rules, then creates a Neutron network with one subnet, finally boots a VM with created security groups, lists the created resources and performs cleanup.
 +
:* [https://review.openstack.org/#/c/139019/ '''''CinderVolumes.create_nested_snapshots_and_attach_volume''''']: creates a volume, its snapshot, and then (recursively) a volume from that snapshot. The recursion depth can be set by the user through the ''--nested_level'' parameter.
 +
* '''Other improvements''':
 +
:* Work on '''Trove support in Rally''' has been started with the [https://review.openstack.org/139047 integration of its client];
 +
:* New [https://review.openstack.org/139022 '''hacking rules'''] are there to provide better codestyle throughout Rally.
  
  
 +
Current work is centered aroud code refactoring (both major, as in the '''benchmark engine''' or the [https://review.openstack.org/139987 '''contexts'''], and minor, as introducing some syntax sugar via '''decorators''' to mark [https://review.openstack.org/138489 deprecated stuff] and [https://review.openstack.org/140033 scenario samples]). We also constantly work on expanding our [https://review.openstack.org/141672 scenario] [https://review.openstack.org/136922 base]. Last but not least, we're about to merge the [https://review.openstack.org/103306 Network Context class] that enables easy Neutron network management.
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
Stay tuned.
+
Stay tuned!
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
==== January 20, 2014 ====
 
  
Hello stackers,
 
  
here are the updates in our project that deserve to be mentioned in the first place:
+
==== December 1, 2014 ====
* Rally now supports the '''automated documentation generation''' using Sphinx. The docs are generated from the docsrings in the code (https://review.openstack.org/#/c/66092/);
+
 
* It is now possible to perform '''Rally installation from DevStack''' (https://review.openstack.org/#/c/65765/). This has been implemented using the so-called ''devstack extras'';
+
Hi stackers!
* We continue to put lots of effort in '''refactoring both the source code of Rally and its unit tests'''. During the past week, we have improved the workflow of the cloud endpoints returned by deploy engines in our system (https://review.openstack.org/#/c/66277/, https://review.openstack.org/#/c/67276/) and improved the code readability for the deploy engine unit tests as well (https://review.openstack.org/#/c/67275/, https://review.openstack.org/#/c/66838/).
 
* Last but not least, certain work has been accomplished concerning the '''Rally CLI''': we've enhanced the CLI output while working with cloud deployments (https://review.openstack.org/#/c/66314/) and fixed a bug related to the ''"task detailed"'' command (https://review.openstack.org/#/c/67830/).
 
  
 +
It's been a while since our last post here, and we've done quite a nice job in Rally during November. Let us share with you new things about Rally:
 +
* '''Autogenerated HTML benchmark reports''' in Rally (which can be created by the '''''"rally task report"''''' command after a benchmark task has completed) have [https://review.openstack.org/#/c/131844/ been] [https://review.openstack.org/#/c/136435/ improved] further within the last month. As of now, the [http://logs.openstack.org/05/131005/28/check/gate-rally-dsvm-rally/f8f3da9/rally-plot/results.html.gz report page] contains an overview table, detailed informations about whether [http://logs.openstack.org/05/131005/28/check/gate-rally-dsvm-rally/f8f3da9/rally-plot/results.html.gz#/Authenticate.validate_cinder SLA (service-level agreement) checks] were successful and also detailed error logs, if any. Rally reports have become a wonderful tool indeed to analyse the benchmarking data as well as to share your results with others!
 +
* Similar improvements have been made for HTML reports generated for the '''Tempest cloud verification''' ('''''"rally verify results --html --output_file <file>"'''''). New [https://review.openstack.org/#/c/135232/ enhanced] report pages have improved styling and refactored JS code.
 +
* We have [https://review.openstack.org/#/c/137502/ changed] the way '''context classes''' in Rally should be declared. Having introduced a new '''''@context''''' decorator, we've made it much easier and also more readable.
 +
* There is a new '''[https://review.openstack.org/127392 "servers" context]''' that allows you to create temporary servers before benchmark scenarios start and use these servers for testing inside these scenarios.
 +
* '''New benchmark scenarios''' in Rally include those for '''[https://review.openstack.org/128631 Nova live migrate]''' and also a '''[https://review.openstack.org/127392 Cinder stress scenario]'''.
 +
* '''Command-line interface improvements''' include an ability to [https://review.openstack.org/131463 refer deployments not only by uuid but also by name]. Please note that the syntax has changed a bit so now you have to supply the ''--deployment'' parameter to commands like ''"rally use deployment"'' (instead of ''--uuid'').
 +
* There has been some '''major refactoring''' of the most critical parts of Rally code: the [https://review.openstack.org/129060 cleanup mechanism] and the [https://review.openstack.org/119297 "users" context code]. We are sure that after refactoring, this code has become both cleaner and less error-prone (as well as very pluggable in the case of cleanups).
  
The main directions of our current work are as follows:
 
* Providing Rally with a '''REST API''': an extensive work is being conducted both on the server side (https://review.openstack.org/#/c/66788/, https://review.openstack.org/#/c/67346/) and on the ''python-rallyclient'' (https://review.openstack.org/#/c/66919/);
 
* Making Rally able to work with a '''predefined set of users''' while launching benchmarking scenarios; this is particularly important when the user does not want to pass the admin user credentials to Rally, but has a set of predefined users which he or she would like to use for benchmarking. Several patches that implement this are ready for review (for reference, see https://blueprints.launchpad.net/rally/+spec/benchmarking-with-predefined-users)
 
* Enhancing Rally with the so-called '''validators''' - special methods checking that certain conditions hold for the cloud before starting benchmarking it. These validations can include the checks for availability of certain resources in the cloud, making sure that the volumes are of the desired size etc. We aim to support fully customizable user-defined validators (https://review.openstack.org/#/c/67157/).
 
  
 +
Current work includes further code refactoring (e.g. in the ''Benchmark engine'' part), further CLI improvements (e.g. for the [https://review.openstack.org/131005 "rally task list" command]) and also new benchmark scenarios (e.g. for [https://review.openstack.org/137661 Murano]). We are also going to introduce a possibility of building Rally images for [https://review.openstack.org/#/c/132556/ Docker].
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
Stay tuned.
+
Stay tuned!
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
==== January 13, 2014 ====
+
 
 +
 
 +
==== October 27, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
we've recovered from the New Year holidays and already accomplished a range a tasks. Here are some recent updates in Rally:
+
much time has passed since our last update and we are happy to announce that we are moving towards making our first official Rally release! Our active recent contribution to Rally has enabled us to make a significant progress. Here are the highlights of the novelties in Rally:
* '''Benchmark scenarios for Cinder''' have been added to Rally. Those include ''creating/deleting volumes'', as well as just ''volume creation'': recall that those volumes will be deleted anyway at the end of benchmarking by the benchmark engine cleanup mechanism (https://review.openstack.org/#/c/61833/);
+
* We have completely '''[https://review.openstack.org/125119 redesigned] the auto-generated benchmark report page''' so that it looks now even nicer than before and is much more easy to navigate. Besides, further improvements of this HTML report page are on their way to being merged soon.
* '''Benchmark scenarios for Keystone''' have been added to Rally. Those include ''creating/deleting users'', as well as just ''users creation'' (https://review.openstack.org/#/c/64329/). The Keystone cleanup mechanism has been implemented as well (https://review.openstack.org/#/c/64220/);
+
* Rally now has an extended support of '''[https://review.openstack.org/103145 plugins]''': in addition to writing custom scenarios, the plugin mechanism now enables to extend Rally with new context classes/scenario runners without actually contributing to the Rally master branch.
* '''Refactoring''' of both the source code (DummyProvider: https://review.openstack.org/#/c/62934/, Benchmark engine: https://review.openstack.org/#/c/64131/) and the unit tests (tests for Nova benchmark scenarios: https://review.openstack.org/#/c/64294/, removing duplicate tests for benchmark engine: https://review.openstack.org/#/c/64128/);
+
* The work on extending the support of OpenStack projects in Rally has been conducted for '''[https://review.openstack.org/128874 Heat]''' and '''[https://review.openstack.org/126900 Sahara]'''.
* Improving the CLI output: modifying the outlook of tables (https://review.openstack.org/#/c/62840/) and performing somewhat more consistent logging (https://review.openstack.org/#/c/64446/).
+
* '''CLI improvements''': the command-line interface gets more and more user-friendly over time: recently, it has begun to support [https://review.openstack.org/124910 detailed] informations about correct commands usage in case of a failure. The ''"rally info"'' command has also been improved so that it now supports '''[https://review.openstack.org/125238 misspellings handling]'''. Finally, there has been some work on '''[https://review.openstack.org/129306 bash completion]''', which hasn't been completely finished yet.
 +
* '''Test code improvements''': we have greatly proceeded in out continuous work on unit/functional test coverage improvement. We also have [https://review.openstack.org/126379 moved] all the tests into a special ''tests/'' directory so that the test code is now organized in a more neat way. We also have removed the ''./run_tests.sh'' script for the sake of using the ''tox'' command to launch the test suite.
  
  
This week we will start implementing the '''REST API''' for Rally, as well as a '''python client''' for it. Besides, there is still much work to do regarding the new '''deploy engines''' and '''server providers''' mentioned in previous reports. We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
In the nearest future, several interesting refactoring patches are going to come to Rally. To be more specific, there will be a vast change of the benchmark engine and cleanup mechanisms.
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
 
 +
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
 +
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 119: Line 130:
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
==== December 23, 2013 ====
+
==== September 26, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
here is the update for the last week. From all the work we've completed this week we would like to highlight the following:
+
here is a brief overview of what has happened in Rally recently:
* A new execution type, namely the '''periodic execution type''' has been added to the benchmark engine (https://review.openstack.org/#/c/57628/). Benchmark engine is now possible to launch a given benchmark scenario once in a specified period of time, thus creating a load which closely resembles the real world load scenarios. For example, you can now ask Rally to lanch the ''"boot and delete server"'' scenario 50 times, booting a new server every 3 minutes. This requires only slight changes in the input configuration file;
+
* We have [https://review.openstack.org/111977 refactored] the code responsible for '''atomic actions processing''' in Rally benchmarks. Let's remind you that each benchmark scenario in Rally consists of a series of atomic actions, whereas the running time of each atomic action is measured in the same way as that of the whole scenario. After refactoring, we now ensure that Rally '''doesn't skip atomic actions that failed''' and '''distinguishes different runs of two atomic actions with the same name''' (see an example of how a typical results table could look [http://paste.openstack.org/show/114111/ before] and looks [http://paste.openstack.org/show/114112/ after] refactoring).
* We've started the work on '''replacing the old FUEL/OSTF-based cloud verification mechanism with a new one, based on Tempest'''. While patches involving Tempest integration are still in progress, we've already get rid of all the FUEL/OSTF stuff in Rally (https://review.openstack.org/#/c/63653/), which has been both a great code cleanup for our project and also has reduced the amount of requirements for Rally.
+
* Another direction of refactoring was the [https://review.openstack.org/118243 SLA code]; it has been modified so that now SLA results are '''stored in the DB''' along with other benchmarking data.
 
+
* Check out a nice [https://github.com/stackforge/rally/blob/master/doc/user_stories/nova/boot_server.rst user story] about '''VMs boot performance''' in Nova with Rally. It shows how Rally can be used in practice to catch reals bugs and improve OpenStack performance.
 
+
* We work hard on achieving '''100% test coverage''' in Rally: last week, additional unit tests for [https://review.openstack.org/122127 contexts] and [https://review.openstack.org/122729 scenarios] have been merged.
  
Our current work is concentrated on:
 
* Adding another execution type to the benchmark engine: '''the stress execution type''', which enables the user to easily specify a benchmark scenario with automatically increasing load of active users (say, from 10 to 100 with step 5). This benchmark scenario will also automatically halt as soon as the cloud starts to fail too frequently: the corresponding ''maximal failure rate'' can be also set in the input configuration file (https://review.openstack.org/#/c/63055/);
 
* Further creation of '''new benchmark scenarios''' for Rally. The most interesting scenario during the past week was presumably the one that boots a server and then allows the user to run a ''custom script'' on that server via SSH (https://review.openstack.org/#/c/63138/).
 
  
 +
Our current priorities are further '''refactoring''' steps, including those in critical parts of Rally (e.g. [https://review.openstack.org/119297 temporary user creation] and [https://review.openstack.org/116269 cloud cleanup] code); we also strive towards making Rally bug-free and continuously issue different bugfixing patches.
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 150: Line 158:
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
==== December 16, 2013 ====
+
==== September 15, 2014 ====
  
 
Hello stackers,
 
Hello stackers,
  
this week has been very fruitful for Rally and below we share with you some of our most important recent results:
+
there has been much diverse and useful contribution to Rally recently. Let us highlight some interesting updates in our project:
* '''Deployment & Benchmark Workflows have now become completely separate things.''' While previously you usually created/specified an OpenStack deployment and also which benchmarks to run on it in one bulk, now Rally requires you first to create a deployment and then reference this deployment while launching benchmark scenarios (https://review.openstack.org/#/c/57455/). This, among other things, allows you to re-use a single deployment for many benchmarking tasks. You are highly encouraged to check out our updated '''[[Rally/HowTo|"How-to"]]''' page where the process of managing deployments and using them in benchmarking tasks is explained in more details;
+
* Rally is on its way to support of '''benchmarking OpenStack clouds using ordinary user accounts that already exist'''. Rally lacked such functionality (it only supported benchmarking either from an admin account or from a bunch of temporarily created users), which posed a problem since some deployments don't allow temporary users creation. There have been [https://review.openstack.org/#/c/116766/ two] [https://review.openstack.org/#/c/119344/ patches] that prepare the code for this new functionality. It is going to come very soon - stay tuned.
* '''Support of resource tracking has been added to LXC server provider'''. This was the last ''server provider'' that didn't support the resource tracking functionality implemented during the previous week and by this patch (https://review.openstack.org/#/c/60930/) we finish the integration of that functionality to the ''server providers'';
+
* We constantly improve our '''gate jobs''' (jobs that run a test suite for every patch pending review in Rally). Recently, we have [https://review.openstack.org/#/c/119584/ introduced] a nice aggregated results page for that gate job (see an [http://logs.openstack.org/84/119584/13/gate/gate-rally-dsvm-rally/d30c028/ example]). It now makes it very easy for developers to navigate through the results of tests agains their patches to Rally.
* '''Adding input config vaildation to several deployment engines and server providers:''' after we've implemented the common config validation procedure last week, the processing of input configuration for deployment engines and server providers has mostly become a matter of writing correct JSON schemas that reflect engine-specific things. Recently, we have merged such schemas for devstack engine (https://review.openstack.org/#/c/57226/), dummy engine (https://review.openstack.org/#/c/57239/) and OpenStack server provider (https://review.openstack.org/#/c/60275/);
+
* We have [https://review.openstack.org/116446 added] a new '''"volumes" context''' which makes it possible to create cinder volumes in the benchmark environment and use them later in the actual benchmark scenarios.
* '''New benchmark scenarios for Nova API.''' We are proud to see that our community starts to grow faster and new interested people come in. A contribution of one of our newcomers (''QianLin'' from ''Huawei'') is a benchmark scenario that exploits Nova server rescue/unrescue API (https://review.openstack.org/#/c/61688/).
+
* Much work has been accomplished on the overall code quality improvement. We have both made several code refactoring patches and introduced several new test suites (say, [https://review.openstack.org/118714 functional tests for CLI]).
 
 
 
 
  
The working plan for this week encompasses:
 
* Adding more diverse '''benchmark scenarios''' to Rally:
 
:* Benchmark scenarios for '''''Nova servers metadata''''' (https://review.openstack.org/#/c/50588/);
 
:* Benchmark scenarios for '''''Cinder''''' (https://review.openstack.org/#/c/61833/);
 
:* Benchmark scenarios for '''''Glance''''' (https://review.openstack.org/#/c/60469/).
 
* Adding out-of-the-box support for '''''stress testing''''': enhancing the benchmarking engine of Rally with the ability to automatically stop when too many benchmarks start to fail. This is often the case when a significant number of benchmark scenarios (i.e. stress testing) is launched on one cloud. This will also require slight changes in the input config format;
 
* Further work on '''deploy engines''': the high-priority work is to finish the implementation of '''''FUEL''''' (https://review.openstack.org/#/c/61963/) and '''''multihost''''' (https://review.openstack.org/#/c/57240/) deploy engines.
 
* '''Code refactoring''', which is this week concentrated on '''''unit tests''''': the goal is to move certain ''"Fake"'' classes commonly used for testing to a special utils-like module (https://review.openstack.org/#/c/62191/), to avoid code duplicate while using these Fake objects (https://review.openstack.org/#/c/62193/) and also to ensure the correct ''decorator'' syntax is used for mocking, which is still not the case for many unit tests.
 
  
 +
Current work centers around introducing benchmarking with pre-created users as mentioned above; much work is currently devoted to new benchmark scenarios as well. Also please note that there will be soon an important update on [http://rally.readthedocs.org/en/latest/ Rally ReadTheDocs page], which will make it much easier to navigate and get through, especially for newbies.
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
 
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
 
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
 
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 188: Line 186:
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
  
 +
==== September 1, 2014 ====
  
==== December 9, 2013 ====
+
Hello stackers,
  
Hello stackers,
+
over the past week, Rally has been extended with the following features:
 +
* Rally now offers a pretty simple [https://review.openstack.org/#/c/116403/ '''feature request''' mechanism]. Everyone who is interested in adding new functionality to Rally now can share his ideas in a standartized format ([https://github.com/stackforge/rally/blob/master/doc/feature_request/historical_performance_data.rst see an example]). All you need to do is to write down your feature request in a separate ''rst-file'' in the ''doc/feature_request'' folder and submit it as a patch to Rally (if you are unsure about how to do this, read our [[Rally/Develop#How_to_contribute|"How to contribute" tutorial]]).
 +
* We added a new benchmark scenario for [https://review.openstack.org/#/c/115929/ '''Cinder quotas''' creation/deletion].
 +
* Designate support in Rally has been extended so that it is posisble now to set up [https://review.openstack.org/#/c/116852/ '''Designate quotas'''].
  
There has been much activity during the past week in Rally, and several significant patches have been merged recently:
 
*  '''Splitting Deploy's & Benchmark's Workflows is coming to the end!'''
 
The last blocker was that we were storing allocated Resources by '''''Server Providers''''' in memory of Rally process instead of permanent storage (e.g. DB). During this week we added new table '''''Resource''''' to DB and switched almost all (except LXC) '''''Server Providers''''' to use DB instead of in-memory storage. Now we should switch LXC provider and then we will be able to merge the final patch addressing the splitting task (https://review.openstack.org/#/c/57455/).
 
  
* '''Generic cloud cleanup after benchmark scenarios.'''
+
There is pretty much work going on this week, including the introduction of [https://review.openstack.org/116766 benchmarking with existing users] (in addition to those created temporarily, which was the only option before), and also improvements in the [https://review.openstack.org/103145 plugin mechanism], so that it will be possible now to write plugins for runners and contexts. We do lots of code refactoring and will do even more in the upcoming weeks.
Performing a '''''generic cleanup''''' after launching benchmark scenarios is essential for guaranteeing the cloud to stay clean after benchmarking and, besides, enables the benchmark scenario writers not to worry about deleting all the resources they create in ''init()'' methods or specific benchmark procedures (https://review.openstack.org/#/c/55695/). "Generic" means that Rally should free all possible kinds of allocated resources: servers, images etc.
 
  
* '''Code refactoring:'''
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
# Fixing a structure issue in the folder with Rally configuration samples (https://review.openstack.org/#/c/59259/);
 
# Renaming the '''''ServerDTO''''' ->  '''''Server''''' entity (used by ''server providers'') to improve code readability (https://review.openstack.org/#/c/59749/).
 
  
 +
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
 +
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 +
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
A wide variety of new contributions to Rally is still under development and pending for review:
 
* '''Enriching the benchmark engine with the mechanism for cloud ''initialization'' before launching benchmark scenarios.''' The support for '''''init()''''' methods in benchmark scenario classes was actually already implemented in Rally 0.1 but has been broken since creating multiple temporary tenants/users for benchmarking had been introduced to Rally (due to the fact that resources - servers, floating IPs etc. - created in ''init()'' did no longer belong to appropriate temporary tenants/users and thus could not be used in benchmark scenarios). There is now a patch (https://review.openstack.org/#/c/59782/) that fixes this issue by calling ''init()'' once for each temporary user and thus creating the appropriate resources (servers, floating IPs etc.) for every temporary OpenStack user that may be involved in benchmarking. This patch also has as a consequence a couple of smaller patches that improve the performance of OpenStack python clients (https://review.openstack.org/#/c/59780/, https://review.openstack.org/#/c/59781/) and thus optimize the procedure of creating resources for temporary users in ''init()'' methods;
 
* '''Glance benchmark scenarios''': while all the previous benchmark scenarios were focused on capabilities provided by Nova API, this patch makes the first contribution for benchmarking other core OpenStack Projects. The patch (https://review.openstack.org/#/c/60469/) currently implements 2 basic scenarios: creating/deleting images and also creating images and using them to boot servers;
 
* '''Code refactoring''': there are currently two patches dedicated to ''deploy engine'' and ''server provider'' code unification; both implement logic for input configuration file validation common for all the deploy engines and server providers respectively (https://review.openstack.org/#/c/57222/, https://review.openstack.org/#/c/60030/). These patches are the followed by the ones concentrated on deploy engine- and server provider-specific things for config validation (https://review.openstack.org/#/c/57239/, https://review.openstack.org/#/c/57226/, https://review.openstack.org/#/c/60275/)
 
  
 +
Stay tuned.
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
 
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Regards,<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
The Rally team
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
+
 
 +
 
 +
==== August 25, 2014 ====
  
 +
Hello stackers,
  
Stay tuned.  
+
here is a quick overview of what has been added to Rally recently:
 +
* Rally now [https://review.openstack.org/#/c/116405/ supports] '''Designate''', which is a ''DNaaS service'' for OpenStack (providing REST API for domain/record management). Several new [https://github.com/stackforge/rally/tree/master/doc/samples/tasks/scenarios/designate bechmark scenarios for Designate] have been added as well;
 +
* In the CLI, we have added a new command called [https://review.openstack.org/102853 '''rally info''']. It is a essentially a special '''search engine''' embedded into Rally, which, for a given search query, prints '''documentation''' for the corresponding benchmark scenario/deploy engine/... as fetched from the source code. Thus you can learn about different Rally entities without leaving the Command Line Interface. For usage samples, see [https://wiki.openstack.org/wiki/Rally/HowTo#Available_Rally_facilities this link];
 +
* We have [https://review.openstack.org/#/c/113253/ extended the SLA output] which indicates whether a benchmark has passed some set of predefined success criteria with customizable messages;
 +
* We have performed a range of nice '''optimizations''':
 +
:* [https://review.openstack.org/#/c/114839/ Reduced gate jobs runtime];
 +
:* [https://review.openstack.org/#/c/104962/ Code clarification in the context manager];
 +
:* [https://review.openstack.org/#/c/116014/ Gates tests fixing].
  
 +
The main work directions for the next week comprise further improvement in the contexts mechanism (both [https://review.openstack.org/116045 the algorithm of context creation] and [https://review.openstack.org/116446 new context classes]), further optimizations (e.g., [https://review.openstack.org/113536 multithreaded objects deletion]) and also code refactoring (e.g., of that for [https://review.openstack.org/111977 tracking atomic actions in benchmark scenarios])
  
Regards,<br />
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
The Rally team
 
  
 +
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
 +
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 +
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
  
 +
Stay tuned.
  
==== December 2, 2013 ====
 
  
Hello stackers,
+
Regards,<br/>
 +
The Rally team
  
below you will find the latest review of our activities in Rally for the past week.
 
  
Our achievements for the end of November comprise:
+
==== August 4, 2014 ====
* Numerous changes in the '''benchmark engine''', the most important among which are:
 
:* Rally is now able not only to perform a specified number of benchmark scenario launches, but also to create a ''continuous load'' on the cloud by running any scenario for the given period of time. For example, you can now boot and delete servers in the cloud continiously from a number of temporary users, say, for 10 minutes, thus simulating in this way a stress load on the cloud. To do so, the only thing you should change in your configuration file is the ''"times"'' field for the benchmark scenario you are going to launch, which now should be replaced with ''"duration"'' field and initialized to the number of minutes the benchmark is expected to run (https://review.openstack.org/#/c/56036/);
 
:* Access to openstack clients with ''administrator permissions'' is now enabled for all the scenarios through the ''admin_clients()'' method of the base ''Scenario'' class. Before this update, this class provided only the ''clients()'' method which returned a reference to a non-admin OpenStack client. This, however, turned out to be not enough for ''keystone''-based benchmark scenarios that are to come in the future releases (https://review.openstack.org/#/c/58381/);
 
:* Bugfix for the ''init()'' methods of benchmark scenarios which now enables benchmark scenario writers to pass through the ''context'' dictionary (which is the dictionary that ''init()'' returns) not only primitive objects like strings or numbers but also the complex ones like references to specially prepared servers or floating ips (https://review.openstack.org/#/c/55465/).
 
* The work on '''separating the ''deployment'' and ''task'' entities''' mentioned in previous updates has now come closely to its successful conclusion. The main results here include:
 
:* The restructurization of the orchestrator workflow that now makes usage of deployment ''make_deploy()'' and ''make_cleanup()'' functions (https://review.openstack.org/#/c/57057/);
 
;* Adding CLI commands for the ''deployment'' entity: ''create'', ''destroy'', ''list'' ''recreate'' etc. (https://review.openstack.org/#/c/56226/);
 
* '''Server provider for OpenStack''': another ''ServerProvider'' class that wraps with the default ServerProvider interface (''create_vms()'' and ''destroy_vms()'' methods) the functionality of ''python-novaclient''. Along with ''lxc'' and ''virsh'' server providers (already present in the system) it constitutes the essential basis for working with different virtualization technologies (https://review.openstack.org/#/c/48811);
 
* The first contribution to '''data processing and visualization''' in Rally: a new CLI command for tasks has been added, namely ''plot aggregated'' which draws plots illustrating the cloud performance on one of the finished benchmark tasks. The CLI command requires the user to indicate the parameter for which the plots will be drawn. For example, if one specifies ''active_users'' as the aggregating parameter, Rally will draw a plot that shows how the number of active users making requests to the cloud affects the runtime of benchmark scenarios. The code uses the ''matplotlib'' library to draw the plots (https://review.openstack.org/#/c/52712/).
 
  
 +
Hello stackers,
  
This week, our work will be concentrated on the following:
+
below you will find the most recent updates in Rally:
 +
* The '''periodic scenario runner''' has been [https://review.openstack.org/#/c/102363/ refactored and renamed], as of now, to '''rpc''' (runs per second). Note that this renaming also affects task configuration files that use that runner type. The runner itself now has been reimplemented with the ''multiprocessing.Process'' class (instead of ''multiprocessing.Pool''), which potentially decreases memory usage on large iterations and reduces errors.
 +
* A nice [https://review.openstack.org/#/c/106031/ optimization to the '''chart generation code'''] ensures that the Rally plots will be rendered fast in browsers even for tasks with a huge number of iterations completed.
 +
* '''New benchmark scenarios''' include those testing the [https://review.openstack.org/#/c/109915 Nova server resize] operation and a set of benchmarks for [https://review.openstack.org/#/c/107962 Sahara group node templates].
 +
* We continue '''extending our gates''' with nice features. A new one are the [https://review.openstack.org/#/c/111347/ SLA checks] that add information about whether benchmark scenarios pass a set of success criteria (see an [http://logs.openstack.org/47/111347/5/gate/gate-rally-dsvm-rally/a14f11b/rally-plot/sla.txt example])
  
* Further '''enhancements in the ''benchmark engine''''': adding facilities for periodic benchmark execution (https://review.openstack.org/#/c/57628/) and enabling the ''init()'' methods of benchmark scenarios to allocate resources like servers or floating ips using OpenStack clients for temporary users (so that these resources can be used further in the bodies of benchmark methods). We are also about to start the work on parallel benchmark execution which will enable Rally to simulate "noise" load on the cloud;
 
* Implementing '''''LXC and multihost OpenStack deployment facilities''''' (https://review.openstack.org/#/c/56222/, https://review.openstack.org/#/c/57240/). Another improvement related to Rally deployment engines is the effort to unify the input configuration file validation for all the engine types (https://review.openstack.org/#/c/57222/);
 
* Finishing the work on '''separating ''deployments'' and ''tasks''''' (https://review.openstack.org/#/c/57455).
 
  
 +
Many interesting improvements are on their path to being merged to Rally. Among them, let us mention the [https://review.openstack.org/#/c/102853 "rally info" command], which prints descriptions for different entities in Rally to the console, further work on the [https://review.openstack.org/110738 Rally documentation] which gets cleaner and cleaner, and also a set of new context classes, including those for [https://review.openstack.org/#/c/104564/ generating images] for VM benchmarks, as well as for creating [https://review.openstack.org/#/c/96300/ Neutron networks].
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 267: Line 269:
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
 
+
==== July 17, 2014 ====
 
 
==== November 25, 2013 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
here is the second report on our activities in Rally development for the past week.
+
here are some recent updates in Rally:
  
The main results that have been recently merged with master are as follows:
+
* Results of scenario runners [https://review.openstack.org/#/c/104518 are now stored asynchronously]. This is precondition for such further features like piecemeal storing of results to DB (to reduce RAM usage), or progress displaying.
* Changes in the ''benchmark engine'': we have '''significantly restructured the format of the input benchmark config''' (https://review.openstack.org/#/c/56035/). The changes make it more transparent to the end-user as well as more flexible. This will enable us to implement new features in the benchmark engine like running tests periodically or for a given amount of time. We have also refactored the test code related to benchmark scenarios by replacing ugly-looking nested with-blocks for mocks with a more readable decorator syntax (https://review.openstack.org/#/c/57732/);
+
* Add [https://review.openstack.org/#/c/97556 check if required services are available] before starting the scenario. This is realized by adding services validation to scenarios. Also this patch includes new validation mechanism.
* Further work on splitting the system logic between the two basic entities, namely the ''deployment'' and the ''benchmark task''. While still having the legacy combined config that contains information both on the deployment and on the benchmarks, we have closely come to the point where we can completely split everything related to these two entites. To be more precise, during the lask week we have made:
+
* Ability to [https://review.openstack.org/#/c/103145/3 cancel task by keyboard interrupts] is availiable now;
:* the integration of the ''deployment'' entity with ''DeploymentEngine'' classes (https://review.openstack.org/#/c/56481/);
+
* Add base to provide [https://review.openstack.org/#/c/102899 interation testing];
:* сode refactoring for ''Task'' and ''Deployment'' classes: making them similarly structured (https://review.openstack.org/#/c/56727) and moving them to a special rally.objects module (https://review.openstack.org/#/c/56480/);
+
* A lot of of patches to improve [https://review.openstack.org/#/q/status:merged+project:stackforge/rally+branch:master+topic:bp/improve-unit-test-coverage-rally,n,z unit tests coverage].
:* test coverage improvement for the ''Task'' class (https://review.openstack.org/#/c/57055/) and for the ''Orchestrator API'' (https://review.openstack.org/#/c/57054/).
 
* Minor updates related to ''deploy engines'' and ''server providers'': better support for Debian/Ubuntu in DevStack engine (https://review.openstack.org/#/c/57181/) and removing legacy code for SSH support (https://review.openstack.org/#/c/57266/)
 
  
  
Our plan for the current week comprises:
+
This week, our work continues with such main novelties as adding support for [https://review.openstack.org/#/c/100579 Keystone API v3], adding context classes for [https://review.openstack.org/#/c/103306 avoiding vm creation if tenant has no network] and [https://review.openstack.org/#/c/104564 prepare an image that will have installed the required programs], mechanism to provide [https://review.openstack.org/#/c/103377 rally as a service], [https://review.openstack.org/#/c/104962 code refactoring in context manager] to make code more clear.
* Finishing the work on separating ''deployments'' and ''benchmark tasks'' in Rally (https://review.openstack.org/#/c/57455/, https://review.openstack.org/#/c/56226/ etc.);
 
* Adding new features to the ''benchmark engine'' like executing benchmarks periodically and for a user-speficied period of time (https://review.openstack.org/#/c/57628/, https://review.openstack.org/#/c/56036/);
 
* Implementing LXC and multihost OpenStack deployment facilities (https://review.openstack.org/#/c/56222/, https://review.openstack.org/#/c/57240/).
 
  
  
We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.
+
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally <br/>
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
 
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
  
Line 304: Line 299:
  
  
Regards,<br />
+
Regards,<br/>
 
The Rally team
 
The Rally team
  
  
 
+
==== July 07, 2014 ====
 
 
==== November 18, 2013 ====
 
  
 
Hello stackers,
 
Hello stackers,
  
here is the first issue of our weekly update notes on Rally, Benchmark-as-a-Service project for OpenStack. Once a week we are going to post a few remarks on what we have done and what we plan to implement in Rally during the next week.
+
here is the update for the last weeks. From all the work we've completed we would like to highlight the following:
  
 +
* New benchmark scenarios:
 +
:* Benchmark to [https://review.openstack.org/#/c/89326 validate a '''keystone''' token N times at service endpoint] that allows to check effect of caching related to tokens;
 +
:* [https://review.openstack.org/#/c/93739 Scenario for benchmark '''Neutron''' ports].
 +
* [https://review.openstack.org/#/c/89555 Add nova floating ip management in VM scenario], before this patch the VM runcommand scenario used the fixed ip to ssh connect to the instance, this could only worked if fixed ip range was directly accessible with was the case in very limited deployments.
 +
* [https://review.openstack.org/#/c/95353 Add image context class] that allows adding images to each user for benchmarks, also this patch provides scenario which tests glance image-list command;
 +
* [https://review.openstack.org/#/c/99304 Large update of documentation] that simplifies its index structure and headings, extends the 'Benchmark' page with info from the "Main concepts" and adds some introductory text on Deploy & Verify parts;
 +
* [https://review.openstack.org/#/c/98158 Add service-level agreement checking] that allows to add section in task configuration that contains the criteria of success, e.g "less then 5% of failure rate", or "faster then 5 seconds for one iteration";
 +
* [https://review.openstack.org/#/c/103706 Use oslo.db module and remove rally/openstack/common/db].
  
During the past week we have been focusing our efforts on two main aspects of Rally development:
 
* Splitting the Rally workflow into 2 parts: the OpenStack deployment part and the benchmark tasks running part. Both have been previously treated by the system as a single process configured once by the end user. Separation of deployment from benchmark tasks, however, allows one to reuse existing deployments. The current results here are:
 
:* Deployment model for SQLAlchemy (https://review.openstack.org/#/c/56185/);
 
:* A separate wrapper class for the deployment model (https://review.openstack.org/#/c/56267/).
 
* Improving the LXC server provider: code refactoring and support of some useful Btrfs features. This will be needed soon for multihost OpenStack deployment implementation. (https://review.openstack.org/#/c/55534/, https://review.openstack.org/#/c/56221/)
 
  
 +
Current work includes such interesting novelties as [https://blueprints.launchpad.net/rally/+spec/benchmark-context-semantic-validation context semantic validation], [https://review.openstack.org/#/c/102363 periodic runner refactoring], [https://review.openstack.org/#/c/102853 "rally info" command] which prints descriptions for different entities in Rally, [https://review.openstack.org/#/c/97556 checking if required services are available before starting the scenario] and refactoring of validation system, continue work on [https://review.openstack.org/94806 "stress" runner].
  
We have also recently received several e-mails notifying us of a possible issue in the soft/hard server reboot benchmark scenario. We would like to thank all of you who reported the problem. We will try to fix it as soon as possible.
 
  
 +
We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.
  
The Rally roadmap for the next week goes as follows:
+
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br/>
* Continue the work on separating OpenStack deployments from benchmark tasks: introduce the necessary CLI commands, integrate the Deployment class with deploy engines, rewrite the orchestrator part to support the separated deployments and benchmarks;
+
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally <br/>
* Implement multihost OpenStack deployment engine using LXC;
+
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
* Add two new capabilities to the benchmark runner:
 
:* Benchmark launching for a given period of time (not a strict amount of times);
 
:* Launching several benchmarks with configured intervals.
 
* Improve the benchmark config format to make it both more flexible and more clear for the end user;
 
* Implement generic cleanup for our benchmark scenarios;
 
* Work on automated output data processing and drawing plots for benchmark results.
 
  
  
 +
Stay tuned.
  
Several patches addressing the above tasks are already available on Gerrit code review. You are welcome to take a look at them.
 
  
Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally<br />
+
Regards,<br/>
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z
+
The Rally team
  
  
Stay tuned.
 
  
 
+
==== Archives ====
Regards,<br />
+
* [[Rally/UpdatesJune2014|Updates archives - June 2014]]
The Rally team
+
* [[Rally/UpdatesMay2014|Updates archives - May 2014]]
 +
* [[Rally/UpdatesApril2014|Updates archives - April 2014]]
 +
* [[Rally/UpdatesMarch2014|Updates archives - March 2014]]
 +
* [[Rally/UpdatesFebruary2014|Updates archives - February 2014]]
 +
* [[Rally/UpdatesJanuary2014|Updates archives - January 2014]]
 +
* [[Rally/UpdatesDecember2013|Updates archives - December 2013]]
 +
* [[Rally/UpdatesNovember2013|Updates archives - November 2013]]

Latest revision as of 12:02, 22 January 2015

Updates

January 22, 2014

Hi stackers!

We are happy to announce that we have completely redesigned our Rally documentation in ReadTheDocs. The docs have now received a simpler structure and have become much easier to get through!

One of the nicest new things is the Rally step-by-step tutorial that explains, in a series of lessons, how to explore the power of Rally in benchmarking your OpenStack clouds.

Since our previous update, there have been many interesting updates in Rally:

  • Rally now has a Network Context class that enables easy Neutron network management.
  • Input task files now can be written using the jinja2-based templates syntax. Very useful if you want, say, parameterize the image name used throughout your complex input task file.
  • Rally scenarios have been 100%-covered with docstrings. That means that now the rally info find <query> command will always output a complete piece information about whatever you ask it.
  • New benchmark scenarios include thos for Cinder (list_volumes, list_snapshots, extend_volume), Nova (cold_migrate)
  • We have moved the directory with samples in our repository to the root level: now it is rally/samples instead of rally/doc/samples (and so much quicker to get to).
  • Rally is on the way to being Python 3 compatible. We have added a Gate job that checks Rally in Python 3 and have produced lots of patches that fix incompability issues. Few changes are left to make Rally fully Python 3 compatible.


Current work includes new benchmark scenarios (Mistral, Murano), new success criteria (SLA) and a lot of refactoring in scenario runners, Rally API etc.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



December 15, 2014

Hi stackers!

Let us share with you our recent accomplishments in Rally:

  • CLI improvements:
  • The rally info command (which is a kind of built-in Rally reference) has been enhanced in such a way that it now prints detailed explanations of main concepts used in Rally whenever you type something like rally info BenchmarkScenarios or rally info SLA. We've also improved the output formatting so that now it is much easier to get through.
  • The rally task list command now supports filters. You can filter the task list either by deployment (using the "... --deployment <deployment_name_or_id>" parameter) or by status ("... --status <status_name>")
  • New benchmark scenarios:
  • Other improvements:


Current work is centered aroud code refactoring (both major, as in the benchmark engine or the contexts, and minor, as introducing some syntax sugar via decorators to mark deprecated stuff and scenario samples). We also constantly work on expanding our scenario base. Last but not least, we're about to merge the Network Context class that enables easy Neutron network management.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



December 1, 2014

Hi stackers!

It's been a while since our last post here, and we've done quite a nice job in Rally during November. Let us share with you new things about Rally:

  • Autogenerated HTML benchmark reports in Rally (which can be created by the "rally task report" command after a benchmark task has completed) have been improved further within the last month. As of now, the report page contains an overview table, detailed informations about whether SLA (service-level agreement) checks were successful and also detailed error logs, if any. Rally reports have become a wonderful tool indeed to analyse the benchmarking data as well as to share your results with others!
  • Similar improvements have been made for HTML reports generated for the Tempest cloud verification ("rally verify results --html --output_file <file>"). New enhanced report pages have improved styling and refactored JS code.
  • We have changed the way context classes in Rally should be declared. Having introduced a new @context decorator, we've made it much easier and also more readable.
  • There is a new "servers" context that allows you to create temporary servers before benchmark scenarios start and use these servers for testing inside these scenarios.
  • New benchmark scenarios in Rally include those for Nova live migrate and also a Cinder stress scenario.
  • Command-line interface improvements include an ability to refer deployments not only by uuid but also by name. Please note that the syntax has changed a bit so now you have to supply the --deployment parameter to commands like "rally use deployment" (instead of --uuid).
  • There has been some major refactoring of the most critical parts of Rally code: the cleanup mechanism and the "users" context code. We are sure that after refactoring, this code has become both cleaner and less error-prone (as well as very pluggable in the case of cleanups).


Current work includes further code refactoring (e.g. in the Benchmark engine part), further CLI improvements (e.g. for the "rally task list" command) and also new benchmark scenarios (e.g. for Murano). We are also going to introduce a possibility of building Rally images for Docker.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned!


Regards,
The Rally team



October 27, 2014

Hello stackers,

much time has passed since our last update and we are happy to announce that we are moving towards making our first official Rally release! Our active recent contribution to Rally has enabled us to make a significant progress. Here are the highlights of the novelties in Rally:

  • We have completely redesigned the auto-generated benchmark report page so that it looks now even nicer than before and is much more easy to navigate. Besides, further improvements of this HTML report page are on their way to being merged soon.
  • Rally now has an extended support of plugins: in addition to writing custom scenarios, the plugin mechanism now enables to extend Rally with new context classes/scenario runners without actually contributing to the Rally master branch.
  • The work on extending the support of OpenStack projects in Rally has been conducted for Heat and Sahara.
  • CLI improvements: the command-line interface gets more and more user-friendly over time: recently, it has begun to support detailed informations about correct commands usage in case of a failure. The "rally info" command has also been improved so that it now supports misspellings handling. Finally, there has been some work on bash completion, which hasn't been completely finished yet.
  • Test code improvements: we have greatly proceeded in out continuous work on unit/functional test coverage improvement. We also have moved all the tests into a special tests/ directory so that the test code is now organized in a more neat way. We also have removed the ./run_tests.sh script for the sake of using the tox command to launch the test suite.


In the nearest future, several interesting refactoring patches are going to come to Rally. To be more specific, there will be a vast change of the benchmark engine and cleanup mechanisms.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better!

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=kilo&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 26, 2014

Hello stackers,

here is a brief overview of what has happened in Rally recently:

  • We have refactored the code responsible for atomic actions processing in Rally benchmarks. Let's remind you that each benchmark scenario in Rally consists of a series of atomic actions, whereas the running time of each atomic action is measured in the same way as that of the whole scenario. After refactoring, we now ensure that Rally doesn't skip atomic actions that failed and distinguishes different runs of two atomic actions with the same name (see an example of how a typical results table could look before and looks after refactoring).
  • Another direction of refactoring was the SLA code; it has been modified so that now SLA results are stored in the DB along with other benchmarking data.
  • Check out a nice user story about VMs boot performance in Nova with Rally. It shows how Rally can be used in practice to catch reals bugs and improve OpenStack performance.
  • We work hard on achieving 100% test coverage in Rally: last week, additional unit tests for contexts and scenarios have been merged.


Our current priorities are further refactoring steps, including those in critical parts of Rally (e.g. temporary user creation and cloud cleanup code); we also strive towards making Rally bug-free and continuously issue different bugfixing patches.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 15, 2014

Hello stackers,

there has been much diverse and useful contribution to Rally recently. Let us highlight some interesting updates in our project:

  • Rally is on its way to support of benchmarking OpenStack clouds using ordinary user accounts that already exist. Rally lacked such functionality (it only supported benchmarking either from an admin account or from a bunch of temporarily created users), which posed a problem since some deployments don't allow temporary users creation. There have been two patches that prepare the code for this new functionality. It is going to come very soon - stay tuned.
  • We constantly improve our gate jobs (jobs that run a test suite for every patch pending review in Rally). Recently, we have introduced a nice aggregated results page for that gate job (see an example). It now makes it very easy for developers to navigate through the results of tests agains their patches to Rally.
  • We have added a new "volumes" context which makes it possible to create cinder volumes in the benchmark environment and use them later in the actual benchmark scenarios.
  • Much work has been accomplished on the overall code quality improvement. We have both made several code refactoring patches and introduced several new test suites (say, functional tests for CLI).


Current work centers around introducing benchmarking with pre-created users as mentioned above; much work is currently devoted to new benchmark scenarios as well. Also please note that there will be soon an important update on Rally ReadTheDocs page, which will make it much easier to navigate and get through, especially for newbies.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


September 1, 2014

Hello stackers,

over the past week, Rally has been extended with the following features:

  • Rally now offers a pretty simple feature request mechanism. Everyone who is interested in adding new functionality to Rally now can share his ideas in a standartized format (see an example). All you need to do is to write down your feature request in a separate rst-file in the doc/feature_request folder and submit it as a patch to Rally (if you are unsure about how to do this, read our "How to contribute" tutorial).
  • We added a new benchmark scenario for Cinder quotas creation/deletion.
  • Designate support in Rally has been extended so that it is posisble now to set up Designate quotas.


There is pretty much work going on this week, including the introduction of benchmarking with existing users (in addition to those created temporarily, which was the only option before), and also improvements in the plugin mechanism, so that it will be possible now to write plugins for runners and contexts. We do lots of code refactoring and will do even more in the upcoming weeks.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


August 25, 2014

Hello stackers,

here is a quick overview of what has been added to Rally recently:

  • Rally now supports Designate, which is a DNaaS service for OpenStack (providing REST API for domain/record management). Several new bechmark scenarios for Designate have been added as well;
  • In the CLI, we have added a new command called rally info. It is a essentially a special search engine embedded into Rally, which, for a given search query, prints documentation for the corresponding benchmark scenario/deploy engine/... as fetched from the source code. Thus you can learn about different Rally entities without leaving the Command Line Interface. For usage samples, see this link;
  • We have extended the SLA output which indicates whether a benchmark has passed some set of predefined success criteria with customizable messages;
  • We have performed a range of nice optimizations:

The main work directions for the next week comprise further improvement in the contexts mechanism (both the algorithm of context creation and new context classes), further optimizations (e.g., multithreaded objects deletion) and also code refactoring (e.g., of that for tracking atomic actions in benchmark scenarios)

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


August 4, 2014

Hello stackers,

below you will find the most recent updates in Rally:

  • The periodic scenario runner has been refactored and renamed, as of now, to rpc (runs per second). Note that this renaming also affects task configuration files that use that runner type. The runner itself now has been reimplemented with the multiprocessing.Process class (instead of multiprocessing.Pool), which potentially decreases memory usage on large iterations and reduces errors.
  • A nice optimization to the chart generation code ensures that the Rally plots will be rendered fast in browsers even for tasks with a huge number of iterations completed.
  • New benchmark scenarios include those testing the Nova server resize operation and a set of benchmarks for Sahara group node templates.
  • We continue extending our gates with nice features. A new one are the SLA checks that add information about whether benchmark scenarios pass a set of success criteria (see an example)


Many interesting improvements are on their path to being merged to Rally. Among them, let us mention the "rally info" command, which prints descriptions for different entities in Rally to the console, further work on the Rally documentation which gets cleaner and cleaner, and also a set of new context classes, including those for generating images for VM benchmarks, as well as for creating Neutron networks.

We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


July 17, 2014

Hello stackers,

here are some recent updates in Rally:


This week, our work continues with such main novelties as adding support for Keystone API v3, adding context classes for avoiding vm creation if tenant has no network and prepare an image that will have installed the required programs, mechanism to provide rally as a service, code refactoring in context manager to make code more clear.


We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


July 07, 2014

Hello stackers,

here is the update for the last weeks. From all the work we've completed we would like to highlight the following:

  • New benchmark scenarios:


Current work includes such interesting novelties as context semantic validation, periodic runner refactoring, "rally info" command which prints descriptions for different entities in Rally, checking if required services are available before starting the scenario and refactoring of validation system, continue work on "stress" runner.


We encourage you to take a look at new patches in Rally pending for review and to help us make Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=juno&metric=commits&project_type=all&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team


Archives