Jump to: navigation, search

Difference between revisions of "Rally/Updates"

(Weekly updates: 09.12 - Updates in benchmark engine)
(December 9, 2013)
Line 6: Line 6:
  
 
There has been much activity during the past week in Rally, but only few patches have been merged recently:
 
There has been much activity during the past week in Rally, but only few patches have been merged recently:
* Following the work on elaborating database entities for '''''Deployments''''' and '''''Tasks''''', we have introduced the '''''Resource''''' entity that allows Rally ''server providers'' to track resources allocated for deployment & benchmarking and safely delete them after all the scheduled benchmark tasks have been finished (https://review.openstack.org/#/c/60168). Recall that ''server providers'' in Rally form a special class of entities that are used by ''deployment engines'' and implement creating and deleting VMs on different architectures (Amazon, Tripple-O etc.), providing a standard interface to accomplish these tasks. Just like the ''deployment engines'' store the configuration and state of a deployment being in use in the database, ''server providers'' now store the configuration of VMs that they have created;
+
* '''Splitting Deploy's & Benchmark's Workflows is coming to the end.'''
* Work on code refactoring: fixing a structure issue in the folder with Rally configuration samples (https://review.openstack.org/#/c/59259/) and renaming the '''''ServerDTO''''' entity (used by ''server providers'') to '''''Server''''' to improve code readability (https://review.openstack.org/#/c/59749/).
+
The last blocker was that we were storing allocated Resources by '''''Server Providers''''' in memory of Rally process instead of permanent storage (e.g. DB). During this week we added new table '''''Resource''''' to DB and switch almost all (except LXC) '''''Server Providers''''' to use DB instead of in-memory storage. Now we should switch LXC provider and we will be able to merge this patch https://review.openstack.org/#/c/57455/ and finish splitting these workflows.
 +
 
 +
 
 +
* '''Code refactoring'''  
 +
# Fixing a structure issue in the folder with Rally configuration samples (https://review.openstack.org/#/c/59259/)
 +
# Renaming the '''''ServerDTO''''' ->  '''''Server''''' entity (used by ''server providers'') to improve code readability (https://review.openstack.org/#/c/59749/).
 +
 
 +
 
 +
* '''Generic Cloud Cleanup is finally fully implement'''
 +
[https://review.openstack.org/#/c/55695/ This patch] add new feature to Rally, now Rally will clean all resources that were created by benchmark scenario, but for some reasons weren't deleted. It is supper useful for writing such tests that just add new resources, like  [[Rally/BenchmarkScenarios#boot_server|boot VMs]]
  
  
Line 28: Line 37:
 
Regards,<br />
 
Regards,<br />
 
The Rally team
 
The Rally team
 
 
 
  
 
==== December 2, 2013 ====
 
==== December 2, 2013 ====

Revision as of 17:35, 9 December 2013

Weekly updates

December 9, 2013

Hello stackers,

There has been much activity during the past week in Rally, but only few patches have been merged recently:

  • Splitting Deploy's & Benchmark's Workflows is coming to the end.

The last blocker was that we were storing allocated Resources by Server Providers in memory of Rally process instead of permanent storage (e.g. DB). During this week we added new table Resource to DB and switch almost all (except LXC) Server Providers to use DB instead of in-memory storage. Now we should switch LXC provider and we will be able to merge this patch https://review.openstack.org/#/c/57455/ and finish splitting these workflows.


  • Code refactoring
  1. Fixing a structure issue in the folder with Rally configuration samples (https://review.openstack.org/#/c/59259/)
  2. Renaming the ServerDTO -> Server entity (used by server providers) to improve code readability (https://review.openstack.org/#/c/59749/).


  • Generic Cloud Cleanup is finally fully implement

This patch add new feature to Rally, now Rally will clean all resources that were created by benchmark scenario, but for some reasons weren't deleted. It is supper useful for writing such tests that just add new resources, like boot VMs


A wide variety of new contributions to Rally is still under development and pending for review:

  • Enriching the benchmark engine with the mechanisms for cloud initialization before and cleanup after launching benchmark scenarios:
  • The support for init() methods in benchmark scenario classes was actually already implemented in Rally 0.1 but has been broken since creating multiple temporary tenants/users for benchmarking had been introduced to Rally (due to the fact that resources - servers, floating IPs etc. - created in init() did no longer belong to appropriate temporary tenants/users and thus could not be used in benchmark scenarios). There is now a patch (https://review.openstack.org/#/c/59782/) that fixes this issue by calling init() once for each temporary user and thus creating the appropriate resources (servers, floating IPs etc.) for every temporary OpenStack user that may be involved in benchmarking. This patch also has as a consequence a couple of smaller patches that improve the performance of OpenStack python clients (https://review.openstack.org/#/c/59780/, https://review.openstack.org/#/c/59781/) and thus optimize the procedure of creating resources for temporary users in init() methods;
  • Performing a generic cleanup after launching benchmark scenarios is essential for guaranteeing the cloud to stay clean after benchmarking procedures and, besides, enables the benchmark scenario writers not to worry about deleting all the resources they create in init() methods or concrete benchmark procedures (https://review.openstack.org/#/c/55695/). "Generic" means that Rally should free all possible kinds of allocated resources: servers, images etc.


We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team

December 2, 2013

Hello stackers,

below you will find the latest review of our activities in Rally for the past week.

Our achievements for the end of November comprise:

  • Numerous changes in the benchmark engine, the most important among which are:
  • Rally is now able not only to perform a specified number of benchmark scenario launches, but also to create a continuous load on the cloud by running any scenario for the given period of time. For example, you can now boot and delete servers in the cloud continiously from a number of temporary users, say, for 10 minutes, thus simulating in this way a stress load on the cloud. To do so, the only thing you should change in your configuration file is the "times" field for the benchmark scenario you are going to launch, which now should be replaced with "duration" field and initialized to the number of minutes the benchmark is expected to run (https://review.openstack.org/#/c/56036/);
  • Access to openstack clients with administrator permissions is now enabled for all the scenarios through the admin_clients() method of the base Scenario class. Before this update, this class provided only the clients() method which returned a reference to a non-admin OpenStack client. This, however, turned out to be not enough for keystone-based benchmark scenarios that are to come in the future releases (https://review.openstack.org/#/c/58381/);
  • Bugfix for the init() methods of benchmark scenarios which now enables benchmark scenario writers to pass through the context dictionary (which is the dictionary that init() returns) not only primitive objects like strings or numbers but also the complex ones like references to specially prepared servers or floating ips (https://review.openstack.org/#/c/55465/).
  • The work on separating the deployment and task entities mentioned in previous updates has now come closely to its successful conclusion. The main results here include:
  • Server provider for OpenStack: another ServerProvider class that wraps with the default ServerProvider interface (create_vms() and destroy_vms() methods) the functionality of python-novaclient. Along with lxc and virsh server providers (already present in the system) it constitutes the essential basis for working with different virtualization technologies (https://review.openstack.org/#/c/48811);
  • The first contribution to data processing and visualization in Rally: a new CLI command for tasks has been added, namely plot aggregated which draws plots illustrating the cloud performance on one of the finished benchmark tasks. The CLI command requires the user to indicate the parameter for which the plots will be drawn. For example, if one specifies active_users as the aggregating parameter, Rally will draw a plot that shows how the number of active users making requests to the cloud affects the runtime of benchmark scenarios. The code uses the matplotlib library to draw the plots (https://review.openstack.org/#/c/52712/).


This week, our work will be concentrated on the following:


We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team



November 25, 2013

Hello stackers,

here is the second report on our activities in Rally development for the past week.

The main results that have been recently merged with master are as follows:

  • Changes in the benchmark engine: we have significantly restructured the format of the input benchmark config (https://review.openstack.org/#/c/56035/). The changes make it more transparent to the end-user as well as more flexible. This will enable us to implement new features in the benchmark engine like running tests periodically or for a given amount of time. We have also refactored the test code related to benchmark scenarios by replacing ugly-looking nested with-blocks for mocks with a more readable decorator syntax (https://review.openstack.org/#/c/57732/);
  • Further work on splitting the system logic between the two basic entities, namely the deployment and the benchmark task. While still having the legacy combined config that contains information both on the deployment and on the benchmarks, we have closely come to the point where we can completely split everything related to these two entites. To be more precise, during the lask week we have made:


Our plan for the current week comprises:


We encourage you to take a look at new patches in Rally pending for review and to help us making Rally better.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
You can track the overall progress in Rally via Stackalytics: http://stackalytics.com/?release=all&metric=commits&project_type=All&module=rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team



November 18, 2013

Hello stackers,

here is the first issue of our weekly update notes on Rally, Benchmark-as-a-Service project for OpenStack. Once a week we are going to post a few remarks on what we have done and what we plan to implement in Rally during the next week.


During the past week we have been focusing our efforts on two main aspects of Rally development:

  • Splitting the Rally workflow into 2 parts: the OpenStack deployment part and the benchmark tasks running part. Both have been previously treated by the system as a single process configured once by the end user. Separation of deployment from benchmark tasks, however, allows one to reuse existing deployments. The current results here are:


We have also recently received several e-mails notifying us of a possible issue in the soft/hard server reboot benchmark scenario. We would like to thank all of you who reported the problem. We will try to fix it as soon as possible.


The Rally roadmap for the next week goes as follows:

  • Continue the work on separating OpenStack deployments from benchmark tasks: introduce the necessary CLI commands, integrate the Deployment class with deploy engines, rewrite the orchestrator part to support the separated deployments and benchmarks;
  • Implement multihost OpenStack deployment engine using LXC;
  • Add two new capabilities to the benchmark runner:
  • Benchmark launching for a given period of time (not a strict amount of times);
  • Launching several benchmarks with configured intervals.
  • Improve the benchmark config format to make it both more flexible and more clear for the end user;
  • Implement generic cleanup for our benchmark scenarios;
  • Work on automated output data processing and drawing plots for benchmark results.


Several patches addressing the above tasks are already available on Gerrit code review. You are welcome to take a look at them.

Source code for Rally is hosted at GitHub: https://github.com/stackforge/rally
Open reviews for Rally: https://review.openstack.org/#/q/status:open+rally,n,z


Stay tuned.


Regards,
The Rally team