Difference between revisions of "GSoC2014/Rally/BenchmarksVirtualMachinesOpenStack"

Revision as of 12:18, 15 June 2014

Introduction

The ability to benchmark a Virtual Machine is an important activity that more and more developers will need to perform as they host their SaaS applications in a cloud. The aim of this project is to integrate into the Rally project the ability to run easily and in an automated manner various benchmarks for measuring the performance of the deployed Virtual Machines of an OpenStack cloud.

Description and Analysis

The goal of this project is to port to Rally existing popular benchmarks used for measuring the performance of a computer system, to the virtual machines of an OpenStack cloud. In order to accomplish that, it is required to develop an architecture that will be flexible to adapt to any benchmark that someone will likely need to use to measure the performance of her Virtual Machine(s).

To achieve that, the architecture is modular and it can be distinguished in the following discrete steps:

1. Create/boot [1, n] VM(s), where n >= 1
2. Inject into the spawned VM(s) the setup.sh script of the specified benchmark. This script builds the benchmark in the VM(s).
3. Inject into the VM(s) the run.py script of the specified benchmark. This script executes the benchmark in the VM(s), process the output of the benchmark, and returns to Rally the output in a form that it can be stored into Rally's database.
4. With the processed results, we choose which kind of chart can visualize better this benchmark, and produce a chart for it in the HTML task report.

The above procedure reveals that a VM benchmark for Rally will be only 2 scripts: one that installs the benchmark, and a second one that executes it. Also, if a benchmark requires more than one VM in order to be run properly (ex. iperf3 that needs two machines) then with this method we can easily isolate the steps, first create the VMs, install into them the benchmark, and then we are ready to execute it. In this case, the run script will be a bit more complex (in contrast with running a benchmark that requires only one machine) but this is inevitable, and we still have all the logic for execution in a separate file without polluting the codebase of Rally with functions for this particular benchmark.

List of Benchmarks to Port

* Blogbench (Disk/io)

Status

Week 01 (May 19 - May 25) Implement the developing/testing environment. Testing environment: Single node architecture (services: keystone, glance, nova, nova-network) on a separate physical machine. Developing environment: Vim with appropriate plugins for Python development, on a FreeBSD desktop, with Rally installed on it (https://review.openstack.org/#/c/95341/). Research available benchmarks; selected primary source of information: Phoronix Test Suite, OpenBenchmarking.org. Tried the PTS Desktop Live.

Week 02 (May 26 - June 01) Design an initial architecture for the project with modularity and extensibility in mind. What is written in the "Description and Analysis" section is a product of this. Search for a good (in respect of popularity and reliable results) disk/io benchmark. Tried locally bonnie++, dbench, blogbench. Implement and test the setup script for blogbench (https://review.openstack.org/#/c/95341/).

Week 03 (June 02 - June 08) Under construction :-)

Source Code

Any code written in the development of this project.

Work In Progress

Add benchmark 'Blogbench' for VMs (https://review.openstack.org/#/c/97030/)
Add the VM scenario 'boot_benchmark_delete' (https://review.openstack.org/#/c/98172/)

Merged

Modify install_rally.sh to install under BSDs (https://review.openstack.org/#/c/95341/)

Links

[1] Blogbench, http://www.pureftpd.org/project/blogbench

@@ Line 4: / Line 4: @@
 == Description and Analysis ==
-The goal of this project is to port to Rally existing popular benchmarks used for measuring the performance of a computing system, to the virtual machines of an OpenStack cloud. In order to accomplish that, it is required to develop an architecture that will be flexible to adapt to any benchmark that someone will likely need to use to measure the performance of his/her VM(s).
+The goal of this project is to port to Rally existing popular benchmarks used for measuring the performance of a computer system, to the virtual machines of an OpenStack cloud. In order to accomplish that, it is required to develop an architecture that will be flexible to adapt to any benchmark that someone will likely need to use to measure the performance of her Virtual Machine(s).
 To achieve that, the architecture is modular and it can be distinguished in the following discrete steps:
 . Create/boot [1, n] VM(s), where n >= 1
-. Inject into the spawned VM(s) the setup_x.sh script, where x is the name of a benchmark. This script builds the benchmark in the virtual machine.
+. Inject into the spawned VM(s) the setup.sh script of the specified benchmark. This script builds the benchmark in the VM(s).
-. Inject into the VM(s) the run_x.sh script, where x is again the name of the benchmark. This script executes the benchmark in the virtual machine, process the output of the benchmark, and returns to Rally the ouput in a form that it can be stored into Rally's database.
+. Inject into the VM(s) the run.py script of the specified benchmark. This script executes the benchmark in the VM(s), process the output of the benchmark, and returns to Rally the output in a form that it can be stored into Rally's database.
-. With the processed results, we choose which kind of chart can visualize better this benchmark, and produce a chart for it in the html task report.
+. With the processed results, we choose which kind of chart can visualize better this benchmark, and produce a chart for it in the HTML task report.
-The above procedure reveals that a VM benchmark for Rally will be only 2 scripts: one that installs the benchmark, and a second one that executes it. Also, if a benchmark requires more than one VM in order to be run properly (ex. iperf3 that needs two machines) then with this method we can easily isolate the steps, first create your VMs, install them the benchmark, and then you are ready to execute it. In this case, the run script will be a bit more complex (in contrast with running a benchmark that requires only one machine) but this is inevitable, and we still have all the logic for execution in a separate file without polluting the codebase of Rally with functions for this particular benchmark.
+The above procedure reveals that a VM benchmark for Rally will be only 2 scripts: one that installs the benchmark, and a second one that executes it. Also, if a benchmark requires more than one VM in order to be run properly (ex. iperf3 that needs two machines) then with this method we can easily isolate the steps, first create the VMs, install into them the benchmark, and then we are ready to execute it. In this case, the run script will be a bit more complex (in contrast with running a benchmark that requires only one machine) but this is inevitable, and we still have all the logic for execution in a separate file without polluting the codebase of Rally with functions for this particular benchmark.
 === List of Benchmarks to Port ===