Jump to: navigation, search

Difference between revisions of "Rally/Concepts"

(Overall page structure)
(Benchmark scenarios notion)
Line 6: Line 6:
 
=== Benchmark scenarios ===
 
=== Benchmark scenarios ===
 
==== Notion of benchmark scenarios ====
 
==== Notion of benchmark scenarios ====
...
+
The concept of '''benchmark scenarios''' is a central one in Rally. Benchmark scenarios are what Rally actually uses to '''test the performance of an OpenStack deployment'''. They also play the role of main building blocks in the configurations of benchmark tasks. Each benchmark scenario performs a small '''set of atomic operations''', thus testing some '''simple use case''', usually that of a specific OpenStack project. For example, the '''''"NovaServers"''''' scenario group contains scenarios that use several basic operations available in '''''nova'''''. The '''''"boot_and_delete_server"''''' benchmark scenario from that group allows to benchmark the performance of a sequence of only '''two simple operations''': it first '''boots''' a server (with customizable parameters) and then '''deletes''' it.
  
  

Revision as of 11:09, 26 May 2014

Main concepts

This article describes in detail several main design concepts used throughout the Rally code (such as benchmark scenarios, contexts etc.). A good understanding of these concepts might be essential for a successful contribution to Rally.


Benchmark scenarios

Notion of benchmark scenarios

The concept of benchmark scenarios is a central one in Rally. Benchmark scenarios are what Rally actually uses to test the performance of an OpenStack deployment. They also play the role of main building blocks in the configurations of benchmark tasks. Each benchmark scenario performs a small set of atomic operations, thus testing some simple use case, usually that of a specific OpenStack project. For example, the "NovaServers" scenario group contains scenarios that use several basic operations available in nova. The "boot_and_delete_server" benchmark scenario from that group allows to benchmark the performance of a sequence of only two simple operations: it first boots a server (with customizable parameters) and then deletes it.


User's view

...


Developer's view

...


Scenario runners

Notion of scenario runners

...


User's view

...


Developer's view

...


Benchmark contexts

Notion of contexts

The notion of contexts in Rally is essentially used to define different types of environments in which benchmark scenarios can be launched. Those environments are usually specified by such parameters as the number of tenants and users that should be present in an OpenStack project, the roles granted to those users, extended or narrowed quotas and so on.


User's view

In user's view, contexts in Rally are manageable via the task configuration files. In a typical configuration file, each benchmark scenario to be run has is not only supplied by the information on with which arguments and how many times it should be launched, but also with a special "context" section. In this section, the user may configure a number of contexts he needs his scenarios to be run within.

In the example below, the "users" context specifies that the "NovaServers.boot_server" scenario should be run from 1 tenant having 3 users in it. Bearing in mind that the default quota for the number of instances is 10 instances pro tenant, it is also reasonable to extend it to, say, 20 instances in the "quotas" context. Otherwise the scenario would eventually fail, since it tries to boot a server 15 times from a single tenant.

{
    "NovaServers.boot_server": [
        {
            "args": {
                "flavor_id": 42,
                "image_id": "73257560-c59b-4275-a1ec-ab140e5b9979"
            },
            "runner": {
                "type": "constant",
                "times": 15,
                "concurrency": 2
            },
            "context": {
                "users": {
                    "tenants": 1,
                    "users_per_tenant": 3
                },
                "quotas": {
                    "nova": {
                        "instances": 20
                    }
                }
            }
        }
    ]
}


Developer's view

In developer's view, contexts management is implemented via Context classes. Each context type that can be specified in the task configuration file corresponds to a certain subclass of the base Context class, located in the rally.benchmark.context module. Every context class should implement a fairly simple interface:

from rally import utils

class YourContext(base.Context):
    """Yet another context class."""

    __ctx_name__ = "your_context"  # Corresponds to the context field name in task configuration files
    __ctx_order__ = xxx            # a 3-digit number specifying the priority with which the context should be set up
    __ctx_hidden__ = False         # True if the context cannot be configured through the task configuration file

    # The schema of the context configuration format
    CONFIG_SCHEMA = {
        "type": "object",
        "$schema": utils.JSON_SCHEMA,
        "additionalProperties": False,
        "properties": {
            "property_1": <SCHEMA>,
            "property_2": <SCHEMA>
        }
    }

    def __init__(self, context):
        super(YourContext, self).__init__(context)
        # Initialize the necessary stuff

    def setup(self):
        # Prepare the environment in the desired way

    def cleanup(self):
        # Cleanup the environment properly

Consequently, the algorithm of initiating the contexts can be roughly seen as follows:

context1 = Context1(ctx)
context2 = Context2(ctx)
context3 = Context3(ctx)

context1.setup()
context2.setup()
context3.setup()

<Run benchmark scenarios in the prepared environment>

context3.cleanup()
context2.cleanup()
context1.cleanup()

- where the order of contexts in which they are set up depends on the value of their __ctx_order__ attribute. Contexts with lower __ctx_order__ have higher priority: 1xx contexts are reserved for users-related stuff (e.g. users/tenants creation, roles assignment etc.), 2xx - for quotas etc.

The __ctx_hidden__ attribute defines whether the context should be a hidden one. Hidden contexts cannot be configured by end-users through the task configuration file as shown above, but should be specified by a benchmark scenario developer through a special @base.scenario(context={...}) decorator. Hidden contexts are typically needed to satisfy some specific benchmark scenario-specific needs, which don't require the end-user's attention. For example, the hidden "allow_ssh" context is used in the VMTasks.boot_runcommand_delete benchmark scenario to enable the SSH access to the servers. The fact that end-users do not have to worry about such details about SSH while launching this benchmark scenarios obviously makes their life easier and shows why hiddent contexts are of great importance in Rally.

If you want to dive deeper, see also the context manager class that actually implements the algorithm described above.