View source for Customizing the OpenStack Compute (nova) Scheduler

Many OpenStack projects allow for customization of specific features using a driver architecture. You can write a driver that conforms to a particular interface and plug it in through configuration. For example, you can easily plug in a new scheduler for Compute. The existing schedulers for Compute are feature full and well documented at [https://docs.openstack.org/ocata/config-reference/compute/schedulers.html Scheduling]. However, depending on your user’s use cases, the existing schedulers might not meet your requirements. You might need to create a new scheduler.

To create a scheduler, you must inherit from the class <code>nova.scheduler.driver.Scheduler</code>. Of the five methods that you can override, you ''must'' override the two methods marked with an asterisk (*) below:

* <code>update_service_capabilities</code>
* <code>hosts_up</code>
* <code>group_hosts</code>
* * <code>schedule_run_instance</code>
* * <code>select_destinations</code>

To demonstrate customizing OpenStack, we’ll create an example of a Compute scheduler that randomly places an instance on a subset of hosts, depending on the originating IP address of the request and the prefix of the hostname. Such an example could be useful when you have a group of users on a subnet and you want all of their instances to start within some subset of your hosts.

{{Warning|header=Warning|body=This example is for illustrative purposes only. It should not be used as a scheduler for Compute without further development and testing.}}

When you join the screen session that <code>stack.sh</code> starts with <code>screen -r stack</code>, you are greeted with many screen windows:

<pre>0$ shell*  1$ key  2$ horizon  ...  9$ n-api  ...  14$ n-sch ...</pre>
; <code>shell</code>
: A shell where you can get some work done
; <code>key</code>
: The keystone service
; <code>horizon</code>
: The horizon dashboard web application
; <code>n-{name}</code>
: The nova services
; <code>n-sch</code>
: The nova scheduler service

'''To create the scheduler and plug it in through configuration'''

<ol style="list-style-type: decimal;">
<li>

The code for OpenStack lives in <code>/opt/stack</code>, so go to the <code>nova</code> directory and edit your scheduler module. Change to the directory where <code>nova</code> is installed:
<pre>$ cd /opt/stack/nova</pre></li>
<li>

Create the <code>ip_scheduler.py</code> Python source code file:
<pre>$ vim nova/scheduler/ip_scheduler.py</pre></li>
<li>

The code shown below is a driver that will schedule servers to hosts based on IP address as explained at the beginning of the section. Copy the code into <code>ip_scheduler.py</code>. When you are done, save and close the file.
<syntaxhighlight lang=python style="border:2px solid #ccc;"># vim: tabstop=4 shiftwidth=4 softtabstop=4
# Copyright (c) 2014 OpenStack Foundation
# All Rights Reserved.
#
#    Licensed under the Apache License, Version 2.0 (the &quot;License&quot;); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an &quot;AS IS&quot; BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

&quot;&quot;&quot;
IP Scheduler implementation
&quot;&quot;&quot;

import random

from oslo_config import cfg

from nova.compute import rpcapi as compute_rpcapi
from nova import exception
from nova.openstack.common import log as logging
from nova.openstack.common.gettextutils import _
from nova.scheduler import driver

CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
LOG = logging.getLogger(__name__)

class IPScheduler(driver.Scheduler):
    &quot;&quot;&quot;
    Implements Scheduler as a random node selector based on
    IP address and hostname prefix.
    &quot;&quot;&quot;

    def __init__(self, *args, **kwargs):
        super(IPScheduler, self).__init__(*args, **kwargs)
        self.compute_rpcapi = compute_rpcapi.ComputeAPI()

    def _filter_hosts(self, request_spec, hosts, filter_properties,
        hostname_prefix):
        &quot;&quot;&quot;Filter a list of hosts based on hostname prefix.&quot;&quot;&quot;

        hosts = [host for host in hosts if host.startswith(hostname_prefix)]
        return hosts

    def _schedule(self, context, topic, request_spec, filter_properties):
        &quot;&quot;&quot;Picks a host that is up at random.&quot;&quot;&quot;

        elevated = context.elevated()
        hosts = self.hosts_up(elevated, topic)
        if not hosts:
            msg = _(&quot;Is the appropriate service running?&quot;)
            raise exception.NoValidHost(reason=msg)

        remote_ip = context.remote_address

        if remote_ip.startswith('10.1'):
            hostname_prefix = 'doc'
        elif remote_ip.startswith('10.2'):
            hostname_prefix = 'ops'
        else:
            hostname_prefix = 'dev'

        hosts = self._filter_hosts(request_spec, hosts, filter_properties,
            hostname_prefix)
        if not hosts:
            msg = _(&quot;Could not find another compute&quot;)
            raise exception.NoValidHost(reason=msg)

        host = random.choice(hosts)
        LOG.debug(&quot;Request from %(remote_ip)s scheduled to %(host)s&quot; % locals())

        return host

    def select_destinations(self, context, request_spec, filter_properties):
        &quot;&quot;&quot;Selects random destinations.&quot;&quot;&quot;
        num_instances = request_spec['num_instances']
        # NOTE(timello): Returns a list of dicts with 'host', 'nodename' and
        # 'limits' as keys for compatibility with filter_scheduler.
        dests = []
        for i in range(num_instances):
            host = self._schedule(context, CONF.compute_topic,
                    request_spec, filter_properties)
            host_state = dict(host=host, nodename=None, limits=None)
            dests.append(host_state)

        if len(dests) &lt; num_instances:
            raise exception.NoValidHost(reason='')
        return dests

    def schedule_run_instance(self, context, request_spec,
                              admin_password, injected_files,
                              requested_networks, is_first_time,
                              filter_properties, legacy_bdm_in_spec):
        &quot;&quot;&quot;Create and run an instance or instances.&quot;&quot;&quot;
        instance_uuids = request_spec.get('instance_uuids')
        for num, instance_uuid in enumerate(instance_uuids):
            request_spec['instance_properties']['launch_index'] = num
            try:
                host = self._schedule(context, CONF.compute_topic,
                                      request_spec, filter_properties)
                updated_instance = driver.instance_update_db(context,
                        instance_uuid)
                self.compute_rpcapi.run_instance(context,
                        instance=updated_instance, host=host,
                        requested_networks=requested_networks,
                        injected_files=injected_files,
                        admin_password=admin_password,
                        is_first_time=is_first_time,
                        request_spec=request_spec,
                        filter_properties=filter_properties,
                        legacy_bdm_in_spec=legacy_bdm_in_spec)
            except Exception as ex:
                # NOTE(vish): we don't reraise the exception here to make sure
                #             that all instances in the request get set to
                #             error properly
                driver.handle_schedule_error(context, ex, instance_uuid,
                                             request_spec)</syntaxhighlight>


There is a lot of useful information in <code>context</code>, <code>request_spec</code>, and <code>filter_properties</code> that you can use to decide where to schedule the instance. To find out more about what properties are available, you can insert the following log statements into the <code>schedule_run_instance</code> method of the scheduler above:
<pre>LOG.debug(&quot;context = %(context)s&quot; % {'context': context.__dict__})
LOG.debug(&quot;request_spec = %(request_spec)s&quot; % locals())
LOG.debug(&quot;filter_properties = %(filter_properties)s&quot; % locals())</pre></li>
<li>

To plug this scheduler into nova, edit one configuration file, <code>/etc/nova/nova.conf</code>:
<pre>$ vim /etc/nova/nova.conf</pre></li>
<li>

Find the <code>scheduler_driver</code> config and change it like so:
<pre>scheduler_driver=nova.scheduler.ip_scheduler.IPScheduler</pre></li>
<li>

Restart the nova scheduler service to make nova use your scheduler. Start by switching to the <code>n-sch</code> screen:
<ol style="list-style-type: decimal;">
<li>Press '''Ctrl+A''' followed by '''9'''.</li>
<li>Press '''Ctrl+A''' followed by '''N''' until you reach the <code>n-sch</code> screen.</li>
<li>Press '''Ctrl+C''' to kill the service.</li>
<li>Press '''Up Arrow''' to bring up the last command.</li>
<li>Press '''Enter''' to run it.</li></ol>
</li>
<li>

Test your scheduler with the nova CLI. Start by switching to the <code>shell</code> screen and finish by switching back to the <code>n-sch</code> screen to check the log output:
<ol style="list-style-type: decimal;">
<li>

Press  '''Ctrl+A''' followed by '''0'''.</li>
<li>

Make sure you are in the <code>devstack</code> directory:
<pre>$ cd /root/devstack</pre></li>
<li>

Source <code>openrc</code> to set up your environment variables for the CLI:
<pre>$ . openrc</pre></li>
<li>

Put the image ID for the only installed image into an environment variable:
<pre>$ IMAGE_ID=`openstack image list | egrep cirros | egrep -v &quot;kernel|ramdisk&quot; | awk '{print $2}'`</pre></li>
<li>

Boot a test server:
<pre>$ openstack server create --flavor 1 --image $IMAGE_ID scheduler-test</pre></li></ol>
</li>
<li>

Switch back to the <code>n-sch</code> screen. Among the log statements, you’ll see the line:
<pre>2014-01-23 19:57:47.262 DEBUG nova.scheduler.ip_scheduler
[req-... demo demo] Request from xx.xx.xx.xx scheduled to devstack-havana
_schedule /opt/stack/nova/nova/scheduler/ip_scheduler.py:76</pre></li></ol>

{{Warning|header=Warning|body=Functional testing like this is not a replacement for proper unit and integration testing, but it serves to get you started.}}

A similar pattern can be followed in other projects that use the driver architecture. Simply create a module and class that conform to the driver interface and plug it in through configuration. Your code runs when that feature is used and can call out to other services as necessary. No project core code is touched. Look for a “driver” value in the project’s <code>.conf</code> configuration files in <code>/etc/&lt;project&gt;</code> to identify projects that use a driver architecture.

When your scheduler is done, we encourage you to open source it and let the community know on the OpenStack mailing list. Perhaps others need the same functionality. They can use your code, provide feedback, and possibly contribute. If enough support exists for it, perhaps you can propose that it be added to the official Compute [https://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler schedulers].