Testing Zones with One Physical Host
I recently had need to set up multiple instance of nova (multiple zones) on one physical host, for testing purposes. This documents how I did it. Note that the zones share a single keystone and glance instance, and I do not document setting that up here.
Separating Zone Configuration/Information
In order to keep the zones as separate as possible, I made separate directories for each one and placed all the configuration in there. This is not strictly necessary—the main difference between the zones is the nova.conf—but I happen to use other files in the directory to automate starting up the nova daemons I need. (I will not document my preferred automation here, but could be convinced to do so under separate cover. -Vek <email@example.com>) If you're using the sqlite backend for storing the nova database, this also provides a convenient place to store the database files, but I use the mysql backend and will document the differences here.
Once you've decided on how you're going to keep your zones separate, the next step is to set up the nova configuration. The important configuration options to pay attention to are:
For generating a value for `--build_plan_encryption_key`, I use a snippet of Python code like the following:
#!highlight python import random charset = "0123456789abcdef" print ''.join([random.choice(charset) for i in range(32)])
Once you have the appropriate configuration file, it is necessary to initialize the database. For mysql, you need to create the databases before they can be initialized; for this, use the `mysql` command to connect to the database server and use "
CREATE DATABASE 'name';" to create the database for each zone. (This step may be safely skipped for sqlite setups.) Then it is time to initialize the database; for each zone you're setting up, do:
$ /path/to/nova-manage --flagfile=/path/to/zone/nova.conf db sync
If you take an update to nova, make sure to re-run this command for each database.
Setting Up Multiple RabbitMQ Instances
Each zone must have its own instance of RabbitMQ. As it turns out, this is easy to do; under Debian-derived systems, create `/etc/default/rabbitmq` and give it the following contents:
The value you choose should be the same as the number of zones you're running on this host. This will cause `NODE_COUNT` instances of the RabbitMQ server to be started with sequential port numbers, starting with 5672; use these values for `--rabbit_port` in the zone's nova.conf.
[Pending] Keystone Configuration
#!wiki caution '''Work in Progress''' Keystone integration is still a work in progress. This discussion should be mostly stable, but does not include all the particulars of setting up Keystone.
Configuring Keystone is actually the most complex part of the operation. To begin with, you need to set up a Keystone tenant for each zone:
$ keystone-manage tenant add zone1tenant $ keystone-manage tenant add zone2tenant ...
Next, you need to set up admin users for the zones:
$ keystone-manage user add zone1admin password zone1tenant $ keystone-manage role grant Admin zone1admin zone1tenant $ keystone-manage user add zone2admin password zone2tenant $ keystone-manage role grant Admin zone2admin zone2tenant ...
Now we come to the hard part—creating endpoint templates. This will look something like this:
$ keystone-manage endpointTemplates add zone1region nova_compat \ http://<IP>:<zone1port>/v1.0 \ http://<IP>:<zone1port>/v1.0 \ http://<IP>:<zone1port>/v1.0 1 0 $ keystone-manage endpointTemplates add zone1region compute \ http://<IP>:<zone1port>/v1.0 \ http://<IP>:<zone1port>/v1.0 \ http://<IP>:<zone1port>/v1.0 1 0 $ keystone-manage endpointTemplates add zone1region nova \ http://<IP>:<zone1port>/v1.1/%tenant_id% \ http://<IP>:<zone1port>/v1.1/%tenant_id% \ http://<IP>:<zone1port>/v1.1/%tenant_id% 1 0 $ keystone-manage endpointTemplates add zone1region compute_v1 \ http://<IP>:<zone1port>/v1.1/%tenant_id% \ http://<IP>:<zone1port>/v1.1/%tenant_id% \ http://<IP>:<zone1port>/v1.1/%tenant_id% 1 0 ...
Take note of the IDs for each of these templates as you create them, because it is also necessary to link them to the tenants you created above:
$ keystone-manage endpoint add zone1tenant 1 $ keystone-manage endpoint add zone1tenant 2 $ keystone-manage endpoint add zone1tenent 3 $ keystone-manage endpoint add zone1tenant 4 ...
#!wiki caution '''Take care in endpoint assignments''' Endpoint templates for the zone1region should only be assigned to zone1tenant, those for zone2region should only be assigned to zone2tenant, etc. This is the mechanism by which the `nova` client tool looks up the actual zone endpoint to contact. Also note that the zones mechanism within nova uses the `nova` client tool, so you must set up endpoint templates and endpoints for all zones you set up.
We're finally to the last step. Start up each of the zones in question and use the `nova` client tool to verify the operation of each zone in turn, using the users you set up above. Then, pick a zone to be the parent zone; child zones don't know anything about their parents (and can in fact have multiple parents), but parent zones must be told about each child zone they are to use. This is done using the `zone-add` command to `nova`. As an example, let's assume `zone1` is the parent zone and `zone2` is the child; the `nova zone-add` invocation will look something like this:
$ export NOVA_USERNAME=zone1admin $ export NOVA_API_KEY=password $ export NOVA_PROJECT_ID=zone1tenant $ export NOVA_URL=http://<keystoneIP>:<keystonePort> $ nova zone-add --zone_username zone2admin --password password $NOVA_URL
Note that `NOVA_URL` is pointing to Keystone, here; keystone uses the endpoint templates to match up the username (`zone2admin`) with the zone endpoint. Subsequent calls to
nova zone-list will display the details about the zones visible to `zone1`. (Note that zone information is updated at regular intervals, so initially values may show up as "n/a" until the first update.)