OpenStack - User contributions [en]

XenServer/XenAndXenServer

2013-02-17T01:13:18Z

Olaph:

= Getting Started =

To get started, take a look at: [[XenServer/GettingStarted|Getting started with XenServer and [[OpenStack]]]]

= Official Docs =

You can find this information in the official docs:
http://docs.openstack.org/trunk/openstack-compute/admin/content/introduction-to-xen.html

= Introduction to using Xen, XCP and [[XenServer]] with [[OpenStack]] =

In this section we will describe Xen, XCP, and [[XenServer]], the
differences between them, and how to use them with [[OpenStack]]. Xen's
architecture is different from KVM's in important ways, and we discuss those
differences and when each might make sense in your [[OpenStack]] cloud.

== Basic terminology ==

=== Xen ===
Xen is a hypervisor. It provides the
fundamental isolation between virtual machines. Xen is open source (GPLv2)
and is managed by Xen.org, an cross-industry organization.
Xen is a component of many different products and projects. The
hypervisor itself is very similar across all these projects, but the way that
it is managed can be different, which can cause confusion if you're not clear
which toolstack you are using. Make sure you know what toolstack you want
before you get started.

=== XCP ===
XCP is an open source (GPLv2) toolstack
for Xen. It is designed specifically as platform for enterprise and cloud
computing, and is well integrated with [[OpenStack]].

=== Citrix [[XenServer]] ===

Citrix [[XenServer]] is a commercial
product. It is based on XCP, and exposes the same toolstack and managment
API. As an analogy, think of [[XenServer]] being based on XCP in the way that Red
Hat Enterprise Linux is based on Fedora. [[XenServer]] has a free version (which
is very similar to XCP) and paid-for versions with additional features
enabled.

=== xapi / XAPI / XenAPI ===

Both [[XenServer]] and XCP include Xen, Linux, and the primary control
daemon known as xapi.

The API shared between XCP and [[XenServer]] is called ""XenAPI"". [[OpenStack]] usually refers to XenAPI, to
indicate that the integration works equally well on XCP and [[XenServer]].
Sometimes, a careless person will refer to [[XenServer]] specifically, but you can
be reasonably confident that anything that works on [[XenServer]] will also work
on the latest version of XCP.

== Privileged and unprivileged domains ==

A Xen host will run a number of virtual machines, VMs, or domains (the
terms are synonymous on Xen). One of these is in charge of running the rest
of the system, and is known as "domain 0", or "dom0". It is the first domain
to boot after Xen, and owns the storage and networking hardware, the device
drivers, and the primary control software. Any other VM is unprivileged, and
are known as a "domU" or "guest". All customer VMs are unprivileged of
course, but you should note that on Xen the [[OpenStack]] control software
(nova-compute) also runs in a domU. This gives a level of security isolation
between the privileged system software and the [[OpenStack]] sofware (much of
which is customer-facing). This architecture is described in more detail
later.

There is an ongoing project to split domain 0 into multiple privileged
domains known as ""driver domains"" and <emphasis
role="bold">stub domains"". This would give even better separation
between critical components. This is an ongoing research project and you
shouldn't worry about it right now. The current architecture just has three
levels of separation: dom0, the [[OpenStack]] domU, and the completely
unprivileged customer VMs.

== Paravirtualized versus hardware virtualized domains ==

A Xen virtual machine can be ""paravirtualized
(PV)"" or ""hardware virtualized
(HVM)"". This refers to the interaction between Xen, domain 0, and
the guest VM's kernel. PV guests are aware of the fact that they are
virtualized and will co-operate with Xen and domain 0; this gives them better
performance characteristics. HVM guests are not aware of their environment,
and the hardware has to pretend that they are running on an unvirtualized
machine. HVM guests have the advantage that there is no need to modify the
guest operating system, which is essential when running Windows.

In [[OpenStack]], customer VMs may run in either PV or HVM mode. However,
the [[OpenStack]] domU (that's the one running nova-compute) <emphasis
role="bold">must"" be running in PV mode.

= Deployment architecture =

When you deploy [[OpenStack]] on XCP or [[XenServer]] you will get something
similar to this:

[[Image:XenServer-dom0-domU.png]]

Key things to note:

* The hypervisor: Xen
* Domain 0: runs xapi and some small pieces from [[OpenStack]] (some xapi plugins and network isolation rules). The majority of this is provided by [[XenServer]] or XCP (or yourself using Kronos).
* [[OpenStack]] domU: The nova-compute code runs in a paravirtualized virtual machine, running on the host under management. Each host runs a local instance of nova-compute. It will often also be running nova-network (depending on your network mode). In this case, nova-network is managing the addresses given to the tenant VMs through DHCP.
* Nova uses the XenAPI Python library to talk to xapi, and it uses the Host Internal Management Network to reach from the domU to dom0 without leaving the host.

Some notes on the networking:
* The above diagram assumes FlatDHCP networking (the [[DevStack]] default).
* There are three main [[OpenStack]] networks: Management traffic (RabbitMQ, MySQL, etc), Tenant network traffic (controlled by nova-network) and Public traffic (floating IPs, public API end points).
* Each network that leaves the host has been put through a separate physical network interface. This is the simplest model, but it's not the only one possible. You may choose to isolate this traffic using VLANs instead, for example.
* nova-compute generally talks to [[XenServer]] using a host local private network called either "Guest Installer Network" or "Host Internal Managmenet Network".

Another example of how to deploy [[OpenStack]] on a system with only two physical network interfaces is:

[[Image:XenServer$$XenXCPAndXenServer$DevStackDiagram.png]]

== [[XenServer]] pools ==

Before [[OpenStack]] 2012.1 ("Essex"), all [[XenServer]] machines used with
[[OpenStack]] are standalone machines, usually only using local storage.

However in 2012.1 and later, the host-aggregates feature allows you to
create pools of [[XenServer]] hosts (configuring shared storage is still an out of
band activity). This move will enable live migration when using shared
storage.

== Xen and libvirt ==

It may possible to manage Xen using libvirt. This would be necessary
for any Xen-based system that isn't using the XCP toolstack, such as SUSE
Linux or Oracle Linux. Unfortunately, this is not well tested or supported
today, and using the XCP or [[XenServer]] toolstacks with the [[OpenStack]] XenAPI
backend is the only recommended method.

= Further reading =

What is Xen? by Xen.org: http://xen.org/files/Marketing/WhatisXen.pdf.

Xen Hypervisor project: http://xen.org/products/xenhyp.html.

XCP project: http://xen.org/products/cloudxen.html.

XenServer/XenAndXenServer

2013-02-17T01:12:14Z

Olaph: Reverted edits by Olaph (talk) to last revision by JohnGarbutt

__NOTOC__

<<[[TableOfContents]]()>>

= Getting Started =

To get started, take a look at: [[XenServer/GettingStarted|Getting started with XenServer and [[OpenStack]]]]

= Official Docs =

You can find this information in the official docs:
http://docs.openstack.org/trunk/openstack-compute/admin/content/introduction-to-xen.html

= Introduction to using Xen, XCP and [[XenServer]] with [[OpenStack]] =

In this section we will describe Xen, XCP, and [[XenServer]], the
differences between them, and how to use them with [[OpenStack]]. Xen's
architecture is different from KVM's in important ways, and we discuss those
differences and when each might make sense in your [[OpenStack]] cloud.

== Basic terminology ==

=== Xen ===
Xen is a hypervisor. It provides the
fundamental isolation between virtual machines. Xen is open source (GPLv2)
and is managed by Xen.org, an cross-industry organization.
Xen is a component of many different products and projects. The
hypervisor itself is very similar across all these projects, but the way that
it is managed can be different, which can cause confusion if you're not clear
which toolstack you are using. Make sure you know what toolstack you want
before you get started.

=== XCP ===
XCP is an open source (GPLv2) toolstack
for Xen. It is designed specifically as platform for enterprise and cloud
computing, and is well integrated with [[OpenStack]].

=== Citrix [[XenServer]] ===

Citrix [[XenServer]] is a commercial
product. It is based on XCP, and exposes the same toolstack and managment
API. As an analogy, think of [[XenServer]] being based on XCP in the way that Red
Hat Enterprise Linux is based on Fedora. [[XenServer]] has a free version (which
is very similar to XCP) and paid-for versions with additional features
enabled.

=== xapi / XAPI / XenAPI ===

Both [[XenServer]] and XCP include Xen, Linux, and the primary control
daemon known as xapi.

The API shared between XCP and [[XenServer]] is called ""XenAPI"". [[OpenStack]] usually refers to XenAPI, to
indicate that the integration works equally well on XCP and [[XenServer]].
Sometimes, a careless person will refer to [[XenServer]] specifically, but you can
be reasonably confident that anything that works on [[XenServer]] will also work
on the latest version of XCP.

== Privileged and unprivileged domains ==

A Xen host will run a number of virtual machines, VMs, or domains (the
terms are synonymous on Xen). One of these is in charge of running the rest
of the system, and is known as "domain 0", or "dom0". It is the first domain
to boot after Xen, and owns the storage and networking hardware, the device
drivers, and the primary control software. Any other VM is unprivileged, and
are known as a "domU" or "guest". All customer VMs are unprivileged of
course, but you should note that on Xen the [[OpenStack]] control software
(nova-compute) also runs in a domU. This gives a level of security isolation
between the privileged system software and the [[OpenStack]] sofware (much of
which is customer-facing). This architecture is described in more detail
later.

There is an ongoing project to split domain 0 into multiple privileged
domains known as ""driver domains"" and <emphasis
role="bold">stub domains"". This would give even better separation
between critical components. This is an ongoing research project and you
shouldn't worry about it right now. The current architecture just has three
levels of separation: dom0, the [[OpenStack]] domU, and the completely
unprivileged customer VMs.

== Paravirtualized versus hardware virtualized domains ==

A Xen virtual machine can be ""paravirtualized
(PV)"" or ""hardware virtualized
(HVM)"". This refers to the interaction between Xen, domain 0, and
the guest VM's kernel. PV guests are aware of the fact that they are
virtualized and will co-operate with Xen and domain 0; this gives them better
performance characteristics. HVM guests are not aware of their environment,
and the hardware has to pretend that they are running on an unvirtualized
machine. HVM guests have the advantage that there is no need to modify the
guest operating system, which is essential when running Windows.

In [[OpenStack]], customer VMs may run in either PV or HVM mode. However,
the [[OpenStack]] domU (that's the one running nova-compute) <emphasis
role="bold">must"" be running in PV mode.

= Deployment architecture =

When you deploy [[OpenStack]] on XCP or [[XenServer]] you will get something
similar to this:

[[Image:XenServer$$XenXCPAndXenServer$XenServer-dom0-domU.png]]

Key things to note:

* The hypervisor: Xen
* Domain 0: runs xapi and some small pieces from [[OpenStack]] (some xapi plugins and network isolation rules). The majority of this is provided by [[XenServer]] or XCP (or yourself using Kronos).
* [[OpenStack]] domU: The nova-compute code runs in a paravirtualized virtual machine, running on the host under management. Each host runs a local instance of nova-compute. It will often also be running nova-network (depending on your network mode). In this case, nova-network is managing the addresses given to the tenant VMs through DHCP.
* Nova uses the XenAPI Python library to talk to xapi, and it uses the Host Internal Management Network to reach from the domU to dom0 without leaving the host.

Some notes on the networking:
* The above diagram assumes FlatDHCP networking (the [[DevStack]] default).
* There are three main [[OpenStack]] networks: Management traffic (RabbitMQ, MySQL, etc), Tenant network traffic (controlled by nova-network) and Public traffic (floating IPs, public API end points).
* Each network that leaves the host has been put through a separate physical network interface. This is the simplest model, but it's not the only one possible. You may choose to isolate this traffic using VLANs instead, for example.
* nova-compute generally talks to [[XenServer]] using a host local private network called either "Guest Installer Network" or "Host Internal Managmenet Network".

Another example of how to deploy [[OpenStack]] on a system with only two physical network interfaces is:

[[Image:XenServer$$XenXCPAndXenServer$DevStackDiagram.png]]

== [[XenServer]] pools ==

Before [[OpenStack]] 2012.1 ("Essex"), all [[XenServer]] machines used with
[[OpenStack]] are standalone machines, usually only using local storage.

However in 2012.1 and later, the host-aggregates feature allows you to
create pools of [[XenServer]] hosts (configuring shared storage is still an out of
band activity). This move will enable live migration when using shared
storage.

== Xen and libvirt ==

It may possible to manage Xen using libvirt. This would be necessary
for any Xen-based system that isn't using the XCP toolstack, such as SUSE
Linux or Oracle Linux. Unfortunately, this is not well tested or supported
today, and using the XCP or [[XenServer]] toolstacks with the [[OpenStack]] XenAPI
backend is the only recommended method.

= Further reading =

What is Xen? by Xen.org: http://xen.org/files/Marketing/WhatisXen.pdf.

Xen Hypervisor project: http://xen.org/products/xenhyp.html.

XCP project: http://xen.org/products/cloudxen.html.

XenServer/XenAndXenServer

2013-02-17T01:08:33Z

Olaph:

= Getting Started =

To get started, take a look at: [[XenServer/GettingStarted|Getting started with XenServer and [[OpenStack]]]]

= Official Docs =

You can find this information in the official docs:
http://docs.openstack.org/trunk/openstack-compute/admin/content/introduction-to-xen.html

= Introduction to using Xen, XCP and [[XenServer]] with OpenStack =

In this section we will describe Xen, XCP, and [[XenServer]], the
differences between them, and how to use them with OpenStack. Xen's
architecture is different from KVM's in important ways, and we discuss those
differences and when each might make sense in your OpenStack cloud.

== Basic terminology ==

=== Xen ===
Xen is a hypervisor. It provides the
fundamental isolation between virtual machines. Xen is open source (GPLv2)
and is managed by Xen.org, an cross-industry organization.
Xen is a component of many different products and projects. The
hypervisor itself is very similar across all these projects, but the way that
it is managed can be different, which can cause confusion if you're not clear
which toolstack you are using. Make sure you know what toolstack you want
before you get started.

=== XCP ===
XCP is an open source (GPLv2) toolstack
for Xen. It is designed specifically as platform for enterprise and cloud
computing, and is well integrated with [[OpenStack]].

=== Citrix [[XenServer]] ===

Citrix [[XenServer]] is a commercial
product. It is based on XCP, and exposes the same toolstack and managment
API. As an analogy, think of [[XenServer]] being based on XCP in the way that Red
Hat Enterprise Linux is based on Fedora. [[XenServer]] has a free version (which
is very similar to XCP) and paid-for versions with additional features
enabled.

=== xapi / XAPI / XenAPI ===

Both [[XenServer]] and XCP include Xen, Linux, and the primary control
daemon known as xapi.

The API shared between XCP and [[XenServer]] is called ""XenAPI"". OpenStack usually refers to XenAPI, to
indicate that the integration works equally well on XCP and [[XenServer]].
Sometimes, a careless person will refer to [[XenServer]] specifically, but you can
be reasonably confident that anything that works on [[XenServer]] will also work
on the latest version of XCP.

== Privileged and unprivileged domains ==

A Xen host will run a number of virtual machines, VMs, or domains (the
terms are synonymous on Xen). One of these is in charge of running the rest
of the system, and is known as "domain 0", or "dom0". It is the first domain
to boot after Xen, and owns the storage and networking hardware, the device
drivers, and the primary control software. Any other VM is unprivileged, and
are known as a "domU" or "guest". All customer VMs are unprivileged of
course, but you should note that on Xen the OpenStack control software
(nova-compute) also runs in a domU. This gives a level of security isolation
between the privileged system software and the OpenStack sofware (much of
which is customer-facing). This architecture is described in more detail
later.

There is an ongoing project to split domain 0 into multiple privileged
domains known as ""driver domains"" and <emphasis
role="bold">stub domains"". This would give even better separation
between critical components. This is an ongoing research project and you
shouldn't worry about it right now. The current architecture just has three
levels of separation: dom0, the OpenStack domU, and the completely
unprivileged customer VMs.

== Paravirtualized versus hardware virtualized domains ==

A Xen virtual machine can be ""paravirtualized
(PV)"" or ""hardware virtualized
(HVM)"". This refers to the interaction between Xen, domain 0, and
the guest VM's kernel. PV guests are aware of the fact that they are
virtualized and will co-operate with Xen and domain 0; this gives them better
performance characteristics. HVM guests are not aware of their environment,
and the hardware has to pretend that they are running on an unvirtualized
machine. HVM guests have the advantage that there is no need to modify the
guest operating system, which is essential when running Windows.

In OpenStack, customer VMs may run in either PV or HVM mode. However,
the OpenStack domU (that's the one running nova-compute) <emphasis
role="bold">must"" be running in PV mode.

= Deployment architecture =

When you deploy OpenStack on XCP or [[XenServer]] you will get something
similar to this:

[[Image:XenServer-dom0-domU.png]]

Key things to note:

* The hypervisor: Xen
* Domain 0: runs xapi and some small pieces from OpenStack (some xapi plugins and network isolation rules). The majority of this is provided by [[XenServer]] or XCP (or yourself using Kronos).
* OpenStack domU: The nova-compute code runs in a paravirtualized virtual machine, running on the host under management. Each host runs a local instance of nova-compute. It will often also be running nova-network (depending on your network mode). In this case, nova-network is managing the addresses given to the tenant VMs through DHCP.
* Nova uses the XenAPI Python library to talk to xapi, and it uses the Host Internal Management Network to reach from the domU to dom0 without leaving the host.

Some notes on the networking:
* The above diagram assumes FlatDHCP networking (the [[DevStack]] default).
* There are three main OpenStack networks: Management traffic (RabbitMQ, MySQL, etc), Tenant network traffic (controlled by nova-network) and Public traffic (floating IPs, public API end points).
* Each network that leaves the host has been put through a separate physical network interface. This is the simplest model, but it's not the only one possible. You may choose to isolate this traffic using VLANs instead, for example.
* nova-compute generally talks to [[XenServer]] using a host local private network called either "Guest Installer Network" or "Host Internal Managmenet Network".

Another example of how to deploy OpenStack on a system with only two physical network interfaces is:

[[Image:DevStackDiagram.png]]

== [[XenServer]] pools ==

Before OpenStack 2012.1 ("Essex"), all [[XenServer]] machines used with
OpenStack are standalone machines, usually only using local storage.

However in 2012.1 and later, the host-aggregates feature allows you to
create pools of [[XenServer]] hosts (configuring shared storage is still an out of
band activity). This move will enable live migration when using shared
storage.

== Xen and libvirt ==

It may possible to manage Xen using libvirt. This would be necessary
for any Xen-based system that isn't using the XCP toolstack, such as SUSE
Linux or Oracle Linux. Unfortunately, this is not well tested or supported
today, and using the XCP or [[XenServer]] toolstacks with the OpenStack XenAPI
backend is the only recommended method.

= Further reading =

What is Xen? by Xen.org: http://xen.org/files/Marketing/WhatisXen.pdf.

Xen Hypervisor project: http://xen.org/products/xenhyp.html.

XCP project: http://xen.org/products/cloudxen.html.

Obsolete:ZoneTesting

2013-02-17T01:02:25Z

Olaph:

= Testing Zones with One Physical Host =

I recently had need to set up multiple instance of nova (multiple ''zones'') on one physical host, for testing purposes. This documents how I did it. Note that the zones share a single keystone and glance instance, and I do not document setting that up here.

== Separating Zone Configuration/Information ==

In order to keep the zones as separate as possible, I made separate directories for each one and placed all the configuration in there. This is not strictly necessary—the main difference between the zones is the nova.conf—but I happen to use other files in the directory to automate starting up the nova daemons I need. (I will not document my preferred automation here, but could be convinced to do so under separate cover. -Vek <kevin.mitchell@rackspace.com>) If you're using the sqlite backend for storing the nova database, this also provides a convenient place to store the database files, but I use the mysql backend and will document the differences here.

Once you've decided on how you're going to keep your zones separate, the next step is to set up the nova configuration. The important configuration options to pay attention to are:

{| border="1" cellpadding="2" cellspacing="0"
| `--sql_connection`
|-
| `--connection_type`
|-
| `--allow_admin_api`
|-
| `--fake_network`
|-
| `--enable_zone_routing`
|-
| `--zone_name`
|-
| `--build_plan_encryption_key`
|-
| `--scheduler_driver`
|-
| `--default_host_filter`
|-
| `--ec2_listen_port`
|-
| `--osapi_listen_port`
|-
| `--rabbit_port`
|}

For generating a value for `--build_plan_encryption_key`, I use a snippet of Python code like the following:

<pre><nowiki>#!highlight python
import random
charset = "0123456789abcdef"
print ''.join([random.choice(charset) for i in range(32)])</nowiki></pre>

== Initializing Databases ==


Once you have the appropriate configuration file, it is necessary to initialize the database. For mysql, you need to create the databases before they can be initialized; for this, use the `mysql` command to connect to the database server and use "<code><nowiki>CREATE DATABASE 'name';</nowiki></code>" to create the database for each zone. (This step may be safely skipped for sqlite setups.) Then it is time to initialize the database; for each zone you're setting up, do:

<pre><nowiki>
$ /path/to/nova-manage --flagfile=/path/to/zone/nova.conf db sync</nowiki></pre>

If you take an update to nova, make sure to re-run this command for each database.

== Setting Up Multiple RabbitMQ Instances ==


Each zone must have its own instance of RabbitMQ. As it turns out, this is easy to do; under Debian-derived systems, create `/etc/default/rabbitmq` and give it the following contents:

<pre><nowiki>
NODE_COUNT=3</nowiki></pre>

The value you choose should be the same as the number of zones you're running on this host. This will cause `NODE_COUNT` instances of the RabbitMQ server to be started with sequential port numbers, starting with 5672; use these values for `--rabbit_port` in the zone's nova.conf.

== [Pending] Keystone Configuration ==

<pre><nowiki>#!wiki caution
'''Work in Progress'''

Keystone integration is still a work in progress. This discussion should be mostly stable, but does not include all the particulars of setting up Keystone.</nowiki></pre>

Configuring Keystone is actually the most complex part of the operation. To begin with, you need to set up a Keystone ''tenant'' for each zone:

<pre><nowiki>
$ keystone-manage tenant add zone1tenant
$ keystone-manage tenant add zone2tenant
...</nowiki></pre>

Next, you need to set up admin users for the zones:

<pre><nowiki>
$ keystone-manage user add zone1admin password zone1tenant
$ keystone-manage role grant Admin zone1admin zone1tenant
$ keystone-manage user add zone2admin password zone2tenant
$ keystone-manage role grant Admin zone2admin zone2tenant
...</nowiki></pre>

Now we come to the hard part—creating endpoint templates. This will look something like this:

<pre><nowiki>
$ keystone-manage endpointTemplates add zone1region nova_compat \
http://<IP>:<zone1port>/v1.0 \
http://<IP>:<zone1port>/v1.0 \
http://<IP>:<zone1port>/v1.0 1 0
$ keystone-manage endpointTemplates add zone1region compute \
http://<IP>:<zone1port>/v1.0 \
http://<IP>:<zone1port>/v1.0 \
http://<IP>:<zone1port>/v1.0 1 0
$ keystone-manage endpointTemplates add zone1region nova \
http://<IP>:<zone1port>/v1.1/%tenant_id% \
http://<IP>:<zone1port>/v1.1/%tenant_id% \
http://<IP>:<zone1port>/v1.1/%tenant_id% 1 0
$ keystone-manage endpointTemplates add zone1region compute_v1 \
http://<IP>:<zone1port>/v1.1/%tenant_id% \
http://<IP>:<zone1port>/v1.1/%tenant_id% \
http://<IP>:<zone1port>/v1.1/%tenant_id% 1 0
...</nowiki></pre>

Take note of the IDs for each of these templates as you create them, because it is also necessary to link them to the tenants you created above:

<pre><nowiki>
$ keystone-manage endpoint add zone1tenant 1
$ keystone-manage endpoint add zone1tenant 2
$ keystone-manage endpoint add zone1tenent 3
$ keystone-manage endpoint add zone1tenant 4
...</nowiki></pre>

<pre><nowiki>#!wiki caution
'''Take care in endpoint assignments'''

Endpoint templates for the zone1region should only be assigned to zone1tenant, those for zone2region should only be assigned to zone2tenant, etc. This is the mechanism by which the `nova` client tool looks up the actual zone endpoint to contact. Also note that the zones mechanism within nova uses the `nova` client tool, so you must set up endpoint templates and endpoints for all zones you set up.</nowiki></pre>

== Connecting Zones ==

We're finally to the last step. Start up each of the zones in question and use the `nova` client tool to verify the operation of each zone in turn, using the users you set up above. Then, pick a zone to be the ''parent'' zone; ''child'' zones don't know anything about their parents (and can in fact have multiple parents), but parent zones must be told about each child zone they are to use. This is done using the `zone-add` command to `nova`. As an example, let's assume `zone1` is the parent zone and `zone2` is the child; the `nova zone-add` invocation will look something like this:

<pre><nowiki>
$ export NOVA_USERNAME=zone1admin
$ export NOVA_API_KEY=password
$ export NOVA_PROJECT_ID=zone1tenant
$ export NOVA_URL=http://<keystoneIP>:<keystonePort>
$ nova zone-add --zone_username zone2admin --password password $NOVA_URL</nowiki></pre>

Note that `NOVA_URL` is pointing to Keystone, here; keystone uses the endpoint templates to match up the username (`zone2admin`) with the zone endpoint. Subsequent calls to <code><nowiki>nova zone-list</nowiki></code> will display the details about the zones visible to `zone1`. (Note that zone information is updated at regular intervals, so initially values may show up as "n/a" until the first update.)

== Resources ==

* http://nova.openstack.org/devref/distributed_scheduler.html
* http://nova.openstack.org/devref/zone.html

Obsolete:GerritJenkinsGit

2013-02-17T00:19:09Z

Olaph:

__NOTOC__
= Gerrit, Jenkins, and GitHub =
For a quick reference, please see [[GerritWorkflow]] instead.

[https://github.com/ GitHub] is a resource for managing Git code repositories and interacting with other developers. [http://jenkins-ci.org/ Jenkins] is used to continuously test all of the components of [[OpenStack]] to ensure functionality and to verify that each change to the code base works as intended. [http://code.google.com/p/gerrit/ Gerrit] is a code review system originally developed for use by the Android Open Source Project and allows us to build a workflow where every change is peer-reviewed and tested by Jenkins before being merged into the main repository.

After making a change in their local Git repository, developers can easily push that change to Gerrit as a proposed change for the project. Jenkins will automatically run functional tests on the code and provide feedback on the change in Gerrit. Any [[OpenStack]] developer can provide feedback (in the form of a comment, or even line-by-line annotations) using Gerrit, and the core developers of the project can indicate whether they approve of the patch as is, or would like to see changes before it is integrated. Once patches are merged by Gerrit, the repository is pushed to the canonical public repository on GitHub.

== Using Gerrit ==
The next sections describe how Gerrit fits into the developer workflow.

=== Gerrit Accounts ===
Gerrit lives at https://review.openstack.org/. Visit and click the '''Sign In''' link at the top-right corner of the page. Log in with your Launchpad ID.

Because Gerrit uses Launchpad OpenID single sign-on, you won't need a separate password for Gerrit, and once you log in to one of Launchpad, Gerrit, or Jenkins, you won't have to enter your password for the others.

Gerrit accounts are automatically synchronized with Launchpad, so your Gerrit account should already have the same username, full name, email address, ssh keys, and group membership.

Some information in Launchpad is not publicly available and so may not be copied over. The first time you log into Gerrit, you should click the '''Settings''' link at the top of the page, and then make sure that your '''Contact Information''', '''SSH Public Keys''', and '''Groups''' look correct. If not, please register your email address and SSH keys. If your group membership is not correct, please email openstack-ci-admins@lists.launchpad.net .

Ensure that you have run these steps to let git know about your email address:

<pre><nowiki>
git config --global user.name "Firstname Lastname"
git config --global user.email "your_email@youremail.com"
</nowiki></pre>

=== Setting up Git for Use with Gerrit ===
For a more comprehensive look at using Gerrit, see [https://review.openstack.org/Documentation/user-upload.html the Gerrit manual].

==== Change-Id Hook ====
Gerrit uses a '''Change-Id''' footer in commits so that it can link Git commits to changes stored in its database. When you upload a revised change (to correct a problem or respond to code review comments), Gerrit will use the Change-Id footer to attach the commit as a new patchset on the existing gerrit change. This works best if the Change-Id is already in the original commit message, before it is even sent to Gerrit.

"git review" installs a commit hook into your repository that automatically adds Change-Id lines to your commits..

The Gerrit manual goes into more detail about [https://review.openstack.org/Documentation/user-changeid.html change IDs].

Install the git-review tool, which is the tool [[OpenStack]] uses to simplify submission of reviews to Gerrit.

<pre><nowiki>
sudo pip install git-review
</nowiki></pre>

==== Pushing Changes from Git ====
Simply running '''git review''' should be sufficient to push your changes to Gerrit, assuming your repository is set up as described above, you don't need to read the rest of this section unless you want to use an alternate workflow.

If you want to push your changes without using git-review, you can push changes to gerrit like you would any other git repository, using the following syntax (assuming "gerrit" is configured as a remote repository):

<pre><nowiki>
git push gerrit HEAD:refs/for/$BRANCH[/$TOPIC]
</nowiki></pre>

Where $BRANCH is the name of the Gerrit branch to push to (usually "master"), and you may optionally specify a Gerrit topic by appending it after a slash character.

==== Git SSH Commands ====
If you find you are frequently executing Gerrit commands via SSH, you may wish to add something like the following to your '''~/.ssh/config''' file:

<pre><nowiki>
Host review
Hostname review.openstack.org
Port 29418
User USERNAME
</nowiki></pre>

Which may shorten some SSH commands; the following are equivalent:

<pre><nowiki>
ssh -p 29418 review.openstack.org gerrit ls-projects
ssh review gerrit ls-projects
</nowiki></pre>

== Reviewing a Change ==
Log in to https://review.openstack.org/ to see proposed changes, and review them.

To provide a review for a proposed change in the Gerrit UI, click on the Review button (it will be next to the buttons that will provide unified or side-by-side diffs in the browser). In the code review, you can add a message, as well as a vote (+1,0,-1).

Any Openstack developer may propose or comment on a change (including voting +1/0/-1 on it). Openstack projects have a policy requiring two positive reviews from core reviewers. A vote of +2 is allowed from core reviewers, and should be used to indicate that they are a core reviewer and are leaving a vote that should be counted as such.

When a review has two +2 reviews and one of the core team believes it is ready to be merged, he or she should leave a +1 vote in the "Approved" category. You may do so by clicking the "Review" button again, with or without changing your code review vote and optionally leaving a comment. When a +1 Approved review is received, Jenkins will run tests on the change, and if they pass, it will be merged.

=== Gerrit Best Practices ===
If you are working on unrelated changes, you should use a [http://progit.org/book/ch3-4.html topic branch] so that there isn't a dependency between the changes.

When you start working on a new change, make sure you have the current repository head from GitHub.

For more information about uploading changes to gerrit, see the [https://review.openstack.org/Documentation/user-upload.html Uploading Changes] section of the Gerrit manual.

=== Gerrit Errors ===
==== missing Change-Id in commit message ====
If you see an error like this:

<pre><nowiki>
! [remote rejected] HEAD -> refs/for/master (missing Change-Id in commit message)
</nowiki></pre>

Make sure that you have the [[#Change-Id_Hook|Change-Id hook]] installed. If you don't, install it now, and the run '''git commit --amend''' and re-save your commit message. The hook will then add a Change-Id line.

If you did have the hook installed, there may be a syntax error with the Change-Id line. It must be in the last paragraph of the commit message, and it must be at the beginning of the line. Your commit message should look like this in your editor:

<pre><nowiki>
The text of your commit message is here.

Change-Id: I5f55e68d1bdb42a0fa6f0b1a5432786d0395da51
</nowiki></pre>

==== squash commits first ====
If you see this message:

<pre><nowiki>
! [remote rejected] HEAD -> refs/for/master (squash commits first)
</nowiki></pre>

It means that you are trying to update an existing change in Gerrit, but you created two separate commits. Normally to update a change you should amend an existing commit (see [[GerritWorkflow#Updating_a_Change|Updating a Change]]). If you have already made a second commit, you will need squash the last two commits in your tree. To do that, run:

<pre><nowiki>
git rebase -i HEAD~2
</nowiki></pre>

Your editor should appear with two commits listed, one per line. Change the word "pick" on the second line to "squash", so that it looks like:

<pre><nowiki>

pick 1458567 Fix bug 1014727
pick 7284e97 Document underlying technologies
pick fac4698 Clean up on section
pick xxxxxxx 2nd commit back
squash yyyyyyy head

</nowiki></pre>

And save. You should then be able to upload your commit with '''git review'''.

==== can not create new references ====

If you try to send a draft with '''git review -D''' and it fails like this:

<pre><nowiki>
! [remote rejected] HEAD -> refs/draft/master (can not create new references)
</nowiki></pre>

Then check that you are running git-review version 1.16 or later. You can install the lastest version with pip:

<pre><nowiki>
pip install git-review
</nowiki></pre>

=== Gerrit Merge Problems ===
Gerrit will fast-forward or merge changes as necessary when they are approved. If a conflict would be created by a merge, gerrit will not merge the change and will record an error message in the comments for the change. In these cases, you may need to [http://www.kernel.org/pub/software/scm/git/docs/git-rebase.html rebase] or [http://www.kernel.org/pub/software/scm/git/docs/git-merge.html merge] the change, or if the repository head has changed significantly, you may need to change the patch.

If you don't already have the patch in your local repository, Gerrit provides commands on the web page for each change indicating how to download that change. You can then use git to correct the problem.

If you encounter other error messages from Gerrit, the [https://review.openstack.org/Documentation/error-messages.html Error Messages] section of the Gerrit manual may offer some tips.

=== Test Failures ===
When a new patchset is uploaded to Gerrit, that project's "check" tests are run on the patchset by Jenkins. Once completed the test results are reported to Gerrit by Jenkins in the form of a Verified: +/-1 vote. After code reviews have been completed and a change receives an Approved: +1 vote that project's "gate" tests are run on the change by Jenkins. Jenkins reports the results of these tests back to Gerrit in the form of a Verified: +/-2 vote. Code merging will only occur after the gate tests have passed successfully and received a Verified: +2. You can view the state of tests currently being run on the [http://status.openstack.org/zuul/ Zuul Status] and [https://jenkins.openstack.org/ Jenkins Dashboard] pages.

If a change fails tests in Jenkins, please follow the steps below:

# Jenkins leaves a comment in the review with links to the log files for the test run. Follow those links and examine the output from the test. It will include a console log, and in the case of unit tests, HTML output from the test runner, or in the case of a devstack-gate test, it may contain quite a large number of system logs.
# Examine the console log or other relevant log files to determine the cause of the error. If it is related to your change, you should fix the problem and upload a new patchset.
# If the problem is due to non-deterministic behavior already merged, and is unrelated to your change, you should do the following to help other developers who may be affected by the same issue, and to focus attention of QA, CI, and other developers working to fix high-impact bugs and improve test systems:
# Visit http://status.openstack.org/rechecks/ to see if one of the bugs listed there matches the error you've seen. If your error isn't there, then:
#* Identify which project(s) are affected, and search for a related bug on [https://bugs.launchpad.net/ Launchpad]. If you do not find an existing bug, file a new one (and be sure to include the error message). If the problem is due to an infrastructure problem (such as Jenkins, Gerrit, etc.), file (or search for) the bug against the "openstack-ci" project.
# Once you have a bug number in hand:
#* To re-run the check jobs (before a change has been approved), leave a comment with the form "recheck bug ####".
#* To re-run the gate jobs (after a change has been approved), leave a comment with the form "reverify bug ####".
# This will update the [http://status.openstack.org/rechecks/ rechecks status page] and help others quickly identify the issue if it occurs again.

If you need to re-run tests and it does not make sense to include a bug number (perhaps there is no error, you've just made a draft change visible and want it tested or you're updating test results because you know that a related branch has changed since the last time they were run), you may leave a comment with the form "recheck no bug" (or "reverify no bug"). Please only do this if you are certain there is no bug that needs to be addressed.

=== Merge Commits ===
When a branch is pushed to gerrit (either manually, or via git-review), each commit in the branch that is not in Gerrit's copy of the tree is turned into a new change for review (unless that commit has a Change-Id that corresponds to an existing change, in which case, the change is updated). This means that if a developer merges a branch from another source into their local repository and pushes to Gerrit, hundreds of new changes may be created. This is almost certainly not what was intended. For that reason, Gerrit is configured not to accept merge commits in the general case.

'''Caution: The following describes an experimental process and is not generally applicable yet:'''

However, merge commits are very useful for feature branches -- development branches that fork from the main line of development, occasionally receiving updates from the main line, and when development is complete, merging back into the main line. For these purposes, members of the $PROJECT-drivers group are permitted to upload merge commits to Gerrit. These users must take special care when dealing with merge commits to avoid the problems mentioned above. The following tips will help avoid problems with merge commits:

* Don't merge branches from gerrit and outside of gerrit (eg, GitHub or personal branches).
* When developing, follow the branching model described in [[GerritWorkflow]], this will avoid creating merge commits locally.
* Use git-review to upload your changes to Gerrit. By default, if git-review would create or update more than one commit, it will list the commits and ask you if that is what you intend to do. Pay attention to that, and if you see that too many commits are being uploaded, say "no" and clean up your local branch.
* To merge master into a feature branch, do the following:

<pre><nowiki>
git checkout feature-branch
git pull --ff-only origin feature-branch
git checkout -b merge-branch
git merge origin/master
git review -R feature-branch
git checkout master
git branch -D merge-branch
</nowiki></pre>

Note that follows the same model described in [[GerritWorkflow]] of using a topic branch for each independent unit of local development. In this case, the "merge-branch" branch is being used to generate the merge commit, and once submitted, is no longer needed and can be (optionally) removed. You may continue to use your local copy of the feature-branch as normal, pulling fast-forward only commits from Gerrit. The following similar process may be used to merge the feature branch into master, terminating development on that branch:

<pre><nowiki>
git checkout master
git pull --ff-only origin master
git checkout -b merge-branch
git merge origin/feature-branch
git review -R
git checkout master
git branch -D merge-branch
</nowiki></pre>

= Release Management =
== Milestones ==
Between the Milestone Branch Point and the release of the corresponding milestone, there needs to be a separate branch in Gerrit for changes destined for the milestone release. Meanwhile, development on the master branch should continue as normal (with the addition that changes proposed for the milestone should also be proposed for master, and some changes for master may need to be applied to milestone-proposed).

This process creates an ephemeral milestone-proposed branch that is only available in Gerrit during the milestone process. When the milestone is released, a tag is applied to the final commit to record the state of the branch at the time.

=== Create milestone-proposed Branch ===
This step should be performed by the [[OpenStack]] Release Manager at the Release Branch Point.

* Go to https://review.openstack.org/ and sign in
* Select '''Admin''', '''Projects''', then the project
* Select '''Branches'''
* Enter '''milestone-proposed''' in the '''Branch Name''' field, and '''HEAD''' as the Initial Revision, then press '''Create Branch'''

In your local checkout:

<pre><nowiki>
git checkout master
git pull
git checkout milestone-proposed
</nowiki></pre>

=== Authoring Changes for milestone-proposed ===
Create topic branches as normal, but branch them from milestone-proposed rather than master.

<pre><nowiki>
git checkout milestone-proposed
git pull
git checkout -b MY-TOPIC-BRANCH
</nowiki></pre>

Changes for milestone-proposed should be submitted with:

<pre><nowiki>
git push gerrit HEAD:refs/for/milestone-proposed
</nowiki></pre>

=== Submit Changes in master to milestone-proposed ===
If a change to master should also be included in milestone-proposed, use this procedure to cherry-pick that change and submit it for review.

<pre><nowiki>
git checkout milestone-proposed
git pull
git checkout -b master-to-mp
git cherry-pick <SHA1 or "master">
git push gerrit HEAD:refs/for/milestone-proposed
git branch -D master-to-mp
</nowiki></pre>

'''git cherry-pick master''' will pick the most recent commit from master to apply, if you want a different patch, use the SHA1 of the commit instead.

=== Submitting Changes in milestone-proposed to master ===
Changes that originate in milestone-proposed should also be submitted to master. Use these commands to make an up-to-date topic branch from master, then cherry-pick changes from milestone-proposed to be applied to master.

<pre><nowiki>
git checkout master
git pull
git checkout -b mp-to-master
git cherry-pick <SHA1 or milestone-proposed>
git review
git branch -D mp-to-master
</nowiki></pre>

'''git cherry-pick milestone-proposed''' will pick the most recent commit from milestone-proposed to apply, if you want a different patch, use the SHA1 of the commit instead.

=== Tagging a Release ===
This step should be performed by the [[OpenStack]] Release Manager when the release is made.

To tag the tip of the milestone-proposed branch with a release tag and push that tag to gerrit and GitHub, run the following commands:

<pre><nowiki>
git checkout milestone-proposed
git pull
git tag -s RELEASE-TAG-NAME
git push gerrit RELEASE-TAG-NAME
</nowiki></pre>

Running `git tag -s` signs the tag using GPG, so it's important to ensure that the person making the release have a valid GPG key.

Make sure you're only adding a single tag when pushing to gerrit.

=== End of Milestone ===
This step should be performed by the [[OpenStack]] Release Manager after the release is tagged.

When the milestone process is complete and the released commit is tagged, remove the milestone-proposed branch. The tag will persist, even after the branch is deleted, making it possible to restore the state of the tree.

* Go to https://review.openstack.org/ and sign in
* Select '''Admin''', '''Projects''', then the project
* Select '''Branches'''
* Select the checkbox next to '''milestone-proposed''' and hit '''Delete'''

= Resources =
See the [https://review.openstack.org/Documentation/index.html Gerrit documentation], especially the User Guide, for more information on how to use Gerrit. It is also available within Gerrit by clicking on the '''Documentation''' link on the top of the page.

The Mahara Project also [https://wiki.mahara.org/index.php/Developer_Area/Developer_Tools uses Git, Gerrit, and Jenkins] in a similar manner (though with Gitorious instead of GitHub).

A description of many of the elements of the [http://sandofsky.com/blog/git-workflow.html git workflow]

GenericMessagingSystem

2013-02-17T00:15:35Z

Olaph:

__NOTOC__




== Generic Messaging System Proposal ==
'''Drafter: ''' [[PaulGuth]]

'''Drafters Email: ''' <<[[MailTo]](paul AT cloudscaling DOT com)>>

'''Status: ''' To be completed by POC

'''Proposal'''

Nova currently supports only RabbitMQ as a messaging system for its internal RPC. We want to implement a generic interface that would allow people to use whatever messaging systems fit best in their environments. This should simplify management for organizations with established messaging infrastructures and also ease adoption for organizations that don't currently use RabbitMQ.

We are addressing this by separating the RPC interface out from the queuing system, essentially creating an API that nova will use to access whatever messaging system is underneath. nova/rpc.py will become generic messaging functionality, simply:

<pre><nowiki>
create_connection(new=True)
create_consumer(conn, topic, proxy, fanout=False)
create_consumer_set(conn, consumers)
call(context, topic, msg)
cast(context, topic, msg)
fanout_cast(context, topic, msg)
multicall(context, topic, msg)
</nowiki></pre>

The specifics of the underlying messaging implementation become a separate driver. As part of this blueprint the driver for the existing AMQP system will be included so no changes will be visible to existing installations or documentation. However, after these changes anyone can plugin a new driver to implement whatever messaging system they prefer.

XMLTemplates

2013-02-17T00:14:58Z

Olaph:

__NOTOC__




== XMLTemplates ==
'''Drafter: '''[[Vek]]

'''Drafters Email: ''' <<[[MailTo]](kevin DOT mitchell AT rackspace DOT com)>>

'''Status: ''' To be completed by POC

Currently, request extensions have to deserialize data, manipulate it, and reserialize it. This happens for each extension. This is generally not much of a problem for JSON data, but XML is expensive to deserialize and reserialize, and it is currently impossible to extend the XML without directly editing the serializer or writing an independent serializer.

This proposal is to establish new middleware which performs the XML serialization, and to also establish a means of creating XML templates, which can be extended by attaching slave templates to describe the extra data that an extension may wish to include.

Much of the code for this is already written, and many elements of nova-api converted over to use template-based serializers in preference to the hand-coded serializers currently utilized. This page is to document what has been done.

=== Lazy Serialization ===

The first and simplest part of the specification is lazy serialization. Basically, this involves the creation of a LazySerializationMiddleware which is added to the openstackapi pipelines, and some modifications to nova.api.openstack.wsgi—specifically, Response is modified to track a lazy_serialization attribute (defaulting to True), and ResponseSerializer's serialize_body() method is modified to store the actual serializer and action (and template, if available) into the request if lazy_serialization is True. The data itself is serialized as a JSON object in order to be passed through the webob pipeline (webob is designed to pass text response bodies, and extending it to allow raw objects to be passed between components would be difficult).

=== Extensions ===

Request extensions are also slightly modified; the loop that calls all the handlers for a particular request deserializes the body and passes that as a third parameter to all the handlers. The handlers can modify the body in place, and additionally modify the template in the wsgi environment (if present; for data requested to be encoded in JSON format, no template will be present in the wsgi environment).

=== XML Templates ===

The most complicated part of the proposal is the introduction of an XML template. XML templates can be constructed the same way an XML tree is constructed using the ElementTree interface of lxml; there exists a TemplateElement class and a SubTemplateElement() helper function that operate in a similar manner to ElementTree. The critical difference is the use of selectors; when an object is serialized against an XML template, the object is passed in, and a selector (basically, a simple callable) is used to select the data to operate on. There is a selector associated with each element, and selectors can also be attached as attribute values and as the text contents of the element; these selectors further refine the element's selector. Subelements will also be passed the object selected by their parent, and have the chance to further refine selection for their own purposes.

Once an element tree has been constructed, it can be wrapped up in a Template subclass. There are two subclasses available—MasterTemplate and SlaveTemplate. Both can serialize an object against the template directly, but MasterTemplate instances can additionally have SlaveTemplate instances attached which can further extend the serialization; the effect is as if the elements in the SlaveTemplate instances were merged with the elements in the MasterTemplate. Additionally, MasterTemplate instances have version numbers, and automatically exclude SlaveTemplate instances which do not apply to that MasterTemplate version (SlaveTemplate instances have a minimum version and an optional maximum version for this selection criteria). This makes it easy for extensions to modify a serializer to include their extended data.

The existing code also includes a TemplateBuilder class, which can be used to ensure that a given template only ever needs to be built once; and an XMLTemplateSerializer, which is similar to the existing XMLDictSerializer.

=== WSGI Environment ===

The lazy serialization potentially adds 3 new variables to the WSGI environment. (Note "potentially"—for JSON serialization, none of these variables are added.) The '''nova.serializer''' variable contains a reference to the body serializer to use (typically a subclass of XMLDictSerializer or, preferably, a subclass of XMLTemplateSerializer), and the '''nova.action''' variable contains the action to pass to the serializer's serialize() method. If the serializer has a get_template() method, the return value of that method will be assigned to the '''nova.template''' variable; extensions may attach their slave templates to this master template as necessary.

----
Blueprint: https://blueprints.launchpad.net/nova/+spec/xml-templates
----
[[Category:Proposal]]

SchedulerRaceReduction

2013-02-17T00:07:36Z

Olaph:

__NOTOC__




== [[SchedulerRaceReduction]] ==

'''Drafter: '''[[belliott]]

== Overview ==

The scheduler is subject to a race condition which can cause it to incorrectly identify available resources on a particular compute host. The problem occurs if multiple scheduler instances/threads concurrently issue an instance build request (i.e. ''run_instance'') to the same compute host. This situation may oversubscribe the given compute host and cause one or more ''run_instance'' requests to fail.

== Example ==

Compute host ''C'' has 3 GB of ram free.

# Scheduler ''A'' sends a ''run_instance'' request ''R1'' to ''C'' trying to build a 2GB instance.
# Scheduler ''B'' sends a ''run_instance'' request ''R2'' to ''C'' trying to build a 2GB instance.
# Assume processing of ''R1'' and ''R2'' begins concurrently on ''C''.

Obviously ''C'' cannot handle both requests, so at least 1 will fail.

== Impact ==

Instance build requests may fail, even if other compute hosts are available with free resources.

== Solution ==

* Compute hosts should have the final say over whether a ''run_instance'' request can be properly serviced. To this end, the compute host must be capable of identify whether it has free resources when a new ''run_instance'' request arrives.
* Compute hosts should serially verify resources available for ''run_instance'' requests to avoid concurrent competition by multiple callers.
* Schedulers should read the response to ''run_instance'' and possibly retry the request at a different compute host.

[https://blueprints.launchpad.net/nova/+spec/scheduler-resource-race Blueprint]

[https://bugs.launchpad.net/nova/+bug/1011852 Launchpad bug]

Nova/LibVirtVolumeMultipathiSCSI

2013-02-17T00:06:51Z

Olaph:

__NOTOC__




== Nova/LibVirtVolumeMultipathiSCSI ==

'''Contributor: '''[[erikzaadi]]

== Summary ==

Currently iSCSI Volumes are connected in LibVirt as single path devices (''/dev/disk/by-path/ip-IP:PORT-iscsi-IQN-lun-X'').

Most storage vendors support multipath connections via iSCSI, allowing more robust and faster volumes.

The iSCSI target can be inquired with the 'sendtargets' discovery method which returns all the available iSCSI ports.

The LibvirtISCSIVolumeDriver class should then login to all the discovered iSCSI targets and use multipath to rescan multipath connections.

After that, a new device will be available (usually ''/dev/dm-X'') which will be used as source device in LibVirt instead of ''/dev/disk/by-path/ip-IP:PORT-iscsi-IQN-lun-X''

The scope of changes intended by this blueprint includes:

* Altering nova/virt/libvirt/volume.py:LibvirtISCSIVolumeDriver's behaviour to deal with multipath if installed on the nova-compute node.
* Should still work if multipath isn't installed
* Should work if the storage provider doesn't support multipath.

=== Gerrit Review: ===
https://review.openstack.org/17946

Nova/LibVirtVolumeMultipathiSCSI

2013-02-17T00:06:16Z

Olaph:

__NOTOC__




== Nova/LibVirtVolumeMultipathiSCSI ==

'''Contributor: '''[[erikzaadi]]

== Summary ==

Currently iSCSI Volumes are connected in [[LibVirt]] as single path devices (''/dev/disk/by-path/ip-IP:PORT-iscsi-IQN-lun-X'').

Most storage vendors support multipath connections via iSCSI, allowing more robust and faster volumes.

The iSCSI target can be inquired with the 'sendtargets' discovery method which returns all the available iSCSI ports.

The LibvirtISCSIVolumeDriver class should then login to all the discovered iSCSI targets and use multipath to rescan multipath connections.

After that, a new device will be available (usually ''/dev/dm-X'') which will be used as source device in [[LibVirt]] instead of ''/dev/disk/by-path/ip-IP:PORT-iscsi-IQN-lun-X''

The scope of changes intended by this blueprint includes:

* Altering nova/virt/libvirt/volume.py:LibvirtISCSIVolumeDriver's behaviour to deal with multipath if installed on the nova-compute node.
* Should still work if multipath isn't installed
* Should work if the storage provider doesn't support multipath.

=== Gerrit Review: ===
https://review.openstack.org/17946

FlatManagerIPv6Support

2013-02-17T00:04:44Z

Olaph:

__NOTOC__
* '''Launchpad Entry''': [[NovaSpec]]:cactus-flatmanager-ipv6-support
* '''Created''':2 February, 2011
* '''Contributors''': Hisaharu Ishii, Nachi Ueno, Koji Iida, Tushar Patil

== Summary ==
IPV6 is already supported for FlatDHCP and VLAN network model in Bexar release. Add support to Flat network model also

== Release Note ==
TBD

== Rationale ==
* IPV4 address exhaustion

== User stories ==
== Assumptions ==
* Currently nova support FlatManager with single bridge that you specify in flat_network_bridge (br100 by default). This bridge needs to be created on all compute hosts by the user.
* Creating multiple networks is currently not supported
* If flat_injected is True, the compute host will attempt to inject network config into the guest. It attempts to modify /etc/network/interfaces and currently only works on debian based systems. To support a wider range of OSes, some other method may need to be devised to let the guest know which ip it should be using so that it can configure itself.

== Design ==
* Add database column netmask_v6 in the networks database table.
* Modify ra_server database column of networks tables to gateway_v6.
** Reason : When we implemented IPv6 support for FlatDHCP and Vlan network model, we added new ra_server column in the networks database table. However we think for Flat network model, this name ra_server column is not suitable. Because for all 3 network models, routers acts as a gateway which forwards the packets from inside and outside of the cloud and for only VlanManager or FlatDHCPManager, routers advertises RA messages. Then in generally, those routers should be called as not "RA server" but just "gateway".
* Populate netmask_v6 when you create networks using nova-manage network create commmand.
* update gateway_v6 column when network host is selected for a project. Get the ipv6 local link address of the bridge you have created on the network node manually and update this local link address in the gateway_v6 column of the network.
* Modify the interfaces.template to include following
** ###Start IPV6 static configuration
*** iface eth0 inet6 static
*** address %(address_v6)s
*** netmask %(netmask_v6)s
*** gateway %(gateway_v6)s
* ###END IPV6
** If use_ipv6 flag is set to True, then only the above IPV6 configuration will be injected in the /etc/network/interfaces of the VM instance.
* Modify libvirt_conn.py for filtering security groups

== Implementation ==
=== UI Changes ===
* There should be no visible changes to the end users, all this work will be behind the network/compute servers.

=== Code Changes ===
=== Migration ===
* Database migration is required for the newly added and modified columns to the networks and fixed_ips database tables.

== Test/Demo Plan ==
* Unit tests will be added as the code is developed. System testing will be on Ubuntu Lucid.

== Unresolved issues ==
None

== BoF agenda and discussion ==
----

[[Category:Spec]]

VMware-vSphere-support

2013-02-17T00:03:44Z

Olaph:

__NOTOC__
* '''Launchpad Nova blueprint''': [[NovaSpec]]:hypervisor-vmware-vsphere-support
* '''Created''': 19 December 2010
* '''Last updated''': 4 February 2010
* '''Contributors''': [https://launchpad.net/~ewanmellor Ewan Mellor], [https://launchpad.net/~ravi-gururaj Ravi Gururaj], [https://launchpad.net/~sateesh-chodapuneedi Sateesh Chodapuneedi]

== Summary ==
This blueprint proposes adding support to Nova for the use of VMware vSphere as a compute provider.

== Requirements ==
R1. Support VMware vSphere 4.1 as a compute provider within Nova. As the deployment architecture, support both ESX and ESXi. VM storage is restricted to VMFS volumes on local drives. Note that vCenter is not required by the current design, and is not currently supported. Instead, Nova Compute talks directly to ESX/ESXi.

R2. Integrate with Glance, so that VM images can be streamed from there for boot on ESXi.

R3. Support Nova's VLAN-based network model (VlanManager).

R4. Support Nova's flat networking model (FlatManager).

R5. When vSphere support is not in use, the presence of this code will add no additional dependencies to nova-compute as a whole. This implies that any new dependencies must be loaded on-demand.

R6. - Removed -

R7. CPU load from Nova on the compute host should be negligible when idle.

== Non-requirements ==
NR1. Shared storage support is excluded.

NR2. Live migration support is excluded.

NR3. Clustering support is excluded.

NR4. Nova is not required to handle any automatic installation or configuration of the underlying hypervisor. The user will be responsible for configuring vSphere appropriately.

NR5. Nova is not required to interrogate vCenter to find compute hosts. The user will be responsible for configuring Nova with the details for each compute host.

NR6. Nova is not required to cope gracefully when compute nodes are concurrently managed by automatic vCenter operations or other third parties. Nova may assume that it "owns" the compute node entirely.

NR7. Nova is not required to cope gracefully when compute nodes are subject to VMware Update Manager dynamic updates. A future blueprint will discuss the interaction between Nova's scheduler and update management on vSphere, to ensure that machines are drained of customer workloads before being upgraded.

NR8. Support for iSCSI volume attach/detach is a future feature, not on the plan for Cactus.

== Assumptions ==
A1. The [[OpenStack]] project will not ship any VMware products or SDKs. In the current design, there is no need for a SDK from VMware (we are using python-suds (version 0.4)to address VMware's SOAP API directly).

A2. Users will require licenses for VMware vSphere Essentials (or greater) for all "clouds" with up to three compute hosts, and VMware vSphere Standard (or greater) for all sensible clouds. See Licensing below.

A3. The QA emphasis will be on ESXi, rather than ESX, with the assumption that anything supported by ESXi is also supported by ESX.

== Rationale ==
Regarding A3's emphasis on ESXi: On http://www.vmware.com/products/vsphere/esxi-and-esx/overview.html, VMware state "vSphere 4.1 and its subsequent update and patch releases are the last releases to include both ESX and ESXi hypervisor architectures. Future major releases of VMware vSphere will include only the VMware ESXi architecture. For this reason, VMware recommends that deployments of vSphere 4.x utilize the ESXi hypervisor architecture." Following this, we shift the QA burden towards ESXi, as we expect this to be the deployment architecture selected in most cases.

Regarding R1's restriction to vSphere 4.1: this is to reduce the QA burden, as there is no known need to support older versions.

NRs 1, 2, 3, 6, and 7 have been added to avoid increasing the QA burden.

NRs 4 and 5 have been added to avoid increasing the development burden.

== Design ==
The requirement to support ESXi implies that nothing can be installed inside the console domain. Instead, we will install nova-compute inside a virtual machine, one per compute node. nova-compute will be given the credentials to the local ESXi (or the controlling vCenter), and with these credentials it will perform all necessary actions through the VMware vSphere API over SOAP (see [http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/index.html VMware vSphere API Reference Documentation] and [http://www.vmware.com/support/pubs/sdk_pubs.html VMware APIs and SDK Documentation]). [http://www.sourceforge.net/projects/python-suds/ python-suds] will be used to make the SOAP calls.

Disk streaming from Glance will also be handled within this VM. The VM will ask for a new virtual disk to be created, and streams the image, using HTTP calls, from Glance server to the virtual disk inside ESXi server. Once the streaming is finished, that disk will be available for the new customer VM.

The connection from Nova to vCenter/ESXi will be through a nova.virt.vmwareapi_conn module, parallel with nova.virt.libvirt_conn and nova.virt.xenapi_conn (i.e. not through libvirt).

== QA ==
Unit testing will be of similar style to that used by XenAPI, with a mock ESX stubbed in at the RPC layer (i.e. below nova.virt.vmware_conn).

The system will be certified using a single host, 7 day stress test:

* Spawn an instance using a random image from Glance server, on the spawned instance, perform VM life cycle operations as listed below
** 1. Reboot the instance and then sleep for sometime.
** 2. Suspend the instance and check for the state to change to “paused” and then sleep for sometime.
** 3. Resume the instance and check for state change to “active” and then sleep for sometime.
** 4. Snapshot the instance and check for image being uploaded to the glance host. Deletes the image from the Glance host (to avoid running out of space on glance host) and sleep for sometime.
** 5. Terminate the instance and check if the instance doesn’t find a listing in the servers’ list.
* Scale: 32 x 1GB VMs, each with a 10GB HDD * up to 2 VMs at a time being deployed

The system test must succeed with zero failures in those 7 days: i.e. no failures to deploy, undeploy, power-on, or power-off VMs, and no nova-compute crashes.

The test results collect so far can be seen at [[attachment:System Testing Output of 48 hours]]

== Future ==
We need to figure out how machines can be updated automatically. Presumably, we want to drain a machine of customer workloads before installing updates and rebooting. It is an open question how best to integrate this with VMware Update Manager.

Support for iSCSI volume attach/detach.

== Development Resources ==
Citrix has committed a development team to implement the functionality specified here. Prototyping is underway, and will be pushed to Launchpad soon.

No commitment has been made with regard to any "Future" features.

== Licensing ==
=== Why we can't use vSphere Hypervisor ===
From http://www.vmware.com/download/eula/esx_esxi_eula.html:

"3. GRANT AND USE RIGHTS FOR SOFTWARE

"3.3 Restrictions. You may not (i) sell, lease, license, sublicense, distribute or otherwise transfer in whole or in part the Software or the Software License Key to another party; (ii) provide, disclose, divulge or make available to, or permit use of the Software in whole or in part by, any third party (except Designated Administrative Access) without VMware’s prior written consent; (iii) modify or create derivative works based upon the Software; or (iv) create, develop, license, install, use, or deploy any third party software or services to circumvent, enable, modify or provide access, permissions or rights which violate the technical restrictions of the Software, any additional licensing terms provided by VMware via product documentation, notification, and/or policy change posted at www.vmware.com, and the terms of this EULA."

Also, from http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1023990:

"vCLI, PowerCLI, and vSphere SDK for Perl are limited to read-only access for the free vSphere Hypervisor edition. To enable full functionality of vCLI on a VMware ESXi host, the host must be licensed with vSphere Essentials, vSphere Essential Plus, vSphere Standard, vSphere Advanced, vSphere Enterprise, or vSphere Enterprise Plus."

We understand this to mean that vSphere Hypervisor (previously known as "Free ESXi") may not be managed programmatically in a read-write manner, since VMware's own tools make that restriction. See also http://www.veeam.com/blog/veeam-and-free-esxi.html and http://www.virtualizationpractice.com/blog/?p=274.

=== Why we can't use vSphere Essentials or Essentials Plus for more than three compute nodes ===
From http://www.vmware.com/download/eula/esx_esxi_eula.html:

"9. SOFTWARE PRODUCT SPECIFIC TERMS AND CONDITIONS

"9.1 ESX/ESXi

"c) VMware vSphere Essentials and VMware vSphere Essentials Plus

"If you have licensed VMware vSphere Essentials or VMware vSphere Essentials Plus, the following additional license terms apply:

"VMware licenses VMware vSphere Essentials or VMware vSphere Essentials Plus editions (collectively "Editions") to you solely for managing up to three (3) server hosts and your use of these Editions is limited to servers with up to two processors. For these Editions, the server hosts must be managed by the VMware vCenter Server that is provided with these Editions, and that same VMware vCenter Server cannot be used to manage other server hosts not included with these Editions."

We understand this to mean that it is forbidden to use Essentials or Essentials Plus for any installation larger than three hosts.

== Release Note ==
Nova may now use VMware vSphere as a compute provider. Docs for developers are available at http://nova.openstack.org/vmwareapi_readme.html.

== Risk Assessment ==
The code changes are encapsulated in a module named vsphereapi that fits inside nova.virt (hypervisor abstraction layer). The risk of destabilization is "low" because the changes are well contained in vsphereapi module, and the bulk of the code will be executed only if the connection type is set to 'vsphereapi_conn', which is the case of VMware ESXi server as backend. The changes to the existing codepaths is minimal.

== Discussion ==
----

[[Category:Spec]] [[Category:Spec]]

NetworkServicePOC

2013-02-17T00:02:30Z

Olaph:

* '''Launchpad Entry''': https://code.launchpad.net/~ntt-pf-lab/nova/network-service
* '''Created''': [https://launchpad.net/~ishii-hisaharu Hisaharu Ishii]
* '''Contributors''': [https://launchpad.net/~ishii-hisaharu Hisaharu Ishii]

== Summary ==
In the current Nova implementation, there are three types of network services: VLAN, flat, and flat DHCP. For each deployment of Nova, only one of these services can be configured to be used at a time. While the current network architecture in Nova works fine with this design, we would like to propose a more flexible design in which these network services are implemented more modularly, allowing them to be 'plugged-in' to Nova. With this design, it would allow third-party network services to be easily integrated into Nova.

In this POC implementation, we will try to solve the following issues in the current networking implementation:

# ''Strong coupling of compute and network services'': In the current implementation, there are fairly large amount of network-related code that exist inside the compute code. This coupling makes it difficult swap the existing services with custom ones.
# ''Association of networking service at the deployment/application level'': Networking services should be associated at the level of smallergranularity so that within one deployment of Nova, it would be possible to have multiple types of networks.
# ''Lack of L2 support'': Because the currently available networking services are all L3-based networking, there is no option to create an L2-based network. By making network services pluggable, it would make it easier to implement and use an L2(or any type of) network.

Finally, it should be noted that despite these changes, all the currently available features should work the same way as they do now.

== Rationale ==
Network services should be implemented in a pluggable manner so that various types of network service can be used with Nova.

== Assumptions ==
* Multiple virtual NICs per instance are supported.
* EC2 API does not support multiple NICs.

== Design ==
=== Network Service POC Overview ===
[[Image:openstack-network-service-apis-generic-vs-specific.png]]

A pluggable network service is typically broken up into the following three parts:

# '''net-api''': REST APIs for network management. Each plugin is responsible for providing a management tool(just like nova-manage) to manage its networks.
# '''net-agent''': Pluggable module that defines generic interfaces called by the compute service. These APIs are meant to be run on the compute node.
# '''net-service''': Pluggable service can optionally implement a network service, which is equivalent to nova-network service in the current Nova implementation, that handles various network tasks like DHCP, NAT, and IP allocation.

Management of the networks, such as creating networks, allocating IPs and virtual NICs are all accomplished through '''net-api'''. All the network-related code that currently exist in the compute code are moved to '''net-agent''', thereby decoupling compute from the network services. Since '''net-service''' is optional, Nova does not have any knowledge of its existence. Only '''net-api''' and '''net-agent''' may access '''net-service'''. For example, for '''nova-compute''' to request a service from '''net-service''', it must go through '''net-agent'''.

=== Nova Network Service Plugins ===
Nova currently comes with three types of network managers: ''VlanManager,'' ''FlatManager'' and''FlatDHCPManager''. These are converted into separate plugins in the new model. These plugins are defined in 'nova/networks' directory as distinct modules. The actual paths of these plugin modules are:

* '''VlanManager''': ''nova.network.vlan''
* '''FlatManager''': ''nova.network.flat''
* '''FlatDHCPManager''': ''nova.network.flat_dhcp''

=== [[OpenStack]] Networking Concept ===
The networking in [[OpenStack]] consists of the following main concepts:

* '''network''': Represents a logical network but does not specify whether it's L3 or L2.
* '''virtual NIC(VNIC)''': Represents a virtual network interface card that is attached to a VM instance. One common example of VNIC is an ethernet card.
* '''port''': Represents a virtual socket to plug the VNIC into for network connectivity. A port belongs to a particular network.

It is assumed that a VM instance is allowed to have multiple VNICs associated with it, with each one connecting to a port that belongs to different networks.

=== Project-Network Service Association ===
A project is associated with a network service. A project cannot be associated with more than one network service. By default, every project is associated with a network service defined by the flag, ''--default_network_service'', where the value is the path to the plugin module. If this flag is not set, it defaults to VLAN(''nova.network.vlan''). New ''nova-manage'' commands are provided to help manage the association.

=== Network Service Configuration File ===
A list of available network services are listed in a network service configuration file. The location of this file is defined in the flag, ''--network_service_conf''. If this flag is not defined, the default is ''/etc/nova/nova-network.conf''. The format of this file is simply a list of Python module paths of the network services. Any blank line and or one that starts with '#' is ignored.

Example:

<pre><nowiki>
# Example network services
nova.network.flat
nova.network.flat_dhcp
nova.network.vlan
</nowiki></pre>

=== Network Management REST API ===
All the network management is done through the REST API implemented by each plugin. In this proposal, only the [[OpenStack]] API is applicable. The EC2 API still works the same way as before, but it does not support multiple NICs for an instance.

Each plugin has its own REST API URL that contains its name in the path. The format of the URL is: '''http://server:port/version/plugin_name/...'''

Flat, FlatDHCP and VLAN define the same APIs. They are:

* Create Network (and the IP addresses for the network)
** URL: '''http://server:port/version/plugin_name/networks'''
** Verb: POST
** Request parameters are the same as those for the 'nova-manage network create' command.
** Returns a list of network IDs created with status 200.
* Create virtual NIC
** URL: '''http://server:port/version/plugin_name/vnics'''
** Verb: POST
** Returns the ID of the new virtual NIC with status 200.
* Delete virtual NIC
** URL: '''http://server:port/version/plugin_name/vnics/vnic_id'''
** Verb: DELETE
** Returns status 200.

The request and response formats are up to the plugins. They should, however, support both JSON and XML.

The routes for these APIs are set by the plugins themselves. The way this is accomplished is explained in the implementation section below.

Note: There is work being done for the new [[OpenStack]] Compute API(version 1.1). It is important to keep the network APIs in sync with the new design. (At the time of this writing, it is NOT in sync).

=== Data Store ===
Nova's database keeps the following data:

* Association of projects and network services
* Association of instances and virtual NICs

The actual data for virtual NICs, networks and ports are all plugin-specific, and therefore they belong in the data storage. If the plugin uses SQL as its data store, then it is the plugin's responsibility to define the migration scripts, data models and DB APIs.

For VLAN, Flat and FlatDHCP, a virtual NIC is an ethernet card, a port is a fixed IP, and a network is the same as that of the current Nova implementation.

* Nova is aware of the existence of Networks and VPorts, where VPorts belong to a Network. VPorts represent ports in which you plug the VNICs into for network connectivity. Network is a group of VPorts that could represent either L2 or L3 network.
* Compute's API takes in a list of list of VNIC IDs to associate with the instances that it is creating.
* [[OpenStack]] API to create a new instance should have a newly defined API that takes in VNICs as a mandatory parameter. An exception should be thrown if VNICs are not supplied or invalid.
* The legacy [[OpenStack]] API to create a new instance(that does not accept VNICs as a parameter), creates a new VNIC and calls the new create VM API with this VNIC as a parameter.
* EC2 API creates a new VNIC for each instance to be created, and calls the Compute's API to create these instances.
* During the VM instantiation, the compute code does not make any RPC call to the network node, and does not perform any task related to networking. Any network-related code that gets executed on the compute node is defined in a separate module called Network Agent.
* Network Service (the service that runs on the network node) is implemented by each plugin. Only Network Agent, which is also implemented by the plugin, can access its own Network Service.
* A generic API to get the IP address for a given VNIC is defined by the Network Agent for file injection.
* A generic API to generate libvirt XML network interface definitions is defined by the Network Agent for generating the libvirt domain XML file.
* Firewall logic is moved from Compute to Network Agent.
* A generic API is defined by Network Agent to bind a VNIC to a VPort(in most cases, this is just assigning an IP address).

== Implementation ==
=== Nova-manage script changes ===
''nova_manage'' script has new Project commands for network service management:

* '''nova-manage project network_set''' ''project_id'' ''network_service''
** Sets a network service module path, ''network_service'', to a project with ID, ''project_id''. If the project already has a network service set, it overwrites it.
* '''nova-manage project network_get''' ''project_id''
** Gets the network service that is associated with the project with ID, ''project_id''. Returns None if no network service is associated.

network commands in ''nova-manage'' are obsolete as all the network management commands are moved to the plugin-specific management tool scripts.

=== Network Service Classes ===
* '''NetworkServiceManager'''
** Defined in ''nova.network.service'', ''NetworkServiceManager'' class handles the loading of the available network service plugins, and provides access to these modules to other Nova services. The configuration file to load the network service is read only once, and these modules are stored in memory.
* '''NetworkServiceRouteMap'''
** ''NetworkServiceRouteMap'' class is defined in ''nova.network.service'', which is a wrapper class to python-route map object that is responsible for adding the appropriate prefix to all the REST API routes. Even though the routes for the API are set by the plugins themselves at the initialization of [[OpenStack]] API module, the route prefix, however, which includes the name of the plugins, is set by this class.

=== Database Changes ===
In the Nova DB, the following new tables are added:

'''ProjectNetworkServiceAssociation'''

<pre><nowiki>
+ id (PK)
+ project_id
+ network_service
+ deleted, created_at, deleted_at, ...
</nowiki></pre>

'''InstanceVirtualNicAssociation'''

<pre><nowiki>
+ id (PK)
+ instance_id(FK - instances.id)
+ virtual_nic_id
+ deleted, created_at, deleted_at, ...
</nowiki></pre>

In the VLAN, FlatDHCP and Flat plugins have EthernetCards table which represents the VNIC:

'''EthernetCards'''

<pre><nowiki>
+ id (PK)
+ mac_address
+ deleted, created_at, deleted_at, ...
</nowiki></pre>

Also, for all the built-in plugins, ''networks'' table is ported over from Nova DB. For VLAN and FlatDHCP, ''fixed_ips'' and ''floating_ips'' tables are copied over. For Flat, ''fixed_ips'' table is ported over and renamed to ''ip_addresses''.

=== Plug-in Generic API ===
As mentioned earlier, each plugin consists of several components, and some of these components are required by Nova. The actual implementation of these components are up to the plugins, but the plugins must define generic APIs to return these modules/classes to Nova.

These generic APIs are:

* get_os_service()
** Returns the module/class that implements generic APIs needed by Nova's [[OpenStack]]/EC2 APIs.
* get_os_api_service()
** Returns the module/class that implements generic APIs needed by Nova's [[OpenStack]] API.
* get_net_agent()
** Returns the module/class that implements generic APIs needed by Nova's compute.

=== Net Agent Generic API ===
All the net agent APIs are generic.

* get_default_vnic_id() : vnic_id
** Gets the default VNIC for the plugin. This API must be implemented by all plugins because the current [[OpenStack]]/EC2 API to launch an instance does not require a parameter for VNICs, which means a VNIC must be created dynamically at the time of VM instantiation. This API is called by the Compute API class, and is defined by the plugin's API module/class returned from ''get_api_service()''. Flat, FlatDHCP and VLAN plugins simply create a new ethernet card. It returns the new VNIC ID if successful, and None if it is not.
* get_project_vpn_address_and_port(project_id) : (vpn_ip, vpn_port)
** Gets a tuple of VPN IP address and VPN port for a given project. This is only applicable if the service has any VPN data(like VLAN plugin). It is used to construct the credential for the user.
* bind_vnics_to_ports(vnic_ids)
** For a list of ''vnic_id''s, the API binds them to the ports if they are not already bound. This binding activates the network connectivity. It is up to the plugin to determine which ports that these VNICs should be plugged into. It does not return anything. It is implemented by the net agent.
* setup_compute_network(vnic_ids)
** This method must be implemented to do any networking setup in the compute node, at the time of VM launch.
* teardown_compute_network(vnic_ids)
** This method must be implemented to perform any clean up on the compute host. This method gets called when the VM terminates.
* requires_file_injection(vnic_id) : True/False
** Returns True if the network service associated with ''vnic_id'' requires injecting network information to the ''/etc/network/interfaces'' file. This is implemented by the net agent.
* get_network_info(vnic_id) : (dictionary: cidr, cidr_v6, netmask, netmask_v6, gateway, gateway_v6, dhcp_server, broadcast, dns, ra_server, mac_address, ip_address, and address_v6)
** Gets network data associated with a VNIC with ID of ''vnic_id''. The return value is a dictionary of network data for the VNIC. There values can be None if they don't apply to that plugin. It is defined in net agent, and is required to be implemented for file injection, Iptables setup, and libvirt XML creation.

== Test/Demo Plan ==
* Unit tests will be added as the code is developed.

== Unresolved issues ==
* Network service components, such as DHCP and Firewall, should be pluggable and optional.
* Firewall logic needs to be refactored. The actual implementation should be plugin-specific, so it should not exist inside the compute/libvirt code as it does now.
* Project-network service association commands should be more user-friendly with validations and sanity checks.
* REST API should check that the project that is sent in the request matches the network service that the URL is asking for.
* Make the Network Management API follow the format of the new Compute API(1.1).
* Integrate with the multi-nic implementation currently underway.
* Consider the possibility of joining the three current network managers into one plugin.

----
[[Category:Spec]]

TroubleshootingNova

2013-02-16T23:58:40Z

Olaph:

= Troubleshooting Tips for [[OpenStack]] Compute (Nova) =

== Starting Points ==

Most of the installation instructions are written for Ubuntu, but there is also information about Centos and Fedora.

You can also check the [https://answers.launchpad.net/nova Answers area of the Launchpad site for Nova] to find asked and answered questions.

== Log Files ==

Log files are stored in /var/log/nova and there is a log file for each service, for example nova-compute.log. You can format the log strings using flags for the nova.log module. The flags used to set format strings are: logging_context_format_string and logging_default_format_string. If the log level is set to debug, you can also specify logging_debug_format_suffix to append extra formatting. For information about what variables are available for the formatter see: http://docs.python.org/library/logging.html#formatter

You have two options for logging for [[OpenStack]] Compute based on configuration settings. In nova.conf, include the --logfile flag to enable logging. Alternatively you can set --use_syslog=1, and then the nova daemon logs to syslog.

== Common Errors ==

The Launchpad Answers site offers a place to ask and answer questions, and you can also mark questions as frequently asked questions. This section describes some errors people have posted to Launchpad Answers and IRC. We are constantly fixing bugs, so online resources are a great way to get the most up-to-date errors and fixes.

=== Credential errors, 401, 403 forbidden errors ===

A 403 forbidden error is caused by missing credentials. Through current installation methods, there are basically two ways to get the novarc file. The manual method requires getting it from within a project zipfile, and the scripted method just generates novarc out of the project zip file and sources it for you. If you do the manual method through a zip file, then the following novarc alone, you end up losing the creds that are tied to the user you created with nova-manage in the steps before.

When you run nova-api the first time, it generates the certificate authority information, including openssl.cnf. If it gets started out of order, you may not be able to create your zip file. Once your CA information is available, you should be able to go back to nova-manage to create your zipfile.

You may also need to check your proxy settings to see if they are causing problems with the novarc creation.

=== Cannot Ping or SSH to an Instance ===

Sometimes a particular instance shows "pending" or you cannot SSH to it. Sometimes networking settings are the problem. Sometimes the image itself is the problem.

<pre><nowiki>#!rst

One of the most commonly missed configuration areas is not allowing the proper access to VMs. Use the 'euca-authorize' command to enable access. Below, you will find the commands to allow 'ping' and 'ssh' to your VMs::

euca-authorize -P icmp -t -1:-1 default
euca-authorize -P tcp -p 22 default

Another common issue is you cannot ping or SSH your instances after issuing the 'euca-authorize' commands. Something to look at is the amount of 'dnsmasq' processes that are running. If you have a running instance, check to see that TWO 'dnsmasq' processes are running. If not, perform the following::

killall dnsmasq
service nova-network restart

</nowiki></pre>

With recent builds of Nova, IPv6 configuration is allowed, but if you cannot SSH to an image, add --use_ipv6=false to your nova.conf.

For example, when using flat manager networking, you do not have a dhcp server, and an ami-tiny image doesn't support interface injection so you cannot connect to it. The fix for this type of problem is to use an Ubuntu image, which should obtain an IP address correctly with FlatManager network settings. To troubleshoot other possible problems with an instance, such as one that stays in a spawning state, first check your instances directory for i-ze0bnh1q dir to make sure it has the following files:

* libvirt.xml
* disk
* disk-raw
* kernel
* ramdisk
* console.log (Once the instance actually starts you should see a console.log.)

Check the file sizes to see if they are reasonable. If any are missing/zero/very small then nova-compute has somehow not completed download of the images from objectstore.

Also check nova-compute.log for exceptions. Sometimes they don't show up in the console output.

Next, check the /var/log/libvirt/qemu/i-ze0bnh1q.log file to see if it exists and has any useful error messages in it.

Finally, from the instances/i-ze0bnh1q directory, try virsh create libvirt.xml and see if you get an error there.

Also, when setting up nodes in [[FlatManger]], be sure to enable ipforward or none your node instances will be able to ping out.

<pre><nowiki>
/etc/sysctl.conf
Net ipv4 ip_forward = 1
</nowiki></pre>

=== Slow Running VMs ===

Why is my VM running slow as molasses? There is a permissions issue where /dev/kvm is not writable, and the VM's drop into qemu mode. This causes the slowdown, and can be addresses by the following:

Verify with:

<pre><nowiki>
cat /var/log/libvirt/qemu/(internal_id).log | grep -i kvm</nowiki></pre>

In the results, look for:

<pre><nowiki>
open /dev/kvm: Permission denied
Could not initialize KVM, will disable KVM support
</nowiki></pre>

Fix with:

<pre><nowiki>
chgrp kvm /dev/kvm
chmod g+rwx /dev/kvm
</nowiki></pre>

=== Filesystem Not Found on Custom-created Images ===

When creating custom images, I get a / (root) filesystem not found error.

This is likely due to your image having / (root) as /dev/sda1, etc…..this needs to be /dev/vda1 for the image to boot properly.

Why /dev/vda1?

In the libvirt xml templates it specifies the virtual drives be made as "vdX" only.

So, vda1 would be the first one generated if you don't have advanced partitioning, and OpenStack can't boot the image.

=== Nova services (nova-api) not starting ===

Nova-api is not starting, so therefore, I can't start nova-compute.

When I try a restart of the nova-api service, such as:

<pre><nowiki>
root@ubuntu2:~# service nova-api restart
</nowiki></pre>

I get a response like so:

<pre><nowiki>
pidfile /var/run/nova/nova-api.pid does not exist. Daemon not running?
Initialized with method overriding = True, and path info altering = True
DEBUG:routes.middleware:Initialized with method overriding = True, and path info altering = True
Initialized with method overriding = True, and path info altering = True
DEBUG:routes.middleware:Initialized with method overriding = True, and path info altering = True
Initialized with method overriding = True, and path info altering = True
DEBUG:routes.middleware:Initialized with method overriding = True, and path info altering = True
Initialized with method overriding = True, and path info altering = True
DEBUG:routes.middleware:Initialized with method overriding = True, and path info altering = True
(7030) wsgi starting up on http://0.0.0.0:8774/
(7030) wsgi starting up on http://0.0.0.0:8773/
</nowiki></pre>

In this case, you should set the daemonize flag to 1 (true) in /etc/nova/nova.conf, like so:
--daemonize=1

Restart the services with this setting enabled and you should be able to start all the services again.

=== Errors with IP addresses for VMs ===

When you see that nova-network or nova-compute shows errors with the IP assigned to the virtual machines themselves, you see something like this:

<pre><nowiki>
2010-12-05 21:48:40-0600 [-] nova.exception.ProcessExecutionError: Unexpected error while running command.
</nowiki></pre>

You can verify with this command:

<pre><nowiki>
sudo -E dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.0.1 --except-interface=lo --dhcp-range=192.168.0.3,static,120s --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro

</nowiki></pre>

where the listen-address is the address that you see errors when sending commands to it.

To verify that this is your error, you should see this type of response after sending the dnsmasq command:

<pre><nowiki>
2010-12-05 21:48:40-0600 [-] Exit code: 2
2010-12-05 21:48:40-0600 [-] Stdout: ''
2010-12-05 21:48:40-0600 [-] Stderr: '\ndnsmasq: failed to bind listening socket for 192.168.0.1: Address already in use\n'
</nowiki></pre>

Fix with these commands:

{{
--killall dnsmasq
--service nova-network restart
</nowiki></pre>

Reboot instances if problem persists.

=== Database Locks ===

If you are not running Nova as a root user, you may get errors that the database is locked, first from nova-scheduler, next from nova-compute. Here is an example:

<pre><nowiki>
OperationalError: (OperationalError) database is locked
</nowiki></pre>

Check your permissions - if you installed as root but are trying to run Nova as another user, these errors may appear.

ReleaseNotes/Bexar

2013-02-16T23:57:43Z

Olaph:

= Release Notes, Bexar =

The Bexar release introduces large file support for OpenStack Object Storage (Swift), the OpenStack Image registry and Delivery service (Glance) and a lot of new features in OpenStack Compute (Nova).

== New Features ==

=== [[OpenStack]] Object Storage (Swift) ===
* Large objects (greater than 5 GB) can now be downloaded using OpenStack Object Storage. While there is still a limit on the size of a single uploaded object, with this release the download size of a single object is virtually unlimited with the concept of segmentation. Refer to [http://docs.openstack.org/openstack-object-storage/admin/content/ch04s08.html Managing Large Objects] for more information.
* An experimental S3 compatibility middleware has been added to OpenStack Object Storage. This middleware intercepts S3 style requests and authorization, and transforms them into swift requests.
* Initial i18n support for logging messages.
* Swauth is a swift compatible authentication and authorization service implemented on top of swift as wsgi middleware. This will replace the dev_auth service in a future release.
* Advanced rate limiting middleware

=== [[OpenStack]] Compute (Nova) ===
* Support for raw disk images with libvirt and XenAPI hypervisors, without the complexity of a separate ramdisk or kernel image.
* IPv6 support in all network modes but FlatManager. Use the new use_ipv6 flag, and refer to [[BexarIpv6supportReadme]] for more information.
* Support for a lot of new volume backends to provide highly available block volumes for VMs: Sheepdog, CEPH/RADOS, and iSCSI (XenAPI only)
* Microsoft Hyper-V support. Refer to [[HypervInstall]] for information.
* Lots of new features have been added around the Openstack API, for example admin features to pause, suspend, lock and password reset instances, but also support for per-instance diagnostics.
* New "rescue" mode allowing an instance to mount affected disks and fix problems (see rescue_image_id, rescue_kernel_id and rescue_ramdisk_id flags)
* Web-based serial console to access instances where networking fails (requires instances with serial console enabled). This is available through the Openstack API or the new euca-get-ajax-console tool (and introduces a new nova-ajax-console-proxy type of node)
* Possibility to do hardware staging: new added services can be specifically load-tested before being made generally available to cloud users (administered by the nova-manage service commands)
* Database versioning and migration support, for painless migration from one version to another
* Instances now use copy-on-write by default for better performance (you can use the use_cow_images flag to control that)
* Support for availability zones, through the introduction of a new scheduler: ZoneScheduler
* A DirectAPI, using introspection to exhibit features, is now available for developers to control Nova locally
* Some features that were partly implemented in the Austin release got finalized in Bexar: IP allocation was moved down the chain, project VPNs are supported and Nova now fully supports iptables-driven security groups.
* Finally, lots of efforts were spent unifying the code around common standards for using the new Glance clients, for i18n, logging, service handling (through eventlet), or using paste.deploy for the API nodes.

=== [[OpenStack]] Image Registry and Delivery service (Glance) ===
* Glance APIs (for registry and delivery) were unified, and a specific client class created
* Support for uploading disk images directly through the glance REST-ful API
* Addition of the glance-upload tool which can register new AMI-like images or raw disk images
* Glance can now fetch image data on a S3-like backend.
* Documentation for Glance is now available at http://glance.openstack.org
* The dependency on Twisted was removed. Glance now uses only Eventlet for its server-side internals.

=== Others ===

There were also deliveries outside those core projects during the Bexar development timeframe, in particular:

* The OpenStack Dashboard, a reference implementation of a web-based console for OpenStack Compute projects is now available. Refer to [[OpenStackDashboard]] for more information.
* A deployment tool to deploy Nova on multiple servers. Refer to [[NovaInstall/NovaDeploymentTool]] for more information.
* An official documentation site at http://docs.openstack.org, containing PDF and HTML versions of manuals plus the capability for user notes on each page.

== Known Issues and Limitations ==

=== Nova ===
* IPv6 is not supported in FlatManager network mode.
* You can't upload images to the [[S3ImageService]] (objectstore) from the Openstack API (only Glance is supported), see Bug 709355
* You can't use FlatDHCPManager network mode on a multiple machines deployment without using multiple interfaces (Bug 710959). Recommended mode of operation for deploying Nova on multiple machines is to use VLAN-supporting managed switches with VlanManager network mode.
* When a compute node dies, its instances may still be reported as running (Bug 661214)
* When a VM dies, a compute node won't realize until it's restarted (Bug 661260)
* Using "setup.py install" does not produce a complete setup (Bugs 683137 and 727794)
* When using XenAPI, the compute node doesn't clean up if a vm fails to spawn (Bug 694935) or if it loses its xapi session (Bug 692994)
* nova-network may fail to start if iptables entries were flushed and a floating IP address is assigned (Bug 711948). You can use the workaround in the bug.

=== Glance ===
* The S3 and Swift backends do not currently support the POST /images/ API command in the Glance API. These backends only currently support fetching disk images via GET calls. Support for storing disk images in S3 and Swift directly through the Glance API is planned for the Cactus release
* There is a bug in the Swift backend that is preventing new-style account:user:key URIs from being used (Bug 717431)
* Some errors/exceptions returned via from the reference registry server implementation (glance-registry) are being improperly consumed by the glance-api server (Bug 704854)

== Blueprints implemented during the Bexar release ==

* Swift: [https://blueprints.launchpad.net/swift/1.2 List of blueprints completed for the Bexar release]
* Nova: [https://blueprints.launchpad.net/nova/bexar List of blueprints completed for the Bexar release]
* Glance: [https://blueprints.launchpad.net/glance/bexar List of blueprints completed for the Bexar release]

== Bugs fixed during the Bexar release ==

* Swift: See the bug list at: https://launchpad.net/swift/1.2/1.2.0
* Nova: See the bug list at: https://launchpad.net/nova/bexar/2011.1
* Glance: See the bug list at: https://launchpad.net/glance/bexar/0.1.7

Obsolete:BexarIpv6supportReadme

2013-02-16T23:54:31Z

Olaph:

<pre><nowiki>#!wiki caution
'''This page is outdated'''

The content of this page has not been updated for a very long time. Sections of this page are incorrect when referring to the current release.
</nowiki></pre>

= [[OpenStack]] Compute (Nova) IPv4/IPv6 dual stack support =

* Date: Dec. 24, 2010 (Updated: Jan. 14, 2010)
* Author: NTT Information Sharing Platform Laboratories

== Release Note ==
In IPv4/IPv6 dual stack mode, instances can use both IPv4 and IPv6
addresses for communication. In IPv4/IPv6 dual stack mode, instances
can acquire their IPv6 global unicast address by stateless address
autoconfiguration mechanism [RFC 4862/2462]. IPv4/IPv6 dual stack mode
works with VlanManager, and FlatDHCPManager. In
VlanManager, different 64bit global routing prefix is used for each
project. In FlatDHCPManager, one 64bit global routing
prefix is used for all instances.

== 1. Pre-requirement ==
* We tested following configuration only.
** Host OS: Ubuntu 10.04
* VM Image requirement:
** IPv6 stateless address autoconfiguration capability is required for each VM.
* Additional python module requirement:
** python-netaddr is required for all nodes which executes nova services.

<pre><nowiki>
$ sudo apt-get install -y python-netaddr
</nowiki></pre>

* Additional software requirement:
** radvd is required on network node

<pre><nowiki>
$ sudo apt-get install -y radvd
</nowiki></pre>

* On network node, you have to type following commands

<pre><nowiki>
$ sudo bash -c "echo 1 > /proc/sys/net/ipv6/conf/all/forwarding"
$ sudo bash -c "echo 0 > /proc/sys/net/ipv6/conf/all/accept_ra"
</nowiki></pre>

== 2. Setting up of nova services ==
If you want to use IPv6 function, please set use_ipv6 flag to True. If not, set this flag to false.
(Please see [[NovaAdminManual]] to know how to configure your services.)

<pre><nowiki>
--use_ipv6
Enable IPv6 dual stack support.
Default: False
</nowiki></pre>

== 3. API changes ==
In this release, some API parameters and responses are modified to
adopt IPv6 feature.

* DescribeInstancesV6
Almost same API as original descibeInstances. You can see IPv6 address of an instance by using this API.
We add dnsNameV6 element for result xml.

* AuthorizeSecurityGroupIngress
You can specify both IPv4 and IPv6 value for cidrIp input parameter.

* RevokeSecurityGroupIngress
You can specify both IPv4 and IPv6 value for cidrIp input parameter.

* DescribeSecurityGroups
You can see both IPv4 and IPv6 value in cidrIp element.

== 4. Support library ==
We extend boto library to [[OpenStack]] ipv6 support.
You can use DescribeInstancesV6 and dnsNameV6 by using boto_v6 library in contrib directory.

== 5. nova-manage command specification ==
New parameter fixed_range_v6 is added to command
'nova-manage network create'.

<pre><nowiki>
nova-manage network create fixed_range num_networks network_size
[vlan_start] [vpn_start] [fixed_range_v6]
</nowiki></pre>

You can set ipv6 global routing prefix by fixed_range_v6 (default: fd00::/48). When you
use FlatDHCPManager, the command uses the original value of fixed_range_v6. When you use [[VlanManager]], the command creates prefixes of subnet
by incrementing subnet id. Guest VMs uses this prefix for generating
his ipv6 global unicast address.

usage example for VlanManager:

<pre><nowiki>
nova-manage network create 10.0.1.0/24 3 32 100 1000 fd00:1::/48
</nowiki></pre>

usage example for FlatDHCPManager:

<pre><nowiki>
nova/bin/nova-manage network create 10.0.2.0/24 3 32 0 0 fd00:1::/48
</nowiki></pre>

Note that [vlan_start] [vpn_start] is not used by FlatDHCPManager

== 6. Supported NWManager ==
* FlatDHCPManager
* VlanManager

== 7. Current limitations ==
* Floating IP feature using IPv6 address is not supported.
* Not tested with [[OpenStack]] API.
* FlatManager is currently not supported.
* VM images which does not use EUI-64 address for stateless address
autoconfiguration will not work correctly in current IPv6
support. Because current IPv6 support supposes that VM instances
generate and configure their IPv6 address based on EUI-64
specification (namely, their MAC address).

== 8. Architecture summary of IPv6 support implementation ==
* In current IPv6 support implementation, IPv6 address assignment is
done by stateless address autoconfiguration [RFC 4862/2462]. For this
purpose, nova-network configures radvd for each VLAN.

== 9. Contributors ==
* Nachi Ueno
* Hisaharu Ishii
* Koji Iida
* Daigoro Yokozeki
* Hiroshi Sakai

Multinic-libvirt

2013-02-16T23:51:47Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:multinic-libvirt
* '''Created''': Ilya Alekseyev
* '''Contributors''': Eldar Nugaev, Ilya Alekseyev

== Summary ==
We need support for multiple network interfaces per instance for libvirt. Our vision based on http://wiki.openstack.org/multi-nic specification. with some additions.

== Release Note ==

This is implementation same functionality as in https://blueprints.launchpad.net/nova/+spec/multi-nic for libvirt.
Multiple NIC for libvirt allows users to have instances connected to several networks.
Implementation of this blueprint is first step in add support multi-nics for libvirt.

== Rationale ==

== User stories ==

== Assumptions ==

== Design ==

https://blueprints.launchpad.net/nova/+spec/multi-nic
GD PoC branches:
https://code.launchpad.net/~ilyaalekseyev/nova/libvirt-multinic-experemental

== Implementation ==

=== UI Changes ===

=== Code Changes ===
I. libvirt connection changes
changes in libvirt_con.to_xml() propagate NIC data
I. firewall rules changes
All firewall drivers would be changed to support multiple networks per instance. iptable rules should be changed.
I. network managers changes
# FlatManager: open question - seems it is not required to be changed, but we need to check it
# FlatDHCPManager should be changed to support multiple networks
# VlanManager should be changed to support multiple networks
I. template changes
Add support of several NICs to template:

<pre><nowiki>#!highlight xml
#for $nic in $nics
<interface type='bridge'>
<source bridge='$nic.bridge_name'/>
<mac address='$nic.mac_address'/>

<filterref filter="nova-instance-$nic.name">
<parameter name="IP" value="$nic.ip_address" />
<parameter name="DHCPSERVER" value="$nic.dhcp_server" />
#if $getVar('extra_params', False)
$nic.extra_params
#end if
#if $getVar('ra_server', False)
<parameter name="RASERVER" value="$nic.ra_server" />
#end if
</filterref>
</interface>
#end for
</nowiki></pre>

=== Migration ===

== Test/Demo Plan ==

We need both unit and integration tests. Second is more important.

== Unresolved issues ==

== BoF agenda and discussion ==
http://ietherpad.com/arRVMd2Lwl

----
[[Category:Spec]]

Nova/Ipv6supportSpec

2013-02-16T23:50:39Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:ipv6-support
* '''Created''': 2010-11-01
* '''Contributors''': Koji Iida, Hisaharu Ishii

== Summary ==

* Support IPv4 and IPv6 dual protocol stacks

http://etherpad.openstack.org/IPV6-Support

== Release Note ==

TBD

== Rationale ==

* IPv4 address exhaustion < >
** especially in Asia Pacific region
** NGN (Next Generation Network) in Japan supports both IPv4 and IPv6.
** all public servers of US Government must be upgraded to IPv6 by FY 2012 < >
http://www.cio.gov/Documents/Transition-to-IPv6.pdf

== User stories ==

== Assumptions ==

* Basically, this blueprint depends on bexar-network-services http://etherpad.openstack.org/0bp9S20wZR blueprint.
* If bexar-network-services blueprint is deferred, we will implement this feature based on codebase of Austin Release.

== Design ==

Bexar release: Limited implementation of IPv6 direct access model

* Configuration flag to select IPv4 mode or IPv4/IPv6 dual stack mode
* In IPv4/IPv6 dual stack mode:
** IPv6 support coexist with current VlanManager
** Each VM is assigned both IPv6 global unicast address and IPv4 private address
*** Not support assignment of only IPv4 address or only IPv6 address to VM.
* Pass-through IPv6 packets on network node
** Firewall rule management for IPv6 traffic
** nova-api support both IPv4 and IPv6 access
* In IPv4 mode, same to current implementation

Followings will be addressed in 'C' release or later.
* Elastic IP feature for IPv6
* IPv6 support in FlatManager

== Implementation ==

=== UI Changes ===

TBD

=== Code Changes ===

TBD

=== Migration ===

TBD

== Test/Demo Plan ==

TBD

== Unresolved issues ==

== BoF agenda and discussion ==
Discussion summary of developer summit Bexar: http://etherpad.openstack.org/IPV6-Support

Slides: [[attachment:IPv6_on_nova_draft_bp.pdf]]

----
[[Category:Spec]]

Obsolete:ArchitecturalOverview

2013-02-16T23:46:09Z

Olaph:

= caution =
'''This page is outdated'''

The content of this page has not been updated for a very long time. Sections of this page are incorrect when referring to the current release.

== Architectural Overview for [[OpenStack]] Compute ==
Live Notes may be taken for this topic at: http://etherpad.openstack.org/Architecture and http://etherpad.openstack.org/nova-archdoc

[[Image:overview.png]]

=== “Small” components, loosely coupled ===
* Queue based (currently AMQP/RabbitMQ)
* Flexible schema for datastore (currently Redis)
* LDAP (allows for integration with MS Active Directory via translucent proxy)
* Workers & Web hooks (be of the web)
* Asynchronous everything (don't block)
* Components (queue, datastore, http endpoints, ...) should scale independently and allow visibility into internal state (for the pretty charts/operations)

=== Development goals ===
* Testing & Continuous Integration
* Fakes (allows development on a laptop)
* Adaptable (goal is to make integration with existing resources at organization easier)

=== Queue ===
* Each worker/agent listens on a general topic, and a subtopic for that node. Example would be "compute" & "compute:hostname"
* Messages in the queue are currently '''Topic''', '''Method''', '''Arguments''' - which maps to a method in the python class for the worker
* exposed via method calls
** '''rpc.cast''' to broadcast the message and not wait for a response
** '''rpc.call''' to send a message and wait for the response

=== Datastore ===
* Pre-Austin, data is stored in Redis 2.0 (RC)
* Do the work on write - make reads FAST
** maintain indexes / lists of common subsets
** use pools (SETs in redis) that are drained for IPs instead of tracking what is allocated

=== Delta ===
* Scheduler does not exist (instances are distributed via the queue to the first worker that consumes the message)
* Object store in Nova is a naive stub which would be replaced with Cloud Files in Production (a simple object store that mimics Cloud Files might be good for development)
* Tornado should be phased out for WSGI-based web framework

=== Networking ===
Currently, there are three strategies for networking, implemented by different managers:

* FlatManager -- ip addresses are grabbed from a network and injected into the image on launch. All instances are attached to the same manually configured bridge.
* FlatDHCPManager -- ip addresses are grabbed from a network, and a single bridge is created for all instances. A dhcp server is started to pass out addresses
* VlanManager -- each project gets its own vlan, bridge and network. A dhcpserver is started for each vlan, and all instances are bridged into that vlan.

The implementation of creating bridges, vlans, dhcpservers, and firewall rules is done by the driver linux_net. This layer of abstraction is so that we can at some point support configuring hardware switches etc. using the same managers.

For more discussion of network architecture, see [[Networking]].

DevQuickstart

2013-02-16T22:37:37Z

Olaph:

__NOTOC__
= [[OpenStack]] Bug Fixing Step-by-Step =

== Step 1: Set Up Your Development Environment ==

* 1.1) The easiest way to build a fully functional development environment is with [http://devstack.org DevStack]. In order to use it, you will need a machine (or Vagrant box) running Ubuntu 11.10. If you've already got a machine up and running, skip to step 1.4. Otherwise, read on.
* 1.2) At this point you need to create a local virtual machine running Ubuntu 11.10. A common method, if you have the capability to run [[VirtualBox]], is to use Vagrant. Head over to https://github.com/bcwaldon/vagrant_devstack for further instructions.
* 1.3) If you can't run [https://www.virtualbox.org/wiki/Downloads VirtualBox] or some other virtualization application locally, you'll need to look to the cloud! You can set up an account on Amazon EC2 or Rackspace Cloud to run a temporary machine.
* 1.4) Follow the guide at http://devstack.org to get your environment up and running.
* NOTE: If you prefer not to use devstack, you can still check out source code on your local machine and develop from there.

== Step 2: Sign the CLA and Sign in to LaunchPad ==

* 2.1) Head over to http://launchpad.net and click 'Log in / Register' in the top-right corner to sign up. Once you've created your account, you need to do a few more things
* 2.2) Sign the [[OpenStack]] CLA. You can sign it electronically at http://wiki.openstack.org/CLA
* 2.3) Add yourself to the Contributors wiki page. See the instructions at the top of this page: http://wiki.openstack.org/Contributors
* 2.4) Join the openstack-cla launchpad group. To do this, navigate to https://launchpad.net/~openstack-cla and click 'Join the team' on the right side of the page. Your request will be forwarded to the core [[OpenStack]] developers, who can then approve your request. Keep in mind, you must have completed the first two steps for your membership to be approved. Once you have submitted your request to join the team, feel free to move on. Just keep in mind that you must be approved before you can submit any code!
* NOTE: At this point, it would be a good idea to join the #openstack-dev IRC channel on Freenode. There are always developers around to help with code reviews or to answer any other general questions you may have. If you find that your request to join the openstack-cla team from step 3 is blocking your ability to submit code, feel free to ping someone in that chatroom!

== Step 3: Claim a Bug ==

* 3.1) The [[OpenStack]] projects keep all of their bug reports on LaunchPad. Follow one of the links below to see a list of open and unclaimed bugs for each specific project:
** [https://bugs.launchpad.net/nova/+bugs?field.searchtext=&orderby=-importance&field.status:list=CONFIRMED&field.status:list=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Nova Bugs]
** [https://bugs.launchpad.net/glance/+bugs?field.searchtext=&orderby=-importance&field.status:list=CONFIRMED&field.status:list=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Glance Bugs]
** [https://bugs.launchpad.net/swift/+bugs?field.searchtext=&orderby=-importance&field.status:list=CONFIRMED&field.status:list=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Swift Bugs]
** [https://bugs.launchpad.net/keystone/+bugs?field.searchtext=&orderby=-importance&field.status:list=CONFIRMED&field.status:list=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Keystone Bugs]
** [https://bugs.launchpad.net/horizon/+bugs?field.searchtext=&orderby=-importance&field.status:list=CONFIRMED&field.status:list=TRIAGED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Horizon Bugs]
* NOTE: If you're still a beginner with [[OpenStack]], it might be a good idea to filter the bugs down to 'low-hanging-fruit'. Look for a link on the right to see this list.
* 3.2) Once you've searched through the list and found something you want to work on, open up the full bug report, click the yellow edit button in the 'Assigned to' column then click 'Assign me' in the box that pops up. You should also set the status to 'In Progress'.

== Step 4: Fix the Bug ==

Now that you've picked out what you're going to work on, head into your development environment and go crazy! We strongly suggest that you first write a failing test that reproduces the behavior detailed in the bug report. Core developers will end up asking for a test that verifies the fix when you get to the code review step, so you might as well take care of it up front. In order to run tests, look for the 'run_test.sh' shell script. It will run all of the tests for the project you are in and check for any style errors.

If you find you can't reproduce the bug from the details in the report, feel free to ask around. Several bugs pop up due to mis-configured environments or only apply in extremely specific situations. Here are the lists of core members for each project:

* Nova: https://launchpad.net/~nova-core/+members#active
* Glance: https://launchpad.net/~glance-core/+members#active
* Swift: https://launchpad.net/~swift-core/+members#active
* Keystone: https://launchpad.net/~keystone-core/+members#active
* Horizon: https://launchpad.net/~horizon-core/+members#active

== Step 5: Propose the Fix ==

* 5.1) There is a bit more work to be done before you can propose your patch. First you need to install 'git-review'. This tool adds a 'review' command to git which helps with some bootstrapping of your repo and individual patches to work with our code review tool. The easiest way to get git-review is to use pip:
<code><nowiki>pip install git-review</nowiki></code>

* 5.2) Next you need to ensure your fix is in a single commit with a link to the bug in the commit message. In order to do that, we suggest that you run 'git rebase -i origin/master'. This will open an interactive editor in which you can 'squash' your commits together. This process also allows you to edit the commit message that will be sent with the review. Your fix can be automatically linked back to the bug on launchpad if you place a string like 'Fixes bug XXXXXX' in your commit message. If you have troubles with this step, please head to (http://wiki.openstack.org/GerritWorkflow) or just grab someone on IRC.
* 5.3) Once you've installed git-review and cleaned up your patch, feel free to run 'git review'. A link to your review should be printed out to the screen. At this point you can move on to another bugfix! If you end up getting any feedback that you need to address, continue on to the next step
* 5.4) Once you fix any code-related problems and you are ready to resubmit your review, you need to once-again squash your new commit(s) into one. Make sure you don't remove the 'Change-Id: ...' line from your commit message as that is the piece of information Gerrit uses to tie your subsequent patches together into one review. You can run the same command, 'git review', to push up your updated code.

NetworkContainers

2013-02-16T22:32:18Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:netcontainers
* '''Created''': 2011-04-06
* '''Contributors''':

__TOC__

== Summary ==
'''Cisco’s Proposal on OpenStack:Network as a Service (NaaS)'''

We are proposing OpenStack:Network as a Service, a first class OpenStack service. Though we share the requirements and many of the design concepts/principles with the other [[NetworkService]] proposal (many of the contributors are participants in both proposals), the main motivation for this blueprint is to provide OpenStack community a preview of the overall proposal and some of the new ideas, concepts for the OpenStack: NaaS consideration. This proposal also extends the [[NetworkService]] requirements specified in its draft.

== Release Note ==
TBD

== Design ==
=== Overview ===
OpenStack: NaaS provides a network abstraction layer and set of APIs (Figure 1) to enable: * Requesting and acquiring network connectivity by OpenStack:Compute for interconnecting two VM instances , both single virtual network (single vnic) or multi vnics to different virtual networks.< > * Network Services (e.g. firewall, load balancers, Wide Area Acceleration Services) insertion at the appropriate virtual networks; and dynamically request “adaptive” network resources.< > * Any OpenStack applications or services that needs Network resources visibility or consumption, examples: DR applications, Network Health / Monitoring Services, Chargeback / Billing services and so on.< > * Network resources that are required to interconnect OpenStack “Zones” or geographically dispersed compute/storage resources.< > * Network resources required to support new Services like Virtual Private Cloud, enterprise extensions.< > * In future, this NaaS could be expanded to support SLA, Network level QoS and other network based auditing/monitoring services.< >

Figure 1< > [[Image:figure1.png]]

Figure 2 shows OpenStack:Network Logical components. [[Image:figure2.png]]

=== Virtual Networks ===
Today’s nova:network supports three “network models” (a)FlatNetwork (b)FlatNetworkwithDHCP (c) VLAN with VPN GW instance. The shortcomings of this model are well documented in other proposals and it’s rather restrictive. In this new Naas model, the Network Abstraction completely hides the technology used for achieving the network segmentation, example: Initially this may be achieved by VLAN tags, in future different vendors may employ their proprietary/ future standards based segmentation schemes, to provide scalability and flexibility. With that in mind, the Virtual networks are primarily L2 segments (with L3 services offered in future). This simplifies the network segment notion for all the other !Openstack services and hides the “platform specific” complexity to the network devices.(both physical and virtual form factors)

=== [[NetContainer]](NC) === A logical entity that is initially created per OpenStack "Project". Each NetContainer contains one or more virtual network segments, with one or more network services and in the future may have one or more NetContainers embedded.

NetContainers can be instantiated based on already defined NC flavors or from ground up.

=== NC Flavors ===
NC Flavors provides a Cloud Service Provider opportunity to “package” its offerings,

* based on Qos , Examples: Gold, Silver or Bronze< > * Flavors that specify application/service specific NC falvors, example: HIPPA complaint Network< > * simple Network resources based definition for a particular tenant or a Project, example: how many virtual networks that the Project’s user allowed to create, Max VMs, type of services supported, ...< >

NC Flavors are envisioned similar to the VM flavors like m1.tiny, m1.large and so on. There will be default basic NC flavors that can be further modified for a particular project requirement.

'''NC Plugins:'''Plug-ins are Network device/vendor specific “wrappers” that are invoked for configuring required network resources at the physical/virtual switches and services.

Figure 3 shows a hierarchical model of NaaS and its layers. Though the NetContainers can be requested and instantiated from already defined and available NC Flavors, NaaS will also provide direct NaaS APIs to request the NC with fine grained parameters, to provide flexibility.

Figure 3 [[Image:figure3.png]]

In the above diagram a logical representation of network segments shown. A cloud Service provider, P1 with a tenant T1, creating a project Pr1, that hosts Application A1 with three virtual networks AT1, AT2 and AT3.

All NetContainers should be instantiated based on a NC flavor. Some flavors will support modification beyond the initial configuration, and some will be locked into the fairly rigid configurations that are initially instantiated. This would depend on the plugin’s capabilities, and gives a lot of flexibility to the plugin implementer. Those that provide more flexible containers will have an advantage, but it won’t preclude those that are more basic from being useful.

Figure 4 shows the OpenStack:Network with “network devices” specific plug-ins and various sets of APIs.

Figure 4< > [[Image:figure4.png]]

NaaS APIs accessibility, in terms of authorization, will follow the same model as of OpenStack:Compute

=== API Set 1: NC APIs ===
These APIs provide the ability to create and "manage" NetContainers for a project; * A NetContainer can be defined/created based on the NC flavors assigned by the CSP(Coud ServiceProvider) to that project * Attach/Create virtual networks for the Net Container * Instantiate/attach Network Services at the "appropriate" Virtual network segment "nodes" * Query/Monitor NetContainer health * Monitoring in future

=== API Set 2: Network Services APIs ===
These are sub-set of APIs that enable us to define services insertion and "bootstrap" parameters for the services.

"Configuring/managing” the services itself is out of scope of this API set. "Configuring/Managing" of each service is vendor specific and its challenging to align them all under one model. Moreover each of the services may have their own SDK and different deployment model to support which is very complex to have it all under this set. However, OpenStack should provide a “generic” Network services configuration for well known services such as Firewall , Load balancer.

Using this API, a tenant can

* Select Services/Types Services that a "project" would like to consume and use< > * "Customize” the services – configure network “insertion” parameters< > * List/identify the services consumed and health< >

=== API Set 3: NC Flavors API ===
This set of APIs primarily used by CSP for defining "resources"/services offering buckets related to Network & Network Services. Open For discussion : how much of these APIs are opened up further for a Project/tenant for customizing the “flavors”.

Example of a flavor : Three tier App with Private, Public Virtual Networks with FW and LB services, VPN GW attached,

Using this API set, CSP/Tenant can: * Create/Manage templates for a project and services offering.< >

In the next section we discuss, how two main layers, NaaS:Core and Naas:Plugins, interact and their respective responsibilities

=== Components responsibilities ===
==== NaaS:Core ====
Hold references to NetContainer instances

* Provides API for container provisioning < > * Exposes catalog of available NetContainers and their attributes < > * Allows for operational/runtime control of NetContainers < > * '''Discussion point:''' Map compute/storage to container? Or does Compute service manage that?
* Choice 1: NaaS owns connections, thus compute registers VMs with NaaS using params
* Choice 2: NaaS advertises container to compute, and compute manages connections

==== NaaS:Plug-ins ====
* Owns what offerings are available (what containers are available)< > * Owns mapping container to Network Devices environment (example : Network segment schemes – VLAN IDs, QinQ and so on)< > * Owns container provisioning, including resource allocation, location, services, etc< > * Support standard but extensible formats for !netcontainer type definitions and !netcontainer instance requests< > * Standard way of reporting errors, such as missing information, unknown element, invalid config, etc< > * APIs for querying actual availability of specific container types (knows what resources are available, and which are consumed)< > * Handles asynchronous conversation with NaaS :Core

=== Use Cases ===
==== Use case 1: Create a Network resource for an Openstack project ====
'''Workflow for provisioning a container:'''

* Customer selects !netcontainer service from NaaS offering,if any< > * Customer provides required parameters to specify config preferences, policies, etc< > * NaaS creates request document from customer information< > * NaaS passes request document to plug-in< > * Plug-in parses document and maps to resource requirements< > * Plug-in evaluates resource req's against available resources< > * Plug-in provisions resources as defined by container request< > * Plug-in returns handle to new container (REST URL)< >

==== Use case 2: Network Services request network resources ====
* Insertion of a firewall requests another internal virtual network and external network segment

==== Use case 3: Virtual Private Cloud/Enterprise Extensions ====
* Create a virtual network in a NC< > * Create a VPN/Acceleration service instance and attach to the “private” network segment< >

==== Use case 4: Network Scheduler ====
* To support multiple !Openstack:Compute Zones and schedule network resources in conjunction with nova-scheduler.

=== API Definitions ===
This set of APIs allow Openstack services and "Applications" to request for Network Containers, Delete Network containers, modify Network containers (NC) attributes, List all the available NC flavors and create new NC flavors. These APIs are consumed programmatically by Openstack compute or other services.

API definitions and data model will try and follow the Openstack:Compute API 1.1 as specified at http://docs.openstack.org/openstack-compute/developer/cs-devguide.pdf

JSON is the default data serialization for [[OpenStack]]:Network APIs.

'''Authentication:'''

[[OpenStack]]: Network APIs will follow the same authentication model as that of [[OpenStack]]:Compute and may support multiple authentication schemes (OAuth, Basic Auth, Token) and so on.

'''Key Concepts:'''

'''Tenants:''' This may be mapped to a project or a org based on the NaaS API invoking system. It’s a unique name/label that identifies the NC ownership. Tenant could be mapped to a Project in Openstack:compute service invocation. The tenant construct could be used in future by “Data Services” to create a Network container for storage related services. This also makes it future proof from changes that may happen at [[OpenStack]]:compute and other consumer apps/services, example: [[OpenStack]]: Computer evolving from projects to ORG/vDC/vApp kind of constructs.

'''Flavor:''' Container definitions in terms - network resources properties. Example : # of Virtual Networks, Network Services attached, Zone affinity in future.

=== Summary of APIs ( work in progress) ===
Tenant Operations:
{| border="1" cellpadding="2" cellspacing="0"
|< >|API Definition
|<class="confluenceTh" style="font-weight: bold;">|Request
|<class="confluenceTh" style="font-weight: bold;">|Request Body
|-
|<class="confluenceTd">|Register Tenant
|<class="confluenceTd">|POST [[OpenStack]]:Networks API URL/config/tenant/create
|<class="confluenceTd">|Tenant info
|-
|<class="confluenceTd">|unregister a tenant
|<class="confluenceTd">|GET [[OpenStack]]:Network API URL/config/tenant/''UID''/delete
|<class="confluenceTd">|Tenant Info
|-
|<class="confluenceTd">|List tenants
|<class="confluenceTd">|GET [[OpenStack]]:Network API URL/tenants GET [[OpenStack]]:Network API URL/tenants/detail
|<class="confluenceTd">|
|-
|<class="confluenceTd">|
|<class="confluenceTd">|
|<class="confluenceTd">|
|}

==== example: Create Tenant ====
POST {''APP-URI}''/config/tenant/create

Request Body

<pre><nowiki>
<tenant>
<name>TenantABC1</name>
<description>Optional description here</description>
</tenant>
</nowiki></pre>

'''Response Codes:'''

* 201 Created – Tenant has been created. Location header gives URL of created tenant. < >400 Bad Request – Something was wrong with the request, such as missing data
* 404 Not Found – Domain specified by domainId was not found.
* 409 Conflict – Name conflicts with existing tenant

'''NC Flavor Operations'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|< >|API Definition
|<class="confluenceTh" style="font-weight: bold;">|Request
|<class="confluenceTh" style="font-weight: bold;">|Request Body
|-
|<class="confluenceTd">|Create NC Flavor
|<class="confluenceTd">|POST [[OpenStack]]:Networks API URL/config/ncflavor/create
|<class="confluenceTd">|NC Flavor info
|-
|<class="confluenceTd">|Modify NC Flavor <actions>
|<class="confluenceTd">|
|<class="confluenceTd">|
|-
|<class="confluenceTd">|Delete NC Flavor
|<class="confluenceTd">|
|<class="confluenceTd">|
|-
|<class="confluenceTd">|List NC Flavors
|<class="confluenceTd">|GET [[OpenStack]]:Network API URL/ncflavors GET [[OpenStack]]:Network API URL/ncflavors/detail
|<class="confluenceTd">|
|}

'''[[NetContainer]] Operations'''
{| border="1" cellpadding="2" cellspacing="0"
|< >|API Definition
|<class="confluenceTh" style="font-weight: bold;">|Request
|<class="confluenceTh" style="font-weight: bold;">|Request Body
|-
|<class="confluenceTd">|Create NetContainer
|<class="confluenceTd">|POST [[OpenStack]]:Networks API URL/config/netcontainer/create
|<class="confluenceTd">|NetContainer info
|-
|<class="confluenceTd">|Modify Net Container <actions>
|<class="confluenceTd">|
|<class="confluenceTd">|
|-
|<class="confluenceTd">|Delete Net Container
|<class="confluenceTd">|
|<class="confluenceTd">|
|-
|<class="confluenceTd">|List NetContainers
|<class="confluenceTd">|GET [[OpenStack]]:Network API URL/netcontainers GET [[OpenStack]]:Network API URL/netcontainers/detail
|<class="confluenceTd">|
|}

''' Network Services attachement'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|<class="confluenceTh" style="font-weight: bold;">|API Definition
|<class="confluenceTh" style="font-weight: bold;">|Request
|<class="confluenceTh" style="font-weight: bold;">|Request Body
|-
|<class="confluenceTd">|attach Services
|<class="confluenceTd">|
|<class="confluenceTd">|Network Services info
|-
|<class="confluenceTd">|Config Services <serviceUID>
|<class="confluenceTd">|
|<class="confluenceTd">|
|-
|<class="confluenceTd">|list Services for a container
|<class="confluenceTd">|
|<class="confluenceTd">|
|}

'''Open''' '''Questions'''

* NetContainers Definitions – How is it related to OVF netspec. How much we can overlap and make use of the OVF spec?< > * How do we resolve "dependency" while instantiating the NC or deleting it?< > * Explicit supported Services Enumeration?< > * How about supporting models like Security groups? Flexibility< >

== BoF agenda and discussion ==
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

----
[[Category:Spec]]

Neutron/ServiceInsertion

2013-02-16T22:27:23Z

Olaph:

__NOTOC__
= WORK IN PROGRESS =
We are integrating all the information related to Services Insertion in only one wiki page

= Services Insertion Model in Quantum =
* '''Launchpad Entries''':

''''' [https://blueprints.launchpad.net/quantum/+spec/quantum-service-type services-type] '''''

''''' [https://blueprints.launchpad.net/quantum/+spec/services-insertion-wrapper services-insertion-wrapper (folsom)] '''''

* '''Contributors''': [https://launchpad.net/~salvatore-orlando Salvatore],[https://launchpad.net/~emagana Edgar Magana], [https://launchpad.net/~enikanorov Eugene], Ram Durairaj, Mani Ramasamy and Quantum Community.

== Description ==
This page contains the design spec for the service insertion blueprint. Despite being quite fleshed out the spec is not finalised yet, as some details are still subject to change, especially as far as the implementation plan is concerned. Please subscribe to this page to get every update, or check it as often as necessary. The blueprint itself can be found at this address:

This specification does not fully respects the blueprint template set for Quantum. This is because we are giving much more space to the design discussion. We will have another wiki page using the blueprint template once the design discussion is Finalized.

== High level description ==
The ''service insertion'' feature aims at defining a framework for running L4/L7 network services on Quantum Logical Topologies. This framework will be immediately leveraged by the LBaaS effort; and in the near future by other advanced services which will be plugged into Quantum (for instance, VPN, and edge firewalls).

== Defining Service Insertion ==
Before going in depth into design and implementation discussion, it might be worth to define in a detailed way what an advanced service is, and how it could be inserted in the Quantum logical topology. The diagram below show Quantum's basic network topology, with a logica router interconnecting internal and external networks; ports, which are used for VIF, DHCP server, router, and floating IP attachments are also shown in the diagram.

[[Image:basic-topology.png]]

From previous discussions it emerged that services can be inserted in the following modes:

* Routed mode - Where the service is attached to a logical router, which then becomes a multi-service appliance.

[[Image:routed-si.png]]

* Floating Mode (In-path) - Where service runs in a standalone way. Please note that the "Floating" insertion is not automatically a network level insertion, as L3 routing might still occur. This mode has also been referred as "in-path or bump-in-the-wire" insertion.

[[Image:in-path-si.png]]

With reference to the diagram above, is is worth noting that an advanced service, even when inserted in floating mode, should still 'plug' into networks. In the example above, the service plugs into the external network and into an internal network. This should be reflected by ports on the relevant Quantum networks ''owned'' by the advanced service itself.

* Re-directed Mode (Out-of-path) - Where the service also runs in a standalone way but in this case the traffic is first sent to the Router entity and then redirected to the Advanced Service, finally send it back to the routed with specific configuration. In particular, this model could be reduced to the first assuming that a standalone service is regarded as a peculiar case of router capable of providing only a specific service. This mode needs specific changes at the routing entity and may not be implemented in Grizzly release.

[[Image:out-of-path-si.png]]

All these three modes are not mutually exclusive. The diagram below shows how they can be combined. With reference to the diagram below, it is worth noting that the services inserted in routing and floating modes could have different implementations. This means for instance, that instances on the same Quantum networks could be load balanced using two different solutions.

[[Image:mixnmatch-si.png]]

== The Service Type concept ==
Just like the Quantum plugins allow for using several technologies for implementing the basic logical topologies, advanced services will use a similar mechanism. However, for advanced services, multiple different implementations of the same kind of service might co-exist in the same deployment. There are a number of reasons for this, most importantly the ability of giving tenants a choice among solutions. The '''Service Type''' concept tries to address the need for multiple, co-existing, service providers.

A '''Service Type''' definitions might be regarded as list of services (and their providers) which can be offered to tenants. Each advanced service, regardless of its insertion mode, should be either directly or indirectly associated with a single service type.

The Service Type resource might be described as follows:

<pre><nowiki>
ServiceType:
{
id: uuid,
name: string,
service_definitions: list<ServiceDefinition>
default: {True¦False} # Only one service_type could be the default one. This is the service type which should be picked if none is specified in the API (think backward compatibility)
}

ServiceDefinition:
{
id: uuid,
name: string,
type: enum(LB, VPN, FW) # or whatever you think is right
provider: string # This ultimately must map to a python class (perhaps through a configuration variable)
capabilities: list of API extensions which are enabled for this specify service provider
}
</nowiki></pre>

Please note that the ''ServiceDefinition'' object might be regarded either as a resource of its own, or as a child resource of ''ServiceType''. The association between a service and a service type can happen in two ways, according to the insertion mode of the service.

* ''Routed'' Insertion mode: The advanced service will be associated with Quantum logical router, which in turn is associated with a service_type resource;

In order to ensure backward compatibility a default service type must be specified. This implies that all the services which will be inserted on a router will share the same service type.

* ''Floating'' Insertion mode: The service type should be explicitly specified on the advanced service being created; if not, the default service type will be used.

When an advanced service is created at the API layer one of the following two should be specified:

# service_type_id # floating or out-of-path insertion
# router_id # routed or in-path insertion

It should not be allowed to specify both parameters.

The logical model for service insertion, augmented with the service type concept, is depicted in the following diagram:

[[Image:svc-type.png]]

The diagram shows a set of services deployed in routed mode (light blue), and a service deployed in floating mode (purple). Light blue and purple services are associated with different service types, which then can map to distinct implementations of the same services.

== Dispatching calls to the plugins ==
The concept of service type implicitly allows for different paths for an API call, as it allows multiple providers to serve the same request. Eugene Nikanorov also addresses this topic in this wiki page: http://wiki.openstack.org/Quantum/ServiceIntegration

There are two design alternatives to be considered:

# A single plugin, augmented with the capability of serving requests for advanced services; call will then be redirected to provider-specific drivers.
# Multiple, self-contained plugins. Each plugin is specific to a given service provider and might implement one or more advanced service interfaces.

For both design alternatives, it is pretty clear that each advanced service should have a 'service plugin interface', which is the plugin-side dual of the tenant-facing APIs. The core Quantum plugin similarly, defines a plugin interface that all plugins must implement.

=== Single Plugin Approach ===
The diagram below show a Quantum Plugin which implements both core and advanced services interface. The plugin is capable of interpreting the service_type attribute, and dispatching the call to the appropriate, provider-specific driver.

[[Image:single-plugin.png]]

For this approach it should be possible to leverage the "mixin" mechanism which proved successful when integrating DHCP and L3 services. The possibility of augmenting the Quantum plugin with the capability of handling advanced services should definitely be allowed. In this case there will still be a single Quantum plugin; Appropriate ''service drivers'' should be specified in the configuration file. For each service multiple drivers might be specified; The service driver should not be seen as a driver for a specific appliance implementing the service. It is rather a driver managing the service for a given provider.

=== Multiple Plugins Approach ===
The diagram below show how service type are mapped to multiple plugins when adopting this approach. Each plugin implements an interface which is specific for the kind of service it implements. The API layer has a dispatcher component which forwards the call to the appropriate plugin, according to the service type associated with the advanced service specified at the API layer.

[[Image:multi-plugin.png]]

It is a Quantum principle that plugins are required only to implement the plugin interface. Hence they are not required to adopt the mixin mechanism; hence another aspect to consider is that there could be multiplie plugins at the same type, and the intelligence of dispatching a call to one plugin or another should reside in the API layer.

In this case several plugins might be configured at the same time. The API layer will need to figure out, for each router and/or advanced service, to which plugin the API call should be dispatched. It is also worth noting that a plugin should not be constrained to implement a single advanced services. It might well be that plugins implement a set of services; this is particularly the case of plugins which will be interfaced with integrated services appliances or application delivery controllers.

In this case the diagram above could be generalised as follow (the reference to the configuration file is an implementation detail at this stage):

[[Image:multi-plugin-2.png]]

=== Evaluation of pro and cons of both Approaches ===
The diagrams below depicts, at a very high level, the different flows of the single versus multiple plugins approaches. The single plugin approach is reported on the right, whereas the multiple plugin approach is reported on the left.

[[Image:single-vs-multi.png]]

* multiple plugins will require the API to manipulate data from different sources. Even if each plugin will return data in the same format, we will still need logic to handle collecting data from multiple sources on GET requests, and dispatching commands to the appropriate destination on POST/PUT/DELETE.
* currently all plugins already implement the "mixin" approach for augmenting the L2 feature with L3 and DHCP features. This mechanism could be extended.
* compatibility between plugins (not something we need to worry from day 1, but still an interesting problem)
* interactions among plugins; with a single plugin, the data model has all the information it needs. With multiple plugins will have to interact with the "base" plugin (or in some cases even among them); this could be achieved throughout the plugin interfaces.
* the single plugin, multiple drivers, model works very well when the Quantum database holds all the required information necessary to operate. This implies that items such as device management, resource allocation, and similars, are described using a model shared by all plugins. Although this might be mitigated by either introducing driver-specific extensions, or moving such capabilities in the driver, this is clearly a case in which the multiple plugins approach is preferred.

== A few notes on the scope of this blueprint ==
This blueprint which focuses on:

* Providing the infrastructure for allowing implementations of these services to serve API requests; multiple implementations for each class of service should be allowed to coexist at the same time in a Quantum deployment.
* Defining a set of API calls (as well supporting logic and data model) for defining which, where, and how services might be attached on the logical topology defined by the basic Quantum topology;
* Allowing providers of such services to expose implementation-specific service extensions and advertising them to API users.

Defining which services might be inserted, and how they should be presented to tenants is beyond the scope of this blueprint. Activity is already ongoing for the Load Balancing service; for more information please refer to lbaas-* blueprints in the Quantum framework.

Most of the concepts expressed in this blueprint are not new to the Quantum community and can be found in this wiki page authored by Edgar Magana from Cisco: http://wiki.openstack.org/QuantumServicesInsertion

''Further notes (or what this design specification is not about):''

* There might be scenarios where service insertion happens at network level, as shown in Sasha Ratkovic's presentation. For the scope of this blueprint we will consider only router-level insertion, assuming that network-level insertion could be reduced to a case in which the service is inserted on a router connected to a single network only.
* There are also definitely scenarios where service insertion happens at the port level; in this case the insertion might be simply represented by attributes applied to the port itself. For an example please refer to the proposed https://review.openstack.org/#/c/14262/.
* The concept of port group, presented by Sasha Ratkovic at the Openstack Design Summit is also outside of the scope of this blueprint.
* Whether advanced services will be realized as new resource (as described in the LBaaS proposal) or as policies (as described by Sasha Ratkovic proposal), is also beyond the scope of this blueprint.

At this stage the careful reader might be wondering whether this blueprint is actually about anything. With all due honesty by defining a (small) set of items we want to address, and a (very large) of items that we definitely do not want to address, it might be argued that this blueprint is actually pretty well scoped, leaving a very thin, blurred, area of items which might or might not be addressed by it.

= Work Tasks =
== API ==
The service insertion API might be implemented as a Quantum extension for the Grizzly release cycle, or be part of the core if there is a total agreement on it. Service insertion support will require the following changes in the Quantum API:

* Service Type management. CRUD operations should be available on a ServiceType object.
* Regular tenants typically should be allowed to read only
* Administrators would have right to create, modify, and delete those object. Also, the provider attribute might be hidden in response returned to regular tenants.

''Note'': default policy settings might always be overridden as they're specified in ''etc/policy.json''

The Router resource should therefore be extended with the following attribute:

<pre><nowiki>
services:service_type_id # which refers to service_type object
</nowiki></pre>

Each advanced service should allow for specifying either a service type id (for floating mode insertion) or a router_id (for routed mode insertion).

If a router supports multiple services, the tenant might be given the ability of enabling or disabling specific services on a router, in accordance with the service_type associated with the router. This feature might be used by tenants to keep advanced service usage within quota limits.

* Option 1: Resource action /routers/<router_id>/enable_service
** /routers/<router_id>/disable_service
* Option 2: List Attribute
* Option 3: Sub-Resource POST /routers/<router_id>/enabled_services

Option 1 is currently the preferred one, as it is consistent with "the Openstack way" of defining APIs executing actions on resources beyond the ones that can be expressed as HTTP methods.

== Service Configuration ==
The configuration file should contains references to the drivers/plugins corresponding to the various advanced services. These information will be used by the plugin manager.

The configuration file might also contain service type definitions, which could alternatively be stored in the Quantum 'core' database (see section on Data Model Changes).

== Plugin Manager ==
A mechanism for loading multiple plugins will be required for the multi-plugin approach. The PluginAwareExtensionManager does not manage plugin loading, but some work will be required on it for handling plugin-specific API extensions, as it currently assumes that there's only a single plugin.

== Data Model Changes ==
ServiceType definition, if modeled into the Quantum DB will require two model classes. The ServiceType class will have a 1:n relationship with the ServiceDefinition class. This class will have an attribute for defining the type of the advanced services provider. This could either be another class in the model or extracted from the Quantum configuration file.

ServiceProvider definition could as well be either another model class or information extracted from the configuration file.

== Changes to the basic plugin Interface ==
The service insertion feature will add a new interface which might be implemented as a new mixin to be inherited by the "core" plugin (a completely standalone class is a viable alternative, albeit less straightforward)

== Managing API dispatching ==
This task is about implementing the single plugin/multiple drivers or multi-plugin approaches, as discussed earlier in this spec document.

= Potential changes to the current implementation =
# Floating IPs: despite being shipped with Folsom, this feature might actually be regarded as an 'advanced service', and should probably fit in the service insertion framework.
# External Gateway: This might be possibly be regarded as an advanced service too. For instance, a SNAT service.
# DHCP: think about how the DHCP service we implement today through the agent fits in the service insertion model. We are not exposing specifically any DHCP API, creating implicitly an instance of this service for each subnet for which ''dhcp_enable=True''. We can either leave DHCP as it is, or include it in the service insertion model.

NOTE: For the cases listed above it is very important to preserve backward compatibility.

= POC Proposal =
A POC of the service insertion model could be offered regardless of actual advanced services being implemented on top of Quantum. This POC might use 'dummy' advanced services plugins. Another options would be reworking the Floating IP and possibly also the external gateway concepts as advanced services and use them for the PoC.

= Mapping tasks to Grizzly milestones =
{| border="1" cellpadding="2" cellspacing="0"
| '''Task'''
| '''Importance'''
|-
| Service Type definition
| Essential
|-
| Plugin/Driver management
| Essential
|-
| API Call Dispatching
| Essential
|-
| PoC with dummy plugins
| High
|-
| PoC with LBaaS
| Essential
|-
| APIs for service type management
| Undecided
|-
| Service types in Quantum DB
| Normal
|-
| Reworking some L3 capabilities as adv svc
| Undecided

Neutron/ServiceInsertion

2013-02-16T22:24:04Z

Olaph: Reverted edits by Olaph (talk) to last revision by Corvus

__NOTOC__
= WORK IN PROGRESS =
We are integrating all the information related to Services Insertion in only one wiki page

= Services Insertion Model in Quantum =
* '''Launchpad Entries''':

''''' [https://blueprints.launchpad.net/quantum/+spec/quantum-service-type services-type] '''''

''''' [https://blueprints.launchpad.net/quantum/+spec/services-insertion-wrapper services-insertion-wrapper (folsom)] '''''

* '''Contributors''': [https://launchpad.net/~salvatore-orlando Salvatore],[https://launchpad.net/~emagana Edgar Magana], [https://launchpad.net/~enikanorov Eugene], Ram Durairaj, Mani Ramasamy and Quantum Community.

== Description ==
This page contains the design spec for the service insertion blueprint. Despite being quite fleshed out the spec is not finalised yet, as some details are still subject to change, especially as far as the implementation plan is concerned. Please subscribe to this page to get every update, or check it as often as necessary. The blueprint itself can be found at this address:

This specification does not fully respects the blueprint template set for Quantum. This is because we are giving much more space to the design discussion. We will have another wiki page using the blueprint template once the design discussion is Finalized.

== High level description ==
The ''service insertion'' feature aims at defining a framework for running L4/L7 network services on Quantum Logical Topologies. This framework will be immediately leveraged by the LBaaS effort; and in the near future by other advanced services which will be plugged into Quantum (for instance, VPN, and edge firewalls).

== Defining Service Insertion ==
Before going in depth into design and implementation discussion, it might be worth to define in a detailed way what an advanced service is, and how it could be inserted in the Quantum logical topology. The diagram below show Quantum's basic network topology, with a logica router interconnecting internal and external networks; ports, which are used for VIF, DHCP server, router, and floating IP attachments are also shown in the diagram.

[[Image:basic-topology.png]]

From previous discussions it emerged that services can be inserted in the following modes:

* Routed mode - Where the service is attached to a logical router, which then becomes a multi-service appliance.

[[Image:routed-si.png]]

* Floating Mode (In-path) - Where service runs in a standalone way. Please note that the "Floating" insertion is not automatically a network level insertion, as L3 routing might still occur. This mode has also been referred as "in-path or bump-in-the-wire" insertion.

[[Image:in-path-si.png]]

With reference to the diagram above, is is worth noting that an advanced service, even when inserted in floating mode, should still 'plug' into networks. In the example above, the service plugs into the external network and into an internal network. This should be reflected by ports on the relevant Quantum networks ''owned'' by the advanced service itself.

* Re-directed Mode (Out-of-path) - Where the service also runs in a standalone way but in this case the traffic is first sent to the Router entity and then redirected to the Advanced Service, finally send it back to the routed with specific configuration. In particular, this model could be reduced to the first assuming that a standalone service is regarded as a peculiar case of router capable of providing only a specific service. This mode needs specific changes at the routing entity and may not be implemented in Grizzly release.

[[Image:out-of-path-si.png]]

All these three modes are not mutually exclusive. The diagram below shows how they can be combined. With reference to the diagram below, it is worth noting that the services inserted in routing and floating modes could have different implementations. This means for instance, that instances on the same Quantum networks could be load balanced using two different solutions.

[[Image:mixnmatch-si.png]]

== The Service Type concept ==
Just like the Quantum plugins allow for using several technologies for implementing the basic logical topologies, advanced services will use a similar mechanism. However, for advanced services, multiple different implementations of the same kind of service might co-exist in the same deployment. There are a number of reasons for this, most importantly the ability of giving tenants a choice among solutions. The '''Service Type''' concept tries to address the need for multiple, co-existing, service providers.

A '''Service Type''' definitions might be regarded as list of services (and their providers) which can be offered to tenants. Each advanced service, regardless of its insertion mode, should be either directly or indirectly associated with a single service type.

The Service Type resource might be described as follows:

<pre><nowiki>
ServiceType:
{
id: uuid,
name: string,
service_definitions: list<ServiceDefinition>
default: {True¦False} # Only one service_type could be the default one. This is the service type which should be picked if none is specified in the API (think backward compatibility)
}

ServiceDefinition:
{
id: uuid,
name: string,
type: enum(LB, VPN, FW) # or whatever you think is right
provider: string # This ultimately must map to a python class (perhaps through a configuration variable)
capabilities: list of API extensions which are enabled for this specify service provider
}
</nowiki></pre>

Please note that the ''[[ServiceDefinition]]'' object might be regarded either as a resource of its own, or as a child resource of ''[[ServiceType]]''. The association between a service and a service type can happen in two ways, according to the insertion mode of the service.

* ''Routed'' Insertion mode: The advanced service will be associated with Quantum logical router, which in turn is associated with a service_type resource;

In order to ensure backward compatibility a default service type must be specified. This implies that all the services which will be inserted on a router will share the same service type.

* ''Floating'' Insertion mode: The service type should be explicitly specified on the advanced service being created; if not, the default service type will be used.

When an advanced service is created at the API layer one of the following two should be specified:

# service_type_id # floating or out-of-path insertion
# router_id # routed or in-path insertion

It should not be allowed to specify both parameters.

The logical model for service insertion, augmented with the service type concept, is depicted in the following diagram:

[[Image:svc-type.png]]

The diagram shows a set of services deployed in routed mode (light blue), and a service deployed in floating mode (purple). Light blue and purple services are associated with different service types, which then can map to distinct implementations of the same services.

== Dispatching calls to the plugins ==
The concept of service type implicitly allows for different paths for an API call, as it allows multiple providers to serve the same request. Eugene Nikanorov also addresses this topic in this wiki page: http://wiki.openstack.org/Quantum/ServiceIntegration

There are two design alternatives to be considered:

# A single plugin, augmented with the capability of serving requests for advanced services; call will then be redirected to provider-specific drivers.
# Multiple, self-contained plugins. Each plugin is specific to a given service provider and might implement one or more advanced service interfaces.

For both design alternatives, it is pretty clear that each advanced service should have a 'service plugin interface', which is the plugin-side dual of the tenant-facing APIs. The core Quantum plugin similarly, defines a plugin interface that all plugins must implement.

=== Single Plugin Approach ===
The diagram below show a Quantum Plugin which implements both core and advanced services interface. The plugin is capable of interpreting the service_type attribute, and dispatching the call to the appropriate, provider-specific driver.

[[Image:single-plugin.png]]

For this approach it should be possible to leverage the "mixin" mechanism which proved successful when integrating DHCP and L3 services. The possibility of augmenting the Quantum plugin with the capability of handling advanced services should definitely be allowed. In this case there will still be a single Quantum plugin; Appropriate ''service drivers'' should be specified in the configuration file. For each service multiple drivers might be specified; The service driver should not be seen as a driver for a specific appliance implementing the service. It is rather a driver managing the service for a given provider.

=== Multiple Plugins Approach ===
The diagram below show how service type are mapped to multiple plugins when adopting this approach. Each plugin implements an interface which is specific for the kind of service it implements. The API layer has a dispatcher component which forwards the call to the appropriate plugin, according to the service type associated with the advanced service specified at the API layer.

[[Image:multi-plugin.png]]

It is a Quantum principle that plugins are required only to implement the plugin interface. Hence they are not required to adopt the mixin mechanism; hence another aspect to consider is that there could be multiplie plugins at the same type, and the intelligence of dispatching a call to one plugin or another should reside in the API layer.

In this case several plugins might be configured at the same time. The API layer will need to figure out, for each router and/or advanced service, to which plugin the API call should be dispatched. It is also worth noting that a plugin should not be constrained to implement a single advanced services. It might well be that plugins implement a set of services; this is particularly the case of plugins which will be interfaced with integrated services appliances or application delivery controllers.

In this case the diagram above could be generalised as follow (the reference to the configuration file is an implementation detail at this stage):

[[Image:multi-plugin-2.png]]

=== Evaluation of pro and cons of both Approaches ===
The diagrams below depicts, at a very high level, the different flows of the single versus multiple plugins approaches. The single plugin approach is reported on the right, whereas the multiple plugin approach is reported on the left.

[[Image:single-vs-multi.png]]

* multiple plugins will require the API to manipulate data from different sources. Even if each plugin will return data in the same format, we will still need logic to handle collecting data from multiple sources on GET requests, and dispatching commands to the appropriate destination on POST/PUT/DELETE.
* currently all plugins already implement the "mixin" approach for augmenting the L2 feature with L3 and DHCP features. This mechanism could be extended.
* compatibility between plugins (not something we need to worry from day 1, but still an interesting problem)
* interactions among plugins; with a single plugin, the data model has all the information it needs. With multiple plugins will have to interact with the "base" plugin (or in some cases even among them); this could be achieved throughout the plugin interfaces.
* the single plugin, multiple drivers, model works very well when the Quantum database holds all the required information necessary to operate. This implies that items such as device management, resource allocation, and similars, are described using a model shared by all plugins. Although this might be mitigated by either introducing driver-specific extensions, or moving such capabilities in the driver, this is clearly a case in which the multiple plugins approach is preferred.

== A few notes on the scope of this blueprint ==
This blueprint which focuses on:

* Providing the infrastructure for allowing implementations of these services to serve API requests; multiple implementations for each class of service should be allowed to coexist at the same time in a Quantum deployment.
* Defining a set of API calls (as well supporting logic and data model) for defining which, where, and how services might be attached on the logical topology defined by the basic Quantum topology;
* Allowing providers of such services to expose implementation-specific service extensions and advertising them to API users.

Defining which services might be inserted, and how they should be presented to tenants is beyond the scope of this blueprint. Activity is already ongoing for the Load Balancing service; for more information please refer to lbaas-* blueprints in the Quantum framework.

Most of the concepts expressed in this blueprint are not new to the Quantum community and can be found in this wiki page authored by Edgar Magana from Cisco: http://wiki.openstack.org/QuantumServicesInsertion

''Further notes (or what this design specification is not about):''

* There might be scenarios where service insertion happens at network level, as shown in Sasha Ratkovic's presentation. For the scope of this blueprint we will consider only router-level insertion, assuming that network-level insertion could be reduced to a case in which the service is inserted on a router connected to a single network only.
* There are also definitely scenarios where service insertion happens at the port level; in this case the insertion might be simply represented by attributes applied to the port itself. For an example please refer to the proposed https://review.openstack.org/#/c/14262/.
* The concept of port group, presented by Sasha Ratkovic at the Openstack Design Summit is also outside of the scope of this blueprint.
* Whether advanced services will be realized as new resource (as described in the LBaaS proposal) or as policies (as described by Sasha Ratkovic proposal), is also beyond the scope of this blueprint.

At this stage the careful reader might be wondering whether this blueprint is actually about anything. With all due honesty by defining a (small) set of items we want to address, and a (very large) of items that we definitely do not want to address, it might be argued that this blueprint is actually pretty well scoped, leaving a very thin, blurred, area of items which might or might not be addressed by it.

= Work Tasks =
== API ==
The service insertion API might be implemented as a Quantum extension for the Grizzly release cycle, or be part of the core if there is a total agreement on it. Service insertion support will require the following changes in the Quantum API:

* Service Type management. CRUD operations should be available on a [[ServiceType]] object.
* Regular tenants typically should be allowed to read only
* Administrators would have right to create, modify, and delete those object. Also, the provider attribute might be hidden in response returned to regular tenants.

''Note'': default policy settings might always be overridden as they're specified in ''etc/policy.json''

The Router resource should therefore be extended with the following attribute:

<pre><nowiki>
services:service_type_id # which refers to service_type object
</nowiki></pre>

Each advanced service should allow for specifying either a service type id (for floating mode insertion) or a router_id (for routed mode insertion).

If a router supports multiple services, the tenant might be given the ability of enabling or disabling specific services on a router, in accordance with the service_type associated with the router. This feature might be used by tenants to keep advanced service usage within quota limits.

* Option 1: Resource action /routers/<router_id>/enable_service
** /routers/<router_id>/disable_service
* Option 2: List Attribute
* Option 3: Sub-Resource POST /routers/<router_id>/enabled_services

Option 1 is currently the preferred one, as it is consistent with "the Openstack way" of defining APIs executing actions on resources beyond the ones that can be expressed as HTTP methods.

== Service Configuration ==
The configuration file should contains references to the drivers/plugins corresponding to the various advanced services. These information will be used by the plugin manager.

The configuration file might also contain service type definitions, which could alternatively be stored in the Quantum 'core' database (see section on Data Model Changes).

== Plugin Manager ==
A mechanism for loading multiple plugins will be required for the multi-plugin approach. The [[PluginAwareExtensionManager]] does not manage plugin loading, but some work will be required on it for handling plugin-specific API extensions, as it currently assumes that there's only a single plugin.

== Data Model Changes ==
[[ServiceType]] definition, if modeled into the Quantum DB will require two model classes. The [[ServiceType]] class will have a 1:n relationship with the [[ServiceDefinition]] class. This class will have an attribute for defining the type of the advanced services provider. This could either be another class in the model or extracted from the Quantum configuration file.

[[ServiceProvider]] definition could as well be either another model class or information extracted from the configuration file.

== Changes to the basic plugin Interface ==
The service insertion feature will add a new interface which might be implemented as a new mixin to be inherited by the "core" plugin (a completely standalone class is a viable alternative, albeit less straightforward)

== Managing API dispatching ==
This task is about implementing the single plugin/multiple drivers or multi-plugin approaches, as discussed earlier in this spec document.

= Potential changes to the current implementation =
# Floating IPs: despite being shipped with Folsom, this feature might actually be regarded as an 'advanced service', and should probably fit in the service insertion framework.
# External Gateway: This might be possibly be regarded as an advanced service too. For instance, a SNAT service.
# DHCP: think about how the DHCP service we implement today through the agent fits in the service insertion model. We are not exposing specifically any DHCP API, creating implicitly an instance of this service for each subnet for which ''dhcp_enable=True''. We can either leave DHCP as it is, or include it in the service insertion model.

NOTE: For the cases listed above it is very important to preserve backward compatibility.

= POC Proposal =
A POC of the service insertion model could be offered regardless of actual advanced services being implemented on top of Quantum. This POC might use 'dummy' advanced services plugins. Another options would be reworking the Floating IP and possibly also the external gateway concepts as advanced services and use them for the PoC.

= Mapping tasks to Grizzly milestones =
{| border="1" cellpadding="2" cellspacing="0"
| '''Task'''
| '''Importance'''
|-
| Service Type definition
| Essential
|-
| Plugin/Driver management
| Essential
|-
| API Call Dispatching
| Essential
|-
| PoC with dummy plugins
| High
|-
| PoC with LBaaS
| Essential
|-
| APIs for service type management
| Undecided
|-
| Service types in Quantum DB
| Normal
|-
| Reworking some L3 capabilities as adv svc
| Undecided

Neutron/ServiceInsertion

2013-02-16T22:22:12Z

Olaph:

__NOTOC__
= WORK IN PROGRESS =
We are integrating all the information related to Services Insertion in only one wiki page

= Services Insertion Model in Quantum =
* '''Launchpad Entries''':

''''' [https://blueprints.launchpad.net/quantum/+spec/quantum-service-type services-type] '''''

''''' [https://blueprints.launchpad.net/quantum/+spec/services-insertion-wrapper services-insertion-wrapper (folsom)] '''''

* '''Contributors''': [https://launchpad.net/~salvatore-orlando Salvatore],[https://launchpad.net/~emagana Edgar Magana], [https://launchpad.net/~enikanorov Eugene], Ram Durairaj, Mani Ramasamy and Quantum Community.

== Description ==
This page contains the design spec for the service insertion blueprint. Despite being quite fleshed out the spec is not finalised yet, as some details are still subject to change, especially as far as the implementation plan is concerned. Please subscribe to this page to get every update, or check it as often as necessary. The blueprint itself can be found at this address:

This specification does not fully respects the blueprint template set for Quantum. This is because we are giving much more space to the design discussion. We will have another wiki page using the blueprint template once the design discussion is Finalized.

== High level description ==
The ''service insertion'' feature aims at defining a framework for running L4/L7 network services on Quantum Logical Topologies. This framework will be immediately leveraged by the LBaaS effort; and in the near future by other advanced services which will be plugged into Quantum (for instance, VPN, and edge firewalls).

== Defining Service Insertion ==
Before going in depth into design and implementation discussion, it might be worth to define in a detailed way what an advanced service is, and how it could be inserted in the Quantum logical topology. The diagram below show Quantum's basic network topology, with a logica router interconnecting internal and external networks; ports, which are used for VIF, DHCP server, router, and floating IP attachments are also shown in the diagram.

[[Image:basic-topology.png]]

From previous discussions it emerged that services can be inserted in the following modes:

* Routed mode - Where the service is attached to a logical router, which then becomes a multi-service appliance.

[[Image:routed-si.png]]

* Floating Mode (In-path) - Where service runs in a standalone way. Please note that the "Floating" insertion is not automatically a network level insertion, as L3 routing might still occur. This mode has also been referred as "in-path or bump-in-the-wire" insertion.

[[Image:in-path-si.png]]

With reference to the diagram above, is is worth noting that an advanced service, even when inserted in floating mode, should still 'plug' into networks. In the example above, the service plugs into the external network and into an internal network. This should be reflected by ports on the relevant Quantum networks ''owned'' by the advanced service itself.

* Re-directed Mode (Out-of-path) - Where the service also runs in a standalone way but in this case the traffic is first sent to the Router entity and then redirected to the Advanced Service, finally send it back to the routed with specific configuration. In particular, this model could be reduced to the first assuming that a standalone service is regarded as a peculiar case of router capable of providing only a specific service. This mode needs specific changes at the routing entity and may not be implemented in Grizzly release.

[[Image:out-of-path-si.png]]

All these three modes are not mutually exclusive. The diagram below shows how they can be combined. With reference to the diagram below, it is worth noting that the services inserted in routing and floating modes could have different implementations. This means for instance, that instances on the same Quantum networks could be load balanced using two different solutions.

[[Image:mixnmatch-si.png]]

== The Service Type concept ==
Just like the Quantum plugins allow for using several technologies for implementing the basic logical topologies, advanced services will use a similar mechanism. However, for advanced services, multiple different implementations of the same kind of service might co-exist in the same deployment. There are a number of reasons for this, most importantly the ability of giving tenants a choice among solutions. The '''Service Type''' concept tries to address the need for multiple, co-existing, service providers.

A '''Service Type''' definitions might be regarded as list of services (and their providers) which can be offered to tenants. Each advanced service, regardless of its insertion mode, should be either directly or indirectly associated with a single service type.

The Service Type resource might be described as follows:

<pre><nowiki>
ServiceType:
{
id: uuid,
name: string,
service_definitions: list<ServiceDefinition>
default: {True¦False} # Only one service_type could be the default one. This is the service type which should be picked if none is specified in the API (think backward compatibility)
}

ServiceDefinition:
{
id: uuid,
name: string,
type: enum(LB, VPN, FW) # or whatever you think is right
provider: string # This ultimately must map to a python class (perhaps through a configuration variable)
capabilities: list of API extensions which are enabled for this specify service provider
}
</nowiki></pre>

Please note that the ''[[ServiceDefinition]]'' object might be regarded either as a resource of its own, or as a child resource of ''[[ServiceType]]''. The association between a service and a service type can happen in two ways, according to the insertion mode of the service.

* ''Routed'' Insertion mode: The advanced service will be associated with Quantum logical router, which in turn is associated with a service_type resource;

In order to ensure backward compatibility a default service type must be specified. This implies that all the services which will be inserted on a router will share the same service type.

* ''Floating'' Insertion mode: The service type should be explicitly specified on the advanced service being created; if not, the default service type will be used.

When an advanced service is created at the API layer one of the following two should be specified:

# service_type_id # floating or out-of-path insertion
# router_id # routed or in-path insertion

It should not be allowed to specify both parameters.

The logical model for service insertion, augmented with the service type concept, is depicted in the following diagram:

[[Image:svc-type.png]]

The diagram shows a set of services deployed in routed mode (light blue), and a service deployed in floating mode (purple). Light blue and purple services are associated with different service types, which then can map to distinct implementations of the same services.

== Dispatching calls to the plugins ==
The concept of service type implicitly allows for different paths for an API call, as it allows multiple providers to serve the same request. Eugene Nikanorov also addresses this topic in this wiki page: http://wiki.openstack.org/Quantum/ServiceIntegration

There are two design alternatives to be considered:

# A single plugin, augmented with the capability of serving requests for advanced services; call will then be redirected to provider-specific drivers.
# Multiple, self-contained plugins. Each plugin is specific to a given service provider and might implement one or more advanced service interfaces.

For both design alternatives, it is pretty clear that each advanced service should have a 'service plugin interface', which is the plugin-side dual of the tenant-facing APIs. The core Quantum plugin similarly, defines a plugin interface that all plugins must implement.

=== Single Plugin Approach ===
The diagram below show a Quantum Plugin which implements both core and advanced services interface. The plugin is capable of interpreting the service_type attribute, and dispatching the call to the appropriate, provider-specific driver.

[[Image:single-plugin.png]]

For this approach it should be possible to leverage the "mixin" mechanism which proved successful when integrating DHCP and L3 services. The possibility of augmenting the Quantum plugin with the capability of handling advanced services should definitely be allowed. In this case there will still be a single Quantum plugin; Appropriate ''service drivers'' should be specified in the configuration file. For each service multiple drivers might be specified; The service driver should not be seen as a driver for a specific appliance implementing the service. It is rather a driver managing the service for a given provider.

=== Multiple Plugins Approach ===
The diagram below show how service type are mapped to multiple plugins when adopting this approach. Each plugin implements an interface which is specific for the kind of service it implements. The API layer has a dispatcher component which forwards the call to the appropriate plugin, according to the service type associated with the advanced service specified at the API layer.

[[Image:multi-plugin.png]]

It is a Quantum principle that plugins are required only to implement the plugin interface. Hence they are not required to adopt the mixin mechanism; hence another aspect to consider is that there could be multiplie plugins at the same type, and the intelligence of dispatching a call to one plugin or another should reside in the API layer.

In this case several plugins might be configured at the same time. The API layer will need to figure out, for each router and/or advanced service, to which plugin the API call should be dispatched. It is also worth noting that a plugin should not be constrained to implement a single advanced services. It might well be that plugins implement a set of services; this is particularly the case of plugins which will be interfaced with integrated services appliances or application delivery controllers.

In this case the diagram above could be generalised as follow (the reference to the configuration file is an implementation detail at this stage):

[[Image:multi-plugin-2.png]]

=== Evaluation of pro and cons of both Approaches ===
The diagrams below depicts, at a very high level, the different flows of the single versus multiple plugins approaches. The single plugin approach is reported on the right, whereas the multiple plugin approach is reported on the left.

[[Image:single-vs-multi.png]]

* multiple plugins will require the API to manipulate data from different sources. Even if each plugin will return data in the same format, we will still need logic to handle collecting data from multiple sources on GET requests, and dispatching commands to the appropriate destination on POST/PUT/DELETE.
* currently all plugins already implement the "mixin" approach for augmenting the L2 feature with L3 and DHCP features. This mechanism could be extended.
* compatibility between plugins (not something we need to worry from day 1, but still an interesting problem)
* interactions among plugins; with a single plugin, the data model has all the information it needs. With multiple plugins will have to interact with the "base" plugin (or in some cases even among them); this could be achieved throughout the plugin interfaces.
* the single plugin, multiple drivers, model works very well when the Quantum database holds all the required information necessary to operate. This implies that items such as device management, resource allocation, and similars, are described using a model shared by all plugins. Although this might be mitigated by either introducing driver-specific extensions, or moving such capabilities in the driver, this is clearly a case in which the multiple plugins approach is preferred.

== A few notes on the scope of this blueprint ==
This blueprint which focuses on:

* Providing the infrastructure for allowing implementations of these services to serve API requests; multiple implementations for each class of service should be allowed to coexist at the same time in a Quantum deployment.
* Defining a set of API calls (as well supporting logic and data model) for defining which, where, and how services might be attached on the logical topology defined by the basic Quantum topology;
* Allowing providers of such services to expose implementation-specific service extensions and advertising them to API users.

Defining which services might be inserted, and how they should be presented to tenants is beyond the scope of this blueprint. Activity is already ongoing for the Load Balancing service; for more information please refer to lbaas-* blueprints in the Quantum framework.

Most of the concepts expressed in this blueprint are not new to the Quantum community and can be found in this wiki page authored by Edgar Magana from Cisco: http://wiki.openstack.org/QuantumServicesInsertion

''Further notes (or what this design specification is not about):''

* There might be scenarios where service insertion happens at network level, as shown in Sasha Ratkovic's presentation. For the scope of this blueprint we will consider only router-level insertion, assuming that network-level insertion could be reduced to a case in which the service is inserted on a router connected to a single network only.
* There are also definitely scenarios where service insertion happens at the port level; in this case the insertion might be simply represented by attributes applied to the port itself. For an example please refer to the proposed https://review.openstack.org/#/c/14262/.
* The concept of port group, presented by Sasha Ratkovic at the Openstack Design Summit is also outside of the scope of this blueprint.
* Whether advanced services will be realized as new resource (as described in the LBaaS proposal) or as policies (as described by Sasha Ratkovic proposal), is also beyond the scope of this blueprint.

At this stage the careful reader might be wondering whether this blueprint is actually about anything. With all due honesty by defining a (small) set of items we want to address, and a (very large) of items that we definitely do not want to address, it might be argued that this blueprint is actually pretty well scoped, leaving a very thin, blurred, area of items which might or might not be addressed by it.

= Work Tasks =
== API ==
The service insertion API might be implemented as a Quantum extension for the Grizzly release cycle, or be part of the core if there is a total agreement on it. Service insertion support will require the following changes in the Quantum API:

* Service Type management. CRUD operations should be available on a [[ServiceType]] object.
* Regular tenants typically should be allowed to read only
* Administrators would have right to create, modify, and delete those object. Also, the provider attribute might be hidden in response returned to regular tenants.

''Note'': default policy settings might always be overridden as they're specified in ''etc/policy.json''

The Router resource should therefore be extended with the following attribute:

<pre><nowiki>
services:service_type_id # which refers to service_type object
</nowiki></pre>

Each advanced service should allow for specifying either a service type id (for floating mode insertion) or a router_id (for routed mode insertion).

If a router supports multiple services, the tenant might be given the ability of enabling or disabling specific services on a router, in accordance with the service_type associated with the router. This feature might be used by tenants to keep advanced service usage within quota limits.

* Option 1: Resource action /routers/<router_id>/enable_service
** /routers/<router_id>/disable_service
* Option 2: List Attribute
* Option 3: Sub-Resource POST /routers/<router_id>/enabled_services

Option 1 is currently the preferred one, as it is consistent with "the Openstack way" of defining APIs executing actions on resources beyond the ones that can be expressed as HTTP methods.

== Service Configuration ==
The configuration file should contains references to the drivers/plugins corresponding to the various advanced services. These information will be used by the plugin manager.

The configuration file might also contain service type definitions, which could alternatively be stored in the Quantum 'core' database (see section on Data Model Changes).

== Plugin Manager ==
A mechanism for loading multiple plugins will be required for the multi-plugin approach. The [[PluginAwareExtensionManager]] does not manage plugin loading, but some work will be required on it for handling plugin-specific API extensions, as it currently assumes that there's only a single plugin.

== Data Model Changes ==
ServiceType definition, if modeled into the Quantum DB will require two model classes. The ServiceType class will have a 1:n relationship with the ServiceDefinition class. This class will have an attribute for defining the type of the advanced services provider. This could either be another class in the model or extracted from the Quantum configuration file.

ServiceProvider definition could as well be either another model class or information extracted from the configuration file.

== Changes to the basic plugin Interface ==
The service insertion feature will add a new interface which might be implemented as a new mixin to be inherited by the "core" plugin (a completely standalone class is a viable alternative, albeit less straightforward)

== Managing API dispatching ==
This task is about implementing the single plugin/multiple drivers or multi-plugin approaches, as discussed earlier in this spec document.

= Potential changes to the current implementation =
# Floating IPs: despite being shipped with Folsom, this feature might actually be regarded as an 'advanced service', and should probably fit in the service insertion framework.
# External Gateway: This might be possibly be regarded as an advanced service too. For instance, a SNAT service.
# DHCP: think about how the DHCP service we implement today through the agent fits in the service insertion model. We are not exposing specifically any DHCP API, creating implicitly an instance of this service for each subnet for which ''dhcp_enable=True''. We can either leave DHCP as it is, or include it in the service insertion model.

NOTE: For the cases listed above it is very important to preserve backward compatibility.

= POC Proposal =
A POC of the service insertion model could be offered regardless of actual advanced services being implemented on top of Quantum. This POC might use 'dummy' advanced services plugins. Another options would be reworking the Floating IP and possibly also the external gateway concepts as advanced services and use them for the PoC.

= Mapping tasks to Grizzly milestones =
{| border="1" cellpadding="2" cellspacing="0"
| '''Task'''
| '''Importance'''
|-
| Service Type definition
| Essential
|-
| Plugin/Driver management
| Essential
|-
| API Call Dispatching
| Essential
|-
| PoC with dummy plugins
| High
|-
| PoC with LBaaS
| Essential
|-
| APIs for service type management
| Undecided
|-
| Service types in Quantum DB
| Normal
|-
| Reworking some L3 capabilities as adv svc
| Undecided

Rebuildforvms

2013-02-16T22:06:51Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:rebuild-for-ha
* '''Created''': 2011-Nov-30
* '''Contributors''': Jason Kim

== Summary ==
Support HA for vms from the failed host.

== Release Note ==
When this is implemented, VMs will be rebuilt from failed host to the other hosts.

== Rationale ==
If the host is failed, we can't run the current vms on the host.
That is to say, we can't HA for vms.
I solved the HA using the rebuild means.

Below steps will be added.

1) get the information of instances on the failed host.
- we can know the state of host by using monitoring system or using the service state (nova-compute, nova-network)

2) get a list of dictionaries of network data of an instance.

3) update the volume db and instance db.

4) setup volumes for block device mapping

5) spawn the instance.

6) associate the floating ip which is used existing the instances.
- set up the iptables and so on.

7) update the instance db.

== User stories ==
* User doesn't know the vm is rebuilt.
* Operator gets the list of failed host from monitoring system or nova network/compute state.
* Operator gets the lists of the instances on the failed hosts.
* Operator should rebuild the instances by using this operation.

== Assumptions ==
The instances were created with volumes attached.

* [https://blueprints.launchpad.net/nova/+spec/auto-create-boot-volumes auto-create-boot-volumes]

== Design ==
[[Image:rebuild_for_vm_concept.png]]

== Implementation ==
* Scheduler should call the compute manager to rebuild the instance.
* Compute Manager should get a list of dictionaries of network data of an instance.
* Compute Manager should update the volume db and instance db.
* Compute Manager should setup volumes for block device mapping by using the volume manager.
* Compute Manager should spawn the instance by using the virt driver.
* Compute Manager should associate the floating ip by using the network manager.
* Compute Manager should update the instance db.
* Compute Manager should restart the network module.

=== Code Changes ===
* /usr/bin/nova-manage
* /nova/compute/api.py, vm_states.py, task_state.py, manager.py
* /nova/api/openstack/common.py
* /nova/network/api.py, manager.py

== Test/Demo Plan ==
This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==
TBD

== BoF agenda and discussion ==
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

----
[[Category:Spec]]

FederatedAuthZwithZones

2013-02-16T22:05:18Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:nova-zones-federated-authz
* '''Created''': [[SandyWalsh]]
* '''Contributors''':
* '''Related''': [[ZonesOauth]] http://etherpad.openstack.org/nova-oauth

== Summary ==
In the current Zones implementation, all operations to child zones are done with an admin account. While this works, it causes problems in certain important use cases. Specifically when a user wishes to see all the instances that he may manage. With the admin-account approach, all instances in child zones are returned when only a subset should be (depending in the users rights).

This page is the culmination of discussions with eday, vish, jorge, khaled and others and attempts to summarize the problems that may arise in other use cases as well as documenting the proposals currently underway for solving them.

== Revision History ==
# First draft - AuthZ servers communicated and synced. [[MyCo]] users were actively added to Resource Groups.
# Realized that SP AuthZ doesn't need to know about users, just Resource Groups. Introduced concept of Behalf Of. Removed AuthZ sync.

== Release Note ==

== Rationale ==

== Assumptions ==

This will be the common scenario for the bulk of these use cases.

[[MyCo]] is a small software development shop. They have a Nova deployment made up of two Zones (parent and child). When these two zones are incapable of providing more instances, [[MyCo]] outsources to Service Provider (SP) to handle the extra work. SP is large and has many Zones geographically separated and nested many levels deep.

There are two main users in our use cases: Alice and Bob. Alice manages her own instances, as does Bob. However, Alice and Bob are also members of a Financial Task Force which gives them the ability to manage any instances under that groups control.

There will be one set of AuthN and AuthZ services per business. [[MyCo]] has it's own AuthN/Z service as does SP. These Auth services will need to be Highly Available.

Whenever a user is added/removed from a security group or permissions changed they are done on the [[MyCo]] side. Only the Resource Group ID's are passed to the SP.

After authenticating, the [[MyCo]] AuthZ will supply a set of data to SP regarding the User. This set of data includes:
* The Resource Group ID's that the user belongs to (and not the Resource ID's within that group, that data only lives in [[MyCo]] AuthZ)
* Optionally, a Resource Group ID that the user is acting on Behalf Of.
* A set of Permission Tuples ([Action Group, ...], [Resource Group, ...]) of the actions the User can perform on which Resource Groups.

For child zones within the same business (ie. all the child zones in Service Provider) group ID's and resource ID's will have to be unique across Zones.

In a multi-tenant scenario such as Service Provider, the Group identifiers passed from one AuthZ system to another will need to be prefixed/namespaced to avoid collisions.

[[Image:UseCaseOverview.gif]]

== Terminology ==

* [[MyCo]].authZ = The AuthZ service running at [[MyCo]]
* SP.authZ = The AuthZ service running at the Service Provider
* Subject = The user performing the action.
* Delegate = A user performing an action on behalf of another user.
* Subject Group = A group of users or delegates.
* Action = The verb being performed on the resource. Such as: boot, halt, pause, rescue, migrate, view, delete, etc.
* Action Group = A collection of Actions.
* Resource = The object the Subject is performing the Action on (aka 'Object' in our previous discussions)
* Resource Group = A group of Resources.

== Design ==

After authenticating Alice with the Service Provider Zones, [[MyCo]].authZ will supply SP.authZ with a set if ''Permission Tuples'' that indicate which operations Alice may perform and which Resources she may perform them on (where applicable). These tuples have the mapping (Action_Groups, Resource_Groups). Note that Subject is implied here since the User had to previously authenticate with the system to gain access.

Side note: There was a discussion of having the tuple be (Action_Group, User_Group) where User_Group was a collection of Users that owned resources. The downside with this approach is it's not fine-grained enough. For example, if Bob has permissions (can_halt, [Alice]) then he can halt ''all'' instances owned by Alice. That's too much privilege. Additionally, the difference between tracking a User_Group and a Resource_Group is nominal. Tracking a Resource_Group closer aligns with

We don't want the tuples to be so fine grained that we are listing individual instance ID's ... it won't scale to thousands of instances. While it would be easier, we don't want permission tuples of (Action_Group, [Resources ID's]) like (can_halt, [ami-AAA, ami-BBB, ami-CCC, ...]). Instead we need to define a Group of resources at the SP.authZ layer and simply refer to it in our permission tuples.

How? In order to do this, [[MyCo]].authZ needs to keep SP.authZ updated with a discrete set of Resource Groups.

[[MyCo]].authZ will have many Resource Groups and much of the contents of these Resource Groups will contain resources that live in [[MyCo]] Zones. We don't need to replicate all of this information to SP.authZ ... only a subset of the group that references resources that live in SP Zones. For example, Bob's permitted groups on the [[MyCo]].authZ side may look like:

<pre><nowiki>
(Bob, [can_halt, can_see, can_boot, ...], [myco-Foo-group, myco-Bar-group, myco-Zoo-group, sp-Alice-group, sp-Bob-group sp-Finance-group])
</nowiki></pre>

but when passing this on to SP.authZ only the following is required:

<pre><nowiki>
([can_halt, can_see, can_boot, ...], [sp-Alice-group, sp-Bob-group sp-Finance-group])
</nowiki></pre>

You can think of this as a formalization of the "override" capability in the authn/z branch proposed by vish & termie.

[[Image:SyncAuthZ.gif]]

=== On Behalf Of ===
One complexity we have is creating new Resources on the SP side. We want to make sure we assign the correct Resource Group to the Resource. There are a couple of possibilities:
# When the user authenticates we add a "On-Behalf-Of=<Resource Group ID>" header, which indicates how new resources should be assigned. This may be tricky to pre-determine.
# We always create a new Resource Group when creating new Resources and later, as needed, nest this Resource Group in other Resource Groups for granting permissions.

== User stories ==

== Implementation ==

=== Migration ===

== Test/Demo Plan ==

== Unresolved issues ==
http://paste.openstack.org/show/1108/

== BoF agenda and discussion ==

----
[[Category:Spec]]

EfficientVolumesFromImage

2013-02-16T22:04:27Z

Olaph:

* '''Launchpad Entry''': [[NovaSpec]]:efficient-volumes-from-images
* '''Created''': 03 April 2012
* '''Contributors''': Josh Durgin, Tommi Virtanen

== Summary ==

Make volume (and instance) creation fast if Nova volumes and Glance
use the same storage backend, and the storage backend supports it.

== Release Note ==

Instance and volume creation can take advantage of underlying storage
to be more efficient in space and time.

== Rationale ==

Storage systems offer a number of features that can save space and
time when storing both images and volumes:

* features like copy-on-write can be used to make volume (and instance creation) near-instantaneous and save space
* deduplication can happen on images and volumes together, reducing hardware requirements
* administrators only have to maintain one storage system

== User stories ==

User creates a bunch of new instances that have no ephemeral disks,
but are all based on the same image. This happens very quickly due to
a shared underlying storage system, and requires very little extra
space.

== Assumptions ==

Depends on
[https://blueprints.launchpad.net/nova/+spec/create-volume-from-image create-volume-from-image]
and a way to query Glance about the backing store of an image.

== Design ==
[[Image:cow_volume_from_image.png]]

== Implementation ==

There are two parts that are volume backend dependent:

# determining whether a given image is stored on the same backend as volumes
# efficiently creating a volume based on the image

These can both be optional volume driver methods - something like
`def cloneable(image_info)` and `def clone(image_info, volume_name)` - if they aren't implemented, or cloneable returns
false, fallback to the open/write/close api from
[https://blueprints.launchpad.net/nova/+spec/create-volume-from-image create-volume-from-image].

=== Migration ===

No database or core API changes.

== Test/Demo Plan ==

This need not be added or completed until the specification is nearing beta.

== BoF agenda and discussion ==

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

----
[[Category:Spec]]

UnifiedInstrumentationMetering

2013-02-16T22:03:46Z

Olaph:

= Unified Instrumentation and Metering =

Last edit: -- [[SandyWalsh]]

== Overview ==

With <code><nowiki>Ceilometer</nowiki></code>, <code><nowiki>Tach</nowiki></code> and <code><nowiki>[[StackTach]]</nowiki></code>, there are some very cool initiatives going on around instrumentation and metering/monitoring within [[OpenStack]] today.

However, due to the incredible speed of development within [[OpenStack]] many of these efforts have been performed in isolation from each other. Now, we are reaching a level of maturity which demands we stop reinventing the wheel and agree upon some shared infrastructure. This need is necessary for a variety of reasons:

# We want to make it easier for new projects to build on the existing [[OpenStack]] notification message bus.
# Less code is good. We shouldn't need three different workers for extracting notifications from [[OpenStack]].
# Notifications are large and there's a lot of them. We only want to process and store that data once.
# Archiving of data is a common problem, we shouldn't need several different ways of doing it.

In this document we'll talk about the State-of-Art with respect to Instrumentation/Metering/Monitoring (IMM) with [[OpenStack]] today and where we might go with it.

Required reading / viewing:
* http://www.openstack.org/summit/san-diego-2012/openstack-summit-sessions/presentation/what-about-billing-an-introduction-to-ceilometer
* http://www.sandywalsh.com/2012/10/debugging-openstack-with-stacktach-and.html
* https://etherpad.openstack.org/grizzly-common-instrumentation (good summary of instrumentation needs)

= Instrumentation vs Metering/Monitoring =

At the very base of this discussion is having a clear understanding of the difference between Instrumentation and Metering/Monitoring.

== Instrumentation ==
Think of instrumentation as the way they test electronics in-circuit. While the device is running, probes are attached to the circuit board and measurements are taken.

[[Image:instrumentation.png]]

There are some key things to consider in this analogy:
* Every technician may want to place their probes in different locations.
* Probes might be placed for long term measuring or transient ("I wonder if ... " scenarios)
* The circuit does not have to change for the testing probes to be placed. No other groups or departments had to be involved for this instrumentation to occur.
* The same probe technology can be used on other circuit boards. Likewise, our instrumentation probe-placement technology should not just be geared towards Nova. It should also work with all other parts of [[OpenStack]].
* When the circuit changes our probe placement may have to change. We have to be aware of that.
* The probes aren't perfect. They might slip off or have a spotty connection. We're looking for trends here, identifying when things are slow or flaky.
* With respect to Python, we may be interested in stack traces as well. Not just single function timings/counts.

== Metering / Monitoring ==
Metering is watching usage of the system, usually for the purposes of Billing. Monitoring is watching the system for critical system changes, performance and accuracy, usually for things like SLA's.

Think of your power meter. You can go outside and watch the dial spin and confirm your monthly bill jives with what the meter is reporting.

[[Image:metering.png]]

The important aspects of metering and monitoring:
* These events/measurements are critical. We cannot risk dropping an event.
* We need to ensure these events are consistent between releases. Their consistency should be considered of equal importance as [[OpenStack]] API consistency.
* We have no idea how people are going to want to use these events, but we can safely assume there will be a lots of other groups interested in them. We don't want these groups talking to the production [[OpenStack]] deployments directly.
* These events may not be nearly as frequent as the instrumentation messages, but they will be a lot larger since the entire context of the message needs to be included (''which'' instance, ''which'' image, ''which'' user, etc)

= The State-of-the-Art =

So, where are we?

== Instrumentation Today ==

There is no formal instrumentation solution for [[OpenStack]] today (other than logging). Logging is insufficient because it's:
* unstructured text
* too much effort to scan, parse and correlate across servers
* requires code changes to collect new information

Rackspace has being using a cocktail of [https://github.com/ohthree/tach Tach], [https://github.com/etsy/statsd Statsd], [http://graphite.wikidot.com/ Graphite] and [http://www.nagios.org/ Nagios] to great success for close to a year.

[[Image:instrumentation_arch.gif]]

Tach is a library that collects timing/count data from anywhere in a Python program. It's not specific to [[OpenStack]], but it has pre-canned config files for the main [[OpenStack]] services. Tach hooks into a programming using monkey-patching and has a concept of Metrics and Notifiers. The Metrics are user-extensible hooks that pull data from the code. The Notifiers take the collected data and send it somewhere. Currently there are Metrics drivers for execution time and counts as well as Notifiers for Statsd, Graphite directly, print and log files. ([[SandyWalsh]] has been working on a replacement for Tach, called [https://github.com/SandyWalsh/scrutinize Scrutinize] which adds cProfile support and easier configuration. It's almost ready for prime-time.)

Tach is launched as: <code><nowiki>tach tach.conf nova-compute nova.conf</nowiki></code> ... so it easily integrates with existing deployments.

The powerful features of statsd are:
* UDP based messaging, so production is not at risk if the collectors die.
* In-memory rollup/aggregate of measurements that get relayed to Graphite. This greatly enhances scalability.
* Written in node.js = fast, fast, fast.

== Monitoring/Metering Today ==

Certainly the face of Monitoring and Metering within [[OpenStack]] today is [https://launchpad.net/ceilometer Ceilometer] ... I'll just refer you to that project for more details.

But there are others. Within Rackspace we use [https://github.com/rackspace/yagi YAGI] to consume the notifications and send them to our internal billing system. Specifically, this data is sent to [http://atomhopper.org/ AtomHopper] where it is turned into an Atom feed for other consumers (one of which is billing). YAGI used to have [https://code.google.com/p/pubsubhubbub/ PubSubHubBub] support, but that's gone dormant due to other motivators. Now, [[AtomHopper]] is the redistribution system of choice. Sadly, [[AtomHopper]] is Java-based, so it may not work well within the [[OpenStack]] ecosystem, per se. The YAGI Worker uses [https://github.com/ask/carrot/ carrot] and has been highly reliable in all of our environments, but there has been discussion of moving to [http://pypi.python.org/pypi/kombu kombu].

[[StackTach]] is a debugging/monitoring tool based on [[OpenStack]] notifications and it too has its own Worker. It is kombu-based and is currently used in production. We've had lots of problems making the stacktach worker reliable but we think the problem has been with combining a threading model with eventlet. Our new scheme uses the multiprocessing library with per-rabbit workers. This is currently being stress tested, stay tuned. We've tried a variety of other schemes and library combinations with little success (more detail if needed). Note: this ''should'' work fine since it's the same scheme Nova uses internally.

[[StackTach]] is quickly moving into Metrics, SLA and Monitoring territory with version 2 and the inclusion of Stacky (the command line interface to [[StackTach]])

[[Image:notifications.gif]]

= The Layers =

With every architecture there are different layers of abstraction/functionality.

IMM has:
* ''low-level'' components that directly interface with [[OpenStack]] via monkeypatching or notification consumption.
* ''mid-level'' components that collect, aggregate and redistribute this collected data to other consumers
* ''high-level'' components that act as the presentation layer.

We currently have overlap on all levels.

[[Image:layers.gif]]

== Unifying the Low Level ==

There is really no way to unify the instrumentation collection with the metrics collection infrastructure within [[OpenStack]]. As stated in the introduction, these are very different animals.

With respect to Monitoring and Metering (notification consumption), the obvious candidate is the notification worker. Using a list of queues in the rabbit notifier is not a solution, it's a big load on the rabbit server.

''Note: we're also using AMQP incorrectly for notifications. Events are published to an Exchange and different queues can be created for different consumers. Currently, [[OpenStack]] creates a different Exchange for each notification destination. This should simply a bug that should be fixed.''

This is the low-hanging fruit. Creating a scalable worker that can work in a multi-rabbit (multi-cell, mult-region) deployment. It should support a pluggable scheme for handling the collected data and support failover/redundancy. The worker has to be fast/reliable enough to keep the notification queue empty since this is easily the fastest growing queue (neck and neck with the cells capacity update queue :)

The worker should be a separate repository so others can use it standalone.

Also, the Ceilometer worker doesn't need to use all of the nova-common code. This is very simple program and can be handled with a single file.

== Unifying the Mid-Level ==

[[Image:architecture.gif]]

=== A Common Database for Collected Data ===

For metering and monitoring, both [[StackTach]] and Ceilometer have a database to store the notifications. [[StackTach]] uses a SQL-based database, while Ceilometer uses a key-value database (Mongo). Each notification is large and contains about ten columns that are required to have fast lookup:
* Deployment ID
* Tenant ID
* UUID
* Request ID
* Timestamp
* Instance State
* Instance Task
* Publisher
* Host ID
* Event name

Additionally the entire JSON payload should be stored for the event for consumers that need something specific.

I have no recommendation on which db to use, but perhaps some load testing should be performed to see how well each works. The important thing is, the entire event gets stored.

However, as these events get collected, there are often additional tables that need to get updated which contain aggregated/summarized information from the events. While these operations could be done in a batch fashion in a separate database it's likely more efficient to provide a means for other applications to hook into the data collection and update these tables in real-time. This is what [[StackTach]] does and it greatly reduces the post-processing required. It does, however, come at a cost of additional processing when event storage is being performed (an expensive operation). Something like Ceilometer's secondary publishing step seems a good solution here, but I think it should be based on an [[AtomHopper]]/PSHB approach vs something proprietary. We should discuss what redistribution system looks like in greater detail.

In the diagram above Ext1, Ext2, Ext3 are user-plugins for doing special aggregation work. (not for sending to the redistribution system, another plugin mechanism could handle that)

For instrumentation, as mentioned above, the statsd in-memory database is a perfect solution here. Probably not much to improve on here.

=== A Common Event Redistribution System ===

As mentioned earlier, we use [[AtomHopper]] internally (and YAGI w/PSHB previously), but it would be nice to use an off-the-shelf framework, ideally something Python-based. This too should be a separate repo and, ideally, optional.

In the [[DreamHost]] use case, The Dude should consume from the [[AtomHopper]] feed based on webhooks vs. having to poll the Ceilometer API for updates.

=== Alerts / Set Points / Thresholds ===

Personally, I don't think this belongs in the Middle Layer ... external components should define/monitor/control these.

== Unifying the Top Layer ==
Every client is different, trying to service the needs of each user with a common user interface might be tricky. [[StackTach]] started as a debugging tool and targets that audience. A Billing tool will need a very different UI.

There are lots of off-the-shelf products available for the presentation of instrumentation data, we shouldn't need to reinvent that wheel.

[[StackTach]] and Stacky should both consume from the Ceilometer API.

== Other Improvements ==

* Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. That said, I'm sure there are specific things that certain deployments will need that aren't covered by this and the existing Ceilometer collector system will still be needed.
* [[StackTach]] has taken major steps in optimizing its REST interface to facilitate caching. The data in the middle layer is largely read-only and there are far more readers than writers. The Ceilometer API should consider these improvements (if it's not dealt with already and I missed it)

UnifiedInstrumentationMetering

2013-02-16T22:02:19Z

Olaph:

__NOTOC__
= Unified Instrumentation and Metering =

Last edit: -- [[SandyWalsh]]
<<[[TableOfContents]]()>>

== Overview ==

With <code><nowiki>Ceilometer</nowiki></code>, <code><nowiki>Tach</nowiki></code> and <code><nowiki>[[StackTach]]</nowiki></code>, there are some very cool initiatives going on around instrumentation and metering/monitoring within [[OpenStack]] today.

However, due to the incredible speed of development within [[OpenStack]] many of these efforts have been performed in isolation from each other. Now, we are reaching a level of maturity which demands we stop reinventing the wheel and agree upon some shared infrastructure. This need is necessary for a variety of reasons:

# We want to make it easier for new projects to build on the existing [[OpenStack]] notification message bus.
# Less code is good. We shouldn't need three different workers for extracting notifications from [[OpenStack]].
# Notifications are large and there's a lot of them. We only want to process and store that data once.
# Archiving of data is a common problem, we shouldn't need several different ways of doing it.

In this document we'll talk about the State-of-Art with respect to Instrumentation/Metering/Monitoring (IMM) with [[OpenStack]] today and where we might go with it.

Required reading / viewing:
* http://www.openstack.org/summit/san-diego-2012/openstack-summit-sessions/presentation/what-about-billing-an-introduction-to-ceilometer
* http://www.sandywalsh.com/2012/10/debugging-openstack-with-stacktach-and.html
* https://etherpad.openstack.org/grizzly-common-instrumentation (good summary of instrumentation needs)

= Instrumentation vs Metering/Monitoring =

At the very base of this discussion is having a clear understanding of the difference between Instrumentation and Metering/Monitoring.

== Instrumentation ==
Think of instrumentation as the way they test electronics in-circuit. While the device is running, probes are attached to the circuit board and measurements are taken.

[[Image:instrumentation.png]]

There are some key things to consider in this analogy:
* Every technician may want to place their probes in different locations.
* Probes might be placed for long term measuring or transient ("I wonder if ... " scenarios)
* The circuit does not have to change for the testing probes to be placed. No other groups or departments had to be involved for this instrumentation to occur.
* The same probe technology can be used on other circuit boards. Likewise, our instrumentation probe-placement technology should not just be geared towards Nova. It should also work with all other parts of [[OpenStack]].
* When the circuit changes our probe placement may have to change. We have to be aware of that.
* The probes aren't perfect. They might slip off or have a spotty connection. We're looking for trends here, identifying when things are slow or flaky.
* With respect to Python, we may be interested in stack traces as well. Not just single function timings/counts.

== Metering / Monitoring ==
Metering is watching usage of the system, usually for the purposes of Billing. Monitoring is watching the system for critical system changes, performance and accuracy, usually for things like SLA's.

Think of your power meter. You can go outside and watch the dial spin and confirm your monthly bill jives with what the meter is reporting.

[[Image:metering.png]]

The important aspects of metering and monitoring:
* These events/measurements are critical. We cannot risk dropping an event.
* We need to ensure these events are consistent between releases. Their consistency should be considered of equal importance as [[OpenStack]] API consistency.
* We have no idea how people are going to want to use these events, but we can safely assume there will be a lots of other groups interested in them. We don't want these groups talking to the production [[OpenStack]] deployments directly.
* These events may not be nearly as frequent as the instrumentation messages, but they will be a lot larger since the entire context of the message needs to be included (''which'' instance, ''which'' image, ''which'' user, etc)

= The State-of-the-Art =

So, where are we?

== Instrumentation Today ==

There is no formal instrumentation solution for [[OpenStack]] today (other than logging). Logging is insufficient because it's:
* unstructured text
* too much effort to scan, parse and correlate across servers
* requires code changes to collect new information

Rackspace has being using a cocktail of [https://github.com/ohthree/tach Tach], [https://github.com/etsy/statsd Statsd], [http://graphite.wikidot.com/ Graphite] and [http://www.nagios.org/ Nagios] to great success for close to a year.

[[Image:instrumentation_arch.gif]]

Tach is a library that collects timing/count data from anywhere in a Python program. It's not specific to [[OpenStack]], but it has pre-canned config files for the main [[OpenStack]] services. Tach hooks into a programming using monkey-patching and has a concept of Metrics and Notifiers. The Metrics are user-extensible hooks that pull data from the code. The Notifiers take the collected data and send it somewhere. Currently there are Metrics drivers for execution time and counts as well as Notifiers for Statsd, Graphite directly, print and log files. ([[SandyWalsh]] has been working on a replacement for Tach, called [https://github.com/SandyWalsh/scrutinize Scrutinize] which adds cProfile support and easier configuration. It's almost ready for prime-time.)

Tach is launched as: <code><nowiki>tach tach.conf nova-compute nova.conf</nowiki></code> ... so it easily integrates with existing deployments.

The powerful features of statsd are:
* UDP based messaging, so production is not at risk if the collectors die.
* In-memory rollup/aggregate of measurements that get relayed to Graphite. This greatly enhances scalability.
* Written in node.js = fast, fast, fast.

== Monitoring/Metering Today ==

Certainly the face of Monitoring and Metering within [[OpenStack]] today is [https://launchpad.net/ceilometer Ceilometer] ... I'll just refer you to that project for more details.

But there are others. Within Rackspace we use [https://github.com/rackspace/yagi YAGI] to consume the notifications and send them to our internal billing system. Specifically, this data is sent to [http://atomhopper.org/ AtomHopper] where it is turned into an Atom feed for other consumers (one of which is billing). YAGI used to have [https://code.google.com/p/pubsubhubbub/ PubSubHubBub] support, but that's gone dormant due to other motivators. Now, [[AtomHopper]] is the redistribution system of choice. Sadly, [[AtomHopper]] is Java-based, so it may not work well within the [[OpenStack]] ecosystem, per se. The YAGI Worker uses [https://github.com/ask/carrot/ carrot] and has been highly reliable in all of our environments, but there has been discussion of moving to [http://pypi.python.org/pypi/kombu kombu].

[[StackTach]] is a debugging/monitoring tool based on [[OpenStack]] notifications and it too has its own Worker. It is kombu-based and is currently used in production. We've had lots of problems making the stacktach worker reliable but we think the problem has been with combining a threading model with eventlet. Our new scheme uses the multiprocessing library with per-rabbit workers. This is currently being stress tested, stay tuned. We've tried a variety of other schemes and library combinations with little success (more detail if needed). Note: this ''should'' work fine since it's the same scheme Nova uses internally.

[[StackTach]] is quickly moving into Metrics, SLA and Monitoring territory with version 2 and the inclusion of Stacky (the command line interface to [[StackTach]])

[[Image:notifications.gif]]

= The Layers =

With every architecture there are different layers of abstraction/functionality.

IMM has:
* ''low-level'' components that directly interface with [[OpenStack]] via monkeypatching or notification consumption.
* ''mid-level'' components that collect, aggregate and redistribute this collected data to other consumers
* ''high-level'' components that act as the presentation layer.

We currently have overlap on all levels.

[[Image:layers.gif]]

== Unifying the Low Level ==

There is really no way to unify the instrumentation collection with the metrics collection infrastructure within [[OpenStack]]. As stated in the introduction, these are very different animals.

With respect to Monitoring and Metering (notification consumption), the obvious candidate is the notification worker. Using a list of queues in the rabbit notifier is not a solution, it's a big load on the rabbit server.

''Note: we're also using AMQP incorrectly for notifications. Events are published to an Exchange and different queues can be created for different consumers. Currently, [[OpenStack]] creates a different Exchange for each notification destination. This should simply a bug that should be fixed.''

This is the low-hanging fruit. Creating a scalable worker that can work in a multi-rabbit (multi-cell, mult-region) deployment. It should support a pluggable scheme for handling the collected data and support failover/redundancy. The worker has to be fast/reliable enough to keep the notification queue empty since this is easily the fastest growing queue (neck and neck with the cells capacity update queue :)

The worker should be a separate repository so others can use it standalone.

Also, the Ceilometer worker doesn't need to use all of the nova-common code. This is very simple program and can be handled with a single file.

== Unifying the Mid-Level ==

[[Image:architecture.gif]]

=== A Common Database for Collected Data ===

For metering and monitoring, both [[StackTach]] and Ceilometer have a database to store the notifications. [[StackTach]] uses a SQL-based database, while Ceilometer uses a key-value database (Mongo). Each notification is large and contains about ten columns that are required to have fast lookup:
* Deployment ID
* Tenant ID
* UUID
* Request ID
* Timestamp
* Instance State
* Instance Task
* Publisher
* Host ID
* Event name

Additionally the entire JSON payload should be stored for the event for consumers that need something specific.

I have no recommendation on which db to use, but perhaps some load testing should be performed to see how well each works. The important thing is, the entire event gets stored.

However, as these events get collected, there are often additional tables that need to get updated which contain aggregated/summarized information from the events. While these operations could be done in a batch fashion in a separate database it's likely more efficient to provide a means for other applications to hook into the data collection and update these tables in real-time. This is what [[StackTach]] does and it greatly reduces the post-processing required. It does, however, come at a cost of additional processing when event storage is being performed (an expensive operation). Something like Ceilometer's secondary publishing step seems a good solution here, but I think it should be based on an [[AtomHopper]]/PSHB approach vs something proprietary. We should discuss what redistribution system looks like in greater detail.

In the diagram above Ext1, Ext2, Ext3 are user-plugins for doing special aggregation work. (not for sending to the redistribution system, another plugin mechanism could handle that)

For instrumentation, as mentioned above, the statsd in-memory database is a perfect solution here. Probably not much to improve on here.

=== A Common Event Redistribution System ===

As mentioned earlier, we use [[AtomHopper]] internally (and YAGI w/PSHB previously), but it would be nice to use an off-the-shelf framework, ideally something Python-based. This too should be a separate repo and, ideally, optional.

In the [[DreamHost]] use case, The Dude should consume from the [[AtomHopper]] feed based on webhooks vs. having to poll the Ceilometer API for updates.

=== Alerts / Set Points / Thresholds ===

Personally, I don't think this belongs in the Middle Layer ... external components should define/monitor/control these.

== Unifying the Top Layer ==
Every client is different, trying to service the needs of each user with a common user interface might be tricky. [[StackTach]] started as a debugging tool and targets that audience. A Billing tool will need a very different UI.

There are lots of off-the-shelf products available for the presentation of instrumentation data, we shouldn't need to reinvent that wheel.

[[StackTach]] and Stacky should both consume from the Ceilometer API.

== Other Improvements ==

* Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. That said, I'm sure there are specific things that certain deployments will need that aren't covered by this and the existing Ceilometer collector system will still be needed.
* [[StackTach]] has taken major steps in optimizing its REST interface to facilitate caching. The data in the middle layer is largely read-only and there are far more readers than writers. The Ceilometer API should consider these improvements (if it's not dealt with already and I missed it)

EfficientVolumesFromImage

2013-02-16T21:57:43Z

Olaph:

__NOTOC__
* '''Launchpad Entry''': [[NovaSpec]]:efficient-volumes-from-images
* '''Created''': 03 April 2012
* '''Contributors''': Josh Durgin, Tommi Virtanen

== Summary ==

Make volume (and instance) creation fast if Nova volumes and Glance
use the same storage backend, and the storage backend supports it.

== Release Note ==

Instance and volume creation can take advantage of underlying storage
to be more efficient in space and time.

== Rationale ==

Storage systems offer a number of features that can save space and
time when storing both images and volumes:

* features like copy-on-write can be used to make volume (and instance creation) near-instantaneous and save space
* deduplication can happen on images and volumes together, reducing hardware requirements
* administrators only have to maintain one storage system

== User stories ==

User creates a bunch of new instances that have no ephemeral disks,
but are all based on the same image. This happens very quickly due to
a shared underlying storage system, and requires very little extra
space.

== Assumptions ==

Depends on
[https://blueprints.launchpad.net/nova/+spec/create-volume-from-image create-volume-from-image]
and a way to query Glance about the backing store of an image.

== Design ==
[[Image:cow_volume_from_image.png]]

== Implementation ==

There are two parts that are volume backend dependent:

# determining whether a given image is stored on the same backend as volumes
# efficiently creating a volume based on the image

These can both be optional volume driver methods - something like
`def cloneable(image_info)` and `def clone(image_info, volume_name)` - if they aren't implemented, or cloneable returns
false, fallback to the open/write/close api from
[https://blueprints.launchpad.net/nova/+spec/create-volume-from-image create-volume-from-image].

=== Migration ===

No database or core API changes.

== Test/Demo Plan ==

This need not be added or completed until the specification is nearing beta.

== BoF agenda and discussion ==

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

----
[[Category:Spec]]

FederatedAuthZwithZones

2013-02-16T21:56:31Z

Olaph:

__NOTOC__
* '''Launchpad Entry''': [[NovaSpec]]:nova-zones-federated-authz
* '''Created''': [[SandyWalsh]]
* '''Contributors''':
* '''Related''': [[ZonesOauth]] http://etherpad.openstack.org/nova-oauth

<<[[TableOfContents]]()>>

== Summary ==
In the current Zones implementation, all operations to child zones are done with an admin account. While this works, it causes problems in certain important use cases. Specifically when a user wishes to see all the instances that he may manage. With the admin-account approach, all instances in child zones are returned when only a subset should be (depending in the users rights).

This page is the culmination of discussions with eday, vish, jorge, khaled and others and attempts to summarize the problems that may arise in other use cases as well as documenting the proposals currently underway for solving them.

== Revision History ==
# First draft - AuthZ servers communicated and synced. [[MyCo]] users were actively added to Resource Groups.
# Realized that SP AuthZ doesn't need to know about users, just Resource Groups. Introduced concept of Behalf Of. Removed AuthZ sync.

== Release Note ==

== Rationale ==

== Assumptions ==

This will be the common scenario for the bulk of these use cases.

[[MyCo]] is a small software development shop. They have a Nova deployment made up of two Zones (parent and child). When these two zones are incapable of providing more instances, [[MyCo]] outsources to Service Provider (SP) to handle the extra work. SP is large and has many Zones geographically separated and nested many levels deep.

There are two main users in our use cases: Alice and Bob. Alice manages her own instances, as does Bob. However, Alice and Bob are also members of a Financial Task Force which gives them the ability to manage any instances under that groups control.

There will be one set of AuthN and AuthZ services per business. [[MyCo]] has it's own AuthN/Z service as does SP. These Auth services will need to be Highly Available.

Whenever a user is added/removed from a security group or permissions changed they are done on the [[MyCo]] side. Only the Resource Group ID's are passed to the SP.

After authenticating, the [[MyCo]] AuthZ will supply a set of data to SP regarding the User. This set of data includes:
* The Resource Group ID's that the user belongs to (and not the Resource ID's within that group, that data only lives in [[MyCo]] AuthZ)
* Optionally, a Resource Group ID that the user is acting on Behalf Of.
* A set of Permission Tuples ([Action Group, ...], [Resource Group, ...]) of the actions the User can perform on which Resource Groups.

For child zones within the same business (ie. all the child zones in Service Provider) group ID's and resource ID's will have to be unique across Zones.

In a multi-tenant scenario such as Service Provider, the Group identifiers passed from one AuthZ system to another will need to be prefixed/namespaced to avoid collisions.

[[Image:UseCaseOverview.gif]]

== Terminology ==

* [[MyCo]].authZ = The AuthZ service running at [[MyCo]]
* SP.authZ = The AuthZ service running at the Service Provider
* Subject = The user performing the action.
* Delegate = A user performing an action on behalf of another user.
* Subject Group = A group of users or delegates.
* Action = The verb being performed on the resource. Such as: boot, halt, pause, rescue, migrate, view, delete, etc.
* Action Group = A collection of Actions.
* Resource = The object the Subject is performing the Action on (aka 'Object' in our previous discussions)
* Resource Group = A group of Resources.

== Design ==

After authenticating Alice with the Service Provider Zones, [[MyCo]].authZ will supply SP.authZ with a set if ''Permission Tuples'' that indicate which operations Alice may perform and which Resources she may perform them on (where applicable). These tuples have the mapping (Action_Groups, Resource_Groups). Note that Subject is implied here since the User had to previously authenticate with the system to gain access.

Side note: There was a discussion of having the tuple be (Action_Group, User_Group) where User_Group was a collection of Users that owned resources. The downside with this approach is it's not fine-grained enough. For example, if Bob has permissions (can_halt, [Alice]) then he can halt ''all'' instances owned by Alice. That's too much privilege. Additionally, the difference between tracking a User_Group and a Resource_Group is nominal. Tracking a Resource_Group closer aligns with

We don't want the tuples to be so fine grained that we are listing individual instance ID's ... it won't scale to thousands of instances. While it would be easier, we don't want permission tuples of (Action_Group, [Resources ID's]) like (can_halt, [ami-AAA, ami-BBB, ami-CCC, ...]). Instead we need to define a Group of resources at the SP.authZ layer and simply refer to it in our permission tuples.

How? In order to do this, [[MyCo]].authZ needs to keep SP.authZ updated with a discrete set of Resource Groups.

[[MyCo]].authZ will have many Resource Groups and much of the contents of these Resource Groups will contain resources that live in [[MyCo]] Zones. We don't need to replicate all of this information to SP.authZ ... only a subset of the group that references resources that live in SP Zones. For example, Bob's permitted groups on the [[MyCo]].authZ side may look like:

<pre><nowiki>
(Bob, [can_halt, can_see, can_boot, ...], [myco-Foo-group, myco-Bar-group, myco-Zoo-group, sp-Alice-group, sp-Bob-group sp-Finance-group])
</nowiki></pre>

but when passing this on to SP.authZ only the following is required:

<pre><nowiki>
([can_halt, can_see, can_boot, ...], [sp-Alice-group, sp-Bob-group sp-Finance-group])
</nowiki></pre>

You can think of this as a formalization of the "override" capability in the authn/z branch proposed by vish & termie.

[[Image:SyncAuthZ.gif]]

=== On Behalf Of ===
One complexity we have is creating new Resources on the SP side. We want to make sure we assign the correct Resource Group to the Resource. There are a couple of possibilities:
# When the user authenticates we add a "On-Behalf-Of=<Resource Group ID>" header, which indicates how new resources should be assigned. This may be tricky to pre-determine.
# We always create a new Resource Group when creating new Resources and later, as needed, nest this Resource Group in other Resource Groups for granting permissions.

== User stories ==

== Implementation ==

=== Migration ===

== Test/Demo Plan ==

== Unresolved issues ==
http://paste.openstack.org/show/1108/

== BoF agenda and discussion ==

----
[[Category:Spec]]

Rebuildforvms

2013-02-16T21:51:38Z

Olaph:

__NOTOC__
* '''Launchpad Entry''': [[NovaSpec]]:rebuild-for-ha
* '''Created''': 2011-Nov-30
* '''Contributors''': Jason Kim

== Summary ==
Support HA for vms from the failed host.

== Release Note ==
When this is implemented, VMs will be rebuilt from failed host to the other hosts.

== Rationale ==
If the host is failed, we can't run the current vms on the host.
That is to say, we can't HA for vms.
I solved the HA using the rebuild means.

Below steps will be added.

1) get the information of instances on the failed host.
- we can know the state of host by using monitoring system or using the service state (nova-compute, nova-network)

2) get a list of dictionaries of network data of an instance.

3) update the volume db and instance db.

4) setup volumes for block device mapping

5) spawn the instance.

6) associate the floating ip which is used existing the instances.
- set up the iptables and so on.

7) update the instance db.

== User stories ==
* User doesn't know the vm is rebuilt.
* Operator gets the list of failed host from monitoring system or nova network/compute state.
* Operator gets the lists of the instances on the failed hosts.
* Operator should rebuild the instances by using this operation.

== Assumptions ==
The instances were created with volumes attached.

* [https://blueprints.launchpad.net/nova/+spec/auto-create-boot-volumes auto-create-boot-volumes]

== Design ==
[[Image:rebuild_for_vm_concept.png]]

== Implementation ==
* Scheduler should call the compute manager to rebuild the instance.
* Compute Manager should get a list of dictionaries of network data of an instance.
* Compute Manager should update the volume db and instance db.
* Compute Manager should setup volumes for block device mapping by using the volume manager.
* Compute Manager should spawn the instance by using the virt driver.
* Compute Manager should associate the floating ip by using the network manager.
* Compute Manager should update the instance db.
* Compute Manager should restart the network module.

=== Code Changes ===
* /usr/bin/nova-manage
* /nova/compute/api.py, vm_states.py, task_state.py, manager.py
* /nova/api/openstack/common.py
* /nova/network/api.py, manager.py

== Test/Demo Plan ==
This need not be added or completed until the specification is nearing beta.

== Unresolved issues ==
TBD

== BoF agenda and discussion ==
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

----
[[Category:Spec]]

UnifiedInstrumentationMetering

2013-02-16T21:50:27Z

Olaph:

__NOTOC__
= Unified Instrumentation and Metering =

Last edit: -- [[SandyWalsh]] <<[[DateTime]](2012-11-01T19:47:57Z)>> <<[[DateTime]](2012-11-01T19:47:57Z)>>

<<[[TableOfContents]]()>>

== Overview ==

With <code><nowiki>Ceilometer</nowiki></code>, <code><nowiki>Tach</nowiki></code> and <code><nowiki>[[StackTach]]</nowiki></code>, there are some very cool initiatives going on around instrumentation and metering/monitoring within [[OpenStack]] today.

However, due to the incredible speed of development within [[OpenStack]] many of these efforts have been performed in isolation from each other. Now, we are reaching a level of maturity which demands we stop reinventing the wheel and agree upon some shared infrastructure. This need is necessary for a variety of reasons:

# We want to make it easier for new projects to build on the existing [[OpenStack]] notification message bus.
# Less code is good. We shouldn't need three different workers for extracting notifications from [[OpenStack]].
# Notifications are large and there's a lot of them. We only want to process and store that data once.
# Archiving of data is a common problem, we shouldn't need several different ways of doing it.

In this document we'll talk about the State-of-Art with respect to Instrumentation/Metering/Monitoring (IMM) with [[OpenStack]] today and where we might go with it.

Required reading / viewing:
* http://www.openstack.org/summit/san-diego-2012/openstack-summit-sessions/presentation/what-about-billing-an-introduction-to-ceilometer
* http://www.sandywalsh.com/2012/10/debugging-openstack-with-stacktach-and.html
* https://etherpad.openstack.org/grizzly-common-instrumentation (good summary of instrumentation needs)

= Instrumentation vs Metering/Monitoring =

At the very base of this discussion is having a clear understanding of the difference between Instrumentation and Metering/Monitoring.

== Instrumentation ==
Think of instrumentation as the way they test electronics in-circuit. While the device is running, probes are attached to the circuit board and measurements are taken.

[[Image:instrumentation.png]]

There are some key things to consider in this analogy:
* Every technician may want to place their probes in different locations.
* Probes might be placed for long term measuring or transient ("I wonder if ... " scenarios)
* The circuit does not have to change for the testing probes to be placed. No other groups or departments had to be involved for this instrumentation to occur.
* The same probe technology can be used on other circuit boards. Likewise, our instrumentation probe-placement technology should not just be geared towards Nova. It should also work with all other parts of [[OpenStack]].
* When the circuit changes our probe placement may have to change. We have to be aware of that.
* The probes aren't perfect. They might slip off or have a spotty connection. We're looking for trends here, identifying when things are slow or flaky.
* With respect to Python, we may be interested in stack traces as well. Not just single function timings/counts.

== Metering / Monitoring ==
Metering is watching usage of the system, usually for the purposes of Billing. Monitoring is watching the system for critical system changes, performance and accuracy, usually for things like SLA's.

Think of your power meter. You can go outside and watch the dial spin and confirm your monthly bill jives with what the meter is reporting.

[[Image:metering.png]]

The important aspects of metering and monitoring:
* These events/measurements are critical. We cannot risk dropping an event.
* We need to ensure these events are consistent between releases. Their consistency should be considered of equal importance as [[OpenStack]] API consistency.
* We have no idea how people are going to want to use these events, but we can safely assume there will be a lots of other groups interested in them. We don't want these groups talking to the production [[OpenStack]] deployments directly.
* These events may not be nearly as frequent as the instrumentation messages, but they will be a lot larger since the entire context of the message needs to be included (''which'' instance, ''which'' image, ''which'' user, etc)

= The State-of-the-Art =

So, where are we?

== Instrumentation Today ==

There is no formal instrumentation solution for [[OpenStack]] today (other than logging). Logging is insufficient because it's:
* unstructured text
* too much effort to scan, parse and correlate across servers
* requires code changes to collect new information

Rackspace has being using a cocktail of [https://github.com/ohthree/tach Tach], [https://github.com/etsy/statsd Statsd], [http://graphite.wikidot.com/ Graphite] and [http://www.nagios.org/ Nagios] to great success for close to a year.

[[Image:instrumentation_arch.gif]]

Tach is a library that collects timing/count data from anywhere in a Python program. It's not specific to [[OpenStack]], but it has pre-canned config files for the main [[OpenStack]] services. Tach hooks into a programming using monkey-patching and has a concept of Metrics and Notifiers. The Metrics are user-extensible hooks that pull data from the code. The Notifiers take the collected data and send it somewhere. Currently there are Metrics drivers for execution time and counts as well as Notifiers for Statsd, Graphite directly, print and log files. ([[SandyWalsh]] has been working on a replacement for Tach, called [https://github.com/SandyWalsh/scrutinize Scrutinize] which adds cProfile support and easier configuration. It's almost ready for prime-time.)

Tach is launched as: <code><nowiki>tach tach.conf nova-compute nova.conf</nowiki></code> ... so it easily integrates with existing deployments.

The powerful features of statsd are:
* UDP based messaging, so production is not at risk if the collectors die.
* In-memory rollup/aggregate of measurements that get relayed to Graphite. This greatly enhances scalability.
* Written in node.js = fast, fast, fast.

== Monitoring/Metering Today ==

Certainly the face of Monitoring and Metering within [[OpenStack]] today is [https://launchpad.net/ceilometer Ceilometer] ... I'll just refer you to that project for more details.

But there are others. Within Rackspace we use [https://github.com/rackspace/yagi YAGI] to consume the notifications and send them to our internal billing system. Specifically, this data is sent to [http://atomhopper.org/ AtomHopper] where it is turned into an Atom feed for other consumers (one of which is billing). YAGI used to have [https://code.google.com/p/pubsubhubbub/ PubSubHubBub] support, but that's gone dormant due to other motivators. Now, [[AtomHopper]] is the redistribution system of choice. Sadly, [[AtomHopper]] is Java-based, so it may not work well within the [[OpenStack]] ecosystem, per se. The YAGI Worker uses [https://github.com/ask/carrot/ carrot] and has been highly reliable in all of our environments, but there has been discussion of moving to [http://pypi.python.org/pypi/kombu kombu].

[[StackTach]] is a debugging/monitoring tool based on [[OpenStack]] notifications and it too has its own Worker. It is kombu-based and is currently used in production. We've had lots of problems making the stacktach worker reliable but we think the problem has been with combining a threading model with eventlet. Our new scheme uses the multiprocessing library with per-rabbit workers. This is currently being stress tested, stay tuned. We've tried a variety of other schemes and library combinations with little success (more detail if needed). Note: this ''should'' work fine since it's the same scheme Nova uses internally.

[[StackTach]] is quickly moving into Metrics, SLA and Monitoring territory with version 2 and the inclusion of Stacky (the command line interface to [[StackTach]])

[[Image:notifications.gif]]

= The Layers =

With every architecture there are different layers of abstraction/functionality.

IMM has:
* ''low-level'' components that directly interface with [[OpenStack]] via monkeypatching or notification consumption.
* ''mid-level'' components that collect, aggregate and redistribute this collected data to other consumers
* ''high-level'' components that act as the presentation layer.

We currently have overlap on all levels.

[[Image:layers.gif]]

== Unifying the Low Level ==

There is really no way to unify the instrumentation collection with the metrics collection infrastructure within [[OpenStack]]. As stated in the introduction, these are very different animals.

With respect to Monitoring and Metering (notification consumption), the obvious candidate is the notification worker. Using a list of queues in the rabbit notifier is not a solution, it's a big load on the rabbit server.

''Note: we're also using AMQP incorrectly for notifications. Events are published to an Exchange and different queues can be created for different consumers. Currently, [[OpenStack]] creates a different Exchange for each notification destination. This should simply a bug that should be fixed.''

This is the low-hanging fruit. Creating a scalable worker that can work in a multi-rabbit (multi-cell, mult-region) deployment. It should support a pluggable scheme for handling the collected data and support failover/redundancy. The worker has to be fast/reliable enough to keep the notification queue empty since this is easily the fastest growing queue (neck and neck with the cells capacity update queue :)

The worker should be a separate repository so others can use it standalone.

Also, the Ceilometer worker doesn't need to use all of the nova-common code. This is very simple program and can be handled with a single file.

== Unifying the Mid-Level ==

[[Image:architecture.gif]]

=== A Common Database for Collected Data ===

For metering and monitoring, both [[StackTach]] and Ceilometer have a database to store the notifications. [[StackTach]] uses a SQL-based database, while Ceilometer uses a key-value database (Mongo). Each notification is large and contains about ten columns that are required to have fast lookup:
* Deployment ID
* Tenant ID
* UUID
* Request ID
* Timestamp
* Instance State
* Instance Task
* Publisher
* Host ID
* Event name

Additionally the entire JSON payload should be stored for the event for consumers that need something specific.

I have no recommendation on which db to use, but perhaps some load testing should be performed to see how well each works. The important thing is, the entire event gets stored.

However, as these events get collected, there are often additional tables that need to get updated which contain aggregated/summarized information from the events. While these operations could be done in a batch fashion in a separate database it's likely more efficient to provide a means for other applications to hook into the data collection and update these tables in real-time. This is what [[StackTach]] does and it greatly reduces the post-processing required. It does, however, come at a cost of additional processing when event storage is being performed (an expensive operation). Something like Ceilometer's secondary publishing step seems a good solution here, but I think it should be based on an [[AtomHopper]]/PSHB approach vs something proprietary. We should discuss what redistribution system looks like in greater detail.

In the diagram above Ext1, Ext2, Ext3 are user-plugins for doing special aggregation work. (not for sending to the redistribution system, another plugin mechanism could handle that)

For instrumentation, as mentioned above, the statsd in-memory database is a perfect solution here. Probably not much to improve on here.

=== A Common Event Redistribution System ===

As mentioned earlier, we use [[AtomHopper]] internally (and YAGI w/PSHB previously), but it would be nice to use an off-the-shelf framework, ideally something Python-based. This too should be a separate repo and, ideally, optional.

In the [[DreamHost]] use case, The Dude should consume from the [[AtomHopper]] feed based on webhooks vs. having to poll the Ceilometer API for updates.

=== Alerts / Set Points / Thresholds ===

Personally, I don't think this belongs in the Middle Layer ... external components should define/monitor/control these.

== Unifying the Top Layer ==
Every client is different, trying to service the needs of each user with a common user interface might be tricky. [[StackTach]] started as a debugging tool and targets that audience. A Billing tool will need a very different UI.

There are lots of off-the-shelf products available for the presentation of instrumentation data, we shouldn't need to reinvent that wheel.

[[StackTach]] and Stacky should both consume from the Ceilometer API.

== Other Improvements ==

* Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. That said, I'm sure there are specific things that certain deployments will need that aren't covered by this and the existing Ceilometer collector system will still be needed.
* [[StackTach]] has taken major steps in optimizing its REST interface to facilitate caching. The data in the middle layer is largely read-only and there are far more readers than writers. The Ceilometer API should consider these improvements (if it's not dealt with already and I missed it)

UnifiedInstrumentationMetering

2013-02-16T21:47:35Z

Olaph:

__NOTOC__
= Unified Instrumentation and Metering =

Last edit: -- [[SandyWalsh]] <<[[DateTime]](2012-11-01T19:47:57Z)>> <<[[DateTime]](2012-11-01T19:47:57Z)>>

<<[[TableOfContents]]()>>

== Overview ==

With <code><nowiki>Ceilometer</nowiki></code>, <code><nowiki>Tach</nowiki></code> and <code><nowiki>[[StackTach]]</nowiki></code>, there are some very cool initiatives going on around instrumentation and metering/monitoring within [[OpenStack]] today.

However, due to the incredible speed of development within [[OpenStack]] many of these efforts have been performed in isolation from each other. Now, we are reaching a level of maturity which demands we stop reinventing the wheel and agree upon some shared infrastructure. This need is necessary for a variety of reasons:

# We want to make it easier for new projects to build on the existing [[OpenStack]] notification message bus.
# Less code is good. We shouldn't need three different workers for extracting notifications from [[OpenStack]].
# Notifications are large and there's a lot of them. We only want to process and store that data once.
# Archiving of data is a common problem, we shouldn't need several different ways of doing it.

In this document we'll talk about the State-of-Art with respect to Instrumentation/Metering/Monitoring (IMM) with [[OpenStack]] today and where we might go with it.

Required reading / viewing:
* http://www.openstack.org/summit/san-diego-2012/openstack-summit-sessions/presentation/what-about-billing-an-introduction-to-ceilometer
* http://www.sandywalsh.com/2012/10/debugging-openstack-with-stacktach-and.html
* https://etherpad.openstack.org/grizzly-common-instrumentation (good summary of instrumentation needs)

= Instrumentation vs Metering/Monitoring =

At the very base of this discussion is having a clear understanding of the difference between Instrumentation and Metering/Monitoring.

== Instrumentation ==
Think of instrumentation as the way they test electronics in-circuit. While the device is running, probes are attached to the circuit board and measurements are taken.

[[Image:instrumentation.png]]

There are some key things to consider in this analogy:
* Every technician may want to place their probes in different locations.
* Probes might be placed for long term measuring or transient ("I wonder if ... " scenarios)
* The circuit does not have to change for the testing probes to be placed. No other groups or departments had to be involved for this instrumentation to occur.
* The same probe technology can be used on other circuit boards. Likewise, our instrumentation probe-placement technology should not just be geared towards Nova. It should also work with all other parts of [[OpenStack]].
* When the circuit changes our probe placement may have to change. We have to be aware of that.
* The probes aren't perfect. They might slip off or have a spotty connection. We're looking for trends here, identifying when things are slow or flaky.
* With respect to Python, we may be interested in stack traces as well. Not just single function timings/counts.

== Metering / Monitoring ==
Metering is watching usage of the system, usually for the purposes of Billing. Monitoring is watching the system for critical system changes, performance and accuracy, usually for things like SLA's.

Think of your power meter. You can go outside and watch the dial spin and confirm your monthly bill jives with what the meter is reporting.

[[Image:metering.png]]

The important aspects of metering and monitoring:
* These events/measurements are critical. We cannot risk dropping an event.
* We need to ensure these events are consistent between releases. Their consistency should be considered of equal importance as [[OpenStack]] API consistency.
* We have no idea how people are going to want to use these events, but we can safely assume there will be a lots of other groups interested in them. We don't want these groups talking to the production [[OpenStack]] deployments directly.
* These events may not be nearly as frequent as the instrumentation messages, but they will be a lot larger since the entire context of the message needs to be included (''which'' instance, ''which'' image, ''which'' user, etc)

= The State-of-the-Art =

So, where are we?

== Instrumentation Today ==

There is no formal instrumentation solution for [[OpenStack]] today (other than logging). Logging is insufficient because it's:
* unstructured text
* too much effort to scan, parse and correlate across servers
* requires code changes to collect new information

Rackspace has being using a cocktail of [https://github.com/ohthree/tach Tach], [https://github.com/etsy/statsd Statsd], [http://graphite.wikidot.com/ Graphite] and [http://www.nagios.org/ Nagios] to great success for close to a year.

[[Image:instrumentation_arch.gif]]

Tach is a library that collects timing/count data from anywhere in a Python program. It's not specific to [[OpenStack]], but it has pre-canned config files for the main [[OpenStack]] services. Tach hooks into a programming using monkey-patching and has a concept of Metrics and Notifiers. The Metrics are user-extensible hooks that pull data from the code. The Notifiers take the collected data and send it somewhere. Currently there are Metrics drivers for execution time and counts as well as Notifiers for Statsd, Graphite directly, print and log files. ([[SandyWalsh]] has been working on a replacement for Tach, called [https://github.com/SandyWalsh/scrutinize Scrutinize] which adds cProfile support and easier configuration. It's almost ready for prime-time.)

Tach is launched as: <code><nowiki>tach tach.conf nova-compute nova.conf</nowiki></code> ... so it easily integrates with existing deployments.

The powerful features of statsd are:
* UDP based messaging, so production is not at risk if the collectors die.
* In-memory rollup/aggregate of measurements that get relayed to Graphite. This greatly enhances scalability.
* Written in node.js = fast, fast, fast.

== Monitoring/Metering Today ==

Certainly the face of Monitoring and Metering within [[OpenStack]] today is [https://launchpad.net/ceilometer Ceilometer] ... I'll just refer you to that project for more details.

But there are others. Within Rackspace we use [https://github.com/rackspace/yagi YAGI] to consume the notifications and send them to our internal billing system. Specifically, this data is sent to [http://atomhopper.org/ AtomHopper] where it is turned into an Atom feed for other consumers (one of which is billing). YAGI used to have [https://code.google.com/p/pubsubhubbub/ PubSubHubBub] support, but that's gone dormant due to other motivators. Now, [[AtomHopper]] is the redistribution system of choice. Sadly, [[AtomHopper]] is Java-based, so it may not work well within the [[OpenStack]] ecosystem, per se. The YAGI Worker uses [https://github.com/ask/carrot/ carrot] and has been highly reliable in all of our environments, but there has been discussion of moving to [http://pypi.python.org/pypi/kombu kombu].

[[StackTach]] is a debugging/monitoring tool based on [[OpenStack]] notifications and it too has its own Worker. It is kombu-based and is currently used in production. We've had lots of problems making the stacktach worker reliable but we think the problem has been with combining a threading model with eventlet. Our new scheme uses the multiprocessing library with per-rabbit workers. This is currently being stress tested, stay tuned. We've tried a variety of other schemes and library combinations with little success (more detail if needed). Note: this ''should'' work fine since it's the same scheme Nova uses internally.

[[StackTach]] is quickly moving into Metrics, SLA and Monitoring territory with version 2 and the inclusion of Stacky (the command line interface to [[StackTach]])

[[Image:notifications.gif]]

= The Layers =

With every architecture there are different layers of abstraction/functionality.

IMM has:
* ''low-level'' components that directly interface with [[OpenStack]] via monkeypatching or notification consumption.
* ''mid-level'' components that collect, aggregate and redistribute this collected data to other consumers
* ''high-level'' components that act as the presentation layer.

We currently have overlap on all levels.

[[Image:layers.gif]]

== Unifying the Low Level ==

There is really no way to unify the instrumentation collection with the metrics collection infrastructure within [[OpenStack]]. As stated in the introduction, these are very different animals.

With respect to Monitoring and Metering (notification consumption), the obvious candidate is the notification worker. Using a list of queues in the rabbit notifier is not a solution, it's a big load on the rabbit server.

''Note: we're also using AMQP incorrectly for notifications. Events are published to an Exchange and different queues can be created for different consumers. Currently, [[OpenStack]] creates a different Exchange for each notification destination. This should simply a bug that should be fixed.''

This is the low-hanging fruit. Creating a scalable worker that can work in a multi-rabbit (multi-cell, mult-region) deployment. It should support a pluggable scheme for handling the collected data and support failover/redundancy. The worker has to be fast/reliable enough to keep the notification queue empty since this is easily the fastest growing queue (neck and neck with the cells capacity update queue :)

The worker should be a separate repository so others can use it standalone.

Also, the Ceilometer worker doesn't need to use all of the nova-common code. This is very simple program and can be handled with a single file.

== Unifying the Mid-Level ==

[[Image:UnifiedInstrumentationMetering$architecture.gif]]

=== A Common Database for Collected Data ===

For metering and monitoring, both [[StackTach]] and Ceilometer have a database to store the notifications. [[StackTach]] uses a SQL-based database, while Ceilometer uses a key-value database (Mongo). Each notification is large and contains about ten columns that are required to have fast lookup:
* Deployment ID
* Tenant ID
* UUID
* Request ID
* Timestamp
* Instance State
* Instance Task
* Publisher
* Host ID
* Event name

Additionally the entire JSON payload should be stored for the event for consumers that need something specific.

I have no recommendation on which db to use, but perhaps some load testing should be performed to see how well each works. The important thing is, the entire event gets stored.

However, as these events get collected, there are often additional tables that need to get updated which contain aggregated/summarized information from the events. While these operations could be done in a batch fashion in a separate database it's likely more efficient to provide a means for other applications to hook into the data collection and update these tables in real-time. This is what [[StackTach]] does and it greatly reduces the post-processing required. It does, however, come at a cost of additional processing when event storage is being performed (an expensive operation). Something like Ceilometer's secondary publishing step seems a good solution here, but I think it should be based on an [[AtomHopper]]/PSHB approach vs something proprietary. We should discuss what redistribution system looks like in greater detail.

In the diagram above Ext1, Ext2, Ext3 are user-plugins for doing special aggregation work. (not for sending to the redistribution system, another plugin mechanism could handle that)

For instrumentation, as mentioned above, the statsd in-memory database is a perfect solution here. Probably not much to improve on here.

=== A Common Event Redistribution System ===

As mentioned earlier, we use [[AtomHopper]] internally (and YAGI w/PSHB previously), but it would be nice to use an off-the-shelf framework, ideally something Python-based. This too should be a separate repo and, ideally, optional.

In the [[DreamHost]] use case, The Dude should consume from the [[AtomHopper]] feed based on webhooks vs. having to poll the Ceilometer API for updates.

=== Alerts / Set Points / Thresholds ===

Personally, I don't think this belongs in the Middle Layer ... external components should define/monitor/control these.

== Unifying the Top Layer ==
Every client is different, trying to service the needs of each user with a common user interface might be tricky. [[StackTach]] started as a debugging tool and targets that audience. A Billing tool will need a very different UI.

There are lots of off-the-shelf products available for the presentation of instrumentation data, we shouldn't need to reinvent that wheel.

[[StackTach]] and Stacky should both consume from the Ceilometer API.

== Other Improvements ==

* Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. That said, I'm sure there are specific things that certain deployments will need that aren't covered by this and the existing Ceilometer collector system will still be needed.
* [[StackTach]] has taken major steps in optimizing its REST interface to facilitate caching. The data in the middle layer is largely read-only and there are far more readers than writers. The Ceilometer API should consider these improvements (if it's not dealt with already and I missed it)

UnifiedInstrumentationMetering

2013-02-16T21:46:30Z

Olaph:

__NOTOC__
= Unified Instrumentation and Metering =

Last edit: -- [[SandyWalsh]] <<[[DateTime]](2012-11-01T19:47:57Z)>> <<[[DateTime]](2012-11-01T19:47:57Z)>>

<<[[TableOfContents]]()>>

== Overview ==

With <code><nowiki>Ceilometer</nowiki></code>, <code><nowiki>Tach</nowiki></code> and <code><nowiki>[[StackTach]]</nowiki></code>, there are some very cool initiatives going on around instrumentation and metering/monitoring within [[OpenStack]] today.

However, due to the incredible speed of development within [[OpenStack]] many of these efforts have been performed in isolation from each other. Now, we are reaching a level of maturity which demands we stop reinventing the wheel and agree upon some shared infrastructure. This need is necessary for a variety of reasons:

# We want to make it easier for new projects to build on the existing [[OpenStack]] notification message bus.
# Less code is good. We shouldn't need three different workers for extracting notifications from [[OpenStack]].
# Notifications are large and there's a lot of them. We only want to process and store that data once.
# Archiving of data is a common problem, we shouldn't need several different ways of doing it.

In this document we'll talk about the State-of-Art with respect to Instrumentation/Metering/Monitoring (IMM) with [[OpenStack]] today and where we might go with it.

Required reading / viewing:
* http://www.openstack.org/summit/san-diego-2012/openstack-summit-sessions/presentation/what-about-billing-an-introduction-to-ceilometer
* http://www.sandywalsh.com/2012/10/debugging-openstack-with-stacktach-and.html
* https://etherpad.openstack.org/grizzly-common-instrumentation (good summary of instrumentation needs)

= Instrumentation vs Metering/Monitoring =

At the very base of this discussion is having a clear understanding of the difference between Instrumentation and Metering/Monitoring.

== Instrumentation ==
Think of instrumentation as the way they test electronics in-circuit. While the device is running, probes are attached to the circuit board and measurements are taken.

[[Image:UnifiedInstrumentationMetering$instrumentation.png]]

There are some key things to consider in this analogy:
* Every technician may want to place their probes in different locations.
* Probes might be placed for long term measuring or transient ("I wonder if ... " scenarios)
* The circuit does not have to change for the testing probes to be placed. No other groups or departments had to be involved for this instrumentation to occur.
* The same probe technology can be used on other circuit boards. Likewise, our instrumentation probe-placement technology should not just be geared towards Nova. It should also work with all other parts of [[OpenStack]].
* When the circuit changes our probe placement may have to change. We have to be aware of that.
* The probes aren't perfect. They might slip off or have a spotty connection. We're looking for trends here, identifying when things are slow or flaky.
* With respect to Python, we may be interested in stack traces as well. Not just single function timings/counts.

== Metering / Monitoring ==
Metering is watching usage of the system, usually for the purposes of Billing. Monitoring is watching the system for critical system changes, performance and accuracy, usually for things like SLA's.

Think of your power meter. You can go outside and watch the dial spin and confirm your monthly bill jives with what the meter is reporting.

[[Image:metering.png]]

The important aspects of metering and monitoring:
* These events/measurements are critical. We cannot risk dropping an event.
* We need to ensure these events are consistent between releases. Their consistency should be considered of equal importance as [[OpenStack]] API consistency.
* We have no idea how people are going to want to use these events, but we can safely assume there will be a lots of other groups interested in them. We don't want these groups talking to the production [[OpenStack]] deployments directly.
* These events may not be nearly as frequent as the instrumentation messages, but they will be a lot larger since the entire context of the message needs to be included (''which'' instance, ''which'' image, ''which'' user, etc)

= The State-of-the-Art =

So, where are we?

== Instrumentation Today ==

There is no formal instrumentation solution for [[OpenStack]] today (other than logging). Logging is insufficient because it's:
* unstructured text
* too much effort to scan, parse and correlate across servers
* requires code changes to collect new information

Rackspace has being using a cocktail of [https://github.com/ohthree/tach Tach], [https://github.com/etsy/statsd Statsd], [http://graphite.wikidot.com/ Graphite] and [http://www.nagios.org/ Nagios] to great success for close to a year.

[[Image:instrumentation_arch.gif]]

Tach is a library that collects timing/count data from anywhere in a Python program. It's not specific to [[OpenStack]], but it has pre-canned config files for the main [[OpenStack]] services. Tach hooks into a programming using monkey-patching and has a concept of Metrics and Notifiers. The Metrics are user-extensible hooks that pull data from the code. The Notifiers take the collected data and send it somewhere. Currently there are Metrics drivers for execution time and counts as well as Notifiers for Statsd, Graphite directly, print and log files. ([[SandyWalsh]] has been working on a replacement for Tach, called [https://github.com/SandyWalsh/scrutinize Scrutinize] which adds cProfile support and easier configuration. It's almost ready for prime-time.)

Tach is launched as: <code><nowiki>tach tach.conf nova-compute nova.conf</nowiki></code> ... so it easily integrates with existing deployments.

The powerful features of statsd are:
* UDP based messaging, so production is not at risk if the collectors die.
* In-memory rollup/aggregate of measurements that get relayed to Graphite. This greatly enhances scalability.
* Written in node.js = fast, fast, fast.

== Monitoring/Metering Today ==

Certainly the face of Monitoring and Metering within [[OpenStack]] today is [https://launchpad.net/ceilometer Ceilometer] ... I'll just refer you to that project for more details.

But there are others. Within Rackspace we use [https://github.com/rackspace/yagi YAGI] to consume the notifications and send them to our internal billing system. Specifically, this data is sent to [http://atomhopper.org/ AtomHopper] where it is turned into an Atom feed for other consumers (one of which is billing). YAGI used to have [https://code.google.com/p/pubsubhubbub/ PubSubHubBub] support, but that's gone dormant due to other motivators. Now, [[AtomHopper]] is the redistribution system of choice. Sadly, [[AtomHopper]] is Java-based, so it may not work well within the [[OpenStack]] ecosystem, per se. The YAGI Worker uses [https://github.com/ask/carrot/ carrot] and has been highly reliable in all of our environments, but there has been discussion of moving to [http://pypi.python.org/pypi/kombu kombu].

[[StackTach]] is a debugging/monitoring tool based on [[OpenStack]] notifications and it too has its own Worker. It is kombu-based and is currently used in production. We've had lots of problems making the stacktach worker reliable but we think the problem has been with combining a threading model with eventlet. Our new scheme uses the multiprocessing library with per-rabbit workers. This is currently being stress tested, stay tuned. We've tried a variety of other schemes and library combinations with little success (more detail if needed). Note: this ''should'' work fine since it's the same scheme Nova uses internally.

[[StackTach]] is quickly moving into Metrics, SLA and Monitoring territory with version 2 and the inclusion of Stacky (the command line interface to [[StackTach]])

[[Image:notifications.gif]]

= The Layers =

With every architecture there are different layers of abstraction/functionality.

IMM has:
* ''low-level'' components that directly interface with [[OpenStack]] via monkeypatching or notification consumption.
* ''mid-level'' components that collect, aggregate and redistribute this collected data to other consumers
* ''high-level'' components that act as the presentation layer.

We currently have overlap on all levels.

[[Image:layers.gif]]

== Unifying the Low Level ==

There is really no way to unify the instrumentation collection with the metrics collection infrastructure within [[OpenStack]]. As stated in the introduction, these are very different animals.

With respect to Monitoring and Metering (notification consumption), the obvious candidate is the notification worker. Using a list of queues in the rabbit notifier is not a solution, it's a big load on the rabbit server.

''Note: we're also using AMQP incorrectly for notifications. Events are published to an Exchange and different queues can be created for different consumers. Currently, [[OpenStack]] creates a different Exchange for each notification destination. This should simply a bug that should be fixed.''

This is the low-hanging fruit. Creating a scalable worker that can work in a multi-rabbit (multi-cell, mult-region) deployment. It should support a pluggable scheme for handling the collected data and support failover/redundancy. The worker has to be fast/reliable enough to keep the notification queue empty since this is easily the fastest growing queue (neck and neck with the cells capacity update queue :)

The worker should be a separate repository so others can use it standalone.

Also, the Ceilometer worker doesn't need to use all of the nova-common code. This is very simple program and can be handled with a single file.

== Unifying the Mid-Level ==

[[Image:UnifiedInstrumentationMetering$architecture.gif]]

=== A Common Database for Collected Data ===

For metering and monitoring, both [[StackTach]] and Ceilometer have a database to store the notifications. [[StackTach]] uses a SQL-based database, while Ceilometer uses a key-value database (Mongo). Each notification is large and contains about ten columns that are required to have fast lookup:
* Deployment ID
* Tenant ID
* UUID
* Request ID
* Timestamp
* Instance State
* Instance Task
* Publisher
* Host ID
* Event name

Additionally the entire JSON payload should be stored for the event for consumers that need something specific.

I have no recommendation on which db to use, but perhaps some load testing should be performed to see how well each works. The important thing is, the entire event gets stored.

However, as these events get collected, there are often additional tables that need to get updated which contain aggregated/summarized information from the events. While these operations could be done in a batch fashion in a separate database it's likely more efficient to provide a means for other applications to hook into the data collection and update these tables in real-time. This is what [[StackTach]] does and it greatly reduces the post-processing required. It does, however, come at a cost of additional processing when event storage is being performed (an expensive operation). Something like Ceilometer's secondary publishing step seems a good solution here, but I think it should be based on an [[AtomHopper]]/PSHB approach vs something proprietary. We should discuss what redistribution system looks like in greater detail.

In the diagram above Ext1, Ext2, Ext3 are user-plugins for doing special aggregation work. (not for sending to the redistribution system, another plugin mechanism could handle that)

For instrumentation, as mentioned above, the statsd in-memory database is a perfect solution here. Probably not much to improve on here.

=== A Common Event Redistribution System ===

As mentioned earlier, we use [[AtomHopper]] internally (and YAGI w/PSHB previously), but it would be nice to use an off-the-shelf framework, ideally something Python-based. This too should be a separate repo and, ideally, optional.

In the [[DreamHost]] use case, The Dude should consume from the [[AtomHopper]] feed based on webhooks vs. having to poll the Ceilometer API for updates.

=== Alerts / Set Points / Thresholds ===

Personally, I don't think this belongs in the Middle Layer ... external components should define/monitor/control these.

== Unifying the Top Layer ==
Every client is different, trying to service the needs of each user with a common user interface might be tricky. [[StackTach]] started as a debugging tool and targets that audience. A Billing tool will need a very different UI.

There are lots of off-the-shelf products available for the presentation of instrumentation data, we shouldn't need to reinvent that wheel.

[[StackTach]] and Stacky should both consume from the Ceilometer API.

== Other Improvements ==

* Remove the Compute service that Ceilometer uses and integrate the existing fanout compute notifications into the data collected by the workers. There's no need for yet-another-worker. That said, I'm sure there are specific things that certain deployments will need that aren't covered by this and the existing Ceilometer collector system will still be needed.
* [[StackTach]] has taken major steps in optimizing its REST interface to facilitate caching. The data in the middle layer is largely read-only and there are far more readers than writers. The Ceilometer API should consider these improvements (if it's not dealt with already and I missed it)

Xenapi-sm-volume-driver

2013-02-16T21:42:22Z

Olaph:

__NOTOC__

* '''Launchpad Entry''': [[NovaSpec]]:xenapi-sm-support
* '''Created''': 1 February 2011
* '''Contributors''': Renuka Apte, Armando Migliaccio

== Summary ==
Design and implement a new driver for Nova-Volume, based on XenAPI Storage Manager. This will provide basic storage functionality (like volume creation, and destruction) on a number of different storage back-ends, but it will enable the capability of using more sophisticated storage back-ends for operations like cloning/snapshotting etc. To have an idea of the benefits of using XenAPI SM to provide back-end storage services, the list below shows some of the storage plugins already supported in [[XenServer]]/XCP:

* NFS VHD: SR plugin which stores disks as VHD files on a remote NFS filesystem
* Local VHD on LVM: SR plugin which represents disks as VHD disks on Logical Volumes within a locally-attached Volume Group
* HBA LUN-per-VDI driver: SR plugin which represents LUNs as VDIs sourced by hardware HBA adapters, e.g. hardware-based iSCSI or FC support
* [[NetApp]]: SR driver for mapping of LUNs to VDIs on a NETAPP server, providing use of fast snapshot and clone features on the filer
* LVHD over FC: SR plugin which represents disks as VHDs on Logical Volumes within a Volume Group created on an HBA LUN, e.g. hardware-based iSCSI or FC support
* iSCSI: Base ISCSI SR driver, provides a LUN-per-VDI. Does not support creation of VDIs but accesses existing LUNs on a target.
* LVHD over iSCSI: SR plugin which represents disks as Logical Volumes within a Volume Group created on an iSCSI LUN
* [[EqualLogic]]: SR driver for mapping of LUNs to VDIs on a EQUALLOGIC array group, providing use of fast snapshot and clone features on the array

== Glossary ==
* [[XenServer]]: Commercial, supported product from Citrix
* Xen Cloud Platform (XCP): Open-source equivalent of [[XenServer]] (and the development project for the toolstack). Everything said about [[XenServer]] below applies equally to XCP
* XenAPI: The management API exposed by [[XenServer]] and XCP
* xapi: The primary daemon on [[XenServer]] and Xen Cloud Platform; the one that exposes the XenAPI

== Release Note ==
No direct impact on user consuming the external HTTP API services. This blueprint enables the use of several storage back-ends by writing a single nova-volume driver, rather than several ones, as many as the specific storage back-ends required to be supported are needed. In due course, leveraging capabilities of specific back-ends, will enable operations like volume snapshotting, cloning, etc. that can be exposed via official API.

== Rationale ==
This new driver will sit alongside drivers already developed like AoE, iSCSI and SAN (both running on commodity boxes running Linux). However, the latter is slightly different as it will still run on commodity hardware, but it access remote controllers on SAN hardware by some protocol like SSH, CLI or custom API. This is clearly undesirable because every time there is a need to support new hardware, a new driver must be designed and developed. The adoption of XenAPI SM will enable a number of storage back-end that are abstracted by a common API. Moreover, this becomes premium value to customers of service providers and may become differentiator for enterprise private clouds.

== User stories ==
The system administrator of a medium sized company deploys a private cloud based on [[OpenStack]]. He had previously bought and deployed a [[NetApp]] filer to provide back-end storage to employees' virtual machines. He now uses XenAPISM volume driver to provide block storage to VMs created using the [[OpenStack]] private cloud.

A service provider wants to deliver premium storage to its customers. Storage back-ends like [[NetApp]] or [[EqualLogic]] can be easily supported using XenAPISM volume driver.

== Assumptions ==
For the design and implementation of the new Nova-Volume driver, XenAPI for [[XenServer]]/XCP is going to be used.

== Design ==
=== Definitions ===
* Backend: A term for a particular storage backend. This could be iSCSI, NFS, Netapp etc.
* Backend-config: All the parameters required to connect to a specific backend. For e.g. For NFS, this would be the server, path, etc.
* Flavor: A user friendly term to specify some notion of quality of service. For example, "gold" might mean that the volumes will use a backend where backups are possible.

The concept of flavor has been introduced in keeping with the LUNR work. No API changes have been introduced to allow for creating a volume of a particular flavor. This will be deferred until the LUNR code is available.

A flavor can be associated with multiple backends. Some kind of policy will decide which backend will be used to create a volume of a particular flavor. Until we have a way of selecting flavor, we will use a simple "first-fit" policy, where the first backend that can successfully create this volume is the one that is used.

The entity relationship diagram is given below:

[[Image:erd-xenapi-sm.png]]

=== Operation ===
Using the nova-manage command detailed in the implementation, an admin can add flavors and backends.

One or more nova-volume service instances will be deployed per availability zone. When an instance is started, it will create storage repositories (SRs) to connect to the backends available within that zone. All nova-volume instances within a zone can see all the available backends. These instances are completely symmetric and hence should be able to service any create_volume request within the zone.

== Implementation ==
=== Commands ===
A category called “sm” has been added to nova-manage in the class '''[[StorageManagerCommands]]'''

The following actions will be added:

# flavor_list
# flavor_create
# flavor_delete
# backend_list
# backend_add
# backend_remove

Usage:

nova-manage sm flavor_create <label> <description>

nova-manage sm flavor_delete<label>

nova-manage sm backend_add <flavor label> <SR type> [config connection parameters]

Note: SR type and config connection parameters are in keeping with the Xen Command Line Interface. http://support.citrix.com/article/CTX124887

nova-manage sm backend_delete <backend-id>

Examples:

nova-manage sm flavor_create gold "Not all that glitters"

nova-manage sm flavor_delete gold

nova-manage sm backend_add gold nfs name_label=toybox-renuka server=myserver serverpath=/local/scratch/myname

nova-manage sm backend_remove 1

=== API Changes ===
As mentioned above, the API changes required to create/delete a volume''' '''''of a particular flavor ''will be deferred till the LUNR code is available. It is therefore not possible to create a volume for a particular flavor yet, although the code/design exists.

The existing euca-create-volume and euca-delete-volume commands should be used.

=== New Tables ===
SM Flavor:
{| border="1" cellpadding="2" cellspacing="0"
|< >|Field
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Type
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Null
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Key
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Default
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|created_at
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|updated_at
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted_at
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|tinyint(1)
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|id
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|int(11)
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NO
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|PRI
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|label
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(255)
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="72px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|description
|<width="83px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(255)
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="63px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="67px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|}

Backend-config:
{| border="1" cellpadding="2" cellspacing="0"
|< >|Field
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Type
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Null
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Key
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Default
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|created_at
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|updated_at
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted_at
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|tinyint(1)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|id
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|int(11)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NO
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|PRI
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|flavor_id
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|int(11)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NO
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|MUL
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|sr_uuid
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(255)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|sr_type
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(255)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="89px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|config_params
|<width="91px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(2047)
|<width="58px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="60px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="64px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|}

SM Volume:
{| border="1" cellpadding="2" cellspacing="0"
|< >|Field
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Type
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Null
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Key
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|Default
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|created_at
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|updated_at
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted_at
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|datetime
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|deleted
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|tinyint(1)
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|id
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|int(11)
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NO
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|PRI
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|backend_id
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|int(11)
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NO
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|MUL
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|-
|<width="79px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|vdi_uuid
|<width="102px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|varchar(255)
|<width="42px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|YES
|<width="69px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|
|<width="70px" style="border-width:1px medium 1px 1px;border-style:solid none solid solid;border-color:rgb(0, 0, 0) -moz-use-text-color rgb(0, 0, 0) rgb(0, 0, 0);padding:0in 0in 0in 0.08in; ">|NULL
|}

=== Code Changes ===
A new volume driver called XenSMDriver has been added to nova/volume. It leverages the xenapi storage manager to control the available backends. Functions have been exposed to create/forget storage repositories and connect to them.

A do_setup function is added to volume driver in order to perform initialization (which includes creating backend repositories) when the service is started.

== Test/Demo Plan ==
Unit tests will be provided as code is developed.

== Unresolved issues ==
None at the time of writing

== BoF agenda and discussion ==
Related topics:

* Having VM's root devices running on block storage
* Thin provisioning
* Fast cloning
* Advanced volume API (Volume as a Service)
* Fibre channel and [[StorageLink]]
* Dedup

----
[[Category:Spec]]

XenServer/NetworkingFlags

2013-02-16T21:41:36Z

Olaph:

__NOTOC__
<<[[TableOfContents]]()>>

= [[XenServer]] Networking Configuration =

Before you look at configuring [[OpenStack]] networking with [[XenServer]], you may find it useful to read up on [[XenServer]] networking:
* http://support.citrix.com/article/CTX117915

Keeping in mind this diagram:

{{http://wiki.openstack.org/XenServer/XenXCPAndXenServer?action=[[AttachFile]]&do=get&target=[[XenServer]]-dom0-domU.png}}

== Key Points ==

[[XenServer]] config:
* We are assuming the [[XenServer]] has three physical interfaces: eth0, eth1, eth2
* This means Dom0 has the following bridges: xenbr0, xenbr1, xenbr2
* The Dom0 also has the host local xenapi network, usually the [[XenServer]] has the address 169.254.0.1

DomU config:
* The DomU is a PV virtual machine (has a kernel with the para-virtualization extensions)
* It generally has four interfaces
** eth0 -> connected to xenapi (xapi trafic)
** eth1 -> xenbr2 Tenant network traffic
** eth2 -> xenbr0 Management traffic (MySQL, RabbitMQ, Glance, etc)
** eth3 -> xenbr1 Public traffic (floating ip, api endpoints)

== Flags you probably want to know about ==

Each have the [[DevStack]] setting and the nova.conf entry

=== Public Interface ===

The interface on ""DomU"" that connects to the public network. Used by nova-network so it sends the floating ip traffic on the correct network.

<pre><nowiki>
PUBLIC_INTERFACE=eth3 # DevStack
public_interface=eth3 # nova.conf
</nowiki></pre>

=== VLAN Interface ===

When using VLAN networking you need to set the following flag:

<pre><nowiki>
vlan_interface=eth2
</nowiki></pre>

This is the [[XenServer]] interface on which a bridge on the correct VLAN will be created, and then the VM will be attached to that bridge. So if the flag is eth2, and your guest network is on VLAN42, a new network bridge may be created on your [[XenServer]] host (unless there is an existing one) and it will be used. or an ex

(Possible bug: it would seem this flag is also used for the DomU interface used to attach the DHCP servers and routers to listen on the appropriate VLAN networks. There used to be a separate flag for that, as required for [[XenServer]])

=== Flat Interface ===

Only needed if you are using FlatDHCP (TODO - or Flat?).

This is the interface on DomU that you want the tenant/VM network traffic. This is the interface to which the DHCP and NAT will be attached by nova.

<pre><nowiki>
flat_interface=eth2
</nowiki></pre>

Note: this interface should not be configured, nova will attach all the required bridges.

=== Flat Network Bridge ===

Only needed if you are using Flat or FlatDHCP.

This is the [[XenServer]] bridge on which the VM instances will have their VIFs attached. This should be the same has the bridge your DomU's Guest Interface is attached to.

<pre><nowiki>
FLAT_NETWORK_BRIDGE=xenbr2
flat_network_bridge=xenbr2
</nowiki></pre>

The above setting can also take the name label instead of the [[XenServer]] name. This is useful when you find on different hosts [[XenServer]] gives your networks different names (such as xapi1 and xapi3). So you can alternatively set it to something like:

<pre><nowiki>
flat_network_bridge=my_vm_network_name_label
</nowiki></pre>

Note: this flag only affects the network bridge written into the database when you add a network using nova-manage, it does not dynamically affect the bridge used when attaching VIFs to your VMs. You may find the need to remove your existing networks and creating a new network to make your new value of the setting apply.

== Networking Modes ==

=== Flat Networking ===

Most details are covered in the manual:
* http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-flat-networking.html

This requires the Network address to be injected into the VM image. This is currently quite error prone (needs appropriate guest agent software, or injects files into an ubuntu file system)

=== FlatDHCP Networking ===

This used DHCP to hand out IP address to the guest VMs.

Most details are covered in the manual:
* http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-flat-dhcp-networking.html

It should look a bit like this, when you have network HA turned on:

[[Image:flatdhcp.png]]

Please note:
* VM DHCP requests go: VM->xenbr2->nova-network->xenbr2->VM
* VM to VM traffic goes VM->eth2->switch->eth2->VM
* floating IP traffic incoming traffic does something like: public->eth1->xenbr1->nova-network->xenbr2->vm

=== VLAN Networking ===

Most details are covered in the manual:
* http://docs.openstack.org/trunk/openstack-compute/admin/content/configuring-vlan-networking.html

There are extra flags for the compute network driver, (so the VLAN network bridges are correctly created on the [[XenServer]]):

<pre><nowiki>
network_driver=nova.virt.xenapi.vif.XenAPIBridgeDriver
or
network_driver=nova.virt.xenapi.vif.XenAPIOpenVswitchDriver
</nowiki></pre>

It could look at bit like this, when you have Network HA turned on (Note: the bridges with dotted lines are automatically created by nova):

[[Image:vlan.png]]

Please note:
* Very similar flow to FlatDHCP, please try to understand that one first
* The nova-network node creates its own bridges in the domU (not shown above). It uses eth1 on the DomU as a trunk port. It ensures appropriate DHCP server instance is correctly configured and only listening on the appropriate VLAN network.

So when you create a VM, this is roughly what happens:
* A PIF identified either by (A) the vlan_interface flag or (B) the bridge_interface column in the networks db table will be used for creating a [[XenServer]] VLAN network
* VIF for VM instances within this network will be plugged in this VLAN network
* The 'Openstack domU', i.e. where nova-network is running, acts as a gateway for multiple VLAN networks, so it has to be attached on a VLAN trunk. For this reason it must have an interface on the parent bridge of the VLAN bridge where VM instances are plugged

Some more pointers are available here:
* [[XenServer/VLANManager]]

=== Network HA ===

This works the same on [[XenServer]] as any other hypervisor: nova-network must be run on every compute host, and the following flag must be set:

<pre><nowiki>
multi_host=True
</nowiki></pre>

It is know to work well with FlatDHCP, but should work with the over modes too (TODO - get confirmation)

= Example Configurations =

== Single NIC with no VLANs ==

Run everything through eth0, make your domU have a single interface too.

<pre><nowiki>
public_interface=eth0
flat_interface=eth0
flat_network_bridge=xenbr0
# update xenapi connection URL to have IP that is available on eth0
</nowiki></pre>

TODO - add a diagram to describe this

TODO - add the [[DevStack]] config for this

== Single NIC with VLANs ==

[[DevStack]] keeps public, managmenet and tenant networks separate by adding extra VLAN networks on [[XenServer]].

TODO - add some more of these

= Example [[DevStack]] localrc configuration =

[[DevStack]] will help create the DomU VM that runs the [[OpenStack]] code.
It will help you and create extra networks on your [[XenServer]] if required.

== Default Configuration: Single NIC on [[XenServer]] with VLANs ==

First the [https://github.com/citrix-openstack/devstack/blob/master/tools/xen/README.md DevStack readme] suggests the default configuration for your localrc file:

<pre><nowiki>
MYSQL_PASSWORD=my_super_secret
SERVICE_TOKEN=my_super_secret
ADMIN_PASSWORD=my_super_secret
SERVICE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=my_super_secret
# This is the password for your guest (for both stack and root users)
GUEST_PASSWORD=my_super_secret

# IMPORTANT: The following must be set to your dom0 root password!
XENAPI_PASSWORD=my_super_secret

# Do not download the usual images
IMAGE_URLS=""
# Explicitly set multi-host
MULTI_HOST=1
# Give extra time for boot
ACTIVE_TIMEOUT=45
# Interface on which you would like to access services
HOST_IP_IFACE=eth0
# First time Ubuntu network install params
NETINSTALLIP="dhcp"
NAMESERVERS=""
NETMASK=""
GATEWAY=""
</nowiki></pre>

Here are the current networking defaults (defined in the [https://github.com/citrix-openstack/devstack/blob/master/tools/xen/xenrc xenrc file]).

<pre><nowiki>
# This is eth0 on the DomU VM is connected to xenapi local host management network and does DHCP

# public interface running on home router, with static IP
# This is eth3 on the DomU VM
PUB_IP=192.168.1.55
PUB_BR=xenbr0
PUB_DEV=eth0
PUB_VLAN=-1
PUB_NETMASK=255.255.255.0

# VM network on VLAN 100
# This is eth1 on the DomU VM
VM_IP=$10.255.255.255 # A host-only ip that let's the interface come up, otherwise unused
VM_NETMASK=255.255.255.0
VM_BR=""
VM_VLAN=100
VM_DEV=eth0

# Management traffic on VLAN 101
# this is eth2 on the DomU VM
MGT_IP=172.16.100.55
MGT_NETMASK=255.255.255.0
MGT_BR=""
MGT_VLAN=101
MGT_DEV=eth0
</nowiki></pre>

This means [[DevStack]] will, if not already there, create two extra networks:
* eth0 and VLAN 100 (vmbr)
* eth0 and VLAN 101 (mgmtbr)

== Using two NICs, with VLANs, using PXE ==

You can [[XenServer/Install/PXE|install XenServer using PXE]]. This will tend to leave you with your management network on eth0 of your [[XenServer]], with DHCP. Then you can have the public network on eth1, and run the VM traffic on a VLAN on eth1.

To achieve this, you probably want something like this:

<pre><nowiki>
# MGMT network params
MGT_IP="dhcp" # use dhcp, as the PXE server is on this network
MGT_NETMASK=255.255.255.0
MGT_BR=xenbr0
MGT_VLAN=-1
MGT_DEV=eth0

# Public network
PUB_IP=172.24.4.10 # static ip in same subnet as your floating ip ranage
PUB_NETMASK=255.255.255.0
PUB_BR=xenbr1
PUB_VLAN=-1
PUB_DEV=eth1

# VM network params
VM_IP=10.255.255.255 # A host-only ip that let's the interface come up, otherwise unused
VM_NETMASK=255.255.255.0
VM_BR=""
VM_VLAN=100
VM_DEV=eth1

</nowiki></pre>

XenServer/XenAndXenServer

2013-02-16T21:39:38Z

Olaph:

__NOTOC__

<<[[TableOfContents]]()>>

= Getting Started =

To get started, take a look at: [[XenServer/GettingStarted|Getting started with XenServer and [[OpenStack]]]]

= Official Docs =

You can find this information in the official docs:
http://docs.openstack.org/trunk/openstack-compute/admin/content/introduction-to-xen.html

= Introduction to using Xen, XCP and [[XenServer]] with [[OpenStack]] =

In this section we will describe Xen, XCP, and [[XenServer]], the
differences between them, and how to use them with [[OpenStack]]. Xen's
architecture is different from KVM's in important ways, and we discuss those
differences and when each might make sense in your [[OpenStack]] cloud.

== Basic terminology ==

=== Xen ===
Xen is a hypervisor. It provides the
fundamental isolation between virtual machines. Xen is open source (GPLv2)
and is managed by Xen.org, an cross-industry organization.
Xen is a component of many different products and projects. The
hypervisor itself is very similar across all these projects, but the way that
it is managed can be different, which can cause confusion if you're not clear
which toolstack you are using. Make sure you know what toolstack you want
before you get started.

=== XCP ===
XCP is an open source (GPLv2) toolstack
for Xen. It is designed specifically as platform for enterprise and cloud
computing, and is well integrated with [[OpenStack]].

=== Citrix [[XenServer]] ===

Citrix [[XenServer]] is a commercial
product. It is based on XCP, and exposes the same toolstack and managment
API. As an analogy, think of [[XenServer]] being based on XCP in the way that Red
Hat Enterprise Linux is based on Fedora. [[XenServer]] has a free version (which
is very similar to XCP) and paid-for versions with additional features
enabled.

=== xapi / XAPI / XenAPI ===

Both [[XenServer]] and XCP include Xen, Linux, and the primary control
daemon known as xapi.

The API shared between XCP and [[XenServer]] is called ""XenAPI"". [[OpenStack]] usually refers to XenAPI, to
indicate that the integration works equally well on XCP and [[XenServer]].
Sometimes, a careless person will refer to [[XenServer]] specifically, but you can
be reasonably confident that anything that works on [[XenServer]] will also work
on the latest version of XCP.

== Privileged and unprivileged domains ==

A Xen host will run a number of virtual machines, VMs, or domains (the
terms are synonymous on Xen). One of these is in charge of running the rest
of the system, and is known as "domain 0", or "dom0". It is the first domain
to boot after Xen, and owns the storage and networking hardware, the device
drivers, and the primary control software. Any other VM is unprivileged, and
are known as a "domU" or "guest". All customer VMs are unprivileged of
course, but you should note that on Xen the [[OpenStack]] control software
(nova-compute) also runs in a domU. This gives a level of security isolation
between the privileged system software and the [[OpenStack]] sofware (much of
which is customer-facing). This architecture is described in more detail
later.

There is an ongoing project to split domain 0 into multiple privileged
domains known as ""driver domains"" and <emphasis
role="bold">stub domains"". This would give even better separation
between critical components. This is an ongoing research project and you
shouldn't worry about it right now. The current architecture just has three
levels of separation: dom0, the [[OpenStack]] domU, and the completely
unprivileged customer VMs.

== Paravirtualized versus hardware virtualized domains ==

A Xen virtual machine can be ""paravirtualized
(PV)"" or ""hardware virtualized
(HVM)"". This refers to the interaction between Xen, domain 0, and
the guest VM's kernel. PV guests are aware of the fact that they are
virtualized and will co-operate with Xen and domain 0; this gives them better
performance characteristics. HVM guests are not aware of their environment,
and the hardware has to pretend that they are running on an unvirtualized
machine. HVM guests have the advantage that there is no need to modify the
guest operating system, which is essential when running Windows.

In [[OpenStack]], customer VMs may run in either PV or HVM mode. However,
the [[OpenStack]] domU (that's the one running nova-compute) <emphasis
role="bold">must"" be running in PV mode.

= Deployment architecture =

When you deploy [[OpenStack]] on XCP or [[XenServer]] you will get something
similar to this:

[[Image:XenServer-dom0-domU.png]]

Key things to note:

* The hypervisor: Xen
* Domain 0: runs xapi and some small pieces from [[OpenStack]] (some xapi plugins and network isolation rules). The majority of this is provided by [[XenServer]] or XCP (or yourself using Kronos).
* [[OpenStack]] domU: The nova-compute code runs in a paravirtualized virtual machine, running on the host under management. Each host runs a local instance of nova-compute. It will often also be running nova-network (depending on your network mode). In this case, nova-network is managing the addresses given to the tenant VMs through DHCP.
* Nova uses the XenAPI Python library to talk to xapi, and it uses the Host Internal Management Network to reach from the domU to dom0 without leaving the host.

Some notes on the networking:
* The above diagram assumes FlatDHCP networking (the [[DevStack]] default).
* There are three main [[OpenStack]] networks: Management traffic (RabbitMQ, MySQL, etc), Tenant network traffic (controlled by nova-network) and Public traffic (floating IPs, public API end points).
* Each network that leaves the host has been put through a separate physical network interface. This is the simplest model, but it's not the only one possible. You may choose to isolate this traffic using VLANs instead, for example.
* nova-compute generally talks to [[XenServer]] using a host local private network called either "Guest Installer Network" or "Host Internal Managmenet Network".

Another example of how to deploy [[OpenStack]] on a system with only two physical network interfaces is:

[[Image:DevStackDiagram.png]]

== [[XenServer]] pools ==

Before [[OpenStack]] 2012.1 ("Essex"), all [[XenServer]] machines used with
[[OpenStack]] are standalone machines, usually only using local storage.

However in 2012.1 and later, the host-aggregates feature allows you to
create pools of [[XenServer]] hosts (configuring shared storage is still an out of
band activity). This move will enable live migration when using shared
storage.

== Xen and libvirt ==

It may possible to manage Xen using libvirt. This would be necessary
for any Xen-based system that isn't using the XCP toolstack, such as SUSE
Linux or Oracle Linux. Unfortunately, this is not well tested or supported
today, and using the XCP or [[XenServer]] toolstacks with the [[OpenStack]] XenAPI
backend is the only recommended method.

= Further reading =

What is Xen? by Xen.org: http://xen.org/files/Marketing/WhatisXen.pdf.

Xen Hypervisor project: http://xen.org/products/xenhyp.html.

XCP project: http://xen.org/products/cloudxen.html.

XenServerMigrations

2013-02-16T21:37:25Z

Olaph:

__NOTOC__
* '''Launchpad Entry'''
* '''Created''':
* '''Contributors''':

== Summary ==

== Release Note ==

We need the ability to move [[XenServer]] instances from one host in a zone to another host within the same zone. This provides a substantial amount of useful functionality, such as the ability to replace a dying or obsolete host with newer hardware by evacuating instances from it. Migrations are also the base dependency for functional resizes.

== Rationale ==

As above, the ability to evacuate a host for any physical reason is highly desirable. Furthermore, the ability for a customer to resize an instance (increasing the RAM and disk allotment) without snapshotting and creating a new instance is useful.

== User stories ==

# As operations, I want to be able to evacuate a host with failing hardware so that I can replace the box with minimal impact to customers
# As a user, I want to be able to migrate my instance so that I can move to faster and/or newer hardware

== Assumptions ==

The ability to snapshot a running [[XenServer]] instance already exists

== Design & Implementation ==

The abstracted websequence diagram is as follows:

[[Image:index.php.png]]

A few issues arise in this design. First, because of the current scheduler implementation, it isn't safe to assume that the scheduler will be aware of the ability to migrate. The "simple" scheduler implementation shows that this is possible, but I don't think we should depend on that ability, at least for the time being. The concession, as above, is to cast the message to the destination first, which simply identifies itself and proxies the message right over to the source.

We can provide for the notion of a smarter scheduler by making the initial migration call context sensitive. If we are the source and a destination argument is also present, we simply begin the Rsync. Otherwise, cast to the source, appending ourself to the message arguments. It feels a little bit inconsistent, but it would be more efficient to have a migration-aware scheduler.

First pass will be to implement it as the sequence diagram above indicates, with a second pass to add the above functionality if deemed necessary.

=== Migration States ===

* No state / No Migration Instance: The destination receives the cast from the API and creates a migration instance with the requisite fields. In this stage, the destination creates a migration instance populated with the current instance attributes and new ones, setting the status to '''pre-migrating''' and then casts a message to the source.
* '''pre-migrating:''' The source has been notified of the intent to migrate the VM, which then snaps the VM and changes the state to '''migrating'''
* '''migrating:''' The source is Rsync'ing the VHD to the destination host machine. Afterwards, the migration status is set to 'migrating_step2'. The source then shuts the instance down, and then starts Rsync'ing the COW to the destination host machine. Afterwards, the source sets the migration status to '''post-migrating''' and casts a message to the destination compute
* '''post-migrating:''' The destination creates a new instance from the Instance table and transferred VHD/COW, updates the instance record with the new hostname, and then sets the migration_status to '''verify-migration'''
* '''verify-migration:''' The migration will exist in this state until a subsequent API call is made. At that point, the source compute will vm-destroy the old instance and mark the migration as status '''finished'''
* '''finished:''' This is the state the migration is in whether it's been "confirmed" or "reverted"
* '''reverted:''' The destination compute will destroy the new instance, and then cast a message to the source to power back on the old instance. The old instance attributes will be restored, and then the migration status will be set to '''finished'''

=== Disk Transfer ===

There are few different ways we could go about transferring the disk to the destination host

# Create a XAPI plugin that does the rsync'ing for us
# SSH into dom0. I'd really prefer to avoid this route, but it may be the best option depending on how long 1 would take

=== Optimization ===
* We could suspend the instance instead of shutting it down, and then migrate the RAM VHD in the process so users don't lose their uptime

=== Code Changes ===

The Openstack API will modify the "action" endpoint to expose "resize" functionality, which is simply a migration with a larger RAM and disk quota.

=== Migration ===

Functionality already exists within the Openstack API, but returns HTTP 501 at this time. Afterwards, existing API clients should be able to successfully migrate through the "resize" functionality present in the API. Additionally, functionality will be exposed through the Admin API for migration without resizing the instance.

== Test/Demo Plan ==

This need not be added or completed until the specification is nearing beta.

----
[[Category:Spec]]

MediaWiki:Common.css

2013-02-05T19:12:12Z

Olaph:

/* CSS placed here will be applied to all skins */

@font-face {
font-family: 'PT Sans';
font-style: normal;
font-weight: 400;
src: local('PT Sans'), local('PTSans-Regular'), url(https://themes.googleusercontent.com/static/fonts/ptsans/v4/LKf8nhXsWg5ybwEGXk8UBQ.woff) format('woff');
}

body.page-Main_Page h1.firstHeading { display:none; }

#p-logo { background-image: url("/w/images/4/4c/OpenStack.png"); }

#p-logo a { display:none; }

body {
background-color: #FFFFFF;
font-family: 'PT Sans';
background-image: display:none;
}

MediaWiki:Common.css

2013-02-05T18:47:19Z

Olaph:

MediaWiki:Common.css

2013-02-05T18:36:38Z

Olaph:

MediaWiki:Common.css

2013-02-05T18:35:52Z

Olaph:

MediaWiki:Common.css

2013-02-05T18:27:43Z

Olaph:

MediaWiki:Common.css

2013-02-05T18:14:37Z

Olaph:

/* CSS placed here will be applied to all skins */

body.page-Main_Page h1.firstHeading { display:none; }

#p-logo { background-image: url("/w/images/4/4c/OpenStack.png"); }

#p-logo a { display:none; }

body { background-color: #FFFFFF }

MediaWiki:Common.css

2013-02-05T17:07:04Z

Olaph:

MediaWiki:Common.css

2013-02-05T16:54:37Z

Olaph:

/* CSS placed here will be applied to all skins */

body.page-Main_Page h1.firstHeading { display:none; }

#p-logo { background-image: url("/w/images/4/4c/OpenStack.png") !important; }

MediaWiki:Common.css

2013-02-05T16:47:31Z

Olaph:

/* CSS placed here will be applied to all skins */

body.page-Main_Page h1.firstHeading { display:none; }

#p-logo { background-image: url("/w/images/4/4c/OpenStack.png"); }

MediaWiki:Common.css

2013-02-05T16:30:41Z

Olaph:

/* CSS placed here will be applied to all skins */

body.page-Main_Page h1.firstHeading { display:none; }

#p-logo { background-image: url("w/skins/common/images/openstack.png"); }

MediaWiki:Common.css

2013-02-05T16:26:25Z

Olaph: Created page with "/* CSS placed here will be applied to all skins */ body.page-Main_Page h1.firstHeading { display:none; }"

/* CSS placed here will be applied to all skins */

body.page-Main_Page h1.firstHeading { display:none; }