Jump to: navigation, search


Revision as of 21:49, 2 March 2013 by Devananda (talk | contribs) (Image Requirements: add shellinabox)

If you are looking for the old page, it has been moved here.


Baremetal is a driver for Openstack Nova Compute which controls physical hardware instead of virtual machines. This hardware is exposed via Openstack's API and in some ways acts like any other compute instance. Provisioning and management of physical hardware can thus be accomplished using common cloud tools. This opens the door for the orchestration of physical deployments using Heat, salt-cloud, and so on. In some ways, baremetal is also very different from other hypervisor drivers for Openstack, and deploying it requires some additional steps be taken and some additional configuration be done.


There is also some terminology which baremetal introduces.

  • Baremetal host and compute host are often used interchangeably to refer to the machine which runs the nova-compute and nova-baremetal-deploy-helper services (and possibly other services as well). This functions like a hypervisor, providing power management and imaging services.
  • Node and baremetal node refer to the physical machines which are controlled by the compute host. When a user requests that Nova start a baremetal instance, it is created on a baremetal node.
  • A baremetal instance is a Nova instance created directly on a physical machine without any virtualization layer running underneath it. Nova retains both power control (via IPMI) and, in some situations, may retain network control (via Quantum and OpenFlow).
  • Deploy image is pair of specialized kernel and ramdisk images which are used by the compute host to write the user-specified image onto the baremetal node.
  • Hardware is enrolled in the baremetal driver by adding its MAC addresses, physical characteristics (# CPUs, RAM, and disk space), and the IPMI credentials into the baremetal database. Without this information, the compute host has no knowledge of the baremetal node.


The current implementation of the Baremetal driver provides the following functionality.

  • A Nova API to enroll & manage hardware in the baremetal database
  • Power control of enrolled hardware via IPMI
  • PXE boot of the baremetal nodes.
  • Support for common CPU architectures (i386, x86_64)
  • FlatNetwork environments are supported and well tested
    • OpenFlow-enabled environments should be supported, but are less well tested at this time
  • Cloud-init is used for passing user data into the baremetal instances after provisioning. Limited support for file-injection also exists, but is being deprecated.

Current limitations include:

  • A separate dnsmasq process must run on the baremetal compute host to control the PXE boot process. This conflicts with quantum-dhcp, which must therefor be disabled.
  • Cloud-init requires an instances' IP be assigned by quantum, and without quantum-dhcp, this requires file injection to set the IP statically.

Future plans include:

  • Improve performance/scalability of PXE deployment process
  • Better support for complex non-SDN environments (eg., static VLANs)
  • Better integration with quantum-dhcp
  • Support snapshot and migrate of baremetal instances
  • Support non-PXE image deployment
  • Support other architectures (arm, tilepro)
  • Support fault-tolerance of baremetal nova-compute node

Key Differences

There are several key differences between the baremetal driver and other hypervisor drivers (kvm, xen, etc).

  • There is no hypervisor running underneath the baremetal instances, so the tenant has full and direct access to the hardware, and that hardware is dedicated to a single instance.
  • Nova does not have any access to manipulate a baremetal instance except for what is provided at the hardware level and exposed over the network, such as IPMI control. Therefor, some functionality implemented by other hypervisor drivers is not available via the baremetal driver, such as: instance snapshots, attach and detach network volumes to a running instance, and so on.
  • It is also important to note that there are additional security concerns created by tenants having direct access to the network (eg., MAC spoofing, packet sniffing, etc).
    • Other hypervisors mitigate this with virtualized networking.
    • Quantum + OpenFlow can be used much to the same effect, if your network hardware supports it.
  • Public cloud images may not work on some hardware, particularly if your hardware requires add'l drivers to be loaded.
  • The PXE driver requires a specialized ramdisk (and a corresponding kernel) for deployment, which is distinct from the cloud image's ramdisk. This can be built via the diskimage-builder project. The Glance UUIDs for these two images should be added to the extra_specs for any flavor (instance_type) that will be deployed onto a bare metal compute host. Alternatively, these UUIDs can also be added to the bare metal compute host's nova.conf file.


Here are a few ideas we have about potential use-cases for the baremetal driver. This isn't an exhaustive list -- there are doubtless many more interesting things which it can do!

  • High-performance computing clusters.
  • Computing tasks that require access to hardware devices which can't be virtualized.
  • Database hosting (some databases run poorly in a hypervisor).
  • Or, rapidly deploying a cloud infrastructure ....

We (the tripleo team) have a vision that Openstack can be used to deploy Openstack at a massive scale. We think the story of getting "from here to there" goes like this:

  • First, do simple hardware provisioning with a base image that contains configuration-management software (chef/puppet/salt/etc). The CMS checks in with a central server to determine what packages to install, then installs and configures your applications. All this happens automatically after first-boot of any baremetal node.
  • Then, accelerate provisioning by pre-installing your application software into the cloud image, but let a CMS still do all configuration.
  • Pre-install KVM and nova-compute into an image, and scale out your compute cluster by using baremetal driver to deploy nova-compute images. Do the same thing for Swift, proxy nodes, software load balancers, and so on.
  • Use Heat to orchestrate the deployment of an entire cloud.
  • Finally, run a mixture of baremetal nova-compute and KVM nova-compute in the same cloud (shared keystone and glance, but different tenants). Continuously deploy the cloud from the cloud using a common API.

The Baremetal Deployment Process

This section is a stub and needs to be expanded.

Differences in Starting a Baremetal Cloud

This section is a stub and needs to be expanded.

This section aims to cover the technical aspects of creating a baremetal deployment without duplicating the information required in general to create an openstack cloud. It starts by assuming you already have all the other services -- MySQL, Rabbit, Keystone, Glance, etc -- up and running, and then covers:

  • Nova configuration changes
  • Additional package requirements
  • Extra services that need to be started
  • Images, Instance types, and metadata that need to be created and defined

Configuration Changes

The following nova configuration options should be set, in addition to any others that your environment requires.

scheduler_host_manager = nova.scheduler.baremetal_host_manager.BaremetalHostManager
firewall_driver = nova.virt.firewall.NoopFirewallDriver
compute_driver = nova.virt.baremetal.driver.BareMetalDriver
# ComputeCapabilityFilter does not work with baremetal. https://bugs.launchpad.net/nova/+bug/1129485
scheduler_default_filters = ComputeFilter,RetryFilter,AvailabilityZoneFilter,ImagePropertiesFilter

net_config_template = /opt/stack/nova/nova/virt/baremetal/net-static.ubuntu.template
tftp_root = /tftpboot
power_manager = nova.virt.baremetal.ipmi.IPMI
driver = nova.virt.baremetal.pxe.PXE
instance_type_extra_specs = cpu_arch:{i386|x86_64}
sql_connection = mysql://{user}:{pass}@{host}/nova_bm

Additional Packages

If using the default baremetal driver (PXE) and default power driver (IPMI), then the baremetal compute host(s) must have the following packages installed to enable image deployment and power management.

 dnsmasq ipmitool open-iscsi syslinux

Additionally, to support PXE image deployments, the following steps should be taken:

 sudo mkdir -p /tftpboot/pxelinux.cfg
 sudo cp /usr/lib/syslinux/pxelinux.0 /tftpboot/
 sudo chown -R $NOVA_USER /tftpboot
 sudo mkdir -p $NOVA_DIR/baremetal/dnsmasq
 sudo mkdir -p $NOVA_DIR/baremetal/console
 sudo chown -R $NOVA_USER $NOVA_DIR/baremetal


At a minimum, Keystone, Nova, Glance, and Quantum must be up and running. The following additional services are currently required for baremetal deployment, and should be started on the nova compute host.

  • nova-baremetal-deploy-helper
    • This service assists with image deployment. It reads all necessary options from nova.conf.
  • dnsmasq
    • This must run on the nova-compute host, and quantum-dhcp must not be answering on the same network. Start it with the following command:
 # Disable any existing dnsmasq service
 sudo service dnsmasq disable && sudo pkill dnsmasq
 # Start dnsmasq for baremetal deployments. 
 # Change IFACE and RANGE as needed.
 sudo dnsmasq --conf-file= --port=0 --enable-tftp --tftp-root=/tftpboot \
   --dhcp-boot=pxelinux.0 --bind-interfaces --pid-file=/var/run/dnsmasq.pid \
   --interface=$IFACE --dhcp-range=$RANGE

A separate database schema must be created for the baremetal driver to store information about the enrolled hardware. Create it first:

 mysql> CREATE DATABASE nova_bm;
 mysql> GRANT ALL ON nova_bm.* TO 'nova_user'@'some_host' IDENTIFIED BY '$password';

Then initialize the database with:

 nova-baremetal-manage db sync

Image Requirements

The diskimage-builder project is provided as a toolchain for customizing and building both run-time images and the deployment images used by the PXE driver. Customization may be necessary if, for example, your hardware requires drivers not enabled or included in the default images.

Diskimage-builder requires the following packages be installed:

 python-lxml python-libvirt libvirt-bin qemu-system

Additionally, if you will be building a deploy image, you will need shellinabox and the following packages:

 qemu-kvm busybox tgt gcc make

To install shellinabox, run:

 wget http://shellinabox.googlecode.com/files/shellinabox-2.14.tar.gz
 tar -xzf shellinabox-2.14.tar.gz
 cd shellinabox-2.14
 sudo make install

To build images, clone the project and run the following:

 git clone https://github.com/stackforge/diskimage-builder.git
 cd diskimage-builder
 # build the image your users will run
 bin/disk-image-create -u base -o my-image
 # and extract the kernel & ramdisk
 bin/disk-image-get-kernel -d ./ -o my -i my-image.qcow2
 # build the deploy image
 # Note that this will build a kernel & ramdisk based on the host it is run on.
 KERNEL=$(uname -r)
 sudo cp /boot/vmlinuz-$KERNEL ./
 sudo chmod a+r vmlinuz-$KERNEL
 bin/ramdisk-image-create deploy -k $KERNEL -o my-deploy-ramdisk

Load all of these images into Glance, and note the glance image UUIDs for each one as it is generated. These are needed for associating the images to each other, and to the special baremetal flavor.

 glance image-create --name my-vmlinuz --public --disk-format aki my-vmlinuz
 glance image-create --name my-initrd --public --disk-format ari my-initrd
 glance image-create --name my-image --public --disk-format qcow2 --container-format bare \
     --property kernel_id=$MY_VMLINUZ_UUID --property ramdisk_id=$MY_INITRD_UUID < my-image
 glance image-create --name deploy-vmlinuz --public --disk-format aki < vmlinuz-$KERNEL
 glance image-create --name deploy-initrd --public --disk-format ari < my-deploy-ramdisk

You will also need to create a special baremetal flavor in Nova.

 # pick a unique number
 # change these to match your hardware
 nova flavor-create my-baremetal-flavor $FLAVOR_ID $RAM $DISK $CPU
 # associate the deploy images with this flavor
 # cpu-arch must match nova.conf, and of course, also must match your hardware
 nova flavor-key my-baremetal-flavor set \
   cpu-arch={i386|x86_64} \
   deploy_kernel_id=$DEPLOY_VMLINUZ_UUID \