Jump to: navigation, search

Difference between revisions of "NovaVMware/DeveloperGuide"

(Get an initial VMDK to work with)
(Get an initial VMDK to work with)
Line 76: Line 76:
download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).   
download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).   
Place the file under your home directory <code><nowiki>~/</nowiki></code>
Place the file under your home directory <code><nowiki>~/</nowiki></code>

Revision as of 11:38, 20 August 2013

Developer's DevStack + vSphere Guide

This is a short guide for developers interested in working on OpenStack and vSphere (ESX, ESXi, and vCenter) drivers.

vSphere Environment and Inventory

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of these inventory types.

a trivial Inventory

... or ...

inventory with a DRS cluster

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

Development Environment

Get an Ubuntu 12.04 (suggested) VM setup

This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that may help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have

  • at least 10 GB disk, with 20+ GB being ideal.
  • at least one vCPU, with 2 being ideal.
  • 4 GB of RAM.

optional setup

If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under Network Troubleshooting

install Devstack (http://devstack.org/) in your new VM

Once booted, run:

sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

Setup localrc for DevStack

create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/localrc


Note: omit the line VMWAREAPI_CLUSTER if you are using the trivial inventory example. Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path where you want to the logs to reside. For example


start stack (first time)


This will take a long time the first time, as it pulls down all code and dependencies. Note: may fail on first start.

Credentials and Environment Variables

get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point.

Glance Initial Setup

Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API.

Get an initial VMDK to work with

There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).


Place the file under your home directory ~/

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

Upload your VMDK to glance

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
   --property vmware_disktype="preallocated" < ~/Debian-flat.vmdk

... or ...

$ glance image-create --name Debian --is-public=True --container-format=bare \
   --disk-format=vmdk --property vmware-disktype="preallocated" < ~/Debian-flat.vmdk 

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

Nova Boot

boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands. For example

$ nova list
| ID                                   | Name  | Status | Task State | Power State | Networks         |
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE  | None       | NOSTATE     | private= |

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

Using Screen

Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

  • "Control + a" enters command mode... enter this for each new command character

characters 0 through 9 switch tabs

    • the 'd' character detaches the session
    • the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

Working the Development Environment

If you want to reset your environment, run:

  • This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
  • After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.


Install python support libraries

apt-get libs

$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential 
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

pip based tool installs

$ sudo pip install --upgrade pip 
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

pip proxy troubleshooting

If you get connection refused here - try this

export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

Install OpenStack libs

$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack  -r requirements.txt -r test-requirements.txt 

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

Using TOX for testing

See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are

tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

Using run_tests.sh for testing

$ cd /opt/stack/nova
$ ./run_tests.sh

... or ...

$ ./run_tests.sh nova.tests.virt.vmwareapi

... to run only the vmware unit tests.

Code Coverage tests and information

with tox

$ cd /opt/stack/nova
$ tox -e cover  (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

with run_tests.sh

 $ cd /opt/stack/nova
 $ ./run_tests.sh --coverage

Additional setup

If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev


Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, setup apt-get with a proxy. You may also have to set other proxy settings depending on your company's particular firewall rules.


You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like screen-$SERVICE_NAME-$TIMESTAMP.log in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use localrc if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you always want log files, in your localrc add the lines ...


The stack.sh script will automatically rotate the log files for you.

MySQL issues

If you get stuck with Keystone try...

$ mysql -u root -pnova

... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server

... you may need to use the name mysql-server-5.5 instead ...

$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

Keystone Issues

If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment


General Database Issues

If you have trouble with testing or other database heavy activities you may need to follow:


rootwrap Issues

If networking suddenly stops working with an error like this

Traceback (most recent call last):
  File "/usr/bin/nova-rootwrap", line 59, in <module>
    from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

Check if nova-compute is running

$ screen -x

Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. ctl+a then 6. If you see:

$ cd /opt/stack/nova && sg  '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist

... then nova-compute did not start. You can manually start nova-compute by editing the command to read:

$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"

... which is permissible for development (but not for production).

Networking Troubleshooting

Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a locally accessible VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

Configure a fake networking interface

Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up 

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your localrc file


Configure a Proxy

Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, setup apt-get with a proxy. You may also have to set other proxy settings depending on your company's particular firewall rules.


Converting Images

  1. Download fresh Precise cloud image from: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
  2. Convert to VMDK using "qemu-img convert". For example
    qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk
  3. Convert your existing vmdk from sparse to thick using vmware CLI

Note: this is currently required, due to a bug we are investigating.

Once you have rCLI installed on your Ubuntu box you can follow the instructions in the link below. You can download rCLI from vmware.com. Go to vSphere downloads on the drivers and tools tab. The latest version from the date this was written is 5.1 and the link is here. You may need to install perldoc and other required libraries but the installer will tell you what to do.

Link to converting sparse vmdk files to thick -


For example - create the new vmdk, rename the old, rename the new to the old name

vmkfstools --server= -i /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_thick.vmdk 
vmkfstools --server= -E /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_sparse.vmdk 
vmkfstools --server= -E /vmfs/volumes/storage1/testVM/testVM_thick.vmdk /vmfs/volumes/storage1/testVM/testVM.vmdk

List of Reviews in progress