NovaVMware/DeveloperGuide

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and vSphere (ESX, ESXi, and vCenter) drivers.

vSphere Environment and Inventory
Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:



Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=

Get an Ubuntu 12.04 (suggested) VM setup
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that may help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
 * at least 10 GB disk, with 20+ GB being ideal.
 * at least one vCPU, with 2 being ideal.
 * 4 GB of RAM.

optional setup
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under Network Troubleshooting

install Devstack (http://devstack.org/) in your new VM
Once booted, run: sudo apt-get install git git clone http://github.com/openstack-dev/devstack.git cd devstack

=Setup localrc for DevStack= create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/localrc ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon VIRT_DRIVER=vsphere VMWAREAPI_IP=$your_vCenter_ip_address VMWAREAPI_USER=root VMWAREAPI_PASSWORD=$password_goes_here VMWAREAPI_CLUSTER=$your_cluster_name DATABASE_PASSWORD=nova RABBIT_PASSWORD=nova SERVICE_TOKEN=nova SERVICE_PASSWORD=nova ADMIN_PASSWORD=nova HOST_IP=$your_development_vm_ip

Note: if you would like to use a specific datastore then configure VMWAREAPI_DATASTORE_REGEX

Note: omit the line VMWAREAPI_CLUSTER if you are using the trivial inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)= ./stack.sh This will take a long time the first time, as it pulls down all code and dependencies. Note: may fail on first start.

=Credentials and Environment Variables= get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup= Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

Get an initial VMDK to work with
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira). http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk Place the file under your home directory

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

Upload your VMDK to glance
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance: https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh upload_image.sh has a dependency: https://github.com/openstack-dev/devstack/blob/master/functions The script will introspect the image provided to extract the metadata. It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk. If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution: $ cd ~/devstack $ ./tools/upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk $ ./tools/upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk $ ./tools/upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is - ; ; .vmdk By default, the Nova driver will be using adapter_type "lsiLogic" and and disk_type "preallocated"

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory. For information about disk types supported by the VMware driver, see http://docs.openstack.org/trunk/config-reference/content/vmware.html. For general information about disk types: http://sanbarrow.com/vmdk-basics.html.

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an  for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

wget "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk" Note the vmware_adaptertype is set to "ide" $ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < "trend-tinyvm1-flat-;ide;.vmdk"

=Nova Boot= boot a VM using the “” from above

$ nova boot --image  --flavor 1 test1

Now you can interact with the image using other nova client commands. For example $ nova list +--+---+++-+--+ | ID                                  | Name  | Status | Task State | Power State | Networks         | +--+---+++-+--+ | 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None       | NOSTATE     | private=10.0.0.2 | +--+---+++-+--+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira.

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen= Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

characters 0 through 9 switch tabs
 * "Control + a" enters command mode... enter this for each new command character
 * the 'd' character detaches the session
 * the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
 * the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment= If you want to reset your environment, run: ./unstack.sh ./stack.sh


 * This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
 * After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe eclipse), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000 --- a/nova/cmd/__init__.py +++ b/nova/cmd/__init__.py @@@ -31,8 -31,4 +31,8 @@@ os.environ['EVENTLET_NO_GREENDNS'] = 'y import eventlet -eventlet.monkey_patch(os=False) +if '--debug-host' and '--debug-port' in sys.argv: +   eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False) +else: +   eventlet.monkey_patch(os=False) +

diff --combined nova/cmd/compute.py index 61cf434,f03a9bd..0000000 --- a/nova/cmd/compute.py +++ b/nova/cmd/compute.py @@@ -33,19 -33,10 +33,19 @@@ from nova.openstack.common import log a from nova import service from nova import utils +cli_opts = [ +       cfg.StrOpt('debug-host', +                    default=None, +                    help='Connect to debug host'), +       cfg.IntOpt('debug-port', +                    default=None, +                    help='Connect to debug host port'), +   ] +  CONF = cfg.CONF CONF.import_opt('compute_topic', 'nova.compute.rpcapi') CONF.import_opt('use_local', 'nova.conductor.api', group='conductor') - +CONF.register_cli_opts(cli_opts) def block_db_access: class NoDB(object): @@@ -72,16 -63,6 +72,16 @@@ def main objects_base.NovaObject.indirection_api = \ conductor_rpcapi.ConductorAPI +   if CONF.debug_host: +       from pydev import pydevd +       print ('Listening on ' + str(CONF.debug_port) + ' for host ' + +                                    CONF.debug_host) +       pydevd.settrace(host=CONF.debug_host, +                        port=CONF.debug_port, +                        stdoutToServer=False, +                        stderrToServer=False) + +     server = service.Service.create(binary='nova-compute',                                      topic=CONF.compute_topic,                                      db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host  and --debug_port. That's it!

=Testing=

apt-get libs
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential $ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

pip based tool installs
$ sudo pip install --upgrade pip $ sudo pip install --upgrade virtualenv $ sudo pip install MySQL-python $ sudo pip install tox $ sudo pip install python-subunit

pip proxy troubleshooting
If you get connection refused here - try this export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command sudo pip install setuptools --no-use-wheel --upgrade

Install OpenStack libs
$ cd /opt/stack/nova $ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

Using TOX for testing
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are tox -e py26,py27 tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

Using run_tests.sh for testing
$ cd /opt/stack/nova $ ./run_tests.sh ... or ... $ ./run_tests.sh nova.tests.virt.vmwareapi ... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

with tox
$ cd /opt/stack/nova $ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

with run_tests.sh
$ cd /opt/stack/nova $ ./run_tests.sh --coverage

Additional setup
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes $ sudo apt-get update $ sudo apt-get install python2.6 python2.6-dev

Disabling verbose logging subsystems
Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting= ''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, setup apt-get with a proxy. You may also have to set other proxy settings depending on your company's particular firewall rules.''

Logging
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like screen-$SERVICE_NAME-$TIMESTAMP.log in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use localrc if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log $ export LOGDAYS=2

... or if you always want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

MySQL issues
If you get stuck with Keystone try... $ mysql -u root -pnova ... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack $ sudo apt-get purge mysql-server ... you may need to use the name mysql-server-5.5 instead ... $ sudo apt-get autoremove $ sudo apt-get install mysql-server-5.5 $ sudo mysql_install_db $ sudo mysqladmin -u root password 'nova' $ sudo service mysql restart $ ./stack

Keystone Issues
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

General Database Issues
If you have trouble with testing or other database heavy activities you may need to follow: http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

rootwrap Issues
If networking suddenly stops working with an error like this

Traceback (most recent call last): File "/usr/bin/nova-rootwrap", line 59, in     from nova.rootwrap import wrapper ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

Check if nova-compute is running
$ screen -x Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. ctl+a then 6. If you see: $ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure" sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist $ ... then nova-compute did not start. You can manually start nova-compute by editing the command to read: $ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure" ... which is permissible for development (but not for production).

Networking Troubleshooting
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a locally accessible VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

Configure a fake networking interface
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap $ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your localrc file FLAT_INTERFACE=tapfoo

Configure a Proxy
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, setup apt-get with a proxy. You may also have to set other proxy settings depending on your company's particular firewall rules.

Self-signed Certificates
In case you wanna connect to a vSphere with a self-signed certificate, since Liberty you have to disable certificate verification for nova and cinder:

$NOVA_CONF [vmware] insecure = true $CINDER_CONF [vmdk] vmware_insecure = true

=Appendix=

Converting other disk images to vmdk using qemu-img
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):

Converting a sparse vmdk to an ESX-compatible format
Due to a bug (fixed with the following proposed patch, currently under review) in how sparse vmdk disks are handled in the VC driver, sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.
The latest version from the date this was written is 5.1 and the link is here.

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

2. Using vmkfstools directly on the ESX host
If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh) vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

3. vmware-vdiskmanager
vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation. Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

Specifying correct adapter type when uploading and image to glance
Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

Obtaining the adapter type associated with a vmdk
If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

List of Reviews in progress
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z