OpenStack - User contributions [en]

NovaVMware/DeveloperGuide

2013-09-28T05:03:45Z

Vui: /* Specifying correct adapter type when uploading and image to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/debian-2.6.32-i686.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide" and vmware_disktype is set to "thin"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2013-09-20T07:05:13Z

Vui: /* Converting Images */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/debian-2.6.32-i686.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide" and vmware_disktype is set to "thin"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, images converted from qemu-img will _always_ if a monolithic sparse ide vmdk. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2013-09-20T07:03:06Z

Vui: /* Converting Images */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/debian-2.6.32-i686.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide" and vmware_disktype is set to "thin"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Convert a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, images converted from qemu-img will _always_ if a monolithic sparse ide vmdk. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2013-08-04T05:15:44Z

Vui: libpq-dev needed to build psycopg2

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and vSphere (ESX, ESXi, and vCenter) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of these inventory types.

[[File:VCenter trivial inventory.png|framed|none|a trivial Inventory]]

... or ...

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.
Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point.

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API.

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
https://www.dropbox.com/s/utvri5bw3zztty6/Debian-flat.vmdk

Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/Debian-flat.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/Debian-flat.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=
==Converting Images==
# Download fresh Precise cloud image from: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
# Convert to VMDK using "qemu-img convert". For example <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>
# Convert your existing vmdk from sparse to thick using vmware CLI

Note: this is currently required, due to a bug we are investigating.

Once you have rCLI installed on your Ubuntu box you can follow the instructions in the link below. You can download rCLI from vmware.com. Go to vSphere downloads on the drivers and tools tab. The latest version from the date this was written is 5.1 and the link is here. You may need to install perldoc and other required libraries but the installer will tell you what to do.

Link to converting sparse vmdk files to thick -
http://www.cosonok.com/2010/08/using-vmware-vsphere-cli-to-convert.html

For example - create the new vmdk, rename the old, rename the new to the old name

vmkfstools --server=10.20.120.61 -i /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_thick.vmdk

vmkfstools --server=10.20.120.61 -E /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_sparse.vmdk

vmkfstools --server=10.20.120.61 -E /vmfs/volumes/storage1/testVM/testVM_thick.vmdk /vmfs/volumes/storage1/testVM/testVM.vmdk

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z