- 1 Developer's DevStack + vSphere Guide
- 2 Development Environment
- 3 Setup localrc for DevStack
- 4 start stack (first time)
- 5 Credentials and Environment Variables
- 6 Glance Initial Setup
- 7 Nova Boot
- 8 Using Screen
- 9 Working the Development Environment
- 10 Testing
- 11 Troubleshooting
- 12 Appendix
Developer's DevStack + vSphere Guide
This is a short guide for developers interested in working on OpenStack and vSphere (ESX, ESXi, and vCenter) drivers.
vSphere Environment and Inventory
Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of these inventory types.
... or ...
Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.
Get an Ubuntu 12.04 (suggested) VM setup
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that may help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.
We suggest your Ubuntu workstation have
- at least 10 GB disk, with 20+ GB being ideal.
- at least one vCPU, with 2 being ideal.
- 4 GB of RAM.
install Devstack (http://devstack.org/) in your new VM
Once booted, run:
sudo apt-get install git git clone http://github.com/openstack-dev/devstack.git cd devstack
Setup localrc for DevStack
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon VIRT_DRIVER=vsphere VMWAREAPI_IP=$your_vCenter_ip_address VMWAREAPI_USER=root VMWAREAPI_PASSWORD=$password_goes_here VMWAREAPI_CLUSTER=$your_cluster_name DATABASE_PASSWORD=nova RABBIT_PASSWORD=nova SERVICE_TOKEN=nova SERVICE_PASSWORD=nova ADMIN_PASSWORD=nova FLAT_INTERFACE=tapfoo HOST_IP=$your_development_vm_ip
Note: omit the line VMWAREAPI_CLUSTER if you are using the trivial inventory example. Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path where you want to the logs to reside. For example
start stack (first time)
This will take a long time the first time, as it pulls down all code and dependencies. Note: may fail on first start.
Credentials and Environment Variables
get the OpenStack API credentials in your environment variables:
$ source openrc demo demo
This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point.
Glance Initial Setup
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API.
Get an initial VMDK to work with
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.
download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
Place the file under your home directory
Upload your VMDK to glance
$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \ --property vmware_disktype="preallocated" < ~/Debian-flat.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \ --disk-format=vmdk --property vmware-disktype="preallocated" < ~/Debian-flat.vmdk
This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.
boot a VM using the “<image-id>” from above
$ nova boot --image <image-id> --flavor 1 test1
Now you can interact with the image using other nova client commands. For example
$ nova list +--------------------------------------+-------+--------+------------+-------------+------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+------------------+ | 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 | +--------------------------------------+-------+--------+------------+-------------+------------------+
Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.
Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .
You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).
Use screen to interact with individual openstack services
$ screen -x stack
See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use
- "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
- the 'd' character detaches the session
- the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
Working the Development Environment
If you want to reset your environment, run:
- This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
- After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)
Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.
Install python support libraries
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential $ sudo apt-get install libxml2-dev libxslt1-dev
pip based tool installs
$ sudo pip install --upgrade pip $ sudo pip install --upgrade virtualenv $ sudo pip install MySQL-python $ sudo pip install tox $ sudo pip install python-subunit
pip proxy troubleshooting
If you get connection refused here - try this
Install OpenStack libs
$ cd /opt/stack/nova $ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt
Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.
Using TOX for testing
Important tests to run before submitting for code review are
tox -e py27 tox -e pep8
Code Coverage tests and information
cd nova tox -e cover (code coverage)
look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report
NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:
$ sudo add-apt-repository ppa:fkrull/deadsnakes $ sudo apt-get update $ sudo apt-get install python2.6 python2.6-dev
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, setup apt-get with a proxy. You may also have to set other proxy settings depending on your company's particular firewall rules.
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.
To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like screen-$SERVICE_NAME-$TIMESTAMP.log in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.
Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use localrc if you always want to have log files appear.
Assuming you have already made a directory...
$ mkdir ~/log
Then, if you only want logs this one time, in your bash shell say...
$ export SCREEN_LOGDIR=~/log $ export LOGDAYS=2
... or if you always want log files, in your localrc add the lines ...
The stack.sh script will automatically rotate the log files for you.
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.
$ ./unstack $ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove $ sudo apt-get install mysql-server-5.5 $ sudo mysql_install_db $ sudo mysqladmin -u root password 'nova' $ sudo service mysql restart $ ./stack
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
General Database Issues
If you have trouble with testing or other database heavy activities you may need to follow:
If networking suddenly stops working with an error like this
Traceback (most recent call last): File "/usr/bin/nova-rootwrap", line 59, in <module> from nova.rootwrap import wrapper ImportError: No module named rootwrap
Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin
- Download fresh Precise cloud image from: http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
- Convert to VMDK using "qemu-img convert". For example
qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk
- Convert your existing vmdk from sparse to thick using vmware CLI
Note: this is currently required, due to a bug we are investigating.
Once you have rCLI installed on your Ubuntu box you can follow the instructions in the link below. You can download rCLI from vmware.com. Go to vSphere downloads on the drivers and tools tab. The latest version from the date this was written is 5.1 and the link is here. You may need to install perldoc and other required libraries but the installer will tell you what to do.
Link to converting sparse vmdk files to thick -
For example - create the new vmdk, rename the old, rename the new to the old name
vmkfstools --server=10.20.120.61 -i /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_thick.vmdk
vmkfstools --server=10.20.120.61 -E /vmfs/volumes/storage1/testVM/testVM.vmdk /vmfs/volumes/storage1/testVM/testVM_sparse.vmdk
vmkfstools --server=10.20.120.61 -E /vmfs/volumes/storage1/testVM/testVM_thick.vmdk /vmfs/volumes/storage1/testVM/testVM.vmdk