OpenStack - User contributions [en]

Glance/JunoCycleMeetup

2014-07-25T22:57:29Z

Arnaud Legendre: /* Schedule and Content */

== Glance Juno Cycle Meetup ==
=== Basic Details ===
==== What ====
The Catalog Program Meetup.

All OpenStack ATCs and associated technical product folks are welcome, however space is limited and only a few spots remain available. If you plan to attend please contact alegendre at vmware dot com.

==== Where ====
VMware, Inc

3425 Hillview Avenue - Building Hilltop A

Palo Alto, CA 94304

==== When ====
[http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140725T09&p1=1240 July 25-26 2014, 9:00 AM - 5:30 PM each day in PST time zone]

=== Schedule and Content ===

Ideas page: https://etherpad.openstack.org/p/glance-juno-mid-cycle-meeting

{| class="wikitable"
|-
! colspan=3 | Thursday, July 25
|-
! Time
! Lead(s)
! width=400 | Description
|-
| align="center" | 8:30 AM
! colspan=2 | Breakfast (provided)
|-
| align="center" | 9:00 AM
| Mark Washenberger and Arnaud legendre
| Summit Kickoff
|-
| align="center" | 9:20 AM
| Mark Washenberger and Brian Rosmaita
| Community Image Sharing
|-
| align="center" | 9:50 AM
| Nikhil Komawar
| Moving testing out of Glance
|-
| align="center" | 10:20 AM
! colspan=2 | Break
|-
| align="center" | 10:40 AM
| Brian Rosmaita
| Glance schemas
|-
| align="center" | 11:20 AM
| Nikhil Komawar, Brian Rosmaita and Arnaud Legendre
| Tasks and Cloning
|-
| align="center" | 12:00 PM
! colspan=2 | Lunch (provided)
|-
| align="center" | 1:30 PM
| Nikhil Komawar and Mark Washenberger
| Transfer Service
|-
| align="center" | 2:30 PM
| Stuart McLaren
| Keystone Composite Auth
|-
| align="center" | 3:15 PM
! colspan=2 | Break
|-
| align="center" | 3:30 PM
! colspan=2 | Post-It session Part 1
|-
| align="center" | 4:15 PM
| Mark Washenberger and Arnaud Legendre
| Improving the development process
|-
! align="center" | 5:30 PM
! colspan=2 | Break For Day
|}

{| class="wikitable"
|-
! colspan=3 | Friday, July 25
|-
! Time
! Lead(s)
! width=400 | Description
|-
| align="center" | 8:30 AM
! colspan=2 | Breakfast (provided)
|-
| align="center" | 9:00 AM
| Alexander Tivelkov
| Artifiacts Part 1
|-
| align="center" | 9:45 AM
! colspan=2 | Break
|-
| align="center" | 10:00 AM
| Alexander Tivelkov
| Artifacts Part 2
|-
| align="center" | 10:45 AM
! colspan=2 | Post-It session Part 2
|-
| align="center" | 11:15 AM
| Erno Kuvaja
| Diagnostic
|-
| align="center" | 12:15 PM
! colspan=2 | Lunch (provided)
|-
| align="center" | 1:30 PM
| Travis Tripp, Lakshmi Sampath, Wayne Okume
| Graffiti Part 1
|-
| align="center" | 2:15 PM
! colspan=2 | Break
|-
| align="center" | 2:30 PM
| Travis Tripp, Lakshmi Sampath, Wayne Okume
| Graffiti Part 2
|-
| align="center" | 3:15 PM
! colspan=2 | Break
|-
| align="center" | 3:30 PM
|
| More Artifacts and Metadata and more...
|-
| align="center" | 4:30 PM
|
| Open Discussion
|-
! align="center" | 5:15 PM
! colspan=2 | Break For Day
|}

==== Open Discussion Format ====
During the meetup, there will be several opportunities for open discussion. To make sure everyone's ideas have an opportunity to be heard,
we will be following a lightweight structure for these sessions. In the first 5 minutes, each attendee will submit ideas on post-it notes and the
whole group will vote on which topics to discuss. Then the remainder of the session will focus on the ideas we chose.

=== Joining Remotely ===
It will be possible to join remotely through Webex. This requires the Webex client installed on the machine and a phone to be able to dial in. Please ping arnaud on irc or send email
in order to get the Webex details (URL and passcode).
Live feedback can be given on IRC (freenode) in the #openstack-glance channel. We will have a local moderator
monitoring the IRC channel and relaying questions and comments as appropriate.

=== Summit Notes ===
Here is a list of etherpads and slides that were used during the summit
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-community-sharing
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-tasks-and-cloning
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-schemas
* https://etherpad.openstack.org/p/image-transfer-service-proposal-juno
* https://etherpad.openstack.org/p/glance-metadata-catalog-juno-mid-cycle-meeting
* https://etherpad.openstack.org/p/glance-tests-move-to-tempest-juno
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-artifacts

Glance/JunoCycleMeetup

2014-07-24T03:20:01Z

Arnaud Legendre: Created page with "== Glance Juno Cycle Meetup == === Basic Details === ==== What ==== The Catalog Program Meetup. All OpenStack ATCs and associated technical product folks are welcome, however..."

== Glance Juno Cycle Meetup ==
=== Basic Details ===
==== What ====
The Catalog Program Meetup.

All OpenStack ATCs and associated technical product folks are welcome, however space is limited and only a few spots remain available. If you plan to attend please contact alegendre at vmware dot com.

==== Where ====
VMware, Inc

3425 Hillview Avenue - Building Hilltop A

Palo Alto, CA 94304

==== When ====
[http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140725T09&p1=1240 July 25-26 2014, 9:00 AM - 5:30 PM each day in PST time zone]

=== Schedule and Content ===

Ideas page: https://etherpad.openstack.org/p/glance-juno-mid-cycle-meeting

{| class="wikitable"
|-
! colspan=3 | Thursday, July 25
|-
! Time
! Lead(s)
! width=400 | Description
|-
| align="center" | 8:30 AM
! colspan=2 | Breakfast (provided)
|-
| align="center" | 9:00 AM
| Mark Washenberger and Arnaud legendre
| Summit Kickoff
|-
| align="center" | 9:20 AM
| Mark Washenberger and Brian Rosmaita
| Community Image Sharing
|-
| align="center" | 9:50 AM
| Nikhil Komawar
| Moving testing out of Glance
|-
| align="center" | 10:20 AM
! colspan=2 | Break
|-
| align="center" | 10:40 AM
| Brian Rosmaita
| Glance schemas
|-
| align="center" | 11:20 AM
| Nikhil Komawar, Brian Rosmaita and Arnaud Legendre
| Tasks and Cloning
|-
| align="center" | 12:00 PM
! colspan=2 | Lunch (provided)
|-
| align="center" | 1:30 PM
| Nikhil Komawar and Mark Washenberger
| Transfer Service
|-
| align="center" | 2:30 PM
| Stuart McLaren
| Keystone Composite Auth
|-
| align="center" | 3:15 PM
! colspan=2 | Break
|-
| align="center" | 3:30 PM
! colspan=2 | Post-It session Part 1
|-
| align="center" | 4:15 PM
| Mark Washenberger and Arnaud Legendre
| Improving the development process
|-
! align="center" | 5:30 PM
! colspan=2 | Break For Day
|}

{| class="wikitable"
|-
! colspan=3 | Friday, July 25
|-
! Time
! Lead(s)
! width=400 | Description
|-
| align="center" | 8:30 AM
! colspan=2 | Breakfast (provided)
|-
| align="center" | 9:00 AM
| Alexander Tivelkov
| Artifiacts Part 1
|-
| align="center" | 9:45 AM
! colspan=2 | Break
|-
| align="center" | 10:00 AM
| Alexander Tivelkov
| Artifacts Part 2
|-
| align="center" | 10:45 AM
! colspan=2 | Post-It session Part 2
|-
| align="center" | 11:15 AM
| Erno Kuvaja
| Diagnostic
|-
| align="center" | 12:15 PM
! colspan=2 | Lunch (provided)
|-
| align="center" | 1:30 PM
| Travis Tripp, Lakshmi Sampath, Wayne Okume
| Graffiti Part 1
|-
| align="center" | 2:15 PM
! colspan=2 | Break
|-
| align="center" | 2:30 PM
| Travis Tripp, Lakshmi Sampath, Wayne Okume
| Graffiti Part 2
|-
| align="center" | 3:15 PM
! colspan=2 | Break
|-
| align="center" | 3:30 PM
| Arnaud Legendre
| Getting teeth into a bug
|-
| align="center" | 4:30 PM
|
| Open Discussion
|-
! align="center" | 5:15 PM
! colspan=2 | Break For Day
|}

==== Open Discussion Format ====
During the meetup, there will be several opportunities for open discussion. To make sure everyone's ideas have an opportunity to be heard,
we will be following a lightweight structure for these sessions. In the first 5 minutes, each attendee will submit ideas on post-it notes and the
whole group will vote on which topics to discuss. Then the remainder of the session will focus on the ideas we chose.

=== Joining Remotely ===
It will be possible to join remotely through Webex. This requires the Webex client installed on the machine and a phone to be able to dial in. Please ping arnaud on irc or send email
in order to get the Webex details (URL and passcode).
Live feedback can be given on IRC (freenode) in the #openstack-glance channel. We will have a local moderator
monitoring the IRC channel and relaying questions and comments as appropriate.

=== Summit Notes ===
Here is a list of etherpads and slides that were used during the summit
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-community-sharing
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-tasks-and-cloning
* https://etherpad.openstack.org/p/glance-juno-mid-cycle-schemas
* https://etherpad.openstack.org/p/image-transfer-service-proposal-juno
* https://etherpad.openstack.org/p/glance-metadata-catalog-juno-mid-cycle-meeting
* https://etherpad.openstack.org/p/glance-tests-move-to-tempest-juno

Sprints

2014-06-19T14:07:58Z

Arnaud Legendre: /* Under discussion */

=== Future sprints ===

Here is a chronological list of future sprints. please keep it ordered and remove past ones.

{| border="1" cellpadding="4" cellspacing="4"
|- bgcolor=#eeeeee
| Date
| Location
| Theme
| More information at
|-
| June 17 - 19
| San Antonio, TX, USA
| LBaaS in Neutron
| [https://etherpad.openstack.org/p/neutron-juno-lbaas-mid-cycle]
|-
| July 2 - 4
| Paris, France
| Ceilometer/All projects
| [[Sprints/ParisJuno2014]]
|-
| July 7 - 9
| San Antonio, TX, USA
| Barbican
| TBD, [[Meetings/Barbican]]
|-
| July 9 - 11
| Bloomington, MN, USA
| Neutron
| [https://etherpad.openstack.org/p/neutron-juno-mid-cycle-meeting]
|-
| July 9 - 11
| San Antonio, TX, USA
| Keystone
| [http://dolphm.com/openstack-keystone-hackathon-for-juno/]
|-
| July 14 - 18
| Darmstadt, Germany
| QA & Infra
| [[Qa_Infra_Meetup_2014]]
|-
| July 14 - 18
| Seattle, USA
| Security Group
| [https://etherpad.openstack.org/p/ossg-juno-meetup]
|-
| July 21 - July 25
| Raleigh, NC, USA
| TripleO & Heat
| [https://etherpad.openstack.org/p/juno-midcycle-meetup]
|-
| July 28 - Jul 30
| Beaverton, OR, USA
| Nova & Ironic
| [[Sprints/BeavertonJunoSprint]]
|-
| Aug 11 - Aug 15
| Fort Collins, CO
| Cinder
| [[https://etherpad.openstack.org/p/CinderMidCycleMeetupAug2014]]
|-
| Aug 20 - Aug 23
| Cambridge, MA, USA
| Trove
| [[Trove/JunoCycleMeetup|Link]]
|-
| July 24 - July 25
| Palo Alto, CA, USA
| Glance
| [[https://etherpad.openstack.org/p/glance-juno-mid-cycle-meeting]]
|}

Sprints

2014-06-19T14:07:37Z

Arnaud Legendre: /* Future sprints */

Sprints

2014-06-05T18:09:35Z

Arnaud Legendre:

Here is a chronological list of future sprints.

{| border="1" cellpadding="4" cellspacing="4"
|- bgcolor=#eeeeee
| Date
| Location
| Theme
| More information at
|-
| June 17 - 19
| San Antonio, TX, USA
| LBaaS in Neutron
| [https://etherpad.openstack.org/p/neutron-juno-lbaas-mid-cycle]
|-
| July 2 - 4
| Paris, France
| Ceilometer/All projects
| [[Sprints/ParisJuno2014]]
|-
| July 7 - 9
| San Antonio, TX, USA
| Barbican
| TBD, [[Meetings/Barbican]]
|-
| July 9 - 11
| Bloomington, MN, USA
| Neutron
| [https://etherpad.openstack.org/p/neutron-juno-mid-cycle-meeting]
|-
| July 9 - 11
| San Antonio, TX, USA
| Keystone
| [http://dolphm.com/openstack-keystone-hackathon-for-juno/]
|-
| July 14 - 18
| Darmstadt, Germany
| QA & Infra
| tbd
|-
| July 28 - Aug 1
| Raleigh, NC, USA
| TripleO & Ironic
| [https://etherpad.openstack.org/p/juno-midcycle-meetup]
|-
| Aug 20 - Aug 23
| Cambridge, MA, USA
| Trove
| TBA
|-
| 2nd half July
| Portland, OR
| Nova
| Survey [https://docs.google.com/forms/d/1ws6iezBwQHvvEP5_9lOI2MclJOLFi75XwbPfji4ea_M/viewform]
|-
| End of July
| CA, USA
| Glance
| [https://etherpad.openstack.org/p/glance-juno-mid-cycle-meeting]
|}

Sprints

2014-06-05T18:09:20Z

Arnaud Legendre:

Here is a chronological list of future sprints.

{| border="1" cellpadding="4" cellspacing="4"
|- bgcolor=#eeeeee
| Date
| Location
| Theme
| More information at
|-
| June 17 - 19
| San Antonio, TX, USA
| LBaaS in Neutron
| [https://etherpad.openstack.org/p/neutron-juno-lbaas-mid-cycle]
|-
| July 2 - 4
| Paris, France
| Ceilometer/All projects
| [[Sprints/ParisJuno2014]]
|-
| July 7 - 9
| San Antonio, TX, USA
| Barbican
| TBD, [[Meetings/Barbican]]
|-
| July 9 - 11
| Bloomington, MN, USA
| Neutron
| [https://etherpad.openstack.org/p/neutron-juno-mid-cycle-meeting]
|-
| July 9 - 11
| San Antonio, TX, USA
| Keystone
| [http://dolphm.com/openstack-keystone-hackathon-for-juno/]
|-
| July 14 - 18
| Darmstadt, Germany
| QA & Infra
| tbd
|-
| July 28 - Aug 1
| Raleigh, NC, USA
| TripleO & Ironic
| [https://etherpad.openstack.org/p/juno-midcycle-meetup]
|-
| Aug 20 - Aug 23
| Cambridge, MA, USA
| Trove
| TBA
|-
| 2nd half July
| Portland, OR
| Nova
| Survey [https://docs.google.com/forms/d/1ws6iezBwQHvvEP5_9lOI2MclJOLFi75XwbPfji4ea_M/viewform]
|-
| End of July
| CA
| Glance
| [https://etherpad.openstack.org/p/glance-juno-mid-cycle-meeting]
|}

NovaVMware/DeveloperGuide

2014-04-18T22:51:45Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack
$ ./tools/upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./tools/upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./tools/upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk
By default, the Nova driver will be using adapter_type "lsiLogic" and and disk_type "preallocated"

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For information about disk types supported by the VMware driver, see http://docs.openstack.org/trunk/config-reference/content/vmware.html . For general information about disk types: http://sanbarrow.com/vmdk-basics.html.

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

wget "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < "trend-tinyvm1-flat-;ide;.vmdk"

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

Python3

2014-03-03T19:15:37Z

Arnaud Legendre: /* Dependencies */

This page tracks the progress of Python 3 effort porting for OpenStack.

== Python 3 ==

[http://techs.enovance.com/6521/openstack_python3 Why should OpenStack move to Python 3 right now?]
:''Python 3 is usually seen as the new Python version which breaks compatibility and raises new Unicode issues. Python 3 is much more than that. It’s a new clean language which has a more consistent syntax. It has many new features, not less than 15 new modules. Python 3 is already well supported by major Linux distributions, whereas Python 2.7 reached its end-of-life. Slowly, some bugs cannot be fixed in Python 2.7 anymore and are only fixed in the latest Python 3 release. Python 3 is now 5 years old and considered as a mature programming language.''

== Port Python 2 code to Python 3 ==

OpenStack project chose to use the same code base for Python 2 and Python 3. The [http://pythonhosted.org/six/ Six: Python 2 and 3 Compatibility Library] helps to write code working on both versions.

=== References to port Python 2 code to Python 3 ===
* [http://python3porting.com/ Porting to Python 3 Book] by Lennart Regebro, especially the [http://python3porting.com/differences.html Language differences and workarounds].
* [http://docs.python.org/dev/howto/pyporting.html HOWTO: Porting Python 2 Code to Python 3] by Brett Cannon
* [https://wiki.python.org/moin/PortingPythonToPy3k Porting Python Code to 3.x]
* [http://code.google.com/p/python-incompatibility/ python-incompatibility]: Demonstrates incompatibilities between Python versions.

=== Common pitfalls ===

==== What is a string ? ====
You should definitely not talk about "strings" in your commit logs/reviews. In Python 2, a 'string' is bytes; in Python 3, it's a Unicode text string. The following code snippet may help in understanding the difference:

Python 2:
>>> type('foo')
<type 'str'>
>>> type(u'foo')
<type 'unicode'>
>>> type(b'foo')
<type 'str'>
>>> isinstance('foo', six.text_type)
False
>>> isinstance(u'foo', six.text_type)
True
>>> bytes is str
True
>>> b'foo'[0]
'f'

Python 3:
>>> type('foo')
<class 'str'>
>>> type(u'foo')
<class 'str'>
>>> type(b'foo')
<class 'bytes'>
>>> isinstance('foo', six.text_type)
True
>>> isinstance(b'foo', six.text_type)
False
>>> bytes is str
False
>>> b'foo'[0]
102

== Python 3 of OpenStack Dependencies ==

'''Blocker Pointer:''' it's not yet possible to specify different list of dependencies for Python 2 and Python 3. For example, mox only works on Python 2, mox3 can be used on Python 3.

* Julien Danjou proposed to add requirements-py3.txt: [https://review.openstack.org/#/c/58770/ openstack/requirements patch] and [https://review.openstack.org/#/c/63236/ openstack-dev/pbr patch]
* An alternative is to support markers in requirements (in pip): [https://github.com/pypa/pip/issues/1433 pip issue: Support markers in setup(install_requires)?]; [https://github.com/pypa/pip/pull/1472 Victor Stinner's pull request: "parse requirements in markers"]

OpenStack Dependencies:

* mox: use mox3 or port tests on mock which works on Python 3 (mock has been integrated in Python 3.3 as unittest.mock)
* eventlet: not available on Python 3 yet, alternatives: [http://docs.python.org/dev/library/asyncio.html asyncio] ([http://code.google.com/p/tulip/ Tulip] for Python 3.3+/[https://bitbucket.org/haypo/trollius Trollius] for Python 2), [http://www.tornadoweb.org/ Tornado]

== Portage in progress ==

* Oslo Messaging: Portage in Progress by Victor Stinner ([https://review.openstack.org/#/dashboard/9107 dashboard])
* glanceclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
* heatclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
* neutronclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])
* glanceclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])

== Portage done ==
* keystoneclient: Portage in Progress by Cyril Roelandt ([https://review.openstack.org/#/dashboard/8122 dashboard])

== Python 3 Status of OpenStack projects ==

See also [[Python3Deps]].

=== OpenStack clients ===
{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-
| [https://pypi.python.org/pypi/python-ceilometerclient python-ceilometerclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color:orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/python-cinderclient python-cinderclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting ||https://review.openstack.org/#/c/73844/ ||
|-
| [https://pypi.python.org/pypi/python-ganttclient python-ganttclient] || ? || ? || ? || ? ||
|-
| [https://pypi.python.org/pypi/python-glanceclient python-glanceclient] || style="background-color: orange;" | In Progress || style="background-color: orange" | Non-voting || || python-keystoneclient ||
|-
| [https://pypi.python.org/pypi/python-heatclient python-heatclient] || style="background-color: orange;" | In progress || style="background-color: orange;" | Non-voting || || || https://review.openstack.org/#/dashboard/8122
|-
| [https://pypi.python.org/pypi/python-ironicclient python-ironicclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen" | Voting || style="background-color: lightgreen;" | On PyPI || ||
|-
| [https://pypi.python.org/pypi/python-keystoneclient python-keystoneclient] || style="background-color: lightgreen;" | Yes || style="background-color:lightgreen;" | Voting || style="background-color: orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/python-marconiclient python-marconiclient]|| style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: red;" | No ||
|-
| [https://pypi.python.org/pypi/python-melangeclient python-melangeclient] || ? || ? || ? || ||
|-
| [https://pypi.python.org/pypi/python-novaclient python-novaclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || https://review.openstack.org/#/c/75964/ ||
|-
| [https://pypi.python.org/pypi/python-neutronclient python-neutronclient] || style="background-color: orange;" | In progress || style="background-color: orange;" | Non-voting || || Differences between Python 2 and 3 ||
|-
| [https://pypi.python.org/pypi/python-openstackclient python-openstackclient] || style="background-color: orange" | In Progress || style="background-color: orange" | Non-Voting || || Works with glanceclient HEAD ||
|-
| [https://pypi.python.org/pypi/python-savannaclient python-savannaclient] || style="background-color: orange;" | In progress || style="background-color: orange" | Non-voting || || python-keystoneclient || https://review.openstack.org/#/c/73128/
|-
| [https://pypi.python.org/pypi/python-swiftclient python-swiftclient] || style="background-color: orange;" | In progress || style="background-color: orange;" | Non-voting || || Differences between Python 2 and 3 ||
|-
| [https://pypi.python.org/pypi/python-ceilometerclient python-tuskarclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: red;" | No|| || https://review.openstack.org/#/c/73130/
|-
| [https://pypi.python.org/pypi/python-troveclient python-troveclient] || style="background-color: lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || https://review.openstack.org/#/c/75075/ || || ||
|}

=== Core OpenStack projects ===
{| class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Trove classifiers !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/ceilometer ceilometer] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* hacking
* mysql-python
* oslo.config
* oslosphinx
* python-ceilometerclient
* python-glanceclient
* python-swiftclient
* sphinxcontrib-docbookrestapi
* sphinxcontrib-httpdomain
* sphinxcontrib-pecanwsme
* sqlalchemy-migrate
* swift
* thrift (which is blocking happybase)
||
|-

| [https://pypi.python.org/pypi/cinder cinder] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* ecdsa (which is blocking paramiko)
* eventlet
* hacking
* mysql-python
* oslo.config
* oslo.rootwrap
* oslo.sphinx
* paste
* python-glanceclient
* python-swiftclient
* rtslib-fb
* sqlalchemy-migrate
* suds
* taskflow
||
|-

| [https://pypi.python.org/pypi/glance glance] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* boto
* eventlet
* hacking
* mysql-python
* oslo.config
* oslo.messaging
* oslosphinx
* paste
* python-cinderclient
* python-swiftclient
* qpid-python
* sqlalchemy-migrate
||
|-

| [https://pypi.python.org/pypi/heat heat] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* ecdsa (which is blocking paramiko)
* eventlet
* hacking
* oslo.config
* oslo.sphinx
* python-ceilometerclient
* python-cinderclient
* python-glanceclient
* python-heatclient
* python-neutronclient
* python-swiftclient
* python-troveclient
* qpid-python
* sqlalchemy-migrate
||
|-

| [https://pypi.python.org/pypi/horizon horizon] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* django-compressor
* django-openstack-auth
* eventlet
* hacking
* nose-exclude
* nosehtmloutput
* nosexcover
* openstack.nose-plugin
* oslo.sphinx
* python-ceilometerclient
* python-cinderclient
* python-glanceclient
* python-heatclient
* python-neutronclient
* python-swiftclient
* python-troveclient
||
|-

| [https://pypi.python.org/pypi/keystone keystone] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* hacking
* oslo.config
* oslo.messaging
* oslosphinx
* pam
* paste
* pycadf
* python-ldap
* sqlalchemy-migrate
||
|-

| [https://pypi.python.org/pypi/neutron neutron] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* eventlet
* hacking
* jsonrpclib
* oslo.config
* oslo.rootwrap
* paste
* python-neutronclient
||
|-

| [https://pypi.python.org/pypi/nova nova] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* boto
* ecdsa (which is blocking paramiko)
* eventlet
* hacking
* mysql-python
* oslo.config
* oslo.messaging
* oslo.rootwrap
* oslosphinx
* paste
* pycadf
* python-cinderclient
* python-glanceclient
* python-neutronclient
* sqlalchemy-migrate
* suds
* websockify
||
|-

| [https://pypi.python.org/pypi/swift swift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No ||
* dnspython
* eventlet
* hacking
* netifaces
* nosehtmloutput
* nosexcover
* openstack.nose-plugin
* python-swiftclient
||
|-

|}

=== Dependencies ===
{|class="wikitable sortable"
|-
! Project !! Python 3 compatibility !! CI tests running? !! Python 3 classifiers ? !! Blocked by !! Comment
|-

| [https://pypi.python.org/pypi/boto boto] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/django-compressor django-compressor] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://github.com/django-compressor/django-compressor/issues/484
|-

| [https://pypi.python.org/pypi/django-openstack-auth django-openstack-auth] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/dnspython dnspython] || style="background-color:lightgreen;" | Yes || N/A|| style="background-color: lightgreen;" | Yes || || Must use the [https://pypi.python.org/pypi/dnspython3/ Python 3 version], see https://github.com/rthalley/dnspython/issues/60
|-

| [https://pypi.python.org/pypi/ecdsa ecdsa] || style="background-color:lightgreen;" | No || N/A || style="background-color: orange;" | In the Git repo || ||Py3 support merge before the 0.10 release (see https://github.com/warner/python-ecdsa/commits/master)
|-

| [https://pypi.python.org/pypi/eventlet eventlet] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/hacking hacking] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/jsonrpclib jsonrpclib] || style="background-color:red;" | No || N/A || style="background-color: red;" | No || || The project seems dead :(
|-

| [https://pypi.python.org/pypi/mysql-python mysql-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/netifaces netifaces] || style="background-color:red;" | No || N/A|| style="background-color: red;" | No || || Patch sent by Victor Stinner (in private), Debian has patches too
|-

| [https://pypi.python.org/pypi/nose-exclude nose-exclude] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/kgrandis/nose-exclude/issue/10/test-failures-with-python-3
|-

| [https://pypi.python.org/pypi/nosehtmloutput nosehtmloutput] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
https://bugs.launchpad.net/ubuntu/+source/python-nosehtmloutput/+bug/1287247
|-
| [https://pypi.python.org/pypi/nosexcover nosexcover] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || There have been Python 3 fixes, but there are no tests, so it's hard to say whether this works
|-
| [https://pypi.python.org/pypi/openstack.nose-plugin openstack.nose-plugin] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.vmware oslo.vmware] || style="background-color:red;" | No || style="background-color: lightgreen;" | Voting || style="background-color: lightgreen;" | Yes || suds ||
|-
| [https://pypi.python.org/pypi/oslo.config oslo.config] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Voting || style="background-color: orange;" | In the git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/oslo.messaging oslo.messaging] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.rootwrap oslo.rootwrap] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | https://review.openstack.org/#/c/77652/ || ||
|-
| [https://pypi.python.org/pypi/oslosphinx oslosphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/oslo.sphinx oslo.sphinx] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/pam pam] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || Dead project, does not work with Py3
|-

| [https://pypi.python.org/pypi/paste paste] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || https://bitbucket.org/ianb/paste/pull-request/9/python-3-support/diff
|-

| [https://pypi.python.org/pypi/pycadf pycadf] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

| [https://pypi.python.org/pypi/python-ldap python-ldap] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || || The project seems dead.
|-

| [https://pypi.python.org/pypi/qpid-python qpid-python] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/rtslib-fb rtslib-fb] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-docbookrestapi sphinxcontrib-docbookrestapi] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-httpdomain sphinxcontrib-httpdomain] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sphinxcontrib-pecanwsme sphinxcontrib-pecanwsme] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/sqlalchemy-migrate sqlalchemy-migrate] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/suds suds] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/taskflow taskflow] || style="background-color:lightgreen;" | Yes || style="background-color: lightgreen;" | Yes || style="background-color: orange;" | In the Git repo, not on PyPI || ||
|-
| [https://pypi.python.org/pypi/thrift thrift] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-
| [https://pypi.python.org/pypi/websockify websockify] || style="background-color:red;" | No || style="background-color: red;" | No || style="background-color: red;" | No || ||
|-

|}

== Reports at OpenStack Summits ==

* Havana summit notes: https://etherpad.openstack.org/p/havana-python3
* Icehouse summit notes: https://etherpad.openstack.org/p/IcehousePypyPy3

== Dependencies ==

This is an attempt to document which OpenStack dependencies work under Python 3. We're starting with oslo, since it's the base of everything else, then gradually including other projects.

The [https://pypi.python.org/pypi/caniusepython3 caniusepython3] tool can be used to do quick yes/no Python 3 support checks.

=== oslo.config ===

==== pip-requires ====
* argparse - included with Python 2.7+. The separate package is only needed for 2.6.

==== test-requires ====
* mox - mox3 supports Python, or replace mox with mock (which is now included in python 3.3)
* nose - supports Python 3
* nose-exclude - 2.6-2.7, 3.1-3.3
* testtools - 2.6-2.7, 3.2-3.3
* coverage - supports 2.3-3.3
* sphinx - supports Python 3

=== oslo-incubator ===

==== pip-requires ====

* PasteDeploy: supports 2.5-3.3
* WebOb: support Python 3
* eventlet: NO (MAJOR PAIN POINT) (gevent doesn't either, though there are some old forks that tried)
* greenlet: supports 2.4-3.2 (lack of 3.3 may be false negative)
* lxml: supports 2.4-3.3
* routes: supports 2.6-3.3
* iso8601: NO (perhaps try [http://labix.org/python-dateutil python-dateutil], specifically the parser module?)
* anyjson: 2.4-3.1 (lack of 3.2-3.3 may be false negatives)
* kombu: supports Python 3 (exact version not given)
* argparse: included in Python 2.7+
* stevedore: supports 2.7, 3.2, and 3.3
* SQLAlchemy: supports Python 3 (exact version not given)
* qpid-python: NO

==== test-requires ====

* distribute: 2.4-3.3
* coverage: 2.3-3.3
* fixtures: supports Python 3 (exact version not given)
* mock: 2.5-3.3
* mox: use mox3, or replace mox with mock (which is now included in python 3.3)
* mysql-python: NO (maybe try pymysql instead?)
* nose: Supports Python 3
* nose-exclude: 2.6-2.7, 3.1-3.3
* nosexcover: NO (bug in tracker from 2012-05; no comments)
* nosehtmloutput: ?
* pep8: Supports Python 3
* pyflakes: Supports Python 3
* pylint: Supports Python 3 (tested with Python 3.2)
* pyzmq: Supports 2.6-2.7, 3.2+
* redis: Supports 2.5-2.7, 3.2+
* setuptools-git: 2.4-2.7, 3.1-3.3
* sphinx: Supports Python 3
* testtools: 2.6-2.7, 3.2-3.3
* webtest: 2.6-2.7, 3.2-3.3

GSoC2014

2014-03-01T01:49:41Z

Arnaud Legendre: /* Mentors */

== Google Summer of Code 2014 ==

[http://www.openstack.org/software/ OpenStack] is applying as a mentoring organization for the [http://www.google-melange.com/gsoc/homepage/google/gsoc2014 2014 Google Summer of Code]. Check [https://developers.google.com/open-source/soc/ Google Developers site] for more information on how the program works.
We need to get in this time!

Link to call for participation: [http://google-opensource.blogspot.com/2014/02/mentoring-organization-applications-now.html here]

Link to FAQs: [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page here]

=== What is Openstack? ===

[http://www.openstack.org/software/ Openstack] is an open-source IaaS cloud computing platform. Its mission is to provide a flexible solution for both public and private clouds of any size, and for this matter two basic requirements are considered: clouds must be simple to implement and massively scalable.

To meet these principles OpenStack is divided into different components that work together. It's [http://www.openstack.org/software/openstack-compute/ computing], [http://www.openstack.org/software/openstack-storage/ storage], [http://www.openstack.org/software/openstack-networking/ networking], and all the other bits that help make this project, '''The Cloud'''.

OpenStack is [http://i.imgur.com/gAyoiF8.png continuously growing] and new and exciting projects are being discussed everyday.

We encourage new contributors to participate and help us make OpenStack the most complete, reliable and flexible open-source cloud service!

=== Mentors ===

If you're interested in mentoring for this round, please add your name, email, IRC handle and the kind of projects you're interested in mentoring below. Please ensure that the projects are official projects in OpenStack and are registered in the [https://github.com/openstack/governance/blob/master/reference/programs.yaml governance projects.yaml]

* Davanum Srinivas - dims - Nova, Oslo
* Debo Dutta - dedutta - Nova, Ceilometer
* [[User:Alejandro_Cabrera|Alejandro Cabrera]] - alcabrera - Marconi, Oslo
* Yathiraj Udupi - Yathi - Nova, Ceilometer
* Fei Long Wang - flwang- Glance
* Boris Pavlovic - boris-42 - Rally
* [[User:Sergey_Lukjanov|Sergey Lukjanov]] - SergeyLukjanov - Savanna (proxy to help find mentor in Savanna community)
* [[User:colinmcnamara|Colin McNamara]] - colinmcnamara - OpenStack Docs, OpenStack Training and ToolChains
* [[User:sriramhere|Sriram Subramanian]] - Fuzz Testing, OpenStack Security
* Balint Kovacs - blint@balabit.hu - Zorp
* Szilard Pfeiffer - floss@pfeifferszilard.hu - Zorp
* Arnaud Legendre - arnaud, alegendre@vmware.com - Oslo
* Joshua Hesketh - jhesketh, joshua.hesketh@rackspace.com - OpenStack Infrastructure

=== Students ===

Students application period opens March 10 and ends on March 21.

If you'd like to get started on your proposal early, go ahead and add your name, location, e-mail, IRC handle and the project you are interested in (if you already know about that!) here:
*Manishanker Talusani,India,shanker.mani0@gmail.com[Project not selected yet]
*Saket Sinha, India, saket.sinha89@gmail.com [project not selected yet]
* Adnan Khan, Canada, khnd06@gmail.com [project not selected yet]
* Anastasios Andronidis, Greece, andronat_asf@hotmail.com. Proposed: [https://docs.google.com/document/d/1mWWSyftZYXxfXKzOBVeVma8KJbQRmYNPnns3oixHVMk/edit?usp=sharing Glance Scalable Image Precaching]
* Artem Shepelev, Russia, e-mail: shepelev.artem@gmail.com, ashepelev at irc.freenode.net, [[GSoC2014/Scheduler/CrossServicesScheduler|OpenStack/Gantt Cross-services Scheduler]].
* Fabio Morais, Brazil, fabio.jorge@gmail.com [Ceilometer; Proposed: [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]]
* [[User:Vkmc|Victoria Martínez de la Cruz]] - Argentina - victoria@vmartinezdelacruz.com - vkmc - OpenStack Message Queuing (Marconi)
* [[User:Damon_Wang|Wei Wang]], China, damon.devops@gmail.com, Neutron & Keystone
* Kumar Rishabh, India, email: shailrishabh@gmail.com [would fill more details later]
* Marc Solanas Tarre, US, email: marc@solanas.cat, mst89, [Monitoring and Telemetry how to detect network anomalies from telemetry data within Openstack]
* Pengfei Zhang, US, lalasjtu@gmail.com, Sparky, [Monitoring and Telemetry Monitoring & Tuning network for QoS within Openstack]
* RobberPhex, China, robberphex@gmail.com, Rally
* Rodrigo Duarte, Brazil, rodrigodsousa@gmail.com, rodrigods at irc.freenode.net [Gantt]
* Rishi Raj Singh, India, rishiraj.devel@gmail.com, [project not selected yet]
* [[User:Telles Nobrega|Telles Nóbrega]], Brazil, tellesnobrega@gmail.com, tellesnobrega at irc.freenode.net [Keystone, Nova, Savanna, Ceilometer]
* Md Ali Ahsan Rana, Canada, aliahsanrana@gmail.com, [Oslo, Rally]
* George Ebbinason, India, ebbinason@hotmail.com, [OpenStack Networking (Neutron) - Implement an application-level FWaaS driver]
* Masaru Nomura, UK, massa.nomura@gmail.com, [OpenStack Incubator (Oslo) - Implement a re-usable shared library for vmware (oslo.vmware)]
* Santosh Iyer, US, mails2santosh@gmail.com, [Nova, Ceilometer, Savanna]
* Chenchong Qin, China, qinchenchong@gmail.com, Neutron
* Andrew Chul, Russia, andymitrich@gmail.com, [project not selected yet]
* Fang Zhen, China, fz1989fz@gmail.com,[Gantt, Ceilometer]
* Abhinav Saxena, India, abhinav.saxena.57@gmail.com [project not yet selected]
* Pranav Singh, India, singh.pranavkumar10@gmail.com, purple_haze, [project not yet selected]
* Demontiê dos Santos, Brazil, demontiejunior@gmail.com, dsantos_ at irc.freenode.net, [Nova]
* [[User:Danielbrunos|Daniel Bruno]], Brazil, danielbrunos@gmail.com, danielbruno, [Heat, Savanna, Nova, Neutron]
* Dániel Csubák, Hungary, cyrrian@gmail.com, [OpenStack Networking (Neutron) - Implement an application-level FWaaS driver]
* Bruno Criado, Brazil, brunocriado@gmail.com, dropped at irc.freenode.net, [Implement a Fuzz testing framework that can be run on Tempest or a similar framework]
* Ondra Machacek, Czech Republic machacek.ondra@gmail.com, omachace [infra]

=== Communication ===

Get in touch with mentors and students through the [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev openstack-dev mailing list].

Also, you can find us at IRC in #openstack-gsoc at irc.freenode.org.

=== Ideas ===

Here are some project suggestions students can choose for their applications. This doesn't mean students have to stick strictly to this list; don't hesitate in propose projects by your own. For the latter, it would be great if you could submit a draft proposal and a estimated timeline.

When writing your proposal, try to estimate your timeline to fit the 4 month period of GSoC coding. Also, take into account that '''GSoC does not consider other projects than coding''', so other ideas (like community tasks or i18n efforts) are not suitable for this internship.

'''Click on the proposed project that caught your attention to get the details about the assumed knowledge, the project goals and more details related to it.'''

==== Fuzz Testing (various projects) ====

Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is commonly used to test for security problems in software or computer systems.

{| class="wikitable"
|-
| Difficulty || Hard
|-
| Topics || testing, tempest
|-
| Mentor || Sriram Subramanian
|}

Proposed ideas,

* [[GSoC2014/Testing/Fuzz|Implement a Fuzz testing framework that can be run on Tempest or a similar framework]]

==== Common Scheduler (Gantt) ====

Gantt provides a common scheduler framework for use by various OpenStack components.

Check out more details about [https://github.com/openstack/gantt Gantt]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || schedulers, python, gantt
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/StandAloneScheduler|Implement a stand-alone scheduler based on the scheduler forklift code]]
* [[GSoC2014/Scheduler/ScalableScheduler|Implement a scalable scheduler]]
* [[GSoC2014/Scheduler/CrossServicesScheduler|Implement a cross-services scheduler]]

==== OpenStack Monitoring and Telemetry (Ceilometer) ====

Ceilometer aims to deliver a unique point of contact for billing systems to acquire all counters they need to establish customer billing, across all current and future OpenStack components. The delivery of counters must be traceable and auditable. The counters must be easily extensible to support new projects, and agents doing data collections should be independent of the overall system.

For more details about Ceilometer project, check out the [[Ceilometer#OpenStack_Telemetry_.28Ceilometer.29|wiki]]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || ceilometer, data science
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/NetworkAnomalyDetection|How to detect network anomalies from telemetry data within Openstack]]
* [[Monitoring & Tuning network for QoS within Openstack]]
* [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]

==== OpenStack Message Queuing Service (Marconi) ====

Openstack Message Queuing Service (Marconi) provides a distributed queue. The basic concept is simple:

* Create a queue
* Post messages
* Read them or claim them

For full details on Marconi project, check out the [https://wiki.openstack.org/wiki/Marconi wiki]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || storage, python, marconi
|-
| Mentor || Alejandro Cabrera
|}

Proposed ideas,

* [[GSoC2014/Queues/Storage | Add a New Storage Backend]]

==== OpenStack Common Libraries (Oslo) ====

The Oslo program produces a set of python libraries containing infrastructure code shared by OpenStack projects. The APIs provided by these libraries should be high quality, stable, consistent and generally useful.

Check out more details about Oslo project on the [https://wiki.openstack.org/wiki/oslo wiki]

{| class="wikitable"
|-
| Difficulty || Medium - Hard
|-
| Topics || storage, python, oslo
|-
| Mentor || Alejandro Cabrera, Davanum Srinivas
|}

Proposed ideas,

* [[GSoC2014/Incubator/Storage|Add a new backend to oslo.cache]]
* [[GSoC2014/Incubator/SharedLib|Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects like Nova, Cinder or Glance]]
* [[GSoC2014/Incubator/Plugin|Define a new layer/abstraction in Nova for plugging-in vCenter and ovirt (since they span multiple hosts)]] (WIP)

==== Benchmarking System (Rally) ====

OpenStack QA team mostly works on CI/CD that ensures that new patches don't break specific single node installation of OpenStack. On the other hand it's clear that such CI/CD is only an indication and does not cover all cases (e.g. if cloud works well on single node installation it doesn't mean that it will work good as well on 1k servers installation under high load).. Rally aims to fix this and help us to get answer on question "How OpenStack works at scale". To make it possible we are going to automate and unify all steps that are required for benchmarking OpenStack at scale: multi node OS deployment, verification, benchmarking & profiling.

[[File:Rally-Actions.png|500px|center]]

* Deploy is not yet another deployer of OpenStack it is just a plugable mechanism that allows to unify & simplify work with different deployers like: DevStack, Fuel, Anvil on hardware/VMs that you have.
* Verify - (work in progress) Use tempest to verify functionality of deployed openstack. In future Rally will support other OS verifiers.
* Benchmark - Smart combination of Framework, Load generation & Big repository of benchmarks.

For more details about Rally project, check out [https://wiki.openstack.org/wiki/rally wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || rally, benchmarks, deploying, python, tempest,
|-
| Mentor || Boris Pavlovic
|}

Proposed ideas,

* Benchmarking OpenStack
* Writing new benchmarks
* Integration of Rally & Tempest
* Processing Results
* Improving Rally deploying system

==== OpenStack Networking (Neutron) ====

Neutron is an OpenStack project to provide Networking-as-a-Service between interface devices (e.g., vNICs) managed by other Openstack services (e.g., Nova).

For full details about Neutron, check out the [https://wiki.openstack.org/wiki/Neutron wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || drivers, python, networking, fwaas, neutron
|-
| Mentor || Balint Kovacs, Szilard Pfeiffer
|}

Proposed ideas,

* [[GSoC2014/Network/Driver|Implement an application-level FWaaS driver]] ([https://github.com/balabit/zorp/wiki/zorp Zorp])

==== OpenStack Data Processing (Savanna) ====

Savanna aims to provide users with simple means to provision a Hadoop cluster by specifying several parameters like Hadoop version, cluster topology, nodes hardware details and a few more. The aim of this project is to enable users provision and management of Hadoop clusters on OpenStack.

Check out the [https://wiki.openstack.org/wiki/Savanna wiki] to learn more about Savanna

{| class="wikitable"
|-
| Difficulty ||
|-
| Topics || plugins, hadoop provision, savanna
|-
| Mentor ||
|}

Proposed ideas,

* [[GSoC2014/DataProcessing/Plugin|Develop a new plugin for Savanna to extend savanna.plugins.provisioning:ProvisioningPluginBase class and implement all the required methods.]] (WIP)

==== OpenStack Infrastructure (infra) ====

The project infrastructure encompasses all of the systems that are used in the day to day operation of the OpenStack project as a whole. This includes development, testing, and collaboration tools. All of the software that we run is open source, and its configuration is public. The project still uses a number of systems that do not yet fall under this umbrella (notably, the main website), but we’re working to incorporate them so that people may just as easily contribute to those areas. All new services used by the project should begin as part of the infrastructure project to ensure easy collaboration from the start.

Proposed ideas,

* [[GSoC2014/ReplaceJenkins|Replace Jenkins with a more scalable solution based on the zuul-gearman protocol]]

=== Projects ===

Current mentors are willing to supervise students in the following projects:

* [https://wiki.openstack.org/wiki/Nova Openstack Compute (Nova)]
* [https://wiki.openstack.org/wiki/Climate Resource Reservation (Climate)]
* [https://github.com/openstack/gantt Common Scheduler (Gantt)]
* [https://wiki.openstack.org/wiki/Marconi OpenStack Message Queuing (Marconi)]
* [https://wiki.openstack.org/wiki/Oslo OpenStack Common Libraries (Oslo)]
* [https://wiki.openstack.org/wiki/Neutron OpenStack Networking (Neutron)]
* [https://wiki.openstack.org/wiki/Swift OpenStack Object Storage (Swift)]
* [https://wiki.openstack.org/wiki/Cinder OpenStack Block Storage (Cinder)]
* [https://wiki.openstack.org/wiki/Heat OpenStack Orchestration (Heat)]
* [https://wiki.openstack.org/wiki/Savanna OpenStack Data Processing (Savanna)]
* [https://wiki.openstack.org/wiki/Ceilometer OpenStack Telemetry (Ceilometer)]
* [https://wiki.openstack.org/wiki/Rally Benchmarking System (Rally) ]

GSoC2014

2014-02-25T22:25:58Z

Arnaud Legendre: /* Mentors */

== Google Summer of Code 2014 ==

[http://www.openstack.org/software/ OpenStack] is applying as a mentoring organization for the [http://www.google-melange.com/gsoc/homepage/google/gsoc2014 2014 Google Summer of Code]. Check [https://developers.google.com/open-source/soc/ Google Developers site] for more information on how the program works.
We need to get in this time!

Link to call for participation: [http://google-opensource.blogspot.com/2014/02/mentoring-organization-applications-now.html here]

Link to FAQs: [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page here]

=== What is Openstack? ===

[http://www.openstack.org/software/ Openstack] is an open-source IaaS cloud computing platform. Its mission is to provide a flexible solution for both public and private clouds of any size, and for this matter two basic requirements are considered: clouds must be simple to implement and massively scalable.

To meet these principles OpenStack is divided into different components that work together. It's [http://www.openstack.org/software/openstack-compute/ computing], [http://www.openstack.org/software/openstack-storage/ storage], [http://www.openstack.org/software/openstack-networking/ networking], and all the other bits that help make this project, '''The Cloud'''.

OpenStack is [http://i.imgur.com/gAyoiF8.png continuously growing] and new and exciting projects are being discussed everyday.

We encourage new contributors to participate and help us make OpenStack the most complete, reliable and flexible open-source cloud service!

=== Mentors ===

If you're interested in mentoring for this round, please add your name, email, IRC handle and the kind of projects you're interested in mentoring below. Please ensure that the projects are official projects in OpenStack and are registered in the [https://github.com/openstack/governance/blob/master/reference/programs.yaml governance projects.yaml]

* Davanum Srinivas - dims - Nova, Oslo
* Debo Dutta - dedutta - Nova, Ceilometer
* [[User:Alejandro_Cabrera|Alejandro Cabrera]] - alcabrera - Marconi, Oslo
* Yathiraj Udupi - Yathi - Nova, Ceilometer
* Fei Long Wang - flwang- Glance
* Boris Pavlovic - boris-42 - Rally
* [[User:Sergey_Lukjanov|Sergey Lukjanov]] - SergeyLukjanov - Savanna (proxy to help find mentor in Savanna community)
* [[User:colinmcnamara|Colin McNamara]] - colinmcnamara - OpenStack Docs, OpenStack Training and ToolChains
* [[User:sriramhere|Sriram Subramanian]] - Fuzz Testing, OpenStack Security
* Balint Kovacs - blint@balabit.hu - Zorp
* Szilard Pfeiffer - floss@pfeifferszilard.hu - Zorp
* Arnaud Legendre - arnaud - Oslo

=== Students ===

Students application period opens March 10 and ends on March 21.

If you'd like to get started on your proposal early, go ahead and add your name, location, e-mail, IRC handle and the project you are interested in (if you already know about that!) here:
*Manishanker Talusani,India,shanker.mani0@gmail.com[Project not selected yet]
*Saket Sinha, India, saket.sinha89@gmail.com [project not selected yet]
* Adnan Khan, Canada, khnd06@gmail.com [project not selected yet]
* Anastasios Andronidis, Greece, andronat_asf@hotmail.com. Proposed: [https://docs.google.com/document/d/1mWWSyftZYXxfXKzOBVeVma8KJbQRmYNPnns3oixHVMk/edit?usp=sharing Glance Scalable Image Precaching]
* Artem Shepelev, Russia, e-mail: shepelev.artem@gmail.com, ashepelev at irc.freenode.net, OpenStack Scheduler.
* Fabio Morais, Brazil, fabio.jorge@gmail.com [Ceilometer; Proposed: [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]]
* Victoria Martínez de la Cruz - Argentina - victoria@vmartinezdelacruz.com - vkmc - OpenStack Message Queuing (Marconi)
* [[User:Damon_Wang|Wei Wang]], China, damon.devops@gmail.com, Neutron & Keystone
* Kumar Rishabh, India, email: shailrishabh@gmail.com [would fill more details later]
* Marc Solanas Tarre, US, email: marc@solanas.cat, mst89, [Monitoring and Telemetry how to detect network anomalies from telemetry data within Openstack]
* Pengfei Zhang, US, lalasjtu@gmail.com, Sparky, [Monitoring and Telemetry Monitoring & Tuning network for QoS within Openstack]
* RobberPhex, China, robberphex@gmail.com, [project not selected yet]
* Rodrigo Duarte, Brazil, rodrigodsousa@gmail.com, OpenStack Scheduler (Gantt)
* Rishi Raj Singh, India, rishiraj.devel@gmail.com, [project not selected yet]
* [[User:Telles Nobrega|Telles Nóbrega]], Brazil, tellesnobrega@gmail.com, tellesnobrega at irc.freenode.net [Keystone, Nova, Savanna, Ceilometer]
* Md Ali Ahsan Rana, Canada, aliahsanrana@gmail.com, [Oslo, Rally]
* George Ebbinason, India, ebbinason@hotmail.com, [OpenStack Networking (Neutron) - Implement an application-level FWaaS driver]
* Masaru Nomura, UK, massa.nomura@gmail.com, [OpenStack Incubator (Oslo) - Implement a re-usable shared library for vmware (oslo.vmware)]
* Santosh Iyer, US, mails2santosh@gmail.com, [Nova, Ceilometer, Savanna]

=== Communication ===

Get in touch with mentors and students through the [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev openstack-dev mailing list].

Also, you can find us at IRC in #openstack-gsoc at irc.freenode.org.

=== Ideas ===

Here are some project suggestions students can choose for their applications. This doesn't mean students have to stick strictly to this list; don't hesitate in propose projects by your own. For the latter, it would be great if you could submit a draft proposal and a estimated timeline.

When writing your proposal, try to estimate your timeline to fit the 4 month period of GSoC coding. Also, take into account that '''GSoC does not consider other projects than coding''', so other ideas (like community tasks or i18n efforts) are not suitable for this internship.

'''Click on the proposed project that caught your attention to get the details about the assumed knowledge, the project goals and more details related to it.'''

==== Fuzz Testing (various projects) ====

Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is commonly used to test for security problems in software or computer systems.

{| class="wikitable"
|-
| Difficulty || Hard
|-
| Topics || testing, tempest
|-
| Mentor || Sriram Subramanian
|}

Proposed ideas,

* [[GSoC2014/Testing/Fuzz|Implement a Fuzz testing framework that can be run on Tempest or a similar framework]]

==== OpenStack Scheduler (Gantt) ====

Gantt provides a common scheduler framework for use by various OpenStack components.

Check out more details about [https://github.com/openstack/gantt Gantt]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || schedulers, python, gantt
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/StandAloneScheduler|Implement a stand-alone scheduler based on the scheduler forklift code]]
* [[GSoC2014/Scheduler/ScalableScheduler|Implement a scalable scheduler]]
* [[GSoC2014/Scheduler/CrossServicesScheduler|Implement a cross-services scheduler]]

==== OpenStack Monitoring and Telemetry (Ceilometer) ====

Ceilometer aims to deliver a unique point of contact for billing systems to acquire all counters they need to establish customer billing, across all current and future OpenStack components. The delivery of counters must be traceable and auditable. The counters must be easily extensible to support new projects, and agents doing data collections should be independent of the overall system.

For more details about Ceilometer project, check out the [[Ceilometer#OpenStack_Telemetry_.28Ceilometer.29|wiki]]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || ceilometer, data science
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/NetworkAnomalyDetection|How to detect network anomalies from telemetry data within Openstack]]
* [[Monitoring & Tuning network for QoS within Openstack]]
* [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]

==== OpenStack Message Queuing Service (Marconi) ====

Openstack Message Queuing Service (Marconi) provides a distributed queue. The basic concept is simple:

* Create a queue
* Post messages
* Read them or claim them

For full details on Marconi project, check out the [https://wiki.openstack.org/wiki/Marconi wiki]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || storage, python, marconi
|-
| Mentor || Alejandro Cabrera
|}

Proposed ideas,

* [[GSoC2014/Queues/Storage | Add a New Storage Backend]]

==== OpenStack Incubator (Oslo) ====

The Oslo program produces a set of python libraries containing infrastructure code shared by OpenStack projects. The APIs provided by these libraries should be high quality, stable, consistent and generally useful.

Check out more details about Oslo project on the [https://wiki.openstack.org/wiki/oslo wiki]

{| class="wikitable"
|-
| Difficulty || Medium - Hard
|-
| Topics || storage, python, oslo
|-
| Mentor || Alejandro Cabrera, Davanum Srinivas
|}

Proposed ideas,

* [[GSoC2014/Incubator/Storage|Add a new backend to oslo.cache]]
* [[GSoC2014/Incubator/SharedLib|Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects like Nova, Cinder or Glance]]
* [[GSoC2014/Incubator/Plugin|Define a new layer/abstraction in Nova for plugging-in vCenter and ovirt (since they span multiple hosts)]] (WIP)

==== OpenStack Benchmarking System (Rally) ====

OpenStack QA team mostly works on CI/CD that ensures that new patches don't break specific single node installation of OpenStack. On the other hand it's clear that such CI/CD is only an indication and does not cover all cases (e.g. if cloud works well on single node installation it doesn't mean that it will work good as well on 1k servers installation under high load).. Rally aims to fix this and help us to get answer on question "How OpenStack works at scale". To make it possible we are going to automate and unify all steps that are required for benchmarking OpenStack at scale: multi node OS deployment, verification, benchmarking & profiling.

[[File:Rally-Actions.png|500px|center]]

* Deploy is not yet another deployer of OpenStack it is just a plugable mechanism that allows to unify & simplify work with different deployers like: DevStack, Fuel, Anvil on hardware/VMs that you have.
* Verify - (work in progress) Use tempest to verify functionality of deployed openstack. In future Rally will support other OS verifiers.
* Benchmark - Smart combination of Framework, Load generation & Big repository of benchmarks.

For more details about Rally project, check out [https://wiki.openstack.org/wiki/rally wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || rally, benchmarks, deploying, python, tempest,
|-
| Mentor || Boris Pavlovic
|}

Proposed ideas,

* Benchmarking OpenStack
* Writing new benchmarks
* Integration of Rally & Tempest
* Processing Results
* Improving Rally deploying system

==== OpenStack Networking (Neutron) ====

Neutron is an OpenStack project to provide Networking-as-a-Service between interface devices (e.g., vNICs) managed by other Openstack services (e.g., Nova).

For full details about Neutron, check out the [https://wiki.openstack.org/wiki/Neutron wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || drivers, python, networking, fwaas, neutron
|-
| Mentor || Balint Kovacs, Szilard Pfeiffer
|}

Proposed ideas,

* [[GSoC2014/Network/Driver|Implement an application-level FWaaS driver]] ([https://github.com/balabit/zorp/wiki/zorp Zorp])

==== OpenStack Data Processing (Savanna) ====

Savanna aims to provide users with simple means to provision a Hadoop cluster by specifying several parameters like Hadoop version, cluster topology, nodes hardware details and a few more. The aim of this project is to enable users provision and management of Hadoop clusters on OpenStack.

Check out the [https://wiki.openstack.org/wiki/Savanna wiki] to learn more about Savanna

{| class="wikitable"
|-
| Difficulty ||
|-
| Topics || plugins, hadoop provision, savanna
|-
| Mentor ||
|}

Proposed ideas,

* [[GSoC2014/DataProcessing/Plugin|Develop a new plugin for Savanna to extend savanna.plugins.provisioning:ProvisioningPluginBase class and implement all the required methods.]] (WIP)

=== Projects ===

Current mentors are willing to supervise students in the following projects:

* [https://wiki.openstack.org/wiki/Nova Openstack Compute (Nova)]
* [https://wiki.openstack.org/wiki/Climate OpenStack Reservation (Climate)]
* [https://github.com/openstack/gantt OpenStack Common Scheduler (Gantt)]
* [https://wiki.openstack.org/wiki/Marconi OpenStack Message Queuing (Marconi)]
* [https://wiki.openstack.org/wiki/Oslo OpenStack Common Libraries (Oslo)]
* [https://wiki.openstack.org/wiki/Neutron OpenStack Networking (Neutron)]
* [https://wiki.openstack.org/wiki/Swift OpenStack Object Storage (Swift)]
* [https://wiki.openstack.org/wiki/Cinder OpenStack Block Storage (Cinder)]
* [https://wiki.openstack.org/wiki/Heat OpenStack Orchestration (Heat)]
* [https://wiki.openstack.org/wiki/Savanna OpenStack Data Processing (Savanna)]
* [https://wiki.openstack.org/wiki/Ceilometer OpenStack Telemetry (Ceilometer)]
* [https://wiki.openstack.org/wiki/Rally OpenStack Benchmarking System (Rally) ]

GSoC2014/Incubator/SharedLib

2014-02-25T04:19:11Z

Arnaud Legendre: /* Project Goals */

= Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects =

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || VMware API, Python, Nova/Glance/Cinder
|-
| Mentor || Arnaud Legendre
|}

The VMware API is consumed by OpenStack using the oslo.vmware library [1]. This library recently created specifically for OpenStack is shared by several projects such as Nova, Glance and Cinder. Internally, it is using SOAP calls to communicate with vCenter server and ESXi.
Instead of consuming the API through SOAP calls, it would be possible to use PyVmomi which is a Python SDK for the VMware vSphere API [2]. This work will give the opportunity to redefine a clean and more performant API.
It involves:

# Deep dive the current API and how it is consumed by the OpenStack services
# Deep dive in PyVmomi
# Define an API
# Provide an implementation for this API using PyVmomi
# Make sure existing tests are working or modify them accordingly

[1] https://github.com/openstack/oslo.vmware/

[2] https://github.com/vmware/pyvmomi

== Assumed Knowledge ==
* Python - basics, class/module management
* Command Line - a little bit of git, code editing, navigation
* Already played with virtual machines
* Experience with API definition
* Some basic knowledge of what VMware does would be a plus

Refer to [https://github.com/cabrera/python-openstack-and-you/ Python, Openstack, and You] if you'd like to get a head start.
I will be able to help you!!

== Project Goals ==

* Provide a clean interface for the OpenStack VMware library
* Provide an implementation for it using PyVmomi
* Provide a test suite

== Project Nice-to-Haves ==

* Performance evaluation of PyVmomi compared to the current codebase
* Provide some documentation for the API

====== Suggestions ======

* vCenter and ESX:

vCenter server: http://www.vmware.com/products/vcenter-server

VMware ESX: http://en.wikipedia.org/wiki/VMware_ESX

* How to write APIs:

https://ep2013.europython.eu/conference/talks/guidelines-to-writing-a-python-api

* OpenStack:

Cinder: https://wiki.openstack.org/wiki/Cinder

Glance: http://docs.openstack.org/developer/glance/

Nova: http://docs.openstack.org/developer/nova/

GSoC2014/Incubator/SharedLib

2014-02-25T04:18:54Z

Arnaud Legendre: /* Project Goals */

= Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects =

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || VMware API, Python, Nova/Glance/Cinder
|-
| Mentor || Arnaud Legendre
|}

The VMware API is consumed by OpenStack using the oslo.vmware library [1]. This library recently created specifically for OpenStack is shared by several projects such as Nova, Glance and Cinder. Internally, it is using SOAP calls to communicate with vCenter server and ESXi.
Instead of consuming the API through SOAP calls, it would be possible to use PyVmomi which is a Python SDK for the VMware vSphere API [2]. This work will give the opportunity to redefine a clean and more performant API.
It involves:

# Deep dive the current API and how it is consumed by the OpenStack services
# Deep dive in PyVmomi
# Define an API
# Provide an implementation for this API using PyVmomi
# Make sure existing tests are working or modify them accordingly

[1] https://github.com/openstack/oslo.vmware/

[2] https://github.com/vmware/pyvmomi

== Assumed Knowledge ==
* Python - basics, class/module management
* Command Line - a little bit of git, code editing, navigation
* Already played with virtual machines
* Experience with API definition
* Some basic knowledge of what VMware does would be a plus

Refer to [https://github.com/cabrera/python-openstack-and-you/ Python, Openstack, and You] if you'd like to get a head start.
I will be able to help you!!

== Project Goals ==

* Provide a clean interface for the OpenStack VMware library
* Provide an implementation for it using PyVmomi
* Provide a Test suite

== Project Nice-to-Haves ==

* Performance evaluation of PyVmomi compared to the current codebase
* Provide some documentation for the API

====== Suggestions ======

* vCenter and ESX:

vCenter server: http://www.vmware.com/products/vcenter-server

VMware ESX: http://en.wikipedia.org/wiki/VMware_ESX

* How to write APIs:

https://ep2013.europython.eu/conference/talks/guidelines-to-writing-a-python-api

* OpenStack:

Cinder: https://wiki.openstack.org/wiki/Cinder

Glance: http://docs.openstack.org/developer/glance/

Nova: http://docs.openstack.org/developer/nova/

GSoC2014

2014-02-25T04:17:26Z

Arnaud Legendre: /* OpenStack Incubator (Oslo) */

== Google Summer of Code 2014 ==

[http://www.openstack.org/software/ OpenStack] is applying as a mentoring organization for the [http://www.google-melange.com/gsoc/homepage/google/gsoc2014 2014 Google Summer of Code]. Check [https://developers.google.com/open-source/soc/ Google Developers site] for more information on how the program works.
We need to get in this time!

Link to call for participation: [http://google-opensource.blogspot.com/2014/02/mentoring-organization-applications-now.html here]

Link to FAQs: [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page here]

=== What is Openstack? ===

[http://www.openstack.org/software/ Openstack] is an open-source IaaS cloud computing platform. Its mission is to provide a flexible solution for both public and private clouds of any size, and for this matter two basic requirements are considered: clouds must be simple to implement and massively scalable.

To meet these principles OpenStack is divided into different components that work together. It's [http://www.openstack.org/software/openstack-compute/ computing], [http://www.openstack.org/software/openstack-storage/ storage], [http://www.openstack.org/software/openstack-networking/ networking], and all the other bits that help make this project, '''The Cloud'''.

OpenStack is [http://i.imgur.com/gAyoiF8.png continuously growing] and new and exciting projects are being discussed everyday.

We encourage new contributors to participate and help us make OpenStack the most complete, reliable and flexible open-source cloud service!

=== Mentors ===

If you're interested in mentoring for this round, please add your name, email, IRC handle and the kind of projects you're interested in mentoring below. Please ensure that the projects are official projects in OpenStack and are registered in the [https://github.com/openstack/governance/blob/master/reference/programs.yaml governance projects.yaml]

* Davanum Srinivas - dims - Nova, Oslo
* Debo Dutta - dedutta - Nova, Ceilometer
* [[User:Alejandro_Cabrera|Alejandro Cabrera]] - alcabrera - Marconi, Oslo
* Yathiraj Udupi - Yathi - Nova, Ceilometer
* Fei Long Wang - flwang- Glance
* Boris Pavlovic - boris-42 - Rally
* [[User:Sergey_Lukjanov|Sergey Lukjanov]] - SergeyLukjanov - Savanna (proxy to help find mentor in Savanna community)
* [[User:colinmcnamara|Colin McNamara]] - colinmcnamara - OpenStack Docs, OpenStack Training and ToolChains
* [[User:sriramhere|Sriram Subramanian]] - Fuzz Testing, OpenStack Security
* Balint Kovacs - blint@balabit.hu - Zorp
* Szilard Pfeiffer - floss@pfeifferszilard.hu - Zorp

=== Students ===

Students application period opens March 10 and ends on March 21.

If you'd like to get started on your proposal early, go ahead and add your name, location, e-mail, IRC handle and the project you are interested in (if you already know about that!) here:

*Saket Sinha, India, saket.sinha89@gmail.com [project not selected yet]
* Adnan Khan, Canada, khnd06@gmail.com [project not selected yet]
* Anastasios Andronidis, Greece, andronat_asf@hotmail.com. Proposed: [https://docs.google.com/document/d/1mWWSyftZYXxfXKzOBVeVma8KJbQRmYNPnns3oixHVMk/edit?usp=sharing Glance Scalable Image Precaching]
* Artem Shepelev, Russia, e-mail: shepelev.artem@gmail.com, ashepelev at irc.freenode.net, OpenStack Scheduler.
* Fabio Morais, Brazil, fabio.jorge@gmail.com [Ceilometer; Proposed: [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]]
* Victoria Martínez de la Cruz - Argentina - victoria@vmartinezdelacruz.com - vkmc - OpenStack Message Queuing (Marconi)
* [[User:Damon_Wang|Wei Wang]], China, damon.devops@gmail.com, Neutron & Keystone
* Kumar Rishabh, India, email: shailrishabh@gmail.com [would fill more details later]
* Marc Solanas Tarre, US, email: marc@solanas.cat, mst89, [Monitoring and Telemetry how to detect network anomalies from telemetry data within Openstack]
* Pengfei Zhang, US, lalasjtu@gmail.com, Sparky, [Monitoring and Telemetry Monitoring & Tuning network for QoS within Openstack]
* RobberPhex, China, robberphex@gmail.com, [project not selected yet]
* Rodrigo Duarte, Brazil, rodrigodsousa@gmail.com, OpenStack Scheduler (Gantt)
* Rishi Raj Singh, India, rishiraj.devel@gmail.com, [project not selected yet]
* [[User:Telles Nobrega|Telles Nóbrega]], Brazil, tellesnobrega@gmail.com, tellesnobrega at irc.freenode.net [Keystone, Nova, Savanna, Ceilometer]

=== Communication ===

Get in touch with mentors and students through the [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev openstack-dev mailing list].

Also, you can find us at IRC in #openstack-gsoc at irc.freenode.org.

=== Ideas ===

Here are some project suggestions students can choose for their applications. This doesn't mean students have to stick strictly to this list; don't hesitate in propose projects by your own. For the latter, it would be great if you could submit a draft proposal and a estimated timeline.

When writing your proposal, try to estimate your timeline to fit the 4 month period of GSoC coding. Also, take into account that '''GSoC does not consider other projects than coding''', so other ideas (like community tasks or i18n efforts) are not suitable for this internship.

'''Click on the proposed project that caught your attention to get the details about the assumed knowledge, the project goals and more details related to it.'''

==== Fuzz Testing (various projects) ====

Fuzz testing or fuzzing is a software testing technique, often automated or semi-automated, that involves providing invalid, unexpected, or random data to the inputs of a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. Fuzzing is commonly used to test for security problems in software or computer systems.

{| class="wikitable"
|-
| Difficulty || Hard
|-
| Topics || testing, tempest
|-
| Mentor || Sriram Subramanian
|}

Proposed ideas,

* [[GSoC2014/Testing/Fuzz|Implement a Fuzz testing framework that can be run on Tempest or a similar framework]]

==== OpenStack Scheduler (Gantt) ====

Gantt provides a common scheduler framework for use by various OpenStack components.

Check out more details about [https://github.com/openstack/gantt Gantt]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || schedulers, python, gantt
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/StandAloneScheduler|Implement a stand-alone scheduler based on the scheduler forklift code]]
* [[GSoC2014/Scheduler/ScalableScheduler|Implement a scalable scheduler]]
* [[GSoC2014/Scheduler/CrossServicesScheduler|Implement a cross-services scheduler]]

==== OpenStack Monitoring and Telemetry (Ceilometer) ====

Ceilometer aims to deliver a unique point of contact for billing systems to acquire all counters they need to establish customer billing, across all current and future OpenStack components. The delivery of counters must be traceable and auditable. The counters must be easily extensible to support new projects, and agents doing data collections should be independent of the overall system.

For more details about Ceilometer project, check out the [[Ceilometer#OpenStack_Telemetry_.28Ceilometer.29|wiki]]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || ceilometer, data science
|-
| Mentor || Debo
|}

Proposed ideas,

* [[GSoC2014/Scheduler/NetworkAnomalyDetection|How to detect network anomalies from telemetry data within Openstack]]
* [[Monitoring & Tuning network for QoS within Openstack]]
* [[GSoC2014/Ceilometer/UnderstandingBurstsLifecycle|Applying OpenStack telemetry to understand the bursts lifecycle in resource usage]]

==== OpenStack Message Queuing Service (Marconi) ====

Openstack Message Queuing Service (Marconi) provides a distributed queue. The basic concept is simple:

* Create a queue
* Post messages
* Read them or claim them

For full details on Marconi project, check out the [https://wiki.openstack.org/wiki/Marconi wiki]!

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || storage, python, marconi
|-
| Mentor || Alejandro Cabrera
|}

Proposed ideas,

* [[GSoC2014/Queues/Storage | Add a New Storage Backend]]

==== OpenStack Incubator (Oslo) ====

The Oslo program produces a set of python libraries containing infrastructure code shared by OpenStack projects. The APIs provided by these libraries should be high quality, stable, consistent and generally useful.

Check out more details about Oslo project on the [https://wiki.openstack.org/wiki/oslo wiki]

{| class="wikitable"
|-
| Difficulty || Medium - Hard
|-
| Topics || storage, python, oslo
|-
| Mentor || Alejandro Cabrera, Davanum Srinivas
|}

Proposed ideas,

* [[GSoC2014/Incubator/Storage|Add a new backend to oslo.cache]]
* [[GSoC2014/Incubator/SharedLib|Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects like Nova, Cinder or Glance]]
* [[GSoC2014/Incubator/Plugin|Define a new layer/abstraction in Nova for plugging-in vCenter and ovirt (since they span multiple hosts)]] (WIP)

==== OpenStack Benchmarking System (Rally) ====

OpenStack QA team mostly works on CI/CD that ensures that new patches don't break specific single node installation of OpenStack. On the other hand it's clear that such CI/CD is only an indication and does not cover all cases (e.g. if cloud works well on single node installation it doesn't mean that it will work good as well on 1k servers installation under high load).. Rally aims to fix this and help us to get answer on question "How OpenStack works at scale". To make it possible we are going to automate and unify all steps that are required for benchmarking OpenStack at scale: multi node OS deployment, verification, benchmarking & profiling.

[[File:Rally-Actions.png|500px|center]]

* Deploy is not yet another deployer of OpenStack it is just a plugable mechanism that allows to unify & simplify work with different deployers like: DevStack, Fuel, Anvil on hardware/VMs that you have.
* Verify - (work in progress) Use tempest to verify functionality of deployed openstack. In future Rally will support other OS verifiers.
* Benchmark - Smart combination of Framework, Load generation & Big repository of benchmarks.

For more details about Rally project, check out [https://wiki.openstack.org/wiki/rally wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || rally, benchmarks, deploying, python, tempest,
|-
| Mentor || Boris Pavlovic
|}

Proposed ideas,

* Benchmarking OpenStack
* Writing new benchmarks
* Integration of Rally & Tempest
* Processing Results
* Improving Rally deploying system

==== OpenStack Networking (Neutron) ====

Neutron is an OpenStack project to provide Networking-as-a-Service between interface devices (e.g., vNICs) managed by other Openstack services (e.g., Nova).

For full details about Neutron, check out the [https://wiki.openstack.org/wiki/Neutron wiki]

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || drivers, python, networking, fwaas, neutron
|-
| Mentor || Balint Kovacs, Szilard Pfeiffer
|}

Proposed ideas,

* [[GSoC2014/Network/Driver|Implement an application-level FWaaS driver]] ([https://github.com/balabit/zorp/wiki/zorp Zorp])

==== OpenStack Data Processing (Savanna) ====

Savanna aims to provide users with simple means to provision a Hadoop cluster by specifying several parameters like Hadoop version, cluster topology, nodes hardware details and a few more. The aim of this project is to enable users provision and management of Hadoop clusters on OpenStack.

Check out the [https://wiki.openstack.org/wiki/Savanna wiki] to learn more about Savanna

{| class="wikitable"
|-
| Difficulty ||
|-
| Topics || plugins, hadoop provision, savanna
|-
| Mentor ||
|}

Proposed ideas,

* [[GSoC2014/DataProcessing/Plugin|Develop a new plugin for Savanna to extend savanna.plugins.provisioning:ProvisioningPluginBase class and implement all the required methods.]] (WIP)

=== Projects ===

Current mentors are willing to supervise students in the following projects:

* [https://wiki.openstack.org/wiki/Nova Openstack Compute (Nova)]
* [https://wiki.openstack.org/wiki/Climate OpenStack Reservation (Climate)]
* [https://github.com/openstack/gantt OpenStack Common Scheduler (Gantt)]
* [https://wiki.openstack.org/wiki/Marconi OpenStack Message Queuing (Marconi)]
* [https://wiki.openstack.org/wiki/Oslo OpenStack Common Libraries (Oslo)]
* [https://wiki.openstack.org/wiki/Neutron OpenStack Networking (Neutron)]
* [https://wiki.openstack.org/wiki/Swift OpenStack Object Storage (Swift)]
* [https://wiki.openstack.org/wiki/Cinder OpenStack Block Storage (Cinder)]
* [https://wiki.openstack.org/wiki/Heat OpenStack Orchestration (Heat)]
* [https://wiki.openstack.org/wiki/Savanna OpenStack Data Processing (Savanna)]
* [https://wiki.openstack.org/wiki/Ceilometer OpenStack Telemetry (Ceilometer)]
* [https://wiki.openstack.org/wiki/Rally OpenStack Benchmarking System (Rally) ]

GSoC2014/Incubator/SharedLib

2014-02-25T04:16:57Z

Arnaud Legendre: /* Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects */

= Implement a re-usable shared library for vmware (oslo.vmware) to be consumed by various OpenStack projects =

{| class="wikitable"
|-
| Difficulty || Medium
|-
| Topics || VMware API, Python, Nova/Glance/Cinder
|-
| Mentor || Arnaud Legendre
|}

The VMware API is consumed by OpenStack using the oslo.vmware library [1]. This library recently created specifically for OpenStack is shared by several projects such as Nova, Glance and Cinder. Internally, it is using SOAP calls to communicate with vCenter server and ESXi.
Instead of consuming the API through SOAP calls, it would be possible to use PyVmomi which is a Python SDK for the VMware vSphere API [2]. This work will give the opportunity to redefine a clean and more performant API.
It involves:

# Deep dive the current API and how it is consumed by the OpenStack services
# Deep dive in PyVmomi
# Define an API
# Provide an implementation for this API using PyVmomi
# Make sure existing tests are working or modify them accordingly

[1] https://github.com/openstack/oslo.vmware/

[2] https://github.com/vmware/pyvmomi

== Assumed Knowledge ==
* Python - basics, class/module management
* Command Line - a little bit of git, code editing, navigation
* Already played with virtual machines
* Experience with API definition
* Some basic knowledge of what VMware does would be a plus

Refer to [https://github.com/cabrera/python-openstack-and-you/ Python, Openstack, and You] if you'd like to get a head start.
I will be able to help you!!

== Project Goals ==

* Provide a clean interface for the OpenStack VMware library
* Provide an implementation for it using PyVmomi
* Test suite

== Project Nice-to-Haves ==

* Performance evaluation of PyVmomi compared to the current codebase
* Provide some documentation for the API

====== Suggestions ======

* vCenter and ESX:

vCenter server: http://www.vmware.com/products/vcenter-server

VMware ESX: http://en.wikipedia.org/wiki/VMware_ESX

* How to write APIs:

https://ep2013.europython.eu/conference/talks/guidelines-to-writing-a-python-api

* OpenStack:

Cinder: https://wiki.openstack.org/wiki/Cinder

Glance: http://docs.openstack.org/developer/glance/

Nova: http://docs.openstack.org/developer/nova/

NovaVMware/DeveloperGuide

2014-02-04T02:17:45Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack
$ ./tools/upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./tools/upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./tools/upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk
By default, the Nova driver will be using adapter_type "lsilogic" and and disk_type "preallocated"

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For more information about disk types: http://sanbarrow.com/vmdk-basics.html

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

wget "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < "trend-tinyvm1-flat-;ide;.vmdk"

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2014-01-25T01:35:48Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack/tools
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For more information about disk types: http://sanbarrow.com/vmdk-basics.html

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

wget "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < "trend-tinyvm1-flat-;ide;.vmdk"

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2014-01-25T01:35:09Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack/tools
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For more information about disk types: http://sanbarrow.com/vmdk-basics.html

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

wget http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2014-01-25T01:34:41Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack/tools
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./upload_image.sh "http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat-;ide;.vmdk"

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For more information about disk types: http://sanbarrow.com/vmdk-basics.html

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2014-01-23T01:48:12Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,n-novnc,n-cauth,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh
upload_image.sh has a dependency:
https://github.com/openstack-dev/devstack/blob/master/functions
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.
If a *-flat.vmdk disk is provided and the descriptor cannot be found, "ide" will be used as the default adapter_type and "preallocated" as the default disk_type.

Example of execution:
$ cd ~/devstack/tools
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

To avoid problems with the metadata, it is highly recommended to provide a monolithicSparse or monolithicFlat disk where the descriptor and flat disk are in the same directory.
For more information about disk types: http://sanbarrow.com/vmdk-basics.html

Alternatively, it is possible to call directly the Glance client (the following commands are using the V1 of the Glance client):

$ glance image-create --name Debian --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

If you get the following error: "pip's wheel support requires setuptools >= 0.8 for dist-info support", try this command
sudo pip install setuptools --no-use-wheel --upgrade

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2013-12-23T23:54:45Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh.
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.

Example of execution:
$ cd ~/devstack/tools
$ ./upload_image.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_image.sh file:///home/user/images/debian-2.6.32-i686.vmdk

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

Alternatively, it is possible to call directly the Glance client:

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/debian-2.6.32-i686.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide" and vmware_disktype is set to "thin"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z

NovaVMware/DeveloperGuide

2013-12-23T23:49:22Z

Arnaud Legendre: /* Upload your VMDK to glance */

=Developer's DevStack + vSphere Guide=

This is a short guide for developers interested in working on OpenStack and [[vSphere]] (ESX, [[ESXi]], and [[vCenter]]) drivers.

==vSphere Environment and Inventory==

Set up a vSphere 5.1 (or better) developer's environment. For development purposes, you'll need to have one of this inventory type:

[[File:VCenter cluster inventory.png|framed|none|inventory with a DRS cluster]]

Note: The cluster should be DRS enabled with "Fully automated" placement turned on. Also, if you can't get two or more ESX hosts to work with, you can still work with one host in the cluster, but you won't be able to observe VM placement behaviors since there won't be anywhere to place the VMs except for the solitary host.

=Development Environment=
== Get an Ubuntu 12.04 (suggested) VM setup==
This Ubuntu workstation can run as a VM itself, but it should have public internet access (ideally direct, but http proxy is workable if you don’t need to commit code back to OpenStack if you do, see the troubleshooting page for steps that ''may'' help). It should also have a reasonably high bandwidth link to the the vSphere hosts, as it will need to stream images from the Ubuntu workstation to the vSphere host.

We suggest your Ubuntu workstation have
* at least 10 GB disk, with 20+ GB being ideal.
* at least one vCPU, with 2 being ideal.
* 4 GB of RAM.

=== optional setup ===
If you are using a remote workstation (not running locally on your physical computer) you may have to follow some of the extra steps under [[NovaVMware/DeveloperGuide#Networking_Troubleshooting|Network Troubleshooting]]

==install Devstack (http://devstack.org/) in your new VM==
Once booted, run:
sudo apt-get install git
git clone http://github.com/openstack-dev/devstack.git
cd devstack

=Setup localrc for DevStack=
create a file named “localrc” in your devstack directory with the content shown below (for each item with $variable_name, you will need to enter values specific to your environment, the rest you can just leave as is).

file ~/devstack/''localrc''
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-cpu,n-net,n-cond,n-sch,rabbit,mysql,horizon
VIRT_DRIVER=vsphere
VMWAREAPI_IP='''$your_vCenter_ip_address'''
VMWAREAPI_USER=root
VMWAREAPI_PASSWORD='''$password_goes_here'''
VMWAREAPI_CLUSTER='''$your_cluster_name'''
DATABASE_PASSWORD=nova
RABBIT_PASSWORD=nova
SERVICE_TOKEN=nova
SERVICE_PASSWORD=nova
ADMIN_PASSWORD=nova
HOST_IP='''$your_development_vm_ip'''

Note: if you would like to use a specific datastore then configure ''VMWAREAPI_DATASTORE_REGEX''

Note: omit the line ''VMWAREAPI_CLUSTER'' if you are using the ''trivial'' inventory example.

Note: If you want the logs to be written to disk (not just accessible via screen), use the define SCREEN_LOGDIR to point to path
where you want to the logs to reside. For example

SCREEN_LOGDIR=/var/log

=start stack (first time)=
./stack.sh
This will take a long time the first time, as it pulls down all code and dependencies.
Note: may fail on first start.

=Credentials and Environment Variables=
get the OpenStack API credentials in your environment variables:

$ source openrc demo demo

This is needed anytime you make a call using an openstack CLI client like ‘nova’, ‘glance’, or ‘quantum’. You can rerun this at any time from the devstack directory if you need to re-login in at a later point. If you want admin credentials then use admin ($ source openrc admin admin)

=Glance Initial Setup=
Get a VMDK to use as a disk image and upload it to OpenStack using the Glance API. The latest code (mid-Aug) has a change which uploads the image when you 1st run devstack. After glance starts, run the following command to see if you have a pre-configured image in glance

$ glance image-list

==Get an initial VMDK to work with==
There are a lot of “gotchas” around what VMDK disks work with OpenStack + vSphere, so we strongly suggest using the provided image to start. In the Appendix there is information about creating/converting your own images.

download the image 1 GB debian disk image (user/password to login this image after booting is nicira/nicira).
http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
Place the file under your home directory <code><nowiki>~/</nowiki></code>

NOTE: if you are using nested hyper-visors you will need to obtain a 32bit image to use or troubleshoot: how to enable nested ESXi & other Hypervisors in vSphere 5.1 - See more at: http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html#sthash.NJ2VNiwt.dpuf

==Upload your VMDK to glance==
It is recommended to use the upload_images.sh script provided in the Devstack to upload images to Glance:
https://github.com/openstack-dev/devstack/blob/master/tools/upload_image.sh.
The script will introspect the image provided to extract the metadata.
It expects a URL as unique argument. The scheme of the URL can be http(s) or file. When a flat disk (*-flat.vmdk) is provided, the script will attempt to retrieve the descriptor (*.vmdk) associated to find the correct metadata. The same way, the user can specify a descriptor: in this case, the script will try to find the flat disk.

Example of execution:
$ ./upload_images.sh http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk
$ ./upload_images.sh file:///home/user/images/debian-2.6.32-i686.vmdk

Specific metadata can be provided: the expected format of the image filename is <name>-<disk type>;<storage adapter>;<network adapter>.vmdk

Alternatively, it is possible to call directly the Glance client:

$ glance add name="Debian" disk_format=vmdk container_format=bare is_public=true \
--property vmware_disktype="preallocated" < ~/debian-2.6.32-i686.vmdk
... or ...
$ glance image-create --name Debian --is-public=True --container-format=bare \
--disk-format=vmdk --property vmware-disktype="preallocated" < ~/debian-2.6.32-i686.vmdk

This will print out an <image-id> for use below. If you forget it, you can always run “glance image-list” to see all existing images.

There is also an IDE based image that can be used.

http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
Note the vmware_adaptertype is set to "ide" and vmware_disktype is set to "thin"
$ glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

=Nova Boot=
boot a VM using the “<image-id>” from above

$ nova boot --image <image-id> --flavor 1 test1

Now you can interact with the image using other nova client commands.
For example
$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------+
| 9bb724bf-298d-4a63-a653-dab7fcbd9ac7 | test1 | ACTIVE | None | NOSTATE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------+-------------+------------------+

Note: if your disk is big, it can take a VERY long time to stream the image from glance to your datastore (e.g., ~7 minutes for a 1 GB thick image). This will happen only the first time a new image UUID is used on a datastore. You can monitor progress by viewing the size of the file in the vmware_base directory of the filestore.

Once this is done, you can view the VM using vCenter. The username/password for the above Debian image is nicira/nicira .

You can access the OpenStack Web GUI (called horizon) by accessing your Ubuntu host on port 80, though currently VNC access via Horizon is broken (see: https://review.openstack.org/#/c/30036/ for a patch to fix this).

=Using Screen=
Use screen to interact with individual openstack services

$ screen -x stack

See: http://www.gnu.org/software/screen/manual/screen.html <- for how to use

* "Control + a" enters command mode... enter this for each new command character
characters 0 through 9 switch tabs
** the 'd' character detaches the session
** the '[' character enters "copy mode" which allows you to navigate the screen trace and 'escape' escapes the copy mode.
** the '"' character shows a list of screens which you can navigate with the up and down arrows

For example the nova-compute service runs on the n-cpu tab which is usually number 6. So to navigate to it, hit Ctrl+a then press '6' and you'll tab to it. You can then interact with the terminal for n-cpu.

=Working the Development Environment=
If you want to reset your environment, run:
./unstack.sh
./stack.sh

* This run of stack.sh will be faster (~1 minute) as it resets all database and service state, but does not need to re-download packages or source code.
* After running ./unstack be sure to clean out all your datastores and delete any files left behind on accident. This will prevent a great number of problems people won't see in production environments. (For example: partially uploaded VMDK.)

Note: the source code is not automatically updated when you reset your environment. This is valuable, as you can manage your source code manually via git, having many branches, etc.

= Remote Debugger =

To use the pycharm remote debugger (any maybe [http://wiki.xbmc.org/index.php?title=How-to:Debug_Python_Scripts_with_Eclipse eclipse]), make the following changes in your nova code

index 0e7ecdc,87edf9b..0000000
--- a/nova/cmd/__init__.py
+++ b/nova/cmd/__init__.py
@@@ -31,8 -31,4 +31,8 @@@
os.environ['EVENTLET_NO_GREENDNS'] = 'y

import eventlet

-eventlet.monkey_patch(os=False)
+if '--debug-host' and '--debug-port' in sys.argv:
+ eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)
+else:
+ eventlet.monkey_patch(os=False)
+

diff --combined nova/cmd/compute.py
index 61cf434,f03a9bd..0000000
--- a/nova/cmd/compute.py
+++ b/nova/cmd/compute.py
@@@ -33,19 -33,10 +33,19 @@@
from nova.openstack.common import log a
from nova import service
from nova import utils

+cli_opts = [
+ cfg.StrOpt('debug-host',
+ default=None,
+ help='Connect to debug host'),
+ cfg.IntOpt('debug-port',
+ default=None,
+ help='Connect to debug host port'),
+ ]
+
CONF = cfg.CONF
CONF.import_opt('compute_topic', 'nova.compute.rpcapi')
CONF.import_opt('use_local', 'nova.conductor.api', group='conductor')
-
+CONF.register_cli_opts(cli_opts)

def block_db_access():
class NoDB(object):
@@@ -72,16 -63,6 +72,16 @@@ def main()
objects_base.NovaObject.indirection_api = \
conductor_rpcapi.ConductorAPI()

+ if CONF.debug_host:
+ from pydev import pydevd
+ print ('Listening on ' + str(CONF.debug_port) + ' for host ' +
+ CONF.debug_host)
+ pydevd.settrace(host=CONF.debug_host,
+ port=CONF.debug_port,
+ stdoutToServer=False,
+ stderrToServer=False)
+
+
server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic,
db_allowed=False)

Then in pycharm, create a remote debug session with host localhost and port whatever you want. When you run it, use --debug_host <your IP> and --debug_port <your port>. That's it!

=Testing=
==Install python support libraries==
===apt-get libs===
$ sudo apt-get install python-dev libmysqlclient-dev python-pip build-essential
$ sudo apt-get install libxml2-dev libxslt1-dev libpq-dev

===pip based tool installs===
$ sudo pip install --upgrade pip
$ sudo pip install --upgrade virtualenv
$ sudo pip install MySQL-python
$ sudo pip install tox
$ sudo pip install python-subunit

====pip proxy troubleshooting====
If you get connection refused here - try this
export PIP_INDEX_URL="http://pypi.openstack.org/openstack"

====Install OpenStack libs====
$ cd /opt/stack/nova
$ sudo pip install -i http://pypi.openstack.org/openstack -r requirements.txt -r test-requirements.txt

Note: make sure everything in the requires lists gets installed. Some installers have individual issues that you may have to clear one at a time. In particular lxslt has some issues that may require attention to properly clear.

==Using TOX for testing==
See: https://wiki.openstack.org/wiki/ProjectTestingInterface

Important tests to run before submitting for code review are
tox -e py26,py27
tox -e pep8

As of this writing the run_tests.sh is the better way to run tests.

== Using run_tests.sh for testing ==
$ cd /opt/stack/nova
$ ./run_tests.sh
... or ...
$ ./run_tests.sh nova.tests.virt.vmwareapi
... to run only the vmware unit tests.

Note: If you want to use the python packages in your normal (non-virtual) environment then pass the -N flag

==Code Coverage tests and information==

=== with tox ===
$ cd /opt/stack/nova
$ tox -e cover (code coverage)

look under the cover directory for html reports on the code coverage for that module. The file index.html contains the overall coverage report

NOTE: tests may run 10, 20, or as long as 65 minutes (depending on your hardware, network, and system) and produce no output until they complete. This is normal. You system may become sluggish but should not lock up entirely, a system hang can be symptomatic of an improperly configured test environment. Tests will run on Mac OSX. 2 Cores with 2GB of RAM on an Ubuntu 12.04 machine that is properly configured should take between 10 and 20 minutes to complete.

=== with run_tests.sh ===
$ cd /opt/stack/nova
$ ./run_tests.sh --coverage

==Additional setup==
If you are on Ubuntu 12.04 and need to use Python 2.6 (which OpenStack needs for some test scripts) install the old version of python by doing the following:

$ sudo add-apt-repository ppa:fkrull/deadsnakes
$ sudo apt-get update
$ sudo apt-get install python2.6 python2.6-dev

== Disabling verbose logging subsystems ==

Add this line to nova.conf to disable the amqp logging (for example)

default_log_levels=amqp=WARN,amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARN,nova.openstack.common.rpc.amqp=WARN

=Troubleshooting=
''Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.''

== Logging ==
You may have a situation where you need to monitor devstack's log files. Here's how you set up logging to file with the least amount of fuss.

To set up logging of screen windows set the shell variable SCREEN_LOGDIR to the directory you want the log files to go. Log files will appear with names like ''screen-$SERVICE_NAME-$TIMESTAMP.log'' in that dir and symbolic link screen-$SERVICE_NAME.log will link to the latest log file. Logs are kept for as long specified in the LOGDAYS variable.

Both SCREEN_LOGDIR and LOGDAYS are shell variables so you need to set them either in your shell or in localrc. Use shell commands if you only want the logs to work this one time. Use ''localrc'' if you always want to have log files appear.

Assuming you have already made a directory...

$ mkdir ~/log

Then, if you only want logs this one time, in your bash shell say...

$ export SCREEN_LOGDIR=~/log
$ export LOGDAYS=2

... or if you ''always'' want log files, in your localrc add the lines ...

SCREEN_LOGDIR=~/log
LOGDAYS=2

The stack.sh script will automatically rotate the log files for you.

== MySQL issues ==
If you get stuck with Keystone try...
$ mysql -u root -pnova
... if that doesn't work you probably have a broken mysql install. If you have problems getting mysql to behave, try purging the OpenStack version of the install and install the database manually. Here’s how to rip everything down and install mysql-server from scratch yourself.

$ ./unstack
$ sudo apt-get purge mysql-server
... you may need to use the name mysql-server-5.5 instead ...
$ sudo apt-get autoremove
$ sudo apt-get install mysql-server-5.5
$ sudo mysql_install_db
$ sudo mysqladmin -u root password 'nova'
$ sudo service mysql restart
$ ./stack

== Keystone Issues ==
If you get stuck when you are checking your keystone connection via curl then it’s your proxy settings. Try setting the following in /etc/environment
no_proxy=localhost,127.0.0.0/8,127.0.1.1,127.0.1.1*,local.home

== General Database Issues ==
If you have trouble with testing or other database heavy activities you may need to follow:
http://codeinthehole.com/writing/how-to-set-up-mysql-for-python-on-ubuntu/

== rootwrap Issues ==
If networking suddenly stops working with an error like this

Traceback (most recent call last):
File "/usr/bin/nova-rootwrap", line 59, in <module>
from nova.rootwrap import wrapper
ImportError: No module named rootwrap

Then something has added a broken rootwrap to /usr/bin. Remove the /usr/bin/nova-rootwrap and it will use the correct one in /usr/local/bin

=== Check if nova-compute is running ===
$ screen -x
Note the tab n-cpu is running in. It is usually #6 but it could be a different tab on occasion. Tab to n-cpu's tab. '''ctl+a''' then '''6'''. If you see:
$ cd /opt/stack/nova && sg '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' || touch "/opt/stack/status/stack/n-cpu.failure"
sg: group '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf' does not exist
$
... then nova-compute did not start. You can manually start nova-compute by editing the command to read:
$ cd /opt/stack/nova && /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf || touch "/opt/stack/status/stack/n-cpu.failure"
... which is permissible for development (but not for production).

== Networking Troubleshooting ==
Remotely accessed workstations need to have an IP for you to SSH or VNC into in order to do work. These instructions mostly assume a ''locally accessible'' VM or physical machine. If you are working with a remotely hosted Ubuntu developer's workstation, you may have special networking concerns. If you find your use of nova-network is stealing your workstations IP in your environment (does not manifest in all environments so it's located here as a troubleshooting step) then set up a second interface for nova-network to use.

=== Configure a fake networking interface ===
Configure a fake interface so that nova-network does not steal the real IP that you need to access the host remotely. On Ubuntu:

$ sudo ip tuntap add dev tapfoo mode tap
$ sudo ifconfig tapfoo $some_ip up

NOTE: assign the interface an ip address that will be appropriate for your environment.

Now that you have a fake ethernet interface, change your localrc and re-run ./stack.sh so that you can start nova-network on the new interface.

add to your ''localrc'' file
FLAT_INTERFACE=tapfoo

=== Configure a Proxy ===
Some workplaces may have to use special networking and/or proxy settings and that these may change when you are mobile. Those issues can cause problems with apt-get, git and other parts of OpenStack. If you are in a place that needs special proxy rules, [https://help.ubuntu.com/community/AptGet/Howto/#Setting_up_apt-get_to_use_a_http-proxy setup apt-get with a proxy]. You may also have to set [http://askubuntu.com/questions/158557/setting-proxy-from-terminal other proxy settings] depending on your company's particular firewall rules.

=Appendix=

==Converting Images==

===Converting other disk images to vmdk using qemu-img===
Disk images in several formats (e.g. qcow2) can be converted to the VMDK format usable by the vmware nova driver using the qemu-img utility.

For example the following command can be used to convert a qcow2 Ubuntu Precise cloud image (downloadable from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img):
 <code><nowiki> qemu-img convert -f raw ~/Downloads/precise-server-cloudimg-amd64-disk1.img -O vmdk precise-server-cloudimg-amd64-disk1.vmdk</nowiki></code>

===Converting a sparse vmdk to an ESX-compatible format===

Due to a bug (fixed with the following proposed [https://review.openstack.org/#/c/43994/ patch], currently under review) in how sparse vmdk disks are handled in the VC driver,
sparse disks have to be pre-converted to thin-provisioned or preallocated disks before they can be uploaded to glance to be used by the driver.

There are several ways to perform such a conversion:

====1. Using vSphere CLI (or sometimes called the remote CLI or rCLI) tools.====

The latest version from the date this was written is 5.1 and the link is [https://my.vmware.com/web/vmware/details?downloadGroup=VSP510-VCLI-510&productId=285 here].

Assuming that the sparse disk is made available on a datastore accessible by an ESX host, the following command converts it to preallocated format:

vmkfstools --server=ip_of_some_ESX_host -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

(note the vifs tool from the same CLI package can be used to upload the disk to be converted and downloaded the converted disk if necessary)

====2. Using vmkfstools directly on the ESX host====

If the SSH service is enabled on an ESX host, the sparse disk can be uploaded to the ESX datastore via scp, and the vmkfstools local to the ESX host can use used to perform the conversion:

(After logging to the host via ssh)
vmkfstools -i /vmfs/volumes/datastore1/sparse.vmdk /vmfs/volumes/datastore1/converted.vmdk

====3. vmware-vdiskmanager====

vmware-vdiskmanager is a utility that comes bundled with VMware Fusion and VMware Workstation.
Below is an example of converting a sparse disk to preallocated format:

'/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager' -r sparse.vmdk -t 4 converted.vmdk

In all the above cases, the converted vmdk is actually a pair of files, the descriptor file converted.vmdk and the actual virtual disk data file converted-flat.vmdk. The file to be uploaded to glance is converted-flat.vmdk.

===Specifying correct adapter type when uploading and image to glance===

Currently, there is a limitation that OS boot vmdk disks with an ide adapter type cannot be attached to a virtual SCSI controller, and likewise disks with one of the scsi adapter types (e.g. busLogic, lsiLogic) cannot be attached to the IDE controller. Therefore it is important when uploading an image to glance that the correct vmware_adaptertype property be set. In particular, vmdk disks converted from other formats via qemu-img will _always_ be a monolithic sparse vmdk with an IDE adapter type. So using the above example of the Precise Ubuntu image, after the qemu-img conversion and conversion to a preallocated disk, the command to upload the vmdk disk should be something like:

glance image-create --name precise-cloud --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="preallocated" --property vmware_adaptertype="ide" < converted-flat.vmdk

====Obtaining the adapter type associated with a vmdk====

If the vmdk is a monolithic file (such as one produced by the qemu-img conversion) the adapter type can be retrieved by

head -20 some_monolithic.vmdk

and looking for the ddb.adapterType=XXX property

If the vmdk has a descriptor/data file pair (foo.vmdk and foo-XXXX.vmdk). The descriptor file foo.vmdk can be examined for the same ddb.adapterType property)

==List of Reviews in progress==
https://review.openstack.org/#/q/message:vmware+OR+message:vcenter+OR+message:vsphere+OR+message:esx+OR+message:vcdriver+OR+message:datastore,n,z