Trove/dev-env

= Trove API and Redstack Dev Environment Set Up on Mac OS X =

Intro
This article describes how to point a local running Trove instance to a Devstack/Redstack deployment in a VM or Cloud instance. You should be able to debug and test your changes without having to deploy.

Installing VMWare Fusion and Trove services
openstack=== VM ===
 * 1) In order to run trove/redstack in a local VM, you need to start by installing a guest with Ubuntu 12.04 server. Download an image file from Ubuntu's web site (http://www.ubuntu.com/download/server) and create a new guest from it in your preferred virtualization solution (VMWare fusion, VirtualBox, Parallels, etc.)
 * 2) Recommended version ubuntu-12.04.1-server-amd64.iso
 * 3) Recommended RAM size is 2GB, but could probably work with 1GB if needed
 * 4) It seems no special network settings are needed in order to be able to access the guest from the host, the defaults work just fine
 * 5) Notice! It seems that the guest-agent will fail to "$ sudo pip install extras" and the below should be run on the redstack guest: $ sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE
 * 6) It seems there is no need for the VMWare tools. They don't solve the copy/paste (or make working with the terminal any easier...) and we simply recommend using ssh instead. see details below.
 * 7) For better performance, make sure you go to the VM Settings -> Processors & Memory -> Advanced Options and check the "Enable hypervisor applications in this virtual machine" (see more about this at the bottom of this page: https://github.com/openstack/trove-integration)
 * 8) The full description of installing Redstack is described here: https://github.com/openstack/trove-integration/blob/master/README.md but we recommend you follow this path:
 * 9) When you set up the guest ubuntu, either create is with the 'ubuntu' user or add it: $ adduser ubuntu  $ visudo
 * 10) Add this line to the file below the root user $ ubuntu  ALL=(ALL:ALL) ALL  **OR use this if you dont want to type your password to sudo a command**  $ ubuntu  ALL=(ALL:NOPASSWD) ALL
 * 11) Linux systems may cache sudo credentials making it appear that sudo is operating without a password, even if there is an error in the /etc/sudoers file.  Verify that sudo operates without a password with sudo -k ls
 * 12) Install ssh server on the guest: $ sudo apt-get install ssh (or openssh)
 * 13) Now, you can run ifconfig on the guest and get it's IP. Use that to ssh (as ubuntu) to it from your mac and continue working from there, it's easier ...
 * 14) Get git: $ apt-get update  $ apt-get install git-core -y
 * 15) This is a good point to take a snapshot of this guest
 * 16) Clone this repo: $ git clone https://github.com/openstack/trove-integration.git
 * 17) Go into the scripts directory: $ cd trove-integration/scripts/
 * 18) Install all the dependencies and then install trove via devstack (this Brings up trove (rd-api rd-tmgr) and initializes the trove database) $ ./redstack install  Should you need to restart the redstack script after it terminating prematurely, you may encounter an error due to a setuptools package being too old;  in that case, update setuptools with the following wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | sudo python
 * 19) Since latest merges of Redstack in to Devstack, i'm not sure what is the exact state of the system at this point, since the screen session is already showing trove-api and trove-taskmanager as running in two of the screens. In any case you can "initialize" trove by:
 * 20) running this, which encapsulate several calls: $ ./redstack kick-start mysql
 * 21) or running the steps manually: $ ./redstack test-init (used to be "initialize"?)  $ ./redstack build-image mysql (or "percona")  $ ./redstack start
 * 22) For more information and usage see here:  https://github.com/openstack/trove-integration/blob/master/README.md
 * 23) There used to be an issue with precise getting jammed. We're not sure if this is still the case, but if you get into such issues, please check the workaround described here:
 * 24) Installing Ubuntu Precise on VMWare Fusion http://wiki.openstack.org/vmware-fusion-precise (just use defaults)

Local Dev
The main idea here, is to have trove-api and/or trove-taskmanager running locally on your mac (in a console or through your IDE so you can actually debug the code) but working against keystone/nova/swift/rabbit-mq/mysql that are all installed on your redstack node (installed above)
 * 1) First, get the code: $ git clone https://github.com/openstack/trove.git
 * 2) You'll need the have mysql client installed. If you don't have it already, install it using homebrew (running $ sudo pip install mysql-python would fail otherwise)
 * 3) Installing homebrew https://github.com/mxcl/homebrew/wiki/Installation
 * 4) Then install mysql $ brew install mysql
 * 5) You're going to use a dedicated python virtual env. Set it up and "activate" it like this: $ trove/tools/install_venv.py  $ source trove/.venv/bin/activate
 * 6) In your new env, install python-swiftclient and mysql-python: $ sudo pip install python-swiftclient  $ sudo pip install mysql-python
 * 7) Due to some issue with openssl and cms (that doesn't work properly on a mac by default) you'll have to apply this fix:
 * 8) Download the openssl tar ball from http://www.openssl.org/source/
 * 9) openssl-1.0.1e.tar.gz seems to work fine: http://www.openssl.org/source/openssl-1.0.1e.tar.gz
 * 10) unzip it ($ tar xvzf filename) and follow the instruction here:
 * 11) Installing openssl on Mac OS X with cms enabled http://stackoverflow.com/questions/9275786/updating-openssl-version-on-osx
 * 12) basically, running: sudo ./Configure --prefix=/usr/ darwin64-x86_64-cc enable-cms  $ sudo make  $ sudo make test  $ sudo make install
 * 13) As an alternative you can run brew install openssl and then back up and replace sudo cp /usr/bin/openssl /usr/bin/openssl_old; sudo cp /usr/local/Cellar/openssl/1.0.1e/bin/openssl /usr/bin/openssl; sudo chown root:wheel /usr/bin/openssl; sudo chmod 755 /usr/bin/openssl
 * 14) Depends on what you've done so far, you might need to "clean up" your env. If you encounter issues with keystone (authenticating) try this:
 * 15) Delete all .pem files under /tmp/keystone-signing-nova
 * 16) Delete all .pem files under /tmp/keystone-signing-trove
 * 17) You might also have to delete ~/.apitoken

Configuring local trove files
For your local processes (api and taskmanager) to find the needed resources on your redstack, you'll have to modify the conifg files. It is recommended you create a copy of trove/etc/trove/trove.conf.sample and trove/etc/trove/trove-taskmanager.conf.sample, modify them and point to those copies when running the api and taskmanager services (details below). You'll also have to modify the api-paste.ini, but since it's currently referenced hard codedly, you'll have to modify this file directly and be carefull not to check it in ;-)  /trove/etc/trove/api-paste.ini  #Auth host will be IP of your restack guest  auth_host = 127.0.0.1   #Tenant name, admin user, and admin password can be retrieved by running  nova credentials on your redstack guest  admin_tenant_name = %SERVICE_TENANT_NAME%   admin_user = %SERVICE_USER%  admin_password = %SERVICE_PASSWORD% sql_connection = mysql://root:e1a2c042c828d3566d0a@localhost/trove - replace localhost with the redstack ip swift_url = http://localhost:8080/v1/AUTH_ - replace localhost with the redstack ip trove_auth_url = http://0.0.0.0:5000/v2.0 - replace the 0.0.0.0 with the redstack ip nova_compute_url = http://localhost:8774/v2 - replace localhost with the redstack ip nova_volume_url = http://localhost:8776/v1 - replace localhost with the redstack ip notifier_queue_hostname = localhost - replace localhost with the redstack ip rabbit_host = localhost - replace localhost with the redstack ip. add this one if it's not there pydev_debug = enabled - this will enable debugging
 * Update fields in api-paste.ini
 * Update the host entries in trove.conf.sample & trove-taskmanager.conf.sample to point to the Devstack/Redstack host (i.e. nova_compute_url, nova_volume_url, notifier_queue_hostname)
 * 1) No need to change the bind_host (i think ...)
 * Update the trove-guestagent.conf.sample as well if you are going to be running this locally.

Getting into the vm
The dev environment, as set up by the diskimage we prepare, adds your ssh key for local development. It uses your existing key, so you should be able to just Once in the vm, the first thing to see is if the trove codebase was deployed If it is in the vm, the initial boot steps worked. If not, its possible you forgot to run the iptables MASQUERADE command. Next step is to see if the guest is running If it is, good, lets check the logs That will tell you what went wrong, or if its still in the install phase
 * ssh IP (ssh 10.0.0.2, if 10.0.0.2 is the ip)
 * ls ~/trove
 * ps aux|grep guest
 * less /tmp/logfile.txt

If the guest did _not_ come online, something else could be wrong. I recommend using your init deamon (upstart...) to restart the guest. If that does not work, try to manually start the guest (the line that is in your init daemon's script).

Running the trove services locally
Trove scripts currently expect to be run from the project root folder. Configure your IDE to do this, and also make sure to use the .venv as the interpreter export OS_AUTH_URL="http://REDSTACK_GUEST_IP:5000/v2.0/" # Run keystone endpoint-list to get region name export OS_REGION_NAME=  # Run nova credentials to get username, password, tenant name, and tenant ID export OS_USERNAME=  export OS_PASSWORD=  export OS_TENANT_NAME=  export OS_TENANT_ID= 
 * 1) $ bin/trove-api --config-file=etc/trove/trove.conf.sample (web service)
 * 2) Notice! if you created a copy of the config file, make sure you point to your copy instead!
 * 3) $ bin/trove-taskmanager --config-file=etc/trove/trove-taskmanager.conf.sample (task manager)
 * 4) Notice! same as the notice above ;)
 * 5) $ bin/trove-guestagent --config-file=etc/trove/trove-guestagent.conf.sample (guest agent)
 * 6) Notice! same as the notice above ;)
 * 7) Set these environment environment variables on local machine in order to use trove-client without having to specify these every call

You can now access your local api server using curl, Chrome's Advanced Rest Cliet or trove-cli (the command line utility - get it by installing python-troveclient) To get things rolling, you start by obtaining an auth token. For some reason, i couldn't do this with the Chrome's rest client, but you can use the two methods below:
 * 1) $ curl -d "{\"auth\":{\"passwordCredentials\":{\"username\": \"radmin\", \"password\": \"radmin\"},\"tenantName\":\"trove\"}}" -H "Content-type: application/json" http://IP_OF_REDSTACK_GUEST:35357/v2.0/tokens
 * 2) Notice, you use the redstack ip since this is were your keystone runs
 * 3) $ trove --username=radmin --apikey=radmin --tenant=trove --os-auth_url=http://IP_OF_REDSTACK_GUEST:5000/v2.0/ --bypass-url=http://localhost:8779/v1.0/YOUR_TENANT_ID
 * 4) You can obtain your tenant_id by running this on the redstack guest: $ nova credentials
 * 5) Notice the use of the --bypass-url

Remember that you also have the api and taskmanager running on your redstack node. So remember to have only one running - those that you start locally, needs to be stopped on the redstack node. ssh into it, run screen -x stack, go the the relevant screen (ctrl a followed by ctrl " - will show you the list of screens and you can select the one you need) and hit ctrl c to stop them. you can then restart them by hitting the up arrow and enter.

Hacking RabbitMQ
The api server talks to the taskmanager via Rabbit, and "knows" the name of the queue from the trove.conf.sample (or your local copy of it, if renamed): taskmanager_queue = taskmanager When running taskmanager under debug, it'll actually create/listen on a queue called pydevd.py - be sure to update your conf file appropriately if you run taskmanager under debug. also, make sure to bounce the api server after you make those changes. Also, notice, that even if you leave the api server running on the redstack node, you'd still have to update this conf file, it should be under /etc/trove/trove.conf

There is a nice management plugin for RabbitMQ, with a web gui console. To use it, do the following:
 * On the Redstack node (the one running Rabbit) run: $ sudo /usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management $ sudo service rabbitmq-server restart
 * You can now access it through port 55672:
 * http://REDSTACK_IP:55672
 * The default user to use is: "guest" and the password is taken from redstack.rc:
 * RABBIT_PASSWORD=f7999d1955c5014aa32c
 * more about this console: http://www.rabbitmq.com/management.html

Accessing the DB
In order to access the admin DB on the redstack guest, you can:
 * Run: $ mysql -u root -p"ADMIN_PASS" mysql> show databases; (to see the databases)  mysql> use trove; (to start using the trove DB)  mysql> show tables; (to see the list of tables)
 * Now run any query you want
 * The admin password is taken from redstack.rc:
 * MYSQL_PASSWORD=e1a2c042c828d3566d0a
 * Alternatively, you can setup a local gui-based client on your mac, such as SequelPro and point to your redstack ip