OpenStack - User contributions [en]

Meetings/TroveBPMeeting

2014-11-17T15:47:54Z

Tim Simpson: /* Nov 17 Meeting CANCELED */

= Weekly Trove Blueprint Meeting =
For those interested, we have blueprint meetings in #openstack-trove weekly, Mondays at 18:00 UTC. 
Feel free to add items in the agenda below if you want to bring a blueprint up for approval. 
Please follow the BP template at https://wiki.openstack.org/wiki/TroveBlueprint 
Please submit RST specs as change requests to https://github.com/openstack/trove-specs 

== Nov 17 Meeting CANCELED==
* Add support for DB2 Express-C datastore (https://review.openstack.org/#/c/133856/) - Mariam John
* Add support for Apache CouchDB datastore (https://review.openstack.org/#/c/133849/) - Mariam John
 
Meeting Agenda History: https://wiki.openstack.org/wiki/Trove/MeetingAgendaHistory#Trove_Blueprint_Meeting_Agenda_History 
Meeting Chat Logs: http://eavesdrop.openstack.org/meetings/trove_bp_review/2014/

== Nov. 24 Meeting ==
[grapex]
* Example Snippet Generator - Tim Simpson
[denis_makogon]
* Clustering action improvements
** https://blueprints.launchpad.net/trove/+spec/deprecate-cluster-action

Meetings/TroveBPMeeting

2014-11-17T15:40:07Z

Tim Simpson: /* Nov. 24 Meeting */

= Weekly Trove Blueprint Meeting =
For those interested, we have blueprint meetings in #openstack-trove weekly, Mondays at 18:00 UTC. 
Feel free to add items in the agenda below if you want to bring a blueprint up for approval. 
Please follow the BP template at https://wiki.openstack.org/wiki/TroveBlueprint 
Please submit RST specs as change requests to https://github.com/openstack/trove-specs 

== Nov 17 Meeting CANCELED==
* Example Snippet Generator - Tim Simpson
* Add support for DB2 Express-C datastore (https://review.openstack.org/#/c/133856/) - Mariam John
* Add support for Apache CouchDB datastore (https://review.openstack.org/#/c/133849/) - Mariam John
 
Meeting Agenda History: https://wiki.openstack.org/wiki/Trove/MeetingAgendaHistory#Trove_Blueprint_Meeting_Agenda_History 
Meeting Chat Logs: http://eavesdrop.openstack.org/meetings/trove_bp_review/2014/

== Nov. 24 Meeting ==
[grapex]
* Example Snippet Generator - Tim Simpson
[denis_makogon]
* Clustering action improvements
** https://blueprints.launchpad.net/trove/+spec/deprecate-cluster-action

Meetings/TroveBPMeeting

2014-11-14T18:42:24Z

Tim Simpson: /* Nov 17 Meeting CANCELED */

Meetings/TroveBPMeeting

2014-11-14T18:42:11Z

Tim Simpson: /* Nov 10 Meeting CANCELED */

Trove/event simulator

2014-09-24T19:51:33Z

Tim Simpson: Created page with "The event simulator is a module used by the Trove integration tests to simulate time and run the tests as quickly as possible while also making sure they run deterministically..."

The event simulator is a module used by the Trove integration tests to simulate time and run the tests as quickly as possible while also making sure they run deterministically.

== Fake Mode Recap ==
To explain the event simulator, a recap of Trove's "fake mode" is in order. Fake mode is just a special configuration of Trove where certain major components, such as the Nova API client, are replaced by long lived test doubles. These "fakes" are more complex than a mock in that they are long lived and work to present a consistent facade of behavior in the components they represent.

For example, when a "fake" Nova server is provisioned, it actually creates an object in a dictionary which is marked as being built. Requests to "show" this object using the Nova client's servers.get method will return a server in a building status. The creation of the fake server will also spawn an eventlet thread that will, after a few seconds, update the status of the fake server object to "active." Because of this, Trove can use this fake Nova client in place of the real thing. Similar fakes are created for Cinder, the guestagent, and other major components.

Because the fake objects need to wait a bit of time before changing their state this means that fake mode takes awhile to run even though nothing is happening. This is exacerbated by the tests, which need to poll the status of Trove objects repeatedly, sleeping each time in between.

One solution would be to rewrite all of the tests to never sleep, and change the fake objects accordingly. This could be done at the expense of being unable to use the same tests against a real deployment of Trove, and conversely being unable to make sure the real tests written for Trove work without running against a real deployment, which may be expensive in terms of time and resources.

This is where the event simulator comes in.

== Event Simulator ==

The root of the problem is time.sleep is getting called and waiting for actions which themselves are being delayed for no reason other than to make sure the tests can order themselves correctly.

Event Simulator fixes things (where "fix" is meant in the Vegas sense of the word) by monkey patching the time.sleep calls as well as calls to spawn eventlet threads.

When the app asks to launch a thread, the event simulator places the function that would be run into a "fake_thread" of which it keeps a collection. The initial fake thread ends up being the tests themselves. All of these fake threads are managed by a central loop (kind of like a reactor pattern). Each thread is run if and only if it doesn't need to sleep anymore. Fake threads acknowledge they wish to sleep using the fake_sleep method of event_simulator which replaces time.sleep during monkey_patching.

This by itself would not work if the code running for these fake threads were associated to regular eventlet threads which might run wherever.

To fix that, and the second problem being addressed, which is that test runs won't be deterministic (i.e. speed of the test machine and other factors could lead to threads running in a different order), event_simulator wraps all of the threads it runs into simulated coroutines. This means only one fake thread gets run at any given time.

The way this works is every eventlet thread is wrapped in a event_simulator.Coroutine object, which waits on a semaphore to run. Code which runs the coroutine is in a different thread, and as soon as it asks the coroutine to run waits for its own semaphore. When the running coroutine finishes or wishes to sleep, it releases the calling thread's semaphore and goes back to waiting on it's own (greenthreads couldn't be used as eventlet itself is built off of them leading to unpredictable results).

The end result is that only one thread of execution ever executes at a time, and that time.sleep *must* be called for events to proceed in fake mode. For example, if you did a busy wait by simply polling for a resource nothing would ever happen and the tests would hang.

== Repl Demos ==

It can be easier to understand how this works if you run a repl to experiment.

Start by entering your Trove directory and running the tests using tox, by running "tox -epy27".

Once they finish, run the following:

.tox/py27/bin/python run_tests.py --stop --group=dbaas.guest.initialize --nocapture --repl

This will execute the run_tests.py script directly. The --repl tells the code to enter an interactive REPL loop as soon as the tests finish. Because the group "nadda" does not exist no tests will be executed. However it can be useful to run a REPL if you're trying to debug test code.

When the repl starts, enter these lines:

>>> from trove.tests.util import test_config
>>> from trove.tests.util import create_dbaas_client
>>> from trove.tests.util.users import Requirements
>>> reqs = Requirements(is_admin=True)
>>> user = test_config.users.find_user(reqs)
>>> client = create_dbaas_client(user)
>>> instance = client.instances.create("Test Instance", 1, {'size':1})
>>> instance.status
u'BUILD'

The last line should cause the status of the instance to be spit out, which will be building.

Try this:

>>> instance = client.instances.get(instance.id)
>>> instance.status
u'BUILD'

Notice no matter how long you run those last two lines in real life, the status will always be building.

Now try this:

>>> import time
>>> time.sleep(3)
CREATING os_admin @ %
>>> instance = client.instances.get(instance.id)
>>> instance.status
u'BUILD'
>>> time.sleep(30)
>>> instance = client.instances.get(instance.id)
>>> time.sleep(30)
>>> instance.status
u'ACTIVE'

The Trove instance eventually does become active, but only after time.sleep, which in this context is just calling the event simulator, is called either enough times or with a high enough argument to pass enough simulated time.

You can use the repl like this in order to explore how Trove works internally and better understand it, which can help not only with testing but also when developing new features.

Here's another example you can paste into the repl which uses the Trove management API to show how the volume, server, and finally guest agent comes online to turn a Trove instance status to ACTIVE:

import time
from trove.tests.util import test_config
from trove.tests.util import create_client
from trove.tests.util.users import Requirements
client = create_client(is_admin=False)
admin_client = create_client(is_admin=True)
instance = client.instances.create("Test Instance", 1, {'size':1})

def print_status():
mgmt_instance = admin_client.management.show(instance.id)
vs = ("None" if mgmt_instance.volume is None
else mgmt_instance.volume['status'])
ns = ("None" if mgmt_instance.server is None
else mgmt_instance.server['status'])
print("Trove API Status = %s" % mgmt_instance.status)
print(" volume status = %s" % vs)
print(" server status = %s" % ns)
print(" agent status = %s" % mgmt_instance.service_status)

print_status()
time.sleep(1)
print_status()
time.sleep(1)
print_status()
time.sleep(1)
print_status()

Trove/ExampleGenerator

2014-08-21T13:23:21Z

Tim Simpson: Generate the API doc request / response snippets

== Description ==
The API docs have a bunch of snippets featuring the JSON request and respond headers and body. Currently these are made by manually with no validation that they resemble what Trove is actually doing.

We can instead generate these docs using simple Python code. Rackspace has used such code for about two years but was unable to submit it as until recently there was no place for it to live. However during the Austin midcycle (now half a year ago) we asked Anne Gentle, guest speaker, if it would be acceptable to move the Trove API docs to the main Trove repo. Through the hard work of Anne, Andreas Jaeger, and others the docs were moved. Following up on that we can finally move the doc generation tool to Trove.

The snippet generator is all Proboscis (sorry) but this means it will fit in with all the tests run with run_tests.py (yay!). We can have it write the snippets to apidocs/src/samples, meaning if something changes due to work by a developer they'll know it and have to check it in to avoid errors in the gate. This will ensure we don't change our API unless we really want to. :)

Here's the breakdown of the work:

* Add snippet generator to tests, let it run every time.
* This is tangential but we'll probably want to remove all the XML snippets, as well as all mention of XML from the docs, to help with this next part:
* In its current form, the snippet generator doesn't generate all snippets (there have been more created since we last updated it) so we'll have to add additional generated snippets until finally *all* the snippets are generated.

==Public API==
No changes- except our contract will be even less likely to change.

==CLI interface==
No changes.

==ReST Part==
No changes.

==Internal API==
No changes.

==RPC API description==
No changes.

==Guest Agent==
No changes.

Meetings/TroveBPMeeting

2014-05-05T17:31:12Z

Tim Simpson: /* BluePrints for next meeting */

= Weekly Trove Blueprint Meeting =
For those interested, we have blueprint meetings in #openstack-trove weekly, Mondays at 18:00 UTC. Feel free to add items in the agenda below.

=== Agenda for Weekly Trove Blueprint Meeting - May 5 ===
* Percona support [mattgriffin]:
** https://blueprints.launchpad.net/trove/+spec/support-percona-xtrabackup-2.2
** https://blueprints.launchpad.net/trove/+spec/support-ps-5.6
** https://blueprints.launchpad.net/trove/+spec/support-pxc-5.5-and-5.6
* Instance database log manipulations [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/dbinstance-log
* Cassandra configuration management support [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/cassandra-configuration-management
* Cassandra incremental backup support [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/incremental-backup-restore-cassandra
* Volume data snapshoting/applying [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
* Update database instance name [nehav]
** https://blueprints.launchpad.net/trove/+spec/update-instance-name
 

=== BluePrints for next meeting ===
* Pluggable conductor manager [boden]
** https://blueprints.launchpad.net/trove/+spec/pluggable-conductor-manager
* Configurable DB plugins [boden]
** https://blueprints.launchpad.net/trove/+spec/configurable-db-plugins
* Multi-path plugin support; added notes to existing BP [boden]
** https://blueprints.launchpad.net/trove/+spec/extensions-update
* Allow configs to be rendered based on datastore version [cp16net]
** https://blueprints.launchpad.net/trove/+spec/datastore-configs-cant-use-multiple-versions
* Trove client for cross-region-backup [esmute]
** https://blueprints.launchpad.net/trove/+spec/cross-region-backup-availability
 

Meeting Agenda History: https://wiki.openstack.org/wiki/Trove/MeetingAgendaHistory#Trove_Blueprint_Meeting_Agenda_History

Meetings/TroveBPMeeting

2014-05-01T13:22:35Z

Tim Simpson: /* BluePrints for next meeting */

Trove/use-datastore-version-when-considering-configs

2014-05-01T13:14:43Z

Tim Simpson: /* Justification/Benefits */

== Description ==
Trove currently can use different config files for each datastore, but uses the same configs for each datastore version. This blueprint is to add the tiny bit of functionality to allow an operator to optionally use different configs each datastore version.

== Justification/Benefits ==
As versions of a datastore change, it makes sense to allow operators to optionally offer different config values for each version.

== Impacts ==

=== Configuration ===
Today a template is loaded by looking in the path "{manager}/config.template".format(manager=datastore.manager).
We can allow for this new functionality by creating instead a list option in the configs which contains different name patterns to look for. Each one will be tried in order to see if there's a match. I propose these defaults:
['{manager}/{version}/{flavor_ram}.config.template',
'{manager}/{version}/config.template',
'{manager}/config.template']

=== Database ===
No impact.

=== Public API ===
No impact.

==== CLI interface====
No impact.

==== ReST Part ====
No impact.

=== Internal API ===
No impact.

==== RPC API description====
No impact.

=== Guest Agent ===
No impact.

Trove/use-datastore-version-when-considering-configs

2014-05-01T13:13:10Z

Tim Simpson: Created page with "== Description == Trove currently can use different config files for each datastore, but uses the same configs for each datastore version. This blueprint is to add the tiny bi..."

== Description ==
Trove currently can use different config files for each datastore, but uses the same configs for each datastore version. This blueprint is to add the tiny bit of functionality to allow an operator to optionally use different configs each datastore version.

== Justification/Benefits ==
As versions of a datastore change, it makes sense to offer different config values.

== Impacts ==

=== Configuration ===
Today a template is loaded by looking in the path "{manager}/config.template".format(manager=datastore.manager).
We can allow for this new functionality by creating instead a list option in the configs which contains different name patterns to look for. Each one will be tried in order to see if there's a match. I propose these defaults:
['{manager}/{version}/{flavor_ram}.config.template',
'{manager}/{version}/config.template',
'{manager}/config.template']

=== Database ===
No impact.

=== Public API ===
No impact.

==== CLI interface====
No impact.

==== ReST Part ====
No impact.

=== Internal API ===
No impact.

==== RPC API description====
No impact.

=== Guest Agent ===
No impact.

Meetings/TroveMeeting

2014-04-09T17:50:32Z

Tim Simpson: /* Agenda for Apr. 9 (weekly meeting) */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 18:00 UTC. Feel free to add items in the agenda below.

Logs available: http://eavesdrop.openstack.org/meetings/trove/2014/

== Agenda for Mar. 19 ==
* https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
* Moving away from mockito (https://blueprints.launchpad.net/trove/+spec/remove-mockito)
* Clustering API Container vs Joins approach.

== Agenda for Mar. 26 ==
* Summit sessions [hub_cap]
* Data Store abstraction start/stop/status/control [amrith]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-agent-datastore-control
* Icehouse, RC1, and juno

== Agenda for Mar. 31 (blueprint meeting) ==
Networking related blueprints:
* Network attribute management [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Neutron Support for Trove [annashen]
** https://blueprints.launchpad.net/trove/+spec/neutron-support
* Trove Managed Instances on Private Network [juice]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-private-network
Others:
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Instance metadata [imsplitbit]
** https://blueprints.launchpad.net/trove/+spec/trove-metadata

== Agenda for Apr. 2 ==
*Icehouse RC1 cut and Juno branch open
*ATL Dev Summit topics
*Open discussion

== Agenda for Apr. 7 (blueprint meeting) ==
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
*Cross-region backups [esmute]
** https://blueprints.launchpad.net/trove/+spec/cross-region-backup-migration
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Neutron Network Support [annashen]
**https://blueprints.launchpad.net/trove/+spec/neutron-support
*Trove-Managed Instances [juice]
**https://blueprints.launchpad.net/trove/+spec/trove-managed-instances

== Agenda for Apr. 9 (weekly meeting)==
* Design Summit Session Proposals
* Refactoring backup/restore strategies [denis_makogon]
** https://bugs.launchpad.net/trove/+bug/1287684
* Moving the docs [grapex]

== Agenda for Apr. 14 (blueprint meeting) ==
* Datastore Capabilities [k-pom]
** https://blueprints.launchpad.net/trove/+spec/capabilities
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Move the Trove Guest Agent to its own module [robert]
** https://blueprints.launchpad.net/trove/+spec/moving-trove-guestagent

Meetings/TroveMeeting

2014-04-09T17:50:17Z

Tim Simpson: /* Agenda for Apr. 9 (weekly meeting) */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 18:00 UTC. Feel free to add items in the agenda below.

Logs available: http://eavesdrop.openstack.org/meetings/trove/2014/

== Agenda for Mar. 19 ==
* https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
* Moving away from mockito (https://blueprints.launchpad.net/trove/+spec/remove-mockito)
* Clustering API Container vs Joins approach.

== Agenda for Mar. 26 ==
* Summit sessions [hub_cap]
* Data Store abstraction start/stop/status/control [amrith]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-agent-datastore-control
* Icehouse, RC1, and juno

== Agenda for Mar. 31 (blueprint meeting) ==
Networking related blueprints:
* Network attribute management [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Neutron Support for Trove [annashen]
** https://blueprints.launchpad.net/trove/+spec/neutron-support
* Trove Managed Instances on Private Network [juice]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-private-network
Others:
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Instance metadata [imsplitbit]
** https://blueprints.launchpad.net/trove/+spec/trove-metadata

== Agenda for Apr. 2 ==
*Icehouse RC1 cut and Juno branch open
*ATL Dev Summit topics
*Open discussion

== Agenda for Apr. 7 (blueprint meeting) ==
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
*Cross-region backups [esmute]
** https://blueprints.launchpad.net/trove/+spec/cross-region-backup-migration
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Neutron Network Support [annashen]
**https://blueprints.launchpad.net/trove/+spec/neutron-support
*Trove-Managed Instances [juice]
**https://blueprints.launchpad.net/trove/+spec/trove-managed-instances

== Agenda for Apr. 9 (weekly meeting)==
* Refactoring backup/restore strategies [denis_makogon]
** https://bugs.launchpad.net/trove/+bug/1287684
* Moving the docs

== Agenda for Apr. 14 (blueprint meeting) ==
* Datastore Capabilities [k-pom]
** https://blueprints.launchpad.net/trove/+spec/capabilities
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Move the Trove Guest Agent to its own module [robert]
** https://blueprints.launchpad.net/trove/+spec/moving-trove-guestagent

Meetings/TroveMeeting

2014-04-09T17:40:57Z

Tim Simpson: /* Agenda for Apr. 14 (blueprint meeting) */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 18:00 UTC. Feel free to add items in the agenda below.

Logs available: http://eavesdrop.openstack.org/meetings/trove/2014/

== Agenda for Mar. 19 ==
* https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
* Moving away from mockito (https://blueprints.launchpad.net/trove/+spec/remove-mockito)
* Clustering API Container vs Joins approach.

== Agenda for Mar. 26 ==
* Summit sessions [hub_cap]
* Data Store abstraction start/stop/status/control [amrith]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-agent-datastore-control
* Icehouse, RC1, and juno

== Agenda for Mar. 31 (blueprint meeting) ==
Networking related blueprints:
* Network attribute management [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Neutron Support for Trove [annashen]
** https://blueprints.launchpad.net/trove/+spec/neutron-support
* Trove Managed Instances on Private Network [juice]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-private-network
Others:
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Instance metadata [imsplitbit]
** https://blueprints.launchpad.net/trove/+spec/trove-metadata

== Agenda for Apr. 2 ==
*Icehouse RC1 cut and Juno branch open
*ATL Dev Summit topics
*Open discussion

== Agenda for Apr. 7 (blueprint meeting) ==
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
*Cross-region backups [esmute]
** https://blueprints.launchpad.net/trove/+spec/cross-region-backup-migration
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Neutron Network Support [annashen]
**https://blueprints.launchpad.net/trove/+spec/neutron-support
*Trove-Managed Instances [juice]
**https://blueprints.launchpad.net/trove/+spec/trove-managed-instances

== Agenda for Apr. 9 (weekly meeting)==
* Refactoring backup/restore strategies [denis_makogon]
** https://bugs.launchpad.net/trove/+bug/1287684

== Agenda for Apr. 14 (blueprint meeting) ==
* Datastore Capabilities [k-pom]
** https://blueprints.launchpad.net/trove/+spec/capabilities
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Move the Trove Guest Agent to its own module [robert]
** https://blueprints.launchpad.net/trove/+spec/moving-trove-guestagent

Meetings/TroveMeeting

2014-04-09T17:33:05Z

Tim Simpson: /* Agenda for Apr. 14 (blueprint meeting) */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 18:00 UTC. Feel free to add items in the agenda below.

Logs available: http://eavesdrop.openstack.org/meetings/trove/2014/

== Agenda for Mar. 19 ==
* https://blueprints.launchpad.net/trove/+spec/refactoring-datastore-options-in-cfg
* Moving away from mockito (https://blueprints.launchpad.net/trove/+spec/remove-mockito)
* Clustering API Container vs Joins approach.

== Agenda for Mar. 26 ==
* Summit sessions [hub_cap]
* Data Store abstraction start/stop/status/control [amrith]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-agent-datastore-control
* Icehouse, RC1, and juno

== Agenda for Mar. 31 (blueprint meeting) ==
Networking related blueprints:
* Network attribute management [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Neutron Support for Trove [annashen]
** https://blueprints.launchpad.net/trove/+spec/neutron-support
* Trove Managed Instances on Private Network [juice]
** https://blueprints.launchpad.net/trove/+spec/trove-guest-private-network
Others:
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Instance metadata [imsplitbit]
** https://blueprints.launchpad.net/trove/+spec/trove-metadata

== Agenda for Apr. 2 ==
*Icehouse RC1 cut and Juno branch open
*ATL Dev Summit topics
*Open discussion

== Agenda for Apr. 7 (blueprint meeting) ==
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
*Cross-region backups [esmute]
** https://blueprints.launchpad.net/trove/+spec/cross-region-backup-migration
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Neutron Network Support [annashen]
**https://blueprints.launchpad.net/trove/+spec/neutron-support
*Trove-Managed Instances [juice]
**https://blueprints.launchpad.net/trove/+spec/trove-managed-instances

== Agenda for Apr. 9 (weekly meeting)==
* Refactoring backup/restore strategies [denis_makogon]
** https://bugs.launchpad.net/trove/+bug/1287684

== Agenda for Apr. 14 (blueprint meeting) ==
* Datastore Capabilities [k-pom]
** https://blueprints.launchpad.net/trove/+spec/capabilities
*Descriptions to Datastore Configuration Group Parameters [cp16net]
** https://blueprints.launchpad.net/trove/+spec/add-descriptions-to-config-parameters-api
*Categorize the trove-manage commands [cp16net]
** https://blueprints.launchpad.net/trove/+spec/categorize-the-trove-manage-commands
*Point in time recovery [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/point-in-time-recovery
*Data volume snapshot [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/volume-snapshot
*Networking [denis_makogon]
** https://blueprints.launchpad.net/trove/+spec/network-manager-spec
* Move the Trove Guest Agent to its own module
** https://blueprints.launchpad.net/trove/+spec/moving-trove-guestagent

Trove-Guest-Agent-Upgrades

2014-03-10T03:55:44Z

Tim Simpson: /* Feedback/Discussion */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX|grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.
* [[User:GrapeX|grapex]] I think the rule should be that we keep things flexible enough that operators can use whatever upgrade strategies works best for them. Whatever goes into CI should be very simple IMO.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.
* [[User:GrapeX|grapex]] The table is called "upgrade" but it seems to me the more useful bit of information to store in the database is the current version of the guest agent as reported by the agent itself during it's heartbeat / status update. We could store that along with a new status which would be, more generally, the status of the guest agent itself. This new status would be set to error states if the upgrade was botched, which the user wouldn't see but admins could peek at. Why not call the table "agent_status"?

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

* [[User:GrapeX|grapex]] I'll be honest- I feel like there are dozens of features in Trove which could use something like this to compliment the logs. I don't think building this audit trail as part of this feature makes sense- we should probably set our sights on making a real audit trail that would be more general. We talked awhile back about having "events" in the database, and Basnight also worked on having something similar awhile back which never was completed.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

* [[User:GrapeX|grapex]]: I agree with Amcrn- rollbacks is very contingent on the packaging schemes. At Rackspace we're using Debian packages, which works well. I'm for supporting whatever people need to use but I think we should try to avoid inventing a new packaging system. At the very least I think the issue of rollbacks should be saved for a later iteration.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

==== Non-Admin Call Initiated Upgrades ====
* [[User:GrapeX|grapex]]: I don't see any talk here about the idea of allowing Trove code to make the upgrade RPC call automatically if it detects an agent is out of date. If we stored the version of the guest in the database it would be very easy to detect this during any of the guest RPC calls. Trove could even wait for the upgrade to finish if it was in the middle of a task that was already long running (such as anything deferred to task manager). In my experience the upgrade time for a guest agent using debian packages is very small, and during an upgrade only a fraction of users are actually hitting the API, so it would be enormously helpful to be able to upgrade the users who need the newer version the most.

Trove-Guest-Agent-Upgrades

2014-03-10T03:48:26Z

Tim Simpson: /* Rollback Support */

Trove-Guest-Agent-Upgrades

2014-03-10T03:43:15Z

Tim Simpson: /* Trove Database Schema */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX|grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.
* [[User:GrapeX|grapex]] I think the rule should be that we keep things flexible enough that operators can use whatever upgrade strategies works best for them. Whatever goes into CI should be very simple IMO.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.
* [[User:GrapeX|grapex]] The table is called "upgrade" but it seems to me the more useful bit of information to store in the database is the current version of the guest agent as reported by the agent itself during it's heartbeat / status update. We could store that along with a new status which would be, more generally, the status of the guest agent itself. This new status would be set to error states if the upgrade was botched, which the user wouldn't see but admins could peek at. Why not call the table "agent_status"?

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

* [[User:GrapeX|grapex]] I'll be honest- I feel like there are dozens of features in Trove which could use something like this to compliment the logs. I don't think building this audit trail as part of this feature makes sense- we should probably set our sights on making a real audit trail that would be more general. We talked awhile back about having "events" in the database, and Basnight also worked on having something similar awhile back which never was completed.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

Trove-Guest-Agent-Upgrades

2014-03-10T03:40:24Z

Tim Simpson: /* Upgrade Status */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX|grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.
* [[User:GrapeX|grapex]] I think the rule should be that we keep things flexible enough that operators can use whatever upgrade strategies works best for them. Whatever goes into CI should be very simple IMO.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.
* [[User:GrapeX|grapex]] The table is called "upgrade" but it seems to me the more useful bit of information to store in the database is the current version of the guest agent as reported by the agent itself during it's heartbeat / status update. We could store that along with a new status which would be, more generally, the status of the guest agent itself. This new status would be set to error states if the upgrade was botched, which the user wouldn't see but admins could peek at. Why not call the table "agent_status"?

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

Trove-Guest-Agent-Upgrades

2014-03-10T03:35:17Z

Tim Simpson: /* Supported Packaging Types */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX|grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.
* [[User:GrapeX|grapex]] I think the rule should be that we keep things flexible enough that operators can use whatever upgrade strategies works best for them. Whatever goes into CI should be very simple IMO.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

Trove-Guest-Agent-Upgrades

2014-03-10T03:33:28Z

Tim Simpson: /* Configs */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX|grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

Trove-Guest-Agent-Upgrades

2014-03-10T03:32:13Z

Tim Simpson: /* Configs */

=== Introduction ===
This article describes the design for Guest Agent upgrades in Trove. Currently Guest Agent upgrades are implemented through external deployment tools that push new code to each guest instance. Usually the same deployment tools for upgrading the control plane handles guest agent upgrades. This can create a bottle neck on the deployment infrastructure.

==== Blueprint ====
https://blueprints.launchpad.net/trove/+spec/upgrade-guestagent

=== Goals ===
# Version the RPC API and tie it to the API version (see nova for examples)
#* This is to help prevent non-backward compatibility between the Trove API and the guest but is not necessarily a dependency for upgrades
# Implement a notification based upgrade path for guest agent
# Allow for different upgrade strategies (swift, jenkins, local disk, rysnc, etc)
# Avoid upgrading during times when guest agents are doing other work (i.e. backups, resize, restart)
#* This doesn't seem to be a concern given that the agent can only handle a single message at a time
# Reduce overall downtime during upgrade cycles

=== Description ===

==== Configuration ====
New properties will be added to the trove configs to allow:
* Enabling/Disabling Guest Agent Upgrades
<pre>
guest_automatic_updates: False
</pre>
* Specifying an upgrade strategy
<pre>
guest_upgrade_strategy: swift
</pre>

==== Affected Trove Components ====
* python-troveclient (optional)
* trove admin API
* guest agent

==== Schema Changes ====
[[File:Upgrades.png]]

==== Workflow ====
1. An external process (outside of Trove) will create an upgrade package or artifact
* This process will be mapped to a defined strategy in the Guest Agent.
* It's possible the admin may want to trigger an automatic backup at this point
[[File:Workflow0.png]]
2. An Admin user will notify a Guest Agent that an upgrade is available through the Trove Management API
3. A new upgrade record will be create in the DB for the request.
4. The Guest Agent will process the RPC message created by the API call and handle the upgrade accordingly
[[File:Workflow1.png]]

5. The Guest Agent will download the package from the location specified in the RPC message
[[File:Workflow2.png]]

==== Guest Agent Message Handling ====
1. Guest agent will handle the message for upgrading to a particular version
2. Simple validation on the message will occur after the message is parsed
a. Check the strategy type, check to see if guest_automatic_updates is enabled
b. Check the location of the package in the message and whether it exists
3. Update the upgrade 'event' in the Trove upgrades database table
4. Execute or process the message

[[File: Message_handling.png]]

==== Guest Agent Process Upgrade Message ====
1. Choose the correct strategy to process the upgrade
2. Download the file from the given location (retry n-times before Failing)
3. Decrypt the package
4. Validate the package
a. check size, check version, checksum, format etc
5. Install the package (pip install)
6. Restart (retry n-times before Failing)

[[File: Ga_processing.png]]

* on start up update the status of the upgrade to SUCCESS
** if start up fails try to install the last known working version, record the status as FAILED
I can see this being a config value that gets updated or a file that is written to disk on the instance

==== Trove Management REST API ====
Create a notification request to upgrade a trove guest agent
<pre>
Relative URL: /v1.0/{admin_tenant_id}/mgmt/upgrades

HTTP Method: POST

HTTP Headers:
Accept: application/json
Content-Type: application/json
User-Agent: python-troveclient
X-Auth-Project-Id: tenant_name
X-Auth-Token: HPAuth10_xxxx

HTTP Post Body
{
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>
* Note: Will need a call return a list of upgrades filtered by state and/or instance_id

==== Trove RPC API ====
unpacked context
<pre>
{
...
"is_admin": True,
"tenant":"<SANITIZED>",
"method": "upgrade",
"instance_id": '<UUID>'
"instance_version": "v1.0.1",
"strategy": "swift",
"location": "http://swift/tenant/container/trove-guestagent-v1.0.1.tar.gz"
}
</pre>

==== Versioning and Package Validation ====
* The Guest Agent will be responsible for validating the package before upgrading

==== Scenarios ====
# What happens when the Guest Agent is in a non-upgradeable state? (backup/restore, resize, restart, error)
#* The message should remain in the queue until the next time the Guest Agent checks and the state is in 'Running'
# What happens when an upgrade fails, and how does that feedback to Trove?
#* Record it as a FAIL in the Trove Database, Admin will have to query.
# Can we rollback or install a previous version?

=== Feedback/Discussion ===

==== Configs ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): "New properties will be added to the trove configs to allow Enabling/Disabling Guest Agent Upgrades and Specifying an upgrade strategy". Is it necessary to add a CONF switch for either of these? If the cloud admin doesn't want to initiate a guestagent upgrade via API/RPC, then as long as they don't issue the request, all is well. Or are you suggesting that users can initiate their own upgrades (i.e. this operation isn't limited to the admin role)? As for the upgrade strategy (e.g. "swift"), this is already in the message payload in your examples, so why the need for the CONF? Is the idea to mimic the 'datastore_registry_ext' concept to allow providers to write and add their own strategies?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): We could leave out the CONF flag for disable/enable upgrades. I'm a little hesitant about removing the strategy key/value though. For instance if for some reason we have different strategies for upgrading guests in dev or staging one day. For now I think we are only considering Admin users for sending upgrade notifications. I think the workflow might resemble the restore from backup concept a little more but yes we want to allow folks to extend this to define their own strategies.
*[[User:Esp|esp]]: I thought of another use for the CONF switch to enable/disable. We might want to use it for disabling REST API calls that are specific to upgrades.
*[[User:GrapeX:grapex]]: Just to make sure, when we talk about a strategy for the guest upgrade this will be on the Trove side, yes?

==== Supported Packaging Types ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Is there a short list of packaging schemes everyone believes we should support? In Austin it looked like some were ok with a simple "pip install", whereas others had strict requirements on package signing, crypto, etc.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Initially we are looking at creating a tarball that would include all dependencies to avoid pulling down other packages. We could come up with a list though. (pypi, tar.gz, etc) The crypto piece should be optional and part of a strategy. The concern was having credentials unencrypted in swift.

==== Upgrade Status ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): It seems this is inferred, but since it's not explicitly mentioned I'll ask: There is no introduction of a new INSTANCE state (e.g. ACTIVE, BACKUP), correct? To take it a step further then, the idea would be that the user sees ACTIVE, despite a possible in-flight upgrade? In the case of an upgrade failure, would the user still see ACTIVE but we'd have FAILED recorded in the upgrade table?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The thought was to 'hide' the upgrade status from the user although I'm not sure we decided that yet. That being said I think we need to create a set of upgrade states that are granular enough to handle various failures. For example if the instance can't talk to swift due to network problem we could just fail/abort the upgrade. An Admin will have to come back and retry again later. The user should not see any issues. But if we botch the install and the guest can't start at all then we got a big problem that the user will see.

==== Trove Database Schema ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Every upgrade attempt will be a new record in the upgrades table, or will each instance have a dedicated row? I would much prefer the former.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): I was thinking a new record for each upgrade event. Kind of like an audit trail or upgrade history table.

* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): What is the purpose of deleted/deleted_at for upgrades? In the scenario of a botched upgrade, a new upgrade request should be fired, incurring a new row to be inserted.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): The deleted flag was put there to stay consistent with some of the existing tables. In the event that we might want to purge the table that would support a logical delete for a batch delete job that would run once in a while. I'd be willing to drop both of these columns as I can't see a scenario for it at the moment.

==== Rollback Support ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): As for whether a rollback to a previous install should be supported: that's likely contingent upon the packaging schemes supported.
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): Agreed. This one is a bit weird. It was my attempt to address the question of what happens when we upgrade and the guest can't start. My current approach was to write to a file (like the config file or a new one) when an upgrade is successful to keep track of that last known running version.

* [[User:Agingh1pster|agingh1pster]]: question 2 - what happens if a agent never upgrades. Is the instance considered dead?
* [[User:Esp|esp]]: I think that if we got into a situation where a Guest Agent failed to update (indefinitely) the cloud admin would have to figure out something else. i.e. contact the user, migrate the instance via backup/restore etc.

==== Instance Version ====
* [[User:Amcrn|amcrn]] ([[User talk:Amcrn|talk]]) 22:56, 4 March 2014 (UTC): Can you elaborate on the instance_version logic?
* [[User:Esp|esp]] ([[User talk:Esp|talk]]) 23:06, 4 March 2014 (UTC): This was kind of a convenience thing to be able to track the instance_version through an entire upgrade cycle. It saves us from parsing the name of the package/file. At first I was thinking, maybe you need to be at a particular version before upgrading. So if for some reason a guest agent is at version 1, version 2 failed but the next upgrade available is version 3. Not sure this is a valid case or not, especially if we just make sure that version 3 has everything it needs from version 1 and 2. Not to mention that if discover that version 2 was crap and we want to install version 3 instead that would definitely not be valid.

==== Trove Managment API ====
*[[User:Agingh1pster|agingh1pster]]: I don't see api for getting the status of the agent upgrades
*[[User:Esp|esp]]: good point. I was gonna add that in but forgot. it would have to be a management API addition if we are really gonna hide the upgrade status from the user
*[[User:Esp|esp]]: We can update the existing /mgmt/instance route (the one that shows the details of the instance). In addition to this we can create a new route that will allow an admin to list all the failed upgrades. Need to document what this might look like.

OpenPGP Web of Trust/Trove Icehouse Sprint

2014-02-19T21:06:37Z

Tim Simpson: /* Trove Mid-Cycle Icehouse Sprint Key Signing */

= Trove Mid-Cycle Icehouse Sprint Key Signing =
If you plan to join us, please bring one or more forms of photo identification, and make sure the real name on your key corresponds to the name on your ID. If you want additional tips, see the [[OpenPGP Web of Trust#Key Signing Process|parent article]].

If you want to bring cards or pieces of paper with your name and key fingerprint then feel free, but if you add your information to the table below we will do our best to bring paper copies (at least enough for the people who are listed here) for use as a checklist so that you won't need to use your computer. Please bring extra pens or pencils however, if you can, since there may not be many of those to go around.

{| class="wikitable"
|+ People participating in identification exchange
! Real Name !! IRC Nick !! Key Fingerprint !! Preferred E-mail Address
|-
| Nikhil Manchanda || slicknik || 5674 7211 71F8 CD59 F285 353D 9692 757E BCD3 D66C || nikhil@manchanda.me
|-
| Steve Leon || esmute || 02F1 F2AE C17D 66D2 1B1B B75C EF3B 2AB9 FFF4 F9EA || steve.leon@hp.com
|-
| Justin Hopper || juice || 7463 498F 7D3B 7DB3 FFBF 4946 C02F 046F '''EA23 8CF3''' || justin.hopper@me.com
|-
|Michael Basnight||hub_cap||9D60 7133 8AFE 4FE5 45D9 625E 9E86 DD8C 3DB5 897B||mbasnight@gmail.com
|-
|Craig Vyvial||cp16net||8975 C9C2 E879 309A 771B 7C01 046B FABA F4BD 6CC2||cp16net@gmail.com
|-
|Dan Nguyen||esp||D9D4 F72E 82FC EFCE 1923 B140 9163 F056 AE67 298E||dan.nguyens.mail@gmail.com
|-
|Daniel Salinas||imsplitbit||54C5 515E 71FB 1533 2332 F115 78EE B83C 964B C17C||imsplitbit@gmail.com
|-
|Andrew Bramley||abramley||7082 00CC 9C6F 7523 C3B1 2CB4 9E67 CA34 2DAA 044F||andrew@tesora.com
|-
|Amrith Kumar||amrith||0D22 859E BE93 63F5 9DD6 35BC 5E48 849A 9D21 A29B||amrith@tesora.com
|-
|Paul Marshall||pdmars||2DCA A0AB 6DEC A717 FC80 A6D1 9F08 2613 795C B691||paul.marshall@rackspace.com
|-
|Vipul Sabhaya||vipul||B7FA 44B3 32C7 1A3A F606 1ED7 A2C7 95BC 02AE 77A2 ||vipul.sabhaya@hp.com
|-
|Doug Shelley||dougshelley66||95FB 1D3D 900D 67F9 FDA9 C378 6B34 FD08 BA22 5C73||doug@tesora.com
|-
|Joe Cruz||jcru||A266 E400 2C05 5F0F 331E 06E6 6CB6 5D33 A00B FEE9||joe.cruz@rackspace.com
|-
|Alex Tomic || atomic77 || C503 BE83 65E4 2672 8BB7 AA1B E480 0C76 EE00 E285 || tomica@usi.ch
|-
|Tim Simpson||grapex||C5A1 91E9 A6E5 AD3F 348C E8FE 7933 CC8C 3839 0CA3||timsimpson4gmail.com
|-}

Trove/IcehouseCycleMeetup

2014-02-17T20:32:39Z

Tim Simpson: /* Event Details */

= Database Program (Trove) - Icehouse Mid-cycle Meetup =

== What ==

There will be a mid-cycle meetup for ATCs (Active Technical Contributors) to the OpenStack Database program in February, 2014. The final schedule of topics will be determined by those that attend, shortly before the event.

== When ==

3 days, February 19-21, 2014

== Where ==

[http://capitalfactory.com/ capital factory]
701 Brazos Street
Suite 1601
Austin, TX 78701

== Event Details ==
{| class="wikitable"
|-
! Time !! Wed !! Thr !! Fri
|-
| 9-11 || Meets -n- greets || State of Tempest || Conductor next steps
|-
| 11-1 || Show -n- tell icehouse || State of Heat || Agent updates
|-
| 1-3 || Current state of blueprints || Clustering/Replication || Capabilities
|-
| 3-5 || Doc support || Clustering/Replication || MVC / Pecan / WSME
|}

Wed Night - Tesora is sponsoring an event at http://www.maggiemaesaustin.com , food and drink provided. Super fun time!!

Thr Night - Rax/HP are sponsoring an event at http://www.buffalobilliards.com/austin/ , food and drink provided. Bring quarters!!

=== Key Signing Party ===
More details at: https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust/Trove_Icehouse_Sprint

=== Registration ===
** Please confirm your attendance ASAP at https://troveicehousemeetup.eventbrite.com
=== Hotels ===
Hotels close to the event (closest on the top of the list). Please note the prices are subject to change and are there only because i clicked thru the reservation system for each hotel!

* [http://www.omnihotels.com/findahotel/austindowntown.aspx omni] - from $260
* [http://www.driskillhotel.com/ driskill] - from $280
* [http://austin.intercontinental.com/ interncontinental] - from $200
* [http://goo.gl/maps/uLZOw We are smack dab in the middle of downtown austin] - from $???

'''If you are in Austin and can provide a discount code / block for rooms, please contact me (hub_cap)'''

=== Travel ===
* Recommended airport: AUS, Austin-Bergstrom Intl

== Things to do ==

Most of the RAX Trove team is from austin. Just ask someone and they can help. When in doubt, talk to hub_cap.

Trove/IcehouseCycleMeetup

2014-02-17T20:31:34Z

Tim Simpson: /* Event Details */

= Database Program (Trove) - Icehouse Mid-cycle Meetup =

== What ==

There will be a mid-cycle meetup for ATCs (Active Technical Contributors) to the OpenStack Database program in February, 2014. The final schedule of topics will be determined by those that attend, shortly before the event.

== When ==

3 days, February 19-21, 2014

== Where ==

[http://capitalfactory.com/ capital factory]
701 Brazos Street
Suite 1601
Austin, TX 78701

== Event Details ==
{| class="wikitable"
|-
! Time !! Wed !! Thr !! Fri
|-
| 9-11 || Meets -n- greets || State of Tempest || Conductor next steps
|-
| 11-1 || Show -n- tell icehouse || State of Heat || Guest updates
|-
| 1-3 || Current state of blueprints || Clustering/Replication || Capabilities
|-
| 3-5 || Doc support || Clustering/Replication || MVC / Pecan / WSME
|}

Wed Night - Tesora is sponsoring an event at http://www.maggiemaesaustin.com , food and drink provided. Super fun time!!

Thr Night - Rax/HP are sponsoring an event at http://www.buffalobilliards.com/austin/ , food and drink provided. Bring quarters!!

=== Key Signing Party ===
More details at: https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust/Trove_Icehouse_Sprint

=== Registration ===
** Please confirm your attendance ASAP at https://troveicehousemeetup.eventbrite.com
=== Hotels ===
Hotels close to the event (closest on the top of the list). Please note the prices are subject to change and are there only because i clicked thru the reservation system for each hotel!

* [http://www.omnihotels.com/findahotel/austindowntown.aspx omni] - from $260
* [http://www.driskillhotel.com/ driskill] - from $280
* [http://austin.intercontinental.com/ interncontinental] - from $200
* [http://goo.gl/maps/uLZOw We are smack dab in the middle of downtown austin] - from $???

'''If you are in Austin and can provide a discount code / block for rooms, please contact me (hub_cap)'''

=== Travel ===
* Recommended airport: AUS, Austin-Bergstrom Intl

== Things to do ==

Most of the RAX Trove team is from austin. Just ask someone and they can help. When in doubt, talk to hub_cap.

Trove/IcehouseCycleMeetup

2014-02-17T20:23:54Z

Tim Simpson: /* Event Details */

= Database Program (Trove) - Icehouse Mid-cycle Meetup =

== What ==

There will be a mid-cycle meetup for ATCs (Active Technical Contributors) to the OpenStack Database program in February, 2014. The final schedule of topics will be determined by those that attend, shortly before the event.

== When ==

3 days, February 19-21, 2014

== Where ==

[http://capitalfactory.com/ capital factory]
701 Brazos Street
Suite 1601
Austin, TX 78701

== Event Details ==
{| class="wikitable"
|-
! Time !! Wed !! Thr !! Fri
|-
| 9-11 || Meets -n- greets || State of Tempest || Conductor next steps
|-
| 11-1 || Show -n- tell icehouse || State of Heat || Guest updates
|-
| 1-3 || Current state of blueprints || Clustering/Replication || Capabilities
|-
| 3-5 || Doc support || Clustering/Rehplication || Pecan / WSME
|}

Wed Night - Tesora is sponsoring an event at http://www.maggiemaesaustin.com , food and drink provided. Super fun time!!
Thr Night - Rax/HP are sponsoring an event at http://www.buffalobilliards.com/austin/ , food and drink provided. Bring quarters!!

=== Key Signing Party ===
More details at: https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust/Trove_Icehouse_Sprint

=== Registration ===
** Please confirm your attendance ASAP at https://troveicehousemeetup.eventbrite.com
=== Hotels ===
Hotels close to the event (closest on the top of the list). Please note the prices are subject to change and are there only because i clicked thru the reservation system for each hotel!

* [http://www.omnihotels.com/findahotel/austindowntown.aspx omni] - from $260
* [http://www.driskillhotel.com/ driskill] - from $280
* [http://austin.intercontinental.com/ interncontinental] - from $200
* [http://goo.gl/maps/uLZOw We are smack dab in the middle of downtown austin] - from $???

'''If you are in Austin and can provide a discount code / block for rooms, please contact me (hub_cap)'''

=== Travel ===
* Recommended airport: AUS, Austin-Bergstrom Intl

== Things to do ==

Most of the RAX Trove team is from austin. Just ask someone and they can help. When in doubt, talk to hub_cap.

Meetings/TroveMeeting

2014-01-29T17:57:27Z

Tim Simpson: /* Agenda for Jan 28nd meeting */

Meetings/TroveMeeting

2014-01-29T17:43:22Z

Tim Simpson: /* Agenda for Jan 22nd meeting */

Meetings/TroveMeeting

2013-12-18T17:41:51Z

Tim Simpson: /* Agenda for the next meeting */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 18:00 UTC. Feel free to add items in the agenda below.

== Agenda for the next meeting ==
* Datastore compat matrix
** https://wiki.openstack.org/wiki/Trove/DatastoreCompatibilityMatrix
* Trove DBInstance Log Operation
** https://wiki.openstack.org/wiki/TroveDBInstanceLogOperation
* Status of next meeting?
=== Open Discussion ===
* Review request
** https://review.openstack.org/#/c/62414/
** https://review.openstack.org/#/c/61969/
** https://review.openstack.org/#/c/61169/
** https://review.openstack.org/#/c/58845/

Meetings/TroveMeeting

2013-11-20T17:52:23Z

Tim Simpson: /* Agenda for the next meeting */

Meetings/TroveMeeting

2013-10-23T20:10:51Z

Tim Simpson: /* Agenda for the next meeting */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 3pm CDT / 1pm PDT, or 20:00 UTC. Feel free to add items in the agenda below.

== Agenda from the previous meeting ==
* User permission request escaping and dots in usernames (datsun180b, demorris)
* Provisioning ancillary resources post ACTIVE
* [https://blueprints.launchpad.net/trove/+spec/dedicated-tenant-db-provisioning Provision database in specific tenant] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/shared-cluster-db-provisioning Provisioning several db processes on the same cluster] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/auto-recovery-of-cluster-replica Auto-recovery of node in replica/cluster] (isviridov)
* Lunchpad meetings (https://launchpad.net/sprints/+new) as tool for topic discussions scheduling (isviridov)

== Agenda for the next meeting ==
* Conductor changes. (datsun180b)
* Tempest for integration tests. (SnowDust)
* Blueprint Rigor / Structure (demorris)
* Datastore types (ashestakov)
* Multiple Node templates (Heat Enabled Instance creation) (yogeshmehra)
* Size of pull requests

Meetings/TroveMeeting

2013-10-23T19:33:08Z

Tim Simpson: /* Agenda for the next meeting */

Meetings/TroveMeeting

2013-10-16T19:46:42Z

Tim Simpson: /* Agenda for the next meeting */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 3pm CDT / 1pm PDT, or 20:00 UTC. Feel free to add items in the agenda below.

== Agenda from the previous meeting ==
* User permission request escaping and dots in usernames (datsun180b, demorris)
* Provisioning ancillary resources post ACTIVE
* [https://blueprints.launchpad.net/trove/+spec/dedicated-tenant-db-provisioning Provision database in specific tenant] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/shared-cluster-db-provisioning Provisioning several db processes on the same cluster] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/auto-recovery-of-cluster-replica Auto-recovery of node in replica/cluster] (isviridov)
* Lunchpad meetings (https://launchpad.net/sprints/+new) as tool for topic discussions scheduling (isviridov)

== Agenda for the next meeting ==
* Conductor changes. (datsun180b)
* Fake mode daemon fix, how to use it. (grapex)
* Tempest for integration tests. (SnowDust)

Meetings/TroveMeeting

2013-10-16T19:44:41Z

Tim Simpson: /* Agenda for the next meeting */

Meetings/TroveMeeting

2013-10-16T19:41:08Z

Tim Simpson: /* Agenda for the next meeting */

= Weekly Trove (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Wednesdays at 3pm CDT / 1pm PDT, or 20:00 UTC. Feel free to add items in the agenda below.

== Agenda from the previous meeting ==
* User permission request escaping and dots in usernames (datsun180b, demorris)
* Provisioning ancillary resources post ACTIVE
* [https://blueprints.launchpad.net/trove/+spec/dedicated-tenant-db-provisioning Provision database in specific tenant] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/shared-cluster-db-provisioning Provisioning several db processes on the same cluster] (isviridov)
* [https://blueprints.launchpad.net/trove/+spec/auto-recovery-of-cluster-replica Auto-recovery of node in replica/cluster] (isviridov)
* Lunchpad meetings (https://launchpad.net/sprints/+new) as tool for topic discussions scheduling (isviridov)

== Agenda for the next meeting ==
* Conductor changes. (grapex)
* Fake mode daemon fix, how to use it. (grapex)

Meetings/TroveMeeting

2013-10-16T19:40:49Z

Tim Simpson: /* Agenda for the next meeting */

Reddwarf-Test-With-XmlLint

2013-05-01T04:57:21Z

Tim Simpson:

It is possible to change the HTTP client to capture the HTTP request and response bodies, save them as files, and run XmlLint to make sure the XML is well formed.

Here's an example test configuration file which will do so:
{
"include-files": ["localhost.test.conf"],
"reddwarf_client_cls": "reddwarf.tests.util.client.XmlLintClient",
"xml_temp_file":"lastclient-body.xml",
"xmllint_bin": "/usr/bin/xmllint"
}

There are some issues which preclude this from running in Tox normally with every run.

First, the path to xmllint is required as shown by the "xmllint_bin" config value. This program would need to be installed somehow by Tox to be make the local testing process simple.

Additionally, for the best results, "xml_temp_file" is used as the path of a temporary file to create which contains the lastest request or response body (piping the xml body to xmllint directly leads to inconsistent behavior and quirky results).

'''Validating an XSD File'''

One additional, but not mandatory, test config value is "xml_xsd" which if present will become an argument following "--schema" which is sent to xml lint.

While this is difficult to set up for Tox, it would be relatively simple to use in the VM gated tests guided by Reddwarf Integration.

'''Obstacles to putting this into VM runs'''

A few obstacles stand in the way of running this in RDI, however:

* If an invalid version is requested, the Reddwarf API currently returns a body of text which does not parse to an XML element (or a JSON body for that matter) which (rightfully) causes the test to fail when xmllint verification is turned on: https://bugs.launchpad.net/reddwarf/+bug/1174960

* There is as of yet no agreed upon XSD schema for Reddwarf.

If you'd like to test currently, simply install xmllint, and then change the contents of the file "etc/tests/xml.localhost.test.conf' to the sample file presented above. After that (assuming tox has been run once) run the following:
$ .tox/py27/bin/python run_tests.py --test-config=etc/tests/xml.localhost.test.conf

Reddwarf-Test-With-XmlLint

2013-05-01T04:47:24Z

Tim Simpson: Describes how to add xmllint verification to test runs

ReddwarfAppliesForIncubation

2013-04-18T21:40:17Z

Tim Simpson: /* Other project developers and qualifications: */

===Project codename:===

RedDwarf

===Summary:===

A control plane and data plane for deploying and operating relational (SQL) and non-relational (NoSQL) database management systems on OpenStack.

===Detailed Description:===

Reddwarf is Database as a Service for Openstack. It's designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. The service focuses on providing database resource isolation while automating complex administrative tasks including deployment, configuration, patching, backups, restores as well as horizontal and vertical scaling. While initially focused on MySQL, the future of Reddwarf will support multiple database technologies, including relational (SQL) and non-relational (NoSQL) technologies such as Redis, MongoDB, MariaDB, PostgreSQL, etc.. The long-term vision for Reddwarf is to provide a framework to control any persistant database management system.

Today, Reddwarf is a consumer of OpenStack. It is built on and leverages core OpenStack infrastructure services. It allows a service provider the flexibility of deployment and configuration of the core OpenStack components, and uses those components as the building blocks of its service. This includes Keystone for auth, Nova for instance creation, Cinder for volume support, Glance for images, Swift to store backups, and networking. In the future, we hope to eventually consume Ceilometer for metrics and Heat for initial image creation/orchestration, and potentially bare metal to spin up Reddwarf instances for the infrastructure database(s) within a OpenStack on OpenStack install.

With the recent adoption of Heat as an integrated project in OpenStack, many will question the need for a defined database service as Heat can orchestrate the setup of complex database clusters as well as handle basic lifecycle management of the database cluster. However, this is only one piece of a managing a database at scale. Reddwarf is layer above Heat, and as the project evolves, we will be able to leverage Heat to set up the database management system, which fits perfectly into our goal of leveraging core OpenStack services. The main differentiator for persistent datastores is that they require tightly integrated and specific operations such as controls to efficiently execute/schedule backups, restore a database to a specific point in time, configuration and optimization of the database, maintenance and management of data replication (masters and slaves) for horizontal scalability, and other database management tasks. A persistent datastore, clustered or not, is not simply a fire and forget resource. It needs constant monitoring and maintenance and we want to use this as the differentiator between Reddwarf and a service for orchestrating the creation of OpenStack resources. However, as Heat continues to evolve, and some basic lifecycle management tasks can be executed through it, we will have the option to leverage those operations.

===Basic roadmap for the project:===

Currently, Reddwarf can deploy multiple MySQL implementations, including core MySQL and Percona MySQL. The API continues to evolve through the addition of new features, but as it stands is very stable in its implementation for the basic tasks of creating and maintaining a single instance database deployment. The system is built such that multiple technologies can be leveraged, from the datastore implementation itself, to the tools used to keep maintenance ongoing. For instance, the backup feature, currently in progress, allows the cloud deployer to choose mysqldump or xtrabackup to backup the datastore. Over time, this will be expanded to support storage level snapshotting for larger volume installations.

From a datastore perspective, Reddwarf's functionality currently is the ability to spin up secured MySQL databases, create/destroy schemas and users, and enable a root user. From a openstack project perspective, Reddwarf also has quota and limit support and consumes Oslo to build out the architecture of the API.

Reddwarf has a healthy backlog of features such as database configuration control, manual backups and restores, and notifications. Others that are not yet started include scheduled backups, point in time database restores, API types/versions, full featured user modification, and replication for horizontal database scalability. The current roadmap can be viewed as blueprints on launchpad and will be continually updated as new feature requests are created. We will be working to add support for NoSQL soon.

There are also many housekeeping blueprints to maintain parity with other OpenStack projects. Reddwarf has strived to keep in line with the ecosystem, but there are some things that are necessary, such as completing integration with devstack. As it stands, reddwarf is integrated as much as possible today, and uses local.sh to instrument the reddwarf infrastructure. It also currently uses vm-gating and runs functional tests on every review.

===Location of project source code:===

Reddwarf operates like any other openstack project. Blueprints are registered and approved in the same fashion as the other projects, release cycles are followed, and code is gated through unit tests and functional tests. The codebase is in stackforge. Reviews are in openstack gerrit.

===Programming language, required technology dependencies: ===

Python. pypi dependencies are listed in pip-requires and test-requires.

===Is project currently open sourced? What license?: ===

Yes. Apache 2.0.

===Level of maturity of software and team:===

The team has followed the openstack ecosystem for a long time. They are well versed in understanding the ecosystem and follow it as closely as possible. Weekly meetings are held, a presence is always in irc, reviews are thoroughly picked through, and they hold a high standard for all code that lands in master. All features require unit and functional tests, and gating is done on these through OpenStack jenkins and a external jenkins for said functional tests. The software is run in a production OpenStack deployment at Rackspace on hundreds of nodes across multiple datacenters encompassing thousands of active database deployments.

===Proposed project technical lead and qualifications:===

'''Michael Basnight (hub_cap on irc)'''. Michael has lead the project since its inception at rackspace. He was the lead of the internal project, but has shifted his role within Rackspace to be fully dedicated to lead the public project. He is focused on the quality of the API, infrastructure, and testability of the project. He follows the openstack community closely and has contributed greatly to the project. He currently manages blueprints, runs the weekly meetings, and evangelizes reddwarf in the community. He has spoken at many of the OpenStack conferences and has attended almost all of the conferences since reddwarf began within Rackspace.

===Other project developers and qualifications:===

Currently HP and Rackspace contribute to the reddwarf project. They operate as a joint team within the open. They are listed below.

'''Tim Simpson (grapex on irc)'''. Tim has worked on Reddwarf since its inception and wrote the initial tests; since then he has worked to ensure every feature reaching trunk is backed by solid testing. Following Mike's transition he has become the lead of the Rackspace Cloud Database product.

'''Vipul Sabhaya (vipul on irc)'''. Vipul has been involved with reddwarf since 2011. He led the effort to get HP Cloud database as a service, based on Reddwarf launched. Vipul also led HP's efforts to refocus all development to the Reddwarf reference implementation. He has also promoted reddwarf by giving talks at the openstack summit, and is a member of the core team.

'''Nikhil Manchanda (slicknik on irc)'''. Nikhil has been involved with OpenStack and reddwarf since 2012. He has led the effort on integrating reddwarf setup with devstack and has also worked on aligning reddwarf testing, packaging, and CI with OpenStack standards. He has worked on all parts of the reddwarf project, including the API, taskmanager, guestagent, client, and documentation.

'''Jim Cooley'''. Jim has been involved with reddwarf since 2012, He has led the efforts to launch multiple Openstack related projects: DNS (moniker), Messaging, Load Balancing, and Monitoring. He initiated the HP move to the Reddwarf reference implementation and abandonment of their proprietary codebase and the migration into the Devstack gate (with the sponsorship of Monty Taylor, checkin currently on hold).

'''Anna Shen (annashen on irc)'''. Anna has been involved with reddwarf since late 2011. She worked primarily on SmartAgent of HP Reddwarf 1.0 and Packaging Reddwarf, she now works on GuestAgent and API side.

'''Dan Nguyen (esp on irc)'''. Dan has been working on Reddwarf since 2012 and contributes code to the reddwarf API, python-reddwarf-client, and reddwarf-integration. He's also started to look at other openstack projects as necessary to support Reddwarf development.

'''Steve Leon (esmute on irc)'''. Steve has been involved with reddwarf since 2012, and has been contributing to both reddwarf and its client.

'''Saurabh Surana (saurabhs on irc)''' Saurabh has been involved with reddwarf since 2012. He mainly works on running reddwarf for HP's cloud.

'''Justin Hopper (juice on irc)''' Justin has been involved with reddwarf since 2012. He incorporated diskimage-builder for guest agent and contributed to the backup/restore feature in reddwarf and its client.

'''Robert Myers (robertmyers on irc)'''. Robert has been involved with Reddwarf since late 2012. He has contributed to the API and integration tests for Reddwarf. He has also contributed to the Openstack Olso project.

'''DJ Johnstone (djohnstone on irc)'''. DJ has been involved with Reddwarf since late 2011. He was the Test Lead for Rackspace Cloud Database for most of 2012 and transitioned over to development on the team in Nov.

'''Joe Cruz (jcru on irc)'''. Joe has been involved with Reddwarf since 2012. He is currently a Software Developer on the Cloud Database team at Rackspace.

'''Ed Cranford (datsun180b on irc)'''. He has been involved with Reddwarf since mid 2011. He is also a Software Developer on the Cloud Database team at Rackspace, contributing to reddwarf and the client.

'''Daniel Salinas (imsplitbit on irc)''' Co-inventor of reddwarf project. Software Developer on the Cloud Database team at Rackspace currently working on extending Nova with OpenVz functionality. He has over five years of DBA experience.

'''Craig Vyvial (cp16net on irc)''' Craig has been involved with Reddwarf since 2011. He is currently a Software Developer on the Cloud Database team at Rackspace. He has contributed to many of the Openstack Ecosystem projects and mainly focused on developing features for Reddwarf. Craig has shared knowledge of Openstack and Reddwarf by giving presentations to user groups.

'''Dave Fecker (davefecker on irc)''' Dave works in Quality Engineering at Rackspace. He has contributed tests to the Reddwarf project.

===Infrastructure requirements (testing, etc): ===

Reddwarf has built its own testing environment. Its currently run as a gating job within by jenkins. Tox runs pep8 and unit tests on each commit through openstack gerrit. The reddwarf developers understand that, over time, they will adopt the openstack testing frameworks for all new functional tests. The core of reddwarf is run on openstack and installed within devstack. Reddwarf-integration, a project on stackforge, controls the installation and setup of reddwarf for development and testing. Tests are run in a openstack/kvm environment.

===Have all current contributors agreed to the OpenStack CLA? ===

Yes.

ReddwarfAppliesForIncubation

2013-04-17T23:53:58Z

Tim Simpson:

Project codename:

reddwarf

Summary (one sentence abstract of the project):

A control plane for managing multiple database technologies.

Detailed Description:

Reddwarf is Database as a Service for Openstack. It's designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. Initially, the service will focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores, and monitoring. The future of Reddwarf will consume multiple database technologies, including relational and non relational technologies. The vision is for Reddwarf to control any datastore that requires ongoing maintanence, data integrity, and replication.

Reddwarf is a consumer of OpenStack. It is built on and leverages core openstack infrastructure services. It allows a service provider the flexibility of deployment and configuration of the core OpenStack components, and uses those components as the building blocks of its service. This includes Keystone for auth, nova for instance creation, cinder for volume support, glance for images, swift to store backups, and quantum for networking. We hope to eventually consume cielometer for metrics and heat for initial image creation, and potentially bare metal to spin up reddwarf instances for the infastructure database(s) within a nova on nova install.

One may ask how is this different from Heat. We will consume heat to set up a initial mysql datastore, which fits perfectly into our vision for using openstack services. The main differentiator for datastores is that they require specific maintainence to back up, restore, and keep continued maintainence of replication, slaves, and yet to be determined management tasks. A datastore is not simply a fire and forget resource. It needs constant monitoring and we want to use this as the differentiator between reddwarf and a service for spinning up fire and forget resources. As heat continues to evolve, and some maintanence tasks can be done through it, such as upgrades and restarts, we will consume those services. Our goal is to focus on the specifics of keeping a datastore online and use other projects as much as possible for the smaller tasks.

Basic roadmap for the project:

Reddwarf can currrently deploy multiple mysql implementations. The API is evolving with new features, but as it stands is very stable in its implementation for the basic tasks. In the codebase as it stands today, both stock mysql and percona mysql are supported. The system is built such that multiple technolgies can be leveraged, from the datastore implementation itself, to the tools used to keep maintiannce ongoing. For instance, the backup feature, which is inflight, allows the cloud deployer to choose mysqldump or xtrabackup to backup the datastore.

From a datastore perspective, Reddwarf's functionality currently is the ability to spin up secured mysql databases, create/destroy schemas and users, and enable a root user. From a openstack project perspective, Reddwarf also has quota and limit support and consumes Oslo to build out the architecture of the api.

Reddwarf has a healthy backlog of features in order to fufil mysql features. Some of the inflight features are configuration (my.cnf) edits, manual backups and restores, notifications. Others that are not yet started include automated backups, including point in time restores, API types/versions, full featured user modification, and replication. this does not encompass all features. The current roadmap can be viewed as blueprints on launchpad.

There are also many housekeeping blueprints to maintain parity with other openstack projects. Reddwarf has strived to keep in line with the ecosystem, but there are some things that are necessary, such as completing integration with devstack. As it stands, reddwarf is integrated as much as possible today, and uses local.sh to instrument the reddwarf infrastructure. It also currently uses vm-gating and runs functional tests on every review.

Location of project source code:

Reddwarf operates like any other openstack project. blueprints are registered and approved in the same fashion as the other projects, release cycles are followed, and code is gated through unit tess and functional tests. The codebase is in stackforge. Reviews are in openstack gerrit.

Programming language, required technology dependencies:

Python. pypi dependencies are listed in pip-requires and test-requires.

Is project currently open sourced? What license?:

Yes. Apache 2.0.

Level of maturity of software and team:

The team has followed the openstack ecosystem for a long time. They are well versed in understanding the ecosystem and follow it as closely as possible. Weekly meetings are held, a presence is always in irc, reviews are thuroughly picked through, and they hold a high standard for all code that lands in master. All features require unit and functional tests, and gating is done on these thru openstack jenkins and a external jenkins for said functional tests. The software is run in production currently by rackspace on thousands of nodes.

Proposed project technical lead and qualifications:

Michael Basnight (hub_cap on irc). Michael has lead the project since its inception at rackspace. He was the lead of the internal project, but has shifted his role within Rackspace to be fully dedicated to lead the public project. He is focused on the quality of the API, infrastructure, and testability of the project. He follows the openstack community closely and has contributed greatly to the project. He currently manages blueprints, runs the weekly meetings, and evangelizes reddwarf in the community. He has spoken at many of the OpenStack conferences and has attended almost all of the conferences since reddwarf began within Rackspace.

Other project developers and qualifications:

Currently HP and Rackspace contribute to the reddwarf project. They operate as a joint team within the open. They are listed below.

Tim Simpson (grapex on irc). Tim has worked on Reddwarf since its inception and wrote the initial tests; since then he has worked to ensure every feature reaching trunk is backed by solid testing. Following Mike's transition he has become the lead of the Rackspace Cloud Database product.

GUYS::::: INSERT YOUR QUALS HERE :::::::

Infrastructure requirements (testing, etc):

Reddwarf has built its own testing environment. Its currently run as a gating job within by jenkins. Tox runs pep8 and unit tests on each commit through openstack gerrit. The reddwarf developers understand that, over time, they will adopt the openstack testing frameworks for all new functional tests. The core of reddwarf is run on openstack and installed within devstack. Reddwarf-integration, a project on stackforge, controls the installation and setup of reddwarf for development and testing. Tests are run in a openstack/kvm environment.

Have all current contributors agreed to the OpenStack CLA?

Yes.

ReddwarfAppliesForIncubation

2013-04-17T23:42:23Z

Tim Simpson:

Project codename:

reddwarf

Summary (one sentence abstract of the project):

A control plane for managing multiple database technologies.

Detailed Description:

Reddwarf is Database as a Service for Openstack. It's designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. Initially, the service will focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores, and monitoring. The future of Reddwarf will consume multiple database technologies, including relational and non relational technologies. The vision is for Reddwarf to control any datastore that requires ongoing maintanence, data integrity, and replication.

Reddwarf is a consumer of OpenStack. It is built on and leverages core openstack infrastructure services. It allows a service provider the flexibility of deployment and configuration of the core OpenStack components, and uses those components as the building blocks of its service. This includes Keystone for auth, nova for instance creation, cinder for volume support, glance for images, swift to store backups, and quantum for networking. We hope to eventually consume cielometer for metrics and heat for initial image creation, and potentially bare metal to spin up reddwarf instances for the infastructure database(s) within a nova on nova install.

One may ask how is this different from Heat. We will consume heat to set up a initial mysql datastore, which fits perfectly into our vision for using openstack services. The main differentiator for datastores is that they require specific maintainence to back up, restore, and keep continued maintainence of replication, slaves, and yet to be determined management tasks. A datastore is not simply a fire and forget resource. It needs constant monitoring and we want to use this as the differentiator between reddwarf and a service for spinning up fire and forget resources. As heat continues to evolve, and some maintanence tasks can be done through it, such as upgrades and restarts, we will consume those services. Our goal is to focus on the specifics of keeping a datastore online and use other projects as much as possible for the smaller tasks.

Basic roadmap for the project:

Reddwarf can currrently deploy multiple mysql implementations. The API is evolving with new features, but as it stands is very stable in its implementation for the basic tasks. In the codebase as it stands today, both stock mysql and percona mysql are supported. The system is built such that multiple technolgies can be leveraged, from the datastore implementation itself, to the tools used to keep maintiannce ongoing. For instance, the backup feature, which is inflight, allows the cloud deployer to choose mysqldump or xtrabackup to backup the datastore.

From a datastore perspective, Reddwarf's functionality currently is the ability to spin up secured mysql databases, create/destroy schemas and users, and enable a root user. From a openstack project perspective, Reddwarf also has quota and limit support and consumes Oslo to build out the architecture of the api.

Reddwarf has a healthy backlog of features in order to fufil mysql features. Some of the inflight features are configuration (my.cnf) edits, manual backups and restores, notifications. Others that are not yet started include automated backups, including point in time restores, API types/versions, full featured user modification, and replication. this does not encompass all features. The current roadmap can be viewed as blueprints on launchpad.

There are also many housekeeping blueprints to maintain parity with other openstack projects. Reddwarf has strived to keep in line with the ecosystem, but there are some things that are necessary, such as completing integration with devstack. As it stands, reddwarf is integrated as much as possible today, and uses local.sh to instrument the reddwarf infrastructure. It also currently uses vm-gating and runs functional tests on every review.

Location of project source code:

Reddwarf operates like any other openstack project. blueprints are registered and approved in the same fashion as the other projects, release cycles are followed, and code is gated through unit tess and functional tests. The codebase is in stackforge. Reviews are in openstack gerrit.

Programming language, required technology dependencies:

Python. pypi dependencies are listed in pip-requires and test-requires.

Is project currently open sourced? What license?:

Yes. Apache 2.0.

Level of maturity of software and team:

The team has followed the openstack ecosystem for a long time. They are well versed in understanding the ecosystem and follow it as closely as possible. Weekly meetings are held, a presence is always in irc, reviews are thuroughly picked through, and they hold a high standard for all code that lands in master. All features require unit and functional tests, and gating is done on these thru openstack jenkins and a external jenkins for said functional tests. The software is run in production currently by rackspace on thousands of nodes.

Proposed project technical lead and qualifications:

Michael Basnight (hub_cap on irc). Michael has lead the project since its inception at rackspace. He was the lead of the internal project, but has shifted his role within Rackspace to be fully dedicated to lead the public project. He is focused on the quality of the API, infrastructure, and testability of the project. He follows the openstack community closely and has contributed greatly to the project. He currently manages blueprints, runs the weekly meetings, and evangelizes reddwarf in the community. He has spoken at many of the OpenStack conferences and has attended almost all of the conferences since reddwarf began within Rackspace.

Other project developers and qualifications:

Currently HP and Rackspace contribute to the reddwarf project. They operate as a joint team within the open. They are listed below.

GUYS::::: INSERT YOUR QUALS HERE :::::::

Infrastructure requirements (testing, etc):

Reddwarf has built its own testing environment. Its currently run as a gating job within by jenkins. Tox runs pep8 and unit tests on each commit through openstack gerrit. The reddwarf developers understand that, over time, they will adopt the openstack testing frameworks for all new functional tests. The core of reddwarf is run on openstack and installed within devstack. Reddwarf-integration, a project on stackforge, controls the installation and setup of reddwarf for development and testing. Tests are run in a openstack/kvm environment.

Have all current contributors agreed to the OpenStack CLA?

Yes.

ReddwarfAppliesForIncubation

2013-04-17T23:38:11Z

Tim Simpson:

Project codename:

reddwarf

Summary (one sentence abstract of the project):

A control plane for managing multiple database technologies.

Detailed Description:

Reddwarf is Database as a Service for Openstack. It's designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. Initially, the service will focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores, and monitoring. The future of Reddwarf will consume multiple database technologies, including relational and non relational technologies. The vision is for Reddwarf to control any datastore that requires ongoing maintanence, data integrity, and replication.

Reddwarf is a consumer of OpenStack. It is built on and leverages core openstack infrastructure services. It allows a service provider the flexibility of deployment and configuration of the core OpenStack components, and uses those components as the building blocks of its service. This includes Keystone for auth, nova for instance creation, cinder for volume support, glance for images, swift to store backups, and quantum for networking. We hope to eventually consume cielometer for metrics and heat for initial image creation, and potentially bare metal to spin up reddwarf instances for the infastructure database(s) within a nova on nova install.

One may ask how is this different from Heat. We will consume heat to set up a initial mysql datastore, which fits perfectly into our vision for using openstack services. The main differentiator for datastores is that they require specific maintainence to back up, restore, and keep continued maintainence of replication, slaves, and yet to be determined management tasks. A datastore is not simply a fire and forget resource. It needs constant monitoring and we want to use this as the differenciator between reddwarf and a service for spinning up fire and forget resources. As heat continues to evolve, and some maintanence tasks can be done through it, such as upgrades and restarts, we will consume those services. Our goal is to focus on the specifics of keeping a datastore online and use other projects as much as possible for the smaller tasks.

Basic roadmap for the project:

Reddwarf can currrently deploy multiple mysql implementations. The API is evolving with new features, but as it stands the API is very stable in its implementation for the basic tasks. In the codebase as it stands today, both stock mysql and percona mysql are supported. The system is built such that multiple technolgies can be leveraged, from the datastore implementation itself, to the tools used to keep maintiannce ongoing. for instance, the backup feature, which is inflight, allows the cloud deployer to choose mysqldump or xtrabackup to backup the datastore.

From a datastore perspective, Reddwarf's functionality currently is the ability to spin up secured mysql databases, create/destroy schemas and users, and enable a root user. From a openstack project perspective, Reddwarf also has quota and limit support and consumes Oslo to build out the architecture of the api.

Reddwarf has a healthy backlog of features in order to fufil mysql features. Some of the inflight features are configuration (my.cnf) edits, manual backups and restores, notifications. Others that are not yet started include automated backups, including point in time restores, API types/versions, full featured user modification, and replication. this does not encompass all features. The current roadmap can be viewed as blueprints on launchpad.

There are also many housekeeping blueprints to maintain parity with other openstack projects. Reddwarf has strived to keep in line with the ecosystem, but there are some things that are necessary, such as completing integration with devstack. As it stands, reddwarf is integrated as much as possible today, and uses local.sh to instrument the reddwarf infrastructure. It also currently uses vm-gating and runs functional tests on every review.

Location of project source code:

Reddwarf operates like any other openstack project. blueprints are registered and approved in the same fashion as the other projects, release cycles are followed, and code is gated through unit tess and functional tests. The codebase is in stackforge. Reviews are in openstack gerrit.

Programming language, required technology dependencies:

Python. pypi dependencies are listed in pip-requires and test-requires.

Is project currently open sourced? What license?:

Yes. Apache 2.0.

Level of maturity of software and team:

The team has followed the openstack ecosystem for a long time. They are well versed in understanding the ecosystem and follow it as closely as possible. Weekly meetings are held, a presence is always in irc, reviews are thuroughly picked through, and they hold a high standard for all code that lands in master. All features require unit and functional tests, and gating is done on these thru openstack jenkins and a external jenkins for said functional tests. The software is run in production currently by rackspace on thousands of nodes.

Proposed project technical lead and qualifications:

Michael Basnight (hub_cap on irc). Michael has lead the project since its inception at rackspace. He was the lead of the internal project, but has shifted his role within rackspace to be fully dedicated to lead the public project. He is focused on the quality of the API, infrastructure, and testability of the project. He follows the openstack community closely and has contributed greatly to the project. He currently manages blueprints, runs the weekly meetings, and evanglises reddwarf in the community. He has spoken at many of the openstack conferences and has attended almost all of the conferences since reddwarf began within rackspace.

Other project developers and qualifications:

Currently HP and rackspace contribute to the reddwarf project. They operate as a joint team within the open. They are listed below.

GUYS::::: INSERT YOUR QUALS HERE :::::::

Infrastructure requirements (testing, etc):

Reddwarf has built its own testing environment. Its currently run as a gating job within by jenkins. Tox runs pep8 and unit tests on each commit through openstack gerrit. The reddwarf developers understand that, over time, they will adopt the openstack testing frameworks for all new functional tests. The core of reddwarf is run on openstack and installed within devstack. Reddwarf-integration, a project on stackforge, controls the installation and setup of reddwarf for development and testing. Tests are run in a openstack/kvm environment.

Have all current contributors agreed to the OpenStack CLA?

Yes.

Meetings/RedDwarfMeeting

2013-04-02T21:13:49Z

Tim Simpson: /* Agenda for the next meeting */

= Weekly RedDwarf (DBaaS) team meeting =

For those interested, we have meetings in #openstack-meeting-alt Weekly, Tuesdays at 21:00 UTC. Feel free to add items in the agenda below.

== Agenda for the next meeting ==
* Update to Action items (http://eavesdrop.openstack.org/meetings/reddwarf/2013/reddwarf.2013-03-26-20.59.html)
* Status of CI int-tests
* Backups Discussion
**Status
**Division of Labor
* Notifications Plan
* RootWrap
* Need to ensure Quotas works with XML, avoid skipping tests if the Quota tests fails.
* Actions / Action Events
* Open Discussion