ReddwarfAppliesForIncubation

Project codename:

RedDwarf

Summary:

A control plane and data plane for deploying and operating relational (SQL) and non-relational (NoSQL) database management systems on OpenStack.

Detailed Description:

Reddwarf is Database as a Service for Openstack. It's designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. The service focuses on providing database resource isolation while automating complex administrative tasks including deployment, configuration, patching, backups, restores as well as horizontal and vertical scaling. While initially focused on MySQL, the future of Reddwarf will support multiple database technologies, including relational (SQL) and non-relational (NoSQL) technologies such as Redis, MongoDB, MariaDB, PostgreSQL, etc.. The long-term vision for Reddwarf is to provide a framework to control any persistant database management system.

Today, Reddwarf is a consumer of OpenStack. It is built on and leverages core OpenStack infrastructure services. It allows a service provider the flexibility of deployment and configuration of the core OpenStack components, and uses those components as the building blocks of its service. This includes Keystone for auth, Nova for instance creation, Cinder for volume support, Glance for images, Swift to store backups, and networking. In the future, we hope to eventually consume Ceilometer for metrics and Heat for initial image creation/orchestration, and potentially bare metal to spin up Reddwarf instances for the infrastructure database(s) within a OpenStack on OpenStack install.

With the recent adoption of Heat as an integrated project in OpenStack, many will question the need for a defined database service as Heat can orchestrate the setup of complex database clusters as well as handle basic lifecycle management of the database cluster. However, this is only one piece of a managing a database at scale. Reddwarf is layer above Heat, and as the project evolves, we will be able to leverage Heat to set up the database management system, which fits perfectly into our goal of leveraging core OpenStack services. The main differentiator for persistent datastores is that they require tightly integrated and specific operations such as controls to efficiently execute/schedule backups, restore a database to a specific point in time, configuration and optimization of the database, maintenance and management of data replication (masters and slaves) for horizontal scalability, and other database management tasks. A persistent datastore, clustered or not, is not simply a fire and forget resource. It needs constant monitoring and maintenance and we want to use this as the differentiator between Reddwarf and a service for orchestrating the creation of OpenStack resources. However, as Heat continues to evolve, and some basic lifecycle management tasks can be executed through it, we will have the option to leverage those operations.

Basic roadmap for the project:

Currently, Reddwarf can deploy multiple MySQL implementations, including core MySQL and Percona MySQL. The API continues to evolve through the addition of new features, but as it stands is very stable in its implementation for the basic tasks of creating and maintaining a single instance database deployment. The system is built such that multiple technologies can be leveraged, from the datastore implementation itself, to the tools used to keep maintenance ongoing. For instance, the backup feature, currently in progress, allows the cloud deployer to choose mysqldump or xtrabackup to backup the datastore. Over time, this will be expanded to support storage level snapshotting for larger volume installations.

From a datastore perspective, Reddwarf's functionality currently is the ability to spin up secured MySQL databases, create/destroy schemas and users, and enable a root user. From a openstack project perspective, Reddwarf also has quota and limit support and consumes Oslo to build out the architecture of the API.

Reddwarf has a healthy backlog of features such as database configuration control, manual backups and restores, and notifications. Others that are not yet started include scheduled backups, point in time database restores, API types/versions, full featured user modification, and replication for horizontal database scalability. The current roadmap can be viewed as blueprints on launchpad and will be continually updated as new feature requests are created. We will be working to add support for NoSQL soon.

There are also many housekeeping blueprints to maintain parity with other OpenStack projects. Reddwarf has strived to keep in line with the ecosystem, but there are some things that are necessary, such as completing integration with devstack. As it stands, reddwarf is integrated as much as possible today, and uses local.sh to instrument the reddwarf infrastructure. It also currently uses vm-gating and runs functional tests on every review.

Location of project source code:

Reddwarf operates like any other openstack project. Blueprints are registered and approved in the same fashion as the other projects, release cycles are followed, and code is gated through unit tests and functional tests. The codebase is in stackforge. Reviews are in openstack gerrit.

Programming language, required technology dependencies:

Python. pypi dependencies are listed in pip-requires and test-requires.

Is project currently open sourced? What license?:

Yes. Apache 2.0.

https://github.com/stackforge/reddwarf/blob/master/LICENSE

Level of maturity of software and team:

The team has followed the openstack ecosystem for a long time. They are well versed in understanding the ecosystem and follow it as closely as possible. Weekly meetings are held, a presence is always in irc, reviews are thoroughly picked through, and they hold a high standard for all code that lands in master. All features require unit and functional tests, and gating is done on these through OpenStack jenkins and a external jenkins for said functional tests. The software is run in a production OpenStack deployment at Rackspace on hundreds of nodes across multiple datacenters encompassing thousands of active database deployments.

Proposed project technical lead and qualifications:

Michael Basnight (hub_cap on irc). Michael has lead the project since its inception at rackspace. He was the lead of the internal project, but has shifted his role within Rackspace to be fully dedicated to lead the public project. He is focused on the quality of the API, infrastructure, and testability of the project. He follows the openstack community closely and has contributed greatly to the project. He currently manages blueprints, runs the weekly meetings, and evangelizes reddwarf in the community. He has spoken at many of the OpenStack conferences and has attended almost all of the conferences since reddwarf began within Rackspace.

Other project developers and qualifications:

Currently HP and Rackspace contribute to the reddwarf project. They operate as a joint team within the open. They are listed below.

Tim Simpson (grapex on irc). Tim has worked on Reddwarf since its inception and wrote the initial tests; since then he has worked to ensure every feature reaching trunk is backed by solid testing. Following Mike's transition he has become the lead of the Rackspace Cloud Database product.

Vipul Sabhaya (vipul on irc). Vipul has been involved with reddwarf since 2011. He led the effort to get HP Cloud database as a service, based on Reddwarf launched. Vipul also led HP's efforts to refocus all development to the Reddwarf reference implementation. He has also promoted reddwarf by giving talks at the openstack summit, and is a member of the core team.

Nikhil Manchanda (slicknik on irc). Nikhil has been involved with OpenStack and reddwarf since 2012. He has led the effort on integrating reddwarf setup with devstack and has also worked on aligning reddwarf testing, packaging, and CI with OpenStack standards. He has worked on all parts of the reddwarf project, including the API, taskmanager, guestagent, client, and documentation.

Jim Cooley (jcooley on irc). Jim has been involved with reddwarf since 2012, He has led the efforts to launch multiple Openstack related projects: DNS (moniker), Messaging, Load Balancing, and Monitoring. He initiated the HP move to the Reddwarf reference implementation and abandonment of their proprietary codebase and the migration into the Devstack gate (with the sponsorship of Monty Taylor, checkin currently on hold).

Anna Shen (annashen on irc). Anna has been involved with reddwarf since late 2011. She worked primarily on SmartAgent of HP Reddwarf 1.0 and Packaging Reddwarf, she now works on GuestAgent and API side.

Dan Nguyen (esp on irc). Dan has been working on Reddwarf since 2012 and contributes code to the reddwarf API, python-reddwarf-client, and reddwarf-integration. He's also started to look at other openstack projects as necessary to support Reddwarf development.

Steve Leon (esmute on irc). Steve has been involved with reddwarf since 2012, and has been contributing to both reddwarf and its client.

Saurabh Surana (saurabhs on irc) Saurabh has been involved with reddwarf since 2012. He mainly works on running reddwarf for HP's cloud.

Justin Hopper (juice on irc) Justin has been involved with reddwarf since 2012. He incorporated diskimage-builder for guest agent and contributed to the backup/restore feature in reddwarf and its client.

Dror Kagan (kagan on irc). Dror has been involved with reddwarf since 2012. He incorporated diskimage-builder for guest agent, contributed to the backup/restore feature in reddwarf and its client, and worked on code coverage reports.

Robert Myers (robertmyers on irc). Robert has been involved with Reddwarf since late 2012. He has contributed to the API and integration tests for Reddwarf. He has also contributed to the Openstack Olso project.

DJ Johnstone (djohnstone on irc). DJ has been involved with Reddwarf since late 2011. He was the Test Lead for Rackspace Cloud Database for most of 2012 and transitioned over to development on the team in Nov.

Joe Cruz (jcru on irc). Joe has been involved with Reddwarf since 2012. He is currently a Software Developer on the Cloud Database team at Rackspace.

Ed Cranford (datsun180b on irc). He has been involved with Reddwarf since mid 2011. He is also a Software Developer on the Cloud Database team at Rackspace, contributing to reddwarf and the client.

Daniel Salinas (imsplitbit on irc) Co-inventor of reddwarf project. Software Developer on the Cloud Database team at Rackspace currently working on extending Nova with OpenVz functionality. He has over five years of DBA experience.

Craig Vyvial (cp16net on irc) Craig has been involved with Reddwarf since 2011. He is currently a Software Developer on the Cloud Database team at Rackspace. He has contributed to many of the Openstack Ecosystem projects and mainly focused on developing features for Reddwarf. Craig has shared knowledge of Openstack and Reddwarf by giving presentations to user groups.

Dave Fecker (davefecker on irc) Dave works in Quality Engineering at Rackspace. He has contributed tests to the Reddwarf project.

Community vendors We have had an uptick in interest from the community with respect to clustering and replication. Percona, Continuent (Tungsten) and CoderShip (Galera) are interested in helping us. Also SkySQL is interested in helping w/ Maria integration and they have a product similar to RedDwarf so they are interested in contributing to our product as well.

Infrastructure requirements (testing, etc):

Reddwarf has built its own testing environment. Its currently run as a gating job within by jenkins. Tox runs pep8 and unit tests on each commit through openstack gerrit. The reddwarf developers understand that, over time, they will adopt the openstack testing frameworks for all new functional tests. The core of reddwarf is run on openstack and installed within devstack. Reddwarf-integration, a project on stackforge, controls the installation and setup of reddwarf for development and testing. Tests are run in a openstack/kvm environment.

Have all current contributors agreed to the OpenStack CLA?

Yes.