Revised on: 4/14/2015 by Rushiagr
Presently, majority of OpenStack relies on relational databases for storing state. There are some limitations of such relational databases which might affect the overall scalability and maintainability (operationally) of a deployed cloud. This wiki tries to list down all the properties which NoSQL databases will bring to OpenStack world, along with some of their trade-offs. This wiki will take Cassandra as the reference NoSQL database, but the concepts should apply to most of the key-value store NoSQL databases.
Relational databases are very good at efficiently storing and retrieving relational data. And understandably so, since they are around since a few decades. However, there are some aspects where relational databases fall short:
Horizontal scalability/centralized architecture
One can only scale a relational database vertically, i.e., by buying a bigger server. This work upto a point, but beyond a point, it becomes infeasible. Even with existing database clustering solutions such as Galera, there is no support for partitioning or sharding data such that the total data is distributed on different servers, allowing them to scale.
NoSQL databases are designed with horizontal scalability as one of the primary goals. These databases distribute data of a table to multiple database servers, based on a hashing or sharding scheme. Moreover, they also come with tooling to add or delete nodes at a later point of time, and distribute the load evenly accordingly.
Since for RDBMS (Relational databases) all of the data resides on one node, availability gets affected in case of node failure. Mitigation of this issue is generally done by setting up a 'cluster'. A cluster can be active-active (meaning data is synchronously replicated) or an active-passive (master-slave, where data from master is asynchronously replicated to the slaves.
A synchronously replicated, active-active clustering solution like Galera has it's own problems. Synchronous replication means data has to be committed on all of the nodes in the cluster before the request is returned to the caller. This causes deadlock issues (which are more accurately race conditions). Also, such kind of replication is not viable if nodes are spread geographically, due to the synchronous nature. With master slave configuration, as the slave is generally behind, be it by a fraction of seconds, data-loss is a possibility when master goes down and a slave is promoted to master.
In NoSQL databases like Cassandra, one can specify replication count per table, so that data is available all the time. One can specify consistency level while writing/reading such that even if some nodes are down of such a NoSQL cluster, all the operations with the database will succeed.
Data center awareness
Cassandra has DC awareness (TODO: elaborate)
How to design NoSQL schema for a SQL database
NoSQL databases don't have joins. So one has to 'denormalize' data, i.e., replicate data in multiple tables in order to fetch all the required data from one table itself. (TODO: an example of join here?)
Problem with OpenStack API
OpenStack had only relational database almost from the start. This has led to an assumption that the database will always support sorting based on any column in a table. For example, for 'list all instances', 'list all images', and 'list all volumes' APIs, one can sort the results based on ANY attribute of an instance, image or volume (e.g. creation time, size, flavor). Most of the NoSQL databases doesn't support this feature. In some of the NoSQL databases, say in Cassandra, this can be done by creating additional tables, one each for each such attribute. But realistically speaking, this is a very ugly solution, performance-wise as well as space-wise. Nobody should be doing it.
Edit by Ajaya
Openstack is backed by relational database from the very beginning. This has led to APIs which assume a relational database in the backend. A very good example of this is "sorting by arbitrary columns in listing resources". Concretely, for 'list all instances', 'list all images', and 'list all volumes' APIs, one can sort the results based on ANY attribute of an instance, image or volume (e.g. creation time, size, flavor). Doing sorting in any distributed database is very expensive and breaks the very basic idea of a distributed database. So databases which partition data across many nodes don't provide this feature at all. For example, in Cassandra it is not possible to sort the rows based on an arbitrary column.
Edit by Rushi
Openstack is backed by relational database from the very beginning. This has led to APIs which assume a relational database in the backend. A very good example of this is "sorting by arbitrary columns in listing resources". Concretely, for 'list all instances', 'list all images', and 'list all volumes' APIs, one can sort the results based on ANY attribute of an instance, image or volume (e.g. creation time, size, flavor). Due to the very nature of distributed databases, sorting based on an arbitrary key is very expensive, and is not directly supported. This feature is expensive in relational databases too, but since all data reside on one node, the database APIs still support it despite it involving a full-table scan.
The logical plausible solution would be, to make sorting based on any arbitrary keys an 'extension' of OpenStack projects, thereby making it an optional feature. This way, people using NoSQL will not have this extension, and as a result, feature available for their customers, while RDBMS-backed deployments can have.. (TODO: need to check if such a functionality actually falls into the category of 'extension' in OpenStack lingo.)
Edit by Ajaya
One possible solution to this problem is to identify such apis and make them extensions, thereby making it an optional feature (TODO: need to check if such a functionality actually falls into the category of 'extension' in OpenStack lingo). Deployers would have freedom and choice to choose their desired backend as well as apis they want to provide to their customers.
Why Cassandra ?
- Also talk about ugly joins in sql, affecting performance?
- Also talk about this: nosql means no sort keys on any random column?
- ^^ in short, means that openstack already developed it's apis with an assumption that all backends are going to have those functionalities, which becomes false once we throw nosql in the mix. So such api's should be made optional/extensions? E.g. sort keys?