|
|
| (12 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| − | === Status ===
| |
| | | | |
| − | * Inital draft --[[User:Isviridov|isviridov]] ([[User talk:Isviridov|talk]]) 17:40, 16 October 2013 (UTC)
| + | == /!\ Project is not maintained anymore == |
| − | * Added more details [[User:Isviridov|isviridov]] ([[User talk:Isviridov|talk]]) 01:10, 30 October 2013 (UTC)
| |
| | | | |
| − | === Scope ===
| + | '''MagnetoDB''' is a key-value store service for OpenStack. It provides horizontally scalable, queryable storage, accessible via REST API. MagnetoDB supports Amazon DynamoDB API as well. MagnetoDB has been designed and developed in order to provide: |
| − | Document describes the idea and technical concept of DB service for OpenStack like [http://aws.amazon.com/dynamodb Amazon DynamoDB].
| + | * ''Easy integration'': REST API. Support Amazon DynamoDB API |
| − | There is no implementation yet.
| + | * ''OpenStack interoperability'': integrated with Keystone. Follows OpenStack design tenets, packaging and distribution |
| | + | * ''Database pluggability'': supports Cassandra,any other databases like MongoDB, HBase could be easily added |
| | + | * ''Horizontal scalability'': MagnetoDB is scalable lineary by amount of data and requests |
| | + | * ''High availability'': just one working API node is enough to continue handling requests |
| | | | |
| − | == What is MagnetoDB? == | + | == Project status == |
| − | MagnetoDB is [http://aws.amazon.com/dynamodb Amazon DynamoDB] [http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/API.html API] implementation for OpenStack world.
| + | Project is not maintained anymore. |
| | | | |
| − | === What is DynamoDB? === | + | == Resources == |
| − | DynamoDB is a fast, fully managed NoSQL database service that makes it simple and cost-effective to store and retrieve any amount of data, and serve any level of request traffic.
| + | * [http://magnetodb.readthedocs.org/ User/Developer Documentation ] |
| − | With DynamoDB, you can offload the administrative burden of operating and scaling a highly available distributed database cluster
| + | * [https://github.com/zaletniy/magnetodb-specs Design Documentation ] |
| | + | * [https://github.com/zaletniy/magnetodb MagnetoDB at GitHub] |
| | + | * [https://github.com/AlexanderChudnovets/magnetodb-test-bench Performace test bench] |
| | + | * [https://launchpad.net/magnetodb Launchpad] |
| | + | * [[MagnetoDB/FAQ|FAQ]] |
| | + | * [https://webchat.freenode.net/?channels=magnetodb #magnetodb at FreeNode] |
| | + | * [[MagnetoDB/WeeklyMeetingAgenda|Weekly IRC meetings]] |
| | | | |
| − | DynamoDB tables do not have fixed schemas, and each item may have a different number of attributes. Multiple data types add richness to the data model. Local secondary indexes add flexibility to the queries you can perform, without impacting performance. [http://aws.amazon.com/dynamodb/ [1<nowiki>]</nowiki>]
| + | == Core Team == |
| | + | * Andrei V. Ostapenko (aostapenko), Mirantis |
| | + | * Charles Wang (charlesw), Symantec |
| | + | * Dmitriy Ukhlov (dukhlov), Mirantis |
| | + | * Ilia Khudoshyn (ikhudoshyn), Mirantis |
| | + | * [PTL] Ilya Sviridov (isviridov), Mirantis |
| | | | |
| − | === Why it is needed? ===
| |
| − | In order to provide high level database service with DynamoDB API interface for OpenStack users.
| |
| | | | |
| − | Just simply create the table and start working with your data. No vm provisioning, database software setup, configuration, patching, monitoring and other administrative routine.
| + | 90 days project statisics http://stackalytics.com/report/contribution/magnetodb/90 |
| | | | |
| − | === Integration with Trove? === | + | == Getting started == |
| − | Initially suggested as [https://blueprints.launchpad.net/trove/+spec/http-accessible-storage blueprint], it separated out as project having different from Trove program mission.
| + | [[MagnetoDB/How_To_Contribute|How To Contribute]] |
| − | | |
| − | MagnetoDB is not one more database, which can be provisioned and managed by Trove. It is one more OpenStack service with own API, keystone based authentication, quota management.
| |
| − | | |
| − | However, the underlaying database will be provisioned and managed by Trove, in order to reuse existing Openstack components and avoid duplication of functionality.
| |
| − | | |
| − | == Overall architecture ==
| |
| − | The shape of the architecture based on Cassandra storage is below. It is the classic 2 layer web application, where each layer is horizontally scalable.
| |
| − | Requests are routed by load balancer to web application layer, and processed there.
| |
| − | | |
| − | [[File:magnetodb_architecture.png|800px|frameless|center]]
| |
| − | | |
| − | In projection to existing OpenStack components
| |
| − | [[File:magnetodb_components_by_layers.png|1000px|frameless|center]] | |
| − | | |
| − | == API ==
| |
| − | The DynamoDB API documented very well here [http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/operationlist.html Operations in Amazon DynamoDB]
| |
| − | However, the key aspects are mentioned below.
| |
| − | | |
| − | The API present HTTP request/response, where JSON is used as data transfer model. The request body contains the payload, and HTTP headers are used for metadata.
| |
| − | | |
| − | Example of such request
| |
| − | | |
| − | POST / HTTP/1.1
| |
| − | host: dynamodb.us-west-2.amazonaws.com
| |
| − | x-amz-date: 20130112T092034Z
| |
| − | x-amz-target: DynamoDB_20120810.CreateTable
| |
| − | Authorization: AWS4-HMAC-SHA256 Credential=AccessKeyID/20130112/us-west-2/dynamodb/aws4_request,SignedHeaders=host;x-amz-date;x-amz-target,Signature=145b1567ab3c50d929412f28f52c45dbf1e63ec5c66023d232a539a4afd11fd9
| |
| − | content-type: application/x-amz-json-1.0
| |
| − | content-length: 23
| |
| − | connection: Keep-Alive
| |
| − | | |
| − | As you can see the operation name and authorization data is sent in HTTP request headers.
| |
| − | | |
| − | === Authorization ===
| |
| − | It is HMAC based on [http://docs.openstack.org/trunk/openstack-ops/content/get_creds.html EC2 Credentials], which is sent in a header of HTTP request as in example above.
| |
| − | | |
| − | === Table CRUD operations ===
| |
| − | With following methods you can create, list, delete tables.
| |
| − | In the first implementation the throughput configuration parameters can be ignored. So UpdateTable operation turns making no effect, because of that.
| |
| − | | |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_ListTables.html ListTables]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DescribeTable.html DescribeTable]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html CreateTable]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateTable.html UpdateTable]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteTable.html DeleteTable]
| |
| − | | |
| − | === Item CRUD operations ===
| |
| − | With items operation you can apply CRUD operation on your data.
| |
| − | In addition to that your request can be conditional, means will be executed if item is in a state you defined in the request
| |
| − | | |
| − | The GetItem operation consistency can be eventual or strong per request.
| |
| − | | |
| − | More details can be found below
| |
| − | | |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_PutItem.html PutItem]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_UpdateItem.html UpdateItem]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html DeleteItem]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html GetItem]
| |
| − | | |
| − | === Data querying ===
| |
| − | In order to search the data there are two ways query and scan operations.
| |
| − | | |
| − | '''Query''' operation searches only primary key attribute values and supports a subset of comparison operators on key attribute values to refine the search process. A Query operation always returns results, but can return empty results.
| |
| − | | |
| − | '''Scan''' operation examines every item in the table. You can specify filters to apply to the results to refine the values returned to you, after the scan has finished. [http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html [2<nowiki>]</nowiki>]
| |
| − | | |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html Query]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html Scan]
| |
| − | | |
| − | === Batch operations ===
| |
| − | The batch operations are also supported.
| |
| − | | |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html BatchGetItem]
| |
| − | * [http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html BatchWriteItem]
| |
| − | | |
| − | == Implementation ==
| |
| − | The technical concept is still in progress, but the shape is pretty clear and key things are mentioned below
| |
| − | | |
| − | === Datasource pluggability ===
| |
| − | | |
| − | For backend storage provisioning and management OpenStack DBaaS Trove will be used.
| |
| − | The backend database should be pluggable to provide flexibility around choosing the solution which matches the best the existing or planned OpenStack installation technology stack. MySQL is one of the obvious options, being de-facto standard for being used with OpenStack and being supported by Trove right away.
| |
| − | | |
| − | [http://cassandra.apache.org/ Apache Cassandra] looks very suitable for that case due to having tunable consistency, easy scalability, key-value data model, ring topology and other features what gives us predictable high performance and fault tolerance. Cassandra data model perfectly fits MagnetoDB needs.
| |
| − | | |
| − | === OpenStack component reusage ===
| |
| − | Being the OpenStack service MagnetoDB must and will use its infrastructure wherever it possible. For most of the core modules the corresponding OpenStack modules are defined in the table below.
| |
| − | | |
| − | {| class="wikitable"
| |
| − | |-
| |
| − | ! Purpose !! OS component
| |
| − | |-
| |
| − | | Provisioning of web tier || [[Heat]], Mistral (Convection)
| |
| − | |-
| |
| − | | Load balancer || [[Neutron/LBaaS|LBaaS, Neutron]]
| |
| − | |-
| |
| − | | Autoscaling of web tier || Heat autoscaling, [[Ceilometer]]
| |
| − | |-
| |
| − | | Data Source provisioning and management || [[Trove]]
| |
| − | |-
| |
| − | | Monitoring || [[Ceilometer]]
| |
| − | |-
| |
| − | | Authentication || [[Keystone]]
| |
| − | |}
| |
| − | | |
| − | === Autoscaling ===
| |
| − | MagnetoDB should scale accoring to the load and data it handle.
| |
| − | Heat autoscaling group is gonna be used to achive it.
| |
| − | | |
| − | The autoscaling of datasource expected to be implemened in Trove.
| |
| − | == CI/CD ==
| |
| − | === CI/CD concept ===
| |
| − | There are several ways to integrate with OpenStack Infra:
| |
| − | * Adding own public cloud to Infra's NodePool as new provider which will be used to provide the specific environment (jenkins slaves) for MagnetoDB project. Adding own Jenkins job and script to Infra for this. For all tasks use Openstack Infra infrastructure (Jenkins, Zuul etc.)
| |
| − | *Adding own Jenkins to Infra's Gearman (Zuul) via Jenkins Gearman plugin. This Jenkins will be contain jobs and slaves for MagnetoDB project. Infra's Zull will be used for the orchestration.
| |
| − | *Gerrit layer integration. Use Jenkins Gerrit Trigger Plugin or Zuul for integration with OpenStack Gerrit and own envoronment for CI. Described here http://ci.openstack.org/third_party.html.
| |
| − | *Full integration. Adding own job to Jenkins Job Builder and some changes to layout.yaml (Zull). Use Infra's slaves for running own job.
| |
| − | | |
| − | :Below is a diagram describing the method of full integration
| |
| − | | |
| − | [[File:Infraconcept.png|1000px|frameless|center|Concept OpenStack Infra integration]]
| |
| − | | |
| − | | |
| − | | |
| − | Diagram shows standard Openstack Infra workflow for main components (Jenkins, Zuul, Nodepool). Red rectangles describes the necessary steps for Infra integration:
| |
| − | * Preparing and adding new Jenkins job to Jenkins Job Builder in .yaml format
| |
| − | * Modify layout.yaml (Zuul) for adding new job in some pipeline.
| |
| − | | |
| − | == Useful links ==
| |
| − | * [https://github.com/stackforge/magnetodb Source code]
| |
| − | * [https://launchpad.net/magnetodb Project space]
| |
| − | * [https://blueprints.launchpad.net/magnetodb Blueprints]
| |
| − | * [https://bugs.launchpad.net/magnetodb Bugs]
| |
| − | * [https://review.openstack.org/#/q/status:open+project:stackforge/magnetodb,n,z Patches on review]
| |
| − | * [http://eavesdrop.openstack.org/irclogs/%23magnetodb/ IRC logs]
| |
| − | * IRC server: '''irc.freenode.net''' channel: <code><nowiki>#magnetodb</nowiki></code>
| |
| − | | |
| − | == Improve MagnetoDB ==
| |
| − | === Where can I discuss & propose changes? ===
| |
| − | # Our IRC channel: IRC server <code><nowiki>#magnetodb</nowiki></code> on '''irc.freenode.net''';
| |
| − | # Openstack mailing list: '''openstack-dev@lists.openstack.org''' (see [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev subscription and usage] instructions);
| |
| − | # [https://launchpad.net/magnetodb MagnetoDB team on Launchpad]: Questions&Answers/Bugs/Blueprints.
| |
| − | | |
| − | === How can I start? ===
| |
| − | It is extremetly simple to participate in different MagnetoDB development lines. [https://launchpad.net/magnetodb MagnetoDB Launchpad page] contains a wide range of tasks perfectly suited for you to start contributing to MagnetoDB. You can choose any unassigned [https://bugs.launchpad.net/magnetodb bug] or [https://blueprints.launchpad.net/magnetodb blueprint]. As soon as you have chosen a bug, just assign it to you, and you can start fixing it. If you would like chose a blueprint, please contact core team at the <code><nowiki>#magnetodb</nowiki></code> IRC channel on '''irc.freenode.net'''.
| |
| − | | |
| − | The most bugs and blueprints contain basic descriptions of what is to be done there; in case you have questions or want to share your ideas, be sure to contanct us in IRC.
| |
| − | | |
| − | === How to contribute? ===
| |
| − | # First of all you need a [http://launchpad.net/ Launchpad] account. Make sure Launchpad has your SSH key, Gerrit (the code review system) uses this;
| |
| − | # Sign the Contributors License Agreement as outlined in section 3 of the [[How_To_Contribute#Contributors_License_Agreement|How To Contribute]] wiki page;
| |
| − | # Tell git your details: <br /> <br /> <code>git config --global user.name "Firstname Lastname"</code> <br /> <code>git config --global user.email "your_email@youremail.com"</code><br /><br />
| |
| − | # Install git-review. This tool takes a lot of the pain out of remembering commands to push code up to Gerrit for review and to pull it back down to edit it. It is installed using: <br /> <br /><code>pip install git-review</code> <br /><br /> NOTE: Several Linux distributions (notably Fedora 16 and Ubuntu 12.04) are also starting to include git-review in their repositories so it can also be installed using the standard package manager;
| |
| − | # Grab the MagnetoDB repository: <br /> <br /> <code> git clone git@github.com:stackforge/magnetodb.git </code> <br /> <br />
| |
| − | # Checkout a new branch to hack on: <br /> <br /> <code> git checkout -b TOPIC-BRANCH </code> <br /> <br />
| |
| − | # Start coding;
| |
| − | # Run the test suite locally to make sure nothing broke, e.g.: <br /> <br /> <code> ./run_tests.sh </code> <br /> <br />or you can use <code>tox</code> test command line tool (install it with <code> pip install tox</code>). <br /> NOTE: If you extend MagnetoDB with new functionality, make sure you also have provided unit tests for it;
| |
| − | # Commit your work using: <br /><br /> <code>git commit -a</code> <br /><br />or you can use the following to edit a previous commit: <br /><br /> <code>git commit -a --amend</code><br /><br /> NOTE: Make sure you have supplied your commit with a neat commit message, containing a link to the corresponding blueprint/bug, if appropriate;
| |
| − | # Push the commit up for code review using: <br /><br /><code>git review</code><br /><br /> That is the awesome tool you installed earlier that does a lot of hard work for you;
| |
| − | # Watch your email or [https://review.openstack.org review site], it will automatically send your code for a battery of tests on our [https://jenkins.openstack.org/ Jenkins setup] and the core team for the project will review your code. If there is any changes that should be made they will let you know;
| |
| − | # When all is good the review site will automatically merge your code.
| |
| − | :<br /> (This tutorial is based on: http://www.linuxjedi.co.uk/2012/03/real-way-to-start-hacking-on-openstack.html)
| |
| − | | |
| − | === Developer Guide ===
| |
| − | [[MagnetoDB/DeveloperGuide|Developer Guide]]
| |
| − | | |
| − | === What are Code Review Guidelines? ===
| |
| − | [[MagnetoDB/Code_Review_Guidelines|Code Review Guidelines]]
| |
| | | | |
| | + | == Presentations & blogposts == |
| | + | [https://www.mirantis.com/blog/introducing-magnetodb-nosql-database-service-openstack/ Introducing MagnetoDB] |
| | == Sub pages == | | == Sub pages == |
| | | | |
| | {{Special:PrefixIndex/:MagnetoDB/}} | | {{Special:PrefixIndex/:MagnetoDB/}} |
