Jump to: navigation, search

Difference between revisions of "Roadmap (Marconi)"

m (Juno-2 (July 24))
m (Juno-3 (Sept 4))
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Let's Do This ==
 
== Let's Do This ==
  
For Juno, let's ship a service that is super fast, has a polished API, and is well-documented.  
+
For Juno, let's ship a service that is super fast, has a polished API, and is well-documented. We can't decide the future of the project without getting more feedback based on real-world usage. If we need to totally reinvent the API based on that feedback, we can do that in K.
  
 
=== Juno Themes ===
 
=== Juno Themes ===
Line 13: Line 13:
 
* Complete a polished version of the 1.0 API, v1.1
 
* Complete a polished version of the 1.0 API, v1.1
 
* Ship a driver for high-throughput use cases
 
* Ship a driver for high-throughput use cases
 +
** 10ms for any operation, per individual client
 +
** 10K/sec aggregate rate per tenant
 
* Ship a driver for a non-AGPL backend that has feature parity with MongoDB
 
* Ship a driver for a non-AGPL backend that has feature parity with MongoDB
* Greatly expanded user guide, published to RTD
+
** Durability is required, so can't simply be in-memory
 +
** Can be the same driver as for high-throughput use cases, if possible
 +
* Expanded user guide, published to RTD
  
  
 
[[File:Juno-marconi-cartoon.png|800px|frameless]]
 
[[File:Juno-marconi-cartoon.png|800px|frameless]]
  
 +
== Upcoming milestones ==
  
 +
=== Juno-3 (Sept 4) ===
  
== Upcoming milestones ==
+
Graduation
 +
* Decide if we are going to try to graduate this cycle, or wait for Kilo.
 +
* Meet on G+ with TC rep to review graduation plan and get it blessed
 +
* Create a non-AGPL alternative to MongoDB. Redis is our best bet:
 +
** Redis supports optional durability via AOF + RDB
 +
** HA available through Redis Cluster, or we can DIY if we need synchronous replication
 +
** Apache 2 License
 +
* Rename the project
 +
* Document all config options in user guide
  
=== Juno-2 (July 24) ===
 
  
* Rename the project
+
Quality Engineering
 +
* MongoDB gate
 +
* Performance tuning (profiling, benchmarking, etc. - may slip to Kilo)
 +
* Publish benchmarks for minimal HA deployment for mongo and redis (perhaps present in design session at Paris summit?)
 
* Threat model for minimal HA deployment
 
* Threat model for minimal HA deployment
* Finish tempest tests for API v1.0
+
* End-game bug fixing and stabilization
* Basic marconi-bench tool and cluster (will be merged into Rally eventually
+
* Decouple control plane backend drivers from data plane backend drivers to facilitate testing and maintenance (may slip to Kilo)
* Queue defaults (API 1.1)
 
* Finish and freeze API 1.1
 
* Functional tests for API 1.1
 
* Live queue migration
 
* Finish POCs for: Redis, AMQP, Elasticsearch (?), LevelDB (?)
 
* Based on POCs, decide on 1-2 to add as an out-of-the-box alternative to MongoDB
 
* Start converting POCs into "real" drivers
 
  
=== Juno-3 (Sept 4) ===
 
  
Here's a rough list--in no particular order--of what we would like to accomplish. This list will be refined the closer we get to the start of J-3.
+
Drivers
 +
* Finish Redis message store driver
 +
** HA (replication + failover)
 +
** Full test coverage (unit/functional)
 +
** Gate?
 +
* Create a feature-complete alternative to MongoDB, with similar durability and perf (as mentioned above, under "Graduation")
  
* Finish work on new backend driver(s)
 
* Finish work on flavors
 
* AuthZ for Admin API
 
* Decouple control plane backend drivers from data plane backend drivers
 
* Mongo DB gate (3rd-party)
 
* Benchmark gate (3rd-party)
 
* Publish User Guide and API Reference on RTD
 
* User Guide 2.0
 
* Cross-service Request ID
 
* Basic security test suite
 
* AMQP gate
 
* Redis gate
 
* DDoS testing and mitigation
 
* Tempest tests for API v1.1
 
* Finish up performance tuning
 
* End-game bug fixing and stabilization
 
  
 +
API v1.1
 +
* Finish work on flavors (may slip to Kilo)
 +
* Finish /health endpoint work
 +
* Finish homedoc changes
 +
* Finish MessagePack support (may slip to Kilo)
 +
**  Depends on changing drivers to store bodies as binary blobs
 +
* Response document changes
 +
* Implement in python-marconiclient (may slip to Kilo)
 +
* Document in user guide (may slip to Kilo)
 +
* Functional tests (may slip to Kilo)
 +
* Tempest tests (may slip to Kilo)
  
 
== Past Milestones ==
 
== Past Milestones ==
 +
 +
=== Juno-2 (July 24) ===
 +
 +
* Finish tempest tests for API v1.0
 +
* Basic marconi-bench tool (will be merged into Rally eventually)
 +
* Finish most of API 1.1 (incl. msgpack support)
 +
* Finish POCs for: Redis, AMQP, Elasticsearch (?), LevelDB (?)
 +
* Start converting POCs into "real" drivers
  
 
=== Juno-1 (June 12) ===
 
=== Juno-1 (June 12) ===
Line 70: Line 86:
 
* Fill out OSSG security FAQ template
 
* Fill out OSSG security FAQ template
 
* Revert read-only queue metadata from API 1.1 (don't forget to update the spec)
 
* Revert read-only queue metadata from API 1.1 (don't forget to update the spec)
 
  
 
== Postponed Work ==
 
== Postponed Work ==
Line 76: Line 91:
 
Here are some things we would love to do this cycle, but don't have enough people to work on them. If you are interested in helping out, please get in touch via the [[Mailing_Lists#Future_Development|openstack-dev mailing list]] (using the prefix "[marconi]" in the subject), or via IRC in #openstack-marconi on Freenode.
 
Here are some things we would love to do this cycle, but don't have enough people to work on them. If you are interested in helping out, please get in touch via the [[Mailing_Lists#Future_Development|openstack-dev mailing list]] (using the prefix "[marconi]" in the subject), or via IRC in #openstack-marconi on Freenode.
  
 +
* AMQP transport driver ([[Marconi/specs/amqp/api/v1|some research]] has been done on this, but we still have to sort out body content type and what to do about claim semantics)
 +
* Delayed message feature? (delay message visibility by x seconds, would help with hybrid/auditing use case)
 +
* Live queue migration
 +
* Rename the project
 +
* AuthZ for Admin API
 +
* Benchmark gate (3rd-party)
 +
* Publish User Guide and API Reference on RTD
 +
* User Guide 2.0
 +
* Cross-service Request ID
 +
* Basic security test suite
 +
* Redis gate?
 +
* DDoS testing and mitigation
 
* Research notifications (meet with 3rd-parties, create a design)
 
* Research notifications (meet with 3rd-parties, create a design)
 
* Start coding notifications
 
* Start coding notifications

Latest revision as of 18:08, 29 July 2014

Let's Do This

For Juno, let's ship a service that is super fast, has a polished API, and is well-documented. We can't decide the future of the project without getting more feedback based on real-world usage. If we need to totally reinvent the API based on that feedback, we can do that in K.

Juno Themes

  • Speed
  • Polish
  • Docs

Major Goals

  • Complete a polished version of the 1.0 API, v1.1
  • Ship a driver for high-throughput use cases
    • 10ms for any operation, per individual client
    • 10K/sec aggregate rate per tenant
  • Ship a driver for a non-AGPL backend that has feature parity with MongoDB
    • Durability is required, so can't simply be in-memory
    • Can be the same driver as for high-throughput use cases, if possible
  • Expanded user guide, published to RTD


Juno-marconi-cartoon.png

Upcoming milestones

Juno-3 (Sept 4)

Graduation

  • Decide if we are going to try to graduate this cycle, or wait for Kilo.
  • Meet on G+ with TC rep to review graduation plan and get it blessed
  • Create a non-AGPL alternative to MongoDB. Redis is our best bet:
    • Redis supports optional durability via AOF + RDB
    • HA available through Redis Cluster, or we can DIY if we need synchronous replication
    • Apache 2 License
  • Rename the project
  • Document all config options in user guide


Quality Engineering

  • MongoDB gate
  • Performance tuning (profiling, benchmarking, etc. - may slip to Kilo)
  • Publish benchmarks for minimal HA deployment for mongo and redis (perhaps present in design session at Paris summit?)
  • Threat model for minimal HA deployment
  • End-game bug fixing and stabilization
  • Decouple control plane backend drivers from data plane backend drivers to facilitate testing and maintenance (may slip to Kilo)


Drivers

  • Finish Redis message store driver
    • HA (replication + failover)
    • Full test coverage (unit/functional)
    • Gate?
  • Create a feature-complete alternative to MongoDB, with similar durability and perf (as mentioned above, under "Graduation")


API v1.1

  • Finish work on flavors (may slip to Kilo)
  • Finish /health endpoint work
  • Finish homedoc changes
  • Finish MessagePack support (may slip to Kilo)
    • Depends on changing drivers to store bodies as binary blobs
  • Response document changes
  • Implement in python-marconiclient (may slip to Kilo)
  • Document in user guide (may slip to Kilo)
  • Functional tests (may slip to Kilo)
  • Tempest tests (may slip to Kilo)

Past Milestones

Juno-2 (July 24)

  • Finish tempest tests for API v1.0
  • Basic marconi-bench tool (will be merged into Rally eventually)
  • Finish most of API 1.1 (incl. msgpack support)
  • Finish POCs for: Redis, AMQP, Elasticsearch (?), LevelDB (?)
  • Start converting POCs into "real" drivers

Juno-1 (June 12)

  • Start coding AMQP
  • Start coding Redis
  • Continue work on API v1.1
  • Rename "shard" to "pool" and note the name change on wiki and in the user manual
  • Start coding KPIs (/health endpoint)
  • Fill out OSSG security FAQ template
  • Revert read-only queue metadata from API 1.1 (don't forget to update the spec)

Postponed Work

Here are some things we would love to do this cycle, but don't have enough people to work on them. If you are interested in helping out, please get in touch via the openstack-dev mailing list (using the prefix "[marconi]" in the subject), or via IRC in #openstack-marconi on Freenode.

  • AMQP transport driver (some research has been done on this, but we still have to sort out body content type and what to do about claim semantics)
  • Delayed message feature? (delay message visibility by x seconds, would help with hybrid/auditing use case)
  • Live queue migration
  • Rename the project
  • AuthZ for Admin API
  • Benchmark gate (3rd-party)
  • Publish User Guide and API Reference on RTD
  • User Guide 2.0
  • Cross-service Request ID
  • Basic security test suite
  • Redis gate?
  • DDoS testing and mitigation
  • Research notifications (meet with 3rd-parties, create a design)
  • Start coding notifications
  • Integrate notifications with 1-2 other programs
  • Demo videos
  • Kafka driver
  • Long polling or WebSocket support in the API
  • Snappy or LZ4 compression of message bodies before storing them
  • Structured logging
  • Temp URL style endpoints with ACL, to give to untrusted parties for limited interactions with a queue
  • Message signing