OpenStack - User contributions [en]

MagnetoDB/specs/requestmetrics

2014-12-03T00:37:53Z

Charles Wang: Undo revision 69425 by Charles Wang (talk)

=== Request Real Time Metrics ===

Real time request metrics including latency/count/etc.

=== Specification status ===
Draft

=== Problem Description ===
To proactively address MagnetoDB operational issues, admin user needs real time visibility to request metrics data on each API node. Including:
* number of requests
* number of failures
* number of errors
* average latency
* median latency
* minimum latency
* maximum latency
* requests per second
* distribution of request latency for each type of REST API call, such as "50%","66%","75%","80%","90%","95%","98%","99%","100%"

=== Proposed Change ===

Request metrics can be reported to StatsD using a sample rate for API calls. Request metrics are either counters or timing data (in units of milliseconds).
 

StatsD can be utilized to expand timing data to min, max, avg, count, and 90th percentile per timing metric.
 

Middleware seems a natural place to collect the request metrics data. However, if admin user needs visibility into requests to each API endpoints, we will need to capture metrics before/after each call to the API endpoints.
 

We propose to introduce a new middleware to all API node services, including REST API and streaming.
 

Initially we will focus on request measurements on API node wide metrics, API endpoints and WSGI processing delays on each API node. Later on, we will expand to cover Cassandra request metrics.
 

Each REST API call made to MagnetoDB will have the following request metrics:
 

* request receive time
* request receive error
* request receive timeout
* request receive bytes
* response send time
* response send error
* response send timeout
* response send bytes
 

It can be further broken down to API endpoints.

WSGI will have the following metrics:
 
* backlog
* waits
* request processing time
* response processing time
* dispatch time

Cassandra metrics is supported by Cassandra python driver and can be enabled. But metrics data is supported through Scales.

==== Alternatives ====
Scales (bundled with Cassandra Python Driver) can be used instead of StatsD.

==== Security Impact ====

* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

==== Notifications Impact ====

==== Other End User Impact ====

==== Performance Impact ====

Performance impact should be minimal since if statsd is used. The metrics sent to statsd is through UDP.

==== Other Deployer Impact ====

A dependency in statsd will be introduced.

==== Developer Impact ====

==== Implementation ====

===== Assignee(s) =====

Charles Wang

===== Work Items =====

===== Dependencies =====

* statsd

==== Documentation Impact ====

==== References ====

MagnetoDB/specs/requestmetrics

2014-12-03T00:33:59Z

Charles Wang: /* Proposed Change */

MagnetoDB/specs/requestmetrics

2014-11-18T19:53:36Z

Charles Wang:

=== Request Real Time Metrics ===

Real time request metrics including latency/count/etc.

=== Specification status ===
Draft

=== Problem Description ===
To proactively address MagnetoDB operational issues, admin user needs real time visibility to request metrics data on each API node. Including:
* number of requests
* number of failures
* number of errors
* average latency
* median latency
* minimum latency
* maximum latency
* requests per second
* distribution of request latency for each type of REST API call, such as "50%","66%","75%","80%","90%","95%","98%","99%","100%"

=== Proposed Change ===

Request metrics can be reported to StatsD per API call. Request metrics are either counters or timing data (in units of milliseconds).
 

StatsD can be utilized to expand timing data to min, max, avg, count, and 90th percentile per timing metric.
 

Middleware seems a natural place to collect the request metrics data.
 

We propose to introduce a new middleware to all API node services, including api/streaming/task executor.
 

Initially we will focus on request measurements on API endpoint, WSGI processing delay, and Cassandra request metrics.
 

Each API endpoint will have the following request metrics:
 

* request receive time
* request receive error
* request receive timeout
* request receive bytes
* response send time
* response send error
* response send timeout
* response send bytes
 

WSGI will have the following metrics:
 
* backlog
* waits
* request processing time
* response processing time
* dispatch time

Cassandra metrics is supported by Cassandra python driver and can be enabled. But metrics data is supported through Scales.

==== Alternatives ====
Scales (bundled with Cassandra Python Driver) can be used instead of StatsD.

==== Security Impact ====

* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

==== Notifications Impact ====

==== Other End User Impact ====

==== Performance Impact ====

Performance impact should be minimal since if statsd is used. The metrics sent to statsd is through UDP.

==== Other Deployer Impact ====

A dependency in statsd will be introduced.

==== Developer Impact ====

==== Implementation ====

===== Assignee(s) =====

Charles Wang

===== Work Items =====

===== Dependencies =====

* statsd

==== Documentation Impact ====

==== References ====

MagnetoDB/specs/requestmetrics

2014-11-18T19:22:49Z

Charles Wang: /* Proposed Change */

=== Request Real Time Metrics ===

Real time request metrics including latency/count/etc.

=== Specification status ===
Draft

=== Problem Description ===
To proactively address MagnetoDB operational issues, admin user needs real time visibility to request metrics data on each API node. Including:
* number of requests
* number of failures
* number of errors
* average latency
* median latency
* minimum latency
* maximum latency
* requests per second
* distribution of request latency for each type of REST API call, such as "50%","66%","75%","80%","90%","95%","98%","99%","100%"

=== Proposed Change ===

Request metrics can be reported to StatsD per API call. Request metrics are either counters or timing data (in units of milliseconds).

StatsD can be utilized to expand timing data to min, max, avg, count, and 90th percentile per timing metric.

Middleware seems a natural place to collect the request metrics data.

We propose to introduce a new middleware to all API node services, including api/streaming/task executor. Each category of REST APIs will have a corresponding request metrics.

==== Alternatives ====
Scales (bundled with Cassandra Python Driver) can be used instead of StatsD.

==== Security Impact ====

* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

==== Notifications Impact ====

==== Other End User Impact ====

==== Performance Impact ====

Performance impact should be minimal since if statsd is used. The metrics sent to statsd is through UDP.

==== Other Deployer Impact ====

A dependency in statsd will be introduced.

==== Developer Impact ====

==== Implementation ====

===== Assignee(s) =====

Charles Wang

===== Work Items =====

===== Dependencies =====

* statsd

==== Documentation Impact ====

==== References ====

MagnetoDB/specs/requestmetrics

2014-11-18T18:46:49Z

Charles Wang:

MagnetoDB/specs/requestmetrics

2014-11-18T18:32:11Z

Charles Wang: MagnetoDB Request Metrics

To proactively address MagnetoDB operational issues, admin user needs real time visibility to request metrics data on each API node. Including:
number of requests
number of failures
number of errors
average latency
median latency
minimum latency
maximum latency
requests per second
distribution of request latency for each type of REST API call, such as "50%","66%","75%","80%","90%","95%","98%","99%","100%"

MagnetoDB/specs/notification

2014-05-27T21:39:45Z

Charles Wang: /* User Stories */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes. This specification defines the notifications that can be used to generate events and aggregate metrics data.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on notifications of table/data item CRUD activities, and the notification mechanism. The monitoring tools and metering metrics will be deferred to later, possibly integrating with Ceilometer.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
* immediate notification
Event is created when a specific increment of usage occurs (such as creation of an table) and notification is sent to notifier immediately
* periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':
<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
The data item events are generated for data item CRUD operations. They have all common attributes of standard MagnetoDB events. In addition, details of read/write operations, such as number of rows impacted, consistency level used, will be in the payload.

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-27T14:59:43Z

Charles Wang: /* Summary */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes. This specification defines the notifications that can be used to generate events and aggregate metrics data.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
* immediate notification
Event is created when a specific increment of usage occurs (such as creation of an table) and notification is sent to notifier immediately
* periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':
<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
The data item events are generated for data item CRUD operations. They have all common attributes of standard MagnetoDB events. In addition, details of read/write operations, such as number of rows impacted, consistency level used, will be in the payload.

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T15:45:33Z

Charles Wang: /* Design and Implementation */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
* immediate notification
Event is created when a specific increment of usage occurs (such as creation of an table) and notification is sent to notifier immediately
* periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':
<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
The data item events are generated for data item CRUD operations. They have all common attributes of standard MagnetoDB events. In addition, details of read/write operations, such as number of rows impacted, consistency level used, will be in the payload.

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T15:04:33Z

Charles Wang: /* Table Events */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':
<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
The data item events are generated for data item CRUD operations. They have all common attributes of standard MagnetoDB events. In addition, details of read/write operations, such as number of rows impacted, consistency level used, will be in the payload.

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:56:31Z

Charles Wang: /* Data Item Events */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
The data item events are generated for data item CRUD operations. They have all common attributes of standard MagnetoDB events. In addition, details of read/write operations, such as number of rows impacted, consistency level used, will be in the payload.

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:51:21Z

Charles Wang: /* Notification Drivers */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
* Log
Log notifier uses MagnetoDB's default logging system
* AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
* AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
* No-OP
No-OP notifier doesn't do anything
* Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:50:11Z

Charles Wang: /* Data Item Events */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

* '''magnetodb.data.getitem'''
* '''magnetodb.data.putitem'''
* '''magnetodb.data.updateitem'''
* '''magnetodb.data.deleteitem'''
* '''magnetodb.data.query,start/.end'''
* '''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:49:42Z

Charles Wang: /* Data Item Events = */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events =====
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

'''magnetodb.data.getitem'''
'''magnetodb.data.putitem'''
'''magnetodb.data.updateitem'''
'''magnetodb.data.deleteitem'''
'''magnetodb.data.query,start/.end'''
'''magnetodb.data.scan,start/.end'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:48:42Z

Charles Wang: /* Detailed Event Types and Payload data: */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* '''magnetodb.table.create.{start,error,end}''':

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* '''magnetodb.table.delete.start/.end''':

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* '''magnetodb.table.usage''':

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events ======
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

'''magnetodb.data.getitem
magnetodb.data.putitem
magnetodb.data.updateitem
magnetodb.data.deleteitem
magnetodb.data.query,start/.end
magnetodb.data.scan,start/.end
'''

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:44:37Z

Charles Wang: /* Event Attributes */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

<pre><nowiki>
{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.
</nowiki></pre>

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* - magnetodb.table.create.{start,error,end}:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* - magnetodb.table.delete.start/.end:

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* - magnetodb.table.usage:

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events ======
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

magnetodb.data.getitem
magnetodb.data.putitem
magnetodb.data.updateitem
magnetodb.data.deleteitem
magnetodb.data.query,start/.end
magnetodb.data.scan,start/.end

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:35:56Z

Charles Wang: /* Event Attributes */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
* priority
* timestamp
* event_type
* tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* - magnetodb.table.create.{start,error,end}:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* - magnetodb.table.delete.start/.end:

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* - magnetodb.table.usage:

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events ======
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

magnetodb.data.getitem
magnetodb.data.putitem
magnetodb.data.updateitem
magnetodb.data.deleteitem
magnetodb.data.query,start/.end
magnetodb.data.scan,start/.end

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:33:34Z

Charles Wang: /* MagnetoDB System Events and Notification Mechanism */

= Summary =
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

= User Stories =
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

= Design and Implementation =
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

== Event Design ==
=== Event Attributes ===
All MagnetoDB events contain the following attributes:
- priority
- timestamp
- event_type
- tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

=== Event Types ===
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

==== Detailed Event Types and Payload data: ====

===== Table Events =====
* - magnetodb.table.create.{start,error,end}:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

* - magnetodb.table.delete.start/.end:

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

* - magnetodb.table.usage:

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

===== Data Item Events ======
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

magnetodb.data.getitem
magnetodb.data.putitem
magnetodb.data.updateitem
magnetodb.data.deleteitem
magnetodb.data.query,start/.end
magnetodb.data.scan,start/.end

=== Enable/Diable Notifications ===

=== Notification Drivers ===
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.

MagnetoDB/specs/notification

2014-05-23T14:29:20Z

Charles Wang: MagnetoDB System Events and Notification

= MagnetoDB System Events and Notification Mechanism =
== Summary ==
MagnetoDB administrators want data on user usage for monitoring, capacity planning, billing, and chargeback purposes.

== User Stories ==
The use case this blueprint tries to address: MongnetoDB administrators want to see user usage data including table creation/delete, data item put/delete/update/retrieval, aggregated into a graph of the total amount of reads, writes, and storage used by all services broken down by domain and project, in order to determine service health, project capacity and billing/chargeback needs.

Initially we will focus on table/data item CRUD events, and event notification mechanism.

== Design and Implementation ==
MagnetoDB system usage data is emitted in the form of usage events from using Oslo Messaging's Notifier component. Notifier is an Oslo incubator project right now. We will use Oslo Notifier to emit events through logging, or send them to a series of AMQP queues (one queue per notification priority), such as table and data item CRUD events. Different types of usage events are distinguished via the notifications' event_type, which is a hierarchical dotted string such as magnetodb.table.create. This way user usage data can be easily grouped for aggregation.

Notifications can be delivered immediately or periodically:
- immediate notification
Notification is created when a specific increment of usage occurs (such as creation of an table)
- periodic notification
Notification is generated by a periodic task, like a cron job, covering usage data for a certain period of time.

=== Event Design ===
==== Event Attributes ====
All MagnetoDB events contain the following attributes:
- priority
- timestamp
- event_type
- tenant_id

The tenant_id is the same as the tenant_id concept in Keystone.

Beside the standard MagnetoDB event attributes, usage events contain a payload of data that will vary depending on the event_type. This is presented as a json-formatted hash of key-value pairs. Some of the keys, such as tenant_id will always be present in any event, others will be data relevant to that event_type (For example, table related notifications will contain data describing the table.)

MagnetoDB events follow the format below:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Note that MagnetoDB itself should not be concerned with billing/charge-backs, but it needs to present a feed of usage data that an external application could use to aggregate the billing data for a requested time period.

Also note that user usage data is for the local MagnetoDB server only, and that data must be aggregated across all MagnetoDB servers to have a full view of the cluster wide usage data.

==== Event Types ====
For a simple and short activity, events are only generated when the activity completes or when error happens.
For a lengthy or asynchronous activity, there are two events generated: an event when the activity starts, and when the activity ends. The events are in the format of:
magnetodb.table.<activity>.start' and 'magnetodb.table.<activity>.end'.
If error happens, instead of .end event, a .error event will be emitted.

Detailed Event Types and Payload data:

- Table Events
magnetodb.table.create.{start,error,end}:

{'message_id': str(uuid.uuid4()),
'publisher_id': 'magnetodb.host1',
'timestamp': timeutils.utcnow(),
'priority': 'INFO',
'event_type': 'magnetodb.table.create',
'tenant_id': user_default_tenant,
'payload': {'schema': ... }}.

Notification upon creation of a new user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "create.error", it will contain error details.

magnetodb.table.delete.start/.end:

Notification upon deletion of a user table.
tenant_id: Tenant ID that owns the this user table (string)
user_id: User ID that owns this user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for this user table (in gb)
message: High-level message describing the notification. If the event type is "delete.error", it will contain error details.

magnetodb.table.usage:

There is no .start/.end event for this activity ... just the 'magnetodb.table.usage' event.

Periodic usage notification generated by the magnetodb-table-usage-audit cron job. These usages are generated for each user table (including secondary indexes) that was active during the specified audit period.
tenant_id: Tenant ID that owns the the user table (string)
user_id: User ID that owns the user table (string)
table_name: user table name (string)
table_schema: user table schema (string)
state: user table status. (string, such as 'active' or 'deleting', 'creating', 'not_exist')
disk_gb: disk allocation for the user table (in gb)
message: High-level message describing the notification. If the event type is "usage.error", it will contain error details.

This notification is disabled by default.

- Data Item Events
These are generated for operations monitoring/profiling purposes and shouldn't directly affect usage/billing. They all follow the same signature as the above calls. Some of these notifications may be called from the above operations (for example, shutdown from delete).

magnetodb.data.getitem
magnetodb.data.putitem
magnetodb.data.updateitem
magnetodb.data.deleteitem
magnetodb.data.query,start/.end
magnetodb.data.scan,start/.end

Enable/Diable Notifications

Notification Drivers
Different notification drivers can be plugged in to define different notifiers. By default, the Log notifier will be used. Some pre-defined notifiers include:
- Log
Log notifier uses MagnetoDB's default logging system
- AMQP queues/RPC
RPC notifier uses AMQP (rabbitmq) queues to send events.
- AMQP queues/RPC with message envelop
RPC notifier uses AMQP (rabbitmq) queues to send events. The only difference between the above one and this one is the message envelop.
- No-OP
No-OP notifier doesn't do anything
- Test
Test notifier stores notification events in memory. This is convenient for unit test purpose.