Jump to: navigation, search

Difference between revisions of "Ceilometer/blueprints/pollster-swift"

 
m (Text replace - "__NOTOC__" to "")
 
(7 intermediate revisions by one other user not shown)
Line 1: Line 1:
−
__NOTOC__
+
 
 
== pollster-swift blueprint ==
 
== pollster-swift blueprint ==
  
Line 17: Line 17:
 
|  '''Resource ID'''  
 
|  '''Resource ID'''  
 
|  '''Volume unit'''  
 
|  '''Volume unit'''  
−
|  '''Payload'''  
+
|  '''Metadata'''  
 
|  '''Type'''  
 
|  '''Type'''  
−
|  '''Note'''
 
 
|-
 
|-
 
|  o1  
 
|  o1  
−
|  obj_volume
+
|  storage.objects
 
|  must  
 
|  must  
 
|  swift account id  
 
|  swift account id  
−
|  Megabytes
+
|  number of objects
 
|   
 
|   
−
|  Cumulative
+
|  Gauge
−
|  total object volume stored
 
 
|-
 
|-
 
|  o2  
 
|  o2  
−
|  obj_in
+
|  storage.objects.size
 
|  must  
 
|  must  
 
|  swift account id  
 
|  swift account id  
−
|  Kbytes
+
|  bytes
 
|   
 
|   
−
|  Cumulative
+
|  Gauge
−
|  volume of data received for this account (int and ext)
 
 
|-
 
|-
 
|  o3  
 
|  o3  
−
|  obj_out
+
|  storage.objects.containers
 
|  must  
 
|  must  
 
|  swift account id  
 
|  swift account id  
−
|  Kbytes
+
|  number of containers
 
|   
 
|   
−
|  Delta
+
|  Gauge
−
|  volume of data sent for this account (int and ext)
 
 
|-
 
|-
 
|  o4  
 
|  o4  
−
|  obj_in_int
+
|  storage.containers.objects
 
|  may  
 
|  may  
−
|  swift account id
+
|  container name
−
|  Kbytes
+
|  number of objects
 
|   
 
|   
−
|  Delta
+
|  Gauge
−
|  volume of data received from internal network source
 
 
|-
 
|-
 
|  o5  
 
|  o5  
−
|  obj_in_ext
+
|  storage.containers.objects.size
 
|  may  
 
|  may  
−
|  swift account id
+
|  container name
−
|  Kbytes
+
|  bytes
 
|   
 
|   
−
|  Delta
+
|  Gauge
−
|  volume of data received from external network source
 
 
|-
 
|-
 
|  o6  
 
|  o6  
−
|  obj_out_int
+
|  storage.objects.upload.internal
 
|  may  
 
|  may  
−
|  swift account id
+
|  container name
−
|  Kbytes
+
|  number of req
 
|   
 
|   
 
|  Delta  
 
|  Delta  
−
|  volume of data sent to internal network dest
 
 
|-
 
|-
 
|  o7  
 
|  o7  
−
|  obj_out_ext
+
|  storage.objects.download.internal
 
|  may  
 
|  may  
−
|  swift account id
+
|  container name
−
|  Kbytes
+
|  number of req
 
|   
 
|   
 
|  Delta  
 
|  Delta  
−
|  volume of data sent to external network destinations
 
 
|-
 
|-
 
|  o8  
 
|  o8  
−
|  obj_number
+
|  storage.objects.update.internal
−
|  must
+
|  may
−
|  swift account id
+
|  container name
 +
|  number of req
 
|   
 
|   
−
|  container
+
|  Delta
−
|  Gauge
 
−
|  Number of objects stored for a container. The resource_id is the container id.
 
 
|-
 
|-
 
|  o9  
 
|  o9  
−
|  obj_containers
+
|  storage.objects.delete.internal
 +
|  may
 +
|  container name
 +
|  number of req
 +
| 
 +
|  Delta
 +
|-
 +
|  o10
 +
|  storage.objects.upload.size.internal
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o11
 +
|  storage.objects.download.size.internal
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o12
 +
|  storage.objects.update.size.internal
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o13
 +
|  storage.objects.delete.size.internal
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o14
 +
|  storage.objects.upload.external
 +
|  may
 +
|  container name
 +
|  number of req
 +
| 
 +
|  Delta
 +
|-
 +
|  o15
 +
|  storage.objects.download.external
 +
|  may
 +
|  container name
 +
|  number of req
 +
| 
 +
|  Delta
 +
|-
 +
|  o16
 +
|  storage.objects.update.external
 +
|  may
 +
|  container name
 +
|  number of req
 +
| 
 +
|  Delta
 +
|-
 +
|  o17
 +
|  storage.objects.delete.external
 +
|  may
 +
|  container name
 +
|  number of req
 +
| 
 +
|  Delta
 +
|-
 +
|  o18
 +
|  storage.objects.upload.size.external
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o19
 +
|  storage.objects.download.size.external
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o20
 +
|  storage.objects.update.size.external
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o21
 +
|  storage.objects.delete.size.external
 +
|  may
 +
|  container name
 +
|  bytes
 +
| 
 +
|  Delta
 +
|-
 +
|  o22
 +
|  storage.objects.incoming.bytes
 
|  must  
 
|  must  
 
|  swift account id  
 
|  swift account id  
 +
|  bytes
 
|   
 
|   
−
|  Number of containers
+
|  Delta
−
|  Gauge
 
−
| 
 
 
|-
 
|-
−
|  o10
+
|  o23
−
|  obj_requests
+
|  storage.objects.outgoing.bytes
 
|  must  
 
|  must  
−
|   swift account id  
+
| swift account id  
−
|  
+
| bytes
−
|  type
+
|   
 
|  Delta  
 
|  Delta  
−
|  Number of HTTP requests, type being the request type (GET/HEAD/PUT/POST…)
 
 
|}
 
|}
  
−
The following counters must be implemented through polling of Swift:
+
The counters o1 to o5 must be implemented through polling of Swift.
−
obj_volume, obj_number, obj_containers, obj_requests, obj_in, obj_out
+
 
 +
For polling Swift, there exists 2 way:
 +
 
 +
1. Go through swift-proxy as an user with [[ResellerAdmin]] role on one of this tenant, using this tenant to get a token
 +
 
 +
2. Use the ring to determine the right swift-<type>-server and request information from it directly. The swift-account-server can provides number and size of objects stored in all containers, and the number of containers, and the list of container. The swift-container-server can provides number number of objects and size of each container.
 +
 
 +
Option 1 should be simpler and cleaner in a first time, but puts more load on Keystone. Option 2 could be preferred, if implemented, via an option.
 +
 
 +
The counters o6 to o21 should be implemented via a Swift middleware counting requests and their size and publishing counters regularly.
  
−
In addition, if we can distinguish between external and internal networks, it would be nice to also have:
+
To distinguish network, it is suggested that we start this implementation by using a configuration file listing internal networks, which by default should list A, B and C private network class (RFC 1918).
−
obj_in_int, obj_in_ext, obj_out_int, obj_out_ext.
 
−
I would suggest that we start this implementation by using a configuration file listing internal networks, which by default should list A, B and C private network class (RFC 1918)
 

Latest revision as of 23:29, 17 February 2013

pollster-swift blueprint

Use Cases

  • Provider wants to retrieve data about swift usage to bill his customers
  • Customer wants to monitor his usage of swift

Proposed implementation

The following is a first list of meters that needs to be collected in order to allow billing systems to perform their tasks on swift. This list must be expandable over time and each administrator must have the possibility to enable or disable each meter based on his local needs

Meter name Priority Resource ID Volume unit Metadata Type
o1 storage.objects must swift account id number of objects Gauge
o2 storage.objects.size must swift account id bytes Gauge
o3 storage.objects.containers must swift account id number of containers Gauge
o4 storage.containers.objects may container name number of objects Gauge
o5 storage.containers.objects.size may container name bytes Gauge
o6 storage.objects.upload.internal may container name number of req Delta
o7 storage.objects.download.internal may container name number of req Delta
o8 storage.objects.update.internal may container name number of req Delta
o9 storage.objects.delete.internal may container name number of req Delta
o10 storage.objects.upload.size.internal may container name bytes Delta
o11 storage.objects.download.size.internal may container name bytes Delta
o12 storage.objects.update.size.internal may container name bytes Delta
o13 storage.objects.delete.size.internal may container name bytes Delta
o14 storage.objects.upload.external may container name number of req Delta
o15 storage.objects.download.external may container name number of req Delta
o16 storage.objects.update.external may container name number of req Delta
o17 storage.objects.delete.external may container name number of req Delta
o18 storage.objects.upload.size.external may container name bytes Delta
o19 storage.objects.download.size.external may container name bytes Delta
o20 storage.objects.update.size.external may container name bytes Delta
o21 storage.objects.delete.size.external may container name bytes Delta
o22 storage.objects.incoming.bytes must swift account id bytes Delta
o23 storage.objects.outgoing.bytes must swift account id bytes Delta

The counters o1 to o5 must be implemented through polling of Swift.

For polling Swift, there exists 2 way:

1. Go through swift-proxy as an user with ResellerAdmin role on one of this tenant, using this tenant to get a token

2. Use the ring to determine the right swift-<type>-server and request information from it directly. The swift-account-server can provides number and size of objects stored in all containers, and the number of containers, and the list of container. The swift-container-server can provides number number of objects and size of each container.

Option 1 should be simpler and cleaner in a first time, but puts more load on Keystone. Option 2 could be preferred, if implemented, via an option.

The counters o6 to o21 should be implemented via a Swift middleware counting requests and their size and publishing counters regularly.

To distinguish network, it is suggested that we start this implementation by using a configuration file listing internal networks, which by default should list A, B and C private network class (RFC 1918).

Retrieved from "https://wiki.openstack.org/w/index.php?title=Ceilometer/blueprints/pollster-swift&oldid=16633"