Ceilometer/blueprints/support-resources-pipeline-item

Generic way to specify the endpoint to poll
Related blueprints:
 * https://blueprints.launchpad.net/ceilometer/+spec/support-resources-pipeline-item

During the openstack HongKong summit session of how to improve the central agent, it is agreed that we should have a way to allow the administrator to specify which endpoints to poll to get the data from, for the central agent. This will gives the administrators much more flexibility and fine grained control of their deployment. Here we propose the following design to support this.

Changes to the YAML pipeline definition file and pipeline manager
The interpretation of the content of the resources list should depend on different pollsters. The content could directly specifying which endpoints the pollster should poll from, or specify a file contains the list of endpoints the pollsters want to poll from. The pipeline manager will only load the resources from the pipeline file and do a very simple type check.
 * Adding new optional list item 'resources' in the pipeline definition

The new pipeline file would be something like the following: - name: meter_pipeline interval: 600 meters: - "meterFoo" resources: - snmp://10.4.4.4 - ipmi://10.5.5.5 - file:///foo/bar/resources_file transformers: publishers: - rpc:// Question: Should we explicitly require that the content of the resources should be URI compatible?

Changes to the Pollster class
def get_samples(self, manager, cache, resources=[]): def verify_resources(self, manager, resources): Verify if the resources are legal to this pollster return True
 * new parameter added to the method get_samples to specify the endpoints to get data from
 * new method added to the pollster to verify the resources is legal to this pollster

Changes to AgentManger and PollingTask
Question: should we make the changes only to the central manager's PollingTask or to the base PollingTask class?
 * The PollingTasks class should be modified to support passing resources defined in the pipeline configuration to the corresponding pollsters, and also to verify the resources.

Detect the changes in the resources
We want to avoid restarting the central agent when the resources are incrementally added into the pipeline configuration file. There are 2 ways to achieve this:
 * Adding reload in the pipeline manager to detect the change of the pipeline file. The PollingTask should try 'reload' the new resources and then pass them to the pollsters in each interval iteration.
 * Question: What if other content of the pipeline changes, e.g. the interval, the publisher, the meter, etc.? That definitely requires a restart to setup a whole different set of PollingTasks.


 * Put all the endpoints definition in a separate file other than the pipeline.yaml, and pass the file name of that file as the resources to the pollsters. Let the pollsters to detect the changes of the content and reload the new endpoints definitions.
 * Question: Should we only allow setting the endpoints in this way, or should we also allow putting the endpoints directly into the resources list?