SearchProject

Summary
Search service provides quick full text search with powerful query language across all the resources.

Why
See the prototype in action - 2 min video screencast

How
Disclaimer: this is 'one way to do it', we are open for alternatives & suggestions.

Consists of: Search index is treated as “cache”: used resource IDs from OpenStack services. Rebuilds in case of failures or corruptions. Must rebuild in seconds (throughput 10K records/sec)
 * 1)    Search      index engine, with query API. Use Solr first, support Elasticsearch, may      be Whoosh, Xapian later. Query language: standard lucene syntax is a good      star
 * 2)    Clients      (Horizon, cli, etc)
 * 3)    Data Publisher – fill up the      index and keep it current

Data publishers:


 * 1) On start – query the resources,      publishes data to index service.
 * 2) Listen to updates on resources -      use notification system, rely on resources publishing CRUD operations.
 * 3) Periodically: poll for resources      not updated via notification system, run consistency checks.

Code
Disclaimer: the code is a working prototype, a starting point for discussion, and may be a reference to a potential future implementation, not the actual implementation.
 * Search backend: https://github.com/StackStorm/search
 * Horizon plugin: https://github.com/StackStorm/search-horizon

Ceilometer integration
(1) is new and specific functionality (2) and (3) is what Ceilometer and others (StackTach) is already doing.

In common:

 * 1) Common - architecture patterns, data flow.
 * 2) Common configuration requirements  - enable publishing notifications.
 * 3) Common problems: some resources don’t publish [some] updates. E.g., Instance ‘rename’ doesn’t send a notification (bug?).

Gaps:

 * 1) Data have different ‘meaning’: meters carry metrics (instance id / duration), search needs updates on properties (instance id / name / power status). But it is part of revised Ceilometer charter: “...a tool to watch for variations in key values...”
 * 2) Current meters are not sufficient – we shall implement the missing ones.

What can we leverage and how?
One way we can think of: Other ideas?
 * Use meters to publish updates on property changes: build more meters.
 * Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index
 * Julien@Ceilometer: instead of using multi-publisher where the metric, Ceilometer can put it to Search index earlier in the chain, where it takes the message from the message queue and saves the data to DB.

Roadmap
Currently, it’s a well working prototype. No proper openstack integration, using hacks and back doors (e.g. direct DB access)

First steps
bring to opensource, make fully functional
 * Clean up and opensource (matter of separating search out as a standalone project) - done

TBD
The roadmap to be defined by community discussion

Questions:

 * 1) Is it relevant to OpenStack vision and directions?
 * 2) Can potentially qualify for OpenStack incubation, when ready?
 * 3) Who is interested to join &  contribute?