Difference between revisions of "SearchProject"
m (→How) |
m (→Why) |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 12: | Line 12: | ||
[[File:Search Horizon Prototype Screen Shot 2013-11-06 at 12.19.03 PM.png|thumbnail|right]] | [[File:Search Horizon Prototype Screen Shot 2013-11-06 at 12.19.03 PM.png|thumbnail|right]] | ||
|} | |} | ||
| + | |||
| + | See the prototype in action - [http://youtu.be/MpEruCij2gc 2 min video screencast] | ||
== How == | == How == | ||
| Line 30: | Line 32: | ||
# Periodically: poll for resources not updated via notification system, run consistency checks. | # Periodically: poll for resources not updated via notification system, run consistency checks. | ||
| − | == Discussion | + | == Code == |
| + | '''Disclaimer:''' the code is a working prototype, a starting point for discussion, and may be a reference to a potential future implementation, not the actual implementation. | ||
| + | * Search backend: https://github.com/StackStorm/search | ||
| + | * Horizon plugin: https://github.com/StackStorm/search-horizon | ||
| + | |||
| + | == Discussion == | ||
| + | === Ceilometer integration === | ||
(1) is new and specific functionality | (1) is new and specific functionality | ||
(2) and (3) is what Ceilometer and others (StackTach) is already doing. | (2) and (3) is what Ceilometer and others (StackTach) is already doing. | ||
| Line 46: | Line 54: | ||
* Use meters to publish updates on property changes: build more meters. | * Use meters to publish updates on property changes: build more meters. | ||
* Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index | * Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index | ||
| + | ** Julien@Ceilometer: instead of using multi-publisher where the metric , Ceilometer can put it to Search index earlier in the chain, where it takes the message from the message queue and saves the data to DB. | ||
Other ideas? | Other ideas? | ||
| Line 53: | Line 62: | ||
==== '''First steps''' ==== | ==== '''First steps''' ==== | ||
bring to opensource, make fully functional | bring to opensource, make fully functional | ||
| − | * Clean up and opensource (matter of separating search out as a standalone project) | + | * Clean up and opensource (matter of separating search out as a standalone project) - '''done''' |
| − | + | ==== TBD ==== | |
| − | + | The roadmap to be defined by community discussion | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | ==== | ||
| − | |||
| − | |||
| − | |||
== Questions: == | == Questions: == | ||
Latest revision as of 00:22, 14 November 2013
Contents
Summary
Search service provides quick full text search with powerful query language across all the resources.
Why
|
Horizon and other openstack admin clients will benefit from “quick search”. At scale, search based UI is the way to go. Clients need queries like “show everything for user John on project Startrack.” Some filtering is available for some resources, but it’s not enough power, flexibility, and responsiveness for cross-resource search implementation. |
See the prototype in action - 2 min video screencast
How
Disclaimer: this is 'one way to do it', we are open for alternatives & suggestions.
Consists of:
- Search index engine, with query API. Use Solr first, support Elasticsearch, may be Whoosh, Xapian later. Query language: standard lucene syntax is a good star
- Clients (Horizon, cli, etc)
- Data Publisher – fill up the index and keep it current
Search index is treated as “cache”: used resource IDs from OpenStack services. Rebuilds in case of failures or corruptions. Must rebuild in seconds (throughput 10K records/sec)
Data publishers:
- On start – query the resources, publishes data to index service.
- Listen to updates on resources - use notification system, rely on resources publishing CRUD operations.
- Periodically: poll for resources not updated via notification system, run consistency checks.
Code
Disclaimer: the code is a working prototype, a starting point for discussion, and may be a reference to a potential future implementation, not the actual implementation.
- Search backend: https://github.com/StackStorm/search
- Horizon plugin: https://github.com/StackStorm/search-horizon
Discussion
Ceilometer integration
(1) is new and specific functionality (2) and (3) is what Ceilometer and others (StackTach) is already doing.
In common:
- Common - architecture patterns, data flow.
- Common configuration requirements - enable publishing notifications.
- Common problems: some resources don’t publish [some] updates. E.g., Instance ‘rename’ doesn’t send a notification (bug?).
Gaps:
- Data have different ‘meaning’: meters carry metrics (instance id / duration), search needs updates on properties (instance id / name / power status). But it is part of revised Ceilometer charter: “...a tool to watch for variations in key values...”
- Current meters are not sufficient – we shall implement the missing ones.
What can we leverage and how?
One way we can think of:
- Use meters to publish updates on property changes: build more meters.
- Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index
- Julien@Ceilometer: instead of using multi-publisher where the metric , Ceilometer can put it to Search index earlier in the chain, where it takes the message from the message queue and saves the data to DB.
Other ideas?
Roadmap
Currently, it’s a well working prototype. No proper openstack integration, using hacks and back doors (e.g. direct DB access)
First steps
bring to opensource, make fully functional
- Clean up and opensource (matter of separating search out as a standalone project) - done
TBD
The roadmap to be defined by community discussion
Questions:
- Is it relevant to OpenStack vision and directions?
- Can potentially qualify for OpenStack incubation, when ready?
- Who is interested to join & contribute?
