SearchProject
Contents
Summary
Search service provides quick full text search with powerful query language across all the resources.
Why
Horizon and other openstack admin clients will benefit from “quick search”. At scale, search based UI is the way to go. Clients need queries like “show everything for user John on project Startrack.” Some filtering is available for some resources, but it’s not enough power, flexibility, and responsiveness for cross-resource search implementation. |
See the prototype in action - 2 min video http://youtu.be/MpEruCij2gc
How
Disclaimer: this is 'one way to do it', we are open for alternatives & suggestions.
Consists of:
- Search index engine, with query API. Use Solr first, support Elasticsearch, may be Whoosh, Xapian later. Query language: standard lucene syntax is a good star
- Clients (Horizon, cli, etc)
- Data Publisher – fill up the index and keep it current
Search index is treated as “cache”: used resource IDs from OpenStack services. Rebuilds in case of failures or corruptions. Must rebuild in seconds (throughput 10K records/sec)
Data publishers:
- On start – query the resources, publishes data to index service.
- Listen to updates on resources - use notification system, rely on resources publishing CRUD operations.
- Periodically: poll for resources not updated via notification system, run consistency checks.
Code
Disclaimer: the code is a working prototype, a starting point for discussion, and may be a reference to a potential future implementation, not the actual implementation.
- Search backend: https://github.com/StackStorm/search
- Horizon plugin: https://github.com/StackStorm/search-horizon
Discussion
Ceilometer integration
(1) is new and specific functionality (2) and (3) is what Ceilometer and others (StackTach) is already doing.
In common:
- Common - architecture patterns, data flow.
- Common configuration requirements - enable publishing notifications.
- Common problems: some resources don’t publish [some] updates. E.g., Instance ‘rename’ doesn’t send a notification (bug?).
Gaps:
- Data have different ‘meaning’: meters carry metrics (instance id / duration), search needs updates on properties (instance id / name / power status). But it is part of revised Ceilometer charter: “...a tool to watch for variations in key values...”
- Current meters are not sufficient – we shall implement the missing ones.
What can we leverage and how?
One way we can think of:
- Use meters to publish updates on property changes: build more meters.
- Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index
- Julien@Ceilometer: instead of using multi-publisher where the metric , Ceilometer can put it to Search index earlier in the chain, where it takes the message from the message queue and saves the data to DB.
Other ideas?
Roadmap
Currently, it’s a well working prototype. No proper openstack integration, using hacks and back doors (e.g. direct DB access)
First steps
bring to opensource, make fully functional
- Clean up and opensource (matter of separating search out as a standalone project)
- Proper StackForge integration
- Replace hacks with proper integration with OpenStack (Keystone, Oslo, may be db / sql-alchemy, etc...)
- Expand to other resources (networks, volumes, images, etc)
Next:
make ‘proper and ready’ for OpenStack incubation
- Query API Facade
- Figure proper ceilometer integration
- Revisit data publisher design, may re-implement
Finally - v1.0
- Index failure detection and recovery
- Extensibility developer guidelines: how to make resource indexable, how to add a property to existing resource, etc.
- Full mutli-tenancy support?
Questions:
- Is it relevant to OpenStack vision and directions?
- Can potentially qualify for OpenStack incubation, when ready?
- Who is interested to join & contribute?