SearchProject

Summary

Search service provides quick full text search with powerful query language across all the resources.

Why

Horizon and other openstack admin clients will benefit from “quick search”. At scale, search based UI is the way to go. Clients need queries like “show everything for user John on project Startrack.” Some filtering is available for some resources, but it’s not enough power, flexibility, and responsiveness for cross-resource search implementation.

See the prototype in action - 2 min video screencast

How

Disclaimer: this is 'one way to do it', we are open for alternatives & suggestions.

Consists of:

1. Search index engine, with query API. Use Solr first, support Elasticsearch, may be Whoosh, Xapian later. Query language: standard lucene syntax is a good star
2. Clients (Horizon, cli, etc)
3. Data Publisher – fill up the index and keep it current

Search index is treated as “cache”: used resource IDs from OpenStack services. Rebuilds in case of failures or corruptions. Must rebuild in seconds (throughput 10K records/sec)

Data publishers:

1. On start – query the resources, publishes data to index service.
2. Listen to updates on resources - use notification system, rely on resources publishing CRUD operations.
3. Periodically: poll for resources not updated via notification system, run consistency checks.

Code

Disclaimer: the code is a working prototype, a starting point for discussion, and may be a reference to a potential future implementation, not the actual implementation.

• Search backend: https://github.com/StackStorm/search
• Horizon plugin: https://github.com/StackStorm/search-horizon

Discussion

Ceilometer integration

(1) is new and specific functionality (2) and (3) is what Ceilometer and others (StackTach) is already doing.

In common:

1. Common - architecture patterns, data flow.
2. Common configuration requirements - enable publishing notifications.
3. Common problems: some resources don’t publish [some] updates. E.g., Instance ‘rename’ doesn’t send a notification (bug?).

Gaps:

1. Data have different ‘meaning’: meters carry metrics (instance id / duration), search needs updates on properties (instance id / name / power status). But it is part of revised Ceilometer charter: “...a tool to watch for variations in key values...”
2. Current meters are not sufficient – we shall implement the missing ones.

What can we leverage and how?

One way we can think of:

• Use meters to publish updates on property changes: build more meters.
• Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index
  • Julien@Ceilometer: instead of using multi-publisher where the metric , Ceilometer can put it to Search index earlier in the chain, where it takes the message from the message queue and saves the data to DB.

Other ideas?

Roadmap

Currently, it’s a well working prototype. No proper openstack integration, using hacks and back doors (e.g. direct DB access)

First steps

bring to opensource, make fully functional

• Clean up and opensource (matter of separating search out as a standalone project) - done

TBD

The roadmap to be defined by community discussion

Questions:

1. Is it relevant to OpenStack vision and directions?
2. Can potentially qualify for OpenStack incubation, when ready?
3. Who is interested to join & contribute?