Difference between revisions of "SearchProject"

Revision as of 06:04, 7 November 2013

Summary

Search service provides quick full text search with powerful query language across all the resources.

Why

Horizon and other openstack admin clients will benefit from “quick search”. At scale, search based UI is the way to go. Clients need queries like “show everything for user John on project Startrack.” Some filtering is available for some resources, but it’s not enough power, flexibility, and responsiveness for cross-resource search implementation.

How

Disclaimer: this is 'one way to do it', we are open for alternatives & suggestions.

Consists of:

1. Search index engine, with query API. Use Solr first, support Elasticsearch, may be Whoosh, Xapian later. Query language: standard lucene syntax is a good star
2. Clients (Horizon, cli, etc)
3. Data Publisher – fill up the index and keep it current

Search index is treated as “cache”: used resource IDs from OpenStack services. Rebuilds in case of failures or corruptions. Must rebuild in seconds (throughput 10K records/sec)

Data publishers:

1. On start – query the resources, publishes data to index service.
2. Listen to updates on resources - use notification system, rely on resources publishing CRUD operations.
3. Periodically: poll for resources not updated via notification system, run consistency checks.

Discussion - ceilometer integration

(1) is new and specific functionality (2) and (3) is what Ceilometer and others (StackTach) is already doing.

In common:

1. Common - architecture patterns, data flow.
2. Common configuration requirements - enable publishing notifications.
3. Common problems: some resources don’t publish [some] updates. E.g., Instance ‘rename’ doesn’t send a notification (bug?).

Gaps:

1. Data have different ‘meaning’: meters carry metrics (instance id / duration), search needs updates on properties (instance id / name / power status). But it is part of revised Ceilometer charter: “...a tool to watch for variations in key values...”
2. Current meters are not sufficient – we shall implement the missing ones.

What can we leverage and how?

One way we can think of:

• Use meters to publish updates on property changes: build more meters.
• Use Multi-Publisher: implement SearchPublisher: filter relevant metrics - resource property updates - and publish to search index

Other ideas?

Roadmap

Currently, it’s a well working prototype. No proper openstack integration, using hacks and back doors (e.g. direct DB access)

First steps

bring to opensource, make fully functional

• Clean up and opensource (matter of separating search out as a standalone project)
• Proper StackForge integration
• Replace hacks with proper integration with OpenStack (Keystone, Oslo, may be db / sql-alchemy, etc...)
• Expand to other resources (networks, volumes, images, etc)

Next:

make ‘proper and ready’ for OpenStack incubation

• Query API Facade
• Figure proper ceilometer integration
• Revisit data publisher design, may re-implement

Finally - v1.0

• Index failure detection and recovery
• Extensibility developer guidelines: how to make resource indexable, how to add a property to existing resource, etc.
• Full mutli-tenancy support?

Questions:

1. Is it relevant to OpenStack vision and directions?
2. Can potentially qualify for OpenStack incubation, when ready?
3. Who is interested to join & contribute?