Difference between revisions of "Watcher"

Revision as of 14:13, 6 October 2015

Mission

OpenStack Watcher provides a flexible and scalable resource optimization service for multi-tenant OpenStack-based clouds. Watcher provides a complete optimization loop—including everything from a metrics receiver, complex event processor and profiler, optimization processor and an action plan applier. This provides a robust framework to realize a wide range of cloud optimization goals, including the reduction of data center operating costs, increased system performance via intelligent virtual machine migration, increased energy efficiency—and more!

Not only does Watcher provide several out-of-box optimization routines for immediate value-add, but it also supports a pluggable architecture by which custom optimization algorithms, data metrics and data profilers can be developed and inserted into the Watcher framework. Additionally, Watcher enables two different modes of execution—advise mode (manual) or active mode (automatic), giving cloud administrators the runtime flexibilities that their clouds require.

Most importantly, administrators of OpenStack-based clouds equipped with Watcher will decrease their Total Cost of Ownership (TCO) by way of more efficient use of their infrastructure and less “hands on” (read: manual) administrator involvement to perform optimizations.

OpenStack Ecosystem

It is important to understand how Watcher is positioned relative to other projects in the OpenStack ecosystem. As mentioned previously, Watcher provides resource optimization for the cloud—complementing projects like Nova and Cinder which are more focused on the initial placement of resources (e.g., VMs and volumes, respectively). Over time, clouds can start running in a sub optimal fashion in terms of application performance, energy consumption, etc. However, this is where Watcher steps into the light and helps cloud administrators tune and re balance their clouds using advanced algorithms to build recommended action plans! This can lead to better application throughput, lower data center operating costs, longer hardware life, less involvement from cloud administrators to troubleshoot and tune the cloud—just to name a few.

Meetings

We meet weekly on Wednesdays at 15:00 UTC in the #openstack-watcher IRC channel Watcher Meeting Agenda

Use cases

The primary objective of this section is to articulate the primary use cases of Watcher and its reference architecture—and how it can be used to optimize an OpenStack cloud; it does not necessarily cover every conceivable use case that Watcher can fulfill.

Use Case 1

As a cloud administrator using Watcher, I want to optimize my data center by performing virtual machine migration based on an optimization goal (e.g., outlet temperature, airflow inlet temperature and power consumption or any other platform-specific measurements or a function of other goals—i.e., multi-dimensional goals).
Dependencies:

• Uses live migration
• Uses Nova scheduler filters (e.g., VM affinity, VM anti-affinity, etc.)
• Correct set of host metrics (e.g., all the temperatures)

Use Case 2

As a cloud administrator using Watcher, I want to specify a threshold (i.e., an objective function) that defines the point at which Watcher needs to optimize the environment.

Use Case 3

As a cloud administrator, I want to be able to define an optimization goal for a set of hosts (e.g., using host aggregates), so that I can have multiple goals within my data center based on the optimization objective of each set of hosts (e.g., “production” vs. “development” vs. “test” host systems).

Use Case 4

As a Watcher plugin developer, I want an easy way to feed in new sources of data, so that I can extend Watcher to work on data sets it doesn’t natively support.

Use Case 5

As a cloud administrator, I want to be able to configure Watcher to run in “advise mode”, so that I can see what optimizations Watcher would make without necessarily allowing Watcher to perform them.

Use Case 6

As a cloud administrator, I want Watcher to understand (read: know about) out-of-the-box host metrics (e.g., # of vcpus, CPU utilization %, memory used), so that I can create optimization goals very easily.
Notes:

• The reference implementation might retrieve this using Ceilometer or some other approach; still T.B.D.

Use Case 7

As a Watcher plugin developer, I want Watcher to provide well-defined interfaces for each of its logical components, so that I can implement the interfaces to provide whatever functionality is important to me.

Use Case 8

As a cloud administrator, I want to be able to easily see what optimizations Watcher has made, so that I know what’s going on in my data center while I’m not necessarily watching Watcher.

Use Case 9

As a cloud administrator, I want to be able to configure Watcher to run in “active mode”, so that I can let Watcher perform its optimizations automatically.

Use Case 10

As a cloud administrator, I want Watcher to support multi-tenancy (e.g., a customer that different SLAs may allow their VMs to be placed on certain hosts).

Architecture

• Watcher run as a set of services within an OpenStack control plane (i.e., be able to access other OpenStack services by way of RPC calls via the OpenStack message queue).
• Watcher is able to communicate with a number of cloud management control points (e.g., OpenStack Nova for VM migration operations, OpenStack Keystone for authentication services, etc.).
• Watcher is able to access resource metrics via a flexible “metrics driver” architecture (e.g., via Ceilometer, Gnocchi, Nagios, proprietary data collectors, etc.).
• For each phase in the overall optimization process, Watcher provides extension points such that different implementations can be swapped in and out to achieve various optimization goals.
• Watcher generates AMQP notifications for any action that it performs via an asynchronous notification system (i.e., this is similar to the “Nova notifications” system), allowing downstream entities to be notified in an event-driven fashion for all Watcher-invoked actions.
  

More detailed architecture info can be found here.

High Level Optimization Process

The various phases of the Watcher optimization process can be found here https://docs.google.com/presentation/d/1sR-k3KdgEMHzpKyvUF7hhkoITe0JDWSd_C6fU0NAby4/edit#slide=id.gca3d12c11_2_52

Project Launchpad

https://launchpad.net/watcher

Licensing

Watcher is licensed under the Apache License v2.

Documentation

Watcher documentation : https://factory.b-com.com/www/watcher/doc/watcher
Watcher CLI documentation : https://factory.b-com.com/www/watcher/doc/python-watcherclient

Source code

Source code of the module is available on Github : https://github.com/stackforge/watcher
We also provide code for the Watcher client : https://github.com/stackforge/python-watcherclient