Jump to: navigation, search


Revision as of 23:26, 3 March 2015 by StackScribe (talk | contribs)

(Moved original to the bottom of the page for reference.)

Proposed Revision

Strategy and assumptions:

  1. Audience is people who have some experience installing OpenStack, not first time users
  2. Focus on installation of OpenStack core services
  3. Structure the guide sequentially -- the steps to take in a reasonable order
  4. Avoid redundancy with the Install Guide; for steps that are identical for HA and non-HA installations, link to appropriate sections in the Install Guide
  5. One guide for all Linux distros/platforms
  6. Emphasize a reasonable, standard deployment based on open source components. We can provide some notes about alternatives as appropriate (for example, using a commercial load-balancer might be a better alternative than relying on HAProxy) and perhaps a link to the OpenStack Marketplace.


HA Intro and Concepts

  1. Redundancy and failover
  2. Stateless/stateful, active/passive, active/active (Keep: http://docs.openstack.org/high-availability-guide/content/stateless-vs-stateful.html)
  3. Quorums; many services should use an odd number of nodes equal to or greater than 3
  4. Single-controller HA mode and scaling up to 3 or more

Hardware setup

  1. Minimal Architecture Example -- Network Layout, styled as in http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites for easy comparison


  1. Link to Install Guide: Install O/S on each node
  2. Install pacemaker, crmsh, corosync, cluster-glue, fence-agents (Fedora only), resource-agents. (Modify: http://docs.openstack.org/high-availability-guide/content/_install_packages.html)
  3. What is needed for LSB/upstart/systemd alternative to OCF scripts (RA) for Pacemaker? See https://bugs.launchpad.net/openstack-manuals/+bug/1349398
  4. Set up and start Corosync and Pacemaker. Stick with 'crm' tool for Ubuntu/Debian and 'pcs' for RHEL/Fedora (Modify http://docs.openstack.org/high-availability-guide/content/_set_up_corosync.html; Modify: http://docs.openstack.org/high-availability-guide/content/_start_pacemaker.html)
  5. Set basic cluster properties (Modify: http://docs.openstack.org/high-availability-guide/content/_set_basic_cluster_properties.html))
  6. Configure fencing for Pacemaker cluster (Links to http://clusterlabs.org/doc/)
  7. Configure the VIP (Keep: http://docs.openstack.org/high-availability-guide/content/s-api-vip.html )
  8. API services -- Do those belong here or in specific sections? (Modify Glance API: http://docs.openstack.org/high-availability-guide/content/s-glance-api.html and Modify Cinder API: http://docs.openstack.org/high-availability-guide/content/s-cinder-api.html )
  9. Schedulers
  10. Memcached service on Controllers (Keep: http://docs.openstack.org/high-availability-guide/content/_memcached.html , which links to http://code.google.com/p/memcached/wiki/NewStart for specifics)

Configure networking on each node

  1. Rather than configuring neutron here, we should simply mention physical network HA methods (e.g., bonding) and additional node/network requirements for L3HA and DVR for planning purposes. As this point the, the networking guide likely won't cover the former.
  2. Link to Networking Guide
  3. (Neutron agents should be described for active/active; deprecate single agent's instances case)
  4. For Kilo and beyond, focus on L3HA and DVR

Install and Configure MySQL

  1. Two nodes plus GARBD.
  2. MySQL with Galera
  3. Pacemaker multistate clone resource for Galera cluster
  4. Pacemaker resource agent for Galera cluster management
  5. Deprecate MySQL DRBD configuration because of split-brain issues

RabbitMQ Message broker

  1. Install and configure message broker on Controller; see http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites
  2. Oslo messaging for active/active
  1. I think services need some special configuration with more than two nodes?
  1. No need for active/passive AMQP; Two-node active/active cluster with mirrored queues instead
  2. Pacemaker multistate clone resource for RabbitMQ cluster
  3. Pacemaker resource agent for RabbitMQ cluster management
  4. Deprecate DRBD for RabbitMQ


Does this go here or in "Prerequisites" section above?

  1. I think Oslo supports hash synchronization so this shouldn't take more than load balancing.


  1. Run NTP servers on every controller and configure other nodes to use all of them for synchronization. Link to http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-ntp

Keystone Identity services

  1. Install Guide for concepts: http://docs.openstack.org/juno/install-guide/install/apt/content/keystone-concepts.html
  2. Install Guide to configure prerequisites, install and configure the components, and finalize the installation: http://docs.openstack.org/juno/install-guide/install/apt/content/keystone-install.html
  3. Configure Keystone for HA MySQL and HA RabbitMQ
  4. Add Keystone resource to Pacemaker
  5. Change bind parameters in keystone.conf
  6. Configure OpenStack services to use HA Keystone

Glance image service

  1. Install Guide for basics (http://docs.openstack.org/juno/install-guide/install/apt/content/ch_keystone.html )
  2. Configure Glance for HA MySQL and HA RabbitMQ
  3. Add OpenStack Image API resource to Pacemaker, Configure OpenStack Image Service API, Configure OpenStack services to use HA Image API (Modify: http://docs.openstack.org/high-availability-guide/content/s-keystone.html )
  4. Should Glance use a redundant storage backend such as Swift?

Cinder Block Storage Service

  1. Install Guide for basic installation
  2. The installation guide covers one API/scheduler node and one volume node.
  3. Add API/scheduler redundancy and multiple volume nodes.
  4. Discuss availability zones?
  5. Need to use Ceph as the storage backend to have data redundancy? We should support at least one open source option such as Ceph and perhaps NFS... and simply mention other options such as NetApp and NFS.

Swift Object Storage

  1. Install Guide for basic installation
  2. The installation guide covers basic storage node redundancy, but only deploys one proxy server. Do we want to discuss the process of adding proxy servers and load balancing them? Also, what about adding storage nodes and perhaps discussing regions/zones?

Nova compute service

  1. Install Guide for basic setup
  2. The installation guide covers multiple compute nodes, but only deploys one instance of API and other services. We should discuss the process of deploying multiple instances of the latter.

Heat Orchestration

  1. Install Guide for basic installation
  2. Add API redundancy
  3. How to set up so that VMs on a failed compute node are quickly migrated to other compute nodes

Ceilometer Telemetry and MongoDB

  1. Install Guide for basic installation
  2. Need one MongoDB node for each Controller node

Database Service (Trove)

  1. Install Guide for basics
  2. Need details about how to apply HA


  1. Install Guide for basics (http://docs.openstack.org/juno/install-guide/install/apt/content/ch_sahara.html )
  2. Should link to Sahara docs for discussion of OpenStack HA versus Hadoop HA and how they work together, although the installation instructions at http://docs.openstack.org/developer/sahara/userdoc/installation.guide.html do not currently mention HA


  1. Configure Pacemaker service group to ensure that the VIP is linked to the API services resource
  2. Systemd alternative to OCF scripts for Pacemaker RA
  3. MariaDB with Galera alternative to MySQL
  4. Install and configure HAProxy for API services and MySQL with Galera cluster load balancing
  5. Mention value of redundant hardware load balancers for stateless services such as REST APIs
  6. Describe scaling single node to 3 nodes HA
  7. Ceph?
  8. Murano?

Original for reference

NOTE: This is the original for us to depart from.

I. Introduction to OpenStack High Availability

  1. Stateless vs. Stateful services
  2. Active/Passive
  3. Active/Active

II. HA Using Active/Passive

1. The Pacemaker Cluster Stack

  1. Installing Packages
  2. Setting up Corosync
  3. Starting Corosync
  4. Starting Pacemaker
  5. Setting basic cluster properties

2. Cloud Controller Cluster Stack

  1. Highly available MySQL
  2. Highly available RabbitMQ

3. API Node Cluster Stack

  1. Configure the VIP
  2. Highly available OpenStack Identity
  3. Highly available OpenStack Image API
  4. Highly available Cinder API
  5. Highly available OpenStack Networking Server
  6. Highly available Ceilometer Central Agent
  7. Configure Pacemaker Group

4. Network Controller Cluster Stack

  1. Highly available Neutron L3 Agent
  2. Highly available Neutron DHCP Agent
  3. Highly available Neutron Metadata Agent
  4. Manage network resources

III. HA Using Active/Active

5. Database

  1. MySQL with Galera
  2. Galera Monitoring Scripts
  3. Other ways to provide a Highly Available database

6. RabbitMQ

  1. Install RabbitMQ
  2. Configure RabbitMQ
  3. Configure OpenStack Services to use RabbitMQ

7. HAproxy Nodes 8. OpenStack Controller Nodes

  1. Running OpenStack API & schedulers
  2. Memcached

9. OpenStack Network Nodes

  1. Running Neutron DHCP Agent
  2. Running Neutron L3 Agent
  3. Running Neutron Metadata Agent