Jump to: navigation, search

Difference between revisions of "HAGuideImprovements/TOC"

Line 22: Line 22:
 
:# Minimal Architecture Example -- Network Layout, styled as in http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites for easy comparison
 
:# Minimal Architecture Example -- Network Layout, styled as in http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites for easy comparison
  
==== Prerequisites ====
+
==== Hardware setup ====
:# Link to Install Guide: Install O/S on each node
+
:# Install O/S on each node (link to Install Guide, e.g http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html
 +
 
 +
==== Basic HA facilities ====
 
:# Install pacemaker, crmsh, corosync, cluster-glue, fence-agents (Fedora only), resource-agents. (Modify: http://docs.openstack.org/high-availability-guide/content/_install_packages.html)
 
:# Install pacemaker, crmsh, corosync, cluster-glue, fence-agents (Fedora only), resource-agents. (Modify: http://docs.openstack.org/high-availability-guide/content/_install_packages.html)
 
:# What is needed for LSB/upstart/systemd alternative to OCF scripts (RA) for Pacemaker?  See https://bugs.launchpad.net/openstack-manuals/+bug/1349398
 
:# What is needed for LSB/upstart/systemd alternative to OCF scripts (RA) for Pacemaker?  See https://bugs.launchpad.net/openstack-manuals/+bug/1349398

Revision as of 23:07, 20 March 2015

(Moved original to the bottom of the page for reference.)

Proposed Revision

Strategy and assumptions:

  1. Audience is people who have some experience installing OpenStack, not first time users
  2. Focus on installation of OpenStack core services
  3. Structure the guide sequentially -- the steps to take in a reasonable order
  4. Avoid redundancy with the Install Guide; for steps that are identical for HA and non-HA installations, link to appropriate sections in the Install Guide
  5. One guide for all Linux distros/platforms
  6. Emphasize a reasonable, standard deployment based on open source components. We can provide some notes about alternatives as appropriate (for example, using a commercial load-balancer might be a better alternative than relying on HAProxy) and perhaps a link to the OpenStack Marketplace.

Structure/Outline

HA Intro and Concepts

  1. Redundancy and failover
  2. Stateless/stateful, active/passive, active/active (Keep: http://docs.openstack.org/high-availability-guide/content/stateless-vs-stateful.html)
  3. Quorums; many services should use an odd number of nodes equal to or greater than 3
  4. Single-controller HA mode and scaling up to 3 or more

Hardware setup

  1. Minimal Architecture Example -- Network Layout, styled as in http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites for easy comparison

Hardware setup

  1. Install O/S on each node (link to Install Guide, e.g http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html

Basic HA facilities

  1. Install pacemaker, crmsh, corosync, cluster-glue, fence-agents (Fedora only), resource-agents. (Modify: http://docs.openstack.org/high-availability-guide/content/_install_packages.html)
  2. What is needed for LSB/upstart/systemd alternative to OCF scripts (RA) for Pacemaker? See https://bugs.launchpad.net/openstack-manuals/+bug/1349398
  3. Set up and start Corosync and Pacemaker. Stick with 'crm' tool for Ubuntu/Debian and 'pcs' for RHEL/Fedora (Modify http://docs.openstack.org/high-availability-guide/content/_set_up_corosync.html; Modify: http://docs.openstack.org/high-availability-guide/content/_start_pacemaker.html)
  4. Set basic cluster properties (Modify: http://docs.openstack.org/high-availability-guide/content/_set_basic_cluster_properties.html))
  5. Configure fencing for Pacemaker cluster (Links to http://clusterlabs.org/doc/)
  6. Configure the VIP (Keep: http://docs.openstack.org/high-availability-guide/content/s-api-vip.html )
  7. API services -- Do those belong here or in specific sections? (Modify Glance API: http://docs.openstack.org/high-availability-guide/content/s-glance-api.html and Modify Cinder API: http://docs.openstack.org/high-availability-guide/content/s-cinder-api.html )
  8. Schedulers
  9. Memcached service on Controllers (Keep: http://docs.openstack.org/high-availability-guide/content/_memcached.html , which links to http://code.google.com/p/memcached/wiki/NewStart for specifics)

Configure networking on each node

  1. Rather than configuring neutron here, we should simply mention physical network HA methods (e.g., bonding) and additional node/network requirements for L3HA and DVR for planning purposes. As this point the, the networking guide likely won't cover the former.
  2. Link to Networking Guide
  3. (Neutron agents should be described for active/active; deprecate single agent's instances case)
  4. For Kilo and beyond, focus on L3HA and DVR

Install and Configure MySQL

  1. Two nodes plus GARBD.
  2. MySQL with Galera
  3. Pacemaker multistate clone resource for Galera cluster
  4. Pacemaker resource agent for Galera cluster management
  5. Deprecate MySQL DRBD configuration because of split-brain issues

RabbitMQ Message broker

  1. Install and configure message broker on Controller; see http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-prerequisites
  2. Oslo messaging for active/active
  1. I think services need some special configuration with more than two nodes?
  1. No need for active/passive AMQP; Two-node active/active cluster with mirrored queues instead
  2. Pacemaker multistate clone resource for RabbitMQ cluster
  3. Pacemaker resource agent for RabbitMQ cluster management
  4. Deprecate DRBD for RabbitMQ

Memcached

Does this go here or in "Prerequisites" section above?

  1. I think Oslo supports hash synchronization so this shouldn't take more than load balancing.

NTP

  1. Run NTP servers on every controller and configure other nodes to use all of them for synchronization. Link to http://docs.openstack.org/juno/install-guide/install/apt/content/ch_basic_environment.html#basics-ntp

Keystone Identity services

  1. Install Guide for concepts: http://docs.openstack.org/juno/install-guide/install/apt/content/keystone-concepts.html
  2. Install Guide to configure prerequisites, install and configure the components, and finalize the installation: http://docs.openstack.org/juno/install-guide/install/apt/content/keystone-install.html
  3. Configure Keystone for HA MySQL and HA RabbitMQ
  4. Add Keystone resource to Pacemaker
  5. Change bind parameters in keystone.conf
  6. Configure OpenStack services to use HA Keystone

Glance image service

  1. Install Guide for basics (http://docs.openstack.org/juno/install-guide/install/apt/content/ch_keystone.html )
  2. Configure Glance for HA MySQL and HA RabbitMQ
  3. Add OpenStack Image API resource to Pacemaker, Configure OpenStack Image Service API, Configure OpenStack services to use HA Image API (Modify: http://docs.openstack.org/high-availability-guide/content/s-keystone.html )
  4. Configure OpenStack Image Service API (http://docs.openstack.org/high-availability-guide/content/_configure_openstack_image_service_api.html)
  5. Configure OpenStack services to use HA Image API (http://docs.openstack.org/high-availability-guide/content/_configure_openstack_services_to_use_high_available_openstack_image_api.html)
  6. Should Glance use a redundant storage backend such as Swift or Ceph?

Cinder Block Storage Service

This section discusses how to configure the Cinder control plane only. The Cinder dataplane is addressed in the "Storage Backend" section.

  1. Install Guide for basic installation
  2. The installation guide covers one API/scheduler node and one volume node.
  3. Install on the controller node
  4. Download resource agent and add Block Storage API to Pacemaker (Keep: http://docs.openstack.org/high-availability-guide/content/_add_block_storage_api_resource_to_pacemaker.html)
  5. Configure Block Storage API service (Expand (it's a bit terse): http://docs.openstack.org/high-availability-guide/content/_add_block_storage_api_resource_to_pacemaker.html)
  6. Configure OpenStack services to use HA Block Storage API (http://docs.openstack.org/high-availability-guide/content/_configure_openstack_services_to_use_highly_available_block_storage_api.html)
  7. Add API/scheduler redundancy and multiple volume nodes.
  8. Discuss availability zones?
  9. Requires at least one additional storage node that provides persistent storage volumes for instances.
  10. Need to use Ceph as the storage backend to have data redundancy? We should support at least one open source option such as Ceph and perhaps NFS... and simply mention other options such as NetApp and NFS.

Swift Object Storage

  1. Install Guide for basic installation
  2. The installation guide covers basic storage node redundancy, but only deploys one proxy server. Do we want to discuss the process of adding proxy servers and load balancing them? Also, what about adding storage nodes and perhaps discussing regions/zones?

Storage Backend

This section contains more concepts than actual procedures; our expectation is that the specific technologies discussed have their own configuration documenation that can be referenced.

This section describes the data plane (infrastructure) elements that factor into the overall HA capabilities of the storage; in other words, how does one ensure that ones data is not lost when systems fail. Topics to be discussed include RAID, Erasure Coding, etc. and describe the protections they do and do not offer.

We will also in a blurb of the options that are available. Finally, we could state that cinder supports multiple storage providers (Ceph, EMC, NetApp, SolidFire, etc.) and you can also get additional details from your storage providers documentation.

Swift combines control and data plane so we would cover some aspects of both.

Nova compute service

  1. Install Guide for basic setup
  2. The installation guide covers multiple compute nodes, but only deploys one instance of API and other services. We should discuss the process of deploying multiple instances of the latter.

Heat Orchestration

  1. Install Guide for basic installation
  2. Add API redundancy
  3. How to set up so that VMs on a failed compute node are quickly migrated to other compute nodes

Ceilometer Telemetry and MongoDB

  1. Install Guide for basic installation
  2. Need one MongoDB node for each Controller node

Database Service (Trove)

  1. Install Guide for basics
  2. Need details about how to apply HA

Sahara

  1. Install Guide for basics (http://docs.openstack.org/juno/install-guide/install/apt/content/ch_sahara.html )
  2. Should link to Sahara docs for discussion of OpenStack HA versus Hadoop HA and how they work together, although the installation instructions at http://docs.openstack.org/developer/sahara/userdoc/installation.guide.html do not currently mention HA

Other

  1. Configure Pacemaker service group to ensure that the VIP is linked to the API services resource
  2. Systemd alternative to OCF scripts for Pacemaker RA
  3. MariaDB with Galera alternative to MySQL
  4. Install and configure HAProxy for API services and MySQL with Galera cluster load balancing
  5. Mention value of redundant hardware load balancers for stateless services such as REST APIs
  6. Describe scaling single node to 3 nodes HA
  7. Ceph?
  8. Murano?

Original for reference

NOTE: This is the original for us to depart from.

I. Introduction to OpenStack High Availability

  1. Stateless vs. Stateful services
  2. Active/Passive
  3. Active/Active

II. HA Using Active/Passive

1. The Pacemaker Cluster Stack

  1. Installing Packages
  2. Setting up Corosync
  3. Starting Corosync
  4. Starting Pacemaker
  5. Setting basic cluster properties

2. Cloud Controller Cluster Stack

  1. Highly available MySQL
  2. Highly available RabbitMQ

3. API Node Cluster Stack

  1. Configure the VIP
  2. Highly available OpenStack Identity
  3. Highly available OpenStack Image API
  4. Highly available Cinder API
  5. Highly available OpenStack Networking Server
  6. Highly available Ceilometer Central Agent
  7. Configure Pacemaker Group

4. Network Controller Cluster Stack

  1. Highly available Neutron L3 Agent
  2. Highly available Neutron DHCP Agent
  3. Highly available Neutron Metadata Agent
  4. Manage network resources

III. HA Using Active/Active

5. Database

  1. MySQL with Galera
  2. Galera Monitoring Scripts
  3. Other ways to provide a Highly Available database

6. RabbitMQ

  1. Install RabbitMQ
  2. Configure RabbitMQ
  3. Configure OpenStack Services to use RabbitMQ

7. HAproxy Nodes 8. OpenStack Controller Nodes

  1. Running OpenStack API & schedulers
  2. Memcached

9. OpenStack Network Nodes

  1. Running Neutron DHCP Agent
  2. Running Neutron L3 Agent
  3. Running Neutron Metadata Agent