Puppet/ceph-blueprint

Overview

This document is intended to capture requirements for a single puppet-ceph module that is going to be written and contributed as a part of the stackforge modules.

Requirements

High level requirements

No complex cross host orchestration

All cross host orchestration should be assumed to be managed outside of Puppet. Provided that it's dependencies have already been configured and are known, each component should support being adding without having to run Puppet on more than one node.

For example

cinder-volume instances should be configured to join a Ceph cluster simply by running Puppet on that node
OSD instances should be configured to join a cluster simply by running puppet agent on a node and targeting that role.

All cross host orchestration should be assumed to be managed outside of Puppet. The Puppet implementation should only be concerned with

what components need to be defined (where these are implemented as classes)
what data is required for those components (where that data is passed in a class parameters)

Architecture to leverage Ceph to its full potential

It means talking to the MON when configuring or modifying the cluster, using ceph-disk as a low level tool to create the storage required for an OSD, creating a minimal /etc/ceph/ceph.conf to allow a client to connect to the Ceph cluster. The MON exposes a very rich API ( either via the ceph cli or a REST API ) and it offers a great flexibility to the system administrator. It is unlikely that the first versions of the puppet module captures all of it. But it should be architectured to allow the casual contributor to add a new feature or a new variation without the need to workaround architectural limitations.

The ceph-deploy utility is developed as part of the ceph project, to help people get up to speed as quickly as possible for test and POCs. Alfredo Deza made a compeling argument against using ceph-deploy as a helper for a puppet module. Because it is designed to hide some of the flexibility ceph offers for the sake of simplicity. An inconvenience that is incompatible with the goal of a puppet module designed to accommodate all use cases.

Components

This section outlines the roles and well as configuration components that are needed:

backend components

ceph specific configuration for cinder/glance (already provided by the puppet-cinder and puppet-glance modules in the volume/rdb and backend/rdb classes)

ceph client implementation

proposed name: ceph::client
purpose: configures the openstack roles as clients by configuring /etc/ceph/ceph.conf
interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers

osd

proposed name: ceph::osd
purpose: configures a ceph OSD using the ceph-disk helper, setup /etc/ceph/ceph.conf with the MONs IPs, declare the OSD to the cluster via the MON, optionaly set the key to allow the OSD to connect to the MONs
interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- directory/disk - a disk or a directory to be used as a storage for the OSD
- key - the secret key for the id user
- id - the id of the user

rgw

proposed name: ceph::rgw
purpose: configures a ceph radosgw , setup /etc/ceph/ceph.conf with the MONs IPs, optionaly set the key to allow the OSD to connect to the MONs
interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- key - the secret key for the id user
- id - the id of the user

mds

proposed name: ceph::mds
purpose: configures a ceph MDS, setup /etc/ceph/ceph.conf with the MONs IPs, declare the MDS to the cluster via the MON, optionaly set the key to allow the MDS to connect to the MONs
interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- key - the secret key for the id user
- id - the id of the user

mon

proposed name: ceph::mon
purpose: configures a ceph MON setup /etc/ceph/ceph.conf with the MONs IPs, optionaly set the key to allow the MON to connect to the other MONs or initialize the MON if it is the first in the cluster
interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- directory/disk - a disk or a directory to be used as a storage for the MON
- key - the secret key for the id user
- id - the id of the user

Related tools and implementations

deploy ceph : ceph-deploy

 for test / POC purposes
 https://github.com/ceph/ceph-deploy

 maintainer: Alfredo Deza

deploy ceph with puppet : puppet-cephdeploy

 relies on ceph-deploy
 https://github.com/dontalton/puppet-cephdeploy/

 maintainer: Don Talton

deploy ceph with puppet : puppet-ceph

 developped in 2012 but still useful, upstream
 https://github.com/enovance/puppet-ceph
 
 maintainer: community
 --
 fork of puppet-ceph, updated recently
 https://github.com/TelekomCloud/puppet-ceph/tree/rc/eisbrecher
 
 maintainer: Deutsche Telekom AG (DTAG)

ceph + openstack : ceph docs

 manual integration
 http://ceph.com/docs/next/rbd/rbd-openstack/
  maintainer: John Wilkins + Josh Durgin

ceph + openstack with puppet : stackforge

 https://github.com/stackforge/puppet-glance/blob/stable/grizzly/manifests/backend/rbd.pp
 https://github.com/stackforge/puppet-cinder/blob/stable/grizzly/manifests/volume/rbd.pp

 maintainer: community

ceph + openstack with puppet : COI

 targeting Cisco use case
 https://github.com/CiscoSystems/puppet-coe/tree/grizzly/manifests/ceph
 http://docwiki.cisco.com/wiki/OpenStack:Ceph-COI-Installation

 maintainer : Don Talton + Robert Starmer

ceph + openstack with puppet : mirantis

 in the context of Fuel
 https://github.com/Mirantis/fuel/tree/master/deployment/puppet/ceph
 https://github.com/Mirantis/fuel/blob/master/deployment/puppet/cinder/manifests/volume/ceph.pp
 https://github.com/Mirantis/fuel/blob/master/deployment/puppet/glance/manifests/backend/ceph.pp

 maintainer : Andrew Woodward

openstack with puppet : openstack-installer

 data driven approach to deploy OpenStack
 https://github.com/CiscoSystems/openstack-installer/

 maintainer: Robert Starmer + Dan Bode