Jump to: navigation, search


< Puppet
Revision as of 14:39, 21 October 2013 by David Moreau Simard (talk | contribs) (I want to spawn a cluster and configure it through a puppetmaster as part of a continuous integration effort)


This document is intended to capture requirements for a single puppet-ceph module.

Very much like vswitch, Ceph is not exclusively used in the context of OpenStack. There is however a significant community relying on puppet to deploy Ceph in the context of OpenStack, as is shown in the [#Related_tools_and_implementations|inventory of the existing efforts] Having a puppet ceph module under the umbrella of the stackforge infrastructure helps federate the efforts while providing a workflow that will improve the overall quality of the module.


Almost each component of this module deserve a discussion and it would take a long time to agree on everything before getting something useful. The following list sets the order in which each module is going to be implemented. Each step must be a useable puppet module, unit tested and including integration tests.

User Stories

I want to try this module, heard of ceph, want to see it in action

   /node/ { 
    ceph::conf { auth_enable: false };
    ceph::osd { '/srv/osd1' }; 
    ceph::osd { '/srv/osd2' }; 
  • install puppet,
  • paste this in site.pp and replace /node/ with the name of your current node,
  • puppet apply site.pp ,
  • ceph -s and see that it works

I want to run benchmarks on three new machines

  • There are four machines, 3 OSD, 1 MON and one machine that is the client from which the user runs commands.
  • install puppetmaster and create site.pp with:
   /ceph-default/ {
     ceph::conf { auth_enable: false };
     ceph::conf { 'mon host': 'node1' }; 
   /node1/ inherits ceph-default { 
    ceph::osd { '/dev/sdb' }; 
   /node2/, /node3/ inherits ceph-default { 
    ceph::osd { '/dev/sdb' }; 
    ceph::osd { '/dev/sdc' }; 
  /client/ inherits ceph-default { 
  • ssh client
  • rados bench
  • interpret the results

I want to spawn a cluster configured with a puppetmaster as part of a continuous integration effort

Leveraging vagrant, vagrant-openstack, openstack

  • Ceph is used as a backend storage for various use cases
  • There are tests to make sure the Ceph cluster was instantiated properly
  • There are tests to make sure various other infrastructure components (or products) can use the Ceph cluster


High level requirements

No complex cross host orchestration

All cross host orchestration should be assumed to be managed outside of Puppet. Provided that it's dependencies have already been configured and are known, each component should support being adding without having to run Puppet on more than one node.

For example

  • cinder-volume instances should be configured to join a Ceph cluster simply by running Puppet on that node
  • OSD instances should be configured to join a cluster simply by running puppet agent on a node and targeting that role.

All cross host orchestration should be assumed to be managed outside of Puppet. The Puppet implementation should only be concerned with

  • what components need to be defined (where these are implemented as classes)
  • what data is required for those components (where that data is passed in a class parameters)

Supporting versions

The Operating System versions supported must be tested with integration on the actual operating system. Although it is fairly to add support for an Operating System, it is prone to regressions if not tested. The per Operating System support strategy mimics the way OpenStack modules do it.

The supported versions of the components that deal with the environment in which Ceph is used ( OpenStack, Cloudstack, Ganeti etc. ) are handled by each component on a case by case basis. There probably is too much heterogeneity to set a rule.

Provide sensible defaults

If the high level components ( osd + mon + mds + rgw for instance ) are included without any parameter, the result must be a functional Ceph cluster.

Architectured to leverage Ceph to its full potential

It means talking to the MON when configuring or modifying the cluster, using ceph-disk as a low level tool to create the storage required for an OSD, creating a minimal /etc/ceph/ceph.conf to allow a client to connect to the Ceph cluster. The MON exposes a very rich API ( either via the ceph cli or a REST API ) and it offers a great flexibility to the system administrator. It is unlikely that the first versions of the puppet module captures all of it. But it should be architectured to allow the casual contributor to add a new feature or a new variation without the need to workaround architectural limitations.

The ceph-deploy utility is developed as part of the ceph project, to help people get up to speed as quickly as possible for test and POCs. Alfredo Deza made a compeling argument against using ceph-deploy as a helper for a puppet module. Because it is designed to hide some of the flexibility ceph offers for the sake of simplicity. An inconvenience that is incompatible with the goal of a puppet module designed to accommodate all use cases.

Prefer cli over REST

The ceph cli is preferred because the rest-api requires the installation of an additional daemon.

Module versioning

Create a branch for each Ceph release ( stable/cuttlefish, stable/dumpling etc. ) and follow the same pattern as the OpenStack modules

Support Ceph versions from cuttlefish

Do not support Ceph versions released before cuttlefish

Integration tests

All scenarios can probably be covered with 2 virtual machines, 2 interfaces and one disk attached to one of the machines. A number of scenarios can be based on a single machine, using directories instead of disks and a single interface.

Loic Dachary (talk) : how are the integration test resource provisioned ? Where to look to learn more ?

Puppet user components

This section outlines the roles and well as configuration components that are visible to the puppet user. They must be understandable for the system administrator willing to deploy Ceph for the first time.


  • proposed name: ceph::conf
  • purpose: keeps and writes config and their options for the top level sections of the ceph config. This includes these sections:
    • [global]
    • [mon]
    • [osd]
    • [mds]
  • interface: every key that is needed to write the base config ... the whole list would be to long to written down here
  • auth_enable - true or false, enables/disables cephx, defaults to true
If enable is true, set the following in the [global] section of the conf file:
       auth cluster required = cephx
       auth service required = cephx
       auth client required = cephx
       auth supported = cephx
If enable is false, set the following in the [global] section of the conf file:
       auth cluster required = none
       auth service required = none
       auth client required = none
       auth supported = none

It should support disabling or enabling cephx when the values change. If it does not support updating, it must fail when changed on an existing Ceph cluster.

Using a inifile child provider ( such as cinder_config ) a setting would look like

   ceph_conf {
     'GLOBAL/fsid': value => $fsid;

And create /etc/ceph/ceph.conf such as:

       fsid = 918340183294812038

Improvements to be implemented later:

  • If a key/value pair is modified in the *mon*, *osd* or *mds* sections, all daemons are notified of the change with *ceph {daemon} tell * ...*.


  • proposed name: ceph::osd
  • purpose: configures a ceph OSD using the ceph-disk helper
  • interface:
    • directory/disk - a disk or a directory to be used as a storage for the OSD
    • bootstrap-osd - the bootstrap-osd secret key
    • dmcrypt - options needed to encrypt disks

Here is what should happen on a node with at least one OSD

       key = AQCUg71RYEi7DxAAxlyC1KExxSnNJgim6lmuGA==
    • The user bootstrap-osd with this key with caps to bootstrap an OSD:
   $ ceph auth list
       key: AQCUg71RYEi7DxAAxlyC1KExxSnNJgim6lmuGA==
       caps: [mon] allow profile bootstrap-osd

At boot time the /var/lib/ceph/osd directory is explored to discover all OSDs that need to be started. Operating systems for which the same logic is not implemented will need an additional script run at boot time to perform the same exploration until the default script is updated to add this capability.


  • proposed name: ceph::mds
  • purpose: configures a ceph MDS, setup /etc/ceph/ceph.conf with the MONs IPs, declare the MDS to the cluster via the MON, optionaly set the key to allow the MDS to connect to the MONs
  • interface:
    • monitor_ips - list of ip addresses used to connect to the monitor servers
    • key - the secret key for the id user
    • id - the id of the user


Creates the files and keyring supporting a mon, runs the daemon

  • proposed name: ceph::mon
  • purpose: configures a ceph MON
  • interface:
    • cluster - cluster name ( defaults to ceph )
    • id - the id of the mon
    • ip_address - the ip addresses of the mon
    • key - the mon. user key
  • add a [mon.$id] section to the conf file (depends on ceph::conf to write the base part of the config)
  • if auth == cephx
    • the mon. key is mandatory and need to be set by the user to be a valid ceph key. The documentation should contain an example key and explanations about how to create an auth key.
    • writes the keyring
  • installs the packages
  • ceph-mon --id 0 --mkfs
  • runs the mon daemon


  • proposed name: ceph::rbd
  • purpose: maps and mounts a rbd image, taking care of dependencies (packages, rbd kernel module, /etc/ceph/rbdmap, fstab)
  • interface:
    • name - the name of the image
    • pool - the pool in which the image is
    • mount_point - where the image will be mounted
    • key - the secret key for the id user
    • id - the id of the user
David Moreau Simard (talk) Should ceph::client be a dependency ?


David Moreau Simard (talk) Should ceph::client be a dependency ?

Implementor components

These components are dependencies of the Puppet user components and can be used by other components. They should be a library of components where the code common to at least two independant components ( think OpenStack and Cloudstack ) is included.


Although some distributions include packages for Ceph, it is recommended to install from the packages available from ceph.com http://ceph.com/docs/next/install/. It is not recommended to install the Ceph package provided by the standard repositories. This will change over time and the need to use the repository provided by ceph.com will gradually become less common.

The use of the repositories from ceph.com should be documented in the README.md with a apt {} based example to clarify that the user is expected to get up to date packages repositories.

ceph client implementation

  • proposed name: ceph::client
  • purpose: setup /etc/ceph/ceph.conf to connect to the Ceph cluster and install the ceph cli
  • interface:
    • monitor_ips - list of ip addresses used to connect to the monitor servers
    • client_id - name of the client to find the correct id for key
    • keypath - path to the clients key file


Keyring management, authentication. It would be a class to create keys for new users (e.g. a user that can create RBDs or use the Objectstore) which may special access rights. But would also be used by the other classes like ceph::mon or ceph::osd to place e.g. the shared 'client.admin' or 'mon.' keys. I would handle here all key related tasks.

  • proposed name: ceph::key
  • purpose: handles ceph keys (cephx), generates keys, creates keyring files, incject keys into or delete keys from the cluster/keyring via ceph and ceph-authtool tools.
  • interface:
    • secret - key secret
    • keyring_path - path to the keyring
    • cap_mon/cap_osd/cap_mds - cephx capabilities
    • user/group/mode: settings for the keyring file if needed
    • inject - options to inject a key into the cluster


  • proposed name: ceph::pool
  • purpose: manage operations on the pools in the cluster such as: create/delete pools, set PG/PGP number
  • interface:
    • pool_name - name of the pool
    • create - if to create a new pool
    • delete - if to delete an existing pool
    • pg_num - number of Placement Groups (PGs) for a pool, if the pool already exists this may increase the number of PGs if the current value is lower
    • pgp_num - same as for pg_num
    • replica_level - increase or decrease the replica level of a pool

OpenStack components

ceph specific configuration for cinder/glance (already provided by the puppet-cinder and puppet-glance modules in the volume/rdb and backend/rdb classes)

RadosGW components

The RadosGW is developped as an integral part of Ceph. It is however not required to deploy a cluster and should be treated as any client application of the cluster.


  • proposed name: ceph::rgw
  • purpose: configures a ceph radosgw , setup /etc/ceph/ceph.conf with the MONs IPs, optionaly set the key to allow the OSD to connect to the MONs
  • interface:
    • monitor_ips - list of ip addresses used to connect to the monitor servers
    • key - the secret key for the id user
    • id - the id of the user
    • rgw_data - the path where the radosgw data should be stored
    • fcgi_file - path to the fcgi file e.g. /var/www/s3gw.fcgi
Danny Al-Gaaf (talk) the monitor_ips are not needed: IMO ceph::conf should provide these information to all other


  • proposed name: ceph::rgw_user
  • purpose: create/remove users and Swift users for the RadosGW S3/Swift API
  • interface:
    • user - username
    • key - secret key (could get generated if needed)
    • swift_user - username for the Swift API user
    • swift_key - secret key for the Swift API user

Related tools and implementations

  • deploy ceph : ceph-deploy
 for test / POC purposes
 maintainer: Alfredo Deza
  • deploy ceph with puppet : puppet-cephdeploy
 relies on ceph-deploy
 maintainer: Don Talton
  • deploy ceph with puppet : puppet-ceph
 developped in 2012 but still useful, upstream
 maintainer: community

 fork of puppet-ceph, updated recently
 maintainer: Deutsche Telekom AG (DTAG)
  • ceph + openstack : ceph docs
 manual integration
  maintainer: John Wilkins + Josh Durgin
  • ceph + openstack with puppet : stackforge
 maintainer: community
  • ceph + openstack with puppet : COI
 targeting Cisco use case
 maintainer : Don Talton + Robert Starmer
  • ceph + openstack with puppet : mirantis
 in the context of Fuel
 maintainer : Andrew Woodward
  • openstack with puppet : openstack-installer
 data driven approach to deploy OpenStack
 maintainer: Robert Starmer + Dan Bode