Difference between revisions of "Puppet/ceph-blueprint"
(→package management) |
(package management) |
||
Line 117: | Line 117: | ||
=== package management === | === package management === | ||
+ | |||
+ | Although some distributions include packages for Ceph, it is recommended to install from the packages available from ceph.com http://ceph.com/docs/next/install/. It is not recommended to install the Ceph package provided by the standard repositories. This will change over time and the need to use the repository provided by ceph.com will gradually become less common. | ||
+ | |||
+ | * Mimic https://github.com/stackforge/puppet-nova/blob/master/manifests/params.pp#L8 for cross distribution package name | ||
* '''proposed name''': ceph::package | * '''proposed name''': ceph::package | ||
Line 124: | Line 128: | ||
::[[User:Danny Al-Gaaf|Danny Al-Gaaf]] ([[User talk:Danny Al-Gaaf|talk]]) This would include supporting classes like for example apt to setup the correct sources/repositories. | ::[[User:Danny Al-Gaaf|Danny Al-Gaaf]] ([[User talk:Danny Al-Gaaf|talk]]) This would include supporting classes like for example apt to setup the correct sources/repositories. | ||
+ | ::[[User:Dachary|Loic Dachary]] ([[User talk:Loic Dachary|talk]]) I would rather handle target ceph release using branches, the same way openstack does with stable/grizzly etc. | ||
=== ceph client implementation === | === ceph client implementation === |
Revision as of 14:11, 20 October 2013
Contents
Overview
This document is intended to capture requirements for a single puppet-ceph module that is going to be written and contributed as a part of the stackforge modules.
Very much like vswitch, Ceph is not exclusively used in the context of OpenStack. There is however a significant community relying on puppet to deploy Ceph in the context of OpenStack, as is shown in the inventory of the existing efforts in this blueprint. Having a puppet ceph module under the umbrela of the stackforge infrastructure would help federate the efforst while providing a workflow that will improve the overall quality of the module.
Requirements
High level requirements
No complex cross host orchestration
All cross host orchestration should be assumed to be managed outside of Puppet. Provided that it's dependencies have already been configured and are known, each component should support being adding without having to run Puppet on more than one node.
For example
- cinder-volume instances should be configured to join a Ceph cluster simply by running Puppet on that node
- OSD instances should be configured to join a cluster simply by running puppet agent on a node and targeting that role.
All cross host orchestration should be assumed to be managed outside of Puppet. The Puppet implementation should only be concerned with
- what components need to be defined (where these are implemented as classes)
- what data is required for those components (where that data is passed in a class parameters)
Supporting versions of everything
It is worth capturing the supported versions of openstack, ceph, and what distros/versions we will be targeting with this work.
Provide sensible defaults
If the high level components ( osd + mon + mds + rgw for instance ) are included without any parameter, the result must be a functional Ceph cluster.
Architectured to leverage Ceph to its full potential
It means talking to the MON when configuring or modifying the cluster, using ceph-disk as a low level tool to create the storage required for an OSD, creating a minimal /etc/ceph/ceph.conf to allow a client to connect to the Ceph cluster. The MON exposes a very rich API ( either via the ceph cli or a REST API ) and it offers a great flexibility to the system administrator. It is unlikely that the first versions of the puppet module captures all of it. But it should be architectured to allow the casual contributor to add a new feature or a new variation without the need to workaround architectural limitations.
The ceph-deploy utility is developed as part of the ceph project, to help people get up to speed as quickly as possible for test and POCs. Alfredo Deza made a compeling argument against using ceph-deploy as a helper for a puppet module. Because it is designed to hide some of the flexibility ceph offers for the sake of simplicity. An inconvenience that is incompatible with the goal of a puppet module designed to accommodate all use cases.
Puppet user components
This section outlines the roles and well as configuration components that are visible to the puppet user. They must be understandable for the system administrator willing to deploy Ceph for the first time.
conf
- proposed name: ceph::conf
- purpose: keeps and writes config and their options for the top level sections of the ceph config. This includes these sections:
- [global]
- [mon]
- [osd]
- [mds]
- interface: every key that is needed to write the base config ... the whole list would be to long to written down here
- Loic Dachary (talk) use inject args in addition to config to update running daemons dynamically
osd
- proposed name: ceph::osd
- purpose: configures a ceph OSD using the ceph-disk helper, setup /etc/ceph/ceph.conf with the MONs IPs, declare the OSD to the cluster via the MON, optionally set the key to allow the OSD to connect to the MONs
- interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- directory/disk - a disk or a directory to be used as a storage for the OSD
- key - the secret key for the id user
- id - the id of the user
- dmcrypt - options needed to encrypt disks
- Danny Al-Gaaf (talk) Not sure if we need an option for the osd keys. This is very well handled by the ceph tools itself since each OSD gets an own key by default.
mds
- proposed name: ceph::mds
- purpose: configures a ceph MDS, setup /etc/ceph/ceph.conf with the MONs IPs, declare the MDS to the cluster via the MON, optionaly set the key to allow the MDS to connect to the MONs
- interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- key - the secret key for the id user
- id - the id of the user
mon
- proposed name: ceph::mon
- purpose: configures a ceph MON setup /etc/ceph/ceph.conf with the MONs IPs, optionaly set the key to allow the MON to connect to the other MONs or initialize the MON if it is the first in the cluster
- interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- directory/disk - a disk or a directory to be used as a storage for the MON
- key - the secret key for the id user
- id - the id of the user
- Danny Al-Gaaf (talk) the ID of the user wouldn't be needed since the MONs have all the same ID for cephx ('mon.')
- Danny Al-Gaaf (talk) the directory/disk information could be obtained via ceph::conf
rbd
- proposed name: ceph::rbd
- purpose: maps and mounts a rbd image, taking care of dependencies (packages, rbd kernel module, /etc/ceph/rbdmap, fstab)
- interface:
- name - the name of the image
- pool - the pool in which the image is
- mount_point - where the image will be mounted
- key - the secret key for the id user
- id - the id of the user
- David Moreau Simard (talk) Should ceph::client be a dependency ?
cephfs
- proposed name: ceph::cephfs
- purpose: mounts a cephfs filesystem, taking care of dependencies (e.g, fstab, packages)
- interface:
- Lots - See http://ceph.com/docs/next/man/8/mount.ceph/
- David Moreau Simard (talk) Should ceph::client be a dependency ?
Implementor components
These components are dependencies of the Puppet user components and can be used by other components. They should be a library of components where the code common to at least two independant components ( think OpenStack and Cloudstack ) is included.
rest-api
run the rest-api daemon
- Loic Dachary (talk) on the same host as the first mon by default ?
- Danny Al-Gaaf (talk) I wouldn't try to define a 'first monitor' this would lead only to trouble in the setup (e.g. dependencies). ATM it's possible to startup all MONs in parallel.
package management
Although some distributions include packages for Ceph, it is recommended to install from the packages available from ceph.com http://ceph.com/docs/next/install/. It is not recommended to install the Ceph package provided by the standard repositories. This will change over time and the need to use the repository provided by ceph.com will gradually become less common.
- Mimic https://github.com/stackforge/puppet-nova/blob/master/manifests/params.pp#L8 for cross distribution package name
- proposed name: ceph::package
- purpose: configures the the necessary repositories/sources, installs required common packages
- interface:
- release - target ceph release (cuttlefish, dumpling, etc)
- Danny Al-Gaaf (talk) This would include supporting classes like for example apt to setup the correct sources/repositories.
- Loic Dachary (talk) I would rather handle target ceph release using branches, the same way openstack does with stable/grizzly etc.
ceph client implementation
- proposed name: ceph::client
- purpose: configures the openstack roles as clients by configuring /etc/ceph/ceph.conf
- interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- client_id - name of the client to find the correct id for key
- keypath - path to the clients key file
key
keyring management, authentication
- proposed name: ceph::key
- purpose: handles ceph keys (cephx), generates keys, creates keyring files, incject keys into or delete keys from the cluster/keyring via ceph and ceph-authtool tools.
- interface:
- secret - key secret
- keyring_path - path to the keyring
- cap_mon/cap_osd/cap_mds - cephx capabilities
- user/group/mode: settings for the keyring file if needed
- inject - options to inject a key into the cluster
- Loic Dachary (talk) idealy the puppet user would only need to activate authentication ( i.e. ceph::conf auth_supported = cephx ) and all key management will happen in the backend. Is it realistic ?
- Danny Al-Gaaf (talk) not sure what you mean exactly. It would be a class to create keys for new users (e.g. a user that can create RBDs or use the Objectstore) with may special access rights. But would also be used by the other classes like ceph::mon or ceph::osd to place e.g. the shared 'client.admin' or 'mon.' keys. I would handle here all key related tasks.
pool
- proposed name: ceph::pool
- purpose: manage operations on the pools in the cluster such as: create/delete pools, set PG/PGP number
- interface:
- pool_name - name of the pool
- create - if to create a new pool
- delete - if to delete an existing pool
- pg_num - number of Placement Groups (PGs) for a pool, if the pool already exists this may increase the number of PGs if the current value is lower
- pgp_num - same as for pg_num
- replica_level - increase or decrease the replica level of a pool
image
- Loic Dachary (talk) what is a ceph image ? I'm not familiar with the concept.
- Danny Al-Gaaf (talk) I'm confused by that one too, for me an image is and RBD and this should be handled by the RBD class. If this about filling the RBD with data, we should have some kind of client class for that like for example ceph::client::image
- proposed name: ceph::image
- purpose: manage operations on the images of a specified pool in the cluster such as: create/delete images
- interface:
- image_name - name of the image
- pool_name - pool in which the image operation is to take place
- create - if to create a new image
- delete - if to delete an existing image
- size - size of the image
OpenStack components
ceph specific configuration for cinder/glance (already provided by the puppet-cinder and puppet-glance modules in the volume/rdb and backend/rdb classes)
RadosGW components
The RadosGW is developped as an integral part of Ceph. It is however not required to deploy a cluster and should be treated as any client application of the cluster.
rgw
- proposed name: ceph::rgw
- purpose: configures a ceph radosgw , setup /etc/ceph/ceph.conf with the MONs IPs, optionaly set the key to allow the OSD to connect to the MONs
- interface:
- monitor_ips - list of ip addresses used to connect to the monitor servers
- key - the secret key for the id user
- id - the id of the user
- rgw_data - the path where the radosgw data should be stored
- fcgi_file - path to the fcgi file e.g. /var/www/s3gw.fcgi
- Danny Al-Gaaf (talk) the monitor_ips are not needed: IMO ceph::conf should provide these information to all other
rgw_user
- proposed name: ceph::rgw_user
- purpose: create/remove users and Swift users for the RadosGW S3/Swift API
- interface:
- user - username
- key - secret key (could get generated if needed)
- swift_user - username for the Swift API user
- swift_key - secret key for the Swift API user
Related tools and implementations
- deploy ceph : ceph-deploy
for test / POC purposes https://github.com/ceph/ceph-deploy
maintainer: Alfredo Deza
- deploy ceph with puppet : puppet-cephdeploy
relies on ceph-deploy https://github.com/dontalton/puppet-cephdeploy/
maintainer: Don Talton
- deploy ceph with puppet : puppet-ceph
developped in 2012 but still useful, upstream https://github.com/enovance/puppet-ceph
maintainer: community
fork of puppet-ceph, updated recently https://github.com/TelekomCloud/puppet-ceph/tree/rc/eisbrecher
maintainer: Deutsche Telekom AG (DTAG)
- ceph + openstack : ceph docs
manual integration http://ceph.com/docs/next/rbd/rbd-openstack/ maintainer: John Wilkins + Josh Durgin
- ceph + openstack with puppet : stackforge
https://github.com/stackforge/puppet-glance/blob/stable/grizzly/manifests/backend/rbd.pp https://github.com/stackforge/puppet-cinder/blob/stable/grizzly/manifests/volume/rbd.pp
maintainer: community
- ceph + openstack with puppet : COI
targeting Cisco use case https://github.com/CiscoSystems/puppet-coe/tree/grizzly/manifests/ceph http://docwiki.cisco.com/wiki/OpenStack:Ceph-COI-Installation
maintainer : Don Talton + Robert Starmer
- ceph + openstack with puppet : mirantis
in the context of Fuel https://github.com/Mirantis/fuel/tree/master/deployment/puppet/ceph https://github.com/Mirantis/fuel/blob/master/deployment/puppet/cinder/manifests/volume/ceph.pp https://github.com/Mirantis/fuel/blob/master/deployment/puppet/glance/manifests/backend/ceph.pp
maintainer : Andrew Woodward
- openstack with puppet : openstack-installer
data driven approach to deploy OpenStack https://github.com/CiscoSystems/openstack-installer/
maintainer: Robert Starmer + Dan Bode