StarlingX Rook Ceph App installation guide

Rook Ceph a orchestrator providing a containerized solution for Ceph Storage with a specialized Kubernetes Operator to automate the management of the cluster. It is an alternative solution to the bare metal Ceph Storage.

Source

Tarball Package

Get the .tgz file located at

   /usr/local/share/applications/helm/

Or build it yourself using the Debian Build environment

   build-pkgs -c -p rook-ceph-helm,rook-ceph-provisioner-helm,python3-k8sapp-rook-ceph,rook-ceph-floating-monitor-helm,stx-rook-ceph-helm

Deployment Models

The deployment model is the topology strategy that defines the storage backend capabilities of the deployment. The deployment model dictates how the storage solution will look like, defining rules for the placement of the storage cluster elements.

Available deployment models

Each deployment model works with different deployment strategies and rules to fit different needs. Choose one of the following models according to the demands of your cluster:

Controller Model (default)

The OSDs must be added only on hosts with controller personality.
The replication factor can be configured up to a size of 3.

Dedicated Model

The OSDs must be added only on hosts with worker personality;
The replication factor can be configured up to a size of 3.

Open Model

The OSD placement does not have any limitation;
The replication factor does not have any limitation.

 IMPORTANT: The Open deployment model allows more flexible configurations. Users should ensure they understand the implications of their settings, as responsibility for correct configuration rests with them.

Changing the deployment model

The deployment models can be changed as long as the system follows the previously estabilished rules

To change to another deployment model, execute the following command

   ~(keystone_admin)$ system storage-backend-modify ceph-rook-store -d <desired_deployment_model>

Replication factor

The replication factor is the number of copies that each piece of data has, spread across the cluster to provide redundancy.

You can change the replication of an existing Rook Ceph storage backend with the following command

   ~(keystone_admin)$ system storage-backend-modify ceph-rook-store replication=<size>

Possible Replication Factors on Deployment Models for platforms.

AIO-Simplex Controller Model

Default: 1
Max: 3

AIO-Simplex Open Model

Default: 1
Max: ANY

AIO-Duplex Controller Model

Default: 2
Max: 2

AIO-Duplex Open Model

Default: 2
Max: ANY

AIO-Duplex+ or Standard Controller Model

Default: 2
Max: 3

AIO-Duplex+ or Standard Dedicated Mode:

Default: 2
Max: 3

AIO-Duplex+ or Standard Open Model

Default: 2
Max: ANY

Minimum Replication Factor

The minimum replication factor is the least number of copies that each piece of data has, spread across the cluster to provide redundancy.

You can assign any number smaller than the replication factor to this parameter. The default value is (replication - 1).

   Note: When the replication factor changes, the minimum replication will be readjusted automatically.

You can change the minimum replication of an existing Rook Ceph storage backend with the command

   ~(keystone_admin)$ system storage-backend-modify ceph-rook-store min_replication=<size>

Monitor, host-fs and controllerfs

Ceph monitors are the central nervous system of the Ceph cluster, ensuring that all components are aware of each other and that data is stored and accessed reliably. To properly set the environment for Rook Ceph monitors, some filesystems are needed: host-fs for fixed monitor and controllerfs for the floating monitor.

   NOTE: All changes in "host-fs" and "controllerfs" need a reapply on the application to properly propagate 
   the modifications in the Rook ceph cluster.

Functions

The functions parameter contains the Ceph cluster function of a given host. A "host-fs" can have monitor and OSD functions, a "controllerfs" can only have the monitor function.

To modify the function of a "host-fs" the complete list of functions desired must be informed.

Examples

host-fs

   #(only monitor)
   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=monitor

   #(only osd)
   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd

   #(no function)
   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=

   #(both functions)
   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd,monitor

controllerfs

   #(only monitor)
   ~(keystone_admin)$ system controllerfs-modify ceph-float --functions=monitor

   #(no function)
   ~(keystone_admin)$ system controllerfs-modify ceph-float --functions=

Monitor Count

Monitors (mons) are allocated on all the hosts that have a "host-fs ceph" with the monitor capability on it.

Possible Monitor Count on Deployment Models for Platforms

AIO-Simplex

Min: 1
Max: 1

AIO-Duplex

Min: 1
Recommended: 3 (using floating monitor)
Max: 3 (using floating monitor)

AIO-Duplex+ or Standard

Min: 1
Recommended: 3
Max: 5

Fixed Monitors

Fixed monitor is the normal monitor that is associated with a given host. Each fixed monitor requires a "host-fs ceph" properly set and configured on the host.

Adding a monitor

To add a monitor to a Rook Ceph cluster, the host-fs ceph must be created or have the function 'monitor' added to its capabilities

When the host has no OSD registered on the platform, you should add a "host-fs ceph" in every node intended to house a monitor. Creating a host-fs this way automatically sets the monitor function. To create a "host-fs ceph", run the following command

   ~(keystone_admin)$ system host-fs-add <hostname> ceph=<size>

When there are OSDs registered on a host you should add the monitor function to the existing "host-fs"

   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd,monitor

After the adding the 'monitor' function, the application must be reapplied

    ~(keystone_admin)$ system application-apply rook-ceph

Removing a monitor

To remove a monitor, the function 'monitor' must be removed from the capabilities list of the "host-fs ceph"

When the host has no OSD registered on the platform, remove the 'monitor' function of the "host-fs ceph" with the following command

   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=

When there are OSDs registered on the same host, only the 'monitor' function should be removed from "host-fs ceph" capabilities list. Use the following command

   ~(keystone_admin)$ system host-fs-modify <hostname> ceph --functions=osd

After the removal of the 'monitor' function, the application must be reapplied

    ~(keystone_admin)$ system application-apply rook-ceph

Floating Monitor (only in AIO-Duplex)

A Floating monitor is possible and recommended on AIO-Duplex platforms. The monitor roams and is always allocated on the active controller, providing redundancy and improving stability.

Adding the floating monitor

   Note: You should lock the inactive controller before add "controllerfs ceph-float" to the platform.

To add the floating monitor follow the procedure

Lock the inactive controller

   ~(keystone_admin)$ system host-lock controller-1 (with controller-0 as the active controller)

With the inactive controller locked, add the controllerfs

   ~(keystone_admin)$ system controllerfs-add ceph-float=<size>

Unlock the inactive controller

   ~(keystone_admin)$ system host-unlock controller-1 (with controller-0 as the active controller)

With inactive controller unlocked and available, reapply the Rook Ceph Application

    ~(keystone_admin)$ system application-apply rook-ceph

Removing the floating monitor

To remove the floating monitor, the function 'monitor' must be removed from the capabilities list of the "controllerfs ceph-float". Use the following command

    ~(keystone_admin)$ system controllerfs-modify ceph-float --functions=

After the removal of the 'monitor' function, the application must be reapplied

    ~(keystone_admin)$ system application-apply rook-ceph

Migrating between AIO-Duplex and AIO-Duplex+

Migrating to AIO-Duplex+

To migrate from AIO-Duplex to AIO-Duplex+ the floating monitor must be removed before the migration and new fixed monitor should be added in a worker after the migration is done.

Migrating to AIO-Duplex

To migrate from AIO-Duplex+ to AIO-Duplex, the fixed monitor should be removed from the cluster before the migration and a floating monitor should be added after the migration is done.

Services

Services are the storage types (or classes) that provides storage to each pod with some mount or storage space allocation.

Available Services

There are four possible services compatible with Rook Ceph. You can combine them following the rules

block (default)
- Not possible to be deployed together with ecblock.
- Will enable the block service in rook, will use cephRBD.

ecblock
- Not possible to be deployed together with block.
- Will enable the ecblock service in rook, will use cephRBD.

filesystem (default)
- Will enable the ceph filesystem and use cephFS.

object
- Will enable the ceph object store (RGW).

   IMPORTANT: A Service cannot be removed or replaced. Services only can be added

Adding New Services

To add a new service to the storage-backend, first choose a possible service compatible with the aforementioned rules.

Get the list of the current services of the storage-backend with the following command

    ~(keystone_admin)$ system storage-backend-show ceph-rook-store

Then, add the desired service to the list via the command

   ~(keystone_admin)$  system storage-backend-modify ceph-rook-store --services=<previous_list>,<new_service>

Reapply Rook Ceph application

    ~(keystone_admin)$ system application-apply rook-ceph

Example: For a storage-backend with the service list "block,filesystem", only 'object' can be added as service

   ~(keystone_admin)$ system storage-backend-modify ceph-rook-store --services=block,filesystem,object

Services Parameterization for the Open Model

In the 'open' deployment model, no specific configurations are enforced. Users are responsible for customizing settings based on their specific needs. To update configurations, a Helm override is required.

When applying a helm-override update, list-type values are completely replaced, not incrementally updated. For example, modifying cephFileSystems (or cephBlockPools, cephECBlockPools, cephObjectStores) via Helm override will overwrite the entire entry.

Here is two examples of how to change a parameter, using failureDomain as an example, for Cephfs and RBD

Cephfs

 # Get the current crush rule information
 ceph osd pool get kube-cephfs-data crush_rule
 
 # Get the current default values
 helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n  '/^cephFileSystems:/,/^[[:alnum:]_-]*:/p;' | sed '$d' > cephfs_overrides.yaml
 
 # Update the failure domain
 sed -i 's/failureDomain: osd/failureDomain: host/g' cephfs_overrides.yaml
 
 # Get the current user override values ("combined overrides" is what will be deployed)
 system helm-override-show rook-ceph rook-ceph-cluster rook-ceph
 
 # Set the new overrides
 system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values cephfs_overrides.yaml
 
 # Get the updated user override values
 system helm-override-show rook-ceph rook-ceph-cluster rook-ceph
 
 # Apply the application
 system application-apply rook-ceph
 
 # Confirm the current crush rule information
 ceph osd pool get kube-cephfs-data crush_rule

RBD

 # Get the current crush rule information
 ceph osd pool get kube-cephfs-data crush_rule
 
 # Retrieve the current values and extract the cephBlockPools section
 helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n '/^cephBlockPools:/,/^[[:alnum:]_-]*:/p;' | sed '$d' > rbd_overrides.yaml
 
 # Modify the failureDomain parameter from osd to host in the rbd_overrides.yaml file
 sed -i 's/failureDomain: osd/failureDomain: host/g' rbd_overrides.yaml
 
 # Set the update configuration
 system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values rbd_overrides.yaml
 
 # Apply the application
 system application-apply rook-ceph
 
 # Confirm the current crush rule information
 ceph osd pool get kube-cephfs-data crush_rule

Disable Helm Chart

   IMPORTANT: Do not disable any of the Rook Ceph Helm charts using "system helm-chart-attribute-modify"
   as this may result in a broken installation.

Installation

Preparing the Environment

Configuring deployment model and services

To configure the deployment model, first certify that there is no no ceph-store storage backend configured on the system

   ~(keystone_admin)$ system storage-backend-list

Create a storage backend for Rook Ceph, choosing your deployment model (controller, dedicated, open), and, optionally, the desired services (block or ecblock, filesystem, object)

   ~(keystone_admin)$ system storage-backend-add ceph-rook --deployment controller --confirmed

Create a host-fs ceph for each host that will house a Rook Ceph monitor (preferably an ODD number of hosts)

   ~(keystone_admin)$ system host-fs-add <hostname> ceph=<size>

Configuring OSDs

Check the uuid of the disks of the desired host that will house the OSDs

   ~(keystone_admin)$ system host-disk-list <hostname>

Note: The OSD placement should follow the chosen deployment model placement rules.

Add the desired disks to the system as OSDs (Preferably an EVEN number of OSDs)

   ~(keystone_admin)$ system host-stor-add <hostname> osd <disk_uuid>

Installing the application

After configuring environment acordingly to the chosen deployment model correctly, Rook Ceph should install automatically.

Verifying the Installation

Verify that rook is running

   [sysadmin@controller-0 ~(keystone_admin)]$ NS=rook-ceph
   [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n ${NS} -o wide -w 
   
   NAME                                            READY   STATUS      RESTARTS   AGE     IP              NODE         
   csi-cephfsplugin-provisioner-86d7d5d9b7-98gn2   6/6     Running     0          11m     172.16.192.101   controller-0
   csi-rbdplugin-nd8xk                             3/3     Running     0          11m     192.168.206.2    controller-0
   csi-rbdplugin-provisioner-c9cd4ffd7-mmfp9       6/6     Running     0          11m     172.16.192.100   controller-0
   rook-ceph-mds-kube-cephfs-a-7b544db87d-4ctrf    1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mds-kube-cephfs-b-5fcbb478fd-qjw4g    1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mgr-a-69fbb9d7b4-tcbt4                1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mon-a-55bbd66678-9cvt6                1/1     Running     0          12m     192.168.206.2    controller-0
   rook-ceph-operator-5ddc68f467-kqbwt             1/1     Running     0          12m     172.16.192.95    controller-0
   rook-ceph-osd-0-bfbf89745-679kp                 1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-osd-1-69799854f5-qvgvg                1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-osd-prepare-controller-0-jmxmz        0/1     Completed   0          11m     192.168.206.2    controller-0
   rook-ceph-provision-g8qfg                       0/1     Completed   0          5m20s   172.16.192.104   controller-0
   rook-ceph-tools-644f4dbc4b-8qc2m                1/1     Running     0          12m     192.168.206.2    controller-0
   stx-ceph-manager-66764cb49d-hxt4d               1/1     Running     0          11m     172.16.192.98    controller-0
   stx-ceph-osd-audit-28484346-r5nfq               0/1     Completed   0          8s      192.168.206.2    controller-0

Verify that the cluster is operational

   [sysadmin@controller-0 ~(keystone_admin)]$ ceph -s

Verify the ceph version

   [sysadmin@controller-0 ~(keystone_admin)]$ceph version
   ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)

Make sure that the disk is reported used

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | device_no | device_ | device_ | size_ | available_ | rpm | device_path                                |
   | de        | num     | type    | gib   | gib        |     |                                            |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | /dev/sda  | 2048    | SSD     | 520.0 | 0.0        | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
   | /dev/sdb  | 2064    | SSD     | 520.0 | 0.0        | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
   | /dev/sdc  | 2080    | SSD     | 520.0 | 519.996    | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+

Make sure that the volume group is reported

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-lvg-list controller-0
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+
   | LVG Name          | State       | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+
   | cgts-vg           | provisioned | wz--n- | 488.406          | 261.593          | 1           | 15          |
   | ceph-2e5d9569-... | provisioned | wz--n- | 519.996          | 0.0              | 1           | 1           |
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+

Make sure that the physical volume is reported

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-pv-list controller-0 --nowrap
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
  | lvm_pv_name | disk_or_part | disk_or_part_device_path                         | pv_state | pv_type   | lvm_vg_name  |
  |             | _device_node |                                                  |          |           |              |
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
  | /dev/sdb    | /dev/sdb     | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0       | provisnd | disk      | ceph-2e5d... |
  | /dev/sda5   | /dev/sda5    | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0-part5 | provisnd | partition | cgts-vg      |
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+

OSD Management

Add OSDs

Obtain the UUIDs of the target disk(s).

~(keystone_admin)$ system host-disk-list compute-0

Add the target disk(s) as OSDs.

~(keystone_admin)$ system host-stor-add compute-0 ed4c51a7-372f-4460-915d-43049cf23dd5
~(keystone_admin)$ system host-stor-add compute-0 ab35233a-9952-4437-87ed-80cc4d6817d7

At this point, the OSDs will be in the configuring-with-app state.

~(keystone_admin)$ system host-stor-list compute-0

Apply rook-ceph application and wait for the application to become applied.

~(keystone_admin)$ system application-apply rook-ceph

The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.

Use the system application-show rook-ceph command to monitor progress.

List the OSDs and verify that they are now in the configured state.

~(keystone_admin)$ system host-stor-list compute-0

Remove OSDs

Obtain the UUIDs of the OSDs to be removed.

~(keystone_admin)$ system host-stor-list compute-0

Mark the OSDs for removal.

~(keystone_admin)$ system host-stor-delete 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397

At this point, the OSDs will be in the deleting-with-app state.

~(keystone_admin)$ system host-stor-list compute-0

Apply rook-ceph application and wait for the application to become applied.

~(keystone_admin)$ system application-apply rook-ceph

The rook-ceph application may take several minutes to apply the changes and provision the OSDs before reaching the applied state.

Use the system application-show rook-ceph command to monitor progress.

Wait until the OSDs are removed.

The OSDs will be removed asynchronously after rook-ceph reaches the applied state. Then, once the OSDs entry disappears from system host-stor-list, the OSDs are successfully removed.

~(keystone_admin)$ system host-stor-list 3

Replace OSDs

Repeat the following steps for each target host, one at a time so that it does not impact Ceph availability.

Follow the steps in the Remove OSDs section to remove the target OSDs from the host.

Note: If after several minutes the OSD entry is reverted to the configured state, then the OSD removal is considered unsafe and is aborted to avoid data loss.

You can proceed using system host-stor-delete --force <uuid>. This will skip data integrity checks and will forcefully remove the OSD. However, this may cause loss of data. Thus, use his only when you are sure that there is enough replicas of data. Otherwise, before using the –force option, always attempt the standard deletion first.

Lock the host

~(keystone_admin)$ system host-lock <host>

After the host is locked, power off the host

~(keystone_admin)$ system host-power-off <host>

Replace the disk(s) while the host is powered off

Power on the host and wait for the host to become online.

Unlock the host

~(keystone_admin)$ system host-unlock <host>

Wait for the host to become unlocked/available.

Follow the steps in Add OSDs to add the new replaced disk(s) as OSDs.

Additional Features and Procedures

Add new OSDs on a running cluster

To add new OSDs to the cluster, just add the new OSD to the platform and reapply the application

       ~(keystone_admin)$ system host-stor-add <host> <disk_uuid>
       ~(keystone_admin)$ system application-apply rook-ceph

Add new monitor on a running cluster

To add a new monitor to the cluster, just add the host-fs to the desired host and reapply the application

       ~(keystone_admin)$ system host-fs-add <host> ceph=<size>
       ~(keystone_admin)$ system application-apply rook-ceph

Enabling Ceph Dashboard

To enable Ceph Dashboard a helm override must be provided before the application apply. You should provide a password coded in base64.

Create the override file

       $ openssl base64 -e <<< "my_dashboard_passwd"
       bXlfZGFzaGJvYXJkX3Bhc3N3ZAo=

       $ cat << EOF >> dashboard-override.yaml
       cephClusterSpec:
         dashboard:
           enabled: true
           password: "bXlfZGFzaGJvYXJkX3Bhc3N3ZAo="
       EOF

Apply the override file

   ~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --values dashboard-override.yaml

Check Rook Ceph pods

You can check the pods of the storage cluster running the following command

   kubectl get pod -n rook-ceph

Ceph Health Status Filter

Some Ceph health statuses can be filtered to avoid generating alarms. The detection of a particular Health error or warning can be disabled.

   IMPORTANT: Disabling the detection of any health error or warning can prevent the system from generating alarms, detecting issues and log generating. This feature must be used at user's discretion. The recommended is to use this feature temporarily during some analysis or procedure and then revert back to the default empty values.

There is two filters: health_filters_for_ignore for filtering at any time (always active) and health_filters_for_upgrade that applies the filter only during an upgrade of the Rook Ceph.

To apply the always-on filter (health_filters_for_ignore), use the following procedure:

Check for the name of any Ceph health issues that might wanted to be filtered out

   ~(keystone_admin)$ ceph health detail

Consult the list of the Ceph health issues currently ignored:

   ~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed -n '/ceph-manager-config.yaml:/,/^[^ ]/p' | sed -n 's/^[ ]*health_filters_for_ignore:[ ]*//p'

Edit the config map adding the name of the all Ceph health issues comma separated and delimited by [] to the list "health_filters_for_ignore"

   #Examples of useful Health Status to ignore: OSD_DOWN,PG_AVAILABILITY,BLUESTORE_SLOW_OP_ALERT,DB_DEVICE_STALLED_READ_ALERT
   ~(keystone_admin)$ health_filters='[<ceph_health_status_1>,<ceph_health_status_2>]' 
   ~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed "s/^\(\s*health_filters_for_ignore:\s*\).*/\1$health_filters/" | kubectl apply -f -

Restart the stx-ceph-manager pod

   ~(keystone_admin)$ kubectl rollout restart -n rook-ceph deployment stx-ceph-manager

To use the upgrade only filter (health_filters_for_upgrade), follow the same procedure but change commands for consult and edit for the following versions:

Consult command for only upgrade

 ~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed -n '/ceph-manager-config.yaml:/,/^[^ ]/p' | sed -n 's/^[ ]*health_filters_for_upgrade:[ ]*//p'

Edit command for only upgrade

 ~(keystone_admin)$ kubectl get configmap ceph-manager-config -n rook-ceph -o yaml | sed "s/^\(\s*health_filters_for_upgrade:\s*\).*/\1$health_filters/" | kubectl apply -f -

Uninstall Rook Ceph

To completely remove Rook Ceph, you must remove the application and clear all the environment configurations to prevent an automatic reinstall.

Note: Before removing the Rook Ceph application(s), unmount all the persistent volumes associated with the Rook Ceph cluster. This is achieved by removing all the applications dependent on StorageClasses provided by the Rook Ceph application. After removing the application(s), delete all PVCs and PVs that were previously provided to the application(s).

If dependent application removal is not desired, then minimal cleanup is needed. All pods with PVCs must be deleted, scaled down to zero, or edited removing the PVC volume mounts. Otherwise, Rook Ceph application removal will fail, volumes may hang, and nodes may require a restart. In this case, a --force option is needed to remove the Rook Ceph application and tear down the Ceph cluster.

Uninstalling