Jump to: navigation, search

Difference between revisions of "StarlingX/Containers/Applications/rook-ceph-migration"

(Created page with "= StarlingX Rook Ceph Migration App installation guide = == Source == * [https://opendev.org/starlingx/rook-ceph Code Repository] * [https://review.opendev.org/q/project:star...")
 
m (→‎Building)
 
Line 9: Line 9:
 
This application is under construction and the contents of the application and this document will likely change.
 
This application is under construction and the contents of the application and this document will likely change.
  
−
== Building ==
+
== Tarball Package ==
−
* Build from the Debian Build environment (if needed):
+
* Get the .tgz file located at:
 +
    /usr/local/share/applications/helm/
 +
* Or build it yourself using the Debian Build environment:
 
     build-pkgs -c -p migration-rook-ceph-helm,python3-k8sapp-migration-rook-ceph,stx-migration-rook-ceph-helm
 
     build-pkgs -c -p migration-rook-ceph-helm,python3-k8sapp-migration-rook-ceph,stx-migration-rook-ceph-helm
  

Latest revision as of 20:46, 26 April 2024

StarlingX Rook Ceph Migration App installation guide

Source


Information

This application is under construction and the contents of the application and this document will likely change.

Tarball Package

  • Get the .tgz file located at:
   /usr/local/share/applications/helm/
  • Or build it yourself using the Debian Build environment:
   build-pkgs -c -p migration-rook-ceph-helm,python3-k8sapp-migration-rook-ceph,stx-migration-rook-ceph-helm

Configuration

  • In a StarlingX system installed without a storage backend, add the Rook storage backend
   sysadmin@controller-0:~$ source /etc/platform/openrc
   [sysadmin@controller-0 ~(keystone_admin)]$ system storage-backend-add ceph-rook
   
   WARNING : THIS OPERATION IS NOT REVERSIBLE AND CANNOT BE CANCELLED. 
   
   Please set the 'confirmed' field to execute this operation for the ceph-rook backend.
   
   [sysadmin@controller-0 ~(keystone_admin)]$ system storage-backend-add ceph-rook --confirmed
   
   System configuration has changed.
   Please follow the administrator guide to complete configuring the system.
   
   +-----------------+-----------+-------------+----------------------------------------+----------+--------------+
   | name            | backend   | state       | task                                   | services | capabilities | 
   +-----------------+-----------+-------------+----------------------------------------+----------+--------------+
   | ceph-rook-store | ceph-rook | configuring | {'controller-0': 'applying-manifests'} | None     |              |
   +-----------------+-----------+-------------+----------------------------------------+----------+--------------+
   
   [sysadmin@controller-0 ~(keystone_admin)]$ system storage-backend-list    
   +-----------------+-----------+------------+------+----------+--------------+
   | name            | backend   | state      | task | services | capabilities |
   +-----------------+-----------+------------+------+----------+--------------+
   | ceph-rook-store | ceph-rook | configured | None | None     |              |
   +-----------------+-----------+------------+------+----------+--------------+
  • Upload the app (if needed)
   [sysadmin@controller-0 ~(keystone_admin)]$ system application-upload ~/rook-ceph-migration-23.09-42.tgz

   Then follow one of the following configurations according to you system:
   [AIO-SX Simplex]
   [AIO-DX Duplex]
   [Standard]

AIO-SX - Simplex

  • Add system labels for rook-ceph
   sysadmin@controller-0:~$ source /etc/platform/openrc

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-0 ceph-mon-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 719c3828-3806-4711-8824-774c1037316d |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$system host-label-assign controller-0 ceph-mgr-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 11bec756-0d76-4fce-89a6-c1f0a0677f4a |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mgr-placement                   |
   | label_value | enabled                              | 
   +-------------+--------------------------------------+

   Proceed to OSD configuring [OSD Configurations]

AIO-DX - Duplex

  • Add system labels for rook-ceph
   sysadmin@controller-0:~$ source /etc/platform/openrc

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-0 ceph-mon-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 719c3828-3806-4711-8824-774c1037316d |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-0 ceph-mgr-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 11bec756-0d76-4fce-89a6-c1f0a0677f4a |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mgr-placement                   |
   | label_value | enabled                              | 
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-1 ceph-mgr-placement=enabled;
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 9d92bac5-ba77-483a-9b2e-dfb57302a54b |
   | host_uuid   | f56121ea-cf37-48c2-9e77-5708417a6963 |
   | label_key   | ceph-mgr-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-1 ceph-mon-placement=enabled;

   system host-label-assign controller-1 ceph-mgr-placement=enabled;
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 9720faf3-7a77-4072-8c1e-ab6f236821a1 |
   | host_uuid   | f56121ea-cf37-48c2-9e77-5708417a6963 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   Proceed to OSD configuring [OSD Configurations]

Standard

  • Add system labels for rook-ceph
   sysadmin@controller-0:~$ source /etc/platform/openrc

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-0 ceph-mon-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 719c3828-3806-4711-8824-774c1037316d |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-0 ceph-mgr-placement=enabled
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 11bec756-0d76-4fce-89a6-c1f0a0677f4a |
   | host_uuid   | aed595fc-9fba-4aab-981a-ae2c04193689 |
   | label_key   | ceph-mgr-placement                   |
   | label_value | enabled                              | 
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-1 ceph-mgr-placement=enabled;
   
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 9d92bac5-ba77-483a-9b2e-dfb57302a54b |
   | host_uuid   | f56121ea-cf37-48c2-9e77-5708417a6963 |
   | label_key   | ceph-mgr-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign controller-1 ceph-mon-placement=enabled;

   system host-label-assign controller-1 ceph-mgr-placement=enabled;
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 9720faf3-7a77-4072-8c1e-ab6f236821a1 |
   | host_uuid   | f56121ea-cf37-48c2-9e77-5708417a6963 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+
  • Add a label for mon placement on your worker
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign compute-0 ceph-mon-placement=enabled;

   system host-label-assign compute-0 ceph-mon-placement=enabled;
   +-------------+--------------------------------------+
   | Property    | Value                                |
   +-------------+--------------------------------------+
   | uuid        | 2e267859-e845-48e1-be61-ca3b376b4b30 |
   | host_uuid   | c102ceb8-7102-4fb7-acd7-0fea2a4fe8a1 |
   | label_key   | ceph-mon-placement                   |
   | label_value | enabled                              |
   +-------------+--------------------------------------+

   Proceed to OSD configuring [OSD Configurations]

OSD configurations

  • Identify the disk to use for bluestore OSD
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list 1
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | device_no | device_ | device_ | size_ | available_ | rpm | device_path                                |
   | de        | num     | type    | gib   | gib        |     |                                            |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | /dev/sda  | 2048    | SSD     | 520.0 | 0.0        | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
   | /dev/sdb  | 2064    | SSD     | 520.0 | 519.996    | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
   | /dev/sdc  | 2080    | SSD     | 520.0 | 519.996    | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
  • Make sure that the disk has been purged
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-wipe -s --confirm controller-0 /dev/sdb
  • Set up the overrides with all desired OSD configurations (e.g.: using sdb as OSD)
   [sysadmin@controller-0 ~(keystone_admin)]$ cat <<EOF > /home/sysadmin/ceph-values.yml
   cephClusterSpec:
     storage:
       useAllNodes: false
       useAllDevices: false
       nodes:
       - name: controller-0
         devices:
         - name: /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0
   EOF
   [sysadmin@controller-0 ~(keystone_admin)]$ cat /home/sysadmin/ceph-values.yml
  • Set the overrides
   [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-list rook-ceph-migration
   [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-update rook-ceph-migration rook-ceph-cluster rook-ceph --values /home/sysadmin/ceph-values.yml

   Proceed to Installation step [Installation]

Installation

  • Verify current overrides
   [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show rook-ceph-migration rook-ceph rook-ceph
   [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show rook-ceph-migration rook-ceph-cluster rook-ceph
   [sysadmin@controller-0 ~(keystone_admin)]$ system helm-override-show rook-ceph-migration rook-ceph-provisioner rook-ceph
  • Apply the app
   [sysadmin@controller-0 ~(keystone_admin)]$ system application-apply rook-ceph-migration
   +---------------+---------------------------------------+
   | Property      | Value                                 |
   +---------------+---------------------------------------+
   | active        | False                                 |
   | app_version   | 23.09-41                              |
   | created_at    | 2023-12-18T23:25:07.201915+00:00      |
   | manifest_file | fluxcd-manifests                      |
   | manifest_name | rook-ceph-migration-fluxcd-manifests  |
   | name          | rook-ceph-migration                   |
   | progress      | None                                  |
   | status        | applying                              |
   | updated_at    | 2023-12-19T16:13:46.924152+00:00      |
   +---------------+---------------------------------------+
   Please use 'system application-list' or 'system application-show rook-ceph-migration' to view the current progress.

   [sysadmin@controller-0 ~(keystone_admin)]$ system application-list
   +--------------------------+----------+-------------------------------------------+----------+-----------+
   | application              | version  | manifest name                             | status   | progress  |
   +--------------------------+----------+-------------------------------------------+----------+-----------+
   | cert-manager             | 1.0-69   | cert-manager-fluxcd-manifests             | applied  | completed |
   | dell-storage             | 1.0-6    | dell-storage-fluxcd-manifests             | uploaded | completed |
   | nginx-ingress-controller | 1.0-50   | nginx-ingress-controller-fluxcd-manifests | applied  | completed |
   | oidc-auth-apps           | 1.0-45   | oidc-auth-apps-fluxcd-manifests           | uploaded | completed |
   | platform-integ-apps      | 1.2-119  | platform-integ-apps-fluxcd-manifests      | uploaded | completed |
   | rook-ceph-migration      | 23.09-41 | rook-ceph-migration-fluxcd-manifests      | applied  | completed |
   +--------------------------+----------+-------------------------------------------+----------+-----------+

Verifying the Installation

  • Verify that rook is running
   [sysadmin@controller-0 ~(keystone_admin)]$ NS=rook-ceph
   [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n ${NS} -o wide -w 
   
   NAME                                            READY   STATUS      RESTARTS   AGE     IP              NODE         
   csi-cephfsplugin-provisioner-86d7d5d9b7-98gn2   6/6     Running     0          11m     172.16.192.101   controller-0
   csi-rbdplugin-nd8xk                             3/3     Running     0          11m     192.168.206.2    controller-0
   csi-rbdplugin-provisioner-c9cd4ffd7-mmfp9       6/6     Running     0          11m     172.16.192.100   controller-0
   rook-ceph-mds-kube-cephfs-a-7b544db87d-4ctrf    1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mds-kube-cephfs-b-5fcbb478fd-qjw4g    1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mgr-a-69fbb9d7b4-tcbt4                1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-mon-a-55bbd66678-9cvt6                1/1     Running     0          12m     192.168.206.2    controller-0
   rook-ceph-operator-5ddc68f467-kqbwt             1/1     Running     0          12m     172.16.192.95    controller-0
   rook-ceph-osd-0-bfbf89745-679kp                 1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-osd-1-69799854f5-qvgvg                1/1     Running     0          11m     192.168.206.2    controller-0
   rook-ceph-osd-prepare-controller-0-jmxmz        0/1     Completed   0          11m     192.168.206.2    controller-0
   rook-ceph-provision-g8qfg                       0/1     Completed   0          5m20s   172.16.192.104   controller-0
   rook-ceph-tools-644f4dbc4b-8qc2m                1/1     Running     0          12m     192.168.206.2    controller-0
   stx-ceph-manager-66764cb49d-hxt4d               1/1     Running     0          11m     172.16.192.98    controller-0
   stx-ceph-osd-audit-28484346-r5nfq               0/1     Completed   0          8s      192.168.206.2    controller-0


  • Verify that the cluster is operational
   [sysadmin@controller-0 ~(keystone_admin)]$ ceph -s
  • Verify the ceph version
   [sysadmin@controller-0 ~(keystone_admin)]$ceph version
   ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable)
  • Make sure that the disk is reported used
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | device_no | device_ | device_ | size_ | available_ | rpm | device_path                                |
   | de        | num     | type    | gib   | gib        |     |                                            |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
   | /dev/sda  | 2048    | SSD     | 520.0 | 0.0        | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 |
   | /dev/sdb  | 2064    | SSD     | 520.0 | 0.0        | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 |
   | /dev/sdc  | 2080    | SSD     | 520.0 | 519.996    | N/A | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 |
   +-----------+---------+---------+-------+------------+-----+--------------------------------------------+
  • Make sure that the volume group is reported
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-lvg-list controller-0
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+
   | LVG Name          | State       | Access | Total Size (GiB) | Avail Size (GiB) | Current PVs | Current LVs |
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+
   | cgts-vg           | provisioned | wz--n- | 488.406          | 261.593          | 1           | 15          |
   | ceph-2e5d9569-... | provisioned | wz--n- | 519.996          | 0.0              | 1           | 1           |
   +-------------------+-------------+--------+------------------+------------------+-------------+-------------+
  • Make sure that the physical volume is reported
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-pv-list controller-0 --nowrap
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
  | lvm_pv_name | disk_or_part | disk_or_part_device_path                         | pv_state | pv_type   | lvm_vg_name  |
  |             | _device_node |                                                  |          |           |              |
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
  | /dev/sdb    | /dev/sdb     | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0       | provisnd | disk      | ceph-2e5d... |
  | /dev/sda5   | /dev/sda5    | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0-part5 | provisnd | partition | cgts-vg      |
  +-------------+--------------+--------------------------------------------------+----------+-----------+--------------+
Retrieved from "https://wiki.openstack.org/w/index.php?title=StarlingX/Containers/Applications/rook-ceph-migration&oldid=184752"