David.desrosiers: David.desrosiers moved page Storage Node Failures and Maintenance to OpsGuide/Storage Node Failures and Maintenance without leaving a redirect

2017-11-14T02:42:27Z

David.desrosiers moved page Storage Node Failures and Maintenance to OpsGuide/Storage Node Failures and Maintenance without leaving a redirect

David.desrosiers: Created page with "==
Storage Node Failures and Maintenance
== Because of the high redundancy of Object Storage, dealing with object stora..."

2017-11-13T19:57:24Z

Created page with "== <div id="storage-node-failures-and-maintenance"> Storage Node Failures and Maintenance</div> == Because of the high redundancy of Object Storage, dealing with object stora..."

New page

== <div id="storage-node-failures-and-maintenance"> Storage Node Failures and Maintenance</div> ==

Because of the high redundancy of Object Storage, dealing with object storage node issues is a lot easier than dealing with compute node issues.

== <div id="rebooting-a-storage-node"> Rebooting a Storage Node</div> ==

If a storage node requires a reboot, simply reboot it. Requests for data hosted on that node are redirected to other copies while the server is rebooting.

== <div id="shutting-down-a-storage-node"> Shutting Down a Storage Node</div> ==

If you need to shut down a storage node for an extended period of time (one or more days), consider removing the node from the storage ring. For example:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># swift-ring-builder account.builder remove <ip address of storage node>
# swift-ring-builder container.builder remove <ip address of storage node>
# swift-ring-builder object.builder remove <ip address of storage node>
# swift-ring-builder account.builder rebalance
# swift-ring-builder container.builder rebalance
# swift-ring-builder object.builder rebalance</syntaxhighlight>
Next, redistribute the ring files to the other nodes:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># for i in s01.example.com s02.example.com s03.example.com
> do
> scp *.ring.gz $i:/etc/swift
> done</syntaxhighlight>
These actions effectively take the storage node out of the storage cluster.

When the node is able to rejoin the cluster, just add it back to the ring. The exact syntax you use to add a node to your swift cluster with <code>swift-ring-builder</code> heavily depends on the original options used when you originally created your cluster. Please refer back to those commands.

== <div id="replacing-a-swift-disk"> Replacing a Swift Disk</div> ==

If a hard drive fails in an Object Storage node, replacing it is relatively easy. This assumes that your Object Storage environment is configured correctly, where the data that is stored on the failed drive is also replicated to other drives in the Object Storage environment.

This example assumes that <code>/dev/sdb</code> has failed.

First, unmount the disk:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># umount /dev/sdb</syntaxhighlight>
Next, physically remove the disk from the server and replace it with a working disk.

Ensure that the operating system has recognized the new disk:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># dmesg | tail</syntaxhighlight>
You should see a message about <code>/dev/sdb</code>.

Because it is recommended to not use partitions on a swift disk, simply format the disk as a whole:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># mkfs.xfs /dev/sdb</syntaxhighlight>
Finally, mount the disk:

<syntaxhighlight lang=bash style="border:2px solid #ccc;"># mount -a</syntaxhighlight>
Swift should notice the new disk and that no data exists. It then begins replicating the data to the disk from the other existing replicas.

← Older revision	Revision as of 02:42, 14 November 2017
(No difference)

OpsGuide/Storage Node Failures and Maintenance - Revision history

David.desrosiers: David.desrosiers moved page Storage Node Failures and Maintenance to OpsGuide/Storage Node Failures and Maintenance without leaving a redirect

David.desrosiers: Created page with "== Storage Node Failures and Maintenance == Because of the high redundancy of Object Storage, dealing with object stora..."

David.desrosiers: Created page with "==
Storage Node Failures and Maintenance
== Because of the high redundancy of Object Storage, dealing with object stora..."