TUTORIAL: Replace a faulty disk on SERVERware
If one of the disks in your SERVERware fails, zpool will be in the DEGRADED state on the primary server.
zpool status # zpool status pool: NETSTOR state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 6 15:10:59 2016 config: NAME STATE READ WRITE CKSUM NETSTOR DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 SW3-NETSTOR-SRV1-1 ONLINE 0 0 0 SW3-NETSTOR-SRV2-1 FAULTED 3 0 0 too many errors errors: No known data errors First, you must make sure that the damaged disk is on a secondary server and not a primary one.
in this case, you can find it out from the line above:
SW3-NETSTOR-SRV2-1 FAULTED
SRV2 this means that server no. 2 has a damaged disk.
If so, you can proceed to the next step.
If the damaged disk is on the primary server SRV1, you must first perform a manual takeover and switch it to the secondary server.
To do it manually, connect to the secondary server and run the following command:
killall -SIGUSR1 sysmonit Then, you will need to physically replace the damaged disk in the server.
In the output of the zpool status command, you can see that SW3-NETSTOR-SRV2-1 is corrupted, for example:
SW3-NETSTOR-SRV2-1 FAULTED 3 0 0 too many errors If so, you must replace the disk labeled SW3-NETSTOR-SRV2-1 with a new one and add it to the zpool mirror.
First, physically remove the faulty disk from the server and replace it with a new disk.
After the replacement, you should see a new disk in the directory /dev/disk/by-id/ (example on line 15):
# ls -lah /dev/disk/by-id total 0 drwxr-xr-x 2 root root 480 Srp 27 08:57 . drwxr-xr-x 7 root root 140 Srp 27 08:13 .. lrwxrwxrwx 1 root root 9 Srp 27 08:13 ata-INTEL_SSDSC2CW060A3_CVCV308402M3060AGN -> ../../sde lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-INTEL_SSDSC2CW060A3_CVCV308402M3060AGN-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-INTEL_SSDSC2CW060A3_CVCV308402M3060AGN-part2 -> ../../sde2 lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-INTEL_SSDSC2CW060A3_CVCV308402M3060AGN-part9 -> ../../sde9 lrwxrwxrwx 1 root root 9 Srp 27 08:13 ata-ST31000520AS_5VX0BZN0 -> ../../sda lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-ST31000520AS_5VX0BZN0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 9 Srp 27 08:13 ata-WDC_WD10JFCX-68N6GN0_WD-WX61A465TH1Y -> ../../sdc lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-WDC_WD10JFCX-68N6GN0_WD-WX61A465TH1Y-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 9 Srp 27 08:13 ata-WDC_WD10JFCX-68N6GN0_WD-WX81EC512Y4H -> ../../sdd lrwxrwxrwx 1 root root 10 Srp 27 08:13 ata-WDC_WD10JFCX-68N6GN0_WD-WX81EC512Y4H-part1 -> ../../sdd1 lrwxrwxrwx 1 root root 9 Srp 27 08:57 ata-WDC_WD10JFCX-68N6GN0_WD-WXK1E6458WKX -> ../../sdb lrwxrwxrwx 1 root root 9 Srp 27 08:13 wwn-0x10076999618641940481x -> ../../sdd lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x10076999618641940481x-part1 -> ../../sdd1 lrwxrwxrwx 1 root root 9 Srp 27 08:13 wwn-0x11689569317835657217x -> ../../sdc lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x11689569317835657217x-part1 -> ../../sdc1 lrwxrwxrwx 1 root root 9 Srp 27 08:57 wwn-0x11769037186453098497x -> ../../sdb lrwxrwxrwx 1 root root 9 Srp 27 08:13 wwn-0x12757853320186451405x -> ../../sde lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x12757853320186451405x-part1 -> ../../sde1 lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x12757853320186451405x-part2 -> ../../sde2 lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x12757853320186451405x-part9 -> ../../sde9 lrwxrwxrwx 1 root root 9 Srp 27 08:13 wwn-0x7847552951345238016x -> ../../sda lrwxrwxrwx 1 root root 10 Srp 27 08:13 wwn-0x7847552951345238016x-part1 -> ../../sda1 Now that you have a block device name, you can create a table, a partition, and prepare the drive for use.
To create a partition table, use parted:
parted /dev/ --script -- mktable gpt Create a new label:
IMPORTANT: the label must be named in the following format: SW3-NETSTOR-SRVx-y.
Where “SRVx” corresponds to the server number and “-y” to the disk number.
Thus, in our example (SW3-NETSTOR-SRV2-1):
- SW3-NETSTOR-SRV2 – this is the virtual disk on SERVER 2
- 1 – is the disk number (disk 1)
Now add a label to the new disk.
Create the partition with the name corresponding to our faulty partition on the server. This name comes from the above:
SW3-NETSTOR-SRV2-1 FAULTED 3 0 0 too many errors
The command in this case will be:
parted /dev/ --script -- mktable ~# parted /dev/ --script -- mkpart "SW3-NETSTOR-SRV2-1" 1 -1 You have now added a new partition and created a label.
To replace the disk, you can use the sw-nvme commands listed below:
| Command | Description |
|---|---|
| sw-nvme list | Lists all connected devices with /dev/nvme-fabrics |
| sw-nvme discover | Discovers all devices exported on the remote host with the given IP and port |
| sw-nvme connect | Imports the remote device from the given IP, port and nqn |
| sw-nvme disconnect | Deletes the imported device from the host |
| sw-nvme disconnect-all | Deletes all imported devices from the host |
| sw-nvme import | For the given file in the appropriate JSON format, imports remote devices |
| sw-nvme reload-import | For the given file in the appropriate JSON format, imports remote devices after disconnecting all current imports |
| sw-nvme enable-modules | This command enables the kernel modules necessary for NVMe/TCP |
| sw-nvme enable-namespace | Enables the namespace with the given identifier |
| sw-nvme disable-namespace | Disables the namespace with the given identifier |
| sw-nvme load | For the given file in the appropriate JSON format, exports remote devices |
| sw-nvme store | If the devices are exported manually, store saves the system configuration in the appropriate JSON format |
| sw-nvme clear | This command removes the exported device from the system configuration. If specified with 'all', it will remove all configurations |
| sw-nvme export | For the given URL parameter, exports the device on the port with the nqn |
| sw-nvme export-stop | Deletes the device currently being exported on the port with the identifier |
| sw-nvme reload-configuration | For the given file in the appropriate JSON format, exports remote devices after deleting all current exports |
| sw-nvme replace-disk | This command combines 'clear all' and reload-configuration to facilitate the disk replacement procedure on SERVERware |
| sw-nvme expand-pool | This command updates the export configuration and adds a new namespace in the sw-mirror subsystem for SERVERware |
Now, you will need to perform the replacement of the old disk with the new one using the following command:
sw-nvme replace-disk --old /dev/disk/by-id/old_disk_id --new /dev/disk/by-id/new_disk_id To find the old disk ID, use the sw-nvme show command.
Example (line 16):
~#sw-nvme show { "config": "/sys/kernel/config/nvmet", "hosts": [ "3cc5c2aa47825e608570a938971bcd7c" ], "subsystems": { "sw-mirror": { "acl": [ "3cc5c2aa47825e608570a938971bcd7c" ], "namespaces": [ { "id": 1, "device": "/dev/disk/by-id/ata-KINGSTON_SA400S37120G_50026B73804B902A", "enabled": true } ], "allow_any_host": false } }, "ports": { "1": { "address": "1.1.1.31", "port": 4420, "address_family": "ipv4", "trtype": "tcp", "subsystems": "sw-mirror" } } } Now that you have the identifier of the old disk and the new disk, your command for the disk replacement will be:
~#sw-nvme replace-disk --old /dev/disk/by-id/ata-KINGSTON_SA400S37120G_50026B73804B902A --new /dev/disk/by-id/ata-WDC_WD10JFCX-68N6GN0_WD-WXK1E6458WKX This marks the end of the procedure on the secondary server.
Then, on the primary server, add a newly created virtual disk to the ZFS pool.
Then, run:
partprobe You can see the zpool status:
~# zpool status pool: NETSTOR state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 6 15:10:59 2016 config: NAME STATE READ WRITE CKSUM NETSTOR DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 SW3-NETSTOR-SRV1-1 ONLINE 0 0 0 SW3-NETSTOR-SRV2-1 FAULTED 3 0 0 too many errors errors: No known data errors The result is as follows:
SW3-NETSTOR-SRV2-1 shows a FAULTED state, indicating a problem with the secondary disk.
You must now proceed to replace the GUID of the old disk with that of the new one, so that zpool can identify the latter.
To make this change in the zpool, you must first identify the new GUID. You can use the zdb command for this purpose (lines 35 & 36):
~# zdb NETSTOR: version: 5000 name: 'NETSTOR' state: 0 txg: 15 pool_guid: 14112818788567273316 errata: 0 hostname: 'HydraA-1' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 14112818788567273316 children[0]: type: 'mirror' id: 0 guid: 17350955661294397060 metaslab_array: 34 metaslab_shift: 33 ashift: 12 asize: 1000164294656 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 11541101181530606692 path: '/dev/disk/by-partlabel/SW3-NETSTOR-SRV1-1' whole_disk: 1 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 12365645279327980714 path: '/dev/disk/by-partlabel/SW3-NETSTOR-SRV2-1' whole_disk: 1 create_txg: 4 features_for_read: com.delphix:hole_birth com.delphix:embedded_data Note the important line for the zdb output:
guid: 12365645279327980714 path: '/dev/disk/by-partlabel/SW3-NETSTOR-SRV2-1' The guid part must be updated in the zpool.
You can update the guid with the command:
zpool replace NETSTOR -f For example:
zpool replace NETSTOR 12365645279327980714 /dev/disk/by-partlabel/SW3-NETSTOR-SRV2-1 -f Now check the zpool status:
~# zpool status pool: NETSTOR state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Tue Dec 6 16:12:53 2016 591M scanned out of 728M at 65,6M/s, 0h0m to go 590M resilvered, 81,14% done config: NAME STATE READ WRITE CKSUM NETSTOR DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 SW3-NETSTOR-SRV1-1 ONLINE 0 0 0 replacing-1 UNAVAIL 0 0 0 old UNAVAIL 0 0 0 corrupted data SW3-NETSTOR-SRV2-1 ONLINE 0 0 0 (resilvering) errors: No known data errors You must now wait until the zpool has finished its resilvering and your disk replacement will be operational.
