Synology, trying to bring back a failed RAID array

so I manage to destroy some of my data… I have a ds916+ and would like to recover the data from the drives, I ordered a new drive because one was failing, and another one failed during resync… so ye, double disk failure.

Here I’m now:

root@NAS:~# mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Sat Mar  3 14:15:17 2018
     Raid Level : raid10
     Array Size : 3897385088 (3716.84 GiB 3990.92 GB)
  Used Dev Size : 1948692544 (1858.42 GiB 1995.46 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Wed Apr 14 21:28:42 2021
          State : clean, FAILED
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

           Name : NAS:2  (local to host NAS)
           UUID : 928d46c6:72c08f44:8997f935:c9c219cf
         Events : 11818

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync set-A   /dev/sda3
       1       8       19        1      active sync set-B   /dev/sdb3
       -       0        0        2      removed
       3       8       51        3      faulty active sync set-B   /dev/sdd3

Now, when I do a:

mdadm --manage /dev/md2 --add /dev/sdc3

This drive is being added as a spare:

root@NAS:~# mdadm --detail /dev/md2
/dev/md2:
        Version : 1.2
  Creation Time : Sat Mar  3 14:15:17 2018
     Raid Level : raid10
     Array Size : 3897385088 (3716.84 GiB 3990.92 GB)
  Used Dev Size : 1948692544 (1858.42 GiB 1995.46 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Wed Apr 14 21:28:12 2021
          State : clean, FAILED
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : near=2
     Chunk Size : 64K

           Name : NAS:2  (local to host NAS)
           UUID : 928d46c6:72c08f44:8997f935:c9c219cf
         Events : 11817

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync set-A   /dev/sda3
       1       8       19        1      active sync set-B   /dev/sdb3
       -       0        0        2      removed
       3       8       51        3      faulty active sync set-B   /dev/sdd3

       4       8       35        -      spare   /dev/sdc3

What I would like to achieve, is that the sdc3 will jump back to it’s place and with that I hope I get access to some of the data, the drive was working before replacement.

Also, the following mdadm command doesn’t work:

root@NAS:~# mdadm --manage /dev/md2 --re-add /dev/sdc3
mdadm: --re-add for /dev/sdc3 to /dev/md2 is not possible

Here is also some additional information which may help:

root@NAS:~# mdadm --examine /dev/sdc2
/dev/sdc2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : fea95269:e7e6ab4e:cced5de7:ca715931 (local to host NAS)
  Creation Time : Sat Jan  1 01:00:09 2000
     Raid Level : raid1
  Used Dev Size : 2097088 (2047.94 MiB 2147.42 MB)
     Array Size : 2097088 (2047.94 MiB 2147.42 MB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 1

    Update Time : Wed Apr 14 21:09:33 2021
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : c01e878e - correct
         Events : 158


      Number   Major   Minor   RaidDevice State
this     2       8       34        2      active sync   /dev/sdc2

   0     0       8        2        0      active sync   /dev/sda2
   1     1       8       18        1      active sync   /dev/sdb2
   2     2       8       34        2      active sync   /dev/sdc2
   3     3       8       50        3      active sync   /dev/sdd2

root@NAS:~# fdisk -l /dev/sd(a-d)
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x27fe7bf8

Device     Boot   Start        End    Sectors  Size Id Type
/dev/sda1          2048    4982527    4980480  2.4G fd Linux raid autodetect
/dev/sda2       4982528    9176831    4194304    2G fd Linux raid autodetect
/dev/sda3       9437184 3906824351 3897387168  1.8T fd Linux raid autodetect
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x26a4ac9c

Device     Boot   Start        End    Sectors  Size Id Type
/dev/sdb1          2048    4982527    4980480  2.4G fd Linux raid autodetect
/dev/sdb2       4982528    9176831    4194304    2G fd Linux raid autodetect
/dev/sdb3       9437184 3906824351 3897387168  1.8T fd Linux raid autodetect
Disk /dev/sdc: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x7eba1232

Device     Boot   Start        End    Sectors  Size Id Type
/dev/sdc1          2048    4982527    4980480  2.4G fd Linux raid autodetect
/dev/sdc2       4982528    9176831    4194304    2G fd Linux raid autodetect
/dev/sdc3       9437184 3906824351 3897387168  1.8T fd Linux raid autodetect
Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xea4cb292

Device     Boot   Start        End    Sectors  Size Id Type
/dev/sdd1          2048    4982527    4980480  2.4G fd Linux raid autodetect
/dev/sdd2       4982528    9176831    4194304    2G fd Linux raid autodetect
/dev/sdd3       9437184 3906824351 3897387168  1.8T fd Linux raid autodetect

Please help! 🙂

Rebuild RAID with Old HDD in Synology NAS

From a Synology NAS with two bays, one of the HDDs has been removed.

The remaining HDD is working fine as remaining part of the degraded RAID 1.

Is it possible to re-add the once removed HDD in the NAS to fix the downgraded RAID 1, like it would with a brand new drive?

Or may this be a problem, because the host remembers the HDD (e.g. by the S/N) and cannot decide which drive is up to date?

cluster – Zero downtime with Kubernetes on top of GlusterFs on top of a ZFS raid – Is this the best solution?

A client requests me to make a robust system to run containerized microservices within their LAN.

Restriction: They give me 2 machines and 6 data-disks. No more hardware. I have to build “the best I can” here.

My worries are the storage and availability. Speed/performance is not an issue.

I’m thinking of:

  • In each machine build a RAID-5 using 3 data-disks, yielding in one data-volume in each machine. ZFS for example.
  • Tie the 2 machines with a distributed filesystem. GlusterFs for example.
  • Then use Kubernetes to create a cluster of 2 nodes pointing their Persistent Volumes to the distributed FS.
  • The fact the kubernetes cluster runs in the same hardware than the distributed filesystem is a mere casuality.

Question is: Any better solution given the restrictions of the client?

Read the context to understand more.

I’m designing a server architecture for running a bunch of 30 microservices locally for a radio station. No AWS, no cloud. We are talking about on-premises server.

For the whole scenario speed is not an issue (traffic is low). The business drivers here are:

  • Data persistance (minimize the risk of loosing data).
  • High availability (minimize the risk of a downtime).

If at any point they are incompatible, avoiding data-loose takes precedence over high-availability: In the limit, I can tell the journalists to stop working for a few minutes; but we can’t loose the interview that was recorded earlier this morning.

They run now all in one server with no containers and no data redundancy beyond backups. They experienced a disaster on 2018 and took them 2 full days to recover. The radio had to stop all the employees working, re-install the full OS, reinstall all the applications by hand, recover all the data from the backups, test everything then thell the journalists “back to write news”.

Of course this is not acceptable in those days (even it was not acceptable in 2018).

To overcome this, they recently bought 2 servers with 1 system disk + 3 data disks on each (total 6 data-disks). The initial idea they had for the data-disks is to make a local software RAID-5 across the 3 data-disks within each server.

Call the servers alpha and beta. The two machines are the same in cpu, ram and system-disk as well as the 3 data-disks. So the computers are exact clones. They will both run Ubuntu linux.

The idea they had is to use alpha as the main server and make beta be a “clone” of alpha so if it dies they can switch over the clients to beta in half an hour by manually reconfiguring the clients to point to another IP.

Nevertheless I think that the current technologies should enable me to create some sort of cluster so they are both alive and fully-in-sync so if anyone of those break, the clients experience zero-downtime.

The radio station initially requested me to build a RAID on alpha, via ZFS and then another RAID on beta via ZFS. Set a bunch of dockers on alpha with --restart=always and then point the clients of the services (running in their journalists’ respective PCs) to alpha (think services as news writing, image uploading, audio recording, program scheduling, web publishing, media transcoding, local live stream feed to the cloud, etc.).

About the storage, their initial though was:

  • For photos and audios, make regular backups from alpha to beta.
  • For MySQL have a master-master in alphabeta so beta is mostly acting as a slave but ready to be used in case of alpha dies.

Then if alpha breaks, switch all the clients to beta.

I’m specifically interested in the storage part of the story.

Instead of “manually switching clients” I was thinking of using kubernetes to make a cluster of 2 worker nodes. As I can’t have separate hardware to act as “kubernetes master” I was thinking of making also alpha and beta be both redundant kubernetes masters of themselves as workers.

So alpha would be a kubernetes master for alpha and beta nodes, as well as beta would be a redundant master of kubernetes also for both alpha and beta.

Up to here no problem.

Until we arrive to the storage.

When it comes to Persistent Volumes in kubernetes, the users of kubernetes launching their pods/containers need to rely that the data will not be lost. Some system administrator (in this case me) needs to “build” the redundancy below to ensure the volume “is” there with the proper data.

So this is what I was thinking:

1. A local ZFS layer

  • In the operating system of alpha (native to the system, forget kubernetes for a second) use ZFS to make a RAID across the 3 data-disks (equal in size). So if each disk is, say 1TB, there are 3TB of which 2TB will be available in the data volume and 1TB is under the hood for redundancy. Let’s call the disks A1, A2 and A3. Let’s call the ZFS volume A.

  • In beta, replicate the structure. Disks B1, B2, B3. Let’s call the ZFS volume B.

Up to here I’d have have 2 independent servers each protected against a single failure of a single disk. No protection against 2-disk simultaneus failure. No protection against a full-node down.

2. A distributed GlusterFs layer

  • Then create GlusterFs across alpha and beta on top of the ZFS volumes. I understand that GlusterFs has to give me some sort of mirroring configuration, so the ZFS volumes A and B are one mirror of the other. So if A is 2TB and B is 2TB the “total available storage” is also 2TB for use.

  • So adding up GlusterFs and ZFS at this moment, from the 6TB in total hardware capacity, 2TB are available for users and therefore 4TB are acting as redundancy.

Up to here, I should have a “distributed disk” that has much more redundancy and allows failure of 2 disks and also node-failure.

I see protection of 2 disks failing in the following manner:

  1. If the two disks pertain to different volumes (say fails A2 and B3) then each NFS separately protects against that and both ZFS volumes A and B are not disrupted (GlusterFs sees no changes).
  2. If the 2 disks failling belong to the same node, then the full volume is failing. For example a failure in A2 and A1 makes A broken. But GlusterFs should be able to balance to use “only 1 node” until the other becomes available (in this case “use only B until A comes back again”).

3. Kubernetes container runtime for service cluster + Kubernetes Persistent Volumes.

  • Finally, use Kubernetes Persistent Volumes would point to the GlusterFs volumes.

  • If I had 4 machines, probably I’d use 2 as kubernetes nodes and 2 for storage acting as a networked storage to the cluster.

  • But we only have 2 physical machines so kubernetes will point “persistent volumes” to “GlusterFs” exactly as if they were “in another remote machine” making it agnostic that the volumes are physically in the same nodes.

I have never physically built a construct like this before. Over the paper, it works. I wonder if the reality is different.

Given the constraints (2 machines, 6 data-disks), question is:

  • Is this topology the best way to create a mini-cluster with zero-downtime and data-redundancy for the client?
  • If not, what changes should I apply and why.

How To Recover A Crashed Raid 5?

How to recover a crashed RAID 5?

Setting up raid hardware with out deleting data

so I have a home lab, and in the beginning, I started with only 1 DISK, and in the raid config, I bypassed the raid because I didn’t have 2 disks to make a RAID 10 or 1, but now, I want some redundancy, and I want to install a new identical disk. My problem is that even though I know how to set up a raid in the BIOS, if I create a new one, it will wipe the disk and all my data, and I was wondering if it was possible to make a new raid 1 but without losing the data.

My server is a HP Proliant DL380 G8 server.

Would rebuild work or would i need to use a different method?

PERC H740p single-disk RAID 0 versus JBOD/Pass-Through/eHBA/IT mode (For ZFS on Linux)

We have a server with PERC H740P mini (embedded) and a 2 disk RAID 1 with EXT4 for the OS (CentOS 7.8) and a 6 disk raidz2 ZFS on Linux setup for the data, all on the same controller.

It’s generally considered bad® to run ZFS with HW RAID, but this controller doesn’t seem to support a mixed RAID/non-RAID setup, so the 6 data drives (for ZFS) are all single disk RAID 0.

We see occasional ZFS panics that I suspect are due to the RAID controller interfering. Where can I read about the exact semantics of single disk RAID 0 for this controller so that I might be able to determine if it is the cause?

Are there any perccli64 incantations or other debuggery I could use to see what the controller might have been doing when ZFS pooped the proverbial bed?

raid – Random json files getting set to 0 bytes

over time I have been noticing some random files json files being set to 0 bytes, at first I thought it was just a bug in one of my programs but its been happening all over my system.

when I try to delete the files it says the file does not exist

I use a 4 disk raid 5 system, the format is refs. im using windows server 22

has anyone else experienced this and what would the fix be

amd ryzen – Is it possible to configure a SATA RAID 1 hardware array on X570 without losing data?

So i basically got a 2nd identical SATA HDD to the one i use to store my bulk data (note OS/system drive is on a different nvme m.2 sdd).

Is it possible to configure a RAID 1 array with the new HDD on X570 bios without losing the existing data on the current HDD?

Kind regards

lvm – Add disk to software raid pool

I have a setup like this there run a mirroring software raid1 but its possible to add a disk more and use its like a raid5 or how do i add more storage to the raid?

Current with 2 disks

"https://serverfault.com/" /var | /usr | /home  |
 --------------------------
|       LVM Volume         |
 --------------------------
|       RAID Volume        |
 --------------------------
|    Disk 1  |   Disk 2    | 

New with 3 disks

"https://serverfault.com/" /var | /usr | /home  |
 --------------------------
|       LVM Volume         |
 --------------------------
|       RAID Volume        |
 --------------------------
| Disk 1 | Disk 2 | Disk 3 |

storage – Unisphere dynamic pools. Does this mean I can lose more than 1 disk with a raid 5 configuration and 20 drives or how are spare space extents treated?

So emc has implemented dynamic pools which do away with hot spares and instead uses some drive extents as spare space on each drive.

When a drive fails, the contents are copied into this spare space (is it usable living on this spare space?).

When setting up a traditional system, you would normally have a couple hot spares so if a drive fails, it rebuilds onto these spares thus saving you a failure. (Raid 5 with a couple hot spares allows you to lose 3 drives at seperate times i believe).

With this dynamic pool, how does this work? If I have a 20 drive system and use raid 5 on a dynamic pool…what number of drives can I lose (at different times of course) and still be operational?