Who and why needs it? (nonraid or unraid, whatever)

[Kamillaova]

[qvr]

I think this HN comment gives a good overview of how unraid differs from traditional raid implementations:

md takes multiple partitions to make a virtual device you can put a file system on, with striping and traditional RAID levels
unRaid takes multiple partitions, dedicates one or two of them to parity, and hands the other partitions through. You can handle those normally, putting different file systems on different partitions in the same array and treating them as completely separate file systems that happen to be protected by the same parity drives

This enables you to easily mix drives of different sizes (as long as the parity drives are at least as large as the largest data partition), add, remove or upgrade drives with relative ease, and means that every read operation only goes to one drive, and writes to that drive plus the parity drives. Depending on how you organize your files you can have drives that are basically never on, while in an md array every drive is used for basically every read or write.

The disadvantages are that you lose out on the performance advantages of a RAID, and that the raid only really protects against losing entire disks. You can't easily repair single blocks the way a zfs RAID could. Also you have a number of file systems you have to balance (which unRaid helps you with, but I don't know how much of that is in this module)

As for who and why of this project itself, at this stage it is for advanced DIY enthusiasts who want to try out the array tech without all the bells and whistles the commercial unRAID OS provides.

[jsebean]

One really clever thing I'd like to try out:
Btrfs isn't really known for having a reliable RAID5/6. It's has some issues that go beyond just the write hole issue (which this also has but isn't much of an issue if you resync after a crash). I'm planning to put a btrfs multidisk filesystem but rebalanced to raid0 for the performance benefits of striping across mixed size drives, and then put all of those member disks into an array with this to provide parity protection. Gives the benefits of striping speed for reads, having a single filesystem to hold the data so you can make atomic snapshots, without relying on actual btrfs redundancy code.

[qvr]

That's an interesting experiment at least!

You're of course giving up on one of the nonraid/unraid benefits: if you lose more disks than parity allows, the whole array is then gone if the disks were used to form an raid0 array.

The default read-modify write algorithm isn't probably optimal if writes are striped over all disks, so you might want to enable reconstruct write: nmdctl set md_write_method 1

I'm also not sure if there are other driver level issues that would make this a Bad Idea, but technically it seems to work though, tested it quickly over 2 disks:

# mkfs.btrfs -f -m raid0 -d raid0 /dev/nmd2p1 /dev/nmd4p1
# btrfs fi show
Label: none  uuid: a581ab53-dca5-47f3-8bc4-58e4bf7eee39
        Total devices 2 FS bytes used 144.00KiB
        devid    1 size 10.00GiB used 1.13GiB path /dev/nmd2p1
        devid    2 size 10.00GiB used 1.13GiB path /dev/nmd4p1
# df -h /mnt/disk2
Filesystem      Size  Used Avail Use% Mounted on
/dev/nmd2p1      20G  5.7M   20G   1% /mnt/disk2

And nmdctl mount still even works, although kinda by accident, because the raid0 btrfs array gets mounted if you try to mount just one of the member disks.

But yeah, you'd be using the driver in a way no-one else ever has (on unraid side), so... have backups. 😅

[rangerco1]

I would be interested in the results of your testing when you get it setup. This sounds like a very interesting use case. I'm wondering how well it would perform when a disk is lost and the system is emulating the missing disk from parity.

[jsebean]

That's an interesting experiment at least!

You're of course giving up on one of the nonraid/unraid benefits: if you lose more disks than parity allows, the whole array is then gone if the disks were used to form an raid0 array.

This is fine by me as you get striping and parity protection which btrfs doesn't do well on it's own.

The default read-modify write algorithm isn't probably optimal if writes are striped over all disks, so you might want to enable reconstruct write: nmdctl set md_write_method 1

This will make writes faster, however if you do have bitrot, RMW instead of turbo write would be safer. Btrfs itself in raid5/6 mode already does RMW. Also reads should still benefit from striping even in RMW. Up to the user to decide which is better, i'd be willing to accept the slow write speeds.

I'm also not sure if there are other driver level issues that would make this a Bad Idea, but technically it seems to work though, tested it quickly over 2 disks:

# mkfs.btrfs -f -m raid0 -d raid0 /dev/nmd2p1 /dev/nmd4p1
# btrfs fi show
Label: none  uuid: a581ab53-dca5-47f3-8bc4-58e4bf7eee39
        Total devices 2 FS bytes used 144.00KiB
        devid    1 size 10.00GiB used 1.13GiB path /dev/nmd2p1
        devid    2 size 10.00GiB used 1.13GiB path /dev/nmd4p1
# df -h /mnt/disk2
Filesystem      Size  Used Avail Use% Mounted on
/dev/nmd2p1      20G  5.7M   20G   1% /mnt/disk2

And nmdctl mount still even works, although kinda by accident, because the raid0 btrfs array gets mounted if you try to mount just one of the member disks.

But yeah, you'd be using the driver in a way no-one else ever has (on unraid side), so... have backups. 😅

There are a few gotchas I'd consider. First is I'm not sure if the nonraid driver changes things, but I could see /dev/nmd* devices colliding with /dev/sd* devices since they still have the same UUID. Do you get any output in dmesg about btrfs complaining about this? I'm guessing since you mkfs the filesystem after starting the array you don't, but if you run btrfs device scan again any time afterwards when mounting/unmounting you're going to run into a problem.
Any time a block device is detected by systemd/udev it automatically runs btrfs device scan so the /dev/sd* devices and MD devices could cause issues I would think. Best to disable the udev rule to run btrfs device scan, and when running device scan, do it on a per device basis. ie btrfs device scan /dev/nmd2p1 since when specifying a device it doesn't just scan all block devices but just the one you specified so the kernel remembers it.

Alternatively you could also just never use btrfs device scan ever, disable the udev rule, and when mounting a multi-device btrfs filesystem you would specify each device manually in the mount options using device= (iirc you just specify this option over and over for each device in the filesystem). The default udev rule on most systems should be /usr/lib/udev/rules.d/64-btrfs.rules, but ofc, don't overwrite this file as the package manager handles it. You can override it in the appropriate /etc directory.

Anyway something worth playing around with :)

Anyway I have done this on unraid itself before lol. it doesn't act nicely though with unraid as it wants to format the disks independently even though they are part of a single btrfs filesystem.

[USBhost]

Btrfs has single raid. It simply writes in one gig chunks (or something like that) throughout each disk without being striped. In this case, I think the normal algorithm will be just fine. The only problem is I wonder how it'll handle RAID 1 metadata.

[jsebean]

Btrfs has single raid. It simply writes in one gig chunks (or something like that) throughout each disk without being striped. In this case, I think the normal algorithm will be just fine. The only problem is I wonder how it'll handle RAID 1 metadata.

It depends which profile you set. If you're using nonraid for redundancy, you don't need to use raid other than raid0 for extra read performance for the filesystem. I don't see much sense in layering raid1 with nonraid other than perhaps metadata to help mitigate fully losing the filesystem due to the write hole (not to be confused with btrfs raid5/6 writehole, but the nonraid write hole, which non-raid has as parity writes are not atomic).