HDD performance decreasing a lot

[Massimo-B]

Hi, I'm running several single device btrfs from 1TB up to 4TB, most are SSD, NVMe, some are 1TB HDDs.
I use btrbk a lot to store snapshots to other btrfs. I'm running bees on all btrfs, especially on the bigger btrfs for deduping snapshots from different machines.

I plan to get one single 20TB HDD for NAS data, server data and also all the snapshots from other machines centralized on this big btrfs.

But as all non-rotating SSDs are fine, the rotating HDDs performance are seriously decreasing. Most probably due to fragmentation of extents? Usually I can measure fragmentation only per file and I did not find any serious high fragmentation. How could I analyze the overall fragmentation of the whole btrfs?

For instance:
My 4TB NVMe receiving snapshots currently currently has 441 subvolumes. btrfs filesystem usage -g says 1759GiB free, min 960GiB. Runs fine.
The 1TB HDD has 74 subvolumes, 200GiB free. But running btrbk, transferring ~100MB takes 10 to 20 minutes and doing a noisy lot of IO. bees is almost never finishing even after days.

The planned 20TB HDD would be connected to a 24/7 thin-client with small CPU and only 8 GB of RAM. Due to the size there could be around 800 subvolumes over time... I doubt that will work if I'm already struggling with the small 1TB disk. I'm quite sure that bees is responsible for the degraded performance. But without bees I would never be able to dedupe snapshots received from different machines.

[kakra]

On COW filesystems, it's usually only useful to measure free space fragmentation because you cannot avoid to fragment files - it's a feature of COW to do exactly that. I'm not sure if free space fragmentation can be measured in btrfs, but it can be optimized by using tools like btrfs-balance-least-used with a parameter like -u80 to let it recombine chunks until each chunk is filled at least 80%. It starts by optimizing the least used chunks first, then walks it way up to the usage limit or 100% (if you don't set a usage limit).

See: https://github.com/knorrie/python-btrfs (maybe this toolset has a utility to measure fragmentation)

Optimizing free space fragmentation should improve write performance because the HDD heads tend to move less. Remember: COW always writes data to unused space.

[Zygo]

Is this with v0.10 or master? See #296. Things have improved a lot recently, particularly for handling snapshots and reducing fragmentation.

[Massimo-B]

Wow, highly interesting. Thanks for the huge amount of development @Zygo and testing by @kakra only 1 week after the initial comment and RC.
Going to try that soon on my old btrfs with many bees versions working over the years.

But already existing fragmentation would not be healed by new extent scan mode?

[kakra]

If I understood correctly, bees will not decrease fragmentation, but due to the way the new extent scanner works, it avoids a lot of additional fragmentation that the old scanner introduced. It will still split extents and thus fragment some files but the impact is greatly reduced. To use it, switch to scan mode 4 (or remove the parameter from your script, it's the default).

[Massimo-B]

As I'm building bees on Gentoo with version (**)9999 taking the latest repository files, how can I see which version I currently have?
/usr/libexec/bees and /usr/sbin/beesd both don't know a --version option.

[Zygo]

It's the first line of output:

# bees
bees version v0.10-33617-gab288ff84

It also appears in beesstats.txt:

Now:     2024-12-11-02-29-22
Uptime:  93379 seconds
Version: v0.10-33617-gab288ff84

[kakra]

As I'm building bees on Gentoo with version (**)9999 taking the latest repository files

I'm maintaining that ebuild. The 9999 version builds from git master. It should have told you about the new scan mode at the start of the build, then you're using the latest build.

Something like bees version v0.10-57-ge403398 if you run beesd --help, meaning it's 57 commits ahead of v0.10.

[Massimo-B]

Today going to start v0.10-57-ge403398-dirty on my problematic 1TB HDD. First tests on SSDs have been successful.

Before I would like to optimize the free space fragmentation:

btrfs-balance-least-used with a parameter like -u80

How would I do that?
btrfs balance start -dusage=80 -musage=80 /mnt/btrfs ?

Wouldn't it be
btrfs balance start -dusage=0 -musage=0 /mnt/btrfs
in order to defrag chunks that are 0% used which is the free space?

[Zygo]

There is a utility in the python3-btrfs package called btrfs-balance-least-used which will target the lowest-usage block groups first, until the limit specified by -u is reached:
btrfs-balance-least-used -u 80

You can also use
btrfs balance start -dusage=80 /mnt/btrfs

Never use -musage or balance metadata when not converting to a different profile. It will delete existing metadata allocations and may allow the filesystem to run out of metadata space.

Generally it's better to run balance after dedupe, since the dedupe will leave behind a lot of random free space fragments. Balancing before dedupe will move the data around before the free space appears.

Balancing also requires bees to scan all the data again to keep the locations in the hash table up to date. Minimizing balance using the usage filter keeps the amount of rescanned data low.

[kakra]

The knorrie version does a better job than plain btrfs balance because it frees least used chunks first, possibly filling other considered chunks to the max so it can skip those later in the process.

@Massimo-B, see above, I posted this:

See: https://github.com/knorrie/python-btrfs (maybe this toolset has a utility to measure fragmentation)

[Massimo-B]

You can also use btrfs balance start -dusage=80 /mnt/btrfs

Never use -musage or balance metadata when not converting to a different profile. It will delete existing metadata allocations and may allow the filesystem to run out of metadata space.

Too late, I already did
btrfs balance start -dusage=80 -musage=80 /mnt/btrfs
on both my SSD/NVMe btrfs and also the HDD.

My knowledge about the internals of btrfs are insufficient. I tried learning from #btrfs on Libera for not polluting tickets here with basic questions.

From what I understood, please correct if I'm wrong: btrfs organizes in chunks and extents. Chunks are physical organisation, extents are logical. Chunks are ~1GiB for data, extents are 4KiB up to several MiB. Chunks contain extents. Chunks are not necessarily contiguous and can be physically fragmented. Extents extents can be fragmented over chunks. Files can be fragmented over extents.

Now what does balance do? Reading man btrfs-balance, I'm confused as it first describes "block groups" for -d[<filters>]. Later it mentions "chunks" for -m[<filters>]. Is that the same thing, different wording?
At least on section FILTERS it mentioned block groups again for both data and metadata, so this would be worth a rewrite of the man-page.

Balance does spread chunks over devices. What it actually does on single devices is not that clear. Section EXAMPLES->MAKING BLOCK GROUP LAYOUT MORE COMPACT says, it takes partially filled chunks and move the extents to other chunks, then removing the empty chunks? That means the extents of a file is not changed, but extents are moved from partially filled chunks for filling existing chunks to have a higher usage? This does defragment the fragmentation of extents over chunks?
Reading further, what does that mean about the usable space? It feels like balance leaves unused chunks, which then must be removed by -dusage=0? So after -dusage=50 and -dusage=85 I need another -dusage=0 to finish? Confusing.

So why do you say "never use -musage"? In the way of -musage=50, -musage=85 the compacting done during the balancing could be run out of free space until the -musage=0 is able to free the empty chunks?

Generally, the balancing on HDD I need to get rid of free space fragmentation. Would I need balancing on SSD/NVMe at all?

What does the knorrie version do better? Is it just the order of processing because it sorts for least used chunks first? Some -dusage=10 and -dusage=20 would also only consider less used chunks and moves their extents to higher used chunks, maybe not sorted.

[lilydjwg]

I need another -dusage=0 to finish?

No.

So why do you say "never use -musage"?

It makes metadata more compact, and thus more likely it want to allocate a new block group metadata---if it's not possible, you may be stuck since deleting things requires updating metadata.

Would I need balancing on SSD/NVMe at all?

I recommend you balance when you have little unallocated space (but still a lot of unused space inside block groups), so when a new metadata block group is needed, you have that space.

My backup HDD performance also decreased quite a lot, but I think it's due to bees "deduplicated" shared data between snapshots and so unshared its metadata.

[kakra]

What does the knorrie version do better? Is it just the order of processing because it sorts for least used chunks first?

It's the order which helps finishing the process faster with less IO pressure.

[Zygo]

The loss of performance is concerning.

Dedupe will leave behind a lot of small free space holes and btrfs isn't great at filling small free space holes. Balance won't fix that--it allocates on...64K? (power-of-2? minimum-length?) boundaries, so the new block groups created during balance are full of tiny holes too. Regular writes do a second allocation pass that eventually fills in the smaller holes so allocation gets fast again, but I don't know of any way to speed that process up.

Unsharing snapshot metadata can also cause a slowdown--or a speedup, at random--because it churns all the metadata and redistributes it across the disk surface. That will also randomly get faster or slower over time with normal writes, as they'll move metadata pages that are modified by normal day-to-day writes closer together and reduce seek time.

---

A dev_extent is a contiguous region of physical space on one device. A chunk is one or more dev_extents organized in a RAID profile. A block group is a chunk with a free space counter. Block groups and chunks are very similar, and the terms are used interchangeably in cases where the presence or absence of the free space counter is not important.

An extent is a contiguous region of logical space in a block group. A file is a list of references to extents, indexed by offset from the beginning of the file. Extents are always contiguous--one fragment is one extent by definition. Extents are created when writing to a file, and can be smaller than the written region if the written region exceeds the maximum size for an extent, or if the largest free space available in any block group is smaller than the written region.

An extent cannot be modified once it is written. Extent reference lists in files can be modified--overwriting part of an extent will result in the reference to the old extent being shortened or split, and a new reference to the new extent will be inserted into the file's extent reference list. nodatacow files are an exception to this rule, but only if the extent that is overwritten is not shared. prealloc files have extent references that can be overwritten exactly once.

Balance moves extents from one block group to others, possibly creating new block groups in the process, or filling in empty space in existing block groups. Each block group is deleted after its contents have been relocated. The product of balance is the free space left behind by the deleted block groups, e.g.

1. if you want more space for metadata, you balance data block groups, and hope more are deleted than created.
2. If you want to expand one drive (replace a small one with a large one) in a 5-drive raid1 filesystem, you balance data block groups on devices other than the one that was expanded, creating equal amounts of free space on the non-expanded drives.
3. If you want less space for metadata, you balance metadata block groups to delete them. Never do this unless you need to delete all the metadata block groups for some reason (e.g. removing a drive from a filesystem, converting to a different metadata profile, or to fix severe dev_extent fragmentation that results from btrfs-convert).

Almost every use case for balance requires a script to balance block groups in the optimal order for that use case. btrfs-balance-least-used is designed for example 1 above. There should be five or six such scripts by now for use cases like "expand one drive in N>2 disk raid1" or "defragment dev_extents after btrfs-convert", but nobody seems to get around to writing and publishing them (myself included).

Extents cannot cross block group boundaries. Balance cannot change the length of any extent. This means balance cannot necessarily fill block groups to 100%. Balance on a filesystem that is over about 90% full (the point where problems start varies from 84% to 99.5% depending on what's on the filesystem) can sometimes run out of space because of these limitations.

Defrag copies data to a new extent, and asks the allocator to allocate the largest extents possible (or exactly the size of the file, if that's smaller). This request may or may not be honored, and in the latter case, defrag will replace a large extent with multiple smaller ones.