inspect-internal list-chunks does not show usage on RAID-6

[rzsn]

Hello,
I am trying to figure out where I got the 700GB of unusable free space on my FS (basically the 0.71% of a 98TiB array):

Overall:
    Device size:                 130.97TiB
    Device allocated:            130.93TiB
    Device unallocated:           40.11GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        129.98TiB
    Free (estimated):            738.63GiB      (min: 721.91GiB)
    Free (statfs, df):           738.59GiB
    Data ratio:                       1.33
    Metadata ratio:                   3.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID6: Size:97.85TiB, Used:97.16TiB (99.29%)

but the suggested list-chunks is not really helpful:

Devid PNumber       Type/profile     PStart     Length       PEnd LNumber     LStart Usage%
----- ------- ------------------ ---------- ---------- ---------- ------- ---------- ------
    1       1        Data/RAID6     1.00MiB    6.00GiB    6.00GiB   14771  119.46TiB   0.00
    1       2        Data/RAID6     1.00GiB    6.00GiB    7.00GiB       1   37.03GiB   0.00
    1       3        Data/RAID6     2.00GiB    6.00GiB    8.00GiB       2   43.03GiB   0.00
    1       4        Data/RAID6     3.00GiB    6.00GiB    9.00GiB       3   49.03GiB   0.00
    1       5        Data/RAID6     4.00GiB    6.00GiB   10.00GiB       4   55.03GiB   0.00
    1       6        Data/RAID6     5.00GiB    6.00GiB   11.00GiB       5   61.03GiB   0.00
    1       7        Data/RAID6     6.00GiB    6.00GiB   12.00GiB       6   67.03GiB   0.00
    1       8        Data/RAID6     7.00GiB    6.00GiB   13.00GiB       7   73.03GiB   0.00
    1       9        Data/RAID6     8.00GiB    6.00GiB   14.00GiB       8   79.03GiB   0.00
    1      10        Data/RAID6     9.00GiB    6.00GiB   15.00GiB       9   85.03GiB   0.00

Can the list-chunks be fixed, or is there another command that works on the logical side of the RAID functionality ?

# btrfs version
btrfs-progs v6.17
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD -UDEV +FSVERITY +ZONED CRYPTO=builtin

# uname -a
Linux ******.***.*******.*** 6.17.6-gentoo-x86_64 #1 SMP PREEMPT_DYNAMIC Sat Nov  1 12:43:31 CET 2025 x86_64 Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz GenuineIntel GNU/Linux

Tried balacing up to 95 but no unallocated space was recovered, however given the figures above the only option is to balance for me is to go straight with 100 - which is sort of a full defragmentation I hope. But I am unable to track any progress due to lack of the usage metrics in the list-chunks for my array.

I am really having hard time understanding what is going on, as the FS keeps DoS-sing me, after I deplete the unallocated portion - further writes go with huge disk activity and at single kB/s of speeds which tells me something is really wrong here.

Does BTRFS have a RMW process in a RAID data chunk, or its doing a SMR-like append-only operation, and the whole 6GiB slice of 8x1GiB chunks has to be copied over in order to be compacted, and allow further writes when the volume is that much full?

I would then really appreciate if the FREE space indicator is split into two distinct metrics - one for TRUE FREE (virgin untouched tail of the chunk) and a FALSE FREE (dirty CoW affected area that is no longer used, but its also not free to direct use and requires rebalance / defragmentation).

Further info:

/dev/***1 on /mnt/******** type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)

And one strange things is that I am getting thousands of such warnings on stderr before list-chunks produces anything:

WARNING: blockgroup 135151418867712 not found
WARNING: blockgroup 135157861318656 not found
WARNING: blockgroup 135164303769600 not found
WARNING: blockgroup 135170746220544 not found

[Forza-tng]

Hi, likely some hidden/not visible snapshots or reflinks.

Try listing all.subvolumes with btrfs sub list /mountpoint

Another very useful tool to quickly find where your 'missing' space is btdu. It will sample your fs and give you a good estimate within a minute of launching. A guide is available at https://wiki.tnonline.net/w/Btrfs/btdu

Does BTRFS have a RMW process in a RAID data chunk

Yes, the btrfs raid56 profileu ses RMW to update its striped. Note that this profile is experimental and not recommended to use for important data. There are important caveats when running in degraded mode or if you have had a crash or power loss.

[Zygo]

A few observations/confirmations:

1. btrfs balance usage filter does not go to 100% - it goes to 99.5% due to rounding. The only way to balance block groups between 99.5% and 100% is to write a python-btrfs script to identify the desired block groups by vaddr.
2. Balance cannot change extent length, so it's not possible to balance once the filesystem has no free spaces of sufficient length to relocate all of the extents within any block group. Practically, this means balance is impossible somewhere between 90-100% full, but the exact threshold where failures start to occur varies by filesystem.
3. The allocator runs in two passes: the first tries to align allocations on N*64K boundaries and allocate without fragmentation, the second tries to allocate again with no restrictions. For non-raid56 profiles, N = 1, but for raid56 profiles N = nr_stripes. This increases the number of small holes and impacts performance on raid56 profiles.
4. btrfs raid56 still has RMW in btrfs for datacow files. This is why the write hole still exists. It causes performance problems as the filesystem fills up, because free space is fragmented more, resulting in more RMW operations. This is an extra separate layer of slowness specific to RAID56, in addition to the slow allocation that occurs in other profiles.
5. In all profiles, the allocator runs in multiple "sweeps" across the filesystem, first filling in the majority of the holes, then, much more slowly, filling in the remainder. This results in a lot of small, isolated writes, which are detrimental to performance. This affects RAID1 and other profiles too, but the effect is worse on RAID56 because of points 3 and 4.
6. At its slowest, the allocator burns a lot of CPU running some algorithm on a single core. The CPU usage becomes the limiting factor when the allocation algorithm is at its peak of poor performance. The point at which this occurs is not only filesystem-dependent, but dependent on recent filesystem history: deleting snapshots or balancing chunks will switch the allocator between clustered and unclustered allocation, and make fast-allocation space available for a while, before falling back to slow-allocation space.
7. There's a lot of confusion between zoned behavior and non-zoned behavior in the OP. The OP's filesystem is raid6, so everything is necessarily non-zoned.
8. Forza might have intended to post that reply to another issue? It seems completely unrelated.
9. list-chunks provides plausible output when I run it, showing chunks between 98 and 99% full on a random raid5 filesystem I tested it with. I don't know why that's not working for OP.

An example of a very poor workload for point 6 is a snapshot rotation scheme based on free space. By deleting snapshots at a specific free space threshold, the filesystem becomes "trained" to slow down at that threshold--the deletion will create fast-allocation space that is promptly filled in by whatever is next written to the filesystem, while leaving slow-allocation space behind. This results in all of the available free space becoming slow-allocation space due to constant adverse selection.

So you can make a reproducibly slow filesystem at any free space threshold you like between about 92% and 99%, by simply repeating snapshot creation (e.g. from backups) and deletion (as long as it uses the same free space threshold each time). Adding some hysteresis (e.g. start deletes at N%, stop deletes at N+1% free) provides longer periods of usable speed in between the slowdowns, but does not ultimately solve the problem unless N >= 10% free.

Balancing does avoid the problem, especially when filtering by usage (with btrfs-balance-least-used or /sys/fs/btrfs/*/allocation/data/bg_reclaim_threshold), because it forces all extents through a variant of the fast allocator, resulting in large contiguous regions of allocated data where every free space hole is too small for fast allocation, and large contiguous regions of free space which can be allocated quickly; however, balancing is only possible when the amount of free space exceeds the minimum threshold for balance to be able to run at all. That threshold is higher than the slow-allocation threshold, at somewhere between 8-15%.

Note that I mention snapshot creation and deletion here because they are common use cases, but the effect is intrinsic to all processes that write and delete data in a btrfs filesystem. You can also get it with rsync and dedupe, or CI build/clean cycles.

There's no pipelining of the RAID56 RMW operations, so they all run sequentially. This could be fixed by doing a SMR-like reclaim operation instead of RMW (which also fixes the raid56 write hole), but btrfs devs so far have flatly refused to implement that. The drawback of SMR-style is that the filesystem in this state would have up to 15% less usable space because slow-allocation space would be entirely unusable, and there has to be enough free space for balance to work.

Removing the special case in the allocator for raid56 doesn't help (it was something I tried as soon as I found it). It slightly changes the point where the first allocator pass (clustered allocation) gives up and the second allocator pass (unclustered allocation) starts; however, by the time the second allocator kicks in, fragmentation of free space holes is so severe that there's just no way to write data at anywhere close to full speed. Spinning drives make this worse, but it's still bad on single-device NVMe SSD too. For raid56 in particular, the allocator spends a greater fraction of its time in the slow state, since allocations shorter than the raid56 stripe size always result in RMW.

The practical workaround I use is to keep the filesystem below 90% full. If maintenance data balances (which we run daily) return ENOSPC due to lack of space, we immediately trigger snapshot deletes, instead of waiting for the 99% threshold. to be reached. That keeps the filesystems in my care around the 92%-94% mark.

[rzsn]

Thanks a lot for the in-depth explanation of the internal workings. Really appreciate that. It now makes sense of what I experience. And also that rounding thing - finally wont chase a ghost here for that last 0.5% !

So it seems it would be better to leave the R6 task on mdraid (given that I have a quite a full array - there is zero benefit of having the parity only on used data when practically everything is used, and also I use same size of drives). Reading the above - may I expect a bit better performance if we avoid the RAID56 code paths? Do I still need to keep metadata duplicated (for recovery/corruption) when the underlaying device will be able to cope with 1-2 disk failures?

There are no snapshots (yet) - only subvolumes - mostly of directories at first and second level.. so 144 subvolumes for easier handling of my data bundles. The CoW wasted space is a result of some activity, eg. git clone/checkouts, video stream split & remux. Judging by the size, I only "changed" the 0.7% of data, the rest is write once / RO data.

One specific of this array is also that it was imported by a send/recv method (by a script/loop over all the snapshots that were from the old array), and then the snapshots were modified into a RW subvolume. I do not expect that it would behave any different due to that way of filling the array.

For the original inquiry - how can we determine why it is not showing the percentage?
(some other dump methods? offline/online)

[Forza-tng]

Forza might have intended to post that reply to another issue? It seems completely unrelated.

I guess I am the one confused....Sorry 🙏

[msmucr]

Hi, just stumbled upon this issue and there seems to be two problems.
The first one is performance under described scenario with very low free space.

The second one is per chunk usage reports from inspect list-chunks with zero value. I just tried to quickly simulate it (Fedora, kernel/btrfs-progs 6.17) and at first look it doesn't seem to be related to RAID6 profile.
The feature, which breaks the usage calculation and reports is BLOCK_GROUP_TREE. Without that feature enabled it seems to report usage correctly. So assuming the OP has this feature enabled (check: /sys/fs/btrfs//features/ or superblock).

Just for my quick comparison, I've used python-btrfs with similar per chunk reports.
https://github.com/knorrie/python-btrfs
eg. btrfs-search-metadata block_groups
It lists usages even with BLOCK_GROUP_TREE enabled.
Although I need to exactly verify its output, when using RAID1 profile for data, it seemed bit off, but it would need some more tests.

[rzsn]

Yes, while that option is not globally enabled:

# cat /sys/fs/btrfs/features/block_group_tree
0

it is enabled on this particular FS instance:

# cat /sys/fs/btrfs/*/features/block_group_tree
1

The Python tools produced nonzero percentages, but they seem to be integer. Histogram on my particular FS is:

100:13387
99:1164
98:1099
97:436
96:133
95:319
94:151
93:52
92:28
91:16
90:13
89:18
88:10
87:5
86:4
84:2
83:3
82:1
81:3
80:1
79:2
77:1
75:1
70:1
68:2
60:1
59:1
39:1
32:1
17:1
0:1

Would be nice to have the usage shown as fractional/decimal :)

So - I am confirming the issue with BLOCK_GROUP_TREE on btrfs inspect-internal list-chunks.
(have not found a issue number here for this particular todo task/item)

[Zygo]

Reading the above - may I expect a bit better performance if we avoid the RAID56 code paths?

Theoretically yes; in practice there might not be much difference. There are many trade-offs:

1. you're gaining speed by replacing slow parts of btrfs with mdadm
2. you're losing speed because btrfs did have a few optimizations for raid56, and they aren't available when btrfs is presented with a single-profile device
3. there's no difference on discontiguous short extent writes because they will trigger RMW cycles in either implementation of raid56
4. at 99.3%, you're already past the threshold for switching from fast to slow allocation on non-raid56 filesystems, so you'd have the same problem on a non-raid56 profile.

You'll get a much bigger performance boost by monitoring unallocated space and reserving enough space to allow balance to make unallocated regions (that's usually 5-10%, but I've pushed it as low as 0.9% with careful monitoring and some tolerance for short bursts of high write latency). Although you can push btrfs past that point for a while, the allocator always claims its debts in the end.

Do I still need to keep metadata duplicated (for recovery/corruption) when the underlaying device will be able to cope with 1-2 disk failures?

mdadm can only cope with device-reported read failures (where the device gives a definitive IO error status). All other paths to corruption (dropped writes, firmware bugs, write hole, etc) aren't handled by mdadm and will result in unrecoverable csum errors in btrfs. You'll still need redundancy in the btrfs metadata (i.e. -mdup) to survive real-world disk failures.

In degraded mode, there might be no working solution for safe writes without a journal device. Without a journal device, in degraded mode most crashes or power failures during a metadata write will lead to stripe corruption, which would unrecoverably damage the metadata tree, even in dup mode. At best, the filesystem would no longer be mountable read-write; at worst, tree roots and interior pages would be damaged, leading to loss of all data.

Current guidance is to never use raid56 profile for metadata, and that includes any non-journalled raid56 implementation in the block layers underneath btrfs.