RAIDZ Expansion feature discussion

[justinclift]

This is carrying on from PR #15022, so we don't crowd the PR out with general discussion traffic. 😄

[yorickdowne]

So to summarize that: Afaik, a copy command will rewrite the data with the new width after expansion. I think I saw a script sloshing around that basically does that for an entire array, cleaning up after itself. Does it file by file for space reasons. Takes a minute, but, you know. Only have to do it once.

From my perspective that’s plenty good enough. Raid expansion is a niche feature, if much beloved by that niche. Adjusting width is then a niche within the niche. Handling that with a community script sounds like an entirely reasonable thing to do. Permissions and paths stay as is, filenames stay, last accessed and creation date changes and width changes.

To explain the niche comments: I am part of that niche. Home user but without the funds to start at full width. I did a poor man’s raid expansion eventually, from 5 to 8. So the niche are people who start at a smaller width than they know they’ll need, and will only run one vdev. By definition home users, and then only a fraction of those.

The niche within the niche are users where rewriting archival data makes an appreciable difference, or that care even though it doesn’t.

That need not be handled in the file system. Handling it with good UX outside seems great. Setups that use ZFS like TrueNAS and Proxmox could even integrate such a script into their UI.

[IvanVolosyuk]

@TurkeyMan
The 'assumed parity' is explained here by Matthew Ahrens after this #12225 (comment)

[Dulanic]

I'm part of that niche crowd, and I'm cool with it. If I decide to expand, I'm fine with keeping the old setup until it gets a rewrite. Storage is pretty cheap these days, which is why I'm looking to expand. That being said, I think my use case is decent. Right now, I've got two vdevs, but one of them has a bunch of older, 8TB drives. It just doesn't make sense to swap an 8TB with another 8TB when a 18TB is way cheaper per TB. As such, it just makes much more sense to expand my current vdev with 18TB drives and end up removing my old vdevof 8TB drives.

So, I much follow the path that you did.

With that said, when I remove my 2nd vdev, my understanding is zfs will shift my data over to the 1 remaining vdev. If this code/process exists, couldn't that process be called, albeit maybe modified, to balance it out again when expanding? Maybe not since its just moving the data from vdev to vdev vs adding a drive to expand which I guess would be a very different flow. This is where I may be a bit lost.

[IvanVolosyuk]

You can swap individually each 8TB with 18TB drives and once all of them are 18TB you can expand them to occupy entire space of the new drives.

[nonlinearsugar]

This script, with caveats (read the readme), helps automate rewriting your data by copying/pasting/deleting/moving at the filesystem level.

https://github.com/markusressel/zfs-inplace-rebalancing

[Dulanic]

FYI, Link is bad - goes to https://github.com/openzfs/zfs/discussions/url

[nonlinearsugar]

Fixed. Thanks.

[the8472]

The script doesn't use direct IO which means the data might end up sitting in caches and you're only hashing stuff from the page cache, not reading back what makes it to disk. I think this leaves a small window for data corruption, especially on systems without ECC memory.

[ahrens]

This is cool, and I think solves the problem well if you don't have snapshots.

If you wanted to preserve space sharing between snapshots and clones, you could use zfs send to copy entire "clone families" (all filesystems and snapshots related by cloning and snapshotting). The downside would be that you'd need enough free space for another copy of your largest "clone family", rather than just your largest file. The zfs send approach would also work with zvols.

[yorickdowne]

@Dulanic you can’t removed a raidz vdev from a pool. Mirror vdevs only (single disk vdev is special case of mirror) can be removed.

You may be able to create a new pool with your 18TB drives, copy data over, then expand that. This really depends on how much you’re using already though. This is why a lot of home users stick with one vdev, lest they are “stuck” with two to manage.

I have one 8x8 raidz2 which will likely last me forever. But in case it ever doesn’t, I intend to replace drives as they fail with larger-capacity ones. Once the last drive has been replaced, I’ll have more space.

[Dulanic]

Oh no! Lol I am still learning! That's a problem lol I have 80TB already! If I have 2 vdevs can I expand 1 if them? I wasn't sure if I needed to match the width.

[TurkeyMan]

My current take on this, the stripe-width expansion seems less of an urgent issue than the fact it only adds one drive at a time.
As advertised, at 50mb/s, it will take 4 days to expand my vdev. I can handle it taking 4 days I suppose, but the real problem is it can only expand one drive at a time. So, expanding 4-wide raidz2 to 8-wide, that'll take 2 or 3 weeks... and it's not clear how much the standard operating performance may be degraded during that time? Is there any additional risk from drive failure during that time?
It's a very long timeframe to expand a vdev.
It seems that this feature really needs to allow adding multiple drives at once...

[yorickdowne]

Unsurprisingly people have been making wider raidz2 vdevs for years, even without official expansion support. The “use sparse files for parity and offline them” trick has gotten good mileage :).

[ahrens]

FYI: Time to expand is limited by CPU usage. Improving it by 10x would be a relatively straightforward follow-on project. Also note that the 50MB/s that I measured was with 512-byte sector size. With 4K sector size it would be around 8x that (~400MB/s).

[justinclift]

Improving it by 10x would be a relatively straightforward follow-on project.

Is that something likely to be done in the near-ish future? 😄

[darkbasic]

Improving it by 10x would be a relatively straightforward follow-on project

I think that would be make raidz expansion much more viable.

[ahrens]

Is that something likely to be done in the near-ish future? 😄

Not by me, but PR's are welcome 😄

[GrayXu]

Is RAIDZ "contraction" support included in this PR? 🤔

[GrayXu]

Reasonable.
But since not all data in a zpool requires the same level of protection, I think the ultimate solution should allow different datasets to have different levels of parity protection. Like in the existing RAIDZ expansion, not all data has the same number of parities for protection.

[yorickdowne]

I’d really rather not see any such thing. ZFS is wondrously complex as is. Per-dataset parity adds another level of complexity for no gain that I can see.

In this expansion PR, all data have the same parity in the expanded vdev. It’s the stripe width that changes, not the parity.

If I recall right there has been discussion of a follow-on PR that’d expand (but not contract) parity, such as from raidz1 to raidz2 or raidz3. That’d be separate work, and whether it will ever be tackled is a good question.

The existing mechanisms for changing pool layouts - make a new pool, and/or add vdevs - work for the use cases I can see. Raidz expansion is a lovely hobbyist use case and I am stoked it’s getting there; raidz parity expansion would be another hobbyist use case but even more niche.

[amotin]

The way expansion is implemented now the contraction is simply impossible. Even in unlikely case that end of the vdev is absolutely empty to be dropped, contraction would not produce sufficient level of redundancy to sustain disk failures.

[GrayXu]

@yorickdowne @amotin thanks for reply!

1. Regarding the issue of data reliability. The number of data dev and parity dev are both related to reliability, and expanding stripe width will also reduce system reliability. Similar topics are widely researched with different coding parameters in erasure coded systems.
2. Regarding different datasets having different levels of protection. Obviously, we can achieve different levels of protection through manual partitioning to different zpools, but this cannot achieve the effect of sharing the same storage space. I think it is reasonable to have different types of files requiring different levels of protection, such as both 3-way replicas and tape replicas in data centers. So I am particularly curious about the engineering challenges of this on ZFS.

[justinclift]

@GrayXu While some of that sounds like a potentially interesting discussion topic, it's probably best to create a new discussion for it. It's not very on-topic for this one. 😄

[yorickdowne]

To @farkeytron: Discussion is here, not in the PR. IxSystems is paying for this work, so it’ll definitely come into TrueNAS. SCALE first likely, since that is where canary features land, then Core in due time.

[kmoore134]

Lol, very appropriate meme :)

This feature will involve more middleware and UI changes to support properly, which means it will be targeted at SCALE only. CORE is mostly feature frozen, so we aren't making big sweeping changes to features / backend / front-end. SCALE is where active development resides.

[MihaMarkic]

"next major update to our ZFS base" What timeframe is this? Next major build after Cobia? Sorry if I've missed this info.

[kmoore134]

24.10 (Electric Eel - October release) is the next major TrueNAS SCALE release that will include ZFS 2.3.

[CBattershillDev]

Reading this makes me want to upgrade my core server to scale but migrating things like nextcloud configurations and data makes me concerned. Is it a certainty that this feature wouldn't make its way to core? (even if it was just for cli access)

[kmoore134]

That feature won't make it into 13.3, so odds are low it will land in CORE in the next 12-24 months. If you need a hand with migrating nextcloud we've had plenty of other folks do that. Might be worth asking questions here:

https://forums.truenas.com/

[KungFuJesus]

we have this issue with ZFS encryption support after Datto stopped contributing to it.

@bghira Is there more info on that around?

Asking because I'm using ZFS encryption on some systems, and hadn't heard mention of potential upcoming problems with it. Would like to know before problems potentially bite. 😄

Hah, I take you don't follow the issue tracking on github too carefully, then. The amount of issues I see with the encryption support (and not just performance problems, but like, swallowed my data or corrupted it on a replication ones) has made me pretty hesitant to recommend ZFS's native encryption support for anyone.

[justinclift]

Hah, I take you don't follow the issue tracking on github too carefully, then.

Yeah, not at all. I think this discussion and it's matching PR are the only things I'm subscribed to in the repo.

swallowed my data or corrupted it on a replication ones

Damn. Sounds like we'll need to investigate that, and potentially find an alternative to ZFS then. Or at least an alternative to native ZFS encryption. 😦

Can't be having unreliable storage backend for production systems. 🤦

[darkbasic]

I use encryption in production: it's not worse than many other features like zvol (#11407).
zfs is not perfect, but what's the alternative? dm-crypt + grub2 patched for argon2? I find it extremely irritating that without out of tree patches your system is as secure as one without encryption. If I have to sign my own bootloader (because I have to build grub from source) I'd rather sign a zfs module instead.

[justinclift]

@don-brady @ahrens Thank you SO much for getting the ZFS expansion PR across the line.

It'll likely be a key capability of ZFS from here on (in the next release onwards of course). 😄

[Sparklingx]

is the expansion option for all setups including stripe?

[mumbleskates]

it's for raidz vdevs specifically, which have 1-3 devices worth of parity.

[KaeTuuN]

This Feature is not for switching from stripe to raidz1-3!

[IvanVolosyuk]

Quick question: if I update to zfs master and enable corresponding feature flag for raidz expansion, will it be active or enabled after expansion completes? Thus, can I downgrade to stable branch after that or there will be some incompatibilities in on disk layout and I'm limited to master branch after that until the next stable release.

[amotin]

On-disk format after expansion is different from one supported before this feature. So I bet the feature stays active forever.

[ahrens]

That's correct. After expansion the feature will be "active", and you can't downgrade the software to an earlier version that doesn't support RAIDZ Expansion.

[emaste]

A couple of days ago @mmatuska updated OpenZFS in FreeBSD main, including RAIDZ Expansion. You can try this out with the next set of 15-CURRENT snapshot images.

freebsd/freebsd-src@e716630

[badmannen]

I tried to read through the DOCS here but still not sure if I can extend my 2-disk VDEV ( only option to create was a mirror ) Can this be expanded with a 3rd disk as RAIDZ-1 ?

[yorickdowne]

No. A mirror vdev is not raidz-1. But this is what you can do:

Break the mirror, remove one drive. Take that drive, a second fresh one, and a sparse file, and make your raidz1 in a new pool, not the same pool! Offline the sparse file.

You now have two pools with one vdev each, and no redundancy. Hope you have a good backup.

zsend/zrecv from the old pool to the new one

verify everything is there, twice. Because the next step means you can’t go back.

destroy the old pool, replace the offlined sparse file in the raidz1 with the now free drive. Let the raidz1 resilver.

This would also work with a raidz2 and two new drives. Which is a bit more future proof. Raidz1 with larger drives has a tendency to break during resilver.

[badmannen]

With my N00b level of understanding you did a great job of explaining. With that said I think I lean towards Backup-copy My 2 disk mirrored Vdev, then create a brand new with my now 3 disks and then copy back the data into the new vdev? ( more failsafe and faster ? or am I missing something , got 14 TB disks)

[yorickdowne]

With 14TB disks raidz-1 is risky when rebuilding. I recommend you do a 4-wide raidz2, which you can then expand later if desired.

[badmannen]

Sounds like a good solution ... just need to squeeze my wallet upsidedown and hope some monetas will fall out from it =) ( but yes If I don´t I will probably regret it later on )

[yorickdowne]

“Buy nice or buy twice” :)

[DurvalMenezes]

This is arriving at a really good moment for me, I was about to expand a 4-disk raidz2 into a 6-disk by the old method (create an entirely new pool then zfs send | recv from the old one into it), so if this is implemented and working, it's going to save me a ton of time and effort.

So I must ask:

1. IIUC what I read here and at the PR, it's implemented on zfs-master, right?
2. Is there an estimate for when this will be available in a main release?
3. Has anyone used it already? How did it go?

TIA!

[DurvalMenezes]

Thanks @yorickdowne and @darkbasic for your responses.

And I just read a little more about the limitations, and it seems the data-to-parity ratio isn't recalculated either, so a ton of space would end up being lost if the expansion is done using this new functionality.

Seriously, we've been waiting since forever for... this? :-( don't want to sound ungrateful but it's really disappointing to see that ZFS remains ITR so far behind what other folks (eg Linux md) have been doing for literally decades.

Anyway, back to the ancient zfs-send-recv-into-new-pool I go...

EDIT: the situation is much less worse than I was led to believe, see my next comment below,

[darkbasic]

it seems the data-to-parity ratio isn't recalculated either

That is just until you rewrite the existing data. If your data is going to change over time (or if you don't care about snapshots and simply rewrite it) you will regain the lost space.

[ahrens]

I might very well be wrong but I think you won't be able to expand existing arrays, only new ones created after the feature has been enabled.

That is incorrect. You can expand existing RAIDZ groups.

[darkbasic]

That is incorrect. You can expand existing RAIDZ groups.

That is awesome, I've never been so happy to be proven wrong :)

[DurvalMenezes]

Thanks @ahrens and @darkbasic for the additional corrections/clarifications. So the situation is much better than I was led to believe.

The major (only?) conses seem to be the (a) inability of the upgraded array to be handled by previous versions of the software not supporting this feature, and (b) desirability of rewriting all data in order to "rebalance" already-existing data (which I understand means both to redistribute its data on the new array including the added vdevs, and recalculate its parity blocks for the new scheme).

(a) is not a major problem (just test the new version well enough to be damn sure no version downgrading will be necessary) and (b) is just an inconvenience for my use case (I presume that, once I have the necessary disk space, I can just shuttle the data inside it from the previous datasets to new ones to force the rewriting with zfs send|recv, and then destroying the pre-rebalance datasets and renaming the new ones to the old names).

I will now mark this on my original comment above.

[jakesteele]

How does expand address fragmentation, if my cap is under 50% does the end result remove most/all fragmentation? What if my capacity is over 50%, does the end result in more fragmentation then before?

[ahrens]

The newly-available space will be one big free chunk, but expansion doesn't change fragmentation of the existing data.

[jakesteele]

Gotcha, I figured expansion when re-writing between disks would fix fragmentation if the free space was greater than 50%.

[GottemHams]

I'm considering upgrading the disks in my system with something faster, and add an extra disk in the process. So let's say I first add the extra disk and let ZFS expand the pool, then swap out the pre-existing disks one by one (or perhaps connect the faster one alongside) and let it resilver each time. Would the resulting pool have the updated data-parity ratio? I would assume so since eventually all disks would have newly written data, but because it happened in response to a resilver it may not take the earlier expansion (fully) into account? As in: disk X is pretty much copied bit-for-bit to replacement disk X, resulting in the same original ratio (only that disk has to be rebuilt so there's no (re)balancing of data-parity). I don't think there's enough room to connect all the new disks alongside the original ones and just send it over. :DD

[Cosine5000]

You realize this functionality is still a year or two away, yes?

[GottemHams]

Ever heard of planning? If the scenario I described wouldn't work then it's probably better to find a way to make room for a new pool and send over the data after all, which I could do right this moment. Otherwise, having to wait for 1-2 years with the current setup is perfectly acceptable.

[Titaniumtown]

I'm thinking about maybe doing something similar in a few years. but more-so for capacity. Have a raidz2 of 4 1tb sata ssds, in a few years when 4tb nvme drives get super cheap. I'm thinking about slowly replacing each 1tb sata ssd with a nvme ssd, then once every 1tb sata ssd is removed, I think it would be able to use the 4tb of each disk. Zpool contraction would be cool!

[Guiorgy]

This is probablty too late to answer your question, but the expansion doesn't change the parity of the pool, i.e. raidz2 will stay as raidz2 after expansion.

[Bommel24]

Would it be possible to use a similar approach to implement RAIDZ Shrinking. So if less space is used we can remove a disk permanently without losing parity?

[IvanVolosyuk]

No. Imagine the current version keeps all block numbers when reshuffling all the data. Conceptually, in shrinking the range of numbers decrease and there is no guarantee that all numbers currently in use will be available in the new layout even if you do the process in reverse order.

[Titaniumtown]

RaidZ shrinking would be absolutely amazing! I bet it would take much longer though than adding a disk due to the extra complexity.

[ghost]

My understanding is that after an expansion, old data remains on the old logical stripe width and only new data gets the benefit of larger logical stripes.

Is that still correct with this feature now being released? If yes, is there a way to convert old data to use the new, larger stripe?

[amotin]

Yes, that is correct. The only way to convert it is overwrite the data one way or another. For example, replicate the datasets locally and delete the original, if you have enough free space. I am dreaming about a way to rewrite data in place, but haven't got to implementing it yet.

[Xyz00777]

@amotin could the issue #15335 be used for that?
I think it would also solve issue #15679

[amotin]

@Xyz00777 Scrub is not designed to modify anything. It just checks and recovers what is already there. #15335 is only a dream. In practice we've already had some "pointer rewrite" code as part of device removal, but it was dropped AFAIK because of inability to properly lock simultaneous accesses from mounted file systems. The design I am dreaming about assumes operation in open context on requests from user-space, alike to read()/write() system calls.

[mirror176]

Does this use scratch space on a full pool and only at the beginning or can that happen again due to a pool being filled partway through the reflow? If it can't be used later, then is there anything that limits pool use to stop normal pool writes from happening at a rate that could consume the space the reflow needs?

[IvanVolosyuk]

For the purposes of writes the pool amid of reflow has the same space as original pool. Only when the reflow finishes the new space becomes available. Thus all extra space is solely available for reflow process when it's underway.

[amotin]

Scratch space is used only at the beginning of reflow. After that, since each new row of block is bigger than the old one (and farther from the beginning), there is no overlap and so reflow may breath freely. And as @IvanVolosyuk said until reflow completes that space is unavailable to the pool and so the process can not be affected by new writes.

[Tomblarom]

I have a rather dumb, but likely vaild question: What happens if a disk dies during the expansion progress? Is the expanding progress considered an unsafe timespan, during which none of the disks should fail or are we safe all the time? 😁

[IvanVolosyuk]

The expansion process is paused until redundancy is restored. In this sense the process should be as safe as normal operation.

[ClockeNessMnstr]

it would have to read the entire logical block and do reconstruction of data being copied from the failed disk to the good disks to continue the expansion. (Continue but still with a missing parity level) reflow is only moving physical stripes so it does not load checksums to be able to reconstruct.

https://youtu.be/tqyNHyq0LYM?t=1185

[diego351]

Is miscalculated available storage after expansion still an issue?

If not I think this is something that needs some attention. Thoughts?

[diego351]

@Xyz00777 Still be wrong

[Xyz00777]

So at least an issue should be created to be fixed at some point? Because I think at least at the moment there is no issue? At least not linked to the conversation

[ClockeNessMnstr]

This issue for Raid Expansion is mentioned/predicted here in an older issue with other features.

#14420

[albino1]

Seems like there should be a new more specific RAIDZ Expansion issue opened that we can follow to avoid cluttering up this discussion. What little I can find is just closed without resolution. For example, ask TrueNAS and they'll just say it's a ZFS upstream problem and not theirs to worry about (even though they funded this in the first place). It's hard for any developer to fix it if they don't even know it exists. Even finding information about this problem is difficult, and it's only going to become more significant and affect more people as users start actually taking advantage of the new expansion feature. There's going to be a lot of home users scratching their heads and frantically googling trying to figure out why the math doesn't add up and everything appears broken.

[Kernald]

Not having an issue tracking this specifically also means more people are going to comment on this thread just to get status updates when anything eventually happens - I know that's what I'm doing right now.

[Wqrld]

[Guiorgy]

No, this has already been talked about above, and it's mentioned in the man page:

Expansion does not change the number of failures that can be tolerated without data loss (e.g. a RAID-Z2 is still a RAID-Z2 even after expansion).

[Wqrld]

That seemed subtly different, so it was worth an ask. Thanks for your response!

No, this has already been talked about above, and it's mentioned in the man page:

Expansion does not change the number of failures that can be tolerated without data loss (e.g. a RAID-Z2 is still a RAID-Z2 even after expansion).

[amotin]

No. Change from RAIDZ1 to RAIDZ2 can not be done without full space reallocation and data rewrite. RAIDZ is not RAID5 (which could be mechanically converted to RAID6) in a sense that several small blocks with their own parity might be stored in one row, one added disk may (and will likely) appear not enough. And that is not even speaking that it would require rewriting all the block pointers on the pool.

[ClockeNessMnstr]

Couldn't Z1->Z2->Z3 work the same way with the expansion where old data is not taking advantage of the redundancy?

I'd think the parity level of "old" blocks be tracked similar to the changes in this expansion where "old" blocks are tracked after expansion to account for width.

P.S. I don't see the original question. assuming it was asking if the current PR expanded z1->z2.
I know that's out of scope but my curiosity is getting the better of me as to how feasible a z1-z2-z3 increase would actually be.

[amotin]

@ClockeNessMnstr I see two problems: 1) you can never rely on the added redundancy, since there will likely always remain some old blocks, and 2) while Z1 pads all allocations to 2 ashifts, Z2 -- 3 ashifts and Z3 -- 4 ashifts, so if the vdev was badly fragmented before conversion, it may be unable to allocate space in the new format, or at least some space will become unusable.

[ClockeNessMnstr]

in an old video Matt Ahrens mentions the need to reconstruct corrupted data from a drive failing before expansion when reading it after 1 or more expansions.
Basically a "diagonal" set of data in a block needs to be checked for each expansion/shift in addition to checking if the data from just current drives fail.

Is the added "combinatorial reconstruction" performed or needed after a successful scrub. or is that additional step only performed 1 time?

[ClockeNessMnstr]

I guess if it checks the drives in the latest configuration first it may not really matter much.
It would presumably be pretty hard to get that kind of failure without an all out drive failure. (which reduces the search dramatically)
The penalty is really bad though if a system actually runs through all the recoverable pre-expansion failure possibilities even though a prior scrub after expanding should rule them out.

[pycaw]

Few questions:

1. Why don't we have an option to disable the pool-wide scrub after the resilvering? Am I madman if cancel that?
2. Was expansion by multiple devices in one go ever considered? Could it be a thing ever?

[amotin]

1. Because RAIDZ expansion does not verify checksums while moving the data. So if one of your drives decide to corrupt the data, scrub started after should be able to recover it. But if instead you run second expansion or have a drive failure, you may lose your data. The same is done for sequential reconstructions of mirrors and draids, that are faster than normal resilver.
2. There is no technical reason to not add several disks to one vdev at once or not expand several vdevs at once. It is just not implemented now. Volunteers are welcome.

[pycaw]

[Guiorgy]

GitHub, WTF?! LOL

I mean, this is the internet, everyone being a bit (sometimes a lot) meaner than usual is normal. But you do you.

[pycaw]

Well, I intended to allude to how the mechanical nature of ZFS's raidz expansion/shrinkage coding problems are fertile ground for AI models of the near future to work some magic.

[Guiorgy]

Of course, I believe an LLM can help in some areas, but it won't be fully replacing a human at such a task.

[IvanVolosyuk]

RaidX to RaidY and shrinking is not a mechanical coding problem - there are conceptual show stoppers which makes it pretty much impossible to deliver. The original raidz expansion was quite a creative approach from idea to the details of implementation. If it looks mechanical to you then creating new LLM is also mechanical, just a bunch of matrix multiplications with sprinkles of other stuff.

[pycaw]

I (and that thread here) was thinking of NwraidzK->MwraidzL transition where M<N, K=L. There's also M-1>N, K=L yet to grasp. I don't want to belittle any of these tasks though! And I am but a novice user of ZFS! One has to acknowledge on the other hand the arc AI is on: LLMs, program optimizations, generative design, etc. Additionally, at this point, you are probably doing it wrong if you don't use an LLM to learn ZFS as a beginner; or, to develop it, as in, for code analysis primarily.

[erikschul]

Any idea about when 2.3.0 will reach debian bookworm-backports?

[Guiorgy]

Wrong place to ask that. This project is not affiliated with Debian in any shape or form. The Debian package maintainers just take the release whenever they see fit, build it, test it, package it and release it into their repos. You should go to the Debian package mailing list to ask them this.

[ahmed00medhat]

Can the source be built and used on Debian without an issue?

[solidsnake1298]

I am unable to attach new disks to my RAIDZ1. I'm running Ubuntu 24.04, but added the ZFS 2.3 PPA and upgraded ZFS.

libzfs6linux/noble,now 2.3.0+release-noble4 amd64 [installed,automatic]
libzfsbootenv1linux/noble,now 2.3.0+release-noble4 amd64 [installed,automatic]
libzfslinux-dev/noble,now 2.3.0+release-noble4 amd64 [installed]
zfs-dkms/noble,now 2.3.0+release-noble4 all [installed]
zfs-zed/noble,now 2.3.0+release-noble4 amd64 [installed,automatic]
zfsutils-linux/noble,now 2.3.0+release-noble4 amd64 [installed]

I enabled the feature flag for raidz_expansion.

root@rat-trap:~# zpool get feature@raidz_expansion library
NAME     PROPERTY                 VALUE                    SOURCE
library  feature@raidz_expansion  enabled                  local

I applied the change to the pool. For good measure I rebooted.

root@rat-trap:~# zpool upgrade -a
This system supports ZFS pool feature flags.

All pools are already formatted using feature flags.

Every feature flags pool already has all supported and requested features enabled.

But I still get this message when attempting to attach a new disk.

root@rat-trap:~# zpool attach -fsw library raidz1-0 wwn-0x5000cca0d2c46644
cannot attach wwn-0x5000cca0d2c46644 to raidz1-0: raidz_expansion feature must be enabled in order to attach a device to raidz

Let me know if you need any additional information.

Here is my full zpool feature list.

root@rat-trap:~# zpool get all library
NAME     PROPERTY                       VALUE                          SOURCE
library  size                           16.4T                          -
library  capacity                       93%                            -
library  altroot                        -                              default
library  health                         ONLINE                         -
library  guid                           15429762156687604509           -
library  version                        -                              default
library  bootfs                         -                              default
library  delegation                     on                             default
library  autoreplace                    off                            default
library  cachefile                      -                              default
library  failmode                       wait                           default
library  listsnapshots                  off                            default
library  autoexpand                     off                            default
library  dedupratio                     1.00x                          -
library  free                           1.13T                          -
library  allocated                      15.2T                          -
library  readonly                       off                            -
library  ashift                         12                             local
library  comment                        -                              default
library  expandsize                     -                              -
library  freeing                        0                              -
library  fragmentation                  2%                             -
library  leaked                         0                              -
library  multihost                      off                            default
library  checkpoint                     -                              -
library  load_guid                      17049638100667828991           -
library  autotrim                       off                            default
library  compatibility                  off                            default
library  bcloneused                     0                              -
library  bclonesaved                    0                              -
library  bcloneratio                    1.00x                          -
library  dedup_table_size               0                              -
library  dedup_table_quota              auto                           default
library  last_scrubbed_txg              0                              -
library  feature@async_destroy          enabled                        local
library  feature@empty_bpobj            active                         local
library  feature@lz4_compress           active                         local
library  feature@multi_vdev_crash_dump  enabled                        local
library  feature@spacemap_histogram     active                         local
library  feature@enabled_txg            active                         local
library  feature@hole_birth             active                         local
library  feature@extensible_dataset     active                         local
library  feature@embedded_data          active                         local
library  feature@bookmarks              enabled                        local
library  feature@filesystem_limits      enabled                        local
library  feature@large_blocks           active                         local
library  feature@large_dnode            enabled                        local
library  feature@sha512                 enabled                        local
library  feature@skein                  enabled                        local
library  feature@edonr                  enabled                        local
library  feature@userobj_accounting     active                         local
library  feature@encryption             enabled                        local
library  feature@project_quota          active                         local
library  feature@device_removal         enabled                        local
library  feature@obsolete_counts        enabled                        local
library  feature@zpool_checkpoint       enabled                        local
library  feature@spacemap_v2            active                         local
library  feature@allocation_classes     enabled                        local
library  feature@resilver_defer         enabled                        local
library  feature@bookmark_v2            enabled                        local
library  feature@redaction_bookmarks    enabled                        local
library  feature@redacted_datasets      enabled                        local
library  feature@bookmark_written       enabled                        local
library  feature@log_spacemap           active                         local
library  feature@livelist               enabled                        local
library  feature@device_rebuild         enabled                        local
library  feature@zstd_compress          enabled                        local
library  feature@draid                  enabled                        local
library  feature@zilsaxattr             active                         local
library  feature@head_errlog            active                         local
library  feature@blake3                 enabled                        local
library  feature@block_cloning          enabled                        local
library  feature@vdev_zaps_v2           active                         local
library  feature@redaction_list_spill   enabled                        local
library  feature@raidz_expansion        enabled                        local
library  feature@fast_dedup             enabled                        local
library  feature@longname               enabled                        local
library  feature@large_microzap         enabled                        local

[amotin]

@solidsnake1298 I wonder if it can be a misreported issue, but -s does not make sense for RAIDZ expansion. The expansion is always sequential. -w I haven't personally tested, so don't know.

[solidsnake1298]

@solidsnake1298 I wonder if it can be a misreported issue, but -s does not make sense for RAIDZ expansion. The expansion is always sequential. -w I haven't personally tested, so don't know.

It was that simple. I removed the -s and -w options and now the raidz is expanding.

[Guiorgy]

Then it seems like it's really misreporting. @solidsnake1298 This probably should be reported as a new Issue?

[solidsnake1298]

Then it seems like it's really misreporting. @solidsnake1298 This probably should be reported as a new Issue?

Will do when I get home.

[MacsedProtoss]

hi, I would like to know why the expansion result in a much less total spaces.
I have a raidz2 pool at the beginning with 4 x 6.99TiB disk, and the total spaces is about 14TB, and is almost full.
then I expanded it for 4 times with the same command:

sudo zpool attach myStorage raidz2-0 /dev/disk/by-id/xxx

after the expansion, the df -h told me I only have 28TiB in total
and the zpool list shows these:

NAME                                                SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
myStorage                                         55.9T  30.6T  25.3T        -         -     0%    54%  1.00x    ONLINE  -
  raidz2-0                                         55.9T  30.6T  25.3T        -         -     0%  54.7%      -    ONLINE

I was expecting that the total spaces should be about 42TiB, but in the end I only got 28TiB.

I assume it's abnormal but I don't know what to do. Any suggestions?

[ClockeNessMnstr]

Ok, your OS should be reporting some weird numbers.

Your total "drive size" should now show as 14.5TB instead of 16.4TB
Your ~10.1TB of data should now show as ~9TB (now that it's re-written)
Your free space should now show ~5.5TB instead of ~6.3TB

"zpool get all" will give the right numbers for allocation across all drives but "used"/"free"/"total" space reported to your OS will not match what's expected of a 4x6TB raidZ1.

[MacsedProtoss]

@solidsnake1298
I'm guessing (from your 16.4TB) your array had a original ratio of 83% (6x3TB -> 7x3TB) and a new ratio of 85.7% meaning your reported 83.3% ratio is only off by ~2.4%
If you had 6TB drives it would be 66%->75% and should be more noticable.
Macsedprotoss' ratio went from 50% to 75% so his reporting is off by 25% which is way more noticable.

I went from 3x6TB to 4x. The lost/regained reported free space was definitely noticeable.

I also used that script in the past two days. And now I got much more free spaces in zpool list myStorage -v but the df -h still reporting the same wrong number.

Thanks to ClockeNessMnstr and those discussions above, now I fully understand why this wired behavior happened. Hopefully someone or myself(if free from work) may create a pr to solve it one day

[DurvalMenezes]

now I got much more free spaces in zpool list myStorage -v but the df -h still reporting the same wrong number

With all due respect to the developers and the efforts they selflessly gift us with, this kinda sucks. Among other things, monitoring scripts will start reporting false alarms.

Is there at least a separate issue for this behavior? It would help focus attention and eventually bring up a fix.

[Guiorgy]

Is there at least a separate issue for this behavior? It would help focus attention and eventually bring up a fix.

Getting this feature to this state alone took a lot of effort, and was finally pushed to a usable state by the folks from TrueNas, from what I heard (or was it a different company?). And since this is primarily aimed at home users who can't afford the hardware for a full migration, the priority is probably low. Honestly, the best thing I can come up with is everyone chipping into a cash pool and putting it out as a bounty/pay for a few developers to dedicate time to work on this. I'd put $50.

Edit: Off topic, but @amotin , does the OpeZFS project accept user donations? I can't find it anywhere.

[amotin]

@Guiorgy While many of bigger features, including this one, were sponsored in one way of another, as you may see in the the commit messages, it usually happens on a corporate level. OpenZFS project as whole never collected donations, as far as I know, while individual developers may be open to it: #16628 (comment) .

[404erreur]

Hello every body,

Small question, I have an zpool created in 2.0 zfs version, this one was upgraded to 2.3 package and zpool upgrade -a.
The extention flag is enable.

I need to do another action before adding a new disk in my vdev ?
My zpool is very big and I prefer to know before bying a disk, I cant make a zfs send / recv for exemple (no suffisant space)

Many thanks in advance for your answer :)

Regards
404

[darkbasic]

@amotin is there currently any other way to get a realistic figure for the used/available space?

[Guiorgy]

Assuming you rewrote all the old data (including snapshots), I think you could try to manually rescale free/total space and get a better approximation. But at the end of the day, if what you want to know is how full your pool is, the current approximation should give a sufficient idea.

[404erreur]

So, we need to rewrite all data (That's why I asked for send/recv previously)?
In this case I will need to rewrite 30 TB of data, I can't do that

[IvanVolosyuk]

You don't have to do send/recv to enjoy extra space after raidz expansion. I don't know your specific layout, but imagine you have 4 disk raidz1, 12TB each disk, and zfs datasets reports 30TB used space. In such case zpool will report something like '30 / (4 - 1) * 4 = 40' used space (including parity).

Total with parity would be 12 * 4 = 48, with free space with parity: 48 - 40 = 8
When adding new disk the total with parity becomes 12 * 5 = 60TB and free space with parity: 60 - 40 = 20TB ,which is one disk of data more: 12TB.
In this example, your data vs data+parity ratio right now is: (4-1) / 4 = 0.75 and will be (5-1) / 5 = 0.8

For writing new data the new parity ratio will be used, thus you can write up-to 20 * 0.8 = 16TB, while before expansion: 8 * 0.75 = 6TB. Thus you can write up-to 10TB more (this is an estimate which works for storing large files both before and after the expansion).

Send/recv can improve efficiency of your data you had before expansion, but it is not required. In this example you can get back up-to '20 * 0.8 / 0.75 - 20 = 1.3TB'. You can gradually sent / recv your data a dataset after dataset, ignore this entirely or wait for data to be rewritten naturally by normal read / writes operations.

[404erreur]

It's very clear many thanks Ivan, and also ll person than read a reply to my post :)

Regards
404

[Sapd]

Im currently expanding a 4-disk zraid-1 pool on Linux to 5 disks.

I however notice extreme spikes in memory doing so. From one second to the other from 64 GB to 128GB (Which is my max ram), which triggers the Linux OOM Killing or freezes the system entirely. Even with a limited ARC of 24 GB (and just having around 50 GB actually in use). I reduced now ARC to even 18GB and scaled down some VMs and enabled SWAP on an nvme, which seemed to help a lot.

Did someone also experience that? And are there ways around that? Whats really surprising is the enormous sudden spikes.

[darkbasic]

I didn't try raid expansion but I had issues with ARC and huge LLM models. If a 120GB model is in a dataset with ARC when you load it in memory the OOM killer will go crazy instead of simply dropping the ARC. I had to store LLM models in a dataset with ARC disabled.

[DurvalMenezes]

Hi Denis, Watching closely for the responses to this (haven't used raidz expansion yet, but I was planning to in the near-future -- and besides the "free/used space numbers going astray" issue, this looks like is another potential problem with raidz expansion, and probably even worse for me, since my largest reliable machine -- with ECC RAM -- only has 64GB RAM). A couple of tips: 1) have you searched the Github issues for OpenZFS? ( https://github.com/openzfs/zfs/issues)? if there are any open issues, you can then add a comment mentioning you're also seeing it, and if there's none, you can then open it so the devs will be aware of the issue (many of them don't read this list, at least no assiduously). 2) It would help if you gave us more details, like OS/Distro/Version, machine architecture especially re: storage, output from `zpool status`, if `dmesg` or syslog shows anything, etc. And a couple of questions/comments:

On Mon, Dec 1, 2025 at 8:05 AM Denis Arnst ***@***.***> wrote: Im currently expanding a 4-disk zraid-1 pool on Linux to 5 disks. I however notice extreme spikes in memory doing so. From one second to the other from 64 GB to 128GB (Which is my max ram), which triggers the Linux OOM Killing or freezes the system entirely.

"freezes entirely" meaning "eventually unfreezes itself and continues after X minutes/hours/days" or "freezes entirely" as in "freezes forever and needs a reboot to get the machine back up"? Even with a limited ARC of 24 GB (and just having around 50 GB actually in

use). I reduced now ARC to even 18GB and scaled down some VMs and enabled SWAP on an nvme, which seemed to help a lot.

Good tip on the NVMe SWAP, could be a viable workaround for my case (will just have to buy another, larger NVMe since the one in the machine is currently fully in use). Did someone also experience that? And are there ways around that? Whats

really surprising is the enormous sudden spikes.

Also very interested in hearing from other folks experiencing raidz expansion issues (this or any other). Cheers, -- Durval.

…

— Reply to this email directly, view it on GitHub <#15232 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJHXRVWJJP6U6WVQEERJ3D37QOFFAVCNFSM6AAAAAA4HOSFKSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKMJSGMZDSMA> . You are receiving this because you commented.Message ID: ***@***.***>

[Guiorgy]

You forgot to ping him: @Sapd
Either that, or this was meant to go under his comment, instead of being a separate one :P