A reminder that zfs recently (past ~5 years) implemented dedicated metadata cache devices ... which allows you to cache either filesystem metadata or even small files to a blazing fast SSD mirror:
This is a quick and easy way to add thousands of iops to even something very slow like a raidz3 zpool.
As always:
"Let's repeat, and emphasize: unlike an SLOG or L2ARC which are merely inconvenient to lose, if you lose your metadata vdev (your "special" vdev) you will lose your entire zpool just as surely as if you lost one of the other full vdevs ..."
If you set up a normal L2arc (read cache device) that will cache both data and metadata. However you can configure it to only cache one of the two. Set it to metadata only and size it appropriately and you have basically a read-only metadata mirror.
If you also want to have fast writes you can get a second SSD and set up a mirrored metadata device (storing metadata on mirrored SSDs, and regular data on whatever the rest of your pool uses)
Last time I stapled l2arc onto a hard drive zfs it was still super slow to do things like ls large directories and the l2arc never filled more than a couple gigabytes.
Yes, the docs[1] tells you to match the redundancy level of the pool, but you could technically do something else.
Note the docs say level and not type. So if you have raid-z1 the a mirror would be appropriate as they both tolerate losing a single disk, and if you have raid-z2 you can use a 3-way mirror etc.
You do, the normal way is to create a special metadata devices of mirrored SSD's. You do not need a 3 way mirror, as ZFS by default stores 2 copies of your metadata.
https://www.rsync.net/resources/notes/2021-q3-rsync.net_tech...
This is a quick and easy way to add thousands of iops to even something very slow like a raidz3 zpool.
As always:
"Let's repeat, and emphasize: unlike an SLOG or L2ARC which are merely inconvenient to lose, if you lose your metadata vdev (your "special" vdev) you will lose your entire zpool just as surely as if you lost one of the other full vdevs ..."