Questions about extended file attributes (xattrs)

[pjotrek-b]

Hello everyone 😄

It's about btrfs-documentation:
I'd like to ask about technical details on extended attributes (xattrs) on btrfs, and found these pages:

• https://docs.kernel.org/6.1/filesystems/btrfs.html
• https://btrfs.readthedocs.io/en/latest/

Which both don't really list or mention extended file attributes handling (xattrs).
I know btrfs can do xattrs, and ArchWiki mentions possible size-constraints, but I'm interested in the specific limits and settings for xattrs on btrfs 😄

I'm moving this as issue as question here, following advice from @kdave in btrfs/linux#1455

If anyone wants to talk about this or point me towards some code/docs, I'd be very grateful! 🍰

[adam900710]

XATTRS is stored with key (<inode number> XATTR_ITEM 0), its related data is btrfs_file_extent_item but only in the inline extent mode.

And xattrs are stored as inline data extents, meaning it's limited by two things:

• Sector size
  Inline data extent can not reach sector size. For most filesystems it's 4K.
  This is just a soft limit, but will make kernel implementation much easier.

• Inline data extent size
  This is limited by node size (metadata size), since we're storing everything inside metadata.
  This is a hard limit, but shouldn't cause any hassle because we're using 16KiB as node size for a long long time.

[pjotrek-b]

@adam900710 : This is great information, thank you!

Do I understand this correctly:

1. 4k sector size:
   Every xattr byte (all keys + all values) must stay below 4k?
   "Soft limit" means: Theoretically adjustable - but not really tested or really laid out for?

2. "16kiB as node size"
   I don't yet fully understand this.
   Do you mean: Internally btrfs could handle up-to-16kiB xattrs - if the sector size would be that big?

So if I'd like to use xattrs like BeFS or HFS/APFS where even binary-data (eg images) are stored: would btrfs' defaults be capped to 4k in total right now?

Thank you so much! 😄

[SaltyKitkat]

So if I'd like to use xattrs like BeFS or HFS/APFS where even binary-data (eg images) are stored: would btrfs' defaults be capped to 4k in total right now?

I've tried with kernel version 6.15.3, and write a 13.5KiB file into a test file's xattr, with key user.testdata. It seems working fine.

However, when I try to write the same file twice(in order to double the size) into xattr, I got a failure message: [Errno 28] No space left on device: b'test_xattr'

But I can write the data into other keys, and it works fine. So if you want to store data more than 16KiB, maybe you should split it into different keys in xattr.

[Forza-tng]

It is possible to mkfs with a node size of 64k, but not change it on an existing fs.

[adam900710]

@adam900710 : This is great information, thank you!

Do I understand this correctly:

1. 4k sector size:
   Every xattr byte (all keys + all values) must stay below 4k?
   "Soft limit" means: Theoretically adjustable - but not really tested or really laid out for?

Sorry I didn't check the code deep enough.
For XATTR items, it doesn't bother sector size, that only affect regular data, not XATTR ones.
And XATTR items are not stored like inlined data, but more like dir items.

So you can ignore this limit.

I'm not sure if this is the intended behavior, but if it allows larger XATTRS, I didn't see a huge problem here.

2. "16kiB as node size"
   I don't yet fully understand this.
   Do you mean: Internally  btrfs could handle up-to-16kiB xattrs - if the sector size would be that big?

Yes, and it's already doing so, and that limit only applies to a single xattr key (if no conflicts).
So you can have multiple XATTRS entries, each up to the nodesize limit (slightly smaller than nodesize, explained later).

So if I'd like to use xattrs like BeFS or HFS/APFS where even binary-data (eg images) are stored: would btrfs' defaults be capped to 4k in total right now?

There is a macro calculating the max XATTR size inside kernel, BTRFS_MAX_XATTR_SIZE(), when fully expanded it is:

nodesize - sizeof(struct btrfs_header) - sizeof(struct btrfs_item) - sizeof(btrfs_dir_item)

Under default mkfs options, it's 16K - 101 - 25 - 30 , which is 16228 bytes.

But please keep in mind, this limit is shared by both the key and data part.
E.g. with user.testdata as key, you only have 16228 - 13 bytes left for data. (No terminating '\0')

Another pitfall is, the key is hashed, and if two different keys have the same hash (aka, hash conflicts), they will share the same item, thus share the same size limit.

In fact there is an error path we didn't properly handle, and if the shared item is too large, it will crash the kernel.

I've tried with kernel version 6.15.3, and write a 13.5KiB file into a test file's xattr, with key user.testdata. It seems working fine.

You're right, I should dig the code a little deeper.

And just explained above, your idea of splitting the contents is also feasible, but the hash conflicts handling is a little tricky and will be very btrfs specific.

In theory, we can add an extra key type to use regular data extents for super large XATTR. If there is a strong demand it may be worthy considering.
But so far I didn't really see such demand yet.

[pjotrek-b]

Thank you all soooo much for these very very useful insights!

@adam900710:
Regarding "we can add an extra key type": You mean something like a new xattr-class?

Quote from Arch xattr man page:

"Currently, the security, system, trusted, and user extended attribute classes are defined as described below. Additional classes may be added in the future."

Regarding demand and use-cases:

I'm working with medium-to-large scale professional collections (mainly cultural hertitage GLAM, but also research-data, etc). There's great need and potential in having proper (large-and-reliable) metadata on the filesystem for offloading cataloging structures (see "AHAlodeck" or "SEN-Labs" (on Haiku))

Since recently, we've been experimenting with xattrs on ext4/btrfs/ZFS and it is mind-blowingly useful.
Even "just" de-embedding metadata from different file-formats (eg using exiftool) has proven very easy and great to handle metadata, annotation and search/retrieval on existing files.

Limits of other filesystems (such as 4kB, 16kB) were already hit in our tests when storing eg preview images or other larger related-metadata "blobs" (eg JSON/XML manifests, like IIIF, etc).

See the IPTC Photo Metadata Reference Sample File for example: Its metadata cannot fully be contained in xattrs with these size-limits at the moment. 😢

Another case is storing RDF-semantic data structures as xattrs.

Given the fact, that once the metadata was moved to xattrs, the filetype/format becomes irrelevant - and allows (equal-)handling of all "datas" from them all (which is brilliant and beautiful! 🍰 🥳 )

Anyways, thanks for your interest and all your contributions to this question!

[pjotrek-b]

It is possible to mkfs with a node size of 64k, but not change it on an existing fs.

May I ask for the commandline/arguments/options for setting up btrfs with its currently largest-possible xattr-limits?

Like, is it:
$ mkfs.btrfs -n 64k -s ??? -M ??? -L "meta+data" ... /dev/sda ?

Taken from man mkfs.btrfs:

• -n|--nodesize
  Specify the nodesize, the tree block size in which btrfs stores
  metadata.

• -s|--sectorsize
  Specify the sectorsize, the minimum data block allocation unit.

• -M|--mixed
  Normally the data and metadata block groups are isolated. The mixed
  mode will remove the isolation and store both types in the same
  block group type.

Thank you!

[Forza-tng]

It is --nodesize that you should use.

[adam900710]

Regarding "we can add an extra key type": You mean something like a new xattr-class?

I mean a btrfs on-disk format change, not related to the XATTR themselves, but how btrfs stores XATTRs.

This won't be a small change, meaning older kernel will not be able to understand the new on-disk format.
Not to mention extra test cases and a lot of extra tests.

May I ask for the commandline/arguments/options for setting up btrfs with its currently largest-possible xattr-limits?

Just --nodesize, so what you want is

mkfs.btrfs -n 64K <device>

The "metadata" terminology used inside btrfs is just to describe its Btree blocks, not the same "metadata" stored inside XATTR in your usecase.

[pjotrek-b]

Thanks for all your replies!

This won't be a small change, meaning older kernel will not be able to understand the new on-disk format.
Not to mention extra test cases and a lot of extra tests.

Roger that.
Well, if one day in the near future, using xattrs will be common and useful like "long-filenames" (compared to 8.3 ASCII UPPERCASE 😉), it may make sense to consider larger changes like that, I guess?

For the time being, I'm very happy to hear that btrfs can have ~64kB information stored in xattrs (given it was formatted with -n 64K:
Quoting @adam900710: nodesize - sizeof(struct btrfs_header) - sizeof(struct btrfs_item) - sizeof(btrfs_dir_item)

Great! 🥳