posix: a file that can never be assimilated is re-hashed forever (no backoff, no terminal state) — sustained 100% disk utilisation

[NickWalters]

Summary

When a file in the posix tree fails to assimilate for a permanent reason, the driver never records the failure. The file stays uncached, so every subsequent scan or directory listing re-assimilates it — including a full-file sha1+md5+adler32 read. There is no backoff, no retry cap, and no "this will never work" state, so the driver reads the same files at full disk bandwidth indefinitely.

I hit this with an xattr permission failure, but the trigger is incidental: any permanent per-file failure produces the same loop. Closed #1585 (file name too long) looks like the same shape with a different cause, and was fixed at the trigger rather than at the retry behaviour.

Possibly related, unconfirmed: #3470 reports recurring ~100% I/O pressure with OpenCloud at only ~80–120 MB RSS and no request load, with no root cause identified. That is what this loop looks like from the outside, though I have not verified that the reporter has any permanently-failing files.

My trigger

opencloud.service runs as User=opencloud; the tree it was pointed at was owned by root:media. Linux only allows user.* xattr writes by the file owner or CAP_FOWNER, so every file ended:

xattr.Set .../00001.m2ts user.oc.id: permission denied
-> failed to assimilate node

followed on the next listing by record not found in cache. Assimilate. and another full re-read.

I understand the docs now require the tree to be owned by the OpenCloud user and reserved exclusively for OpenCloud, and that my setup violated that. The report is not "support my layout" — it is that violating it degrades into an unbounded read loop rather than a bounded, visible error.

Impact

• 117 MB/s sustained for 46 hours, 517 assimilate attempts per file per 24 h.
• Starved every other consumer of that array.
• STORAGE_USERS_POSIX_MAX_CONCURRENCY=2 did not help — it bounds parallelism, not repetition.
• Hard to attribute: load 11 with 4% user CPU and 47% iowait, so anything watching CPU reports the service as idle.

Where it is in the code

vendor/github.com/opencloud-eu/reva/v2/pkg/storage/fs/posix/tree/assimilation.go on main:

• workScanQueue (~L186): failure is log.Error(...) then continue. Nothing is persisted about the failure.
• The _errSkipAlreadyKnown short-circuit (~L496) compares the node's metadata mtime — which lives in the xattrs that were never written. A file that failed to assimilate can never reach this path.
• node.CalculateChecksums (~L770) runs on every assimilation, so each retry is a full-file read.

Suggested fix (any one would be enough)

1. Track consecutive failures per path (in the existing ID cache) and apply exponential backoff, or stop retrying after N attempts until mtime changes.
2. Distinguish permanent errors (EPERM/EACCES on xattr set, ENAMETOOLONG) from transient ones and mark those nodes as un-assimilable rather than retrying.
3. Fail fast and loudly at startup: probe the tree for xattr writability under the service user and refuse to start (or log one clear fatal) instead of degrading silently.
4. Cheapest mitigation: skip the checksum recompute when the previous attempt on the same path+mtime already failed.

Environment

• OpenCloud 7.3.0 (rolling) when observed; code paths above unchanged on main as of 7.5.0.
• Debian in an unprivileged LXC, native systemd install (no Docker), ZFS-backed storage.
• STORAGE_USERS_DRIVER=posix, STORAGE_USERS_POSIX_WATCH_FS=true, SCAN_FS=true.