posixfs: every upload sticks in processing (425) when fs revisions are enabled — blobstore reads source after renaming it away

[TCS-UK]

Summary

Since OpenCloud 7.2.4 (reva v2.46.8), every upload to a posixfs storage stops being
accessible when STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS=true. The bytes transfer correctly and
land at the right path, but finalization fails, the node keeps
user.oc.nodestatus="processing:<blobid>" forever, and the file can no longer be opened,
downloaded or deleted — everything returns 425 Too Early.

The cause is in pkg/storage/fs/posix/blobstore/blobstore.go. reva#720 changed the blob move to
use os.Rename(source, tempName), but the later block that writes the "current version" copy still
does os.Open(source) — a path the rename has just removed.

This only triggers when the upload area and the blobstore are on the same filesystem (the
recommended layout) and fs revisions are enabled. Cross-device installations hit EXDEV, fall
back to copying, and keep working — so correctly configured servers are the ones that break.

Affected versions

OpenCloud	reva	Status
7.2.2	v2.46.7	clean
7.2.3	v2.46.7	clean
7.2.4	v2.46.8	affected
7.3.0	v2.47.0	affected
7.4.0	v2.48.0	affected
7.5.0	v2.49.0	affected
main	—	affected

Verified by the presence of the offending code in each tag:

$ for t in v2.46.7 v2.46.8 v2.47.0 v2.48.0 v2.49.0; do
    printf "%-9s " $t
    curl -s https://raw.githubusercontent.com/opencloud-eu/reva/$t/pkg/storage/fs/posix/blobstore/blobstore.go \
      | grep -c "could not open source file"
  done
v2.46.7   0
v2.46.8   1
v2.47.0   1
v2.48.0   1
v2.49.0   1

Environment

• OpenCloud 7.2.4 (stable), reva v2.46.8, single container
• STORAGE_USERS_DRIVER=posix
• STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS=true
• POSTPROCESSING_STEPS=policies,virusscan (ClamAV)
• The whole /var/lib/opencloud tree — including storage/users/uploads and the spaces — is on a
  single filesystem
• Clients reproducing it: web, desktop sync client, and iOS app (all equally)

Steps to reproduce

1. Use the posix driver with STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS=true.
2. Ensure the upload area and the storage tree are on the same filesystem.
3. Upload any file containing at least one byte.

Expected

The upload finalizes; the file becomes accessible.

Actual

Finalization fails immediately and the node stays in processing forever:

{"level":"error","spaceid":"<space-id>","nodeid":"<node-id>",
 "error":"failed to upload file to blobstore: could not open source file
 '/var/lib/opencloud/storage/users/uploads/<upload-id>' for reading:
 open /var/lib/opencloud/storage/users/uploads/<upload-id>: no such file or directory",
 "message":"could not finalize upload"}

The node keeps user.oc.nodestatus="processing:<blobid>". From then on the file returns 425 to
every operation — POST /app/open, WebDAV GET, and WebDAV DELETE. The user cannot open the
file, download it, or delete it, and sync clients silently diverge because their delete is refused.

An unacked event is retried roughly hourly, re-running the virus scan and failing identically each
time, so the log fills with the same pair of errors indefinitely.

Zero-byte files are unaffected — they have no blob, so the failing path never runs. That is a
convenient way to confirm the diagnosis.

Root cause

Blobstore.Upload(n *node.Node, source, copyTarget string) in
pkg/storage/fs/posix/blobstore/blobstore.go.

Before reva#720, the function opened source once, copied it to the blob temp file, then rewound
that same handle with Seek(0, 0) and copied it again to copyTarget.

reva#720 replaced the first copy with a move:

if bs.canUseRenameForUpload {
    err := os.Rename(source, tempName)     // source no longer exists after this
    switch {
    case err == nil:
        // continue
    case errors.Is(err, syscall.EXDEV):
        bs.canUseRenameForUpload = false   // different device -> fall back to copying
    default:
        return fmt.Errorf(...)
    }
}

if !bs.canUseRenameForUpload {
    sourceFile, err := os.Open(source)     // copy path; source still present here
    ...
}

but the block further down, which writes the "current version" copy, still reads source
unconditionally:

// also "upload" the file to a local path, e.g., for keeping the "current" version of the file
sourceFile, err := os.Open(source)
if err != nil {
    return errors.Wrapf(err, "could not open source file '%s' for reading", source)
}

After a successful rename that path is gone, so this always fails.

copyTarget is the .CURRENT file, which is only written when fs revisions are enabled — which is
why the failure presents as "fs revisions are broken" rather than "uploads are broken".

Why the three conditions matter

• Same filesystem + revisions on → rename succeeds, source is gone, the copyTarget block
  runs → always fails.
• Cross-device → EXDEV, falls back to copying, source survives → works. (This is why
  Error when uploading files #1885, a cross-device report, is a different failure with a different message.)
• Revisions off → copyTarget block never runs → works.

Evidence

inotifywait on the upload area and the destination during a single upload — the blob is moved out
of uploads/ before finalization reads it:

21:25:36 CREATE      uploads/<upload-id>
21:25:37 CLOSE_WRITE uploads/<upload-id>
21:25:37 MOVED_FROM  uploads/<upload-id>
21:25:37 MOVED_TO    <space>/.oc-tmp/<upload-id>
21:25:37 MOVED_FROM  <space>/.oc-tmp/<upload-id>
21:25:37 MOVED_TO    <space>/<folder>/<filename>

Server log for that same upload:

21:25:37 storage-users  UploadFinished
21:25:37 antivirus      File scanned      virus:""  outcome:"continue"
21:25:37 error          could not finalize upload
   could not open source file '.../uploads/<upload-id>': no such file or directory

Note this is the first and only finalize attempt — not a retry of an earlier partial success.

The stored file is complete and correct: size matches user.oc.blobsize, and the user.oc.cs.md5,
user.oc.cs.sha1 and user.oc.cs.adler32 attributes all match the bytes on disk. No data is
lost — only the bookkeeping fails, and the file is then gated behind a flag that can never clear.

Impact

• Every upload with content becomes permanently inaccessible.
• Affected files cannot be deleted either (425 on DELETE), so users cannot clean up, and sync
  clients enter a divergent state where a locally deleted file persists on the server.
• The hourly retry re-runs the antivirus scan for every stuck upload forever.
• opencloud storage-users uploads sessions --processing cannot see these sessions — it only
  enumerates sessions whose blob still exists, and the blob being gone is the whole problem. On one
  affected server it listed 3 of 202 sessions and returned empty for --processing while 33 nodes
  were genuinely stuck. --resume, --restart and --clean are therefore all unavailable. This
  matches [Bug] Cant remove old stuck files from postprocessing #2416.
• opencloud posixfs scan <path> reports success but does not clear the state.

Workaround

Set STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS=false and restart. Uploads work again immediately,
at the cost of file version history.

Already-stuck files must be repaired manually — the CLI cannot reach them:

setfattr -x user.oc.nodestatus "<file>"                                # clear the flag
mv /var/lib/opencloud/storage/users/uploads/<blobid>.info <elsewhere>  # stop the hourly retry
docker restart <opencloud-container>                                   # required

The restart is mandatory: posixfs keeps its filemetadata cache in-process, so the file keeps
returning 425 until the process restarts even after the xattr is cleared.

Verify integrity before clearing. Note os.getxattr / getfattr --only-values return the checksum
digests raw, while getfattr -d displays them base64-encoded with an 0s prefix.

Suggested fix

In the copyTarget block, read from tempName (which holds the bytes after the rename) rather
than from source — or write the copyTarget copy before performing the rename.

Related

• enabling posix fs revisions breaks async processing #868 — "enabling posix fs revisions breaks async processing" (closed 2025-05-28 by reva#228).
  Same user-visible symptom, different cause; worth reopening or cross-referencing, since anyone
  searching will land there first.
• Error when uploading files #1885 — same could not finalize upload message and the same .oc-tmp path, but a cross-device
  (EXDEV) failure, so a genuinely different bug.
• [Bug] Cant remove old stuck files from postprocessing #2416 — stuck postprocessing files cannot be removed; consistent with the CLI blindness above.
• reva#720 — the change that introduced this.