[martinconic]

Problem

Two valid single owner chunks can share the same address, batch and stamp while wrapping different content. The pullsync want-check (ReserveHas(addr, batchID, stampHash)) is blind to content, so a node holding one variant never requests the other. Neighborhoods holding divergent SOCs can never converge, and their reserve samples produce different commitments in the redistribution game.

Change (SWIP-101)

Offers now carry a 16-byte checksum per chunk instead of batch ID + stamp hash:

• CAC: keccak256(batchID ‖ stampHash)[:16]
• SOC: keccak256(batchID ‖ stampHash ‖ wrappedCacAddress)[:16]

The receiver wants a chunk iff !Has(address, sum), answered in O(1) by a new existence index (ChunkSumItem). Sums are precomputed at write time and stored on ChunkBinItem. On delivery the sum is recomputed, and a mismatch is rejected as unsolicited. Offer entries shrink from 96 to 48 bytes.

When one SOC address holds several payloads, the reserve keeps one of them deterministically, whatever the arrival order:

• Same stamp, different payload: the payload wrapping the lexicographically lower CAC address wins.
• Different stamps: the strictly higher stamp timestamp wins; equal timestamps go to the lower stamp hash. The losing stamp is kept as an entry against the winning payload, so it still keeps the chunk paid for.
• Same slot re-stamped: a strictly newer timestamp is required, as on master. An equal timestamp with a different stamp hash is rejected, because anyone can derive such a stamp from an existing one through signature malleability.

When a SOC payload is replaced, the sums of the entries under the other stamps are refreshed, keeping the index consistent with deliverable content.

Breaking change

Pullsync protocol version 1.4.0 → 2.0.0. Old and new nodes cannot pullsync with each other in either direction, which needs a coordinated release. See the open question below.

Migration

• Backfill (step_09): backfills sums for existing reserve entries, paginated and without loading stamps. It removes dangling, unreadable, untyped and duplicate entries, plus leftovers in the legacy format, instead of failing startup.
• Version 8 is a noop: it was removeRogueChunks (v2.8.3). Run on a node upgrading from v2.8.2 or older, it would decode ChunkBinItems in the pre-Sum format with the new decoder and fail. step_09 re-verifies every reserve entry against the chunkstore, so it covers that step.
• Repair: db repair-reserve rebuilds the sum index alongside the chunk bin items.
• Downtime: the node is offline during the backfill. It costs one chunkstore read per reserve entry, about 107 µs each with a warm cache (roughly 7.5 min per 4M chunks), and longer on HDD.

Merge blocker

.github/workflows/beekeeper.yml pins BEEKEEPER_BRANCH to feat/pullsync-chuns-convergence, because the socmatrix check only exists there (beekeeper PR 611). The pin goes back to master once that PR is merged.

Open questions for reviewers

1. Protocol version. Instead of the hard 2.0.0, we could bump to 1.5.0 and keep BatchID = 2 and StampHash = 3 next to a new Sum = 4. Old nodes could then still pull from new ones, as with every earlier pullsync bump. The cost is larger offers until a later release drops the old fields. (Suggested by @gacevicljubisa in review.)
2. Divergence that already exists. Sync intervals survive the upgrade, so chunks already synced under 1.4.0 are not offered again, and only divergence created after the upgrade converges. Resetting the pullsync intervals once would heal the existing network, at the cost of one full offer pass per neighbor.

Known limitations

• Replaying an old payload. A stamp signs the address, not the content. Anyone holding an older payload the owner signed can therefore pair it with the owner's current stamp. If that payload wraps a lower CAC address, it wins the same-stamp tie-break. A deterministic rule keeps the neighborhood's reserve samples consistent, but any rule based only on content can be steered this way. Feeds are not affected, since each update uses a new id.
• Overwrites follow stamp timestamps. An update stamped by an issuer whose clock is behind the stored stamp loses to it, while pushsync still returns a receipt.
• Settled payload is not revisited. When the stamp that decided a payload later leaves its slot, the decision is not reconsidered. The tests for this log as known unresolved and fail only with RESERVE_STRICT_CONVERGENCE=1.
• Sibling refresh cost. Replacing a SOC payload rewrites the sums of all co-resident stamps, so the cost is O(N) in the number of stamps on that address.
• One-way migration. After a downgrade to v2.8.3 or older, pullsync serving and the reserve sampler fail on the new record format. After a downgrade and re-upgrade, run bee db repair-reserve.

Testing

• Unit tests:
  • the divergent-SOC sum property;
  • the sum index staying in lockstep with the reserve;
  • sibling-sum refresh on both replacement paths;
  • rejection of a re-stamp with a malleated signature;
  • the migration against genuine pre-migration records: the old 106-byte serialization, dangling entries, orphans, a zero stamp hash, and a duplicate (bin, binID) across a page boundary.
• Checksum vectors for a CAC and for two SOCs sharing an address. They were computed independently of the implementation, so other clients can check their sums against them.
• Pullsync between two real reserves. Each starts with a different payload of one SOC, under one stamp or under stamps of two batches, and the test syncs in both orders. Both nodes end on the same payload and sums, and a second round puts nothing.
• Beekeeper ci-socmatrix runs the retention rules end to end on a local cluster.

[acud]

with the change from @janos we can just backward support the old version through the different semver (but we'd need to duplicate the protocol code for this). let's discuss this.

[gacevicljubisa]

We could even keep the protocol version 1.4.0, by adapting the proto message for Chunk:

message Chunk {
  bytes Address = 1;
  bytes BatchID = 2;
  bytes StampHash = 3;
  bytes Sum = 4;
}

where len(Sum) == 0 can identify the legacy peer, and select the old batchID/stampHash check.

[acud]

what is this about? :) i think this would be the first appearance of a self-authored storage wipe mechanism. why do we need such a thing? and why does a migration not cover what we need it to cover?

[acud]

iff when a chunk changes on the disk we need to bump the binIDs for the indices relating to that chunk

[acud]

remove this branch, test case and document the parent Put method

[acud]

please document in a nutshell what this method does

[acud]

if this is true, why do we need to do anything instead of returning early here?

[martinconic]

We can’t return early because the incoming chunk brings a new, valid stamp (different batch or index) for this SOC address. Returning early would break arrival-order convergence:

If V1 arrives before V2: both stamps end up in the reserve, with V1 updated to V2's payload via refreshSiblingSums.
If V2 arrives before V1: returning early would drop V1's stamp completely, causing the node to diverge from nodes that received them in reverse order.
So we must still record the stamp in the reserve, but index it with storedSum (the winning payload on disk) so pullsync offers match what we actually serve.

[martinconic]

Findings fixed in the last 8 commits
bee-5544-fixes.md
bee-5544-open-findings.md

[acud]

bee-5544-fixes.md

B1 how is this possible? isn't this the most basic case we are catering for here? reading the first sentence A SOC address already holds a payload under stamp A. A second stamp B arrives with a payload that loses the tie-break (for example an older timestamp). the only logical continuation is "nothing should happen", no?

B2 seems hypothetical and i'm not sure whether we should address it

B3 seems to change significantly the migration performance - we should test whether this makes sense and whether this has a significant impact on the migration time. there are other ways to achieve this same fix i suppose.

B4 seems hypothetical. is this even possible?

[martinconic]

bee-5544-fixes.md

B1 how is this possible? isn't this the most basic case we are catering for here? reading the first sentence A SOC address already holds a payload under stamp A. A second stamp B arrives with a payload that loses the tie-break (for example an older timestamp). the only logical continuation is "nothing should happen", no?

B2 seems hypothetical and i'm not sure whether we should address it

B3 seems to change significantly the migration performance - we should test whether this makes sense and whether this has a significant impact on the migration time. there are other ways to achieve this same fix i suppose.

B4 seems hypothetical. is this even possible?

Hi, here is the context behind each of those points:

B1 (Why "nothing should happen" breaks convergence)

In Swarm, postage stamps sign the chunk address, not the payload content. A SOC address can be funded simultaneously by multiple batches.
If a new stamp arrives with a payload that loses the tie-break:

1. Arrival-order divergence:
   • Node 1 receives Stamp A (older) then Stamp B (newer) → retains the newer payload and holds 2 stamps (A and B).
   • Node 2 receives Stamp B (newer) then Stamp A (older) → if it does "nothing", it only holds 1 stamp (B only).
     Their reserve sizes, radiuses, and pullsync offers diverge based purely on network arrival order.
2. Premature eviction: If Node 2 drops Stamp A, then when Batch B expires, Node 2 deletes the chunk from disk even though Batch A is active and fully paid for.
3. The B1 fix: We keep the winning payload on disk and record Stamp B's entry with entrySum = ChunkSum(Stamp B, winningPayload). Because Stamp B now references the stored payload, we must call ChunkStore().Put(ctx, stored) to increment the refcount; otherwise, evicting Batch A would delete the payload from under Stamp B.

B2 (Self-healing missing sum index rows)

Under clean operation this is rare, but if a node crashes mid-migration or during db repair-reserve, an entry can exist without its ChunkSumItem. Previously, putChunk treated this as a divergence and rejected identical SOCs with ErrDivergentChunkRejected or CACs with "not a single owner chunk", causing pullsync to spin in an infinite re-want loop. restoreEntrySum is a lightweight self-healing fallback that repairs the index row when the payload matches.

B3 (Migration performance)

Migration performance is not impacted for standard nodes:

• The 99.999% happy path (valid chunks being backfilled with sums) remains 100% batched in 1,000-item transactions (st.Run).
• Single-item transactions are only executed when an entry is marked for deletion (if remove), which is rare (orphans/corrupted chunks).
• Removals require single-item transactions because batched transactions cannot see pending refcount decrements on shared SOC addresses, which would leak Sharky slots (identical to how EvictBatchBin isolates deletions).

B4 (Zero-hash stamp cleanup)

The check bytes.Equal(item.StampHash, swarm.EmptyAddress.Bytes()) was already added to step_08.go in July to prune legacy records with unpopulated stamp hashes. However, the existing code called RemoveChunkWithItem, which looked up stamps by hash; with an all-zero hash, it returned nil and silently skipped deleting the stampindex row. This left poisoned slots that caused subsequent writes to fail with not found. B4 simply ensures the cleanup that step_08 was already doing completes cleanly.

[gacevicljubisa]

if no error is returned, maybe metrics that count skipped could be beneficial to check offer. Also, at the end of the method we could still write the error and return top most?

[gacevicljubisa]

unused parameter: sum

[gacevicljubisa]

This is already called in the soc.Valid(ch)

[gacevicljubisa]

Instead of last write wins, would it be better that we have smaller hash wins, which is deterministic for all, something like this:

			} else if bytes.Compare(oldStampIndex.StampHash, stampHash) < 0 {
				return false, fmt.Errorf("stamp collision chunk %s lost stamp-hash tie-break: %w", chunk.Address(), storage.ErrDivergentChunkRejected)
			}

[sbackend123]

I do not remember exactly why, but we decided do not compare stamp hashes. I remember there was long discussion on calls with Elad also. May be @martinconic remembers better?

[gacevicljubisa]

If swarm address is not necesary to be print out in the log, by using struct{}, you could save over 300MB of heap memory...