fix putData : detect collision on specific provided version id - #6230
fix putData : detect collision on specific provided version id#6230SylvainSenechal wants to merge 1 commit into
Conversation
Hello sylvainsenechal,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
| errorInstances.BadRequest.customizeDescription('bad request: invalid x-scal-version-id header'), | ||
| ); | ||
| } | ||
| if (objMd && objMd.versionId === incomingVersionIdDecoded) { |
There was a problem hiding this comment.
No need for objMd.versionId === incomingVersionIdDecoded anymore as the middleware is already fetching objMd for the specific versionID provided in the header
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files
@@ Coverage Diff @@
## development/9.4 #6230 +/- ##
===================================================
+ Coverage 86.39% 86.40% +0.01%
===================================================
Files 212 212
Lines 14577 14582 +5
===================================================
+ Hits 12594 12600 +6
+ Misses 1983 1982 -1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
148ca98 to
18d305d
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes Backbeat putData collision detection for replication of non-current object versions by ensuring the metadata lookup targets the specific replicated version (from x-scal-version-id) rather than always using the master version.
Changes:
- Decode
x-scal-version-idin the Backbeat router forPUT /_/backbeat/dataand use it as theversionIdfor metadata lookup soobjMdcorresponds to the replicated version. - Simplify collision detection in
putDatato treat the presence of that specific version’sobjMdas a collision (while explicitly excludingExternalNullVersionId/"null"). - Add functional test coverage for collisions on a non-current version and adjust the
"null"version test to cover cases where a master already exists.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/functional/backbeat/putData.js | Adds/updates functional tests to validate collision behavior for non-current versions and "null" version behavior. |
| lib/routes/routeBackbeat.js | Uses decoded x-scal-version-id to fetch version-specific metadata for putData, fixing collision detection semantics. |
maeldonn
left a comment
maeldonn
left a comment
There was a problem hiding this comment.
Logic is fine, need a little refactor
| } | ||
| if (objMd && objMd.versionId === incomingVersionIdDecoded) { | ||
| // Data already at destination for this version; return 409 with the existing | ||
| if (objMd) { |
There was a problem hiding this comment.
if (objMd) only means "this version exists" because the router's isPutDataApi block fetched the header version instead of the master; that coupling is easy to break. Clearer to drop the router block and fetch the version here with metadataGetObject (returns undefined when missing):
const incomingVersionId = decode(incomingVersionIdEncoded);
// ... BadRequest on decode error ...
return metadataGetObject(request.bucketName, request.objectKey, incomingVersionId, null, log, (err, versionMd) => {
if (err) {
return callback(err);
}
if (versionMd) {
// existing 409 conflict path, using versionMd.microVersionId
}
return writeData();
});There was a problem hiding this comment.
Mhh I'll let other reviewers comment on that but I don't fully agree :
Middleware/router is usually a good place to pre-extract generic info, here, the router already get the object after validating query params (bucket/object names), and it does it for all backbeat routes.
If we re add a metadataGetObject in this specific api, we are gonna call it twice because it's already called in the router.
I think it's an ok design to have the router handle the get object, the only thing a bit dirty is indeed that new logic where the router fetches an object specific version id only for putData
There was a problem hiding this comment.
I think this is not a theoretical problem, we need to be practical:
- the router makes one metadata call already; we must not make a second metadata to avoid modifying the router on principle. Metadata calls are costly at scale, we should not make a useless one.
- in addition, the router does auth check on the object it requests. So pointing at the wrong object could lead to incorrect auth (esp. in cross-account case)
- modifying the router must be done carefully though, to ensure we don't break any case: neither
putData, nor other cases - I think there is a case in putMetadata where we also make the call without versionId, and eventually need to make a second metadata call to get actual version. Bonus point if we can fix it at the same time (but anyway should look at it, as it could steer the change in another direction)
There was a problem hiding this comment.
Ok so lets keet it here, will recheck putmetadata
| // For putData api: the version to check is passed | ||
| // via x-scal-version-id header, not the URL query. Fetch that specific | ||
| // version so objMd matches the replicated version, not always the master. | ||
| const isPutDataApi = request.method === 'PUT' && request.resourceType === 'data'; |
There was a problem hiding this comment.
The router shouldn't special-case one handler, and these guards duplicate putData's. Move the version lookup into putData itself (see handler comment) so this block goes away.
There was a problem hiding this comment.
This is debatable if we consider that the router acts as a middleware that fetches objMd for all apis, in that case, we wouldn't want each apis functions to have their own extra ways of dealing with fetching again objMd.
Let's discuss with the first comment
There was a problem hiding this comment.
the router already has many cases, which are kind of hard-coded.
Maybe we can do this based on the presence of the header (if versionIdHeader ...)? I.e. set the query.versionId field from this header if it was not present already, before calling decodeVersionId ?
either way, even if we keep the "is put api" guard, no point doing both decodeVersionId and handling if this header:
| const isPutDataApi = request.method === 'PUT' && request.resourceType === 'data'; | |
| const versionId = ...... ? // whatever condition: header present, put api, ... | |
| decodeVersionId(request.query) : decodeScalVersionId(header); |
(or move it in a separate function, out of here)
There was a problem hiding this comment.
I checked cloudserverClient, we only have 3 other apis (MPUxx) using the versionId in header, and these request are treated differently. We can remove the "if isPutData" and do something simpler
c3d0e4b to
c9814a4
Compare
There was a problem hiding this comment.
General comment regarding Maël 3 feedbacks : I mostly agree with him that there is weirdness in the code changes, and that the current solution doesn't feel fully satisfying, but I don't think there is any perfect way to do it, else we would need a bigger refactoring
| incomingVersionIdEncoded !== 'null' ? decode(incomingVersionIdEncoded) : 'null'; | ||
| if (incomingVersionIdDecoded instanceof Error) { | ||
| // ExternalNullVersionId means a pre-versioning null object: collision detection is not applicable. | ||
| if (incomingVersionIdEncoded !== undefined && incomingVersionIdEncoded !== ExternalNullVersionId) { |
There was a problem hiding this comment.
abit hard to read but the diff is equivalent, in both case, the logic is :
"do not try to decode a 'null' versionId", as this would throw an error, and we dont wanna throw on 'null' versionId, they are just special case.
Prefer this new code, to avoid weird ternary and start using ExternalNullVersionId arsenal constant which will remind developers that 'null' versionId is a thing 👀
|
@francoisferrand added as Maël not here |
| if (objMd && objMd.versionId === incomingVersionIdDecoded) { | ||
| // Data already at destination for this version; return 409 with the existing | ||
| if (objMd) { | ||
| // objMd is the specific version (fetched by versionId from x-scal-version-id header) |
There was a problem hiding this comment.
not true : in case of decode error, you fallback to the "old" path.... so you may still receive an objMD from the master, even though a version id was provided.
i.e. same as https://github.com/scality/cloudserver/pull/6230/changes#r3637279080 : on header decode error the API should fail. It should not fallback to requesting master if a versionId is provided.
There was a problem hiding this comment.
you're right, considering that versionIdHeader is provided, it means the client specifically asks for a version ID, so now the middleware will return an error when failing to decode
| const incomingVersionIdDecoded = | ||
| incomingVersionIdEncoded !== 'null' ? decode(incomingVersionIdEncoded) : 'null'; | ||
| if (incomingVersionIdDecoded instanceof Error) { | ||
| // ExternalNullVersionId means a pre-versioning null object: collision detection is not applicable. |
There was a problem hiding this comment.
pre-versioning null object
that is called a "null version"
collision detection is not applicable
why? and especially how is it relevant to this block: here the condition just skip the decode() call, there is nothing about collision detection here.
| // ExternalNullVersionId means a pre-versioning null object: collision detection is not applicable. | |
| // ExternalNullVersionId means a null version, which does not need decoding |
I don't really mind removing the ternary, but it conveying the indent better: "if a versionId is set, decode it" (null version does not have a different meaning/intent, it is really an implementation detail of the decoding).
However I wonder if it really needs to be "decoded" as "null" : don't we use an empty string as the decoded null-version marker? (not an issue with this PR, but I wonder if there wasn't a bug here...)
There was a problem hiding this comment.
Yeah so ideally, decode should handle null version without throwing error, and be able to differentiate for us nullVersion/validDecoded/ErrorDecoded but I don't feel like changing decode, sounds like it could break a bunch of other stuff
There was a problem hiding this comment.
And you're right about the intent, its better to have
if versionId is defined
if versionId is a null version: continue
decode version id
than
if versionId is defined and versionId is not a null version
decode version id
result is the same but intent feels different
| // via x-scal-version-id header, not the URL query. Fetch that specific | ||
| // version so objMd matches the replicated version, not always the master. | ||
| const isPutDataApi = request.method === 'PUT' && request.resourceType === 'data'; | ||
| if (isPutDataApi) { |
There was a problem hiding this comment.
don't we have the same issue in putMetadata ?
e.g. if the router is not modified, that may also receive the "master" instead of the specific version, and thus not behave as expected...
There was a problem hiding this comment.
putMetadata is safe, versionId is provided by backbeat through query parameters, and objMd is queried based on that specific versionid.
Although, the current design is weird, we have a backbeat api that accepts versionIDs sometimes through headers (putdata), sometimes through query params (putMetadata)...
The header was actually added for crr cascade, I think the reason we used header was thats the encoded characters are better handled than in query params (because versoin id can contains whitespace), and also maybe something about header being directly readable without having to decode the whole body but Im not even sure this is true. Still, we end up with something thats not super clean, and had we use query params instead of header, wouldnt even need this pr
c9814a4 to
7129a18
Compare
7129a18 to
031c3c4
Compare
| ); | ||
| // ExternalNullVersionId means a null version, which does not need decoding | ||
| if (incomingVersionIdEncoded !== ExternalNullVersionId) { | ||
| if (objMd) { |
There was a problem hiding this comment.
Actually removed the whole decode here : It's already done in the middleware, we could almost do the whole remaining logic in the middleware but its probably not the right place
Issue: CLDSRV-953
The putData collision detection previously fetched the master object and compared its versionId against the incoming x-scal-version-id header. This caused issues when replicating a non-current version : the master's versionId wouldn't match, so no collision was detected and data was re-uploaded unnecessarily
Fix : decode the x-scal-version-id header in the router and use it as the metadata lookup key, so objMd is always the specific version being replicated