Built cloudserver client with Smithy #1

SylvainSenechal · 2025-10-02T14:39:04Z

Related : scality/backbeat#2679
Replacing the "Backbeat client" built with the AWS SDK V2 in Backbeat, with this new client built with Smithy, related to our migration to the AWS SDK V3

ISSUE: CLDSRVCLT-1

LICENSE

SylvainSenechal · 2025-10-02T14:50:33Z

legacy/backbeat-2017-07-01.api.json

I think its nice to have this here as a reference, let me know if you think we should remove it

it is still in (backbeat) git history, should be enough I think.
would be nice however to link to backbeat file on the PR message and/or commit message

👍 agree to delete it

build/smithy/classpath.json

localTests/initBucketForTests.ts

francoisferrand · 2025-10-02T16:22:30Z

build/smithy/source/sources/getMetadata.smithy

should this "build" folder be committed?

looking at package.json, it seems we rebuild them anyway...

We need to have smithy installed to generate the build folder, I feel like it would be more complex if it's not comited ? At what moment would the build generation happen ?
I'm not too sure about this

Should be in the CI to be honest and the pushed with the release

localTests/initBucketForTests.ts

francoisferrand · 2025-10-02T16:30:21Z

service/cloudserver.smithy

+@restJson1
+@sigv4(name: "s3")
+@service(sdkId: "cloudserver")
+service cloudserver {


looking at the bigger picture (i.e. all the APIs supported by cloudserver), this is only one part : i.e. the "Backbeat routes".

eventually (not in this PR ;) we should have other APIs I guess, i.e.

custom listing and other "extensions" to s3 api with extra attributes/... (typically done with a middleware, not sure if/how we want to override or just provide an ready-to-use middleware)

metadata routes https://github.com/scality/cloudserver/blob/development/9.1/lib/routes/routeMetadata.js

backbeat api routes (as used in zenkoclient, i.e. just frontend for backbeat/api extension)

quota API

...

→ given this context, is cloudserver the right name for this specific service?

IMO cloudserver is fine. It's a first iteration so we don't have all of them and I think it's ok. Maybe we should add a TODO.md with which API is missing, the cleanup of the tests (assert of process.env) and the CI? And link them to JIRA ticket

README.md

package.json

build/smithy/classpath.json

francoisferrand · 2025-10-02T16:37:53Z

legacy/backbeat-2017-07-01.api.json

it is still in (backbeat) git history, should be enough I think.
would be nice however to link to backbeat file on the PR message and/or commit message

francoisferrand · 2025-10-02T16:43:08Z

tests/package.json

+  },
+  "dependencies": {
+    "@aws-sdk/client-s3": "^3.896.0",
+    "@scality/cloudserverclient": "file:../build/smithy/source/typescript-codegen"


this works in the same repo (file:..); how will it work from another repo, i.e. with git reference?

depending on the "internal" generation path of the package does not look very good..... could we/should we "wrap" this with a top-level file -maybe manually generated- re-exporting tuff from build/smithy/source/typescript-codegen ? So that other packages could directly use the package.json defined at root directory?

alternatively, we may push the registry... c.f. scality/Guidelines#206
(may be done in a followup PR, adding github actions workflows to build/... everything)

I tried importing it from backbeat
It worked with this :
"@scality/cloudserverclient": "file:../cloudserverclient"
What I did here in the package.json is if you look at "main", "types", "modules", I provided the path to the build folder :
"main": "build/smithy/source/typescript-codegen/dist-cjs"
So when we release a version of this repo, we should be able to use the library this way :
"@scality/cloudserverclient": "github:scality/cloudserverclient#v1.0.0"

Ended up changing it :
We now have a client wrapper setup at the root project, it was needed to be able to provide this client with the error wrapper needed to handle html/xml error parsing

francoisferrand

publishing some overall comments, did not yet review the tests or models yet

README.md

localTests/testApis.ts

README.md

build/smithy/source/build-info/smithy-build-info.json

tests/testMultipleBackend.test.ts

tests/testRaftApis.test.ts

francoisferrand · 2025-10-08T07:37:05Z

tests/testRaftApis.test.ts

+            const raftIdData = await client.send(getRaftIdCommand);
+            console.log('GetRaftId succeeded:', raftIdData);
+            console.log('Raft ID:', raftIdData.RaftId);
+        } catch (err) {


we should not catch the error, as it prevents the test framework from noticing : so the test will always succeed...

(if this is expected, best to just skip the test for now...)

Ideally these tests should be run in a pipeline as functional tests, for now I ran them manually and I think they are fine this way. I have another ticket for setting up a pipeline, although I think this will not be straightforward to test : We need cloudserver to be started with different setup depending on the api we want to test

tests/testSetup.ts

SylvainSenechal · 2025-10-23T07:51:34Z

src/index.ts

+                    });
+                }
+
+                // S3C has an nginx proxy that can return HTML error responses


Logic and comment from https://github.com/scality/backbeat/blob/66c19b092af86f90fe0cf22eb4ce25b6a28cddbf/lib/clients/BackbeatClient.js#L30

francoisferrand · 2025-10-29T17:46:49Z

models/multipleBackendPutObject.smithy

+
+list LocationMDList {
+    member: LocationMDObj
+}
+
+structure LocationMDObj {
+    /// Storage key for this location
+    key: String,
+    /// Size of the data stored at this location
+    size: Integer,
+    /// Start position/offset for this data segment
+    start: Integer,
+    /// Name of the data store where this is located
+    dataStoreName: String,
+    /// Type of the data store (e.g., file, mem, etc.)
+    dataStoreType: String,
+    /// ETag from the data store for this location
+    dataStoreETag: String,
+    /// Version ID in the data store for this location
+    dataStoreVersionId: String
+}


nit: these are shared types, can they be moved to a separate smithy file? (locationMD.smithy)

francoisferrand · 2025-10-29T17:48:14Z

models/multipleBackendPutMPUPart.smithy

+blob StreamingBlob
+
+/// Uploads a part for a multipart upload to multiple backend storage
+@idempotent


does this not create a new part on every call, making it non-idempotent?

models/multipleBackendPutObject.smithy

models/multipleBackendCompleteMPU.smithy

models/multipleBackendAbortMPU.smithy

models/listLifecycleOrphans.smithy

models/listLifecycleNonCurrents.smithy

models/getObjectList.smithy

francoisferrand · 2025-10-29T18:03:46Z

models/getObject.smithy

+    output: GetObjectOutput,
+}
+
+structure GetObjectInput {


is this not the same type used in AWS ?
could we import those types and inherit from them (for custom/extra fields we support)?

something like:

structure GetObjectInput extends aws.GetObjectInput { @httpHeader("X-Scal-Request-Uids") RequestUids: String }

it look similar as the regular getObject input but it doesn't seem possible to do such an "extend". I prefer to keep this api fully custom. What we should've maybe looked at is if we can remove this api and use a classic getObject

models/deleteObjectFromExpiration.smithy

francoisferrand · 2025-10-29T18:06:57Z

models/deleteBucketIndexes.smithy

+$version: "2.0"
+namespace cloudserver.client
+
+@http(method: "POST", uri: "/_/backbeat/index/{Bucket}?operation=delete")


don't we need this:

@idempotent @suppress(["HttpMethodSemantics.UnexpectedPayload"])

(it has a body in the definition, and delete should be idempotent...)
(or maybe there should not be a body?)

I added idempotent, suppress not needed, there is no warning as the api verb is not delete/put

francoisferrand · 2025-10-29T18:09:30Z

tsconfig.json

+    "target": "es2020",
+    "module": "commonjs",
+    "lib": ["es2020"],
+    "outDir": "./dist",


nit: should we output to ./buid instead, so all the generated code is in a single place?

francoisferrand · 2025-10-29T18:19:16Z

LGTM, some cleanup (extra space / missing line in models...) to do.

Not sure what is the effect of @idempotent, but in some case this seems unclear if appropriate: so we may want to dig a bit more

DarkIsDude · 2025-10-31T14:21:08Z

tests/testApis.test.ts

@@ -0,0 +1,105 @@
+import {


As we are in a test folder, we can just use test.apis.ts or test.apisjs.js. spec.ts is used when test file are located next to the src ?

DarkIsDude · 2025-10-31T14:23:26Z

service/cloudserver.smithy

+@restJson1
+@sigv4(name: "s3")
+@service(sdkId: "cloudserver")
+service cloudserver {


IMO cloudserver is fine. It's a first iteration so we don't have all of them and I think it's ok. Maybe we should add a TODO.md with which API is missing, the cleanup of the tests (assert of process.env) and the CI? And link them to JIRA ticket

SylvainSenechal · 2025-11-07T15:49:43Z

Checklist:

Cloudserver client renaming ?
Extracting the error middleware ?
Group api by theme or a-z ?
Idempotent impact

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch from 920e18a to 356c165 Compare October 2, 2025 14:46

SylvainSenechal commented Oct 2, 2025

View reviewed changes

LICENSE Show resolved Hide resolved

SylvainSenechal commented Oct 2, 2025

View reviewed changes

build/smithy/classpath.json Show resolved Hide resolved

SylvainSenechal commented Oct 2, 2025

View reviewed changes

localTests/initBucketForTests.ts Outdated Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch from 8101fc0 to 103d2ed Compare October 2, 2025 15:22

francoisferrand reviewed Oct 2, 2025

View reviewed changes

README.md Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 3 times, most recently from c8442a6 to c6c998f Compare October 2, 2025 20:27

francoisferrand reviewed Oct 3, 2025

View reviewed changes

localTests/testApis.ts Outdated Show resolved Hide resolved

BourgoisMickael reviewed Oct 3, 2025

View reviewed changes

README.md Outdated Show resolved Hide resolved

BourgoisMickael reviewed Oct 3, 2025

View reviewed changes

README.md Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 3 times, most recently from 766fbca to 125bc0d Compare October 6, 2025 12:49

francoisferrand reviewed Oct 8, 2025

View reviewed changes

build/smithy/source/build-info/smithy-build-info.json Show resolved Hide resolved

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testMultipleBackend.test.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testRaftApis.test.ts Outdated Show resolved Hide resolved

francoisferrand reviewed Oct 8, 2025

View reviewed changes

tests/testSetup.ts Outdated Show resolved Hide resolved

SylvainSenechal force-pushed the improvement/CLDSRVCLT-1 branch 5 times, most recently from 0e5f323 to dfc7d0f Compare October 23, 2025 07:48

SylvainSenechal commented Oct 23, 2025

View reviewed changes