How do I upload an object to S3 in a cloudflare worker? #4256

vinaysshenoy · 2022-10-28T07:12:51Z

vinaysshenoy
Oct 28, 2022

I am trying to upload a file to S3 using the v3 JavaScript SDK. I have been following the example on https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/s3-example-photo-album.html#s3-example-photo-album-adding-photos to setup the integration.

However at the time of making the request, the server fails with the following error:

✘ [ERROR] [plugin checkForNodeBuiltins] Detected a Node builtin module import while Node compatibility is disabled.
Add node_compat = true to your wrangler.toml file to enable Node compatibility.

    node_modules/@aws-sdk/util-user-agent-node/dist-cjs/index.js:5:21:
      5 │ const os_1 = require("os");
        ╵                      ~~~~

  This error came from the "onResolve" callback registered here:

    node_modules/wrangler/wrangler-dist/cli.js:142347:11:
      142347 │     build6.onResolve(
             ╵            ~~~~~~~~~

It seems like the AWS SDK is trying to upload the file using the NodeJS APIs instead of the browser APIs. I am not sure if I am missing something about how I am using it. Here are the relevant parts of my code (setting up of the S3 client and the actual upload parts)

const {
    S3Client,
    PutObjectCommand
} = require("@aws-sdk/client-s3")

const S3 = new S3Client({
    region: 'auto',
    endpoint: AWS_S3_ENDPOINT,
    credentials: {
        accessKeyId: AWS_ACCESS_KEY_ID,
        secretAccessKey: AWS_SECRET_ACCESS_KEY
    }
})

const uploadObjectParams = {
    	Bucket: 'prod-assets',
    	Key: `projects/templates/${projectInfo.id}`,
        // This is a `File` object read from `FormData`
    	Body: body['project_archive']
}

await S3.send(new PutObjectCommand(uploadObjectParams))

ajredniwja · 2022-12-10T05:22:25Z

ajredniwja
Dec 10, 2022

Hey @vinaysshenoy thanks for opening this discussion, I see you are using a V3 of the SDK. Unlike SDK for JavaScript version 2 (V2), V3 is not provided as a JavaScript file with support included for a default set of services. Instead V3 enables you to bundle and include in the browser only the SDK for JavaScript files you require, reducing overhead.

With that being said I am not sure at what step do you see this error, the error doesn't indicate it to be SDK error. I was able to take the example and work through it with some troubles with the dependencies. Below is the files I used. Can you try to run this by changing bucket name and region in s3.js.
Then running:
npm install
npm run build
open index.html

index.html

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <script src="./bundle.js"></script>
</head>

<body>
    <p><input type="file" accept="image/*" name="image" id="photoupload" onchange="addPhoto()"></p>
    <h1>My photo </h1>
</body>

</html>

s3.js

async function addPhoto() {
    const files = document.getElementById("photoupload").files;
    try {
        // const albumPhotosKey = encodeURIComponent(albumName) + "/";
        const file = files[0];
        const fileName = file.name;
        const photoKey = fileName;
        const uploadParams = {
            Bucket: "bucket",
            Key: photoKey,
            Body: file
        };
        try {
            const S3 = new S3Client({
                region: 'us-west-2',
                // endpoint: AWS_S3_ENDPOINT,
                credentials: credentials,
            })
            await S3.send(new PutObjectCommand(uploadParams));
            alert("Successfully uploaded photo.");
            // viewAlbum(albumName);
        } catch (err) {
            console.log(err)
            return alert("There was an error uploading your photo: ", err.message);
        }
    } catch (err) {
        if (!files.length) {
            return alert("Choose a file to upload first.");
        }
    }
};
// Make addPhoto function available to the browser
window.addPhoto = addPhoto;

webpack.config.js

// Import path for resolving file paths
var path = require("path");
const webpack = require('webpack');
module.exports = {


    // Specify the entry point for our app.
    entry: [path.join(__dirname, "./s3.js")],
    // Specify the output file containing our bundled code.
    output: {
        path: __dirname,
        filename: 'bundle.js',
        // library: 'test',
        // libraryTarget: 'window',
        libraryExport: 'default',
    },
    // Enable WebPack to use the 'path' package.
    resolve: {
        fallback: { path: require.resolve("path-browserify") }
    },

    mode: 'development',

    plugins: [
        new webpack.ProvidePlugin({
            Buffer: ['buffer', 'Buffer'],
        })
    ],
};

package.json

{
    "name": "browser-simple",
    "version": "1.0.0",
    "description": "",
    "main": "s3.js",
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "build": "webpack --config webpack.config.js"
    },
    "author": "",
    "license": "ISC",
    "devDependencies": {
        "path-browserify": "^1.0.1",
        "webpack": "^5.75.0",
        "webpack-cli": "^5.0.1"
    },
    "dependencies": {
        "@aws-sdk/client-s3": "^3.226.0",
        "@aws-sdk/lib-storage": "^3.226.0",
        "buffer": "^6.0.3"
    }
}

Bundling the SDK for browsers
Bundling applications with webpack
Cloudflare issue

4 replies

vinaysshenoy Dec 10, 2022
Author

@ajredniwja Thanks for the reply!

One thing which I forgot to mention is that I was trying to do this in a "Worker" environment (specifically Cloudflare workers). This seems to have been the issue since Cloudflare workers are neither a browser nor a NodeJS environment. The node compatibility flag which they provide does not seem to do what I assumed it would do (i.e, run it in a full node environment).

We tried a couple of things to make it work (including using the v2 SDK) but that failed as well. We took the decision to migrate to a full NodeJS environment (which we are running smoothly on Elastic Beanstalk now!), which is working as expected!

vinaysshenoy Dec 12, 2022
Author

The error also happens at the time of build because the CF Worker CLI tool (Wrangler) does some sort of compile time checks to verify whether the app is using any unsupported APIs (which is where this error was being thrown).

ajredniwja Jan 24, 2023

I am not familiar with using Cloudflare worker, I am glad you were able to run it well on Elastic Beanstalk. However I would need to reproduce it to try to find the problem if there is any for that would need some reproduction steps.

vinaysshenoy Jan 25, 2023
Author

The only way you can reproduce it is by running the SDK on Cloudflare Workers. This is because Workers there are neither a Node nor a Browser environment.

I guess I didn't articulate the resolution correctly, but the crux of the matter is that the problem here is not with the AWS SDK but with Cloudflare Workers. Unless of course, Workers are a supported target for the AWS SDK, which I don't think is the case.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

How do I upload an object to S3 in a cloudflare worker? #4256

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

How do I upload an object to S3 in a cloudflare worker? #4256

Uh oh!

Uh oh!

vinaysshenoy Oct 28, 2022

Replies: 1 comment · 4 replies

Uh oh!

ajredniwja Dec 10, 2022

Uh oh!

vinaysshenoy Dec 10, 2022 Author

Uh oh!

vinaysshenoy Dec 12, 2022 Author

Uh oh!

ajredniwja Jan 24, 2023

Uh oh!

vinaysshenoy Jan 25, 2023 Author

vinaysshenoy
Oct 28, 2022

Replies: 1 comment 4 replies

ajredniwja
Dec 10, 2022

vinaysshenoy Dec 10, 2022
Author

vinaysshenoy Dec 12, 2022
Author

vinaysshenoy Jan 25, 2023
Author