S3 V4 chunked upload - endpoint with bucket as directory (of the style s3.amazonaws.com/bucket.name/)

[d-a-s]

Type of issue

• Bug report

Uploader type

• S3

canonical header is not being generated correctly if the endpoint is of the style `s3.amazonaws.com/bucket.name/`

Fine Uploader version

{example: 5.15.0}

Browsers where the bug is reproducible

Chrome or Firefox (presumably all browsers)

Operating systems where the bug is reproducible

(presumably all, tested on Windows 10)

All relevant Fine Uploader-related code that you have written

    var getCpHeaders = function(si, key){
        var i = parseInt(si);
        if(i!==NaN) {
            var file = uploader.getFile(i);
            if(file && file.cpHeaders) {
                if(typeof key==='string') return file.cpHeaders[key] ||'';
                return file.cpHeaders;
            }
        }
        return null;
    };
    var uploader = new qq.s3.FineUploader({
        debug: true,
        element: document.getElementById('fine-uploader'),
        objectProperties: {
            key: function (i) {
                return getCpHeaders(i,'key');
            },
            serverSideEncryption: true
        },
        chunking: {
            enabled: true,
            concurrent: {
                enabled: true
            }
        },
        request: {
            accessKey: @Html.RawSerialize(Emr.CasePacer.Data.Services.S3.AccessKey)
        },
        signature: {
            customHeaders: function (i) {
                return getCpHeaders(i);
            },
            endpoint: '/Record/S3Signature',
            version: 4
        },
        uploadSuccess: {
            endpoint: '/Record/S3Success'
        },
        iframeSupport: {
            localBlankPagePath: '/success.html'
        },
        callbacks: {
            onSubmit: function (i, name) {
                var file = uploader.getFile(i);
                var promise = new qq.Promise();
                $.ajax({
                    method: 'POST',
                    url: '/Record/S3Initialize',
                    data: {
                        lawsuitId: $('#ActiveLawsuitId').val(),
                        fileName: file.name,
                        fileLength: file.size //, file.type
                    },
                    success: function (headers) {
                        if (headers && headers.key) {
                            file.cpHeaders = headers;
                            uploader.setEndpoint(headers.Endpoint, i);
                            promise.success();
                        } else promise.failure();
                    },
                    error: function () {
                        promise.failure();
                    }
                });
                return promise;
            }
        }
    });

Detailed explanation of the problem

If the endpoint is of the style s3.amazonaws.com/bucket.name/, the V4 signature expects the canonical header to be in the form:

PUT
/Bucket.Name/Path/To/File.txt
uploads=
host:s3.amazonaws.com
x-amz-content-sha256:...

However with the current version (5.15.0) the canonical header looks like this:

PUT
/Path/To/File.txt
uploads=
host:s3.amazonaws.com/Bucket.Name
x-amz-content-sha256:...

This causes the signature to be invalid. I have written a server-side workaround to fix the canonical header, but eventually it seems like it would be best to fix this in the code, at the following location possibly:

fine-uploader/client/js/s3/request-signer.js

Line 134 in a513777

getCanonicalRequest: function(signatureSpec) {

            getCanonicalRequest: function(signatureSpec) {
                return qq.format("{}\n{}\n{}\n{}\n{}\n{}",
                    signatureSpec.method,
                    v4.getCanonicalUri(signatureSpec.endOfUrl), // this should include /Bucket.Name/ at the beginning
                    v4.getCanonicalQueryString(signatureSpec.endOfUrl),
                    signatureSpec.headersStr || "\n", // if there is a host header, it should exclude the /Bucket.Name
                    v4.getSignedHeaders(signatureSpec.headerNames),
                    signatureSpec.hashedContent);
},

[bibidon]

Are there any updates or workarounds?

[d-a-s]

I ended up manually altering the canonical header on my server, just before i did all the hashing and other stuff. As I recall, it was as simple as taking the bucket name from the "host" line and moving it to the beginning of the path, which i think is the next line.

[bagr001]

It is the same problem as here #2010

[felipedeboni]

@bibidon replace the following functions:

getCanonicalRequest: function(signatureSpec) {
    return qq.format("{}\n{}\n{}\n{}\n{}\n{}", signatureSpec.method, v4.getCanonicalUri(signatureSpec.bucket,signatureSpec.endOfUrl), v4.getCanonicalQueryString(signatureSpec.endOfUrl), signatureSpec.headersStr || "\n", v4.getSignedHeaders(signatureSpec.headerNames), signatureSpec.hashedContent);
},
getCanonicalUri: function(bucket, endOfUri) {
    var path = endOfUri, queryParamIdx = endOfUri.indexOf("?");
    if (queryParamIdx > 0) {
        path = endOfUri.substr(0, queryParamIdx);
    }
    return "/" + bucket + "/" + path;
},

Also, look for requestInfo.headers.Host = requestInfo.host and replace with:

requestInfo.headers.Host = requestInfo.host.indexOf(requestInfo.bucket) > -1 ? requestInfo.host.replace("/" + requestInfo.bucket, '') : requestInfo.host;