Skip to content

feat: Allow File adapter to create file with specific locations or dynamic filenames #9557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: alpha
Choose a base branch
from
Open

feat: Allow File adapter to create file with specific locations or dynamic filenames #9557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
0b79b20
Allow file adapter to override file name and location + give createFi…
AdrianCurtin Jan 16, 2025
2036369
Only call getFileLocation if no url is provided
AdrianCurtin Jan 16, 2025
1df8f1a
Modify description and typescript
AdrianCurtin Jan 17, 2025
a9381bc
Add optional location element too
AdrianCurtin Jan 17, 2025
46b1e14
Update FilesAdapter.js
AdrianCurtin Jan 17, 2025
7c4e69f
Remove trialing space
AdrianCurtin Jan 17, 2025
c87b86d
change to undefined as a fallback
AdrianCurtin Jan 17, 2025
ce2c8f9
Add tests for different return conditions
AdrianCurtin Jan 17, 2025
6298c0c
Update description of config slightly
AdrianCurtin Jan 17, 2025
657dba6
Modify mock adapter initialization to use struct in beginning
AdrianCurtin Feb 1, 2025
39f48bd
Update to include config in arguments for createFileSpy
AdrianCurtin Feb 1, 2025
59ed527
Remove space + move preserve filename args to files controller initia…
AdrianCurtin Feb 7, 2025
3fee23a
Split into separate test
AdrianCurtin Aug 13, 2025
623d083
Limit the file adapter config to just the minimum it needs to get loc…
AdrianCurtin Aug 13, 2025
a0a1782
Match the actual test keys
AdrianCurtin Aug 13, 2025
ff23067
Change domain to example
AdrianCurtin Aug 14, 2025
6114fbf
Use config and stripped down config
AdrianCurtin Aug 14, 2025
028103a
Rename to basicServerConfig
AdrianCurtin Aug 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions spec/CloudCode.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3695,11 +3695,21 @@ describe('saveFile hooks', () => {
foo: 'bar',
},
};
// Get the actual config values that will be used
const config = Config.get('test');

const expectedConfig = {
applicationId: config.applicationId,
mount: config.mount,
fileKey: config.fileKey
};

expect(createFileSpy).toHaveBeenCalledWith(
jasmine.any(String),
newData,
'text/plain',
newOptions
newOptions,
expectedConfig
);
});

Expand Down Expand Up @@ -3727,11 +3737,20 @@ describe('saveFile hooks', () => {
foo: 'bar',
},
};
const config = Config.get('test');

const expectedConfig = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you import this from the test suite where it's defined?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated it, but previously the checks were unhappy with the full config being passed through, I don't recall why off the top of my head

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it but i still stripped it down because previously the config had been modified through the process or something. We'll see if it passes CI, but Maybe the check just needs to be loosened here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you don't mind assisting,

when i revert to the config('test") i end up with these errors

`

  1. saveFile hooks beforeSave(Parse.File) should change values of uploaded file by editing fileObject directly
  • Expected spy createFile to have been called with:
    [ <jasmine.any(String)>, Buffer [ 1, 2, 3 ], 'text/plain', Object({ tags: Object({ tagA: 'some-tag' }), metadata: Object({ foo: 'bar' }) }), Object({ applicationId: 'test', mount: undefined, fileKey: 'test' }) ]
    but actual calls were:
    [ '7e441dcb3fc497b19592367415376ea1_popeye.txt', Buffer [ 1, 2, 3 ], 'text/plain', Object({ metadata: Object({ foo: 'bar' }), tags: Object({ tagA: 'some-tag' }) }), Object({ applicationId: 'test', mount: 'http://localhost:8378/1', fileKey: 'test' }) ].

Call 0:
Expected $[4].mount = 'http://localhost:8378/1' to equal undefined.

  1. saveFile hooks beforeSave(Parse.File) should contain metadata and tags saved from client
  • Expected spy createFile to have been called with:
    [ <jasmine.any(String)>, <jasmine.any(Buffer)>, 'text/plain', Object({ metadata: Object({ foo: 'bar' }), tags: Object({ bar: 'foo' }) }), Object({ applicationId: 'test', mount: undefined, fileKey: 'test' }) ]
    but actual calls were:
    [ '352ace761fac45c85ba556c85548888d_popeye.txt', Buffer [ 1, 2, 3 ], 'text/plain', Object({ metadata: Object({ foo: 'bar' }), tags: Object({ bar: 'foo' }) }), Object({ applicationId: 'test', mount: 'http://localhost:8378/1', fileKey: 'test' }) ].

Call 0:
Expected $[4].mount = 'http://localhost:8378/1' to equal undefined.

  1. saveFile hooks beforeSave(Parse.File) should change values by returning new fileObject
  • Expected spy createFile to have been called with:
    [ <jasmine.any(String)>, Buffer [ 4, 5, 6 ], 'application/pdf', Object({ tags: Object({ tagA: 'some-tag' }), metadata: Object({ foo: 'bar' }) }), Object({ applicationId: 'test', mount: undefined, fileKey: 'test' }) ]
    but actual calls were:
    [ 'b048f76b35f8628bf9336c5efdd0eb95_donald_duck.pdf', Buffer [ 4, 5, 6 ], 'application/pdf', Object({ metadata: Object({ foo: 'bar' }), tags: Object({ tagA: 'some-tag' }) }), Object({ applicationId: 'test', mount: 'http://localhost:8378/1', fileKey: 'test' }) ].

Call 0:
Expected $[4].mount = 'http://localhost:8378/1' to equal undefined.
`

applicationId: config.applicationId,
mount: config.mount,
fileKey: config.fileKey
};

expect(createFileSpy).toHaveBeenCalledWith(
jasmine.any(String),
newData,
newContentType,
newOptions
newOptions,
expectedConfig
);
const expectedFileName = 'donald_duck.pdf';
expect(file._name.indexOf(expectedFileName)).toBe(file._name.length - expectedFileName.length);
Expand All @@ -3757,11 +3776,20 @@ describe('saveFile hooks', () => {
metadata: { foo: 'bar' },
tags: { bar: 'foo' },
};
const config = Config.get('test');

const expectedConfig = {
applicationId: config.applicationId,
mount: config.mount,
fileKey: config.fileKey
};

expect(createFileSpy).toHaveBeenCalledWith(
jasmine.any(String),
jasmine.any(Buffer),
'text/plain',
options
options,
expectedConfig
);
});

Expand Down
98 changes: 98 additions & 0 deletions spec/FilesController.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,4 +218,102 @@ describe('FilesController', () => {
expect(gridFSAdapter.validateFilename(fileName)).not.toBe(null);
done();
});

it('should return filename and url when adapter returns both', async () => {
const config = Config.get(Parse.applicationId);
const adapterWithReturn = { ...mockAdapter };
adapterWithReturn.createFile = () => {
return Promise.resolve({
name: 'newFilename.txt',
url: 'http://example.com/newFilename.txt'
});
};
adapterWithReturn.getFileLocation = () => {
return Promise.resolve('http://example.com/file.txt');
};
const controllerWithReturn = new FilesController(adapterWithReturn, null, { preserveFileName: true });

const result = await controllerWithReturn.createFile(
config,
'originalFile.txt',
'data',
'text/plain'
);

expect(result.name).toBe('newFilename.txt');
expect(result.url).toBe('http://example.com/newFilename.txt');
});

it('should use original filename and generate url when adapter returns nothing', async () => {
const config = Config.get(Parse.applicationId);
const adapterWithoutReturn = { ...mockAdapter };
adapterWithoutReturn.createFile = () => {
return Promise.resolve();
};
adapterWithoutReturn.getFileLocation = (config, filename) => {
return Promise.resolve(`http://example.com/${filename}`);
};

const controllerWithoutReturn = new FilesController(adapterWithoutReturn, null, { preserveFileName: true });
const result = await controllerWithoutReturn.createFile(
config,
'originalFile.txt',
'data',
'text/plain',
{}
);

expect(result.name).toBe('originalFile.txt');
expect(result.url).toBe('http://example.com/originalFile.txt');
});

it('should use original filename when adapter returns only url', async () => {
const config = Config.get(Parse.applicationId);
const adapterWithOnlyURL = { ...mockAdapter };
adapterWithOnlyURL.createFile = () => {
return Promise.resolve({
url: 'http://example.com/partialFile.txt'
});
};
adapterWithOnlyURL.getFileLocation = () => {
return Promise.resolve('http://example.com/file.txt');
};

const controllerWithPartial = new FilesController(adapterWithOnlyURL, null, { preserveFileName: true });
const result = await controllerWithPartial.createFile(
config,
'originalFile.txt',
'data',
'text/plain',
{}
);

expect(result.name).toBe('originalFile.txt');
expect(result.url).toBe('http://example.com/partialFile.txt');
});

it('should use adapter filename and generate url when adapter returns only filename', async () => {
const config = Config.get(Parse.applicationId);
const adapterWithOnlyFilename = { ...mockAdapter };
adapterWithOnlyFilename.createFile = () => {
return Promise.resolve({
name: 'newname.txt'
});
};
adapterWithOnlyFilename.getFileLocation = (config, filename) => {
return Promise.resolve(`http://example.com/${filename}`);
};

const controllerWithOnlyFilename = new FilesController(adapterWithOnlyFilename, null, { preserveFileName: true });
const result = await controllerWithOnlyFilename.createFile(
config,
'originalFile.txt',
'data',
'text/plain',
{}
);

expect(result.name).toBe('newname.txt');
expect(result.url).toBe('http://example.com/newname.txt');
});
});
8 changes: 5 additions & 3 deletions src/Adapters/Files/FilesAdapter.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,12 +31,14 @@ export class FilesAdapter {
* @discussion the contentType can be undefined if the controller was not able to determine it
* @param {object} options - (Optional) options to be passed to file adapter (S3 File Adapter Only)
* - tags: object containing key value pairs that will be stored with file
* - metadata: object containing key value pairs that will be sotred with file (https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-object-metadata.html)
* - metadata: object containing key value pairs that will be stored with file (https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-object-metadata.html)
* @discussion options are not supported by all file adapters. Check the your adapter's documentation for compatibility
* @param {Config} config - (Optional) server configuration
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would the whole Parse Server config be passed into the adapter? Or am I misreading this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mtrezza
Allowing the server config to be passed to the createFile would allow a couple of things.

  1. files can be named based on serer attributes (ex: generateKey could store log files with the app id)
  2. getFileLocation can be called within createFile, either to test or check that the file has been created (especially if getFileLocation depends on the server config which it always has access to)

There may be other situations in which the server config may be useful apart from customizing file paths and running getFileLocation, but I don't see why createFile shouldn't have access to it if getFileLocation does.

If this was a smaller project I would probably have refactored adapter.createFile to use config as the first argument, but that had the potential to be fairly breaking and maybe its nicer for some people to use X.createFile('filename')

In any case tacking it at the end was the compromise I made.

In the current S3-adapter PR, this only allows getFileLocation to run inside and I didn't pass the parameters to generateKey.

I would say though that removing that expectation and removing getFileLocation from the adapter would make the changes more explicit, but then [location] = createFile would not always match the call from getFileLocation (if it depended in any way on server parameters)

Copy link
Member

@mtrezza mtrezza Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is passing the server config to the adapter l a good approach? It creates a dependency on a specific config structure. If that structure changes, it could break the adapter, right? Or is the config already used in other parts of the adapter?

Copy link
Author

@AdrianCurtin AdrianCurtin Aug 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mtrezza

The file location is based on the server config as is, so when you call getFileLocation you need the server config.
You only need the mount and the app id though for most getFileLocation calls, but the adapter gets the full server config in those cases.

So we can't actually know where the file ends up without the server config unfortunately (if it ends up being different from the getFileLocation)

We can opt instead for the full server config to pass through but that was making the tests unhappy a little and I haven't seen any instances of getFileLocation use more than just those parameters.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the basic idea:

FilesController.createFile returns url (location) and filename

File location is determined by the file name and the server configuration

but adapter.createFile does not have access to the server config and therefore cannot return the stored file config if the filename or any other part has changed.

With this change, the expectation is that createFile might return an updated filename (if for instance generatekey was used to alter the filepath), and optionally, createFile could return the entire location (presumably by calling getFileLocation internally.

If createFile returns nothing (current implementation), the original filename will be returned and getFileLocation will be called using the original filename (after creation though instead of before). <- no breaking changes here

If a file adapter chooses to alter the filename, but does not internally implement getFileLocation and return a url, this version will simply call getFileLocation but with the revised filename so that it can be correctly found. Adapters would be responsible for returning altered filenames when they are changed and this is the current issue with the s3 adapter issue.

If a file adapter chooses to internally call getFileLocation or a similar function to return the URL, then we can simply return the location that was part of createFile instead of calling a separate function. (File adapters would be responsible for ensuring that the returned url would be equivalent to the getFileLocation url, again this is opt-in).

If a file adapter chose to alter the filename or some other aspect of file creation based on the server configuration (ex: s3 folders based on application id), it would be the file adapter's responsibility to ensure that the create file behavior was not brittle. However, this complexity is especially opt-in and would not be advised for anything beyond altering the filename if desired.

For most adapters, the best course of action here is to simply update them to return the filename as stored (if the adapter is one that might change the filename).

For some adapters the getFileLocation might simply be returning that file directly with some modification, or others like presigning might be more involved. Implementing this internally may result in performance increases (small likely) but also is not strictly necessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mtrezza passing the whole config to an adapter don't feel strange to me, it allow a true inversion of control, yes it can break in case of config change, but a developer is responsible of testing its custom adapter basically

* @discussion config may be passed to adapter to allow for more complex configuration and internal call of getFileLocation (if needed). This argument is not supported by all file adapters. Check the your adapter's documentation for compatibility
*
* @return {Promise} a promise that should fail if the storage didn't succeed
* @return {Promise<{url?: string, name?: string, location?: string}>|Promise<undefined>} Either a plain promise that should fail if storage didn't succeed, or a promise resolving to an object containing url and/or an updated filename and/or location (if relevant)
*/
createFile(filename: string, data, contentType: string, options: Object): Promise {}
createFile(filename: string, data, contentType: string, options: Object, config: Config): Promise {}

/** Responsible for deleting the specified file
*
Expand Down
16 changes: 13 additions & 3 deletions src/Controllers/FilesController.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,20 @@ export class FilesController extends AdaptableController {
filename = randomHexString(32) + '_' + filename;
}

const location = await this.adapter.getFileLocation(config, filename);
await this.adapter.createFile(filename, data, contentType, options);
// Create a clean config object with only the properties needed by file adapters
const basicServerConfig = {
applicationId: config.applicationId,
mount: config.mount,
fileKey: config.fileKey
};
Comment on lines +33 to +37
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: why passing a partial config, we could land with inconsistencies, i think the whole config should be passed for a true inversion of control


const createResult = await this.adapter.createFile(filename, data, contentType, options, basicServerConfig);
filename = createResult?.name || filename; // if createFile returns a new filename, use it

const url = createResult?.url || await this.adapter.getFileLocation(basicServerConfig, filename); // if createFile returns a new url, use it otherwise get the url from the adapter

return {
url: location,
url: url,
name: filename,
}
}
Expand Down