[timelapse / a2] Support large file uploads for ETL script access

[rudokemper]

Feature Request

As previously noted here and here, we need to establish a process for uploading very large archive files to cloud storage and enabling access to those files by the Timelapse and Auditor2 ETL scripts (which does not exist yet - #97).

Using the file input in a Windmill app will not do for large files, so we'll likely need to handle uploads outside of Windmill. For example, we could upload files to Azure Blob Storage and then use the Python Azure Storage SDK within the scripts to access the files.

At this point, it's unclear whether support for Azure alone will be sufficient, or whether we should aim to support other cloud storage providers (e.g., S3, GCS) as well. The workflow should ideally be flexible enough to accommodate different backends if needed. But given that CMI is likely to be the only user of these scripts in the short-to-medium term, it's fine to start with Azure only.

In terms of the actual workflow: do we just upload files to blob storage manually, and provide whatever path and credentials are needed for the Python SDK when running or scheduling a script in Windmill? @IamJeffG I'm interested to learn what you've done in the past (feel free to just link to a code snippet if that's easiest).

To close this issue, let's:

• Agree upon the workflow
• Modify the scripts accordingly
• Provide necessary documentation

[rudokemper]

To look into -- does Windmill have a built-in ability to handle large files?

[rudokemper]

Some kind of process to delete a file (either usage of SDK with a write permission credential, or just a reminder to delete the file). Document this in README or maybe even the script YAML

[IamJeffG]

I created a blob container a2export in our bcmplayground Azure storage account.

In the windmill.demo deployment I also created an example Windmill script ("azure-blob-demo") and Windmill azure_blob resource that reads this container.

More docs about reading files from Azure Blob Storage.

[IamJeffG]

To look into -- does Windmill have a built-in ability to handle large files?

While https://www.windmill.dev/docs/core_concepts/files_binary_data mentions the "stream to browser" option we both saw before (a nonstarter for huge files), it also states you may "Have a dedicated storage for binary data: S3 or Azure Blob. Windmill has a first class integration with S3 buckets or Azure Blob containers."

Is that available on the free plan? I know we haven't set it up yet. It would mean we ought to dedicate a specific Azure Blob container for all Windmill uploads (which is fine). If indeed this works, very promising, and maybe preferred to the soln in my previous comment.

[rudokemper]

Workspace object storage does support Azure Blob containers, but there are in fact limitations on the free plan that will prevent this from being useful to us:

Windmill S3 bucket browser will not work for buckets containing more than 20 files and uploads are limited to files < 50MB. Consider upgrading to Windmill Enterprise Edition to use this feature with large buckets.

I have verified that this is the case:

In addition to the "stream to browser" file upload component, is also an S3 file input component that uploads a file in chunks to a storage bucket, but it seems to only take (AWS) S3 resources as an input, unfortunately it is not compatible with Azure Blob storage. The documentation doesn't say anything about any file upload limitations, so I wonder if using "S3 file input" with an AWS S3 bucket is therefore our best bet, short of just expecting a file to already exist in blob storage when running the script.

To be sure, I did ask on Windmill Discord if there is a workaround to use S3 file input with Azure Blob storage.

[rudokemper]

To be sure, I did ask on Windmill Discord if there is a workaround to use S3 file input with Azure Blob storage.

Regarding the use of the S3 file input component with Azure Blob storage, Ruben answered on Discord:

@rudo Kemper it's a todo but otherwise it will work by default with your workspace storage set in the instance settings if you do not specify resource

I will give this a try later!

[rudokemper]

Yes, it is possible to use the S3 File Input component with Azure blob storage if you don't set a resource as Ruben suggested while having Azure set as a default for Workspace storage, but we still run into the 50mb file size limitation 🫤

I think this means that we have exhausted all options, and there is no easy way to upload a >50mb file to any blob storage service on the Windmill free plan.

[rudokemper]

For the purposes of the 11th Hour project, it's acceptable to expect users to upload data to Azure Blob Storage using Azure Storage Explorer, given that the initial user persona is a CMI team member (like an Abram or a Rudo).

Therefore, the best course of action appears to be the initially suggested approach:

We could upload files to Azure Blob Storage and then use the Python Azure Storage SDK within the scripts to access the files.

[rudokemper]

I might be wrong about "there is no easy way to upload a >50mb file to any blob storage service on the Windmill free plan", reading https://www.windmill.dev/docs/core_concepts/object_storage_in_windmill#workspace-object-storage:

Instance and workspace object storage are different from using S3 resources within scripts, flows, and apps, which is free and unlimited.

At the workspace level, what is exclusive to the Enterprise version is using the integration of Windmill with S3 that is a major convenience layer to enable users to read and write from S3 without having to have access to the credentials.

So it might well be possible to write to S3-compatible storage, but not directly to Azure (...yet)

EDIT: Nope, Ruben says "I think regardless it will still be limited to 50MB".