Proposal: Enable the possibility to store in memory video/images/raw-bytes through common.io xcom backend

[jolin1337]

Proposal: I suggest that we should be able to store in memory video/images/raw-bytes through common.io xcom backend. After opening a PR: #61777 I realized that this might not be the implementation the community would do in the common.io provider as @Nataneljpwd pointed out. What data-types should be supported to store in this provider and what is the actual puprose of it?

Current behaviour: We only support json serializable objects. And I could not find any previous discussions about supporting other serializatoin methods.

My optinion: I think it is pretty convenient that it exists both as a fallback that we do not accidentally store big files in S3 but also as a reliable feature to standardize the bucket paths in s3 for all dags that are running in the airflow instance.

Suggestion: I think the purest form of data we can store is raw bytes, and don't care about the contents. Although since json is already implemented and for the sake of not destroy backward compatibility we would support both JSON and raw bytes data.

What are your takes on the topic?

[Nataneljpwd]

Hello, as I said, I think that it is a good Idea, yet we need to take into acount that it can only work for when we store the data on s3, as otherwise, we are required to change the schema of the xcom table, and I think that storing raw bytes only without the option for json in the table will add to the current overhead of pulling an xcom, as most xcoms use json schemas, and if we change it to be binary only, we will need to know what the underlying data type actually is, if it is a serialized json or a video/audio file.

I also think that in no case we should save binary data (blobs) in the metadata database itself, as it is not recomended by any of the database vendors, in addition to there bein a hard limit for the binary data size (in pgsql around 1gb without plugins, in maria/mysql up to 4gb), and so I think the implementation should target the object storage xcom backend instead

[jolin1337]

I agree with you that this could be an edge case for the object key storage scenario. Would it be possible to add a simple assert in the xcom backend just before we try to insert bytes to the database instead of uploading it to the object store? Or would we need a more sophisticated approach?

[Nataneljpwd]

I think it needs to be implemented in the xcom backend mainly while having as little logic in the common provider to allow for various types other than json

[jolin1337]

Another approach also suggested in the PR is to encode the raw bytes as strings such that they become json serializable. Do you have a suggestion of how one would go approaching it that way? Both approaches has different pros and cons. Concerning data allocation, complexity and convenience.

[Nataneljpwd]

it is an approach, but having a b64 encoded binary fine might not be the best way to do that