R2

[gedw99]

Would you accept an R2 PR?

[funkyshu]

I'm generally open to the idea. The biggest concern obviously is maintaining another backend (for which we don't currently have an account to test against). I'll have to confer with my team.
Alternatively, vfs is meant to allow for external backends. See https://github.com/C2FO/vfs/blob/main/docs/backend.md. If you chose to self-publish, I'm happy to review any code and link your lib in our docs.

Finally, as far as I know, R2 is S3-compatible. I've not tested this but I'd assume you could use the s3 vfs backend doing something like:

package main

import (
	"log"

	"github.com/c2fo/vfs/v6"
	"github.com/c2fo/vfs/v6/backend/s3"
	"github.com/c2fo/vfs/v6/utils"
)

func main() {
	// Cloudflare R2 credentials and endpoint
	r2AccessKey := "YOUR_ACCESS_KEY"
	r2SecretKey := "YOUR_SECRET_KEY"
	r2Endpoint := "https://<your-account-id>.r2.cloudflarestorage.com" // Replace <your-account-id>
	r2Bucket := "YOUR_BUCKET_NAME"
	filePath := "example/file.txt"

	// Set up environment variables for the S3 backend
	utils.MustSetEnv("AWS_ACCESS_KEY_ID", r2AccessKey)
	utils.MustSetEnv("AWS_SECRET_ACCESS_KEY", r2SecretKey)
	utils.MustSetEnv("AWS_REGION", "auto") // R2 uses "auto" as a placeholder for the region
	utils.MustSetEnv("AWS_S3_ENDPOINT", r2Endpoint)
	utils.MustSetEnv("AWS_S3_FORCE_PATH_STYLE", "true") // Ensure path-style addressing

	// Create an S3 location
	location, err := s3.NewLocation(r2Bucket, "")
	if err != nil {
		log.Fatalf("Failed to create S3 location: %v", err)
	}

	// Get a file reference in the R2 bucket
	file, err := location.NewFile(filePath)
	if err != nil {
		log.Fatalf("Failed to create file reference: %v", err)
	}

	// Write a file to R2
	content := "Hello, Cloudflare R2!"
	if err := file.Write([]byte(content)); err != nil {
		log.Fatalf("Failed to write file: %v", err)
	}
	log.Printf("File written to %s/%s", r2Bucket, filePath)

	// Read the file back
	data, err := file.ReadAll()
	if err != nil {
		log.Fatalf("Failed to read file: %v", err)
	}
	log.Printf("File contents: %s", string(data))
}

The key difference with S3 vs R2, I believe, is setting the s3 backend options, Endpoint and (maybe) ForcePathStyle.

Is there somethink known in R2 that the s3 backend won't do?

[gedw99]

Hey @funkyshu

Thanks and I will try it out .

I don’t know what limitations R2 has compared to S3 yet as I have only just started using it. Will update here if I find more than what you have.

The reason I am interested in this is because R2 is much cheaper than all the other S3 systems .

Also it’s because I run a ton of golang WASM as Cloud flare workers for free ( under the 3mb limit ).

It’s a cache layer in front of my origin servers , and when blobs change in my origin , I force an update into R2, ensuring that R2 is always up to date in all regions ( about 300 of them ).

The workers can then do read only computation from the images in R2.

all mutations of blobs goes back always to Origin , which uses s3 for blob storage.

So your system would allow me to use the same api , from 3 different locations , keeping all the blob data in sync. I would of course use a queue to ensure eventual consistency.

I just wanted to explain my use case to justify this issue and because it might be useful to others .

https://developers.cloudflare.com/r2/api/workers/workers-api-usage/ Explains this to a degree

[funkyshu]

Assuming, our s3 backend doesn't work as is, I'm happy to accept a PR to accommodate or entertain an R2-specific backend, if necessary. Looking forward to hearing if it works.

[gedw99]

The golang Cloudflare package is a good option ?

https://github.com/cloudflare/cloudflare-go/blob/master/r2_bucket.go

https://github.com/cloudflare/cloudflare-go/blob/1ad4079e9348210fc2ff7b809e288675fa6eeb80/pages_project.go#L169 Is slightly related but it’s for allowing web pages , workers access to the R2 buckets .