Run prebuilds in the storage-bucket's region

[meysholdt]

Running a prebuild in the same geographical region of the storage bucket to which the prebuild will be uploaded saves us costs in two places:

• for all prebuilds, the compute egress costs from uploading the completed prebuild to its bucket.
• for incremental prebuilds, the storage egress of downloading a prebuild with the intention of updating it.

proposed solution: ws-manager-bridge/server should be aware of the geographical region of the prebuild's storage bucket and route the prebuild to a workspace cluster in the same region.

While we may not have capacity to fix this immediately it would be great to have this on the radar.

Related:

• https://github.com/gitpod-io/ops/issues/120
• Cache prebuilds locally (to a region) #6145
• https://github.com/gitpod-io/ops/issues/146
• https://github.com/gitpod-com/gitpod/issues/5773

[jankeromnes]

This sounds like a sensible and really good idea. 👍

(Unfortunately, it does not solve the egress due to popular prebuilds getting consumed in all regions -- for example, our starter templates typically have one prebuild that gets opened in many thousands of workspaces in all regions -- if a popular prebuild is stored in a single region, then all other regions will constantly transfer the same prebuild multiple times across oceans)

[JanKoehnlein]

We're a bit swamped at the moment. Moving this to our backlog to bring it up later

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaal]

If this issue becomes stale, will it be difficult to remember/find this feature-request in the future?

[jankeromnes]

I think we should raise the priority of this issue, because:

• We see a sharp uptake in Prebuilds, some of them with Incremental Prebuilds enabled
• Many projects with Incremental Prebuilds enabled do this:

1. Start new prebuild in US
2. Download old prebuild (base) from the EU bucket ($$$)
3. Build the new increment in US
4. Upload the new snapshot back to the EU bucket ($$$)

... repeated on every single push to such a repository (and some projects have a very high push-rate).

Also, is team WebApp the correct owner for this?

Cc @geropl

[geropl]

Ah, didn't look long enough to find this one, but created a fresh one: #9872

@jankeromnes Could you please replace the new issue with this one? And place in on top ot the "scheduled" column? 🙏