Skip to content

Dotfiles install script execution should be cached #7592

@schickling

Description

@schickling

Is your feature request related to a problem? Please describe

As mentioned in this issue executing the install.sh script of my dotfiles currently takes 2-3 minutes (mostly by running Nix home-manager). This means using Gitpod with my dotfiles enabled is basically not practical for my workflows (i.e. waiting 2-3 minutes every time I'm opening a workspace).

Describe the behaviour you'd like

One solution to this problem I can think of is to cache the result of the dotfiles install script.

Describe alternatives you've considered

Maybe some kind of layered file systems (e.g. via Docker images)?

Additional context

Activity

stemount

stemount commented on Jan 15, 2022

@stemount

I've not used Nix, would it be more suitable in this scenario for you to fork the GitPod default workspace (or similar) as I am guessing it's not the dotfiles themselves but just that this environment is being built everytime you use GitPod.

schickling

schickling commented on Jan 15, 2022

@schickling
Author

I've not used Nix, would it be more suitable in this scenario for you to fork the GitPod default workspace (or similar) as I am guessing it's not the dotfiles themselves but just that this environment is being built everytime you use GitPod.

Would that approach still work for every repo/container I'm opening regardless of whether I have control over its .gitpod.yml?

added
team: workspaceIssue belongs to the Workspace team
and removed
team: webappIssue belongs to the WebApp team
on Jan 24, 2022
csweichel

csweichel commented on Jan 24, 2022

@csweichel
Contributor

Caching the dotfile execution in such long-running cases would be sensible indeed. I'm wondering how this would work if it were based on a filesystem layer model: in filesystem layers files cannot really be changed, merely overwritten, we'd need to run the install script at least once for every workspace image - and when the dotfile repo changes. I'm not sure this would yield a good user experience, as users would still be waiting a lot for their dotfile scripts.

Part of what makes this wait so bad is that right now the dotfile install script has to complete before we start the IDE (it's likely the dotfiles modify $PATH and the likes). What if your dotfile install script did some of those install operations outside of its "hot path", i.e. did not block until they're complete?

axonasif

axonasif commented on Jan 24, 2022

@axonasif
Member

@csweichel: What if your dotfile install script did some of those install operations outside of its "hot path", i.e. did not block until they're complete?

Can we just spawn a subprocess for long running commands? (E.g sudo apt-get upgrade &)
Would the IDE launch process still be blocked and wait for the dotfiles install script to complete/exit?

schickling

schickling commented on Jan 24, 2022

@schickling
Author

I'm wondering how this would work if it were based on a filesystem layer model

Since I'm using nix for my system setup, I think this approach would actually work really well. cc @ghuntley

axonasif

axonasif commented on Jan 30, 2022

@axonasif
Member

@csweichel: we'd need to run the install script at least once for every workspace image - and when the dotfile repo changes. I'm not sure this would yield a good user experience, as users would still be waiting a lot for their dotfile scripts.

I can think of two approaches(more like init prebuilds):

  • A dedicated script (I.e install-init.sh) that only runs once on change. A filesystem change snapshot may be taken for file modifications performed by this script.
  • A gp cli command where you pass your one-time commands from your install.sh, then take the snapshots of the file changes and combine multiple snapshots together (Might be a rather complicated solution)
    It could be like: gp snap <identifier-name> <some-cli-command>.
    An example could be like: gp snap install-firefox sudo apt-get install firefox

What if your dotfile install script did some of those install operations outside of its "hot path", i.e. did not block until they're complete?

I went through experimenting with a bunch of hacks but either way, you have to wait to get started with your work. Whether you stay stuck in the IDE loading(Starting G) screen for a while or inject the shell changes in terminals after the installation is done, you have to wait either way.

stale

stale commented on Apr 30, 2022

@stale

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

added
meta: staleThis issue/PR is stale and will be closed soon
on Apr 30, 2022
schickling

schickling commented on May 1, 2022

@schickling
Author

Still relevant.

removed
meta: staleThis issue/PR is stale and will be closed soon
on May 1, 2022
david-bakin

david-bakin commented on Jul 9, 2022

@david-bakin

When implemented, the gitpod env vars including user defined env vars (incl per project env vars) should be part of the cache key. So you could pass an env var (either set in the gitpod settings or via URL) and have that affect the dotfile execution. See for example the discussion at https://discord.com/channels/816244985187008514/839379835662368768/995280980812255322

loujaybee

loujaybee commented on Aug 12, 2022

@loujaybee
Member

I wonder if the request to optimise dotfiles points to the absence of user-specific image configurations. Right now, Gitpod assumes that a team collaborates together on getting the right tools installed for a project, however in some cases users will want their own per-user tools or configurations for themselves (just as they do on their local machine). It might be worth exploring having a per-use base image setup where a user can define their own tools that's cached and incorporated with prebuilds, rather than being bundled in the dotfiles (which is the current workaround for per-user installations).

axonasif

axonasif commented on Aug 12, 2022

@axonasif
Member

I personally prefer the to install/configure things from my dotfiles, I use my dotfiles locally too and I would usually run my installation script for the first time I setup a machine. It's easier to manage one singe source of my dotfiles instead of having a detached Gitpod-specific configuration elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @stemount@schickling@csweichel@david-bakin@loujaybee

        Issue actions

          Dotfiles install script execution should be cached · Issue #7592 · gitpod-io/gitpod