[Feature Request] watch-exec

[andoriyu]

watch-store is nice when running locally, but it would be nice to have something similar to cachix's watch-exec.

It's essentially the same, but automatically stops watching when sub-process finishes.

[colemickens]

Another approach is to hook the post build process with a script that adds the built path to a queue. (Tested in a long-abandoned self-host nix cache tool)

[colemickens]

To elaborate (I opened up some old code). It would be something like:

nix-build --option post-build-hook attic-queue and attic-queue would take paths from stdin and queue for upload.

So in my case, I had a command that I would prescribe users to use that wrapped nix-build and supplied this extra argument with the appropriate environment set to get cache information from the wrapper through to the post-build script/command.

It's more elegant than watching the store since you might catch intermediates or other build outputs. It also ensures that separate builds, pushing to different caches, go to the right place.

[andoriyu]

In my case, i actually want all of the dependencies and intermediate paths. My main use case is speeding up builds for CI/CD.

…

On Wed, May 10, 2023, 7:14 PM Cole Mickens ***@***.***> wrote: To elaborate (I opened up some old code). It would be something like: nix-build --option post-build-hook attic-queue and attic-queue would take paths from stdin and queue for upload. So in my case, I had a command that I would prescribe users to use that wrapped nix-build and supplied this extra argument with the appropriate environment set to get cache information from the wrapper through to the post-build script/command. It's more elegant than watching the store since you might catch intermediates or other build outputs. It also ensures that separate builds, pushing to different caches, go to the right place. — Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABEFGZY4NMVZL3G7PPQPILXFRDQNANCNFSM6AAAAAAWUX7RZQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[colemickens]

Makes sense, I hope I didn't distract. I can split this out into a new feature request as well.

[Pythoner6]

+1 to this. I'm in the process of setting up some CI for my own nix builds and having this would be super useful for caching - with the current options I don't think attic is ideal for this usecase because either you wait until the build is done and upload everything, or use watch-store, but with watch-store I don't have a way to know when everything is done being uploaded.

[alisonjenkins]

I am currently using a workaround for this. I background watch-store using bash's jobs system and then at the end of the build I have a 30 second sleep and then kill the job.

This is working but not ideal as the builds have a potentially unnecessary 30s wait at the end of them.

Here are the 2 code snippets if people want to use this workaround:

# Setup attic caching
attic use cachename
attic watch-store cachename &

and at the end of the build:

# Wait for attic to finish pushing
echo "Waiting 30s to allow Attic to finish pushing to cache"
sleep 30
kill %1
wait

[Pythoner6]

Probably going to take a stab at a PR for this in the near-ish future hopefully, have a POC of this that I think is working (I still need to get my actual attic server working properly for pushing to confirm though 😂) main...Pythoner6:attic:watch-exec

[dhess]

I agree that watch-exec would be useful.

As a workaround, unless I’m missing something, I think the method described here and replacing cachix with attic would effectively do the same thing as watch-exec, just less elegantly, yes?

[TECHNOFAB11]

Just saw this issue again and wanted to mention my PR #97 which adds exactly that :)