feat: Create Individual Virtual Env for the Kernel

[dasmy]

This addresses #6 and #62 - at least partially.

Whenever the kernel is invoked, we start with an empty directory and set up a virtual environment with the standard packages. If an additional package is needed, the corresponding !pip install PACKAGE call can be executed and packages do not pollute the system environment.

Note, that the approach considerably dilates kernel startup time. - Thus, debugging is not always fun with this.

An example prompt for this is

Use rdkit to plot the molecule with the SMILES string 'CC1CCC/C(C)=C1/C=C/C(C)=C/C=C/C(C)=C/C=C/C=C(C)/C=C/C=C(C)/C=C/C2=C(C)/CCCC2(C)C'.

I think (!), this could be a good baseline for addressing #67 as well. - At least, running kernels with separate environments would be possible this way.

[ricklamers]

This is really nice! But can we make it so that restarting the app will pick up on a created kernel venv?

[dasmy]

This is really nice! But can we make it so that restarting the app will pick up on a created kernel venv?

I'll give it a try. - Definitely makes sense (at least during development the slow startup times are annoying :-D ).

Currently, I am working on a multi-client version, where everybody needs to have their own kernel. Will have to think how to get both requirements into a common approach.

Possibly, creating one template venv and re-using it for all kernels (via copy/symlink) might be an option to make things quicker...

[dasmy]

Hi @ricklamers,
this is a fresh attempt, to make things considerably more smooth (and faster).

We first create a base venv that is not to be deleted and can be re-used easily.
There we install all default packages. (Thus, the very first start after applying
the change is still slow).

Then, we create another, kernel-specific, venv without any additional packages.
Instead, we create a link that points to the packages installed in the base image.

The base image is intended to stay immutable. All newly installed packages end up
in the kernel-specific venv. While they are lost on restart, the packages inside the
base image are not.

The overall idea is coming from https://stackoverflow.com/a/75545634 and - reading
the corresponding documentation - feels technically sound. What do you think?

[dasmy]

Closing this for now to cleanup git history. - Will re-open afterwards.

[ricklamers]

Although the .pth approach is somewhat exposed to the user, it does feel like it's breaking a line of abstraction provided by regular virtual environment use (venv or virtualenv). I'm not opposed to it entirely, but this approach could lead to unexpected consequences. For one, I'm not aware what kind of limitations there are for such "derived" virtual environments compared to a "regular virtual environment". Open to exploring this direction! Just stating my initial thoughts

[dasmy]

Although the .pth approach is somewhat exposed to the user, it does feel like it's breaking a line of abstraction provided by regular virtual environment use (venv or virtualenv). I'm not opposed to it entirely, but this approach could lead to unexpected consequences. For one, I'm not aware what kind of limitations there are for such "derived" virtual environments compared to a "regular virtual environment". Open to exploring this direction! Just stating my initial thoughts

Thank you for your comment. Answering here: #74 (comment)