Recommended best practice for including jupyter notebooks **without** their outputs, but enabling use of the _freeze cache

[jamesowers-cohere]

Description

I would like contributors of my quarto backed site to be able to include jupyter notebooks as pages. However, I have some requirements:

• The outputs of jupyter notebooks are very large so I do not want to include the output of the notebooks in the version controlled files pushed to github
• I want to ensure execute: enabled: true, such that every render will rerun the notebook from scratch if there has been a change in the source code
• I want my contributors to be able to develop using a jupyter notebook rather than a qmd file
• (this seems like a novel approach - it's not one that is in your documentation) the code which creates the virtual environment and jupyter kernel to use for the notebook is executed prior to the running of the notebook itself (this is generally done in a qmd)

However, because the "file source" which matches the _freeze cache hash check is that of the notebook including its outputs, it doesn't seem like I can satisfy my requirements (i.e. I am forced to version control notebook outputs if I want to be able to use _freeze cache).

Is there any other current approach that I'm missing that would enable me to use the cache without having to version control jupyter notebook outputs.

Some solutions I'm currently considering involve making use of a pre render script which could do one of the following:

• use quarto convert to convert all ipynb files to qmd pre-render, and prevent the rendering of ipynb altogether (I worry about weird edge cases where something would work in a notebook that doesn't in qmd)
• accept that freeze cache cannot be used for jupyter notebooks and either:
  • force execution of the recipes which create the jupyter kernels upon every render
  • roll my own cache solution and copy the notebook with outputs from some shared location if the notebook code has not changed

All advice well received, and I'm happy to provide more information about my setup as needed.

[cscheid]

The freezer contents, by definition, include the contents of the notebook you executed. So if I understand your situation correctly, the essential mismatch between your requirements and Quarto features need to be solved by disabling the freezer for the pages which you don't want to check in to version control.

Of course, in that case, everyone will need to re-execute those .ipynb files at every quarto render. But that's somewhat inevitable: either quarto has access to the computation results (in which case they need to be somewhere like version control), or quarto doesn't have access to the computation results (in which case Quarto needs to execute the code).

accept that freeze cache cannot be used for jupyter notebooks

It can be used for notebooks. It's just that freeze is, by design, meant to be used in conjunction with source control.

You might be able to split the difference by using the contents shortcode (referring to a file that you know to be in .gitignore or whatever the equivalent for your version control system) or a custom Lua filter that pulls contents in a similar manner. In this case, your .ipynb notebooks being rendered by Quarto would not contain the large outputs. They would simply contain placeholders that would be filled out by the Lua filter. As a result, the freezer contents would be relatively small (and you would become responsible for ensuring that the large notebook results are available somewhere).

[jamesowers-cohere]

To put it another way, say I quarto render an ipynb. The ipynb will contain both code and outputs, and the results will be cached in _freeze. If I think clear the outputs in the ipynb, it will no longer match the file hash for the cache in _freeze, and it will be executed when I run quarto render on it, even though none of the code has changed, so the cache could have been used.

[jamesowers-cohere]

Did my clarification here land at all? I think the misunderstanding is that my problem is not about what is stored in the _freeze directory, but the hash function that determines whether the file has changed since it was cached i.e. the process that determines whether the _freeze cached outputs should be used.

[cscheid]

Your misunderstanding is your assumption that there is a hash function, and your expectation that there should be a hash function. That's simply not how the freezer is designed.

[jamesowers-cohere]

Aha, thank you @cscheid !!!

Please may you explain how quarto decides whether or not to use the freezer cache? My experience is as documented, i.e.

You can use the freeze option to denote that computational documents should ... only be re-rendered when their source file changes:

https://quarto.org/docs/projects/code-execution.html#freeze

How does quarto decide whether the source file has changed?

[jamesowers-cohere]

To link this back to my question specifically - I'm wanting quarto to consider a jupyter notebook file unchanged if it is the same in all ways other than outputs.