Control order of rendering

[allefeld]

Description

I have a sequence of qmd files, where the code in some creates results (disk files) which are used by others. These files therefore need to be executed in a specific order by quarto render to work properly.

I tried to influence the order in which Quarto renders documents: Explicit listing of contents: instead of auto, putting order: into the metadata of the files, putting numbers at the beginning of the documents' titles, putting numbers at the beginning of the documents' file names – and none of them work reliably. The order seems to be neither alphabetical by file name nor alphabetical by title; my guess is that quarto render simply processes files in the order they are in the directory (ls -U).

My request is: Please provide a way to control the rendering order of qmd files. Optimally that would not depend on filenames or titles, but e.g. on metadata like order, but alphabetical file name order would be okay, too.

Note that this is similar to but different from #3568; I'd be happy if the rendering order would be identical to the order of appearance. In order to accomodate both wishes, maybe a new metadata field execution-order: would be needed.

I'd also be interested in a short-term workaround. Any tips?

[mcanouil]

This could be possible with:

• Epic: Execution Planner #6518

But definitely not before.

[allefeld]

A workaround is to list the files explicitly in _quarto.yml under project / render.

[cscheid]

That's a good trick!

It's very much in the "clever workaround" category, and I hadn't realize it would work. I think we respect array order everywhere it would matter, but I wouldn't lean on that too hard.

With that said, this is very practical and we'll start recommending it to folks in the meantime, thank you for the idea!

[steffilazerte]

This may be implied, but I was happy to realize that I didn't have to explicitly list every file:

I number my qmd files in the order they should be rendered (01_setup.qmd, 02_load_data.qmd, etc). And adding the following was sufficient for them to be rendered in order 🎉

project:
  render: 
  - "*.qmd"

[allefeld]

In my experience, that does not work reliably. My best guess is that using wildcards processes the files in the order they are stored in the directory (*nix: ls -U).

[steffilazerte]

Hmm, although ls -U definitely does not list the files in alpha-numerical order, I find they're still rendered in order when I use this work around. Perhaps it depends on the file system? I'm on Ubuntu 22.04.

[allefeld]

@cscheid: Unfortunately, this workaround now seems to be broken, see #10490. Since it made it into the documentation

If you don’t want to render all of the target documents in a project, or you wish to control the order of rendering more precisely, you can add a project: render: [files] entry to your project metadata.

I guess this should now be considered a bug? Also because the epic execution scheduler still seems to be a way away.

I only noticed this recently with 1.5.56 when I returned to a complex project. But I checked previous stable 1.5 versions, and they do not respect the project / render order either.

[eculler]

I am having this problem as well on GitHub Actions with the ubuntu-latest runner, despite:

• listing files individually in project/render and
• Numbering the files so that alphanumeric order should be the correct one

The render runs correctly on my system (MacOS Sonoma 14.5).

I need the notebooks to render in order, because they use jupyter storemagic to carry variables over from one to the next.

As an example, when I run:

project:
  render:
    - pages/03-git-github/02-github-collaboration/02-pull-requests.qmd
    - notebooks/01-climate/climate-31.qmd
    - notebooks/01-climate/climate-32.qmd
    - notebooks/01-climate/climate-33.qmd
    - notebooks/01-climate/climate-34.qmd

the actual run order is:
- notebooks/01-climate/climate-31.qmd
- pages/03-git-github/02-github-collaboration/02-pull-requests.qmd
- notebooks/01-climate/climate-33.qmd
- notebooks/01-climate/climate-32.qmd
- notebooks/01-climate/climate-34.qmd

Note that it doesn't even finish rendering a full directory before skipping around!

[mattpollock]

I have a similar use case to the OP. Using the render: work-around seems to work for website projects, but not book projects, where the rendering order is the chapter order. This makes doing something like an executive summary up front very difficult.

Looking forward to #6518