quarto::render() fails in parallel il project.render is "*.qmd"

[ColinFay]

Bug description

Rendering in parallel fails if project.render is "*.qmd" in _quarto.yaml

Reprex

Let's assume the following _quarto.yaml:

project:
  render:
    - "*.qmd"

Which is found on the doc at https://quarto.org/docs/projects/quarto-projects.html#render-targets

If I create a series of qmd and tries to render them in parallel using {future}, it fails :

options(
  future.rng.onMisuse = "ignore"
)
future::plan(future::multisession)

for (i in 1:50){
  write(file = sprintf("test%s.qmd", i),
        sprintf("---
title: 'test%s'
---

## Quarto

Quarto enables you to weave together content and executable code into a finished presentation. To learn more about Quarto presentations see <https://quarto.org/docs/presentations/>.
", i)
  )
}

list.files(pattern = "\\.qmd$", full.names = TRUE) |>
  furrr::future_map(
    .progress = TRUE,
    \(x){
      quarto::quarto_render(
        x,
        quiet = FALSE
      )
    }
  )

With the following error:

ERROR: NotFound: No such file or directory (os error 2): stat '/Users/colinfay/git/poubelle/multirenderquarto/test15_files/mediabag'

Stack trace:
    at Object.statSync (ext:deno_fs/30_fs.js:425:3)
    at createWalkEntrySync (file:///Applications/quarto/bin/quarto.js:2028:23)
    at walkSync (file:///Applications/quarto/bin/quarto.js:2068:15)
    at walkSync.next (<anonymous>)
    at advanceMatch (file:///Applications/quarto/bin/quarto.js:2191:20)
    at advanceMatch.next (<anonymous>)
    at expandGlobSync (file:///Applications/quarto/bin/quarto.js:2207:24)
    at expandGlobSync.next (<anonymous>)
    at expandGlobs (file:///Applications/quarto/bin/quarto.js:4972:24)
    at resolveGlobs (file:///Applications/quarto/bin/quarto.js:5049:26)
Error in (function (.x, .f, ..., .progress = FALSE)  : 
  ℹ In index: 3.
Caused by error in `quarto::quarto_render()`:
✖ Error running quarto cli.
Caused by error:
! System command 'quarto' failed
Calls: <Anonymous> ... resolve.list -> signalConditionsASAP -> signalConditions
Execution halted

Notes

• If I modify the yaml on the fly and add all the qmd names, it works :

y <- yaml::read_yaml(
  "_quarto.yaml"
)
old_render <- y$project$render
y$project$render <- list.files(pattern = "qmd")
yaml::write_yaml(y, "_quarto.yaml")

list.files(pattern = "\\.qmd$", full.names = TRUE) |>
  furrr::future_map(
    .progress = TRUE,
    \(x){
      quarto::quarto_render(
        x,
        quiet = FALSE
      )
      return(x)
    }
  )
  
y$project$render <- old_render
yaml::write_yaml(y, "_quarto.yaml")

• Removing the _quarto.yml works

• A _quarto.yml with other content will work

format:
  html

Session information

% Rscript -e "packageVersion('quarto')"
[1] ‘1.4.4’
% quarto --version
1.6.40

[cderv]

Is everything running in same working directory ? It seems that all quarto render will happen at the same time and could try to render the same project at the same time, or the same document at the same time.

This does not seem like something that will work very well unfortunately.
First reason is that already with rmarkdown::render() and how knitr works, this was something that could have conflicts. (accessing like reading / writing same file at same time)
Then, I don't think quarto render logic is designed for parallel rendering yet. I mean quarto render will write internal files, name intermediates, and do other things which could very easily interact badly between two quarto render

Though, I understand this works ok for you except using *.qmd as glob pattern. So let's say the above is working ok you (no bad side effect), and let's focus on the issue.

Stack Trace shows that the problem happens while resolving and expanding the glob, which is directly related to *.qmd. Error happens in expandGlobSync() which is a Deno fs function.

It seems directly related to conflict between processes because a path to mediabag dir like /Users/colinfay/git/poubelle/multirenderquarto/test15_files/mediabag is a path that is among the one cleaned during each quarto render. So it could exists when resolveGlobs() is called, but no more when createWalkEntrySync() is ran.

I see in Deno's code for expandGlobSync() that they sometimes do avoid throwing for NotFound error
like

  let fixedRootInfo: WalkEntry;
  try {
    fixedRootInfo = createWalkEntrySync(fixedRoot);
  } catch (error) {
    return throwUnlessNotFound(error);
  }

But some other times they don't like in walkSync()

  if (includeDirs && include(root, exts, match, skip)) {
    yield createWalkEntrySync(root);
  }

So it seems like a Deno bug at the root when running in parallel some deno process that will try to read / write / remove /expand same path at different times.

Not sure exactly what Quarto should do here - especially since we know that quarto render my-project/ is not something you can really call safely in parallel to operate on the same files. 🤔
To be clear, I am thinking that even we make an exception here to no error when file not found, there will probably be other issues, with other intermediates in different place. 😥

[ColinFay]

Hello,

Yes, the rendering is done from the same working directory.

Here is a reprex (with a Github action) of the code provided in the issue : https://github.com/ColinFay/quartoreprex/actions/runs/13507065321

And yes, this does work if there is no *qmd in the yml, without any bad side effect.

[cderv]

And yes, this does work if there is no *qmd in the yml, without any bad side effect.

Sad that we are hit by this Deno behavior then... Not sure how to prevent it. 🤔