quarto render on shared file storage fails with locked database

[schochastics]

After updating quarto to 1.7.31, we run in the following problem:
We have a mounted shared file system on which we get the following error when rendering a qmd file.

ERROR: database is locked: Error code 5: The database file is locked

Stack trace:
    at async Object.openKv (ext:deno_kv/01_db.ts:10:15)
    at async ProjectCacheImpl.init (file:///opt/quarto/bin/quarto.js:51070:22)
    at async createProjectCache (file:///opt/quarto/bin/quarto.js:51197:5)
    at async projectContext (file:///opt/quarto/bin/quarto.js:82381:32)
    at async render (file:///opt/quarto/bin/quarto.js:91820:19)
    at async Command.actionHandler (file:///opt/quarto/bin/quarto.js:92001:32)
    at async Command.execute (file:///opt/quarto/bin/quarto.js:8253:13)
    at async Command.parseCommand (file:///opt/quarto/bin/quarto.js:8143:20)
    at async quarto (file:///opt/quarto/bin/quarto.js:128476:9)
    at async file:///opt/quarto/bin/quarto.js:128506:9

if the file is stored in the home directory of a user, the error does not appear.
We did not have issues rendering qmd files on the shared file system before the update of quarto.

I have tried to remove the deno cache but the error remains.

[cscheid]

We generally discourage using Quarto in a shared network drives for this reason. In the same way that it's not generally safe eg to use git in a Dropbox, there are parts of Quarto that will not work well in a shared drive because these shared drives behave very differently from file systems. That's not something we can fix, unfortunately.

[cderv]

We have a mounted shared file system on which we get the following error when rendering a qmd file.

Just curious to confirm our thoughts:
Is this a shared drive where you are working on several users at the same time on the same file? Or running several quarto render on the same files?

What exactly is your environment configuration?

Quarto 1.7 uses Deno.kv to have caching of project information, and the Deno.kv mechanism requires locking by Windows, which means only one access.

[schochastics]

Quarto is running in a docker container with Ubuntu 22.04.
I'll have to confirm, but the error appears on any new quarto file by any user on the shared drive.

What I can see is that the render process creates ~/.cache/deno/ with *-wal and *-shm files in the respective users home directory. I can't locate such a folder on the shared drive.

I guess I'll work around this in the future, but just out of curiosity:
Is there any way to "unlock" this situation? Also: Were we just lucky that this never happened with quarto < 1.7 or is this due to the introduction of Deno.kv in 1.7?

[cderv]

Were we just lucky that this never happened with quarto < 1.7 or is this due to the introduction of Deno.kv in 1.7?

I do believe this is the introduction of using Deno.kv for caching. This is what requires locking and unlocking.

What I can see is that the render process creates ~/.cache/deno/

I am curious about this path. Our createProjectCache() should target project-cache inside .quarto folder in a Quarto project. 🤔

Quarto do use ~/.cache when created from

quarto-cli/src/core/appdirs.ts

Lines 140 to 144 in 85b8193

	function xdgUserCacheDir(appName: string) {
	const cacheHome = Deno.env.get("XDG_CACHE_HOME") ||
	join(Deno.env.get("HOME") || "", ".cache");
	return join(cacheHome, appName);
	}

which is used for User cache directory on Linux

quarto-cli/src/core/appdirs.ts

Lines 74 to 90 in 85b8193

	export function userCacheDir(appName: string) {
	switch (platformOs) {
	case "darwin":
	return darwinUserCacheDir(appName);
	case "windows":
	return windowsUserDataDir(appName);
	default: // assume generic unix
	// in 1.32.5 this is:
	// case "linux":
	// case "netbsd":
	// case "aix":
	// case "solaris":
	// case "illumos":
	// case "freebsd":
	return xdgUserCacheDir(appName);
	}
	}

However, I am curious about the deno/ part, as Quarto should be using quarto/

quarto-cli/src/core/appdirs.ts

Lines 19 to 21 in 85b8193

	export function quartoCacheDir(subdir?: string) {
	return quartoDir(userCacheDir, subdir);
	}

Quarto is running in a docker container with Ubuntu 22.04.

Would you be able to share a Docker container that has the issue? What would we need to reproduce the shared folder setting?

I'll let Calors add his insights on this.

[schochastics]

The ~/.cache/deno dir seems to be for Deno’s internal SQLite usage. That is the default location. It is possible to redirect it by

export DENO_DIR=<path>

I did test that and quarto did indeed write the cache to this location but didnt solve the issue. Could deleting the quarto cache "unlock" the database?

Would you be able to share a Docker container that has the issue? What would we need to reproduce the shared folder setting?

I am afraid that I cannot share any docker containers since they are for internal use only. I will try to derive a minimal repro from it if possible though.

[cderv]

I will try to derive a minimal repro from it if possible though.

That would be helpful. Right now I am quite puzzled by the fact that stack trace shows problem happens in ProjectCacheImpl which should initialize the db using Deno.kv() in the scratch directory (<project root>/.quarto/project-cache

quarto-cli/src/core/cache/cache.ts

Lines 69 to 72 in 85b8193

	async init(): Promise<void> {
	const indexPath = join(this.projectScratchDir, "project-cache");
	ensureDirSync(indexPath);
	this.index = await Deno.openKv(join(indexPath, "deno-kv-file"));

For context, all previous issues we had with Deno.kv and locking were on Windows, as Windows does have a specific locking mechanism that requires explicit unlock.
For example, deleting the cache dir with the DB wouldn't be possible if it was not unlocked beforehand.

Understanding the setting and where the files are really created to cause this type of issue would help us better understand and get to the bottom of this.

Right now, our POV, is shared drive means shared file with multiple access at the same time, which does not play well with Deno.kv it seems.

Thanks a lot.

[schochastics]

No worries. It might take some time on my end but since I would like to understand this issue better too, I'll try to create a minimal repro example.

[cderv]

Awesome. Don't hesitate to reach out (even for quick pairing if this can help). We would like to be sure this is what we think this is, and not some other edge case. Thank you !