Save level.dat asynchronously on the I/O thread to avoid blocking the server thread

[Ghost-chu]

Is your feature request related to a problem?

I currently have 27 worlds loaded on my Paper instance, but aside from the three main worlds, the rest are archive worlds in past 10 years. The three main worlds run on local high-speed NVMe SSD storage, while the archive worlds are mounted to the world path via Samba/CIFS from a slow HDD NAS on the local network since my players only visit these worlds during nostalgia events, there is no player activity on them at other times.

Due to the NAS's heavy workload (continuous backups, deduplication, compression, and checksumming) and the fact that it runs on HDDs, its I/O performance is poor. Although Paper's chunk system handles these slow read operations effectively, preventing them from affecting the game server itself, the operation to save level.dat still runs on the server thread. This causes the server to experience a spike in lag every so often (when you have a lots worlds, it will keep and continuously). Although these spikes last only a few seconds, they are still long enough to trigger the watchdog.

Stacktrace: https://mclo.gs/Np1vX9r

Describe the solution you'd like.

The saving of level.dat should be performed asynchronously on an I/O thread and should not block the server thread anyway.

Performing I/O operations on a server thread is definitely a bad idea.

Describe alternatives you've considered.

Maybe I should mount level.dat on a local high-speed HDD, but that's a hassle.
Or unload those worlds from instance.

Other

No response

[Ghost-chu]

Tried move those worlds to local HDD and there doesn't seem to be any significant improvement.

[Rothes]

Since they are "archive" worlds, you can set chunks.auto-save-interval to -1 in paper-world.yml to disable auto save.

Btw, player data saving is also always on server thread. It operates 3 files once for each player, that's heavier than level data auto save.

[Ghost-chu]

Since they are "archive" worlds, you can set chunks.auto-save-interval to -1 in paper-world.yml to disable auto save.

Btw, player data saving is also always on server thread. It operates 3 files once for each player, that's heavier than level data auto save.

Its still need to save since players will play a while on archived worlds.

Furthermore, adjusting the auto-save interval doesn't seem to solve the problem; it just makes it happen less frequently.

[Rothes]

Set chunks.auto-save-interval to -1 should disable level data saving and chunks auto save, if there's no other plugins trying to save or unload the worlds. Chunks will still save on unload. Level data saves on shutdown. Both can be saved by /save-all or World.save().

There are cases that some chunk data have to syncLoad on server thread, mostly poi searching, can be triggered by for example using eye of ender / entity AI. It may be alleviated by increasing region file cache.

About the async issue.. Mojang does have a async saving logic for other level data (maps, raids, etc.), however for level.dat it's on server thread though it does not require much changes for async. Personally I think there's no reason, maybe Mojang thinks it's a crucial file. However, Their file operations is not atomicly, so it's possible to leave no level.dat file if the process get terminated.