Repository navigation
fix: hold spill temp files by path instead of open file descriptor - #27602
Conversation
Each RefCountedTempFile kept a NamedTempFile—and with it one open file descriptor—for the entire lifetime of the spill file. Operators that retain many spill runs (such as a hash aggregate before its final merge) hold one descriptor per live run and can hit EMFILE (Too many open files) under tight memory limits with the default ulimit -n of 1024. Store TempPath in RefCountedTempFile instead and open the file on demand in append mode in open_writer (read_stream already opens the path on demand). Live spill files then hold no file descriptors, so descriptor usage scales with active writers and readers instead of total live spill files. Removes RefCountedTempFile::inner(&self) -> &NamedTempFile, which is unused in the workspace.
Other tests in the same process open files concurrently, so a total descriptor count with slack is flaky. The test now reads each descriptor's target from /proc/self/fd and asserts that none points at a live spill file. This makes the test Linux-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counting descriptors in a shared test process is either flaky or platform-specific. The PR description holds a datafusion-cli script that fails with EMFILE on main and passes with this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jayzhan211
left a comment
There was a problem hiding this comment.
@adriangb thanks for this! Nothing in the PR checks that a live spill file holds no fd, so a change back to holding the file would still pass CI. If the test is the only test in its own binary, nothing else opens files while it counts, and /dev/fd works on both Linux and macOS. The test below fails on main (left: 404, right: 4) and passes on this branch.
datafusion/execution/tests/spill_file_descriptors.rs (add the ASF license header)
//! This file is its own test binary with a single test, so no other test
//! opens files while it counts this process's descriptors.
use std::io::Write;
use std::sync::Arc;
use datafusion_common::Result;
use datafusion_execution::disk_manager::DiskManagerBuilder;
#[cfg(unix)]
fn open_fd_count() -> usize {
std::fs::read_dir("/dev/fd").unwrap().count()
}
#[cfg(unix)]
#[test]
fn live_spill_files_hold_no_file_descriptors() -> Result<()> {
const FILES: usize = 400;
let dm = Arc::new(DiskManagerBuilder::default().build()?);
// The first file creates the spill directory.
drop(dm.create_tmp_file("warm up")?);
let before = open_fd_count();
let mut files = Vec::with_capacity(FILES);
for _ in 0..FILES {
let file = dm.create_tmp_file("fd bound")?;
let mut writer = file.open_writer()?;
writer.write_all(b"spilled")?;
writer.finish()?;
files.push(file);
}
assert_eq!(open_fd_count(), before);
for file in &files {
assert_eq!(std::fs::read(file.path().unwrap())?, b"spilled");
}
Ok(())
}Removing RefCountedTempFile::inner() is a breaking change, so it needs the api change label for the release notes, plus a short entry in docs/source/library-user-guide/upgrading/57.0.0.md. The entry can be the sentence you already have under user-facing changes: use path() for the path and the SpillFile methods to read or write.
Optional: the Rationale could open with the failing query instead of the mechanism, so a reviewer sees the need first. For example: "A sort or aggregate that keeps many spill runs alive fails with Too many open files (os error 24), because each live spill file keeps one fd open even when idle (400 live spill files = 400 fds on main). This PR holds spill files by path, so only open writers and readers use fds."
|
Thanks for the review @jayzhan211. I had actually removed the test in 558abf5 (which I think landed before your review? not sure). I couldn't find a good way to make this a reproducible test since things like file descriptors are global and also OS dependent. I offered a reproducer in the PR description instead. |
|
@adriangb I see. |
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
|
Thanks @jayzhan211 ! |
Which issue does this PR close?
Rationale for this change
Each
RefCountedTempFilecurrently holds anArc<NamedTempFile>, which keeps an open OS file descriptor for the entire lifetime of the spill file even while it sits idle on disk.Under tight memory limits (e.g., 256 MB–512 MB with 16 partitions), operators that accumulate many spill runs—such as a hash aggregate before its final merge—hold one file descriptor per live spill file. On ClickBench Q18 at a 256 MB limit, this reaches ~1,000 live runs and fails with
Too many open files(EMFILE) under the defaultulimit -nof 1024.RefCountedTempFile::read_streamalready opens the file by path on demand. HoldingTempPathinstead ofNamedTempFilemeans live spill files hold no file descriptors, so descriptor usage scales with active open writers and readers rather than total live spill files.What changes are included in this PR?
Arc<TempPath>instead ofArc<NamedTempFile>inRefCountedTempFile(datafusion/execution/src/disk_manager.rs), converting the created file via.into_temp_path().OpenOptions::new().append(true)mode inRefCountedTempFile::open_writer, preserving shared-offset append behavior across multiple writers on the same spill file.RefCountedTempFile::inner(&self) -> &NamedTempFile(unused in the workspace).Are these changes tested?
Existing
datafusion-executiontests cover spill read, write and cleanup. A unit test for descriptor usage is either flaky (other tests share the process) or platform-specific, so this PR uses a reproduction instead.The script below sorts 100M rows with a 64 MB pool, which keeps about 30 spill runs alive. Capping the merge fan-in at 4 means only a few readers are open at a time. So the live spill files themselves exceed
ulimit -n 32. It works on Linux and macOS.ulimit -n 32main(a72c60b)IO error: Too many open files (os error 24)100000000Also ran
cargo test -p datafusion-execution --lib,cargo clippy -p datafusion-execution --all-targets -- -D warningsandcargo fmt --all -- --check.Are there any user-facing changes?
Breaking public API change:
RefCountedTempFile::inner(&self) -> &NamedTempFileis removed becauseRefCountedTempFileno longer holds an openNamedTempFile. Callers that need the path can useRefCountedTempFile::path(), and callers that need to write or read can use theSpillFiletrait methods.