Skip to content

fix: hold spill temp files by path instead of open file descriptor - #27602

Merged
adriangb merged 4 commits into
apache:mainfrom
adriangb:fix/spill-file-hold-path-not-fd
Nov 21, 2026
Merged

adriangb merged 4 commits into
apache:mainfrom
adriangb:fix/spill-file-hold-path-not-fd

Conversation

@adriangb

@adriangb adriangb commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Each RefCountedTempFile currently holds an Arc<NamedTempFile>, which keeps an open OS file descriptor for the entire lifetime of the spill file even while it sits idle on disk.

Under tight memory limits (e.g., 256 MB–512 MB with 16 partitions), operators that accumulate many spill runs—such as a hash aggregate before its final merge—hold one file descriptor per live spill file. On ClickBench Q18 at a 256 MB limit, this reaches ~1,000 live runs and fails with Too many open files (EMFILE) under the default ulimit -n of 1024.

RefCountedTempFile::read_stream already opens the file by path on demand. Holding TempPath instead of NamedTempFile means live spill files hold no file descriptors, so descriptor usage scales with active open writers and readers rather than total live spill files.

What changes are included in this PR?

  • Store Arc<TempPath> instead of Arc<NamedTempFile> in RefCountedTempFile (datafusion/execution/src/disk_manager.rs), converting the created file via .into_temp_path().
  • Open the path on demand in OpenOptions::new().append(true) mode in RefCountedTempFile::open_writer, preserving shared-offset append behavior across multiple writers on the same spill file.
  • Remove RefCountedTempFile::inner(&self) -> &NamedTempFile (unused in the workspace).

Are these changes tested?

Existing datafusion-execution tests cover spill read, write and cleanup. A unit test for descriptor usage is either flaky (other tests share the process) or platform-specific, so this PR uses a reproduction instead.

The script below sorts 100M rows with a 64 MB pool, which keeps about 30 spill runs alive. Capping the merge fan-in at 4 means only a few readers are open at a time. So the live spill files themselves exceed ulimit -n 32. It works on Linux and macOS.

cargo build --release -p datafusion-cli
cat > /tmp/spill_fds.sql <<'SQL'
SET datafusion.runtime.max_spill_merge_fan_in = '4';
SET datafusion.execution.target_partitions = 1;
SELECT max(r) FROM (
  SELECT row_number() OVER (ORDER BY v DESC) AS r
  FROM generate_series(1, 100000000) t(v)
);
SQL
(ulimit -n 32; ./target/release/datafusion-cli -m 64m -q -f /tmp/spill_fds.sql)
Build Result at ulimit -n 32
main (a72c60b) IO error: Too many open files (os error 24)
This PR 100000000

Also ran cargo test -p datafusion-execution --lib, cargo clippy -p datafusion-execution --all-targets -- -D warnings and cargo fmt --all -- --check.

Are there any user-facing changes?

Breaking public API change: RefCountedTempFile::inner(&self) -> &NamedTempFile is removed because RefCountedTempFile no longer holds an open NamedTempFile. Callers that need the path can use RefCountedTempFile::path(), and callers that need to write or read can use the SpillFile trait methods.

Each RefCountedTempFile kept a NamedTempFile—and with it one open file descriptor—for the entire lifetime of the spill file. Operators that retain many spill runs (such as a hash aggregate before its final merge) hold one descriptor per live run and can hit EMFILE (Too many open files) under tight memory limits with the default ulimit -n of 1024.

Store TempPath in RefCountedTempFile instead and open the file on demand in append mode in open_writer (read_stream already opens the path on demand). Live spill files then hold no file descriptors, so descriptor usage scales with active writers and readers instead of total live spill files.

Removes RefCountedTempFile::inner(&self) -> &NamedTempFile, which is unused in the workspace.
@github-actions github-actions Bot added the execution Related to the execution crate label Nov 20, 2026
@adriangb
adriangb marked this pull request as ready for review November 20, 2026 06:04
adriangb and others added 2 commits November 20, 2026 10:21
Other tests in the same process open files concurrently, so a total
descriptor count with slack is flaky. The test now reads each
descriptor's target from /proc/self/fd and asserts that none points
at a live spill file. This makes the test Linux-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counting descriptors in a shared test process is either flaky or
platform-specific. The PR description holds a datafusion-cli script
that fails with EMFILE on main and passes with this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb thanks for this! Nothing in the PR checks that a live spill file holds no fd, so a change back to holding the file would still pass CI. If the test is the only test in its own binary, nothing else opens files while it counts, and /dev/fd works on both Linux and macOS. The test below fails on main (left: 404, right: 4) and passes on this branch.

datafusion/execution/tests/spill_file_descriptors.rs (add the ASF license header)
//! This file is its own test binary with a single test, so no other test
//! opens files while it counts this process's descriptors.

use std::io::Write;
use std::sync::Arc;

use datafusion_common::Result;
use datafusion_execution::disk_manager::DiskManagerBuilder;

#[cfg(unix)]
fn open_fd_count() -> usize {
    std::fs::read_dir("/dev/fd").unwrap().count()
}

#[cfg(unix)]
#[test]
fn live_spill_files_hold_no_file_descriptors() -> Result<()> {
    const FILES: usize = 400;
    let dm = Arc::new(DiskManagerBuilder::default().build()?);
    // The first file creates the spill directory.
    drop(dm.create_tmp_file("warm up")?);

    let before = open_fd_count();
    let mut files = Vec::with_capacity(FILES);
    for _ in 0..FILES {
        let file = dm.create_tmp_file("fd bound")?;
        let mut writer = file.open_writer()?;
        writer.write_all(b"spilled")?;
        writer.finish()?;
        files.push(file);
    }
    assert_eq!(open_fd_count(), before);
    for file in &files {
        assert_eq!(std::fs::read(file.path().unwrap())?, b"spilled");
    }
    Ok(())
}

Removing RefCountedTempFile::inner() is a breaking change, so it needs the api change label for the release notes, plus a short entry in docs/source/library-user-guide/upgrading/57.0.0.md. The entry can be the sentence you already have under user-facing changes: use path() for the path and the SpillFile methods to read or write.

Optional: the Rationale could open with the failing query instead of the mechanism, so a reviewer sees the need first. For example: "A sort or aggregate that keeps many spill runs alive fails with Too many open files (os error 24), because each live spill file keeps one fd open even when idle (400 live spill files = 400 fds on main). This PR holds spill files by path, so only open writers and readers use fds."

@adriangb

Copy link
Copy Markdown
Contributor Author

Thanks for the review @jayzhan211. I had actually removed the test in 558abf5 (which I think landed before your review? not sure). I couldn't find a good way to make this a reproducible test since things like file descriptors are global and also OS dependent. I offered a reproducer in the PR description instead.

@jayzhan211

Copy link
Copy Markdown
Contributor

@adriangb I see.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
@adriangb

Copy link
Copy Markdown
Contributor Author

Thanks @jayzhan211 !

@adriangb
adriangb enabled auto-merge November 21, 2026 18:51
@adriangb
adriangb added this pull request to the merge queue Nov 21, 2026
Merged via the queue into apache:main with commit c5361f0 Nov 21, 2026
41 of 42 checks passed
@adriangb
adriangb deleted the fix/spill-file-hold-path-not-fd branch November 21, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

execution Related to the execution crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants