Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major compaction API #105

Open
marvin-j97 opened this issue Jan 31, 2025 · 0 comments
Open

Major compaction API #105

marvin-j97 opened this issue Jan 31, 2025 · 0 comments
Labels
api enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@marvin-j97
Copy link
Contributor

marvin-j97 commented Jan 31, 2025

This is already partially implemented as a hidden function, however it needs to be exposed as a method in AbstractTree (so it can be called for BlobTree as well, which will require a full rewrite of the entire value log (...maybe not)).

lsm-tree/src/tree/mod.rs

Lines 462 to 478 in 3a2e3e1

// TODO: Expose as public function, however:
// TODO: Right now this is somewhat unsafe to expose as
// major compaction needs ALL segments, right now it just takes as many
// as it can, which may make the LSM inconsistent.
// TODO: There should also be a function to partially compact levels and individual segments
/// Performs major compaction, blocking the caller until it's done.
///
/// # Errors
///
/// Will return `Err` if an IO error occurs.
#[doc(hidden)]
pub fn major_compact(&self, target_size: u64, seqno_threshold: SeqNo) -> crate::Result<()> {
log::info!("Starting major compaction");
let strategy = Arc::new(crate::compaction::major::Strategy::new(target_size));
self.compact(strategy, seqno_threshold)
}

It's important that the major compaction can get all segments, so it needs a write lock and wait until it can hide all segments.

@marvin-j97 marvin-j97 added enhancement New feature or request api labels Jan 31, 2025
@marvin-j97 marvin-j97 added good first issue Good for newcomers help wanted Extra attention is needed labels Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant