Skip to content

Conversation

@ripatel-fd
Copy link
Contributor

@ripatel-fd ripatel-fd commented Oct 31, 2025

This patch integrates the snapshot loader with Firedancer's new disk-backed account database.
Adds a line-rate capable O_DIRECT write path with cache bypass.
Has minimal impact to the existing funk-based database logic, except for minor cleanups along the way.

  • restore: separate out funk DB logic
  • restore: initial vinyl integration
  • restore: async O_DIRECT via worker thread

@ripatel-fd ripatel-fd force-pushed the ripatel/vinyl-io-wd-2 branch 2 times, most recently from 72ab702 to ca1e6cc Compare October 31, 2025 20:46
@github-actions
Copy link

Performance Measurements ⏳

Suite Baseline New Change
backtest mainnet-368528500-perf per slot 0.075966 s 0.076496 s 0.698%
backtest mainnet-368528500-perf snapshot load 2.637 s 2.623 s -0.531%
backtest mainnet-368528500-perf total elapsed 75.966062 s 76.496284 s 0.698%
firedancer mem usage with mainnet.toml 974.13 GiB 974.13 GiB 0.000%

@ripatel-fd ripatel-fd force-pushed the ripatel/vinyl-io-wd-2 branch from ca1e6cc to ce878a1 Compare October 31, 2025 20:58
@github-actions
Copy link

Performance Measurements ⏳

Suite Baseline New Change
backtest mainnet-368528500-perf per slot 0.073036 s 0.072985 s -0.070%
backtest mainnet-368528500-perf snapshot load 2.678 s 2.619 s -2.203%
backtest mainnet-368528500-perf total elapsed 73.035807 s 72.985296 s -0.069%
firedancer mem usage with mainnet.toml 974.13 GiB 974.13 GiB 0.000%

lidatong
lidatong previously approved these changes Oct 31, 2025
@ripatel-fd ripatel-fd force-pushed the ripatel/vinyl-io-wd-2 branch from 63795be to 1c93fc2 Compare October 31, 2025 21:19
@ripatel-fd ripatel-fd requested a review from amass-jump October 31, 2025 21:19
@github-actions
Copy link

Performance Measurements ⏳

Suite Baseline New Change
backtest mainnet-368528500-perf per slot 0.073056 s 0.072782 s -0.375%
backtest mainnet-368528500-perf snapshot load 2.663 s 2.634 s -1.089%
backtest mainnet-368528500-perf total elapsed 73.055607 s 72.782402 s -0.374%
firedancer mem usage with mainnet.toml 974.13 GiB 974.13 GiB 0.000%

fputs( "--------------------------------------------", stdout );
if( snapwr_tile ) fputs( "--------------", stdout );
fputs( "[ct],[ld],[dc],[in]--------[ct],[ld],[dc],[in]", stdout );
if( snapwr_tile ) fputs( ",[wr]" , stdout );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about similar changes in watch?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the list of dummy tiles in fd_tile_unit_test.c important? (I was just looking at files I changed when adding the snapld tile)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I'm aware of since fd_tile_unit_test.c is probably not supposed to test the snapshot loader

handle_control_frag( fd_snapwr_t * ctx,
ulong meta_ctl,
ulong meta_sig ) {
switch( meta_ctl ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it make sense to FD_TEST that we are in IDLE state when receiving any of these?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, snapin->snapwr is simplified. snapin just sends a SHUTDOWN while snapin is still in PROCESSING state, because snapwr just blindly writes to offsets specified by snapin. It isn't aware what kind of snapshot is being written and whether anything was reset.


populate_sock_filter_policy_fd_snapin_tile( out_cnt, out, (uint)fd_log_private_logfile_fd() );
return sock_filter_policy_fd_snapin_tile_instr_cnt;
(void)topo;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOC, are we not supposed to use FD_PARAM_UNUSED?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kevin prefers (void).

@ripatel-fd ripatel-fd merged commit bf71b6f into main Nov 1, 2025
12 checks passed
@ripatel-fd ripatel-fd deleted the ripatel/vinyl-io-wd-2 branch November 1, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants