Add file-based blob store #2554

MarkJr94 · 2019-01-25T12:49:34Z

This is a work in progress replacement for the storage functionality of DbLedger.

See Issue #2566

upcoming subtasks:

src/blob_store.rs

garious · 2019-01-28T17:08:18Z

Can you create an issue (maybe several) to track this work and add it to your PR description?

MarkJr94 · 2019-01-28T17:17:10Z

@garious on it

rob-solana · 2019-01-30T17:43:18Z

src/blob_store/store_impl.rs

+        let mut buf = [0u8; INDEX_RECORD_SIZE as usize];
+        while let Ok(_) = index_file.read_exact(&mut buf) {
+            let index = BigEndian::read_u64(&buf[0..8]);
+            if index == blob_index {


this code takes N/2 to find the index record, it could be constant time

Wouldn't I have to ensure that indexes were written in-order to make that possible?

no, you can seek() to the spot you need to write() at in the index file. the file will grow automatically, and will be filled with zeros (which you can call "invalid" offsets)

Got it. I'm already doing this in another place, will make this change tomorrow

rob-solana · 2019-01-30T17:48:04Z

benches/blob_store.rs

+    // Generate a num_reads sized random sample of indexes in range [0, total_blobs - 1],
+    // simulating random reads
+    let mut rng = rand::thread_rng();
+    let indexes: Vec<usize> = (0..num_reads)


are you intending to verify that caching helps, once implemented?

Yes. I've also experimented with parallelizing writing (partitioned based on slots) but I couldn't get it to work safely without needing to copy so much memory that it was much slower overall.

The little caching I've done so far made benchmark ns/iter ~70-80% of db_ledger benchmarks when before it was about ~110-120% .

I tried several different ways of exploiting concurrency including using tokio and futures, creating a separate writer thread that communicated over std::sync::mpsc and crossbeam-channel:: channels, etc.

But I could not do it both 1) safely 2) without so much copying that it made everything much slower.

aeyakovenko · 2019-01-31T14:55:28Z

can you guys sync up with @sakridge on sakridge@aea2639

sakridge · 2019-02-04T22:46:18Z

src/blob_store/recordfile.rs

+pub struct RecordFile<T> {
+    file: File,
+    current_offset: u64,
+    buf: RefCell<Vec<u8>>,


Why is buf in the RecordFile struct and not just a stack variable in the calling functions if needed?

That was my original goal, but I ran into the fact that there's no way to use an associated consts from a trait in an array expression currently [1] so I decided to use a single vector that was sized appropriately.

[1] rust-lang/rust#42863

pgarg66 · 2019-02-05T18:57:27Z

@MarkJr94 , I am trying to build this PR locally, and getting these build errors. Does it need rebase?

  Compiling solana v0.12.0 (/home/pankaj/pgarg66/solana)
error[E0658]: imports can only refer to extern crate names passed with `--extern` on stable channel (see issue #53130)
  --> src/blob_store.rs:21:5
   |
21 | use store::Key;
   |     ^^^^^
...
25 | pub mod store;
   | -------------- not an extern crate passed with `--extern`
   |
note: this import refers to the module defined here
  --> src/blob_store.rs:25:1
   |
25 | pub mod store;
   | ^^^^^^^^^^^^^^

error[E0658]: imports can only refer to extern crate names passed with `--extern` on stable channel (see issue #53130)
   --> src/blob_store.rs:135:13
    |
25  | pub mod store;
    | -------------- not an extern crate passed with `--extern`
...
135 |         use store::Named;
    |             ^^^^^
    |
note: this import refers to the module defined here
   --> src/blob_store.rs:25:1
    |
25  | pub mod store;
    | ^^^^^^^^^^^^^^

error[E0658]: imports can only refer to extern crate names passed with `--extern` on stable channel (see issue #53130)
   --> src/blob_store.rs:220:13
    |
25  | pub mod store;
    | -------------- not an extern crate passed with `--extern`
...
220 |         use store::Named;
    |             ^^^^^
    |
note: this import refers to the module defined here
   --> src/blob_store.rs:25:1
    |
25  | pub mod store;
    | ^^^^^^^^^^^^^^

error[E0658]: use of unstable library feature 'int_to_from_bytes' (see issue #52963)
  --> src/blob_store/store_impl.rs:15:22
   |
15 |     let splat = slot.to_be_bytes();
   |                      ^^^^^^^^^^^

error: aborting due to 4 previous errors

For more information about this error, try `rustc --explain E0658`.
error: Could not compile `solana`.

MarkJr94 · 2019-02-05T20:28:06Z

@pgarg66 I'm looking into it right now, issue is likely because I have been using the nightly compiler locally.

…nd blob accessors unimplemented

only accepted as individual raw Vec<u8> buffers for now

… are

about 35% faster now, faster than DbLedger in benchmarks.

blob.index and blob.slot no longer return `Result`s

…a facade

MarkJr94 · 2019-02-05T21:15:10Z

@pgarg66 should now build

pgarg66

@MarkJr94 , I think I am making assumptions while reviewing the code. Do you happen to have a small writeup/diagram that explains how different pieces interact with each other? My current understanding is this:

BlobStore is the top level interface, and it exports functionality to store blobs per slot. Also it tracks the meta information for the slot (received, consumed, num blobs etc)
Store provides get/put APIs for blobs and meta
StoreImpl is providing caching and storing functionality
RecordFile is filesystem wrapper

Also, there's a ton of use of Generics. Wondering if it's really needed?

I think some sort of documentation will help me understand the code better.

pgarg66 · 2019-02-06T19:14:07Z

src/blob_store/store.rs

+    }
+
+    #[inline]
+    pub fn put_dyn<T>(&mut self, column: &str, key: Key, obj: T) -> Result<()>


These methods (gets/puts) are mostly called with the same type of T. Any good reason to use generics here?

The intention was just to make things generic so that I wasn't duplicating code. The traits are definitely excessive I think. They'll be replaced with methods that just accept any T: Serialize for put and returns bytes or any T: Deserialize for gets.

sakridge · 2019-02-07T18:58:32Z

@MarkJr94 Can we also try to split this PR into something smaller? Looks like RecordFile can be a separate PR, then look at the layers on top of that that can be merged.

MarkJr94 · 2019-02-13T23:39:13Z

@pgarg66 Yes I should've started with a design document. I'll have one up tomorrow, linked in the issue, (possibly as a PR to the book).

@sakridge Yes I think I'll split this up. I'm not sure if should close this PR for now then? The issue #2554 is where I'm tracking what I"m doing now.

…labs#2554)

MarkJr94 added the work in progress This isn't quite right yet label Jan 25, 2019

MarkJr94 requested a review from rob-solana January 25, 2019 12:49

rob-solana reviewed Jan 25, 2019

View reviewed changes

src/blob_store.rs Outdated Show resolved Hide resolved

rob-solana reviewed Jan 25, 2019

View reviewed changes

src/blob_store.rs Show resolved Hide resolved

rob-solana reviewed Jan 25, 2019

View reviewed changes

src/blob_store.rs Outdated Show resolved Hide resolved

MarkJr94 mentioned this pull request Jan 28, 2019

Replace DbLedger with simple blob store #2566

Closed

22 tasks

MarkJr94 force-pushed the blob-store branch 2 times, most recently from 3bc54e4 to 8dd82a5 Compare January 30, 2019 03:11

rob-solana reviewed Jan 30, 2019

View reviewed changes

MarkJr94 force-pushed the blob-store branch from 8dd82a5 to 77d1ca8 Compare January 31, 2019 08:28

MarkJr94 force-pushed the blob-store branch from 6e1408c to 00e687a Compare February 2, 2019 06:46

garious changed the title ~~File based blob store~~ Add file-based blob store Feb 4, 2019

rob-solana requested review from pgarg66 and sakridge February 4, 2019 20:25

sakridge reviewed Feb 4, 2019

View reviewed changes

mark-solana added 9 commits February 5, 2019 14:28

initial work on simplified blob store. erasure unimplemented; entry a…

da7a5ac

…nd blob accessors unimplemented

Add basic storage and retrieval of erasure codes

d6ef456

only accepted as individual raw Vec<u8> buffers for now

split slot paths (/root/0x7788 => /root/0x77/88)

153ef75

Add and store more per-slot metadata

448efc1

Add per-slot entry retrieval

5151b53

moved blob_store tests to integration test folder as that's what they…

5860094

… are

created benchmark for blob_store matching db_ledger

2599ec7

Batch index and blob writing

bbb8198

about 35% faster now, faster than DbLedger in benchmarks.

Separated out per-slot I/O in prep for async

819c3e4

mark-solana added 9 commits February 5, 2019 14:28

Move to more free functions; remove excess unwrap

f3f46ab

blob.index and blob.slot no longer return `Result`s

added append vec; small refactoring

ce65462

more generic store; support for unique and multiple per-slot values

35c3acc

Make backing store generic over what it stores. BlobStore now mostly …

5c8220a

…a facade

simplify keys and remove generics

3a83665

added cross-slot retrieval and tracking of occupied slots

1d5af3f

fix non-looping loop lint; simplify/correct record iteration logic

6c3e6f0

fix rebase errors

6c45524

Rebase and fix clippy lints + Rust edition

032fb3f

MarkJr94 force-pushed the blob-store branch from 00e687a to 032fb3f Compare February 5, 2019 21:13

pgarg66 reviewed Feb 6, 2019

View reviewed changes

MarkJr94 added the noCI Suppress CI on this Pull Request label Feb 14, 2019

MarkJr94 closed this Feb 14, 2019

yihau pushed a commit to yihau/solana that referenced this pull request Aug 13, 2024

unified_scheduler_logic: replace get_account_locks_unchecked (solana-…

fc208a0

…labs#2554)

Add file-based blob store #2554

Add file-based blob store #2554

Uh oh!

Conversation

MarkJr94 commented Jan 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

garious commented Jan 28, 2019

Uh oh!

MarkJr94 commented Jan 28, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aeyakovenko commented Jan 31, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarkJr94 Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgarg66 commented Feb 5, 2019

Uh oh!

MarkJr94 commented Feb 5, 2019

Uh oh!

MarkJr94 commented Feb 5, 2019

Uh oh!

pgarg66 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sakridge commented Feb 7, 2019

Uh oh!

MarkJr94 commented Feb 13, 2019

Uh oh!

Uh oh!

MarkJr94 commented Jan 25, 2019 •

edited

Loading

MarkJr94 Feb 5, 2019 •

edited

Loading