Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Headerwithproof migration and prune #3053

Draft
wants to merge 2 commits into
base: remove-union-in-blockheaderwithproof
Choose a base branch
from
Draft

Headerwithproof migration and prune #3053

wants to merge 2 commits into from

Conversation

kdeme
Copy link
Contributor

@kdeme kdeme commented Feb 7, 2025 •
edited
Loading

Playing around with options regarding database changes that occur in #3019

kdeme added 2 commits February 7, 2025 16:42
@kdeme
Header with proof migration and pruning: naive approach
0bf3a78 
Naive as it iterates over all content items, checks serialization
and then updates or deletes per individual item.

This takes too long to be run at start of fluffy and would only be
useful for small database and separate pruning, e.g. in fcli.
@kdeme
Quicker prune version by using one sql delete statement
8008aef 
Add custom function to be able to get all keys which hold headers
without a proof and delete in one sql statement.

Does not resolve updating of headers with proof.
@kdeme
Copy link
Contributor Author

kdeme commented Feb 7, 2025

Naive version is way too slow even for relative small databases.
It highly depends on how many actual headers with proof are stored, which when launching a new node might not be the same for some of our long running nodes. Especially as when testing I use the portal bridge to locally dump a lot of data in it.

But to give an example:
Database filled from the network: 1.1G Jan 14 17:25 fluffy.sqlite3

NTC 2025-01-29 15:28:10.989+01:00 ContentDB migration: iterating over all content
NTC 2025-01-29 15:55:48.725+01:00 ContentDB migration done:                  contentAltered=46903 contentDeleted=223548

-> 27:38 mins
The important value is here 46903 updates (well rather INSERT OR REPLACE) and + 223548 deletes.

@kdeme
Copy link
Contributor Author

kdeme commented Feb 7, 2025

Delete version (8008aef) is fine in terms of speed, but will not update the existing values that could be migrated.

On a big database with 953240 required deletions:

NTC 2025-02-07 16:31:32.686+01:00 ContentDB migration: deleting all headers without proof
NTC 2025-02-07 16:32:02.274+01:00 ContentDB migration done

Might be able to do a similar update statement, not sure if supported with current API and what it would do in terms of memory.

Simpler option could be to just delete all, and re-inject all headers-only via our portal_bridge.

Secondary question is, whether to move this in to fcli as separate action to take or not?
In case of not, we probably need to add a version table or use sqlite PRAGMA user_version. The latter seems however already used for https://github.com/status-im/nim-eth/blob/master/eth/db/kvstore_sqlite3.nim#L595C32-L595C44

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kdeme