[DB] Introduce EVO database by random-zebra · Pull Request #2271 · PIVX-Project/PIVX

random-zebra · 2021-03-25T21:28:27Z

Extracted from #2267.
This introduces the new database, used for all DMN-related things, and updates the dbWrapper.

Builds on top of:

>>> backports dash@4531f6b896119b2b8f33c03a50cb4d4ee62a6acb Allows easier commit/rollback handling, especially useful when AcceptBlock fails and things need to be reverted.

>>> backports dash@c9a72e8880fe88f07b64aecfc059f08ec14440fc Also add transaction handling to ConnectTip and DisconnectTip and a few other places where blocks are processed.

>>> adapted from dash@378dadd0f7c73285ab5a13d1f6fb96314822170b

>>> Backports dash@a1cca1b9a11975b8a57d916d1325dc8d855217b7 the boolean value will lose its meaning in the next commit

>>> from dash@521d4ae08fe0e35c30bf05ffe8b9d611e86c3bfd CDBTransaction is changed to allow CDBBatch, CDBWrapper and other CDBTransactions as parent instead of just CDBWrapper. This in turn allows to implement multi-staged commits in CEvoDB. We now have the "current transaction" which is started and ended (commit or rollback) for each call to Connect-/DisconnectBlock. When the current transaction is committed, it moves its contents into the "root transaction" instead of directly writing to CDBWrapper. CommitRootTransaction() then handles the final commitment to CDBWrapper. It is called at the same time when the chainstate is flushed to disk, which guarantees consistency between chainstate and CEvoDB.

>>> backports dash@5b4fe43c2577f3f39fd174c84a8c23c99646903d This has the wanted side effect of proper locking of "cs" inside CommitCurTransaction and RollbackCurTransaction, which was not easily possible to implement in the generic version. This fixes some rare crashes.

This allow to pre-serialize keys in advance and pass the serialized form into these classes.

...instead of comparing by using the keys < operator. There are 2 reasons for this: 1. This better mimics the behavior of CDBWrapper. Otherwise there is a chance of discrepancy when it comes to key equality. 2. Next commit will introduce CDBTransactionIterator, which relies on CDBTransaction and CDBWrapper being compatible in the way keys are compared.

This iterator allows merged iteration of the key/values from the parent and the not-yet-committed key/values from the transaction. This also works for nested transactions (as used in CEvoDB). It's interface mimics CDBIterator.

>>> backports dash@1b3d0d9a347f1e40e9efa246ff3691c8e995383f

>>> backports dash@849bf3c76e62a980e89f2be2877ea2cc3728017c

it will be locked from outside when iterating mined commitments (ref. dash@c68b5f68aa41881c3bf36d8881b60b6855868605)

random-zebra · 2021-04-11T09:23:04Z

Rebased on master. Next in line for #2267

Fuzzbawls

Code ACK c27bbed

furszy · 2021-04-09T21:57:33Z

src/validation.cpp

future: would be good to maintain the best block hash cached inside a manager (manager that should be created, and the evoDb should be encapsulated there), preventing the disk access on every new connected block (slow operation).

src/dbwrapper.h

furszy

Code ACK c27bbed.

Have to add that spent some time thinking about this and will return to this area after finishing with #2267, so we can talk and brainstorm more about it after reviewing the complete set of changes.

6dbdf37 [Consensus] Serialize also shield inputs for ProReg inputshash (random-zebra) e0ff60f [Consensus] Serialize the whole scriptPayout for ProReg sign string (random-zebra) e659d7e [Trivial] Fix styling/logging nits (random-zebra) 170ab92 [Tests] Check for protx inputs hash in evo_specialtx_tests (random-zebra) a1cb227 [BUG] Add proper virtual dtor for CActiveDeterministicMasternodeManager (random-zebra) 0520ea9 [Init] Add -mnoperatorprivatekey to flag the active mn as deterministic (random-zebra) dcb5df9 [Refactoring] CADMM: GetLocalAddress static (random-zebra) 6cd5b0c [Refactoring] encapsulate activeMNINfo inside activeMNManager (random-zebra) ca9d10c Implement CActiveDeterministicMasternodeManager (Alexander Block) 66302c3 [Tests] Check deterministic masternodes unique properties (random-zebra) ddad010 [Validation] Check duplicate unique-properties for masternodes (random-zebra) 41869ec [Refactoring] Add IsDIP3Enforced/LegacyMNObsolete funcs to DMNManager (random-zebra) c013bf1 [Refactoring] Add evo Notification Interface (random-zebra) 180a311 [Validation] Connect deterministic manager to block processing (random-zebra) 7f2d2b5 [MOVE-ONLY] move special_tx_validation_test inside evo_* test file (random-zebra) 9ac0f37 [Tests] Introduce evo_deterministicmns_tests.cpp (random-zebra) 48c66dc Conflict handling for ProRegTx in mempool (Alexander Block) ae33972 [Core] Deterministic Masternode List implementation (random-zebra) 50c277b Implement std::unordered_map/set compatible hasher classes for salted hashes (Alexander Block) 0c064f8 [Tests] Add tests for SetTxPayload/GetTxPayload and CheckStringSig (random-zebra) 693772a [Core] Introduce ProRegTx payload (random-zebra) 6fb2496 Define hash function for uint256 to be used in STL-like containers (random-zebra) 3f74c88 [Build] CMake: add immer headers (random-zebra) e067443 [Build] Add "immer" functional/immutable containers library (random-zebra) b92de4d [Consensus] Reject special txes before V6 enforcement (random-zebra) Pull request description: Extracted from #2267. This is the first fundamental PR. Here we introduce the `PROREG` transaction type, the DMN manager (and active-masternode manager), and all the required classes (`CDeterministicMN`, `CDeterministicMNState`, `CDeterministicMNStateDiff`, `CDeterministicMNList` and `CDeterministicMNListDiff`). This contains also the headers-only immer library for persistent immutable data structures. Reasons for the inclusion are explained in #2267 (under "code architecture"). The headers are copied over, at the same commit used by Dash (`0a718d2d76bab6ebdcf43de943bd6c7d2dbfe2f9`). Later on, we might want to properly include this as a git subtree. Built on top of: - [x] #2271 ACKs for top commit: furszy: Code review ACK 6dbdf37. Fuzzbawls: Code ACK 6dbdf37 Tree-SHA512: 3b175eef5cd03c73ee4d125063a4a5f5a5eff6064fb6ff3f54c01b6f2d48562c6181cbfa2687025a2fc9a65a80b17d799581cba1f2937333dfee4c3212b5cfac

random-zebra added Refactoring UTXO DBs and Indexes Masternode Upstream Dash labels Mar 25, 2021

random-zebra added this to the 6.0.0 milestone Mar 25, 2021

random-zebra self-assigned this Mar 25, 2021

This was referenced Mar 25, 2021

[Doc] Introducing Deterministic Masternodes #2267

Merged

[TierTwo] Deterministic masternode lists #2273

Merged

random-zebra force-pushed the 202103-evo-db branch from 314ac22 to edbb529 Compare March 26, 2021 16:12

random-zebra force-pushed the 202103-evo-db branch 4 times, most recently from 3eb908c to b48163e Compare April 7, 2021 17:22

random-zebra and others added 17 commits April 11, 2021 10:20

[Trivial] Remove unused variable in ProcessNewBlock

f3a7cfb

[Build] CMake: Introduce evo headers

da6a2c1

Implement CDBTransaction and CScopedDBTransaction

2496933

>>> backports dash@4531f6b896119b2b8f33c03a50cb4d4ee62a6acb Allows easier commit/rollback handling, especially useful when AcceptBlock fails and things need to be reverted.

Introduce CEvoDB for all evo related things, e.g. DIP3

d6c56d1

>>> backports dash@c9a72e8880fe88f07b64aecfc059f08ec14440fc Also add transaction handling to ConnectTip and DisconnectTip and a few other places where blocks are processed.

[DB][BUG] Add virtual destructor for KeyValueHolder

ad7f5d7

[Tests] Fix functional test suite with new directory 'evodb'

cab50d3

[Refactoring] migrate evoDb to unique pointer

be85c9a

[DB] Ensure evoDB consistency by storing best block hash

fec56b0

>>> adapted from dash@378dadd0f7c73285ab5a13d1f6fb96314822170b

[Refactoring] Let Commit() return void

e6c7efe

>>> Backports dash@a1cca1b9a11975b8a57d916d1325dc8d855217b7 the boolean value will lose its meaning in the next commit

[Refactoring] Properly add explicit specifier where appropriate

34a7541

[Cleanup] Drop unneeded casts

cdd44d4

Support passing CDataStream as key into CDBWrapper/CDBBatch/CDBIterator

7094a60

This allow to pre-serialize keys in advance and pass the serialized form into these classes.

Implement CDBTransactionIterator

f259774

This iterator allows merged iteration of the key/values from the parent and the not-yet-committed key/values from the transaction. This also works for nested transactions (as used in CEvoDB). It's interface mimics CDBIterator.

Track memory usage in CDBTransaction and CEvoDB

ed0d4f9

>>> backports dash@1b3d0d9a347f1e40e9efa246ff3691c8e995383f

random-zebra added 3 commits April 11, 2021 10:22

Take memory used by CEvoDB/CDBTransaction into account when flushing

304fc46

>>> backports dash@849bf3c76e62a980e89f2be2877ea2cc3728017c

[Refactoring] Make CEvoDB mutex public

8850b3e

it will be locked from outside when iterating mined commitments (ref. dash@c68b5f68aa41881c3bf36d8881b60b6855868605)

[Cleanup] functions impl: branch on new line

c27bbed

random-zebra force-pushed the 202103-evo-db branch from b48163e to c27bbed Compare April 11, 2021 09:22

Fuzzbawls approved these changes Apr 13, 2021

View reviewed changes

furszy reviewed Apr 13, 2021

View reviewed changes

furszy approved these changes Apr 13, 2021

View reviewed changes

furszy merged commit 75ed795 into PIVX-Project:master Apr 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DB] Introduce EVO database#2271

[DB] Introduce EVO database#2271
furszy merged 20 commits intoPIVX-Project:masterfrom
random-zebra:202103-evo-db

random-zebra commented Mar 25, 2021 •

edited

Loading

Uh oh!

random-zebra commented Apr 11, 2021

Uh oh!

Fuzzbawls left a comment

Uh oh!

furszy Apr 9, 2021

Uh oh!

Uh oh!

furszy left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

random-zebra commented Mar 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

random-zebra commented Apr 11, 2021

Uh oh!

Fuzzbawls left a comment

Choose a reason for hiding this comment

Uh oh!

furszy Apr 9, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

furszy left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

random-zebra commented Mar 25, 2021 •

edited

Loading