REP-6586 Store rechecks locally rather than in metaDB #136
Conversation
|
|
||
| const ( | ||
| schemaVersionKey = "formatVersion" | ||
| schemaVersion = uint16(3) |
There was a problem hiding this comment.
Similar to the comparable metaDB mechanism. How we prevent inadvertent breakage from using an outdated localDB across verifier updates.
There was a problem hiding this comment.
It sounds like each time we change the schema we should update the schema version. Is it documented or tested anywhere?
| if errors.Is(err, badger.ErrKeyNotFound) { | ||
| return nil | ||
| } | ||
|
|
There was a problem hiding this comment.
Why are we not handling other errors?
There was a problem hiding this comment.
key-not-found is expected when the DB is new. There should be no other errors.
There was a problem hiding this comment.
Could we add an assertion that no other error should occur?
Co-authored-by: Jian Guan <[email protected]>
…igration-verifier into REP-6586-local-rechecks-badger
internal/verifier/localdb/recheck.go
Outdated
| return errors.Wrapf( | ||
| ldb.db.Update(func(tx *badger.Txn) error { | ||
| var addedRechecks uint64 | ||
| for _, chunk := range lo.Chunk(rechecks, 16384) { |
There was a problem hiding this comment.
chunk size could be a const.
|
This doesn’t actually work very well. It’s more performant to spin up a local MongoDB replset. |
This makes Migration Verifier enqueue rechecks to a local BadgerDB datastore rather than the metadata DB. This avoids problems with latency and allows the metadata to reside on the destination cluster without materially impeding handling of change events. The local DB is removed whenever the
--cleanflag is given.Since it is now sensible to store metadata on the destination cluster, this changeset also makes the
--metaURIparameter default to the--dstURIparameter’s value.It may be sensible in the future to migrate all of metadata to BadgerDB.