F05dump #1
Open
F05dump #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently various groups still rely on this, and spend hours generating it via lotus. Doubles as a demo of lotus-less API implementation doing same work in ~6 minutes from a car file, or ~20 minutes from a lotus node via RPC ( vastly unoptimized, can be made still MUCH faster ) Usage: ./filexp dump-statemarketdeals --rpc-fullnode=ws://localhost:1234 > SMD.json
Processing ~70 million entries using an inefficient "swarm of goroutines" puts exortbitant pressure on GC, especially notable in low-memory environments. While one can not solve all of the problems (due to low-level IPLD libs being meh), a number of reductions can help: - switch to a static number of consumer-workers - reuse a single static encode-buffer per worker (json.Encode does not help) - keep as much of possible on the stack - reuse the variables within loops where it makes sense - pass the from-IPLD struct by ref where possible - use a single mutex syncpoint, remove atomic incs entirely
Nearly all use cases of filexp (except for the FVM cmds) do not really benefit from counting unique CIDs. In cases of large scale dumps this counting becomes so expensive that it dominates the execution time. Make the counting optional except under FVM, gated by --count-unique-cids
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.