Improved Particle Synchronization and Distributed Cancellation #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… PI and BPI now use sync_banks. Not fully working.
* added sync_banks * Fixed Spelling errors * Changed sync_banks to work with MPI_Send and MPI_Recv * Rearrangements and addition of comments * Verified changes --------- Co-authored-by: Hunter Belanger <[email protected]>
…o work with new MPI.
* added sync_banks * Fixed Spelling errors * Changed sync_banks to work with MPI_Send and MPI_Recv * Rearrangements and addition of comments * Verified changes * Removed unused method verify * Made initial setKeys * improved setAllKeys and renamed to sync_keys * Added some clears and comments * Implemented perform_cancellation_vector and perform_cancellation_loop * clang-formatted the files * Added vector Bcast * Made necessary changes to perform_cancellation and sync_keys, still need to test * Fixed clear * Working perform_cancellation_loop and perform_cancellation_vector with main tests * Updated with comments * Updated main test cases * Changed stuff for testing * Made little changes * Fixed bug vector * Added Uint16 and working loop version with updated particle weights * Working vector version with NDArray and updated cancellation, main test cases looking better * Small naming and formatting changes. * Fixed warnings * Small floating point optimizations. * First version of comb with bug * Fixed comb bugs * Fix reproducibility bug with new combing * Fixes some small bugs and stuffs. * Added pair and Key type * Started sync_keys * Fixed comb to do negative and positive at the same time * Comb working * Added spaceship to Key, Uint32 to mpi, changed naming of key and pair, implemented sync_keys and cancellation across all nodes for exact cancellator * Fixed silly bug * Made comb prettier * Small changes to formatting, and updates a use of vector to NDArray * Fix bug/typo introduced in last commit * Only master has uniform weights for generating uniform particles. * Only add uniform particles to next bank if there are any. * Adds kahan summation to try to improve combing reproducibility. * Replaced kahan with std::accumulate, kahan was producing inconsistent results * Adds fixes for ExactMGCancelator. * Added kahan babushka sums * Reviewed comments and changed calculations of Nneg and Npos * Fixed * Add inline keywords to kahan functions. * Use kahan for PowerIterator particle normalization. * Clean up some of BranchlessPowerIterator changes. * Fix bug in MPI type create of Key int pair. * Small changes to uint types in Key. * Fix potential bug in MPI version of ExactMGCancelation. NOT WORKING ! * Added some prints to debug * Added first draft of number_of_cell_instances and get_all_contained_cells * Revert "Added first draft of number_of_cell_instances and get_all_contained_cells" This reverts commit ec85960. * Fix bugs in MPI DataType creation. * Fix many bugs in ExactMGCancelator. Should be working with MPI now. * Clean up some things after the merge. Still working. * Update Noise simulation mode for new cancellation and fission bank syncing. * Increase buffer for estimate of time to complete simulation. * Some clean up for future integration. Needs a clang format. * Applies a clang-format. --------- Co-authored-by: Hunter Belanger <[email protected]>
…tor in Approximate cancelator.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR incorporates an improved MPI communication scheme for synchronizing the particle banks between nodes. Instead of sending all particles to master and then scattering (which was before needed for cancellation), particles are now sent between nodes, minimizing the amount of data which needs to be transferred. Each node will just send particles left or right to their neighbor.
To accommodate this, weight cancellation has also been modified to work in a distributed manner, so that all nodes can help in performing the cancellation computation. Additionally, this means particles no longer need to be sent to master for cancellation, just to be sent back to all the other nodes.
The
BasicExactMGCancellatorclass will be removed in favor of theExactMGCancellatorclass. This is because the former is not actually exact in ALL cases, and its additional runtime options are less efficient than the optimized implementation in the later. It will also be very difficult to implement a version ofBasicExactMGCancellatorwhich generates results independent of the parallelization scheme.Lastly, a distributed particle comb algorithm has also been implemented for use with branchless collisions. Before, particles needed to be sent to master for combing. While the cancellators and the comb generate results which are independent of the parallelization scheme, it appears that using both of them at the same time leads to too much variation in the lower precision places, and will give different results for different parallelization schemes. At the moment, it does not appear as though there is a realistic solution for this problem, which would ensure reproducibility.
TODO:
BasicExactMGCancellator