sync layer norms #272
Merged
sync layer norms #272
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Draft
We should be able to store a |
thomasw21
reviewed
May 24, 2022
younesbelkada
pushed a commit
to younesbelkada/Megatron-DeepSpeed
that referenced
this pull request
Sep 28, 2022
* sync layer norms * all_reduce is an in_place operation * Make dataloader use another random generator (bigscience-workshop#276) * do all_reduce op.AVG directly * add eval dataloader deadlock workaround * revert generator sync * make auto-sync configurable; basic test; cleanup * test with updated AMI image * fix unrelated test Co-authored-by: thomasw21 <[email protected]>
adammoody
pushed a commit
to adammoody/Megatron-DeepSpeed
that referenced
this pull request
Dec 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Extracting just the fix for now from #271 - since currently the test suite runs against deepspeed master which is missing the new bf16 code.
Hopefully we won't merge this, as I'm concerned with us imposing an unnecessary code for other users - if we do merge we will have to parametrize the
if 1branch.edit: decided to merge this but to make it configurable so that we could run other PR branches and not having a headache of merging multiple branches all the time, which is too error prone.
cc: @thomasw21