Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[UB] Adding support for multinode nvlink #815

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

[UB] Adding support for multinode nvlink #815

wants to merge 26 commits into from

Conversation

shamisp
Copy link
Contributor

@shamisp shamisp commented Apr 26, 2024 •
edited
Loading

This adds support for multi-node nvlink architecture. In addition it includes changes for making CE deadlock checker configurable at the runtime.

@shamisp shamisp changed the title [UB] Adding support for multinode nvlink [WIP] [UB] Adding support for multinode nvlink Jun 4, 2024
@shamisp
Copy link
Contributor Author

shamisp commented Jun 4, 2024

@denera I think this branch is ready for upstream but I know u have outstanding changes as well.

@denera
Copy link
Collaborator

denera commented Jun 4, 2024

@shamisp Let's review and merge this branch. I will rebase my PR on top later.

@timmoon10 timmoon10 requested a review from denera June 12, 2024 21:19
timmoon10
timmoon10 reviewed Jun 13, 2024
View reviewed changes
Copy link
Collaborator

@timmoon10 timmoon10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked closely at the implementation and my changes are mostly stylistic. Also, DCO is complaining about an unsigned commit.

@timmoon10 timmoon10 self-requested a review June 13, 2024 00:26
@timmoon10
Copy link
Collaborator

/te-ci pytorch

shamisp added 22 commits June 26, 2024 21:28
@shamisp
Addingi basic MNNVL
b49ed66 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixes for unit-tests
afab890 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Integration APIs
5cdd0db 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Adding code for collecting info about NVML domain
42fccd0 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Adding macro to provide an option to enable/disable CE deadlock detector
e228210 
Since CE deadlock detector launch and additional increment kernel it
may introduce an overhead to the communication flow. Adding in
option to disable it.

Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Adding integration with NVML interface
dfbeda0 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Improving the code and removing assumtions
42ef57b 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Improving error code checking
c6dbeae 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Support for multiple TP per dimension
2b2bc3c 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Minor fixes
80613d7 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing MPI error macro
4c8bab2 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing lint warnings
e95f2df 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Restoring cast to void
3fa0a69 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Adding multi-tensor group support
79a2cff 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Changes in CE deadlock detection
61c69e9 
Making CE deadlock detection a runtime option, which is disabled by
defaults

Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Lint cleanup
af36b50 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Addressing build issue on system with mismatching cuda/driver versions
038f560 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing config flow for MNNVL
115046c 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Post rebase adjustment
1283664 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing clique detection
922308e 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing Bcast logic
80d3bba 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Increasing MAX regions to accommodate large domains
34db0d7 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
shamisp added 2 commits June 26, 2024 21:28
@shamisp
Increasing max nvlink size
3227f5e 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Addressing reviewers comments
90d371c 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
shamisp added 2 commits June 28, 2024 12:02
@shamisp
Adding debug prints
8878121 
Signed-off-by: Pasha (Pavel) Shamis <[email protected]>
@shamisp
Fixing issues on H100 systems
ae28fa6 
Signed-off-by: Pavel Shamis (Pasha) <[email protected]>
@denera denera mentioned this pull request Aug 2, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@denera denera Awaiting requested review from denera

@timmoon10 timmoon10 Awaiting requested review from timmoon10

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shamisp @denera @timmoon10