Skip to content

[DSV3] Forward and backward pass for single GPU #1320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jun 23, 2025
Merged

[DSV3] Forward and backward pass for single GPU #1320

merged 4 commits into from
Jun 23, 2025

Conversation

wwwjn
Copy link
Contributor

@wwwjn wwwjn commented Jun 19, 2025 •
edited
Loading

Command to run: NGPU=1 CONFIG_FILE="./torchtitan/models/deepseek_v3/train_configs/debug_model.toml" ./run_train.sh

Context

  1. Added model args for 4 model settings, and training config for debug model
  2. Debugged the forward pass, and the backward pass works out of pocket.
  3. Reused c4-test dataset, and tiktokenizer from llama3 model for current testing

Screenshot 2025-06-20 at 11 52 49 AM

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jun 19, 2025
@wwwjn wwwjn requested review from H-Huang and tianyu-l June 19, 2025 15:28
tianyu-l
tianyu-l reviewed Jun 19, 2025
View reviewed changes
Copy link
Contributor

@tianyu-l tianyu-l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks quite good! left some comments

H-Huang
H-Huang reviewed Jun 20, 2025
View reviewed changes
@wwwjn wwwjn requested review from H-Huang and tianyu-l June 20, 2025 19:09
H-Huang
H-Huang approved these changes Jun 20, 2025
View reviewed changes
Copy link
Member

@H-Huang H-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks for getting working so quickly!

@wwwjn wwwjn merged commit 968a889 into deepseek-v3 Jun 23, 2025
5 checks passed
@tianyu-l tianyu-l deleted the dsv3-configs branch June 24, 2025 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@H-Huang H-Huang H-Huang approved these changes

@tianyu-l tianyu-l Awaiting requested review from tianyu-l

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wwwjn @H-Huang @tianyu-l @facebook-github-bot