Skip to content

allow disabling ft checkpoints #1810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 12, 2025
Merged

allow disabling ft checkpoints #1810

merged 1 commit into from
Oct 12, 2025

Conversation

tushar00jain
Copy link
Contributor

@tushar00jain tushar00jain commented Oct 7, 2025 •
edited
Loading

Summary:
Allows disabling the storage of checkpoints related to torchft.

Users don't really have to rely on any external storage. So it reduces set up time to get things up and running. Since we also don't really need model checkpoints when we have torchft. And if checkpoint storage has issues, this can work as a killswitch to completely disable the storage so it doesn't impact training.

Stack created with Sapling. Best reviewed with ReviewStack.

This was referenced Oct 7, 2025
Merged
Draft
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 7, 2025
@tushar00jain tushar00jain mentioned this pull request Oct 7, 2025
Merged
@tushar00jain tushar00jain force-pushed the pr1810 branch 2 times, most recently from 6ffcdf6 to 4fbe143 Compare October 7, 2025 21:54
fegin
fegin requested changes Oct 8, 2025
View reviewed changes
Copy link
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comments as other PRs, it is hard to review because this PR contains changes from the previous PRs. I would suggest that you use ghstack, which is the standard tool used by most PyTorch developers to stack PRs.

@tushar00jain tushar00jain force-pushed the pr1810 branch 2 times, most recently from 1c1c5a2 to 634d838 Compare October 8, 2025 17:44
fegin
fegin requested changes Oct 8, 2025
View reviewed changes
Copy link
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make the option more specific to allow retrain the data and add some warning.

@tushar00jain tushar00jain force-pushed the pr1810 branch 3 times, most recently from a013c35 to 0beadec Compare October 8, 2025 18:46
tianyu-l
tianyu-l requested changes Oct 8, 2025
View reviewed changes
@tushar00jain tushar00jain force-pushed the pr1810 branch 2 times, most recently from 22239d9 to 9333989 Compare October 8, 2025 19:09
tushar00jain added a commit that referenced this pull request Oct 8, 2025
@tushar00jain
enhance profiler config (#1809)
21739fd 
Summary:
allow users to specify the profiler schedule

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1809).
* #1811
* #1810
* #1812
* __->__ #1809

Co-authored-by: Tushar Jain <[email protected]>
@tushar00jain tushar00jain requested a review from tianyu-l October 8, 2025 20:00
@tushar00jain tushar00jain mentioned this pull request Oct 8, 2025
Closed
@tushar00jain tushar00jain requested a review from fegin October 8, 2025 20:06
tushar00jain added a commit that referenced this pull request Oct 10, 2025
@tushar00jain
add script to train with ft (#1812)
42fe05f 
Summary:
the script adds configuration options to run training locally with ft
enabled

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1812).
* #1840
* #1811
* #1810
* __->__ #1812
* #1809

---------

Co-authored-by: Tushar Jain <[email protected]>
fegin
fegin approved these changes Oct 10, 2025
View reviewed changes
Copy link
Contributor

@fegin fegin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to move TorchFT logic out of checkpointer in the future. A better design is that TorchFT has its own train.py which customizes Trainer to use two checkpointers, one for the regular checkpoint and another one for dataloader.

@tushar00jain
allow disabling ft checkpoints
bcbc707 
Summary:
Allows disabling the storage of checkpoints related to torchft.

Users don't really have to rely on any external storage. So it reduces set up time to get things up and running. Since we also don't really need model checkpoints when we have torchft. And if checkpoint storage has issues, this can work as a killswitch to completely disable the storage so it doesn't impact training.
@tushar00jain tushar00jain mentioned this pull request Oct 11, 2025
Draft
@tianyu-l tianyu-l merged commit a82b77a into pytorch:main Oct 12, 2025
12 of 13 checks passed
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 13, 2025
@tushar00jain @githubsgi
enhance profiler config (pytorch#1809)
e7f4294 
Summary:
allow users to specify the profiler schedule

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1809).
* pytorch#1811
* pytorch#1810
* pytorch#1812
* __->__ pytorch#1809

Co-authored-by: Tushar Jain <[email protected]>
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 13, 2025
@tushar00jain @githubsgi
add script to train with ft (pytorch#1812)
f63037f 
Summary:
the script adds configuration options to run training locally with ft
enabled

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1812).
* pytorch#1840
* pytorch#1811
* pytorch#1810
* __->__ pytorch#1812
* pytorch#1809

---------

Co-authored-by: Tushar Jain <[email protected]>
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 13, 2025
@tushar00jain @githubsgi
allow disabling ft checkpoints (pytorch#1810)
358622f 
Summary:
Allows disabling the storage of checkpoints related to torchft.

Users don't really have to rely on any external storage. So it reduces
set up time to get things up and running. Since we also don't really
need model checkpoints when we have torchft. And if checkpoint storage
has issues, this can work as a killswitch to completely disable the
storage so it doesn't impact training.

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1810).
* pytorch#1856
* pytorch#1811
* __->__ pytorch#1810

Co-authored-by: Tushar Jain <[email protected]>
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 15, 2025
@tushar00jain @githubsgi
add script to train with ft (pytorch#1812)
79c7690 
Summary:
the script adds configuration options to run training locally with ft
enabled

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1812).
* pytorch#1840
* pytorch#1811
* pytorch#1810
* __->__ pytorch#1812
* pytorch#1809

---------

Co-authored-by: Tushar Jain <[email protected]>
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 16, 2025
@tushar00jain @githubsgi
add script to train with ft (pytorch#1812)
087dc88 
Summary:
the script adds configuration options to run training locally with ft
enabled

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1812).
* pytorch#1840
* pytorch#1811
* pytorch#1810
* __->__ pytorch#1812
* pytorch#1809

---------

Co-authored-by: Tushar Jain <[email protected]>
githubsgi pushed a commit to githubsgi/torchtitan that referenced this pull request Oct 16, 2025
@tushar00jain @githubsgi
add script to train with ft (pytorch#1812)
ff832d2 
Summary:
the script adds configuration options to run training locally with ft
enabled

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/pytorch/torchtitan/pull/1812).
* pytorch#1840
* pytorch#1811
* pytorch#1810
* __->__ pytorch#1812
* pytorch#1809

---------

Co-authored-by: Tushar Jain <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fegin fegin fegin approved these changes

@tianyu-l tianyu-l tianyu-l approved these changes

@wwwjn wwwjn Awaiting requested review from wwwjn wwwjn is a code owner

@wconstab wconstab Awaiting requested review from wconstab wconstab is a code owner

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tushar00jain @fegin @tianyu-l