feat: add warmup support to cosine scheduler#896
Open
Packsod wants to merge 990 commits into
Open
Conversation
…g gradient checkpointing
* Initial support for ltx 2.3. Still needs a lot of testing to make sure it is all right. * bump version * Handle lora renaming keys for new ltx 2.3 layers
…o convert surround sound audio to stereo. Invalidate old caches just to be safe so they recache now.
* Fix Qwen Image mask handling * Fix Qwen attention mask crash with diffusers >=0.37 diffusers v0.37 (PR #12987) optimizes all-ones attention masks to None in encode_prompt() when there is no padding. This breaks ai-toolkit's Qwen extensions which call .to() on the mask unconditionally. Fix: reconstruct the all-ones mask at the boundary (get_prompt_embeds) right after encode_prompt() returns. This keeps the rest of the code unchanged and works with both old and new diffusers versions. Also removes redundant duplicate mask assignments in qwen_image_edit and qwen_image_edit_plus. Fixes ostris#740
Apply random_noise_multiplier to noise.
… videos. Varous ltx 2.3 VAE optimizations such as removing tiling articacts, and doing frame split encoding to reduce vram on encoding/decoding.
…l for DFE and other methods that target pixelspace losses.
…ted format notes better.
…ely resemble the full pipeline results. I pushed a finetuned unconditional lora to the hub as an adapter.
Read num_warmup_steps from lr_scheduler_params to enable custom warmup length. Fallback to standard CosineAnnealingLR if no warmup steps are specified.
- extends the `cosine` learning rate scheduler to support a configurable cosine decay window via `cosine_decay_steps`. - If `cosine_decay_steps` is not provided, it defaults to `steps` to preserve current behavior. - If `cosine_decay_steps` is smaller than the total training steps, the scheduler finishes cosine annealing early and keeps the learning rate at `eta_min` until training ends. - If `cosine_decay_steps` exceeds `steps`, it is clamped to `steps` with a warning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds warmup support to the
cosinelearning rate scheduler.Previously, configuring a cosine schedule with warmup steps wasn't fully supported or would skip warmup if parameters were misplaced. By properly checking
num_warmup_stepswithinlr_scheduler_params, users can now smoothly scale up their learning rate before entering the cosine decay phase.Changes
get_lr_schedulerinscheduler.pyto handlenum_warmup_stepsfor thecosinename branch.LambdaLRlogic whenwarmup_steps > 0to combine linear warmup with cosine annealing, reusing the automatically passedtotal_iters/T_max.CosineAnnealingLRseamlessly if no warmup steps are provided.Usage Example