feat: NemotronH PP/EP SFT integration + fixed-length SQuAD supervision fix

[jasont314]

What does this PR do ?

Enable stable NemotronH PP+EP SFT with fixed-length SQuAD by integrating the EP runtime path, hardening PP/FSDP synchronization/schedule behavior, and fixing label-token supervision masking so training no longer degenerates to NaN/zero-label-token steps.

Changelog

• Add NemotronH EP runtime integration and safety guards in distributed parallelization paths.
• Add EP shard mesh null-guard behavior and mesh-aware EP axis propagation in training setup.
• Add PP schedule robustness updates:
  • explicit invalid-style error handling,
  • guarded NEMOAUTOMODEL_PP_SKIP_OUTPUT_MERGE patching with compatibility checks,
  • safer underfill handling interactions in tests.
• Normalize AutoPipeline device handling (torch.device | int | str -> torch.device) and update call sites.
• Fix FSDP2 divisibility error text to match actual tp * cp * pp logic.
• Make warning filtering opt-in via env (NEMOAUTOMODEL_FILTER_WARNINGS) for better debugging visibility.
• Fix fixed-length SQuAD supervision path:
  • truncation-aware prompt/answer masking,
  • fixed-length truncation behavior that preserves answer supervision tokens,
  • avoid all-masked label cases (num_label_tokens == 0).
• Add optimized SQuAD PP+EP config:
  • examples/llm_finetune/nemotron/nemotron_nano_v3_pp_ep_squad.yaml
• Add patch documentation:
  • NEMOTRON_PP_EP_SQUAD_PATCH_NOTES.md
• Add/update unit tests for:
  • PP schedule/style/skip-merge behavior,
  • AutoPipeline device normalization,
  • EP shard-axis + missing moe_mesh safety.
• Include SFT training artifacts for baseline vs optimized throughput comparison:
  • checkpoints/baseline_training.jsonl
  • checkpoints/optimized_training.jsonl

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

• Related to: N/A
• Validation run:
  • python -m pytest -q tests/unit_tests/distributed/pipelining/test_functional.py tests/unit_tests/distributed/pipelining/test_autopipeline.py tests/unit_tests/moe/test_parallelizer.py
  • Result: 79 passed, 5 skipped, 6 warnings

Acknowledgements

Thanks to collaborators:

• Nazar Ospanov
• Zoir Imomaliev
• Sanjay Adhikesaven

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.