[QEff. Finetune]: Enabled FT CI tests. #420

quic-meetkuma · 2025-05-21T05:35:20Z

Enabled CI tests for Finetuning.
Updated Jenkins file to install torch_qaic as it is required during FT tests.
Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag.

ochougul · 2025-06-24T16:11:53Z

scripts/Jenkinsfile

@@ -25,6 +25,7 @@ pipeline {
                   pip install junitparser pytest-xdist &&
                   pip install librosa==0.10.2 soundfile==0.13.1 && #packages needed to load example for whisper testing
                   pip install --extra-index-url https://download.pytorch.org/whl/cpu timm==1.0.14 torchvision==0.19.1+cpu einops==0.8.1 && #packages to load VLMs
+                   pip install /opt/qti-aic/integrations/torch_qaic/py310/torch_qaic-0.1.0-cp310-cp310-linux_x86_64.whl && # For finetuning tests


Will this replace the torch that is getting installed on line 27?
If yes, we should install this package only at the finetuning tests stage.

No, it won't update existing pytorch installation. The torch_qaic is the add-on for qaic backend.

Did e2e CI run pass post this change?

But most of qeff users won't have this package installed.
We should reflect the exact env that is going to be used by customers.
I understand it should not affect anything, but why should we deviate from golden standard?

Did e2e CI run pass post this change?
Yes, it ran.

But most of qeff users won't have this package installed. We should reflect the exact env that is going to be used by customers. I understand it should not affect anything, but why should we deviate from golden standard?

Thanks for pointing it out. We will have subsequent PRs for FT testcases. We will try to take care this in it by having a new stage before FT test cases.

Signed-off-by: meetkuma <[email protected]>

…on of tests for FT. Removed samsum test case from FT. Signed-off-by: meetkuma <[email protected]>

Signed-off-by: meetkuma <[email protected]>

quic-hemagnih · 2025-07-01T06:31:11Z

Are we good to merge this code? Anything pending on this?

Please reply to @ochougul comments

vbaddi · 2025-07-01T06:54:13Z

QEfficient/finetune/dataset/samsum_dataset.py

@@ -9,7 +9,7 @@


 def get_preprocessed_samsum(dataset_config, tokenizer, split, context_length=None):
-    dataset = datasets.load_dataset("Samsung/samsum", split=split, trust_remote_code=True)
+    dataset = datasets.load_dataset("knkarthick/samsum", split=split, trust_remote_code=True)


Is this approved dataset?

Will be removed in #482

vbaddi · 2025-07-01T06:54:28Z

scripts/Jenkinsfile

@@ -25,6 +25,7 @@ pipeline {
                   pip install junitparser pytest-xdist &&
                   pip install librosa==0.10.2 soundfile==0.13.1 && #packages needed to load example for whisper testing
                   pip install --extra-index-url https://download.pytorch.org/whl/cpu timm==1.0.14 torchvision==0.19.1+cpu einops==0.8.1 && #packages to load VLMs
+                   pip install /opt/qti-aic/integrations/torch_qaic/py310/torch_qaic-0.1.0-cp310-cp310-linux_x86_64.whl && # For finetuning tests


Did e2e CI run pass post this change?

quic-hemagnih · 2025-07-01T08:26:04Z

Hi Meet - Lets do following -

Create a separate enviornment variable for Finetunning tests.
Activate the env variable and run all the FT UTs and then once done
Gracefully exit the env
This way we will be able to overcome all the version mismatches or any custom env changes required for FT in future also.

Thanks
Hem

quic-meetkuma · 2025-07-01T08:28:27Z

Thanks, Hem.
This will be done in subsequent PR.

- Enabled CI tests for Finetuning. - Updated Jenkins file to install torch_qaic as it is required during FT tests. - Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag. --------- Signed-off-by: meetkuma <[email protected]> Co-authored-by: Meet Patel <[email protected]>

- Enabled CI tests for Finetuning. - Updated Jenkins file to install torch_qaic as it is required during FT tests. - Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag. --------- Signed-off-by: meetkuma <[email protected]> Co-authored-by: Meet Patel <[email protected]> Signed-off-by: Amit Raj <[email protected]>

- Enabled CI tests for Finetuning. - Updated Jenkins file to install torch_qaic as it is required during FT tests. - Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag. --------- Signed-off-by: meetkuma <[email protected]> Co-authored-by: Meet Patel <[email protected]>

- Enabled CI tests for Finetuning. - Updated Jenkins file to install torch_qaic as it is required during FT tests. - Added finetune as a new pytest flag and updated other existing tests not to trigger for this flag. --------- Signed-off-by: meetkuma <[email protected]> Co-authored-by: Meet Patel <[email protected]> Signed-off-by: Dhiraj Kumar Sah <[email protected]>

quic-meetkuma requested review from quic-rishinr, ochougul, quic-hemagnih and quic-amitraj as code owners May 21, 2025 05:35

quic-meetkuma requested review from vbaddi, quic-swatia and quic-mamta May 21, 2025 05:35

quic-meetkuma force-pushed the ci_enable branch from 742e8f1 to e67c70a Compare May 21, 2025 05:39

quic-meetkuma force-pushed the ci_enable branch 5 times, most recently from c615ad9 to 80f8cd3 Compare June 24, 2025 13:14

ochougul requested changes Jun 24, 2025

View reviewed changes

quic-meetkuma changed the title ~~Enabled FT CI tests.~~ [QEff. Finetune]: Enabled FT CI tests. Jun 27, 2025

quic-meetkuma force-pushed the ci_enable branch from 4804574 to 2554792 Compare June 27, 2025 05:41

quic-meetkuma added 9 commits July 1, 2025 10:45

Enabled FT CI tests.

d6007d4

Signed-off-by: meetkuma <[email protected]>

Expanded test cases to cover samsum, gsm8k, alpaca and imdb dataset

e699541

Signed-off-by: meetkuma <[email protected]>

Fixed ruff formatting.

327ac13

Signed-off-by: meetkuma <[email protected]>

Fixed samsum dataset.

cd6d580

Signed-off-by: meetkuma <[email protected]>

Updated test id names.

d385f48

Signed-off-by: meetkuma <[email protected]>

Disabled other tests temp. PR will be merged after enabling these tests.

b15257f

Signed-off-by: meetkuma <[email protected]>

Fixed failing tests.

94ae920

Signed-off-by: meetkuma <[email protected]>

Disabled other tests in Jenkins temporarily. Removed parallel executi…

66fb91f

…on of tests for FT. Removed samsum test case from FT. Signed-off-by: meetkuma <[email protected]>

Fixed Jenkins stages.

c39238f

Signed-off-by: meetkuma <[email protected]>

quic-meetkuma force-pushed the ci_enable branch from d4da3a9 to c39238f Compare July 1, 2025 05:15

quic-hemagnih approved these changes Jul 1, 2025

View reviewed changes

quic-hemagnih merged commit 6c64d35 into quic:main Jul 1, 2025
4 checks passed

vbaddi reviewed Jul 1, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[QEff. Finetune]: Enabled FT CI tests. #420

[QEff. Finetune]: Enabled FT CI tests. #420

Uh oh!

quic-meetkuma commented May 21, 2025

Uh oh!

ochougul Jun 24, 2025

Uh oh!

quic-meetkuma Jul 1, 2025

Uh oh!

vbaddi Jul 1, 2025

Uh oh!

ochougul Jul 1, 2025

Uh oh!

quic-meetkuma Jul 1, 2025

Uh oh!

quic-meetkuma Jul 1, 2025

Uh oh!

quic-hemagnih commented Jul 1, 2025

Uh oh!

Uh oh!

vbaddi Jul 1, 2025

Uh oh!

quic-meetkuma Jul 1, 2025

Uh oh!

vbaddi Jul 1, 2025

Uh oh!

quic-hemagnih commented Jul 1, 2025

Uh oh!

quic-meetkuma commented Jul 1, 2025

Uh oh!

Uh oh!

[QEff. Finetune]: Enabled FT CI tests. #420

[QEff. Finetune]: Enabled FT CI tests. #420

Uh oh!

Conversation

quic-meetkuma commented May 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-hemagnih commented Jul 1, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quic-hemagnih commented Jul 1, 2025

Uh oh!

quic-meetkuma commented Jul 1, 2025

Uh oh!

Uh oh!