[Tests]: Adding dummy causal models for testing in regular CI run #427
+685
−292
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Abukhoyer Shaik <[email protected]>
abukhoy
requested review from
quic-rishinr,
ochougul,
quic-hemagnih and
quic-amitraj
as code owners
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
quic-rishinr
requested changes
Jul 15, 2025
| "hpcai-tech/grok-1", | ||
| ] | ||
|
|
||
| test_dummy_model_configs = [ |
There was a problem hiding this comment.
Can we move this outside this file? may be we can maintain a CSV file for better readability.
| "hpcai-tech/grok-1", | ||
| ] | ||
|
|
||
| test_dummy_model_configs = [ | ||
| # model_name, model_type, max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size, additional_params | ||
| ("TinyLlama/TinyLlama-1.1B-Chat-v1.0", "llama", 128, 1, 2, 64, 256, 32000, {"num_key_value_heads": 1}), |
There was a problem hiding this comment.
are we following any criteria for selecting these configs?
|
|
||
| if model_hf is None: | ||
| model_hf, _ = load_causal_lm_model(model_config) | ||
| model_hf_cb = copy.deepcopy(model_hf) |
| @pytest.mark.cli | ||
| @pytest.mark.parametrize("config", configs) | ||
| def test_export_compile_execute_qnn_fb(mocker, config): | ||
| # testing export -> compile -> infer with full_batch_size in QNN enviroment |
| @pytest.mark.qnn | ||
| @pytest.mark.cli | ||
| @pytest.mark.parametrize("config", configs) | ||
| def test_export_compile_execute_qnn(mocker, config): |
There was a problem hiding this comment.
- Both test_export_compile_execute_qnn and test_export_compile_execute_qnn_fb is currently having same configs right? Ideally in test_export_compile_execute_qnn we should be providing BS and in test_export_compile_execute_qnn_fb we should be providing FBS.
- Rename test_export_compile_execute_qnn_fb -> test_export_compile_execute_qnn_fbs for better readability
- Typo in 'enviroment'
tests/cloud/test_infer.py
Outdated
| ) | ||
| check_infer(mocker=mocker, generation_len=20, **local_config) |
There was a problem hiding this comment.
Can we have a vlm qnn test as well?
| mxfp6=ms.mxfp6, | ||
| mxint8=ms.mxint8, | ||
| full_batch_size=ms.full_batch_size, | ||
| enable_qnn=ms.enable_qnn, | ||
| image_url=kwargs["image_url"], | ||
| ) |
There was a problem hiding this comment.
how can we make sure the infer is running as expected? Please include proper asset for checking, export, compile and generation is running proper.
| # testing for CB models | ||
| model_hf, _ = load_causal_lm_model(model_config) | ||
| model_hf = model_hf_cb | ||
| model_hf.eval() |
There was a problem hiding this comment.
do we need model_hf.eval()?
| ``Mandatory`` Args: | ||
| :model_name (str): Hugging Face Model Card name, Example: ``gpt2`` | ||
| """ | ||
| if test_dummy_model_name in { |
There was a problem hiding this comment.
We should avoid putting such constants. May be have a separate test for quantized models
| @@ -292,6 +515,35 @@ def test_causal_lm_pytorch_vs_kv_vs_ort_vs_ai100_qnn(model_name): | |||
|
|
|||
|
|
|||
| @pytest.mark.skip() # remove when the SDK 1.20.0 issue solved for compiling this model | |||
There was a problem hiding this comment.
Can we remove it now? same on line 545 as well
Signed-off-by: Abukhoyer Shaik <[email protected]>
Signed-off-by: Abukhoyer Shaik <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of this PR:
This update aims to reduce test execution time for causal language model inference. Previously, tests were run using full-scale models with one or two layers, which was inefficient and time-consuming. Refactoring CLI api testing for independent testing and redundant conftest code.
What’s Changed:
Introduced dummy models with significantly smaller configurations by adjusting parameters such as
max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size and additional_params.These lightweight models are used exclusively for testing purposes to ensure faster execution without compromising test coverage.
And CLI testing has two test scripts one is for export, compile, and execute, another is for infer cli api.
Note: This optimization is applied only to causal language models.