[Testing / CI/CD] Ability to automate scale testing with a mock server and test different datasets, loadgen, etc. and run it as a part of CI/CD (#274)

[huaxig]

Summary

This introduces an e2e testing using a mock server to enable automated scale testing as part of our CI/CD pipeline.

Key changes include:

• Mock Client: Mock client is used to simulate the behavior of a real API client. It has a configurable latency to simulate different response times.
• E2e Test case: A test case - basic_mock_client_benchmark is added. This test runs a benchmark with the mock client, generates reports, and asserts key metrics like achieved request rate and success count.
• E2e Test Utility: A run_benchmark_minimal function is created to simplify the process of running benchmarks in tests. It handles configuration, execution, and result parsing.
• CI/CD Integration: updated with a new pdm run test:e2e command to easily run the end-to-end tests in our CI/CD workflow or Dev Env.

Note: there are few more commits to add test to CI workflow

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: huaxig
Once this PR has been reviewed and has the lgtm label, please assign achandrasekar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[huaxig]

#274

[jjk-g]

overall lgtm

Thoughts on creating a top level tests directory and putting the e2e directory under tests?

[jjk-g]

I think this makes more sense in the tests/e2e dir and probably should have a more detailed name. Something like "e2e_simple_mock_client.yaml"

[jjk-g]

sgtm on the e2e top level dir, still think this belongs in e2e dir instead of examples with a more descriptive name

[huaxig]

it makes sense, I will move and rename the file

[huaxig]

Thoughts on creating a top level tests directory and putting the e2e directory under tests?

Thanks for raising this — great point.

In most open-source Python projects, there are two common patterns for organizing E2E tests:

1. A dedicated top-level e2e/ directory — preferred when E2E tests are heavier or depend on external services. this approach keeps runtime environments and CI configurations isolated.

2. A unified tests/ hierarchy — e.g. tests/unit/ and tests/e2e/ — used when all tests share the same tooling and runtime.

Given our current setup — with tests/ already containing unit tests and some E2E tests being relatively heavy — the first approach fits better here.

Keeping E2E tests in a separate top-level e2e/ directory:

• Avoids refactoring existing unit test paths.

• Allows independent pytest configs, fixtures, or environment setup. (e.g., pdm run test for unit test, pdm run test:e2e for e2e test).

CC: @jjk-g