Integration test coverage for PR validation

[miroslavpojer]

Feature Description

Integrate automated integration tests into the project CI pipeline to ensure that pull request (PR) changes do not break or regress existing project features.

Problem / Opportunity

Currently, changes delivered via PR could unintentionally break critical workflows, since there is no enforced integration validation of project functionality. This lack of automated coverage increases risk for production incidents and slows down the release process due to manual checking.

Specification

Testing is divided into two main parts:

• Mocked integration tests (broad, CI-friendly): Run the tool's real entrypoint with all GitHub API calls stubbed using libraries such as responses or respx. These tests use canned JSON payloads representing issues, labels, and pagination, and cover a wide range of project scenarios.

  • Validate grouping, sorting, error handling, and output formatting based on fixtures.
  • Assert error behaviors (e.g., 401, 403, 429/5xx simulated errors).
  • Run on all PRs, including forks (safe, deterministic, and secret-free).

• Real API smoke E2E (minimal, trusted context only):

  • One job in CI runs the project against a real repository using secrets. Verifies that the project can fetch real issues and generate non-empty release notes.
  • Must run only on push to main and same-repo PRs, never on forks.
  • Asserts exit code is 0 and output is present (optionally, check known section in output).

Acceptance Criteria

1. Integration tests are triggered automatically on every PR.
2. Mocked tests cover: happy path, no issues, pagination, error handling, determinism.
3. Real API test checks that project still works end-to-end on a real repo with secrets.
4. Documentation is updated in DEVELOPER.md (this repo uses DEVELOPER.md) next to the unit tests section, describing how to run both kinds of tests locally and in CI.

Proposed Solution

Implement integration test scripts using Python/Shell to cover end-to-end operations of the GitHub Action. Update the workflow configuration so PRs are validated by these tests. Add documentation for maintainers in DEVELOPER.md next to the unit tests section.

Task List (with Acceptance Criteria and File References)

• Discover current test structure

  • References:
    • CI workflow for PR checks: .github/workflows/test.yml
    • Existing integration tests: tests/integration/integration_test.py, tests/integration/test_release_notes_snapshot.py
    • Action entry definition: action.yml (composite action)
    • Developer docs file present: DEVELOPER.md
  • Acceptance criteria:
    • Project entrypoint(s) for the action/CLI are identified (starting from action.yml) and documented.
    • Existing integration tests are inventoried and classified (real API vs mock/snapshot).
    • Target structure for new mocked integration tests is agreed.

• Remove/replace existing integration tests

  • References:
    • tests/integration/integration_test.py (real API smoke requiring GITHUB_TOKEN)
    • tests/integration/test_release_notes_snapshot.py (snapshot-like behavior)
  • Acceptance criteria:
    • Existing integration tests are reviewed for overlap with the new design.
    • Any real-API-calling test is moved behind the Smoke E2E job or refactored to mocked API.
    • Project ends with: mocked integration suite + one real API smoke E2E (trusted context only).

• Implement mocked integration tests

  • References:
    • Existing integration test directory: tests/integration/
    • Suggested new/refactored test file: tests/integration/test_generate_release_notes_mocked.py
  • Acceptance criteria:
    • Mocking reliably stubs all GitHub API requests used by the tool.
    • Fixtures cover: happy path, empty, pagination, auth failure, rate-limit/5xx.
    • Output assertions are deterministic (golden files or strict expectations).
    • Tests run on every PR without secrets.

• Add mocked integration tests job to CI

  • References:
    • .github/workflows/test.yml
  • Acceptance criteria:
    • A new job is added into the same .github/workflows/test.yml that runs pytest ... tests/integration (or equivalent).
    • Job runs on PRs (including forks) and fails the workflow on failure.
    • Job reuses the standard Python setup/install approach from existing jobs.

• Add real API smoke E2E job to CI (automatic, in same workflow file)

  • References:
    • .github/workflows/test.yml
  • Acceptance criteria:
    • Smoke E2E is implemented as an additional job inside .github/workflows/test.yml (not a new workflow file).
    • Job runs automatically on:
      • push to main
      • PRs from same repo only (not forks)
    • Uses GITHUB_TOKEN securely and performs minimal stable assertions (exit code 0 + non-empty output).

• Update documentation

  • References:
    • DEVELOPER.md
  • Acceptance criteria:
    • DEVELOPER.md includes how to run mocked integration tests locally and how Smoke E2E is executed automatically in CI.

Dependencies / Related

No response

Additional Context

No response

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• Add manual integration test into doc #180

Related Issues

• Fix Aquasec Issues - commit sha vs tags #213
• .github File Upgrade #210
• Dependency Dashboard #231

🔗 Related PRs

#144 - Feature/141 add support for coderabbit summary in pr description [merged]
#192 - Feature/182 review of unit tests [merged]
#200 - Doc/spec kit and project sync [merged]
#207 - Feature/206 fix issues detected by aquasec scanner [merged]
#214 - Fix Aquasec Issues - commit sha vs tags [merged]

👤 Suggested Assignees

• miroslavpojer
• tmikula-dev

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[miroslavpojer]

Implemented by this PR.