Excessive memory usage when processing history leads to Javascript Heap Exhaustion

[HenrikDK]

Problem, we have a pretty large test suite in python with around 1250+ tests, and when we try to use the github-test-reporter action on main, we get:

Run ctrf-io/github-test-reporter@v1
📜 Preparing CTRF report
⏮️ Processing previous results
  Starting workflow runs processing...
  (node:402) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
  (Use `node --trace-deprecation ...` to show where the warning was created)
  
  <--- Last few GCs --->
  
  [402:0x2ecac000]   100072 ms: Mark-Compact 4089.6 (4126.8) -> 4089.6 (4130.8) MB, pooled: 0 MB, 50.99 / 0.00 ms  (average mu = 0.450, current mu = 0.031) allocation failure; scavenge might not succeed
  [402:0x2ecac000]   100155 ms: Mark-Compact 4090.0 (4131.1) -> 4090.0 (4131.1) MB, pooled: 0 MB, 82.29 / 0.00 ms  (average mu = 0.206, current mu = 0.005) allocation failure; scavenge might not succeed
  
  FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
  ----- Native stack trace -----

Note that this is not a problem on branches and pr's it is exclusively a problem on the main branch.

I have tried limiting the length of the history from the default 10 to 2, but still the problem persists.

Config:

    steps:
      - uses: actions/checkout@v6

      - name: Install pytest
        run: pip3 install pytest pytest-json-ctrf pytest-cov pytest-xdist pytest-mock pyfakefs
        shell: bash

      - name: Test all workflows
        id: run-pytest
        run: | 
          pytest -n auto -m "not serial" --cov=scripts --cov-report=xml --ctrf ctrf-report.json
          rm -f github-summary.env
        shell: bash

      - name: Publish Test Report
        uses: ctrf-io/github-test-reporter@v1
        if: always()
        with:
          report-path: 'ctrf-report.json'
          summary-delta-report: true
          insights-report: true
          flaky-rate-report: true
          upload-artifact: true
          pull-request-report: true
          overwrite-comment: true
          previous-results-max: 2
          comment-tag: PR Test Report
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

When i run the test suite with json output enabled locally the resulting report produced is 60mb large vs 170kb for junit.

I guess this could help explain the data volumes, but it still seems excessive to use 4GB of ram to process 2x60 mb of json?

Relevant logs:

• 6_Test all workflows.txt
• 7_Publish Test Report.txt

[Ma11hewThomas]

I think two separate things are going on here:

• This action is using too much memory when processing large historical reports.
• The input CTRF report itself looks unusually large for the number of tests.

On the second point first: a CTRF report of roughly 60 MB for around 1250 tests does not look normal. In our own testing, CTRF report sizes have generally been on par with JUnit while still carrying much more infromation, so this suggests something specific in the report content is inflating it.

If you’re able to inspect the report contents, it would be useful to look for large fields under the test entries, that should tell us pretty quickly what is driving the size.

The heap exhaustion still looks like a real issue on our side, because the action should handle large historical reports better than it currently does. I’m going to assess that side further and work out the underlying cause.

[HenrikDK]

On the second point first: a CTRF report of roughly 60 MB for around 1250 tests does not look normal. In our own testing, CTRF report sizes have generally been on par with JUnit while still carrying much more information, so this suggests something specific in the report content is inflating it.

Hmmm, seems like it's a problem with the python lib 'pytest-json-ctrf' thats generating the json, we're using a-lot of parameterized test's, and it's including a serialized version of the test function's parameters as a "tag" on the test. In this case that includes a lot of data as we collect all information about the test run into the "test" object that's a param to the calling function.

            {
                "name": "tests/workflow/resources/test_ar_validation.py::test_azure_validation",
                "status": "passed",
                "duration": 1068,
                "start": 1782863280764,
                "stop": 1782863281832,
                "retries": 0,
                "file_path": "tests/workflow/resources/test_ar_validation.py",
                "tags": [
                    "parametrize::test::[" + between 20kb - 300kb more data depending on the test
                ],
                "extra": {
                    "worker": "gw5"
                }
            },

I'll take it up with the maintainer of pytest-json-ctrf