-
-
Notifications
You must be signed in to change notification settings - Fork 18
Open
Labels
bugSomething isn't workingSomething isn't working
Description
If a user has used print()
or other debugging methods in their code and the tests for that code are using unittest.subtest
, the captured output for the subtests is being written into the parent output
property in the results.json
file -- one set of of output for each subtest. Not an ideal situation.
Steps to reproduce:
Using the tests/example-partial-failure-with-subtests/
test:
- Add
print("User output is captured!")
on line 8 ofexample_partial_failure_with_subtests.py
- Run ./bin/run.sh example-partial-failure-with-subtests
- Note that
results.json
has an entry that looks like this forExampleSuccessTest.test_abc
:
"name": "ExampleSuccessTest.test_abc",
"status": "pass",
"test_code": "input_data = ['frog', 'fish', 'coconut', 'pineapple', 'carrot', 'cucumber', 'grass', 'tree']\nresult_data = [(\"Hello, World!\", param) for param in input_data]\nnumber_of_variants = range(1, len(input_data) + 1)\n\nfor variant, param, result in zip(number_of_variants, input_data, result_data):\n with self.subTest(f\"variation #{variant}\", param=param, result=result):\n self.assertEqual(hello(param), result,",
"task_id": 1,
"output": "User output is captured!\nUser output is captured!\nUser output is captured!\nUser output is captured!\nUser output is captured!\nUser output is captured!\nUser output is captured!\nUser output is captured!"
}
Ideally, we'd write captured output for a subtest into the output
property for that subtest. In cases where there is not a subtest report (pytest only reports failed subtests), we would annotate the parent test output
to indicate it came from a successful subtest.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working