Save and output number of samples of each task #851
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR closes #804 .
What does this PR do?
This PR adds the
num_samples
field to both the results_dict that is saved as json, but also the final_dict that is passed tomake_results_table()
as requested in the issue. All existing elements in these dicts are left unchanged.The keys in

num_samples
are the exact same as the keys inresults
(meaning we calculate the number of samples for each individual task, as well as all grouped tasks by summing their subtasks, and the "all" task), allowing us to add the number of samples to the markdown table created inmake_results_table()
like so:To guarantee backwards compatibility in
make_results_table()
, the "Number of Samples" fields will just be empty in the case that the result_dict does not containnum_samples
.The samples are counted via the length of each entry in details_logger.details.
Changes
calculate_num_samples()
method in EvaluationTrackernum_samples
field to results_dict in EvaluationTracker.save()num_samples
field to final_dict in EvaluationTracker.generate_final_dict()make_results_table()
results.json
in docs to include the new entryTests
All tests passed locally.