Allow secret test cases to use .statement and .download #516
Conversation
| The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). |
There was a problem hiding this comment.
I disagree. full_feedback test cases differ from sample in that are not given apriori, only if you fail on them.
| The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). |
There was a problem hiding this comment.
Ah, I think I meant to describe that the test case with full feedback should be available for download, only when the feedback is given.
The test case would've already be shown in that case, and allowing it to be downloaded would just make it easier.
Do you agree with this?
There was a problem hiding this comment.
I agree that it is a very good idea, I think we should be careful with requiring specific methods for delivering data. E.g. one could imagine a contest system that makes the files available on disk. That would technically not be "available for download".
There was a problem hiding this comment.
| The test case with full feedback should also be available for download; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). | |
| For a submission that fails a test case with `full_feedback` set to `true`, the relevant feedback files for that test case should be available for download for that submission. These files are not made available otherwise; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). |
| Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`. | ||
| ### Test Cases with Full Feedback |
There was a problem hiding this comment.
This section seems to mostly talk about sample. Specifically the sections on downloads which doesn't really say that all full_feedback cases should be downloadable (and I don't think they should), but the text makes this unclear.
There was a problem hiding this comment.
Yes that was unclear, I have changed some things (and some mistakes where it mentioned sample).
Although, I think it should be the following: The test cases with full_feedback should be downloadable for a specific submission, if the submission failed on that test case.
I agree that the test cases secret should not be available for download otherwise. (Is that what you meant?)
Co-authored-by: Fredrik Niemelä <[email protected]>
Co-authored-by: Fredrik Niemelä <[email protected]>
| Note that it is an error to provide both a `.out` and a `.ans.statement` file. | ||
| By default, the `.in`, `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. |
There was a problem hiding this comment.
| By default, the `.in`, `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. | |
| By default, the `.in`, `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. Test cases with full_feedback are only available for download if the submission failed on that specific test case. These files are not made available otherwise. |
Fixes #508 and fixes #515.
I have basically copied from the sections "Samples Shown in the Problem Statement", and "Samples Available for Download" and rewritten them
I am unsure if this is the best way to do it. One other way is to put the section I added together with the sections "Samples Shown in the Problem Statement", and "Samples Available for Download". However, I think I prefer what I have done.