-
Notifications
You must be signed in to change notification settings - Fork 11
Description
When authors write randomized questions, it is possible that their code for some reason or the other fails to produce good values 5% of the time. For these situations, authors should have the ability to run tests in their code to verify that the values they produced are good. If they aren't good, they should have a way to programmatically tell quadbase to throw out this instance (meaning don't show this instance to the requestor or always show a different instance (one that works) when the problematic instance is requested). The code mechanism should probably be the throwing of some sort of error exception (e.g. LogicError).
E.g. q451v3?seed=4 is requested but the randomized values produced for the question are inconsistent or otherwise bad. The logic tests the values and detects this problem. It then raises a LogicError. Quadbase catches the LogicError and then regenerates the question with a different random seed. Quadbase should note which different random seed is used so that it can always be the seed that is used in the future (because when someone requests q451v3?seed=4 they should always always always get the same question back).
If the second seed tried also raises an error, we could maybe try one more. If that one also raises an error, this question should be marked as defunct and the author should be notified. The author should also be notified the first time the LogicError is raised for a particular seed so that they can come in and see what happened and maybe improve the question in a new version.
This issue spawned by some great comments from @waynepadgett