Specifying runtime limit (and memory limit as well?) inside kata creator

[Voileexperiments]

So I was wondering, since the runtime limit for running a solution is hardcoded in the runner config, is it possible to expose this parameter in the kata creator so that kata authors can customize the limit for each available language?

This would make performance tests much easier to implement, much easier to balance the relative speed between different languages, and allows some katas that naturally require longer runtime than usual (e.g this and this kata) to be implemented easily.

Also, that makes CW one step closer to an OJ site :P

┆Issue is synchronized with this Clickup by Unito

[kazk]

I'm moving this to Codewars issue tracker because the new runner supports this (time limit) already and it's been used on Qualified for a while.

[hobovsky]

I have a related question. Would it be possible to exclude the setup time from time limit? For some languages, significant part of available time is consumed by compilation,. For example empty Scala file taka 6 seconds to run, and some C++ solutions take ages to compile.

I know that for some languages it cannot be reduced because time spent for runtime setup (jvm warm-up etc. ) cannot be easily separated from actual execution, but maybe just excluding runner setup and compilation would help for some languages?

[kazk]

@hobovsky No, it's not possible because all code is executed with a single command (sh -c is used if it needs multiple stages) and the time is based on that. This design keeps the runner simple and uniform, so I don't think it'll change.

I know that for some languages it cannot be reduced because time spent for runtime setup (jvm warm-up etc. ) cannot be easily separated from actual execution, but maybe just excluding runner setup and compilation would help for some languages?

The overhead of the runner are not included except for container start up time (a few hundred milliseconds).
Languages that requires a significant amount of time to run anything already have a longer time limit because of it. Some code does take significantly longer to compile than others, but allowing authors to extend the limit when that's expected should be enough.

Also, the time limit is used to limit the amount of resources required to handle the load. So even if we did support it, we'll limit the maximum allowed time by total time. Being able to specify separate limits for compile and execute will be nice, but I don't think we'll support it because of the required changes.

We won't be allowing to extend the time limit significantly either. Maybe allow up to 25s or so, and will require a valid reason and explicit permission from us.

---

I don't think we'll ever support limiting memory. This is because unlike many of the other platforms, we run a lot more than the submitted code and it's not possible to tell the amount of resources used by the solution by itself. We'll need to introduce a separate controlled benchmark step and I don't think that's going to happen.