Applying priority for the whole analyses and treating it as a singular task

[aiste-kalinauskaite]

Let's assume we are running just one model for simplicity (as I know it will get more complicated, if I add another model). Let's say I have 2 workers on the environment and can only run two analyses simultaneously. I know with true horizontal scaling, it's not going to be quite the same, but hopefully, this illustrates the purpose.
This is the order in which the user submits analyses:

• Analysis 1, priority (P) = 2
• Analysis 2, P = 6
• Analysis 3, P = 4
• Analysis 4, P = 6
• Analysis 5, P = 3
• Analysis 6, P = 4

Let's assume there was nothing running on in the queue, when those analyses were sent to the queue.
As there was nothing running, I would expect Analysis 1 to start first and then Analysis 2 to follow. I expect Analysis 1 to complete entirely (all the way to finished) before a new analysis is picked up (even though there are higher priority analyses still waiting in the queue):

• Once Analysis 1 is complete, resources are released and Analysis 4 starts (as it's P = 6)
• Once Analysis 2 is complete, then Analysis 3 starts (the next highest available priority and an earlier submission time)
• Then whichever completes next, those resources be released to Analysis 6
• And afterwards to Analysis 5

Of course, in reality, based on the portfolio size, say Analysis 2 could have completed before Analysis 1 is - so that would be ok. The key part is to ensure that Analysis 1 does not get paused after input gen (or at some other stage of a sub-task), because there is Analysis 3, 4, 5, and 6 waiting in the queue and all having P > 2 (which is the priority of Analysis 1.

If we add a bit of complexity with the true horizontal scaling in place, allowing for the resources to be released after a sub-task is complete and as there are sub-tasks that only use 1 core (so technically the rest of the cores can be released), then they can be released, BUT they should be used on the analyses that are running instead of starting a new one (unless there is no running analyses). AND that when the resource is released back, the original analyses gets to complete the rest of its sub-tasks before the new one starts. I.e. it's the same principal, as I expressed before: to ensure that Analysis 1 does not get paused after input gen (or at some other stage of a sub-task), because there is Analysis 3, 4, 5, and 6 waiting in the queue and all having P > 2 (which is the priority of Analysis 1.

Ideas for solutions: assigning max priority for the remaining subtasks, once the first task is started.