python API for jobsub_lite

[vhewes]

i'm working on overhauling NOvA's grid infrastructure, and i have a feature request for jobsub_lite that would significantly streamline our job submission infrastructure – namely, the ability to invoke it from python directly.

from reading the jobsub_lite code, my understanding is that the lib directory defines an importable python package, in which the guts of the machinery are located, and then the command line interface is provided via jobsub_submit in the bin directory. according to its docstring, this script is responsible for the following:

• parse args
• get credentials
• handle tarfile options
• set added values from environment, etc.
• convert/render template files to submission files
• launch

NOvA users typically interface with jobsub_lite via python scripts, which provide a layer of abstraction to assemble the raw jobsub_submit command for a few common workflows (mostly Art and CAFAna jobs), and then execute it via a call to subprocess. so essentially, we run python scripts that assemble bash commands that execute a python script.

my request is this: would it be possible to take all of the functionality of the main function in jobsub_submit that comes after argument parsing and migrate it into lib, so it can be imported and invoked from python scripts, taking arguments as kwargs? then the jobsub_submit executable itself would simply parse the command line args and then pass them as kwargs into this submission function. that way, our python submission scripts can import and run this submission function directly, without needing to invoke the subprocess package.

the ability to run jobsub submission and query commands from python natively would be a game-changer. i'm happy to implement these changes and submit them as a PR for review, if this idea is something you'd be open to. thank you!

[marcmengel]

This sounds like a good topic for our next jobsub meeting on the 11th. It is an excellent idea, however there are a few concerns:

• jobsub currently lives in the OS system python3 environment; most of the experiments have their own python environment which doesn't include system RPMs like:
  • condor python bindings
  • distributed tracing
  • jinja2 templating

However, if the experiment is willing to perform the PYTHONPATH gymnastics needed to access those system python packages, this could work.

[vhewes]

is this true in the AL9 era, given that fermi spack pulls in the system python as an external package? if we always use the system python, this may not even be an issue. i'll do some local tests to see whether we'd need to worry about this – but yes, in general i think we'd be happy to figure out any gymnastics

i'm not sure whether the jobsub meetings are internal, but for what it's worth, i'm happy to attend and discuss further if that would be useful. thanks for the quick response!

[marcmengel]

In particular much of the work for this was discussed before, in #484, the idea of making the various jobsub commands a single jobsub command with verb arguments.. (i.e. jobsub submit ... rather than jobsub_submit) which also involved pulling the mainlines out into a side file.

[marcmengel]

is this true in the AL9 era, given that fermi spack pulls in the system python as an external package? if we always use the system python, this may not even be an issue.

While Spack can pull in the system python, I believe most experiment stacks currently build their own; if only because they used to do so on SL7, where the system python was ancient. Of course, one can also build a python script to use the system python even in the experiment environment, as we do with the jobsub scripts themselves.

[shreyb]

Hi @vhewes - thanks for your feedback! As Marc mentioned, something along these lines has always been on the roadmap, but has taken a back seat due to available effort. I talked to line management about this as well, and I don't know how much effort from CSAID will be forthcoming in the next couple of months, but I expect not that much.

That being said, I think, since we would eventually like to do this, I think it's worth discussing at the next meeting. It was originally supposed to be today, but I'll be rescheduling it for next week - the 18th. I'm happy to send you an invitation for it, and if you're available, please do join us (they're generally internal, but not strictly so; we have invited stakeholders like yourself as-needed). If you're not available, or if we need more time to discuss, I can schedule an additional meeting for that. I think it'll be useful to at least set out a structure of what we want to do, and discuss the technical issues like you and Marc have here, so maybe we can break the larger rewrite into smaller tasks that can be handled as you or we have time to knock them out.

[shreyb]

It was noted in the jobsub meeting today that #484 is heavily related to this effort.

[shreyb]

Alright - I've started making this into a github project, and I'll start breaking out the work items.