Skip to content

Interchange: launch as a fork/exec not Python multiprocessing #3373

@benclifford

Description

@benclifford

Is your feature request related to a problem? Please describe.
The interchange is launched using Python multiprocessing but mostly wants to be its own freestanding process: for example, it awkwardly works around inheriting the main process logger by avoiding the "parsl" logging topic; and it communicates with the main process mostly via ZMQ (with only a small bit of multiprocessing-based communication near the start).

Multiprocessing-fork is a Bad Thing (see issue #2343) so launching the interchange this way should go away.

Launching as a fresh process would also allow more normal logging (for example, logging everything to the interchange log file rather than carefully avoiding what was inherited).

The only use of multiprocessing queues, which is to communicate the ZMQ port numbers to be used by workers, could be transmitted in a different way: for example add a WORKER_PORTS command to the command channel.

Describe the solution you'd like
Use normal fork/exec so this behaves like a freestanding process

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions