DataLad hangs waiting for input in non-interactive job

[bpoldrack]

Origin: INM-7 data channel

User reported datalad get hanging in a compute job. Caveat is, that the datalad special remote is trying to ask for credentials assuming it's running in an interactive shell when it isn't. Furthermore, providing credentials is not a solution, since it is perfectly fine to fail. There are multiple sources for the content and failure with this particular source is expected (leading to try another that would succeed).
The issue is recorded in datalad/datalad#7345 and addressed by datalad/datalad#7344 (unmerged as of now). A patch allowing the user to use the configuration provided by this PR was provided to the user and confirmed to solve the issue. Actual release with a fix should be announced at the original channel once it exists.

TODO (not necessarily to be performed in this order)

• Inform OP/Add reference to this issue at origin
• Clarifying Qs asked or not needed
• Nature of the issue is understood
• Inform OP about resolution

[adswa]

I'm revisiting this, to see if we can prevent old issues from staying open forever. The associated issue in core also has "severity-critical", which warrants elevated attention.

The current state of things is this: The PR in datalad-core is somewhat abandoned. As Ben left, he likely won't work on it anymore. Yarik is unhappy with the change the PR introduces because, if I read it correctly, it would break authentication workflows because interactivity detection defaults to non-interactive unless users explicitly set environment or config variables (I tend to agree that this would be suboptimal and confusing). And Michael pointed in datalad/datalad#7344 (comment) to an alternative solution in datalad-next for setting configurations that make it into the subprocesses, which would prevent the default-to-non-interactivity that is questioned in the PR review.

I think the situation is not resolvable out of the box. If I read everything correctly, it needs some changes from the PR that allow our special remotes to be non-interactive. I believe this is the NoDialogUnderAnnexUI backend introduced there. But the pieces that make processes default to non-interactivity and cause behavior changes need to go. In their place, it needs documentation how to explicitly configure non-interactivity via next, and a KBI how to detect this kind of stalling and what to do in response. So we need to keep a few things, rip out a few things, and add a few things. Sadly this isn't really trivial. I would like to discuss this in the next datalad-devcall, and have put it on the agenda.