escaping pass-through arguments is too eager

[soletan]

Background

When trying to use concurrently with --passthrough-arguments option, any passed arguments get escaped too eagerly.

The issue has been observed while developing an application when trying to concurrently run a local web server and gherkin-testcafe for testing with the latter requring tags to start with @ character.

Since that application can't be shared here for legal reasons I've tried a simpler scenario which is reproducing the issue, too.

Scenario

• windows platform
• npm 10.5.2
• node 20.13.1
• concurrently 8.2.2
• files:
  • package.json contains scripts:

    {"scripts": {
        "foo": "concurrently -k -n \"bar,baz\" -P npm:foo:bar \"npm:foo:baz -- {@}\"",
        "foo:bar": "node block.js",
        "foo:baz": "node log.js",
    }}

  • file block.js looks like this:

    setTimeout(() => console.log("delayed"), 1000);

  • file log.js looks like this:

    console.dir(process.argv);

Test

invoke without arguments

$ npm run foo

> logistics-client@0.0.0 foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}"

[bar] 
[bar] > logistics-client@0.0.0 foo:bar
[bar] > node block.js
[bar]
[baz]
[baz] > logistics-client@0.0.0 foo:baz
[baz] > node log.js
[baz]
[baz] [
[baz]   'C:\\Program Files\\nodejs\\node.exe',
[baz]   'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js'
[baz] ]
[baz] npm run foo:baz --  exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1

looks good ... setup is working as expected

invoke with arguments to pass

$ npm run foo -- -- --tags=@foo

> logistics-client@0.0.0 foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}" -- --tags=@foo

[bar]
[bar] > logistics-client@0.0.0 foo:bar
[bar] > node block.js
[bar]
[baz]
[baz] > logistics-client@0.0.0 foo:baz
[baz] > node log.js --tags\=\@foo
[baz]
[baz] [
[baz]   'C:\\Program Files\\nodejs\\node.exe',
[baz]   'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz]   '--tags\\=\\@foo'
[baz] ]
[baz] npm run foo:baz -- --tags\=\@foo exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1

Problem here:

• the assignment operator gets escaped
• the @ gets escaped

Escaping seems to be eligible due to cmd.exe being involved. But eventually that escaping is found in invoked sub-process, too. Trying to fix it in that sub-process is not an option for it is a third-party application (gherkin-testcafe).

gherkin-testcafe illustrates the provision of tags without assignment operator, so I tried that one, too:

$ npm run foo -- -- --tags @foo

> logistics-client@0.0.0 foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {@}" -- --tags @foo

[baz] 
[baz] > logistics-client@0.0.0 foo:baz
[baz] > node log.js --tags \@foo
[baz] 
[bar] 
[bar] > logistics-client@0.0.0 foo:bar
[bar] > node block.js
[bar] 
[baz] [
[baz]   'C:\\Program Files\\nodejs\\node.exe',
[baz]   'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz]   '--tags',
[baz]   '\\@foo'
[baz] ]
[baz] npm run foo:baz -- --tags \@foo exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1

The @ character still ends up escaped.

I've also tried using {*} on invoking concurrently. So, the script in package.json is now:

"foo": "concurrently -k -n \"bar,baz\" -P npm:foo:bar \"npm:foo:baz -- {*}\"",

Invoking as before prevents the escaping, but yields a different issue:

$ npm run foo -- -- --tags @foo

> logistics-client@0.0.0 foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {*}" -- --tags @foo

[baz] 
[baz] > logistics-client@0.0.0 foo:baz
[baz] > node log.js '--tags @foo'
[baz]
[bar]
[bar] > logistics-client@0.0.0 foo:bar
[bar] > node block.js
[bar]
[baz] [
[baz]   'C:\\Program Files\\nodejs\\node.exe',
[baz]   'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz]   "'--tags",
[baz]   "@foo'"
[baz] ]
[baz] npm run foo:baz -- '--tags @foo' exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1

So, the {*} seems to wrap all additional arguments in a single pair quotes. This may be intentional, but it definitely doesn't help here:

$ npm run foo -- -- --tags @foo --debug-on-fail

> logistics-client@0.0.0 foo
> concurrently -k -n "bar,baz" -P npm:foo:bar "npm:foo:baz -- {*}" -- --tags @foo --debug-on-fail

[baz] 
[baz] > logistics-client@0.0.0 foo:baz
[baz] > node log.js '--tags @foo --debug-on-fail'
[baz]
[bar]
[bar] > logistics-client@0.0.0 foo:bar
[bar] > node block.js
[bar]
[baz] [
[baz]   'C:\\Program Files\\nodejs\\node.exe',
[baz]   'C:\\Users\\cepharum\\PhpstormProjects\\UID\\Logistics\\WebClient\\Source\\UiNew\\log.js',
[baz]   "'--tags",
[baz]   '@foo',
[baz]   "--debug-on-fail'"
[baz] ]
[baz] npm run foo:baz -- '--tags @foo --debug-on-fail' exited with code 0
--> Sending SIGTERM to other processes..
[bar] npm run foo:bar exited with code 1

[soletan]

I've just cloned concurrently locally to create a build off current main branch, then linked it into that other project.

$ git clone https://github.com/open-cli-tools/concurrently.git
$ cd concurrently
$ npm i
$ npm build
$ npm link

Then, in my project:

$ npm link concurrently

The outcome is the same. I thought it might have been related to that windowVerbatimArguments option set. But nope. I'm a fool ...

[soletan]

I've kept on trying to nail down the issue. From concurrently perspective, relying on shell-quote to do the job seems to cause this issue. As pointed out in the linked issue (comment1 comment2), maybe shell-quote isn't meant to support Windows' cmd.exe. What about concurrently? ... just asking.

[gustavohenke]

Hey, thanks for the report.
Yes, concurrently should work on Windows. We have, however, been naive not to test with certain special characters such as @ and =, like you did.

I'm willing to be more lenient with escaping/wrapping in quotes than the shell-quote package if it means having less bugs like yours. E.g. arguments with a space need wrapping, unless they're already wrapped, of course. But I'm not sure about the others, might need to go through the list below and reimplement it in concurrently:

https://github.com/ljharb/shell-quote/blob/9eecafc0486c9321be223415cf3fb76a5bd07dda/quote.js#L14

(that said, I'm admittedly ignorant of risks of getting this wrong 😅 )

[soletan]

Well, I was hesitant to provide a solution I found based on my follow-up investigations as documented in the related issue at shell-quote repository for I was hoping to get a solution done there. However, since I wasn't able to wait for that with the original issue I was facing I've created a patched version of concurrently with shell-quote being replaced with my package. Maybe, I should provide it as a PR here.

[nathanpovo]

I ran into this same issue.

This is the concurrently command I have in my package.json:

"build": "npx concurrently --passthrough-arguments \"npm:build-cms -- {@}\" \"npm:build-lms -- {@}\" -- ",

When used with the command npm run build -- --env=baseUrl='/' I get the following output:

❯  npm run build -- --env=baseUrl='/'

> project@1.0.0 build
> npx concurrently --passthrough-arguments "npm:build-cms -- {@}" "npm:build-lms -- {@}" -- --env=baseUrl=/

[build-cms]
[build-cms] > project@1.0.0 build-cms
[build-cms] > cd ./CMS && npx webpack --env\=baseUrl\=/
[build-cms]
[build-lms]
[build-lms] > project@1.0.0 build-lms
[build-lms] > cd ./LMS && npx webpack --env\=baseUrl\=/
[build-lms]
[build-cms] [webpack-cli] Error: Unknown option '--env\=baseUrl\=/'
[build-lms] [webpack-cli] Error: Unknown option '--env\=baseUrl\=/'
[build-cms] [webpack-cli] Did you mean '--env'?
[build-lms] [webpack-cli] Did you mean '--env'?
[build-cms] [webpack-cli] Run 'webpack --help' to see available commands and options
[build-lms] [webpack-cli] Run 'webpack --help' to see available commands and options
[build-lms] npm run build-lms -- --env\=baseUrl\=/ exited with code 2
[build-cms] npm run build-cms -- --env\=baseUrl\=/ exited with code 2

Note how = is escaped and the ' characters are removed.

In my case, wrapping the pass-through argurments in double quotes results in everything being passed-through as expected:

Wrapping the pass-through arguments in double quotes results in everything looking like it would be ok:

❯  npm run build -- "--env=baseUrl='/'"

> project@1.0.0 build
> npx concurrently --passthrough-arguments "npm:build-cms -- {@}" "npm:build-lms -- {@}" -- --env=baseUrl='/'

[build-lms]
[build-lms] > project@1.0.0 build-lms
[build-lms] > cd ./LMS && npx webpack --env=baseUrl='/'
[build-lms]
[build-cms]
[build-cms] > project@1.0.0 build-cms
[build-cms] > cd ./CMS && npx webpack --env=baseUrl='/'
[build-cms]
[build-cms] Base URL has been set to: '/'
[build-cms] API URL has been set to: '/'api

but the single quotes are erroneously being passed to the sub-command. When I run the webpack command directly in the relevant directory you can see the correct usage of the slash:

❯  npx webpack --env=baseUrl='/'
Base URL has been set to: /
API URL has been set to: /api

---

Version information:

Node: 18.20.8
npm: 10.8.2
concurrently: 8.2.2
PowerShell 7.4.7
Windows Terminal 1.22.10731.0

[soletan]

I haven't had much time since reporting this issue and there is a PR I have to take another look into though basically it is working as I have tested myself in a fork which has been in use at our site since then.

[nathanpovo]

No problem. I just found this github issue while investigating my issue and wanted to document the issue on my side.

I actually updated my comment because my workaround didn't actually resolve the problem.