[🐛 Bug]: Selenium 4.32.0 ruby bindings fail to start firefox in parallel testing

[tomhughes]

Description

Selenium 4.32.0 attempts to address issues with Firefox 136 and later failing when more that one instance is started (see mozilla/geckodriver#2218 for details) by allocating a unique bidi port for each instance in #15458.

That works fine if all the instances are started in a single thread but fails if two threads are starting instances simultaneously as PortProber works by allocating and then immediately closing a port leaving another thread free to do the same thing in the gap before firefox starts and binds it.

Reproducible Code

Not available

Debugging Logs

I added puts command.inspect to the build_process method in ServiceManager in order to see what processes were being started and then ran my test suite single threaded:

% PARALLEL_WORKERS=1 bundle exec rails test:system

Run options: --seed 22623

# Running:

["/usr/bin/geckodriver", "--port=4444", "--websocket-port", "9223"]
.["/usr/bin/geckodriver", "--port=4445", "--websocket-port", "9224"]
............................................................["/usr/bin/geckodriver", "--port=4446", "--websocket-port", "9225"]
...................................................................................................................

Finished in 110.831997s, 1.5880 runs/s, 7.3084 assertions/s.
176 runs, 810 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Minitest to /home/tom/src/osm/rails/coverage.
Line Coverage: 46.57% (3587 / 7702)
Lcov style coverage report generated for Minitest to coverage/lcov.info

and then I repeated the same with multiple threads:

% bundle exec rails test:system
Running 176 tests in parallel using 24 processes
Run options: --seed 57887

# Running:

["/usr/bin/geckodriver", "--port=4444", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4445", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4446", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4447", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4448", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4449", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4450", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4451", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4452", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4453", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4454", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4455", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4456", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4457", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4458", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4459", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4460", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4461", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4462", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4463", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4464", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4465", "--websocket-port", "9225"]
.["/usr/bin/geckodriver", "--port=4466", "--websocket-port", "9225"]
["/usr/bin/geckodriver", "--port=4467", "--websocket-port", "9225"]
.....E

Error:
IndexTest#test_should_not_add_overlay_when_not_closing_right_menu_popup:
Selenium::WebDriver::Error::NoSuchWindowError: Browsing context has been discarded
    test/system/index_test.rb:26:in 'block in <class:IndexTest>'

As you can see in the second case multiple instances were started using the same bidi port and tests then started failing en mass as most of the firefox instances had failed to start.

---

ℹ️ Last known working version: 4.31.0 with firefox < 136 or a monkey patch to add `--websocket-port=0`

[selenium-ci]

@tomhughes, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[titusfortner]

Ah, right. There's a race condition between asking for an unused port over 9222, and one of the processes actually locking that port. @p0deje is there a reason we don't make this random? (Doesn't actually fix the race condition, but might make it harder to hit). I know the other bindings use random ports in their implementations.

The only way to ensure there is no collision is to manage the websocket ports in the client, code.

[selenium-ci]

This issue is looking for contributors.

Please comment below or reach out to us through our IRC/Slack/Matrix channels if you are interested.

[tomhughes]

Does the selenium code actually need to know the port number? As far as I could see nothing actually connects to it?

If that is the case then --websocket-port=0 will cause firefox to pick a random port.

[titusfortner]

I thought @whimboo said that using 0 wasn't sufficient.

[whimboo]

Using port 0 would work for desktop but could cause issues for Android given that geckodriver uses adb to forward/reverse ports and for HTTP based sockets it requires the same port number for the device and host as explained in https://bugzilla.mozilla.org/show_bug.cgi?id=1754016.

We do not have a solution for that yet.