pytest-xdist interop: signal only works in main thread

Original report by Buck Evan (Bitbucket: bukzor, GitHub: bukzor).

I'm getting this kind of error with increasing frequency as I add tests to my project.

#!python

INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 59, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 72, in run_tests
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 70, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 95, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

Full detail:

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

Oops, apologies to miss the original report!

It seems like execnet must have changed where xdist's DSession is run and it is no longer run in the main thread. Not immediately sure what to do about this but I think you should be able to work around this by using --timeout-method=thread.

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

Any update on this issue? hitting this regularly

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

@karthikborkar does the work-around fix it for you?

@flub

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

@flub From https://pypi.python.org/pypi/pytest-timeout the thread method has few downsides such as relatively large overhead for running each test and that test runs are not completed. This means that other py.test features, like e.g. JUnit XML output or fixture teardown, will not function normally. fixture teardown and JUnit XML are important features which i need them to work correctly

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

But it is the most reliable way of using pytest-timeout. pytest-timeout aims to be a last-resort attempt at finding problems that you otherwise would just be stuck with and not get any useful infor from. It's kind of misleading to think you can still continue your test run after a timeout occurred. And I do wonder if the default method is wrong because of this.

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

I am getting the same kind of errors when using pytest-timeout with pytest-xdist:

https://travis-ci.org/Peque/osbrain/jobs/197634358

Using --timeout-method=thread fixes this issue (i.e.: test suite completes) but, when a timeout actually occurs, the whole test suite gets stuck (just as if I didn't use pytest-timeout) and no useful stack-trace information is given. The output does not tell me which test timed-out either.

Original comment by Bulat Gaifullin (Bitbucket: bgaifullin, GitHub: bgaifullin).

the issue affects me too.

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

So currently it seems pytest-timeout and pytest-xdist seem to work together fine, I ran this:

import pytest, time

@pytest.mark.parametrize('a', [1, 2, 3, 4])
@pytest.mark.timeout(1)
def test_foo(a):
    time.sleep(2)
    assert a

And the timeouts work. So I'll close this unless someone reopens with a reproducible example.

Thanks

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

Can still reproduce this. With pytest < 3.3 it would happen nearly every run, now it happens every fifth run or so.

#!python
from time import sleep

import pytest


@pytest.mark.timeout(10000)
@pytest.mark.parametrize("value", range(0, 20))
def test_timeout(value):
    sleep(5)
    pass

Full output:

#!python

(venv) darius@darius-ubuntu:~/dev/repro$ pytest test_timeout.py -v -n4
============================================================================================================================ test session starts ============================================================================================================================
platform linux2 -- Python 2.7.12, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /home/darius/dev/repro/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/darius/dev/repro, inifile:
plugins: xdist-1.22.2, timeout-1.2.1, forked-0.2
[gw0] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw1] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw2] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw3] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw0] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw3] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw1] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw2] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
gw0 [20] / gw1 [20] / gw2 [20] / gw3 [20]
scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session

test_timeout.py::test_timeout[1] 
test_timeout.py::test_timeout[3] 
test_timeout.py::test_timeout[2] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 65, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 84, in run_one_test
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 175, in _multicall
INTERNALERROR>     next(gen)   # first yield
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 76, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 115, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 116, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 139, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 181, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_timeout.py::test_timeout[0]', <WorkerController gw2>)
INTERNALERROR> assert not 'test_timeout.py::test_timeout[0]'

======================================================================================================================= no tests ran in 5.64 seconds ========================================================================================================================

Keep in mind, I'm not actually triggering the timeout, it's much larger than the test can reasonably take. Runs fine without -n4.

pip list:

#!python
Package        Version
-------------- -------
apipkg         1.4    
attrs          18.1.0 
execnet        1.5.0  
funcsigs       1.0.2  
more-itertools 4.1.0  
pip            10.0.1 
pluggy         0.6.0  
py             1.5.3  
pytest         3.5.1  
pytest-forked  0.2    
pytest-timeout 1.2.1  
pytest-xdist   1.22.2 
setuptools     39.1.0 
six            1.11.0 
wheel          0.31.1

Running on Ubuntu 16.04. I have not noticed this issue on Windows or Mac.

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

Sorry, with your sample file in a loop I still can't reproduce this. I did a pip install pytest pytest-timeout pytest-xdist. I even added pytest-forked as maybe a trigger but running the test in a loop still didn't trigger anything.

Have you tried this in a complete clean environment? Or is there anything else hidden, like a config file or so (as otherwise pytest-forked wouldn't do anything)

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

Ok, so I tried redoing everything from scratch, these are my steps:

#!python
virtualenv venv
source venv/bin/activate
pip install pytest pytest-timeout pytest-xdist
pytest test_timeout.py -v -n4
pytest test_timeout.py -v -n4

It failed on the second run. I also checked which pytest just in case and it's the one in my virtual environment.

I'm not sure what other system information could be relevant. I'm seeing this both on my Ubuntu 16.04 VM (running in Parallels on a Mac) and on our Jenkins servers, also Ubuntu 16.04.

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

I too think this should be reopened. See:

https://travis-ci.org/opensistemas-hub/osbrain/jobs/379614919

With:

pytest==3.5.1
pytest-timeout==1.2.1
pytest-xdist==1.22.2

That is the result of trying to reintegrate pytest-timeout in osBrain with no other changes.

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

I am not sure that pytest-timeout has anything to do with this directly.

FWIW I am getting the same kind of errors in scancode-toolkit test runs... but ONLY on Travis/Linux and not always.... kinda like every other run like others have reported here... and re-running the failed run makes the issue go away.

See https://travis-ci.org/nexB/scancode-toolkit/jobs/458977684#L1189

The tests I run are using pytest and xdist AND the underlying code supports timeout-based interruption in multiprocessing .
See https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py#L121

It could well be that the code I used for timeouts is using the same calls that pytest-timeout... which could explain why I am facing this too. In any case, I thought posting here could help

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

ok, my timeout handling code is functionally the same set of calls that pytest-timeout uses. So it makes sense to experience the same issues.

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

Now the interesting part is that pytest-xdist does not seem to be part of the problem either as Darius said above https://bitbucket.org/pytest-dev/pytest-timeout/issues/8/pytest-xdist-interop-signal-only-works-in#comment-45439827

So this seems to boil down t an issue between pytest and timout signals

@ndevenish

Hi @ndevenish, thanks for the explorations and additional info! The commit you point to is very interesting indeed. I'm a bit surprised that with xdist threaded mode can hang, I that'd be interesting to understand better as well.

Certainly do take this to a PR, at first glance I'm a bit torn on automatically switching to threaded vs just erroring out and forcing someone to decide this explicitly. But it would be nice to improve usability somehow and you found a pretty promising way I think.

(also, apologies I'm not always the most active, responses can take a while as you just noticed)

(also, apologies I'm not always the most active, responses can take a while as you just noticed)

I have no expectations for reply times for open source projects! I'll try to pull a PR together, for discussion if nothing else.

@flub

@flub this should be fixed (mostly) on execnet==1.8.0

see also pytest-dev/pytest-xdist#620

Ah, that's good to hear. I was settling on "Error, but with a more helpful suggestion" as the least-worst way to "fix", and didn't want to dig into execnet. Evidently it annoyed someone further than it annoyed me :)

@graingert

@graingert oh, that's very nice work you did! Might finally fix this issue properly and would mean we don't need a workaround here.

use thread method if not main thread

Yes, we've gotten ever since using xdist + timeout together. It's fairly random:

internal error
def worker_internal_error(self, node, formatted_error):
        """
        pytest_internalerror() was called on the worker.
    
        pytest_internalerror() arguments are an excinfo and an excrepr, which can't
        be serialized, so we go with a poor man's solution of raising an exception
        here ourselves using the formatted message.
        """
        self._active_nodes.remove(node)
        try:
>           assert False, formatted_error
E           AssertionError: Traceback (most recent call last):
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/_pytest/main.py", line 257, in wrap_session
E                 session.exitstatus = doit(config, session) or 0
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/_pytest/main.py", line 313, in _main
E                 config.hook.pytest_runtestloop(session=session)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
E                 return self._hookexec(self, self.get_hookimpls(), kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
E                 return self._inner_hookexec(hook, methods, kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
E                 return outcome.get_result()
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
E                 raise ex[1].with_traceback(ex[2])
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 187, in _multicall
E                 res = hook_impl.function(*args)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/remote.py", line 75, in pytest_runtestloop
E                 self.run_one_test(torun)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/remote.py", line 89, in run_one_test
E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
E                 return self._hookexec(self, self.get_hookimpls(), kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
E                 return self._inner_hookexec(hook, methods, kwargs)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
E                 self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 208, in _multicall
E                 return outcome.get_result()
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 80, in get_result
E                 raise ex[1].with_traceback(ex[2])
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pluggy/callers.py", line 182, in _multicall
E                 next(gen)  # first yield
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pytest_timeout.py", line 104, in pytest_runtest_protocol
E                 timeout_setup(item)
E               File "/opt/h2oai/dai/python/lib/python3.8/site-packages/pytest_timeout.py", line 207, in timeout_setup
E                 signal.signal(signal.SIGALRM, handler)
E               File "/opt/h2oai/dai/python/lib/python3.8/signal.py", line 47, in signal
E                 handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
E             ValueError: signal only works in main thread
E           assert False

/opt/h2oai/dai/python/lib/python3.8/site-packages/xdist/dsession.py:187: AssertionError

use thread method if not main thread

@flub

@flub Can you close this?

I guess so, pytest-timeout now recognises this situation and will refuse to use the signal timeout method, falling back to the thread method.

Navigation Menu

Repository navigation

pytest-xdist interop: signal only works in main thread #8

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

pytest-xdist interop: signal only works in main thread #8

Description

Activity

pytest-timeout-bot commented on Jun 9, 2015

pytest-timeout-bot commented on Dec 16, 2015

pytest-timeout-bot commented on Dec 16, 2015

pytest-timeout-bot commented on Dec 16, 2015

pytest-timeout-bot commented on Dec 16, 2015

pytest-timeout-bot commented on Feb 3, 2017

pytest-timeout-bot commented on May 12, 2018

pytest-timeout-bot commented on May 13, 2018

pytest-timeout-bot commented on May 15, 2018

pytest-timeout-bot commented on May 15, 2018

pytest-timeout-bot commented on May 16, 2018

pytest-timeout-bot commented on May 16, 2018

pytest-timeout-bot commented on Nov 24, 2018

pytest-timeout-bot commented on Nov 24, 2018

pytest-timeout-bot commented on Nov 24, 2018

11 remaining items

flub commented on Jan 27, 2021

ndevenish commented on Feb 1, 2021

graingert commented on Feb 3, 2021

ndevenish commented on Feb 3, 2021

flub commented on Feb 3, 2021

pseudotensor commented on Apr 27, 2021

xoviat commented on May 18, 2021

flub commented on Jun 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions