pytest-xdist interop: signal only works in main thread

[pytest-timeout-bot]

Original report by Buck Evan (Bitbucket: bukzor, GitHub: bukzor).

---

I'm getting this kind of error with increasing frequency as I add tests to my project.

#!python

INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "<remote exec>", line 59, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 72, in run_tests
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 413, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 424, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/_pytest/core.py", line 315, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 70, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/travis/build/Yelp/venv-update/.tox/test/lib/python3.4/site-packages/pytest_timeout.py", line 95, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

Full detail:

• http://paste.pound-python.org/show/qnK7qab1vZEYbYIzwLXD/
• http://paste.pound-python.org/show/5SnhQMlELjJgiE3Lwpim/

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

Oops, apologies to miss the original report!

It seems like execnet must have changed where xdist's DSession is run and it is no longer run in the main thread. Not immediately sure what to do about this but I think you should be able to work around this by using --timeout-method=thread.

[pytest-timeout-bot]

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

---

Any update on this issue? hitting this regularly

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

@karthikborkar does the work-around fix it for you?

[pytest-timeout-bot]

Original comment by Karthik Borkar (Bitbucket: [Karthik Borkar](https://bitbucket.org/Karthik Borkar), ).

---

@flub From https://pypi.python.org/pypi/pytest-timeout the thread method has few downsides such as relatively large overhead for running each test and that test runs are not completed. This means that other py.test features, like e.g. JUnit XML output or fixture teardown, will not function normally. fixture teardown and JUnit XML are important features which i need them to work correctly

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

But it is the most reliable way of using pytest-timeout. pytest-timeout aims to be a last-resort attempt at finding problems that you otherwise would just be stuck with and not get any useful infor from. It's kind of misleading to think you can still continue your test run after a timeout occurred. And I do wonder if the default method is wrong because of this.

[pytest-timeout-bot]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

---

I am getting the same kind of errors when using pytest-timeout with pytest-xdist:

https://travis-ci.org/Peque/osbrain/jobs/197634358

Using --timeout-method=thread fixes this issue (i.e.: test suite completes) but, when a timeout actually occurs, the whole test suite gets stuck (just as if I didn't use pytest-timeout) and no useful stack-trace information is given. The output does not tell me which test timed-out either.

[pytest-timeout-bot]

Original comment by Bulat Gaifullin (Bitbucket: bgaifullin, GitHub: bgaifullin).

---

the issue affects me too.

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

So currently it seems pytest-timeout and pytest-xdist seem to work together fine, I ran this:

import pytest, time

@pytest.mark.parametrize('a', [1, 2, 3, 4])
@pytest.mark.timeout(1)
def test_foo(a):
    time.sleep(2)
    assert a

And the timeouts work. So I'll close this unless someone reopens with a reproducible example.

Thanks

[pytest-timeout-bot]

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

---

Can still reproduce this. With pytest < 3.3 it would happen nearly every run, now it happens every fifth run or so.

#!python
from time import sleep

import pytest


@pytest.mark.timeout(10000)
@pytest.mark.parametrize("value", range(0, 20))
def test_timeout(value):
    sleep(5)
    pass

Full output:

#!python

(venv) darius@darius-ubuntu:~/dev/repro$ pytest test_timeout.py -v -n4
============================================================================================================================ test session starts ============================================================================================================================
platform linux2 -- Python 2.7.12, pytest-3.5.1, py-1.5.3, pluggy-0.6.0 -- /home/darius/dev/repro/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/darius/dev/repro, inifile:
plugins: xdist-1.22.2, timeout-1.2.1, forked-0.2
[gw0] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw1] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw2] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw3] linux2 Python 2.7.12 cwd: /home/darius/dev/repro
[gw0] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw3] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw1] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
[gw2] Python 2.7.12 (default, Dec  4 2017, 14:50:18)  -- [GCC 5.4.0 20160609]
gw0 [20] / gw1 [20] / gw2 [20] / gw3 [20]
scheduling tests via LoadScheduling
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session

test_timeout.py::test_timeout[1] 
test_timeout.py::test_timeout[3] 
test_timeout.py::test_timeout[2] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 65, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 84, in run_one_test
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 175, in _multicall
INTERNALERROR>     next(gen)   # first yield
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 76, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pytest_timeout.py", line 115, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR> ValueError: signal only works in main thread

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 107, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 145, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 116, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 139, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/darius/dev/repro/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 181, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_timeout.py::test_timeout[0]', <WorkerController gw2>)
INTERNALERROR> assert not 'test_timeout.py::test_timeout[0]'

======================================================================================================================= no tests ran in 5.64 seconds ========================================================================================================================

Keep in mind, I'm not actually triggering the timeout, it's much larger than the test can reasonably take. Runs fine without -n4.

pip list:

#!python
Package        Version
-------------- -------
apipkg         1.4    
attrs          18.1.0 
execnet        1.5.0  
funcsigs       1.0.2  
more-itertools 4.1.0  
pip            10.0.1 
pluggy         0.6.0  
py             1.5.3  
pytest         3.5.1  
pytest-forked  0.2    
pytest-timeout 1.2.1  
pytest-xdist   1.22.2 
setuptools     39.1.0 
six            1.11.0 
wheel          0.31.1 

Running on Ubuntu 16.04. I have not noticed this issue on Windows or Mac.

[pytest-timeout-bot]

Original comment by Floris Bruynooghe (Bitbucket: flub, GitHub: flub).

---

Sorry, with your sample file in a loop I still can't reproduce this. I did a pip install pytest pytest-timeout pytest-xdist. I even added pytest-forked as maybe a trigger but running the test in a loop still didn't trigger anything.

Have you tried this in a complete clean environment? Or is there anything else hidden, like a config file or so (as otherwise pytest-forked wouldn't do anything)

[pytest-timeout-bot]

Original comment by Darius Lapūnas (Bitbucket: DariusL, GitHub: DariusL).

---

Ok, so I tried redoing everything from scratch, these are my steps:

#!python
virtualenv venv
source venv/bin/activate
pip install pytest pytest-timeout pytest-xdist
pytest test_timeout.py -v -n4
pytest test_timeout.py -v -n4

It failed on the second run. I also checked which pytest just in case and it's the one in my virtual environment.

I'm not sure what other system information could be relevant. I'm seeing this both on my Ubuntu 16.04 VM (running in Parallels on a Mac) and on our Jenkins servers, also Ubuntu 16.04.

[pytest-timeout-bot]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque).

---

I too think this should be reopened. See:

https://travis-ci.org/opensistemas-hub/osbrain/jobs/379614919

With:

• pytest==3.5.1
• pytest-timeout==1.2.1
• pytest-xdist==1.22.2

That is the result of trying to reintegrate pytest-timeout in osBrain with no other changes.

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

I am not sure that pytest-timeout has anything to do with this directly.

FWIW I am getting the same kind of errors in scancode-toolkit test runs... but ONLY on Travis/Linux and not always.... kinda like every other run like others have reported here... and re-running the failed run makes the issue go away.

See https://travis-ci.org/nexB/scancode-toolkit/jobs/458977684#L1189

The tests I run are using pytest and xdist AND the underlying code supports timeout-based interruption in multiprocessing .
See https://github.com/nexB/scancode-toolkit/blob/develop/src/scancode/interrupt.py#L121

It could well be that the code I used for timeouts is using the same calls that pytest-timeout... which could explain why I am facing this too. In any case, I thought posting here could help

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

ok, my timeout handling code is functionally the same set of calls that pytest-timeout uses. So it makes sense to experience the same issues.

[pytest-timeout-bot]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne).

---

Now the interesting part is that pytest-xdist does not seem to be part of the problem either as Darius said above https://bitbucket.org/pytest-dev/pytest-timeout/issues/8/pytest-xdist-interop-signal-only-works-in#comment-45439827

So this seems to boil down t an issue between pytest and timout signals