Recursion Crashes Under Windows in Python 3.12 and 3.13

[mrbean-bremen]

Discussed in #1095

^{Originally posted by MikeTheHammer December 3, 2024}
I think this is a bug, but it might be something I'm doing wrong.

I've run the following smoke test under Linux (Ubuntu 24.04) and Windows 10 LTSC, using Pythons 3.9 - 3.13 .

def test__pyfakefs_smoke(fs):
    filename = r"C:\source_file"
    fs.create_file(filename)
    with open(filename) as source_file:
        assert True

Under Linux, this test passes on all Pythons 3.9 - 3.13. Under Windows, it passes on 3.9 - 3.11, but crashes with a RecursionError: maximum recursion depth exceeded on Python 3.12 and 3.13.

On Python 3.12, the stack trace is:

============================= test session starts =============================
platform win32 -- Python 3.12.4, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\Users\Hammer\pyfakefs_smoke
plugins: pyfakefs-5.7.2
collected 1 item

test_pyfakefs_smoke.py F                                                 [100%]

================================== FAILURES ===================================
____________________________ test__pyfakefs_smoke _____________________________

filename = 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv312\\Scripts\\pytest.exe\\__main__.py'
module_globals = None

    def updatecache(filename, module_globals=None):
        """Update a cache entry and return its list of lines.
        If something's wrong, print a message, discard the cache entry,
        and return an empty list."""
    
        if filename in cache:
            if len(cache[filename]) != 1:
                cache.pop(filename, None)
        if not filename or (filename.startswith('<') and filename.endswith('>')):
            return []
    
        fullname = filename
        try:
>           stat = os.stat(fullname)
E           FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv312\\Scripts\\pytest.exe\\__main__.py'

C:\Program Files\Python312\Lib\linecache.py:93: FileNotFoundError

That error repeats 74 times, followed by:

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000019388945EE0>

    def test__pyfakefs_smoke(fs):
        filename = r"C:\source_file"
        fs.create_file(filename)
>       with open(filename) as source_file:

C:\Users\Hammer\pyfakefs_smoke\test_pyfakefs_smoke.py:4: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python312\Lib\traceback.py:232: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python312\Lib\traceback.py:395: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python312\Lib\traceback.py:438: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python312\Lib\traceback.py:323: in line
    self._line = linecache.getline(self.filename, self.lineno)
C:\Program Files\Python312\Lib\linecache.py:30: in getline
    lines = getlines(filename, module_globals)
C:\Program Files\Python312\Lib\linecache.py:46: in getlines
    return updatecache(filename, module_globals)
C:\Program Files\Python312\Lib\linecache.py:101: in updatecache
    data = cache[filename][0]()
<frozen zipimport>:196: in get_source
    ???
<frozen zipimport>:534: in _get_data
    ???
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:124: in open_code
    return self._io_module.open_code(path)
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv312\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python312\Lib\traceback.py:232: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python312\Lib\traceback.py:395: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python312\Lib\traceback.py:438: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python312\Lib\traceback.py:323: in line
    self._line = linecache.getline(self.filename, self.lineno)
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
=========================== short test summary info ===========================
FAILED test_pyfakefs_smoke.py::test__pyfakefs_smoke - RecursionError: maximum...
============================== 1 failed in 0.64s ==============================

The stack trace in 3.13 is similar:

============================= test session starts =============================
platform win32 -- Python 3.13.0, pytest-8.3.4, pluggy-1.5.0
rootdir: C:\Users\Hammer\pyfakefs_smoke
plugins: pyfakefs-5.7.2
collected 1 item

test_pyfakefs_smoke.py F                                                 [100%]

================================== FAILURES ===================================
____________________________ test__pyfakefs_smoke _____________________________

filename = 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py'
module_globals = None

    def updatecache(filename, module_globals=None):
        """Update a cache entry and return its list of lines.
        If something's wrong, print a message, discard the cache entry,
        and return an empty list."""
    
        # These imports are not at top level because linecache is in the critical
        # path of the interpreter startup and importing os and sys take a lot of time
        # and slows down the startup sequence.
        import os
        import sys
        import tokenize
    
        if filename in cache:
            if len(cache[filename]) != 1:
                cache.pop(filename, None)
        if not filename or (filename.startswith('<') and filename.endswith('>')):
            return []
    
        fullname = filename
        try:
>           stat = os.stat(fullname)

C:\Program Files\Python313\Lib\linecache.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = ('C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py',)
kwargs = {}, should_use_original = True

    @functools.wraps(f)
    def wrapped(*args, **kwargs):
        should_use_original = FakeOsModule.use_original
    
        if not should_use_original and args:
            self = args[0]
            fs: FakeFilesystem = self.filesystem
            if self.filesystem.patcher:
                skip_names = fs.patcher.skip_names
                if is_called_from_skipped_module(
                    skip_names=skip_names,
                    case_sensitive=fs.is_case_sensitive,
                ):
                    should_use_original = True
    
        if should_use_original:
            # remove the `self` argument for FakeOsModule methods
            if args and isinstance(args[0], FakeOsModule):
                args = args[1:]
>           return getattr(os, f.__name__)(*args, **kwargs)
E           FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Hammer\\pyfakefs_smoke\\venv313\\Scripts\\pytest.exe\\__main__.py'

C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_os.py:1454: FileNotFoundError

This is repeated 60 times, followed by:

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x0000024459FD9E80>

    def test__pyfakefs_smoke(fs):
        filename = r"C:\source_file"
        fs.create_file(filename)
        assert os.path.exists(filename)
>       with open(filename) as source_file:

C:\Users\Hammer\pyfakefs_smoke\test_pyfakefs_smoke.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python313\Lib\traceback.py:260: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python313\Lib\traceback.py:445: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python313\Lib\traceback.py:492: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python313\Lib\traceback.py:369: in line
    self._set_lines()
C:\Program Files\Python313\Lib\traceback.py:350: in _set_lines
    lines.append(linecache.getline(self.filename, lineno).rstrip())
C:\Program Files\Python313\Lib\linecache.py:25: in getline
    lines = getlines(filename, module_globals)
C:\Program Files\Python313\Lib\linecache.py:41: in getlines
    return updatecache(filename, module_globals)
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_filesystem_unittest.py:702: in updatecache
    return self.linecache_updatecache(filename, module_globals)
C:\Program Files\Python313\Lib\linecache.py:108: in updatecache
    data = cache[filename][0]()
C:\Program Files\Python313\Lib\linecache.py:189: in get_lines
    return get_source(name, *args, **kwargs)
<frozen zipimport>:196: in get_source
    ???
<frozen zipimport>:617: in _get_data
    ???
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:124: in open_code
    return self._io_module.open_code(path)
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_io.py:93: in open
    return fake_open(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\fake_open.py:89: in fake_open
    if is_called_from_skipped_module(
C:\Users\Hammer\pyfakefs_smoke\venv313\Lib\site-packages\pyfakefs\helpers.py:494: in is_called_from_skipped_module
    stack = traceback.extract_stack()
C:\Program Files\Python313\Lib\traceback.py:260: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
C:\Program Files\Python313\Lib\traceback.py:445: in extract
    return klass._extract_from_extended_frame_gen(
C:\Program Files\Python313\Lib\traceback.py:492: in _extract_from_extended_frame_gen
    f.line
C:\Program Files\Python313\Lib\traceback.py:369: in line
    self._set_lines()
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)
=========================== short test summary info ===========================
FAILED test_pyfakefs_smoke.py::test__pyfakefs_smoke - RecursionError: maximum...
============================== 1 failed in 1.20s ==============================

Versions:

• pytest 8.3.4

• pyfakefs 5.7.2

• Linux Ubuntu 24.04 - Linux 6.8.0-49-generic x86_64:

  • Python 3.9.19
  • Python 3.10.14
  • Python 3.11.9
  • Python 3.12.4
  • Python 3.13.0

• Windows 10 LTSC 21H2 (OS Build 19044.5131) :

  • Python 3.9.13
  • Python 3.10.11
  • Python 3.11.9
  • Python 3.12.4
  • Python 3.13.0

[mrbean-bremen]

I have seen a somewhat similar behavior with #1086, but labeled it as a limitation due to another context.
But if that happens with such a basic test, it is certainly a bug, at least as far as I can see now.

I did not really understand the cause of the problem (it has to do with the virtual environment and the Python path, but I was not able to find a good solution to prevent this).

An easy workaround to fix this is to use python -m pytest instead of calling pytest directly. This is generally considered good practice, but the direct call should also work. I cannot promise a timely fix, as I already have been struggling with this without success, but I will see what I can do.

[MikeTheHammer]

Yes, running it as python -m pytest works for me with both 3.12 and 3.13.

[mrbean-bremen]

@MikeTheHammer - I added a workaround that should mitigate the issue. Can you please check if the main branch works for you?

[mrbean-bremen]

I've made a new patch release with the workaround. I'm not happy with the solution and will try to understand the underlying issue better, but I'm closing this issue for now. Please re-open if the issue persists with the current version.

[b-morgenthaler]

@mrbean-bremen
Hi, I recently ran into this issue as well (Python 3.12 with pyfakefs 5.8.0 on Windows 10) and wondered why your workaraound patch didn't work. I noticed this in my traceback:

FileNotFoundError: [WinError 3] Das System kann den angegebenen Pfad nicht finden: 'D:\\path\\to\\.venv\\Scripts\\pytest.EXE\\__main__.py'

Please notice, the upper case letters for 'EXE' which prevents your workaround to actually work in this scenario.

Patching (locally) the workaround line

            # workaround for updatecache problem with pytest under Windows, see #1096
            if not filename.endswith(r"pytest.exe\__main__.py"):
                return self.linecache_updatecache(filename, module_globals)
            return []

to

            # workaround for updatecache problem with pytest under Windows, see #1096
            if not filename.endswith(r"pytest.exe\__main__.py") and not filename.endswith(r"pytest.EXE\__main__.py"):
                return self.linecache_updatecache(filename, module_globals)
            return []

ultimately fixed it for me as well (I assume there are more elegant ways to accomplish this - I just wanted to proof my point).

Since I couldn't re-open this issue, I just wanted to let you know that the workaround could be improved. Until then, we are going to call pytest by python -m pytest.

[mrbean-bremen]

Thanks - that makes total sense. We should have just ignored the capitalization, given that Windows is not case-sensitive.
I'm still not happy about the workaround in general, but I will at least adapt it so that it will work for you.

[b-morgenthaler]

Thank you

[mrbean-bremen]

I couldn't think of a test for this, but as it is a very trivial change I think we can live with this.
Should now work in main branch, let me know if you need a patch release.