Repository navigation
Python 3.12: TypeError: cannot create weak reference to 'typing.TypeVar' object #507
Description
python3.12 -m venv _e
. _e/bin/activate
pip install -e .
pip install -rdev-requirements.txt
python -m pytest
Sample test failure:
_______________________________________ CloudPickleTest.test_generic_subclass ________________________________________
self = <tests.cloudpickle_test.CloudPickleTest testMethod=test_generic_subclass>
def test_generic_subclass(self):
T = typing.TypeVar('T')
class Base(typing.Generic[T]):
pass
class DerivedAny(Base):
pass
class LeafAny(DerivedAny):
pass
class DerivedInt(Base[int]):
pass
class LeafInt(DerivedInt):
pass
class DerivedT(Base[T]):
pass
class LeafT(DerivedT[T]):
pass
klasses = [
Base, DerivedAny, LeafAny, DerivedInt, LeafInt, DerivedT, LeafT
]
for klass in klasses:
> assert pickle_depickle(klass, protocol=self.protocol) is klass
tests/cloudpickle_test.py:2423:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/cloudpickle_test.py:81: in pickle_depickle
return pickle.loads(cloudpickle.dumps(obj, protocol=protocol))
cloudpickle/cloudpickle_fast.py:73: in dumps
cp.dump(obj)
cloudpickle/cloudpickle_fast.py:632: in dump
return Pickler.dump(self, obj)
cloudpickle/cloudpickle.py:909: in _typevar_reduce
return (_make_typevar, _decompose_typevar(obj))
cloudpickle/cloudpickle.py:899: in _decompose_typevar
_get_or_create_tracker_id(obj),
cloudpickle/cloudpickle.py:107: in _get_or_create_tracker_id
class_tracker_id = _DYNAMIC_CLASS_TRACKER_BY_CLASS.get(class_def)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <WeakKeyDictionary at 0x7f089efba2a0>, key = ~T, default = None
def get(self, key, default=None):
> return self.data.get(ref(key),default)
E TypeError: cannot create weak reference to 'typing.TypeVar' object
/usr/lib64/python3.12/weakref.py:452: TypeError
List of test failures:
============================================== short test summary info ===============================================
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_generic_subclass - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_generic_type - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_locally_defined_class_with_type_hints - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_module_importability - ModuleNotFoundError: No module named 'distutils'
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar_memoization - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::CloudPickleTest::test_pickle_dynamic_typevar_tracking - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_generic_subclass - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_generic_type - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_locally_defined_class_with_type_hints - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_module_importability - ModuleNotFoundError: No module named 'distutils'
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_constructs_from_module_registered_for_pickling_by_value - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar_memoization - TypeError: cannot create weak reference to 'typing.TypeVar' object
FAILED tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_dynamic_typevar_tracking - TypeError: cannot create weak reference to 'typing.TypeVar' object
============================== 16 failed, 227 passed, 17 skipped, 12 warnings in 8.26s ===============================
The No module named 'distutils' failures are a separate issue; distutils is removed from the standard library in Python 3.12, and it is necessary to either port away from it or introduce an explicit dependency on setuptools, which still provides it.
Activity
ogrisel commentedon Jul 5, 2023
@musicinmybrain would you be interested in building cpython from the python/cpython#106403 branch by @encukou to check that this is enough for cloudpickle?
musicinmybrain commentedon Jul 5, 2023
test.log
That fixed the issues reported here, but there were others, and there was an abort related to a weak reference. Next I’ll try cherry-picking JelleZijlstra/cpython@df59646 onto
v3.12.0b3, since I suspect some of the new problems could related to other Python 3.13 work.musicinmybrain commentedon Jul 5, 2023
Nope, it fails similarly to Python 3.13: test-py312.log
Four tests fail,
and a traceback for a weakref-related segfault is printed during
tests/cloudpickle_test.py::Protocol2CloudPickleTest::test_pickle_constructs_from _module_registered_for_pickling_by_value.If I skip these four tests (
python -m pytest -k 'not (CloudPickleTest and test_pickle_dynamic_typevar_tracking) and not (CloudPickleTest and test_pickle_constructs_from_module_registered_for_pickling_by_value) and not (Protocol2CloudPickleTest and test_pickle_constructs_from_module_registered_for_pickling_by_value) and not (Protocol2CloudPickleTest and test_pickle_dynamic_typevar_tracking)') the tests seem to pass, and the traceback disappears, but there a segfault is still reported at the end:ogrisel commentedon Jul 13, 2023
Pseudo-random ping to fellow cloudpickle contributors/maintainers: @mrocklin @jrbourbeau @jakirkham @robertnishihara @ueshin @HyukjinKwon @JoshRosen.
This issue with Python 3.12 will likely be disruptive for your downstream projects / businesses if left unattended. But I am personally running a bit out of spare cycles to investigate and report to CPython developers what's the root cause of the regression.
It would be great if some of you could have a look :)
ogrisel commentedon Oct 9, 2023
I tried to run the cloudpickle tests with a fresh new conda env running Python 3.12.0 from conda-forge on my macOS laptop and I can not reproduce the segfault when running the tests.
Furthermore #514 has an updated CI that runs Python 3.12 on linux / windows / macos and all tests are green.
I think we can close this issue. Thanks @musicinmybrain for your work with upstream.