Skip to content

Commit f365314

Browse files
authored
Enable Conversions Between Native Python Enum Types and C++ Enums (#5555)
* Apply smart_holder-branch-based PR #5280 on top of master. * Add pytest.skip("GraalPy does not raise UnicodeDecodeError") * Add `parent_scope` as first argument to `py::native_enum` ctor. * Replace `operator+=` API with `.finalize()` API. The error messages still need cleanup. * Resolve clang-tidy performance-unnecessary-value-param errors * Rename (effectively) native_enum_add_to_parent() -> finalize() * Update error message: pybind11::native_enum<...>("Fake", ...): MISSING .finalize() * Pass py::module_ by reference to resolve clang-tidy errors (this is entirely inconsequential otherwise for all practical purposes). * test_native_enum_correct_use_failure -> test_native_enum_missing_finalize_failure * Add test_native_enum_double_finalize(), test_native_enum_value_after_finalize() * Clean up public/protected API. * [ci skip] Update the Enumerations section in classes.rst * Rename `py::native_enum_kind` → `py::enum_kind` as suggested by gh-henryiii: #5555 (comment) * Experiment: StrEnum enum.StrEnum does not map to C++ enum: * https://chatgpt.com/share/67d5e965-ccb0-8008-95b7-0df2502309b3 ``` ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0 C++ Info: 13.3.0 C++20 __pybind11_internals_v10000000_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__ PYBIND11_SIMPLE_GIL_MANAGEMENT=False PYBIND11_NUMPY_1_ONLY=False configfile: pytest.ini plugins: parallel-0.1.1, xdist-3.6.1 collected 40 items / 39 deselected / 1 selected test_native_enum.py F [100%] =================================== FAILURES =================================== ________________________ test_native_enum_StrEnum_greek ________________________ def test_native_enum_StrEnum_greek(): assert not hasattr(m, "greek") > m.native_enum_StrEnum_greek(m) test_native_enum.py:150: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3.12/enum.py:764: in __call__ return cls._create_( boundary = None cls = <enum 'StrEnum'> module = None names = [('Alpha', 10), ('Omega', 20)] qualname = None start = 1 type = None value = 'greek' values = () /usr/lib/python3.12/enum.py:917: in _create_ return metacls.__new__(metacls, class_name, bases, classdict, boundary=boundary) _ = <class 'str'> bases = (<enum 'StrEnum'>,) boundary = None class_name = 'greek' classdict = {'_generate_next_value_': <function StrEnum._generate_next_value_ at 0x701ec1711e40>, 'Alpha': 10, 'Omega': 20, '__module__': 'test_native_enum'} cls = <enum 'StrEnum'> first_enum = <enum 'StrEnum'> item = ('Omega', 20) member_name = 'Omega' member_value = 20 metacls = <class 'enum.EnumType'> module = 'test_native_enum' names = [('Alpha', 10), ('Omega', 20)] qualname = None start = 1 type = None /usr/lib/python3.12/enum.py:606: in __new__ raise exc.with_traceback(tb) __class__ = <class 'enum.EnumType'> __new__ = <function StrEnum.__new__ at 0x701ec1711da0> _gnv = <staticmethod(<function StrEnum._generate_next_value_ at 0x701ec1711e40>)> _order_ = None _simple = False bases = (<enum 'StrEnum'>,) boundary = None classdict = {'Alpha': <enum._proto_member object at 0x701ebc74f9b0>, 'Omega': <enum._proto_member object at 0x701ebc74cce0>, '__module__': 'test_native_enum', '_all_bits_': 0, ...} cls = 'greek' exc = TypeError('10 is not a string') first_enum = <enum 'StrEnum'> ignore = ['_ignore_'] invalid_names = set() key = '_ignore_' kwds = {} member_names = {'Alpha': None, 'Omega': None} member_type = <class 'str'> metacls = <class 'enum.EnumType'> name = 'Omega' save_new = False tb = <traceback object at 0x701ebc7a6cc0> use_args = True value = 20 /usr/lib/python3.12/enum.py:596: in __new__ enum_class = super().__new__(metacls, cls, bases, classdict, **kwds) __class__ = <class 'enum.EnumType'> __new__ = <function StrEnum.__new__ at 0x701ec1711da0> _gnv = <staticmethod(<function StrEnum._generate_next_value_ at 0x701ec1711e40>)> _order_ = None _simple = False bases = (<enum 'StrEnum'>,) boundary = None classdict = {'Alpha': <enum._proto_member object at 0x701ebc74f9b0>, 'Omega': <enum._proto_member object at 0x701ebc74cce0>, '__module__': 'test_native_enum', '_all_bits_': 0, ...} cls = 'greek' exc = TypeError('10 is not a string') first_enum = <enum 'StrEnum'> ignore = ['_ignore_'] invalid_names = set() key = '_ignore_' kwds = {} member_names = {'Alpha': None, 'Omega': None} member_type = <class 'str'> metacls = <class 'enum.EnumType'> name = 'Omega' save_new = False tb = <traceback object at 0x701ebc7a6cc0> use_args = True value = 20 /usr/lib/python3.12/enum.py:271: in __set_name__ enum_member = enum_class._new_member_(enum_class, *args) args = (10,) enum_class = <enum 'greek'> member_name = 'Alpha' self = <enum._proto_member object at 0x701ebc74f9b0> value = 10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cls = <enum 'greek'>, values = (10,) def __new__(cls, *values): "values must already be of type `str`" if len(values) > 3: raise TypeError('too many arguments for str(): %r' % (values, )) if len(values) == 1: # it must be a string if not isinstance(values[0], str): > raise TypeError('%r is not a string' % (values[0], )) E TypeError: 10 is not a string cls = <enum 'greek'> values = (10,) /usr/lib/python3.12/enum.py:1322: TypeError =========================== short test summary info ============================ FAILED test_native_enum.py::test_native_enum_StrEnum_greek - TypeError: 10 is... ======================= 1 failed, 39 deselected in 0.07s ======================= ERROR: completed_process.returncode=1 ``` * Remove StrEnum code. * Make enum_kind::Enum the default kind. * Catch redundant .export_values() calls. * [ci skip] Add back original documentation for `py::enum_` under new advanced/deprecated.rst * [ci skip] Add documentation for `py::enum_kind` and `py::detail::type_caster_enum_type_enabled` * Rename `Type` to `EnumType` for readability. * Eliminate py::enum_kind, use "enum.Enum", "enum.IntEnum" directly. This is still WIP. * EXPERIMENTAL StrEnum code. To be removed. * Remove experimental StrEnum code: My judgement: Supporting StrEnum is maybe nice, but not very valuable. I don't think it is worth the extra C++ code. A level of indirection would need to be managed, e.g. RED ↔ Python "r" ↔ C++ 0 Green ↔ Python "g" ↔ C++ 1 These mappings would need to be stored and processed. * Add test with enum.IntFlag (no production code changes required). * First import_or_getattr() implementation (dedicated tests are still missing). * Fix import_or_getattr() implementation, add tests, fix clang-tidy errors. * [ci skip] Update classes.rst: replace `py::enum_kind` with `native_type_name` * For "constructor similar to that of enum.Enum" point to https://docs.python.org/3/howto/enum.html#functional-api, as suggested by gh-timohl (#5555 (comment)). * Advertise Enum, IntEnum, Flag, IntFlags are compatible stdlib enum types in the documentation (as suggested by gh-timohl, #5555 (review)); add test for enum.Flag to ensure that is actually true.
1 parent 48eb5ad commit f365314

18 files changed

+1154
-54
lines changed

CMakeLists.txt

+2
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,7 @@ set(PYBIND11_HEADERS
136136
include/pybind11/detail/dynamic_raw_ptr_cast_if_possible.h
137137
include/pybind11/detail/init.h
138138
include/pybind11/detail/internals.h
139+
include/pybind11/detail/native_enum_data.h
139140
include/pybind11/detail/struct_smart_holder.h
140141
include/pybind11/detail/type_caster_base.h
141142
include/pybind11/detail/typeid.h
@@ -162,6 +163,7 @@ set(PYBIND11_HEADERS
162163
include/pybind11/gil_safe_call_once.h
163164
include/pybind11/iostream.h
164165
include/pybind11/functional.h
166+
include/pybind11/native_enum.h
165167
include/pybind11/numpy.h
166168
include/pybind11/operators.h
167169
include/pybind11/pybind11.h

docs/advanced/cast/custom.rst

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
.. _custom_type_caster:
2+
13
Custom type casters
24
===================
35

docs/advanced/deprecated.rst

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
Deprecated
2+
##########
3+
4+
.. _deprecated_enum:
5+
6+
``py::enum_``
7+
=============
8+
9+
This is the original documentation for ``py::enum_``, which is deprecated
10+
because it is not `PEP 435 compatible <https://peps.python.org/pep-0435/>`_
11+
(see also `#2332 <https://github.com/pybind/pybind11/issues/2332>`_).
12+
Please prefer ``py::native_enum`` (added with pybind11v3) when writing
13+
new bindings. See :ref:`native_enum` for more information.
14+
15+
Let's suppose that we have an example class that contains internal types
16+
like enumerations, e.g.:
17+
18+
.. code-block:: cpp
19+
20+
struct Pet {
21+
enum Kind {
22+
Dog = 0,
23+
Cat
24+
};
25+
26+
struct Attributes {
27+
float age = 0;
28+
};
29+
30+
Pet(const std::string &name, Kind type) : name(name), type(type) { }
31+
32+
std::string name;
33+
Kind type;
34+
Attributes attr;
35+
};
36+
37+
The binding code for this example looks as follows:
38+
39+
.. code-block:: cpp
40+
41+
py::class_<Pet> pet(m, "Pet");
42+
43+
pet.def(py::init<const std::string &, Pet::Kind>())
44+
.def_readwrite("name", &Pet::name)
45+
.def_readwrite("type", &Pet::type)
46+
.def_readwrite("attr", &Pet::attr);
47+
48+
py::enum_<Pet::Kind>(pet, "Kind")
49+
.value("Dog", Pet::Kind::Dog)
50+
.value("Cat", Pet::Kind::Cat)
51+
.export_values();
52+
53+
py::class_<Pet::Attributes>(pet, "Attributes")
54+
.def(py::init<>())
55+
.def_readwrite("age", &Pet::Attributes::age);
56+
57+
58+
To ensure that the nested types ``Kind`` and ``Attributes`` are created within the scope of ``Pet``, the
59+
``pet`` ``py::class_`` instance must be supplied to the :class:`enum_` and ``py::class_``
60+
constructor. The :func:`enum_::export_values` function exports the enum entries
61+
into the parent scope, which should be skipped for newer C++11-style strongly
62+
typed enums.
63+
64+
.. code-block:: pycon
65+
66+
>>> p = Pet("Lucy", Pet.Cat)
67+
>>> p.type
68+
Kind.Cat
69+
>>> int(p.type)
70+
1L
71+
72+
The entries defined by the enumeration type are exposed in the ``__members__`` property:
73+
74+
.. code-block:: pycon
75+
76+
>>> Pet.Kind.__members__
77+
{'Dog': Kind.Dog, 'Cat': Kind.Cat}
78+
79+
The ``name`` property returns the name of the enum value as a unicode string.
80+
81+
.. note::
82+
83+
It is also possible to use ``str(enum)``, however these accomplish different
84+
goals. The following shows how these two approaches differ.
85+
86+
.. code-block:: pycon
87+
88+
>>> p = Pet("Lucy", Pet.Cat)
89+
>>> pet_type = p.type
90+
>>> pet_type
91+
Pet.Cat
92+
>>> str(pet_type)
93+
'Pet.Cat'
94+
>>> pet_type.name
95+
'Cat'
96+
97+
.. note::
98+
99+
When the special tag ``py::arithmetic()`` is specified to the ``enum_``
100+
constructor, pybind11 creates an enumeration that also supports rudimentary
101+
arithmetic and bit-level operations like comparisons, and, or, xor, negation,
102+
etc.
103+
104+
.. code-block:: cpp
105+
106+
py::enum_<Pet::Kind>(pet, "Kind", py::arithmetic())
107+
...
108+
109+
By default, these are omitted to conserve space.
110+
111+
.. warning::
112+
113+
Contrary to Python customs, enum values from the wrappers should not be compared using ``is``, but with ``==`` (see `#1177 <https://github.com/pybind/pybind11/issues/1177>`_ for background).

docs/classes.rst

+68-40
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,8 @@ you can use ``py::detail::overload_cast_impl`` with an additional set of parenth
459459
other using the ``.def(py::init<...>())`` syntax. The existing machinery
460460
for specifying keyword and default arguments also works.
461461

462+
.. _native_enum:
463+
462464
Enumerations and internal types
463465
===============================
464466

@@ -487,76 +489,102 @@ The binding code for this example looks as follows:
487489

488490
.. code-block:: cpp
489491
492+
#include <pybind11/native_enum.h> // Not already included with pybind11.h
493+
490494
py::class_<Pet> pet(m, "Pet");
491495
492496
pet.def(py::init<const std::string &, Pet::Kind>())
493497
.def_readwrite("name", &Pet::name)
494498
.def_readwrite("type", &Pet::type)
495499
.def_readwrite("attr", &Pet::attr);
496500
497-
py::enum_<Pet::Kind>(pet, "Kind")
501+
py::native_enum<Pet::Kind>(pet, "Kind")
498502
.value("Dog", Pet::Kind::Dog)
499503
.value("Cat", Pet::Kind::Cat)
500-
.export_values();
504+
.export_values()
505+
.finalize();
501506
502507
py::class_<Pet::Attributes>(pet, "Attributes")
503508
.def(py::init<>())
504509
.def_readwrite("age", &Pet::Attributes::age);
505510
506511
507-
To ensure that the nested types ``Kind`` and ``Attributes`` are created within the scope of ``Pet``, the
508-
``pet`` ``py::class_`` instance must be supplied to the :class:`enum_` and ``py::class_``
509-
constructor. The :func:`enum_::export_values` function exports the enum entries
510-
into the parent scope, which should be skipped for newer C++11-style strongly
511-
typed enums.
512+
To ensure that the nested types ``Kind`` and ``Attributes`` are created
513+
within the scope of ``Pet``, the ``pet`` ``py::class_`` instance must be
514+
supplied to the ``py::native_enum`` and ``py::class_`` constructors. The
515+
``.export_values()`` function is available for exporting the enum entries
516+
into the parent scope, if desired.
512517

513-
.. code-block:: pycon
518+
``py::native_enum`` was introduced with pybind11v3. It binds C++ enum types
519+
to native Python enum types, typically types in Python's
520+
`stdlib enum <https://docs.python.org/3/library/enum.html>`_ module,
521+
which are `PEP 435 compatible <https://peps.python.org/pep-0435/>`_.
522+
This is the recommended way to bind C++ enums.
523+
The older ``py::enum_`` is not PEP 435 compatible
524+
(see `issue #2332 <https://github.com/pybind/pybind11/issues/2332>`_)
525+
but remains supported indefinitely for backward compatibility.
526+
New bindings should prefer ``py::native_enum``.
514527

515-
>>> p = Pet("Lucy", Pet.Cat)
516-
>>> p.type
517-
Kind.Cat
518-
>>> int(p.type)
519-
1L
528+
.. note::
520529

521-
The entries defined by the enumeration type are exposed in the ``__members__`` property:
530+
The deprecated ``py::enum_`` is :ref:`documented here <deprecated_enum>`.
522531

523-
.. code-block:: pycon
532+
The ``.finalize()`` call above is needed because Python's native enums
533+
cannot be built incrementally — all name/value pairs need to be passed at
534+
once. To achieve this, ``py::native_enum`` acts as a buffer to collect the
535+
name/value pairs. The ``.finalize()`` call uses the accumulated name/value
536+
pairs to build the arguments for constructing a native Python enum type.
524537

525-
>>> Pet.Kind.__members__
526-
{'Dog': Kind.Dog, 'Cat': Kind.Cat}
538+
The ``py::native_enum`` constructor supports a third optional
539+
``native_type_name`` string argument, with default value ``"enum.Enum"``.
540+
Other types can be specified like this:
527541

528-
The ``name`` property returns the name of the enum value as a unicode string.
542+
.. code-block:: cpp
529543
530-
.. note::
544+
py::native_enum<Pet::Kind>(pet, "Kind", "enum.IntEnum")
531545
532-
It is also possible to use ``str(enum)``, however these accomplish different
533-
goals. The following shows how these two approaches differ.
546+
Any fully-qualified Python name can be specified. The only requirement is
547+
that the named type is similar to
548+
`enum.Enum <https://docs.python.org/3/library/enum.html#enum.Enum>`_
549+
in these ways:
534550

535-
.. code-block:: pycon
551+
* Has a `constructor similar to that of enum.Enum
552+
<https://docs.python.org/3/howto/enum.html#functional-api>`_::
536553

537-
>>> p = Pet("Lucy", Pet.Cat)
538-
>>> pet_type = p.type
539-
>>> pet_type
540-
Pet.Cat
541-
>>> str(pet_type)
542-
'Pet.Cat'
543-
>>> pet_type.name
544-
'Cat'
554+
Colors = enum.Enum("Colors", (("Red", 0), ("Green", 1)))
555+
556+
* A `C++ underlying <https://en.cppreference.com/w/cpp/types/underlying_type>`_
557+
enum value can be passed to the constructor for the Python enum value::
558+
559+
red = Colors(0)
560+
561+
* The enum values have a ``.value`` property yielding a value that
562+
can be cast to the C++ underlying type::
563+
564+
underlying = red.value
565+
566+
As of Python 3.13, the compatible `types in the stdlib enum module
567+
<https://docs.python.org/3/library/enum.html#module-contents>`_ are:
568+
``Enum``, ``IntEnum``, ``Flag``, ``IntFlag``.
545569

546570
.. note::
547571

548-
When the special tag ``py::arithmetic()`` is specified to the ``enum_``
549-
constructor, pybind11 creates an enumeration that also supports rudimentary
550-
arithmetic and bit-level operations like comparisons, and, or, xor, negation,
551-
etc.
572+
In rare cases, a C++ enum may be bound to Python via a
573+
:ref:`custom type caster <custom_type_caster>`. In such cases, a
574+
template specialization like this may be required:
552575

553576
.. code-block:: cpp
554577
555-
py::enum_<Pet::Kind>(pet, "Kind", py::arithmetic())
556-
...
557-
558-
By default, these are omitted to conserve space.
578+
#if defined(PYBIND11_HAS_NATIVE_ENUM)
579+
namespace pybind11::detail {
580+
template <typename FancyEnum>
581+
struct type_caster_enum_type_enabled<
582+
FancyEnum,
583+
std::enable_if_t<is_fancy_enum<FancyEnum>::value>> : std::false_type {};
584+
}
585+
#endif
559586
560-
.. warning::
587+
This specialization is needed only if the custom type caster is templated.
561588

562-
Contrary to Python customs, enum values from the wrappers should not be compared using ``is``, but with ``==`` (see `#1177 <https://github.com/pybind/pybind11/issues/1177>`_ for background).
589+
The ``PYBIND11_HAS_NATIVE_ENUM`` guard is needed only if backward
590+
compatibility with pybind11v2 is required.

docs/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
advanced/pycpp/index
3737
advanced/embedding
3838
advanced/misc
39+
advanced/deprecated
3940

4041
.. toctree::
4142
:caption: Extra Information

0 commit comments

Comments
 (0)