[WIP] Preparation for v3.0.0 release: Upgrade guide & Changelog

docs/changelog.rst

@@ -10,29 +10,224 @@ Changes will be added here periodically from the "Suggested changelog entry"
 block in pull request descriptions.
 
 
-IN DEVELOPMENT
---------------
+IN DEVELOPMENT (v3.0.0)
+-----------------------
 
 Changes will be summarized here periodically.
 
+
 New Features:
 
-* Support for Python 3.7 was removed. (Official end-of-life: 2023-06-27).
-  `#5191 <https://github.com/pybind/pybind11/pull/5191>`_
+* The ``smart-holder`` branch has been merged, enabling ``py::class_<T,
+  py::smart_holder>``, which handles two-way conversion with
+  ``std::unique_ptr<T>`` and ``std::shared_ptr<T>`` (simultaneously), disowning
+  a Python object being passed to ``std::unique_ptr<T>``, trampoline objects,
+  and ``std::enable_shared_from_this``.
+  `#5542 <https://github.com/pybind/pybind11/pull/5542>`_
+
+* Remove support for pybind11 v2 internals versions (4, 5, 6).
+  (The internals version number has been bumped for pybind11 v3.)
+  `#5512 <https://github.com/pybind/pybind11/pull/5512>`_ |
+  `#5530 <https://github.com/pybind/pybind11/pull/5530>`_
+
+* (CMake) Enable FindPython mode by default, with a ``COMPAT`` mode that sets some of the old variables to ease transition.
+  `#5553 <https://github.com/pybind/pybind11/pull/5553>`_
+
+* ``py::native_enum`` was added, for conversions between Python's native (stdlib) enum types and C++ enums.
+  `#5555 <https://github.com/pybind/pybind11/pull/5555>`_
+
+* A ``py::release_gil_before_calling_cpp_dtor`` option (for ``py::class_``) was added to resolve the long-standing issue #1446.
+  `#5522 <https://github.com/pybind/pybind11/pull/5522>`_
+
+* Add ``dtype::normalized_num`` and ``dtype::num_of``.
+  `#5429 <https://github.com/pybind/pybind11/pull/5429>`_
+
+* Add support for ``array_t<handle>`` and ``array_t<object>``.
+  `#5427 <https://github.com/pybind/pybind11/pull/5427>`_
+
+* Added ``py::warnings`` namespace with ``py::warnings::warn`` and ``py::warnings::new_warning_type`` that provides the interface for Python warnings.
+  `#5291 <https://github.com/pybind/pybind11/pull/5291>`_
 
 * stl.h ``list|set|map_caster`` were made more user friendly: it is no longer
   necessary to explicitly convert Python iterables to ``tuple()``, ``set()``,
   or ``map()`` in many common situations.
   `#4686 <https://github.com/pybind/pybind11/pull/4686>`_
 
+* The ``array_caster`` in pybind11/stl.h was enhanced to support value types that are not default-constructible.
+  `#5305 <https://github.com/pybind/pybind11/pull/5305>`_
+
+* ``pybind11/conduit/pybind11_platform_abi_id.h`` was factored out, to maximize reusability of ``PYBIND11_PLATFORM_ABI_ID`` (for other Python/C++ binding systems).
+  `#5375 <https://github.com/pybind/pybind11/pull/5375>`_
+
+* Changed ``PYBIND11_MODULE`` macro implementation to perform multi-phase module initialization (PEP 489) behind the scenes.
+  `#5574 <https://github.com/pybind/pybind11/pull/5574>`_
+
+* Added support for finding pybind11 using pkgconf distributed on pypi.
+  `#5552 <https://github.com/pybind/pybind11/pull/5552>`_
+
+* Support ``--extension-suffix`` on the pybind11 command.
+  `#5360 <https://github.com/pybind/pybind11/pull/5360>`_
+
+
+New Features (typing):
+
+.. feat(types)
+
+* Added option for different arg/return type hints to ``type_caster``. Updated
+  ``stl/filesystem`` to use correct arg/return type hints. Updated
+  ``pybind11::typing`` to use correct arg/return type hints for nested types.
+  `#5450 <https://github.com/pybind/pybind11/pull/5450>`_
+
+
+* Updated type hint for ``py::capsule`` to ``type.CapsuleType``.
+  `#5567 <https://github.com/pybind/pybind11/pull/5567>`_
+
+* Adds support for ``typing.SupportsInt`` and ``typing.SupportsFloat``.
+  Update ``Final`` to be narrower type hint. Make ``std::function`` match
+  ``Callable`` type. Fix ``io_name`` bug in ``attr_with_type_hint``.
+  `#5540 <https://github.com/pybind/pybind11/pull/5540>`_
+
+* Rework of arg/return type hints to support ``.noconvert()``.
+  `#5486 <https://github.com/pybind/pybind11/pull/5486>`_
+
+* Add ``attr_with_type`` for declaring attribute types and ``Final``, ``ClassVar`` type annotations.
+  `#5460 <https://github.com/pybind/pybind11/pull/5460>`_
+
+* Allow annotate methods with ``py::pos_only`` when only have the ``self`` argument. Make arguments for auto-generated dunder methods positional-only.
+  `#5403 <https://github.com/pybind/pybind11/pull/5403>`_
+
+* Added ``py::Args`` and ``py::KWArgs`` to enable custom type hinting of ``*args`` and ``**kwargs`` (see PEP 484).
+  `#5357 <https://github.com/pybind/pybind11/pull/5357>`_
+
+* Switched to ``numpy.typing.NDArray`` and ``numpy.typing.ArrayLike``.
+  `#5212 <https://github.com/pybind/pybind11/pull/5212>`_
+
+.. fix(types)
+
+* Use ``numpy.object_`` instead of ``object``.
+  `#5571 <https://github.com/pybind/pybind11/pull/5571>`_
+
+* Fix module type hint.
+  `#5469 <https://github.com/pybind/pybind11/pull/5469>`_
+
+
+Bug fixes:
+
+* Set ``__file__`` on submodules.
+  `#5584 <https://github.com/pybind/pybind11/pull/5584>`_
+
+* pybind11-bound functions are now pickleable.
+  `#5580 <https://github.com/pybind/pybind11/pull/5580>`_
+
+* Fix bug in ``attr_with_type_hint`` to allow objects to be in ``attr_with_type_hint``.
+  `#5576 <https://github.com/pybind/pybind11/pull/5576>`_
+
+* A ``-Wmaybe-uninitialized`` warning suppression was added in ``pybind11/eigen/matrix.h``.
+  `#5516 <https://github.com/pybind/pybind11/pull/5516>`_
+
+* ``PYBIND11_WARNING_POP`` was incorrectly defined as ``PYBIND11_PRAGMA(clang diagnostic push)``.
+  `#5448 <https://github.com/pybind/pybind11/pull/5448>`_
+
+* ``PYBIND11_PLATFORM_ABI_ID`` (which is used in composing ``PYBIND11_INTERNALS_ID``) was modernized to reflect actual ABI compatibility more accurately.
+  `#4953 <https://github.com/pybind/pybind11/pull/4953>`_ |
+  `#5439 <https://github.com/pybind/pybind11/pull/5439>`_
+
+* Fix buffer protocol implementation.
+  `#5407 <https://github.com/pybind/pybind11/pull/5407>`_
+
+* Fix iterator increment operator does not skip first item.
+  `#5400 <https://github.com/pybind/pybind11/pull/5400>`_
+
+* When getting or deleting an element in a container bound by ``bind_map``, print the key in ``KeyError`` if it does not exist.
+  `#5397 <https://github.com/pybind/pybind11/pull/5397>`_
+
+* ``pybind11::builtin_exception`` is now explicitly exported when linked to libc++.
+  `#5390 <https://github.com/pybind/pybind11/pull/5390>`_
+
+* Allow subclasses of ``py::args`` and ``py::kwargs``.
+  `#5381 <https://github.com/pybind/pybind11/pull/5381>`_
+
+* Disable false-positive GCC 12 Bound Check warning.
+  `#5355 <https://github.com/pybind/pybind11/pull/5355>`_
+
+* fix: using ``__cpp_nontype_template_args`` instead of ``__cpp_nontype_template_parameter_class``.
+  `#5330 <https://github.com/pybind/pybind11/pull/5330>`_
+
+* Properly translate C++ exception to Python exception when creating Python buffer from wrapped object.
+  `#5324 <https://github.com/pybind/pybind11/pull/5324>`_
+
+
+.. fix(cmake)
+
+Bug fixes (CMake):
+
+* Add an author warning that auto-calculated ``PYTHON_MODULE_EXTENSION`` may not respect ``SETUPTOOLS_EXT_SUFFIX`` during cross-compilation.
+  `#5495 <https://github.com/pybind/pybind11/pull/5495>`_
+
+* Don't strip with ``CMAKE_BUILD_TYPE`` None.
+  `#5392 <https://github.com/pybind/pybind11/pull/5392>`_
+
+* Fix an issue with ``NO_EXTRAS`` adding ``pybind11::windows_extras`` anyway.
+  `#5378 <https://github.com/pybind/pybind11/pull/5378>`_
+
+
+Bug fixes (free-threading):
+
+.. fix(free-threading)
+
+* Fix data race in free threaded CPython when accessing a shared static variable.
+  `#5494 <https://github.com/pybind/pybind11/pull/5494>`_
+
+* A free-threading data race in ``all_type_info()`` was fixed.
+  `#5419 <https://github.com/pybind/pybind11/pull/5419>`_
+
+* Added exception translator specific mutex used with ``try_translate_exceptions`` in the free-threaded build for internal locking.
+  `#5362 <https://github.com/pybind/pybind11/pull/5362>`_
+
+
+Documentation:
+
+* Improved ``reference_internal`` policy documentation.
+  `#5528 <https://github.com/pybind/pybind11/pull/5528>`_
+
+* A new "Double locking, deadlocking, GIL" document was added.
+  `#5394 <https://github.com/pybind/pybind11/pull/5394>`_
+
+* Adds an answer (FAQ) for "What is a highly conclusive and simple way to find memory leaks?".
+  `#5340 <https://github.com/pybind/pybind11/pull/5340>`_
+
+
+Tests:
+
+* Download the final Catch2 2.x release if Catch download is requested.
+  `#5568 <https://github.com/pybind/pybind11/pull/5568>`_
+
+* Explicitly used ``signed char`` for two numpy dtype tests. As seen when
+compiling using ``clang`` on Linux with the ``-funsigned-char`` flag.
+  `#5545 <https://github.com/pybind/pybind11/pull/5545>`_
+
+* Test PyPy3.11 in CI.
+  `#5534 <https://github.com/pybind/pybind11/pull/5534>`_
+
+
+New and removed platforms:
+
+* Added support for GraalPy Python implementation (https://github.com/oracle/graalpython).
+  `#5380 <https://github.com/pybind/pybind11/pull/5380>`_
+
+* Support for PyPy 3.11 added.
+  `#5508 <https://github.com/pybind/pybind11/pull/5508>`_
+
+* Support for PyPy 3.8 and 3.9 was dropped.
+  `#5578 <https://github.com/pybind/pybind11/pull/5578>`_
+
+* Support for Python 3.7 was removed. (Official end-of-life: 2023-06-27).
+  `#5191 <https://github.com/pybind/pybind11/pull/5191>`_
+
 * Support for CMake older than 3.15 removed. CMake 3.15-3.30 supported.
   `#5304 <https://github.com/pybind/pybind11/pull/5304>`_
 
-* The ``array_caster`` in pybind11/stl.h was enhanced to support value types that are not default-constructible.
-  `#5305 <https://github.com/pybind/pybind11/pull/5305>`_
 
-* Added ``py::warnings`` namespace with ``py::warnings::warn`` and ``py::warnings::new_warning_type`` that provides the interface for Python warnings.
-  `#5291 <https://github.com/pybind/pybind11/pull/5291>`_
 
 Version 2.13.6 (September 13, 2024)
 -----------------------------------

docs/classes.rst

@@ -580,7 +580,7 @@ As of Python 3.13, the compatible `types in the stdlib enum module
         template <typename FancyEnum>
         struct type_caster_enum_type_enabled<
             FancyEnum,
-            std::enable_if_t<is_fancy_enum<FancyEnum>::value>> : std::false_type {};
+            enable_if_t<is_fancy_enum<FancyEnum>::value>> : std::false_type {};
         }
         #endif
 

docs/upgrade.rst

@@ -8,6 +8,153 @@ to a new version. But it goes into more detail. This includes things like
 deprecated APIs and their replacements, build system changes, general code
 modernization and other useful information.
 
+.. _upgrade-guide-3.0:
+
+v3.0
+====
+
+pybind11 v3.0 introduces major new features, but the vast majority of
+existing extensions are expected to build and run without modification. Minor
+adjustments may be needed in rare cases, and any such changes can be easily
+wrapped in preprocessor conditionals to maintain compatibility with the
+2.x series.
+
+However, due to new features and modernizations, extensions built with
+pybind11 v3.0 are not ABI-compatible with those built using v2.13. To ensure
+cross-extension-module compatibility, it is recommended to rebuild all
+pybind11-based extensions with v3.0.
+
+CMake support now defaults to the modern FindPython module. If you haven't
+moved, we do some some backward compatibility ``PYTHON_*`` variables, but
+please update to using ``Python_*`` variables (and setting ``PYTHON_*``
+variables will not affect the build anymore).
+
+A major new feature in this release is the integration of
+``py::smart_holder``, which improves support for ``std::unique_ptr``
+and ``std::shared_ptr``, resolving several long-standing issues. See
+:ref:`smart_holder` for details. Closely related is the addition
+of ``py::trampoline_self_life_support``, documented under
+:ref:`overriding_virtuals`.
+
+This release includes a major modernization of cross-extension-module
+ABI compatibility handling. The new implementation reflects actual ABI
+compatibility much more accurately than in previous versions. The details
+are subtle and complex; see
+`#4953 <https://github.com/pybind/pybind11/pull/4953>`_ and
+`#5439 <https://github.com/pybind/pybind11/pull/5439>`_.
+
+Also new in v3.0 is ``py::native_enum``, a modern API for exposing
+C++ enumerations as native Python types — typically standard-library
+``enum.Enum`` or related subclasses. This provides improved integration with
+Python's enum system, compared to the older (now deprecated) ``py::enum_``.
+See #5555 <https://github.com/pybind/pybind11/pull/5555>_ for details.
+
+Functions exposed with pybind11 are now pickleable. This removes a
+long-standing obstacle when using pybind11-bound functions with Python features
+that rely on pickling, such as multiprocessing and caching tools.
+See #5580 <https://github.com/pybind/pybind11/pull/5580>_ for details.
+
+Migration Recommendations
+-------------------------
+
+We recommend migrating to pybind11 v3.0 promptly, while keeping initial
+changes to a minimum. Most projects can upgrade simply by updating the
+pybind11 version, without altering existing binding code.
+
+After a short stabilization period — enough to surface any subtle issues —
+you may incrementally adopt new features where appropriate:
+
+* Use ``py::smart_holder`` and ``py::trampoline_self_life_support`` as needed,
+  or to improve code health. Note that ``py::classh`` is available as a
+  shortcut — for example, ``py::classh<Pet>`` is shorthand for
+  ``py::class_<Pet, py::smart_holder>``. This is designed to enable easy
+  experimentation with ``py::smart_holder`` without introducing distracting
+  whitespace changes. In many cases, a global replacement of ``py::class_``
+  with ``py::classh`` can be an effective first step. Build failures will
+  quickly identify places where ``std::shared_ptr<...>`` holders need to be
+  removed. Runtime failures (assuming good unit test coverage) will highlight
+  base-and-derived class situations that require coordinated changes.
+
+  Note that ``py::bind_vector`` and ``py::bind_map`` (in pybind11/stl_bind.h)
+  have a ``holder_type`` template parameter that defaults to
+  ``std::unique_ptr``. If ``py::smart_holder`` functionality is desired or
+  required, use e.g. ``py::bind_vector<VecType, py::smart_holder>``.
+
+* Gradually migrate from ``py::enum_`` to ``py::native_enum`` to improve
+  integration with Python's standard enum types.
+
+There is no urgency to refactor existing, working bindings — adopt new
+features as the need arises or as part of ongoing maintenance efforts.
+
+If you are using CMake, update to FindPython variables (mostly changing
+variables from ``PYTHON_*`` -> ``Python_*``). You should see if you can use
+``set(PYBIND11_FINDPYTHON ON)``, which has been supported for years and will
+avoid setting the compatibly mode variables.
+
+Potential stumbling blocks when migrating to v3.0
+-------------------------------------------------
+
+The following issues are very unlikely to arise, and easy to work around:
+
+* In rare cases, a C++ enum may be bound to Python via a
+  :ref:`custom type caster <custom_type_caster>`. In such cases, a
+  template specialization like this may be required:
+
+  .. code-block:: cpp
+
+      #if defined(PYBIND11_HAS_NATIVE_ENUM)
+      namespace pybind11::detail {
+      template <typename FancyEnum>
+      struct type_caster_enum_type_enabled<
+          FancyEnum,
+          enable_if_t<is_fancy_enum<FancyEnum>::value>> : std::false_type {};
+      }
+      #endif
+
+  This specialization is needed only if the custom type caster is templated.
+
+  The ``PYBIND11_HAS_NATIVE_ENUM`` guard is needed only
+  if backward compatibility with pybind11v2 is required.
+
+* Similarly, template specializations like the following may be required
+  if there are custom
+
+  * ``pybind11::detail::copyable_holder_caster`` or
+
+  * ``pybind11::detail::move_only_holder_caster``
+
+  implementations that are used for ``std::shared_ptr`` or ``std::unique_ptr``
+  conversions:
+
+  .. code-block:: cpp
+
+      #if defined(PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT)
+      namespace pybind11::detail {
+      template <typename ExampleType>
+      struct copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled<
+          ExampleType,
+          enable_if_t<is_example_type<ExampleType>::value>> : std::false_type {};
+      }
+      #endif
+
+  .. code-block:: cpp
+
+      #if defined(PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT)
+      namespace pybind11::detail {
+      template <typename ExampleType>
+      struct move_only_holder_caster_unique_ptr_with_smart_holder_support_enabled<
+          ExampleType,
+          enable_if_t<is_example_type<ExampleType>::value>> : std::false_type {};
+      }
+      #endif
+
+  The ``PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT`` guard is needed only
+  if backward compatibility with pybind11v2 is required.
+
+  (Note that ``copyable_holder_caster`` and ``move_only_holder_caster`` are not
+  documented, although they existed since 2017.)
+
+
 .. _upgrade-guide-2.12:
 
 v2.12