`UnicodeEncodeError` in pluggy with surrogate escape in parametrization and `--debug`

[The-Compiler]

On Linux, with the current git main of pytest (8.5.0.dev60+g67738403d), pluggy 1.6.0 and Python 3.13.1, this test runs fine:

import pytest

@pytest.mark.parametrize("s", ["\ud800"])
def test_x(s):
    pass

but, somewhat contrary to its name, --debug=debug.log causes a bug instead:

_____________________ ERROR collecting test_surrogate.py _____________________
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:240: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:466: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:573: in call_extra
    return self._hookexec(self.name, hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:114: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)
src/_pytest/python.py:1274: in parametrize
    ids = self._resolve_parameter_set_ids(
src/_pytest/python.py:1395: in _resolve_parameter_set_ids
    return id_maker.make_unique_parameterset_ids()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:902: in make_unique_parameterset_ids
    resolved_ids = list(self._resolve_ids())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:945: in _resolve_ids
    yield "-".join(
src/_pytest/python.py:946: in <genexpr>
    self._idval(val, argname, idx)
src/_pytest/python.py:957: in _idval
    idval = self._idval_from_hook(val, argname)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:985: in _idval_from_hook
    id: str | None = self.config.hook.pytest_make_parametrize_id(
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:470: in traced_hookexec
    before(hook_name, hook_impls, caller_kwargs)
.venv/lib/python3.13/site-packages/pluggy/_manager.py:495: in before
    hooktrace(hook_name, kwargs)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:69: in __call__
    self.root._processmessage(self.tags, args)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:44: in _processmessage
    self._writer(self._format_message(tags, args))
E   UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 140: surrogates not allowed

Reporting here as it's somewhat unclear whether a proper fix for this should be in pytest or in pluggy (possibly the latter though, given that it can probably be reproduced with any hook execution with a string argument that has surrogate escapes?).

[RonnyPfannschmidt]

Seems like we need to configure surrogate escape error handling in pluggy

[bluetech]

The fact that there are strings in Python that cannot be UTF-8 encoded is such a massive gotcha that no one knows about.

I agree with @RonnyPfannschmidt that the fix should be in pluggy, it should use errors='replace' in its tracing function.