-
Notifications
You must be signed in to change notification settings - Fork 903
feat(config): recursively convert parsed dicts to typed dataclasses in loader #5269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
MikeGoldsmith
wants to merge
11
commits into
open-telemetry:main
Choose a base branch
from
MikeGoldsmith:mike/config-recursive-dict-conversion
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
e5d6932
recursively convert parsed dicts to typed dataclasses in loader
MikeGoldsmith 582c37f
rename changelog fragment to PR #5269
MikeGoldsmith b302f93
tighten typing on conversion module
MikeGoldsmith 3a6fd21
isolate typing.get_type_hints call to placate astroid 3.x on py3.14
MikeGoldsmith 131378c
inline the typing.get_type_hints wrap
MikeGoldsmith b6e4702
use ExemplarFilter for enum coercion test fixture; allow 'astroid' in…
MikeGoldsmith 70c93d9
add end-to-end loader tests covering YAML -> typed config -> factory
MikeGoldsmith f6e0eac
Merge branch 'main' into mike/config-recursive-dict-conversion
MikeGoldsmith fc07187
verify sampler and span processor wiring in factory test
MikeGoldsmith 4d24140
Merge remote-tracking branch 'origin/mike/config-recursive-dict-conve…
MikeGoldsmith 7e49de9
Merge branch 'main' into mike/config-recursive-dict-conversion
MikeGoldsmith File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| `opentelemetry-sdk`: declarative config loader now recursively converts parsed dicts into typed dataclass instances, including nested dataclasses, lists of dataclasses, and enum values. End-to-end YAML/JSON → SDK configuration now works via the factory functions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| [codespell] | ||
| # skipping auto generated folders | ||
| skip = ./.tox,./.mypy_cache,./docs/_build,./target,*/LICENSE,./venv,.git,./opentelemetry-semantic-conventions,*-requirements*.txt | ||
| ignore-words-list = ans,ue,ot,hist,ro | ||
| ignore-words-list = ans,ue,ot,hist,ro,astroid |
113 changes: 113 additions & 0 deletions
113
opentelemetry-sdk/src/opentelemetry/sdk/_configuration/_conversion.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # Copyright The OpenTelemetry Authors | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Recursive dict-to-dataclass conversion for parsed config data. | ||
|
|
||
| The YAML/JSON loader produces nested dicts. Factory functions expect typed | ||
| dataclass instances (e.g. ``TracerProvider``, ``SpanProcessor``). This module | ||
| walks each field's type annotation and converts nested dicts into their | ||
| corresponding dataclass types. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import dataclasses | ||
| import enum | ||
| import types | ||
| import typing | ||
| from collections.abc import Mapping | ||
| from typing import Any, TypeVar, Union, get_args, get_origin | ||
|
|
||
| _T = TypeVar("_T") | ||
|
|
||
|
|
||
| def _unwrap_optional(type_hint: Any) -> Any: | ||
| """Strip ``None`` from a ``X | None`` / ``Optional[X]`` annotation. | ||
|
|
||
| Returns the unwrapped type, or the original hint if not a Union with None. | ||
| """ | ||
| origin = get_origin(type_hint) | ||
| if origin is Union or origin is types.UnionType: | ||
| non_none = [t for t in get_args(type_hint) if t is not type(None)] | ||
| if len(non_none) == 1: | ||
| return non_none[0] | ||
| return type_hint | ||
|
|
||
|
|
||
| def _convert_value(value: Any, type_hint: Any) -> Any: | ||
| """Convert a value according to its type hint. | ||
|
|
||
| Recursively converts dicts to dataclasses and lists of dicts to lists of | ||
| dataclasses. Other values (primitives, enums, ``dict[str, Any]`` aliases) | ||
| pass through unchanged. | ||
| """ | ||
| if value is None: | ||
| return None | ||
|
|
||
| unwrapped = _unwrap_optional(type_hint) | ||
| origin = get_origin(unwrapped) | ||
|
|
||
| # list[X] — recurse on each element | ||
| if origin is list and isinstance(value, list): | ||
|
MikeGoldsmith marked this conversation as resolved.
|
||
| args = get_args(unwrapped) | ||
| if args: | ||
| item_type = args[0] | ||
| return [_convert_value(item, item_type) for item in value] | ||
| return value | ||
|
|
||
| # Direct dataclass type — recurse | ||
| if ( | ||
| isinstance(unwrapped, type) | ||
| and dataclasses.is_dataclass(unwrapped) | ||
| and isinstance(value, dict) | ||
| ): | ||
| return _dict_to_dataclass(value, unwrapped) | ||
|
|
||
| # Enum type — coerce string/value to the Enum member | ||
| if ( | ||
| isinstance(unwrapped, type) | ||
| and issubclass(unwrapped, enum.Enum) | ||
| and not isinstance(value, unwrapped) | ||
| ): | ||
| return unwrapped(value) | ||
|
|
||
| return value | ||
|
MikeGoldsmith marked this conversation as resolved.
|
||
|
|
||
|
|
||
| def _dict_to_dataclass(data: Mapping[str, Any], cls: type[_T]) -> _T: | ||
| """Recursively convert a mapping to a dataclass instance. | ||
|
|
||
| For each key in ``data``: | ||
| - If it matches a known dataclass field, the value is converted according | ||
| to that field's type annotation (recursing for nested dataclasses). | ||
| - Unknown keys are passed through as kwargs; classes decorated with | ||
| ``@_additional_properties`` will capture them on the instance's | ||
| ``additional_properties`` attribute. | ||
|
|
||
| ``ClassVar`` fields (e.g. the ``additional_properties`` annotation on | ||
| decorated dataclasses) are ignored as expected. | ||
|
|
||
| Raises: | ||
| TypeError: If ``cls`` is not a dataclass type. | ||
| """ | ||
| if not dataclasses.is_dataclass(cls): | ||
| raise TypeError(f"{cls.__name__} is not a dataclass") | ||
|
|
||
| # Annotated as ``dict[str, Any]`` so astroid stops tracing into | ||
| # ``typing.get_type_hints`` — under pylint 3.x that path leads into | ||
| # Python 3.14's ``annotationlib`` (which uses t-strings) and crashes. | ||
| hints: dict[str, Any] = dict( | ||
| typing.get_type_hints(cls, include_extras=False) | ||
| ) | ||
| known_fields = {f.name for f in dataclasses.fields(cls)} | ||
| kwargs: dict[str, Any] = {} | ||
|
|
||
| for key, value in data.items(): | ||
| if key in known_fields: | ||
| type_hint = hints.get(key) | ||
| kwargs[key] = _convert_value(value, type_hint) | ||
| else: | ||
| # Unknown key — @_additional_properties decorator will capture it. | ||
| kwargs[key] = value | ||
|
|
||
| return cls(**kwargs) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
110 changes: 110 additions & 0 deletions
110
opentelemetry-sdk/tests/_configuration/test_conversion.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,110 @@ | ||
| # Copyright The OpenTelemetry Authors | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| # Tests access private members of SDK classes to assert correct configuration. | ||
| # pylint: disable=protected-access | ||
|
|
||
| import unittest | ||
| from dataclasses import dataclass | ||
| from typing import Any, ClassVar | ||
|
|
||
| from opentelemetry.sdk._configuration._common import _additional_properties | ||
| from opentelemetry.sdk._configuration._conversion import _dict_to_dataclass | ||
| from opentelemetry.sdk._configuration.models import ExemplarFilter | ||
|
|
||
|
|
||
| @dataclass | ||
| class _Inner: | ||
| value: int | None = None | ||
|
|
||
|
|
||
| @dataclass | ||
| class _Middle: | ||
| inner: _Inner | None = None | ||
| items: list[_Inner] | None = None | ||
|
|
||
|
|
||
| @dataclass | ||
| class _Outer: | ||
| middle: _Middle | None = None | ||
| name: str | None = None | ||
|
|
||
|
|
||
| @_additional_properties | ||
| @dataclass | ||
| class _WithExtras: | ||
| known: str | None = None | ||
| additional_properties: ClassVar[dict[str, Any]] | ||
|
|
||
|
|
||
| @dataclass | ||
| class _WithEnum: | ||
| filter: ExemplarFilter | None = None | ||
|
|
||
|
|
||
| class TestDictToDataclass(unittest.TestCase): | ||
| def test_raises_on_non_dataclass(self): | ||
| # _dict_to_dataclass is internal and assumes cls is a dataclass. | ||
| with self.assertRaises(TypeError) as ctx: | ||
| _dict_to_dataclass({"x": 1}, dict) | ||
| self.assertIn("not a dataclass", str(ctx.exception)) | ||
|
|
||
| def test_converts_flat_dict(self): | ||
| result = _dict_to_dataclass({"value": 42}, _Inner) | ||
| self.assertIsInstance(result, _Inner) | ||
| self.assertEqual(result.value, 42) | ||
|
|
||
| def test_converts_nested_dataclass(self): | ||
| result = _dict_to_dataclass( | ||
| {"middle": {"inner": {"value": 7}}}, _Outer | ||
| ) | ||
| self.assertIsInstance(result, _Outer) | ||
| self.assertIsInstance(result.middle, _Middle) | ||
| self.assertIsInstance(result.middle.inner, _Inner) | ||
| self.assertEqual(result.middle.inner.value, 7) | ||
|
|
||
| def test_converts_list_of_dataclasses(self): | ||
| result = _dict_to_dataclass( | ||
| {"middle": {"items": [{"value": 1}, {"value": 2}]}}, _Outer | ||
| ) | ||
| self.assertEqual(len(result.middle.items), 2) | ||
| self.assertIsInstance(result.middle.items[0], _Inner) | ||
| self.assertEqual(result.middle.items[0].value, 1) | ||
| self.assertEqual(result.middle.items[1].value, 2) | ||
|
|
||
| def test_none_value_preserved(self): | ||
| result = _dict_to_dataclass({"middle": None, "name": "test"}, _Outer) | ||
| self.assertIsNone(result.middle) | ||
| self.assertEqual(result.name, "test") | ||
|
|
||
| def test_missing_optional_fields_default_to_none(self): | ||
| result = _dict_to_dataclass({}, _Outer) | ||
| self.assertIsNone(result.middle) | ||
| self.assertIsNone(result.name) | ||
|
|
||
| def test_unknown_keys_routed_to_additional_properties(self): | ||
| result = _dict_to_dataclass( | ||
| {"known": "yes", "my_plugin": {"opt": True}}, _WithExtras | ||
| ) | ||
| self.assertEqual(result.known, "yes") | ||
| self.assertEqual( | ||
| result.additional_properties, {"my_plugin": {"opt": True}} | ||
| ) | ||
|
|
||
| def test_primitive_values_pass_through(self): | ||
| result = _dict_to_dataclass({"name": "hello"}, _Outer) | ||
| self.assertEqual(result.name, "hello") | ||
|
|
||
| def test_empty_list_converted(self): | ||
| result = _dict_to_dataclass({"middle": {"items": []}}, _Outer) | ||
| self.assertEqual(result.middle.items, []) | ||
|
|
||
| def test_enum_value_coerced_from_string(self): | ||
| result = _dict_to_dataclass({"filter": "always_on"}, _WithEnum) | ||
| self.assertIs(result.filter, ExemplarFilter.always_on) | ||
|
|
||
| def test_enum_value_already_enum_passes_through(self): | ||
| result = _dict_to_dataclass( | ||
| {"filter": ExemplarFilter.trace_based}, _WithEnum | ||
| ) | ||
| self.assertIs(result.filter, ExemplarFilter.trace_based) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.