Skip to content

Commit 565af3e

Browse files
authored
Merge branch 'main' into andystaples-improve-ci-validation
2 parents 56fce4e + 6a52e1d commit 565af3e

9 files changed

Lines changed: 812 additions & 37 deletions

File tree

CHANGELOG.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,16 @@ ADDED
1414

1515
CHANGED
1616

17+
- Importing `durabletask` no longer eagerly imports the worker implementation and its
18+
dependencies (gRPC, protobuf, entities, serialization, OpenTelemetry). The public names
19+
re-exported from the package — `ActivityWorkItemFilter`, `ConcurrencyOptions`,
20+
`EntityWorkItemFilter`, `GrpcChannelOptions`, `GrpcRetryPolicyOptions`,
21+
`LargePayloadStorageOptions`, `OrchestrationWorkItemFilter`, `PayloadStore`,
22+
`VersioningOptions`, and `WorkItemFilters` — are now resolved on first use, so
23+
`import durabletask` is substantially faster and loads far fewer modules. This
24+
measurably reduces cold-start time for client-only applications, including those using
25+
`durabletask.azuremanaged`, which shares the same `durabletask` namespace. All existing
26+
import paths, `__all__`, `dir()`, and star-imports behave exactly as before.
1727
- **Breaking:** `FailureDetails.error_type` — and the `errorType` value sent over the wire — is now the fully-qualified type name (`module.ClassName`, e.g. `builtins.ValueError`, `durabletask.task.TaskFailedError`) instead of the bare class name, matching the .NET and Java SDKs. Code that compared `error_type` against a bare name (for example `== "ValueError"`) must be updated to the qualified name or, preferably, switched to `FailureDetails.is_caused_by()`. Because this value is persisted and crosses the orchestration boundary, failures produced by older workers may still carry a bare name; `is_caused_by()` accepts both.
1828
- **Breaking:** Retired the process-global history-export context. `durabletask.extensions.history_export.bind_context()` and `clear_context()` have been removed; the export activities now resolve their `HistoryExportContext` per invocation via a resolver captured at registration. `ExportHistoryClient.register_worker()` continues to wire this up automatically, so most callers are unaffected. Code that called `bind_context(HistoryExportContext(client, writer))` directly should instead register the activities with `durabletask.extensions.history_export.activities.register(worker, lambda: HistoryExportContext(client, writer))` (or a lazier resolver), or use `ExportHistoryClient.register_worker()`.
1929

durabletask-azuremanaged/CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,10 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
1313
result, credential failures (for example an unavailable managed identity or a misconfigured
1414
`DefaultAzureCredential`) now surface from the first request rather than from the constructor.
1515
The exception type and message are unchanged; only the timing differs.
16+
- Importing `durabletask.azuremanaged.preview.sandboxes` no longer loads the sandbox worker
17+
runtime or `azure-identity` up front. The package's public names are now resolved on first
18+
use, which roughly halves import cost for callers that only declare sandbox worker profiles
19+
or use the sandbox client. Exported names, `__all__`, and import paths are unchanged.
1620
- `FailureDetails.error_type` now carries the fully-qualified type name (e.g. `durabletask.task.TaskFailedError`) instead of the bare class name, and the new `FailureDetails.is_caused_by()` helper is available (both inherited from durabletask). See the core `durabletask` changelog for details, including the breaking-change notes.
1721
- Improved async access token refresh concurrency handling to avoid duplicate
1822
refresh operations under concurrent access, matching the existing sync

durabletask-azuremanaged/durabletask/azuremanaged/internal/durabletask_grpc_interceptor.py

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Copyright (c) Microsoft Corporation.
22
# Licensed under the MIT License.
33

4+
from functools import lru_cache
45
from importlib.metadata import version
56

67
import grpc
@@ -17,6 +18,22 @@
1718
)
1819

1920

21+
@lru_cache(maxsize=1)
22+
def _get_sdk_version() -> str:
23+
"""Return the installed version of the azuremanaged package.
24+
25+
Resolving the version walks distribution metadata on disk, so the result is
26+
cached and shared by every interceptor instance instead of being recomputed
27+
on each client or worker construction. Falls back to ``"unknown"`` when the
28+
version cannot be determined.
29+
"""
30+
try:
31+
return version('durabletask-azuremanaged')
32+
except Exception:
33+
# Fallback if version cannot be determined
34+
return "unknown"
35+
36+
2037
class DTSDefaultClientInterceptorImpl (DefaultClientInterceptorImpl):
2138
"""The class implements a UnaryUnaryClientInterceptor, UnaryStreamClientInterceptor,
2239
StreamUnaryClientInterceptor and StreamStreamClientInterceptor from grpc to add an
@@ -27,13 +44,7 @@ def __init__(
2744
token_credential: TokenCredential | None,
2845
taskhub_name: str,
2946
worker_id: str | None = None):
30-
try:
31-
# Get the version of the azuremanaged package
32-
sdk_version = version('durabletask-azuremanaged')
33-
except Exception:
34-
# Fallback if version cannot be determined
35-
sdk_version = "unknown"
36-
user_agent = f"durabletask-python/{sdk_version}"
47+
user_agent = f"durabletask-python/{_get_sdk_version()}"
3748
self._metadata = [
3849
("taskhub", taskhub_name),
3950
("x-user-agent", user_agent)] # 'user-agent' is a reserved header; use 'x-user-agent'
@@ -81,13 +92,7 @@ class DTSAsyncDefaultClientInterceptorImpl(DefaultAsyncClientInterceptorImpl):
8192
(task hub name, user agent, and authentication token) to all async calls."""
8293

8394
def __init__(self, token_credential: AsyncTokenCredential | None, taskhub_name: str):
84-
try:
85-
# Get the version of the azuremanaged package
86-
sdk_version = version('durabletask-azuremanaged')
87-
except Exception:
88-
# Fallback if version cannot be determined
89-
sdk_version = "unknown"
90-
user_agent = f"durabletask-python/{sdk_version}"
95+
user_agent = f"durabletask-python/{_get_sdk_version()}"
9196
self._metadata = [
9297
("taskhub", taskhub_name),
9398
("x-user-agent", user_agent)]

durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/__init__.py

Lines changed: 54 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,22 @@
1313
SandboxWorker,
1414
SandboxActivitiesClient,
1515
)
16+
17+
The exports below are resolved lazily on first attribute access, so importing
18+
this package does not load the sandbox worker runtime (and its Azure Identity
19+
and gRPC dependencies) unless those APIs are actually used.
1620
"""
1721

18-
from durabletask.azuremanaged.preview.sandboxes.client import SandboxActivitiesClient
19-
from durabletask.azuremanaged.preview.sandboxes.helpers import SandboxActivity
20-
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import SandboxWorkerProfile
21-
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import SandboxWorkerProfileOptions
22-
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import sandbox_worker_profile
23-
from durabletask.azuremanaged.preview.sandboxes.worker import SandboxWorker
22+
from importlib import import_module
23+
from typing import TYPE_CHECKING, Any
24+
25+
if TYPE_CHECKING:
26+
from durabletask.azuremanaged.preview.sandboxes.client import SandboxActivitiesClient
27+
from durabletask.azuremanaged.preview.sandboxes.helpers import SandboxActivity
28+
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import SandboxWorkerProfile
29+
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import SandboxWorkerProfileOptions
30+
from durabletask.azuremanaged.preview.sandboxes.worker_profiles import sandbox_worker_profile
31+
from durabletask.azuremanaged.preview.sandboxes.worker import SandboxWorker
2432

2533
__all__ = [
2634
"SandboxWorker",
@@ -30,3 +38,43 @@
3038
"SandboxActivitiesClient",
3139
"sandbox_worker_profile",
3240
]
41+
42+
# Public export name -> submodule of this package that defines it.
43+
_LAZY_EXPORTS: dict[str, str] = {
44+
"SandboxWorker": "worker",
45+
"SandboxActivity": "helpers",
46+
"SandboxWorkerProfile": "worker_profiles",
47+
"SandboxWorkerProfileOptions": "worker_profiles",
48+
"SandboxActivitiesClient": "client",
49+
"sandbox_worker_profile": "worker_profiles",
50+
}
51+
52+
# Submodules that eager imports previously bound as attributes of this package.
53+
# They remain reachable through attribute access without an explicit import.
54+
_LAZY_SUBMODULES: frozenset[str] = frozenset({
55+
"client",
56+
"helpers",
57+
"profile_builder",
58+
"transport",
59+
"worker",
60+
"worker_messages",
61+
"worker_profiles",
62+
})
63+
64+
65+
def __getattr__(name: str) -> Any:
66+
"""Import public sandbox exports on first access (PEP 562)."""
67+
submodule = _LAZY_EXPORTS.get(name)
68+
if submodule is not None:
69+
value = getattr(import_module(f".{submodule}", __name__), name)
70+
elif name in _LAZY_SUBMODULES:
71+
value = import_module(f".{name}", __name__)
72+
else:
73+
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
74+
75+
globals()[name] = value
76+
return value
77+
78+
79+
def __dir__() -> list[str]:
80+
return sorted(set(globals()) | set(__all__) | _LAZY_SUBMODULES)

durabletask-azuremanaged/durabletask/azuremanaged/preview/sandboxes/profile_builder.py

Lines changed: 92 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,6 @@
77
from durabletask.azuremanaged.internal import sandbox_service_pb2 as pb
88
from durabletask.azuremanaged.preview.sandboxes.helpers import (
99
SandboxActivity,
10-
activities_overlap,
1110
format_activity,
1211
normalize_required,
1312
resolve_activities,
@@ -91,22 +90,109 @@ def _build_sandbox_worker_profile(
9190
return worker_profile
9291

9392

93+
class _ActivityOwnerSlot:
94+
"""Earliest worker profiles recorded for a single activity overlap bucket.
95+
96+
Only two owners ever need to be retained to answer "which worker profile
97+
first claimed an activity in this bucket, ignoring `worker_profile_id`":
98+
the very first owner, plus the first owner that differs from it.
99+
"""
100+
101+
__slots__ = ("_first", "_first_other")
102+
103+
def __init__(self) -> None:
104+
self._first: Optional[tuple[int, str]] = None
105+
self._first_other: Optional[tuple[int, str]] = None
106+
107+
def add(self, order: int, worker_profile_id: str) -> None:
108+
if self._first is None:
109+
self._first = (order, worker_profile_id)
110+
elif self._first_other is None and worker_profile_id != self._first[1]:
111+
self._first_other = (order, worker_profile_id)
112+
113+
def first_owner_other_than(self, worker_profile_id: str) -> Optional[tuple[int, str]]:
114+
if self._first is not None and self._first[1] != worker_profile_id:
115+
return self._first
116+
return self._first_other
117+
118+
119+
class _ActivityOwnerIndex:
120+
"""Indexes activity ownership so overlap checks cost O(1) per activity.
121+
122+
Reproduces the semantics of
123+
:func:`durabletask.azuremanaged.preview.sandboxes.helpers.activities_overlap`
124+
exactly: activity names are compared case insensitively, an unversioned
125+
activity overlaps every version of the same name, and two activities with
126+
the same name overlap when their explicit versions are equal. Registration
127+
order is tracked so the reported conflict matches the first overlapping
128+
activity, exactly as a linear scan would report it.
129+
"""
130+
131+
def __init__(self) -> None:
132+
self._registration_count = 0
133+
self._by_name: dict[str, _ActivityOwnerSlot] = {}
134+
self._by_name_and_version: dict[tuple[str, Optional[str]], _ActivityOwnerSlot] = {}
135+
136+
def find_conflicting_profile(
137+
self,
138+
activity: SandboxActivity,
139+
worker_profile_id: str) -> Optional[str]:
140+
"""Return the profile that first claimed an overlapping activity, if any."""
141+
name_key = activity.name.casefold()
142+
if activity.version is None:
143+
# An unversioned activity overlaps every version of the same name.
144+
owner = _first_owner_other_than(self._by_name.get(name_key), worker_profile_id)
145+
else:
146+
# A versioned activity overlaps the same version plus any
147+
# unversioned registration of the same name.
148+
owner = _earlier_owner(
149+
_first_owner_other_than(
150+
self._by_name_and_version.get((name_key, None)), worker_profile_id),
151+
_first_owner_other_than(
152+
self._by_name_and_version.get((name_key, activity.version)), worker_profile_id))
153+
return None if owner is None else owner[1]
154+
155+
def add(self, activity: SandboxActivity, worker_profile_id: str) -> None:
156+
"""Record `worker_profile_id` as an owner of `activity`."""
157+
name_key = activity.name.casefold()
158+
order = self._registration_count
159+
self._registration_count += 1
160+
self._by_name.setdefault(name_key, _ActivityOwnerSlot()).add(order, worker_profile_id)
161+
self._by_name_and_version.setdefault(
162+
(name_key, activity.version), _ActivityOwnerSlot()).add(order, worker_profile_id)
163+
164+
165+
def _first_owner_other_than(
166+
slot: Optional[_ActivityOwnerSlot],
167+
worker_profile_id: str) -> Optional[tuple[int, str]]:
168+
return None if slot is None else slot.first_owner_other_than(worker_profile_id)
169+
170+
171+
def _earlier_owner(
172+
left: Optional[tuple[int, str]],
173+
right: Optional[tuple[int, str]]) -> Optional[tuple[int, str]]:
174+
if left is None:
175+
return right
176+
if right is None:
177+
return left
178+
return left if left[0] <= right[0] else right
179+
180+
94181
def build_sandbox_worker_profiles() -> list[pb.SandboxWorkerProfile]:
95182
"""Build sandbox worker_profiles from worker profile configuration."""
96183
worker_profiles: list[pb.SandboxWorkerProfile] = []
97-
activity_owners: list[tuple[SandboxActivity, str]] = []
184+
activity_owners = _ActivityOwnerIndex()
98185
for profile in registered_sandbox_worker_profiles():
99186
activities = resolve_activities(profile.activities)
100187

101188
for activity in activities:
102-
existing_profile = next((owner_profile for owner_activity, owner_profile in activity_owners
103-
if activities_overlap(owner_activity, activity)
104-
and owner_profile != profile.worker_profile_id), None)
189+
existing_profile = activity_owners.find_conflicting_profile(
190+
activity, profile.worker_profile_id)
105191
if existing_profile:
106192
raise ValueError(
107193
f"Sandbox activity '{format_activity(activity)}' is assigned to both worker profile "
108194
f"'{existing_profile}' and '{profile.worker_profile_id}'.")
109-
activity_owners.append((activity, profile.worker_profile_id))
195+
activity_owners.add(activity, profile.worker_profile_id)
110196

111197
worker_profiles.append(_build_sandbox_worker_profile(
112198
activities=activities,

durabletask/__init__.py

Lines changed: 70 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,16 +3,19 @@
33

44
"""Durable Task SDK for Python"""
55

6-
from durabletask.grpc_options import GrpcChannelOptions, GrpcRetryPolicyOptions
7-
from durabletask.payload.store import LargePayloadStorageOptions, PayloadStore
8-
from durabletask.worker import (
9-
ActivityWorkItemFilter,
10-
ConcurrencyOptions,
11-
EntityWorkItemFilter,
12-
OrchestrationWorkItemFilter,
13-
VersioningOptions,
14-
WorkItemFilters,
15-
)
6+
from typing import TYPE_CHECKING, Any
7+
8+
if TYPE_CHECKING:
9+
from durabletask.grpc_options import GrpcChannelOptions, GrpcRetryPolicyOptions
10+
from durabletask.payload.store import LargePayloadStorageOptions, PayloadStore
11+
from durabletask.worker import (
12+
ActivityWorkItemFilter,
13+
ConcurrencyOptions,
14+
EntityWorkItemFilter,
15+
OrchestrationWorkItemFilter,
16+
VersioningOptions,
17+
WorkItemFilters,
18+
)
1619

1720
__all__ = [
1821
"ActivityWorkItemFilter",
@@ -28,3 +31,60 @@
2831
]
2932

3033
PACKAGE_NAME = "durabletask"
34+
35+
# Public names are resolved lazily so that merely importing the ``durabletask``
36+
# package - which also happens implicitly when importing anything from the
37+
# ``durabletask.azuremanaged`` distribution, since both share this namespace -
38+
# does not pull in the worker dependency graph (gRPC, protobuf, entities,
39+
# serialization, OpenTelemetry). Client-only applications would otherwise pay
40+
# that cost at process startup.
41+
_LAZY_EXPORTS: dict[str, str] = {
42+
"ActivityWorkItemFilter": "durabletask.worker",
43+
"ConcurrencyOptions": "durabletask.worker",
44+
"EntityWorkItemFilter": "durabletask.worker",
45+
"GrpcChannelOptions": "durabletask.grpc_options",
46+
"GrpcRetryPolicyOptions": "durabletask.grpc_options",
47+
"LargePayloadStorageOptions": "durabletask.payload.store",
48+
"OrchestrationWorkItemFilter": "durabletask.worker",
49+
"PayloadStore": "durabletask.payload.store",
50+
"VersioningOptions": "durabletask.worker",
51+
"WorkItemFilters": "durabletask.worker",
52+
}
53+
54+
# Submodules that the previous eager imports bound as attributes of this package
55+
# as a side effect. Attribute access keeps them reachable without an explicit
56+
# ``import durabletask.<name>``, exactly as before, but they are now imported
57+
# only when actually touched. This list mirrors the old surface and must not be
58+
# extended: ``durabletask.client``, for instance, was never bound this way.
59+
_LAZY_SUBMODULES: frozenset[str] = frozenset({
60+
"entities",
61+
"grpc_options",
62+
"internal",
63+
"payload",
64+
"serialization",
65+
"task",
66+
"worker",
67+
})
68+
69+
70+
def __getattr__(name: str) -> Any:
71+
"""Import and return a lazily exported public name or submodule (PEP 562)."""
72+
module_name = _LAZY_EXPORTS.get(name)
73+
if module_name is None and name not in _LAZY_SUBMODULES:
74+
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
75+
76+
from importlib import import_module
77+
78+
if module_name is not None:
79+
value = getattr(import_module(module_name), name)
80+
else:
81+
value = import_module(f".{name}", __name__)
82+
83+
# Cache on the module so subsequent lookups bypass this hook entirely.
84+
globals()[name] = value
85+
return value
86+
87+
88+
def __dir__() -> list[str]:
89+
"""Include lazily exported names that have not been resolved yet."""
90+
return sorted(set(globals()) | set(__all__) | _LAZY_SUBMODULES)

0 commit comments

Comments
 (0)