Background
In the AIP-108 dev@ thread (2026-05-13 reply), Maciej
Obuchowski outlined what OpenLineage needs from the Java SDK:
- Generic task lifecycle events — when OL is enabled, every task
execution emits OL start / complete / failure events via the listener
framework, driven by the Python task runner. For Java tasks, the listener
calls still fire from the Python side around the Java subprocess (e.g.
near on_task_instance_running / on_task_instance_success in
task-sdk/src/airflow/sdk/execution_time/task_runner.py:1196 and
:1921). No code change is required to keep this working for the
"task ran, task succeeded" signal.
- Operator/hook-specific lineage data — the part that currently relies
on Python operators/hooks being in-process and reading state from the
TaskInstance after execute(). This does not work for a Java task
because the user code runs in a JVM subprocess that the Python listener
cannot introspect.
Maciej's conclusion: v1 of the Java SDK does not need OL emission from
inside the Java task, but the IPC and Java-side API must not block a
future lineage interface from being added. Concretely:
"able to send serialized data back from the task execution to Python;
and an API in the Java SDK for users to be able to specify that data."
What needs to happen
- Reserve a lineage channel on the coordinator IPC. When the Java
subprocess returns task results to the supervisor, the protocol should
allow an optional serialized lineage payload alongside the existing
result message. The base BaseCoordinator interface needs to expose a
hook the supervisor calls with that payload (no-op by default).
- Expose a Java SDK API for users to declare lineage data. Minimal
shape, mirroring how Python tasks can attach lineage to a TaskInstance
today. Exact API to be decided once the IPC channel exists, but it
should be:
- opt-in (no overhead for tasks that don't use it),
- language-idiomatic on the Java side,
- resolvable to whatever serialized form the Python listener expects.
- Wire the payload into the existing listener pipeline. The Python
supervisor should forward the serialized lineage data into the same
get_listener_manager().hook.* call chain that already runs around
Java task execution, so OL providers don't need any Java-specific code
path.
- Document the v1 boundary clearly. The coordinator user guide should
state that OL start/complete events fire for Java tasks today, but
Java-side lineage extraction is a follow-up.
Acceptance criteria
- A user enabling OpenLineage sees start/complete events for Java stub
tasks the same way as Python tasks (no regression from current
behavior).
- The coordinator IPC has a documented optional lineage field; a Java
task that does not emit lineage produces exactly the same wire traffic
as today.
- A Java SDK user can attach lineage data from a task and see it land in
the Python supervisor as a serialized payload available to OL listeners.
- The OL provider's listener does not need a Java-specific branch — the
Java payload reaches it through the same listener hooks as Python.
- The coordinator user guide states the v1 boundary and links to this
issue for follow-up work.
Context
Background
In the AIP-108 dev@ thread (2026-05-13 reply), Maciej
Obuchowski outlined what OpenLineage needs from the Java SDK:
execution emits OL start / complete / failure events via the listener
framework, driven by the Python task runner. For Java tasks, the listener
calls still fire from the Python side around the Java subprocess (e.g.
near
on_task_instance_running/on_task_instance_successintask-sdk/src/airflow/sdk/execution_time/task_runner.py:1196and:1921). No code change is required to keep this working for the"task ran, task succeeded" signal.
on Python operators/hooks being in-process and reading state from the
TaskInstanceafterexecute(). This does not work for a Java taskbecause the user code runs in a JVM subprocess that the Python listener
cannot introspect.
Maciej's conclusion: v1 of the Java SDK does not need OL emission from
inside the Java task, but the IPC and Java-side API must not block a
future lineage interface from being added. Concretely:
What needs to happen
subprocess returns task results to the supervisor, the protocol should
allow an optional serialized lineage payload alongside the existing
result message. The base
BaseCoordinatorinterface needs to expose ahook the supervisor calls with that payload (no-op by default).
shape, mirroring how Python tasks can attach lineage to a
TaskInstancetoday. Exact API to be decided once the IPC channel exists, but it
should be:
supervisor should forward the serialized lineage data into the same
get_listener_manager().hook.*call chain that already runs aroundJava task execution, so OL providers don't need any Java-specific code
path.
state that OL start/complete events fire for Java tasks today, but
Java-side lineage extraction is a follow-up.
Acceptance criteria
tasks the same way as Python tasks (no regression from current
behavior).
task that does not emit lineage produces exactly the same wire traffic
as today.
the Python supervisor as a serialized payload available to OL listeners.
Java payload reaches it through the same listener hooks as Python.
issue for follow-up work.
Context
— Maciej Obuchowski reply on 2026-05-13 ("Generic task information and
specific lineage data ..."). Jarek Potiuk's prior message on the same
day pinged Maciej and Kacper for OL input.
(Coordinator layer).
the mechanism (or share machinery) for shipping lineage data back to
Python.
BaseCoordinatorinterface to support pluggable communication channels #66838 — Pluggable communication channels. The IPC reservationin step 1 should be designed against whatever shape
BaseCoordinatorsettles on there.
optional lineage field needs to fit the forward-compat contract being
defined there.