Oracle ingestion fails after upgrading to DataHub 1.3: TaskError setting up virtual environment

[vivafung]

Describe the bug
After upgrading to 1.3, we got an error while ingesting oracle : which is
"acryl.executor.execution.task.TaskError: Failed to set up virtual environment: Command '['/usr/bin/uv', 'pip', 'install', '-r', "
"'/tmp/datahub/ingest/58f10f86-f290-4041-aa75-30c77d219e6c/venv-oracle-d77916d56bd699e1/requirements.txt']' returned non-zero exit status "

datahub is running on aws eks. I logged into pods and run above cmd :
/usr/bin/uv pip install -r /tmp/datahub/ingest/d34611be-f417-45e9-bc74-c03b87c6f102/venv-oracle-d77916d56bd699e1/requirements.txt

it worked without issue. but just not working while running ingestion via GUI

Datahub version : 1.3.0
Helm chart version : 0.6.16

Issue CAN NOT be reproduced on v1.0.0

Oracle ingestion in v1.2.0 has a different problem which is : #14191 , so we had to upgrade to 1.3.0

Recipe is straightforward , see below :

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[zPhenomm]

I have the exact same problem with Iceberg ingestions in 1.3.0 (Helm chart version 0.0.7). Pipeline fails when trying to create the Python venv. I did not have that problem with 1.2.0.1.

As previously mentioned, manually installing from the actions pod works:

But via the GUI it fails

Full logs from the ingestion pipeline:

Execution finished with errors.
{'exec_id': '5ad55a42-7d1c-4f2e-adf5-0266d5a07503',
 'infos': ['2025-10-14 10:18:06.608657 INFO: Starting execution for task with name=RUN_INGEST',
           '2025-10-14 10:18:09.703486 INFO: Caught exception EXECUTING task_id=5ad55a42-7d1c-4f2e-adf5-0266d5a07503, name=RUN_INGEST, stacktrace=  '
           '+ Exception Group Traceback (most recent call last):\n'
           '  |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 278, in collapse_excgroups\n'
           '  |     yield\n'
           '  |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 318, in execute\n'
           '  |     async with self._process, anyio.create_task_group() as tg:\n'
           '  |   File "/home/datahub/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 781, in __aexit__\n'
           '  |     raise BaseExceptionGroup(\n'
           '  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)\n'
           '  +-+---------------- 1 ----------------\n'
           '    | Traceback (most recent call last):\n'
           '    |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 246, in '
           '_setup_venv\n'
           '    |     venv_ref = await setup_venv(\n'
           '    |                ^^^^^^^^^^^^^^^^^\n'
           '    |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 498, in setup_venv\n'
           '    |     await runner.execute(\n'
           '    |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 317, in execute\n'
           '    |     with collapse_excgroups():\n'
           '    |   File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__\n'
           '    |     self.gen.throw(typ, value, traceback)\n'
           '    |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 284, in collapse_excgroups\n'
           '    |     raise exc\n'
           '    |   File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 338, in execute\n'
           '    |     raise subprocess.CalledProcessError(\n'
           "    | subprocess.CalledProcessError: Command '['/usr/bin/uv', 'pip', 'install', '-r', "
           "'/tmp/datahub/ingest/5ad55a42-7d1c-4f2e-adf5-0266d5a07503/venv-iceberg-48492ddf5c148a4c/requirements.txt']' returned non-zero exit "
           'status 2.\n'
           '    +------------------------------------\n'
           '\n'
           'During handling of the above exception, another exception occurred:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 246, in '
           '_setup_venv\n'
           '    venv_ref = await setup_venv(\n'
           '               ^^^^^^^^^^^^^^^^^\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 498, in setup_venv\n'
           '    await runner.execute(\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 317, in execute\n'
           '    with collapse_excgroups():\n'
           '  File "/usr/lib/python3.11/contextlib.py", line 158, in __exit__\n'
           '    self.gen.throw(typ, value, traceback)\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 284, in collapse_excgroups\n'
           '    raise exc\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/runner.py", line 338, in execute\n'
           '    raise subprocess.CalledProcessError(\n'
           "subprocess.CalledProcessError: Command '['/usr/bin/uv', 'pip', 'install', '-r', "
           "'/tmp/datahub/ingest/5ad55a42-7d1c-4f2e-adf5-0266d5a07503/venv-iceberg-48492ddf5c148a4c/requirements.txt']' returned non-zero exit "
           'status 2.\n'
           '\n'
           'The above exception was the direct cause of the following exception:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/default_executor.py", line 153, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete\n'
           '    return future.result()\n'
           '           ^^^^^^^^^^^^^^^\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 324, in execute\n'
           '    await self._execute_with_debug(validated_args, ctx, exec_id)\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 371, in '
           '_execute_with_debug\n'
           '    ingest_process = await self._create_subprocess(\n'
           '                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 274, in '
           '_create_subprocess\n'
           '    venv_ref = await self._setup_venv(\n'
           '               ^^^^^^^^^^^^^^^^^^^^^^^\n'
           '  File "/home/datahub/.venv/lib/python3.11/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 260, in '
           '_setup_venv\n'
           '    raise TaskError(f"Failed to set up virtual environment: {e}") from e\n'
           "acryl.executor.execution.task.TaskError: Failed to set up virtual environment: Command '['/usr/bin/uv', 'pip', 'install', '-r', "
           "'/tmp/datahub/ingest/5ad55a42-7d1c-4f2e-adf5-0266d5a07503/venv-iceberg-48492ddf5c148a4c/requirements.txt']' returned non-zero exit "
           'status 2.\n'],
 'errors': []}

~~~~ Ingestion Logs ~~~~

[deepgarg760]

@vivafung, Thanks for reporting!! Will surface this to our internal team to have look and will update you

[amidani-inovany]

I have the same issue with different ingestion sources, found this when I took a look at the pod's logs

There's a timeout when installing packages, in my case we need to use a proxy and it seems the created venv do not inherit global ENV vars such as HTTPS_PROXY.

Hope this will help resolving the issue faster

[treff7es]

@amidani-inovany can you set the custom env variable on the ingestion ui at the advanced section? -> https://docs.datahub.com/docs/ui-ingestion#advanced-configuration-options

[treff7es]

@vivafung can you check the datahub-actions pod logs when this happens? There, you should see why the virtualenv creation failed.

[amidani-inovany]

@amidani-inovany can you set the custom env variable on the ingestion ui at the advanced section? -> https://docs.datahub.com/docs/ui-ingestion#advanced-configuration-options

@treff7es thanks for the hint, works well as a temp workaround

[amidani-inovany]

I just got a new issue related to this, the PIP_GLOBAL_EXTRA_PACKAGES is not picked up by the created venv, in my case I have installed the package from a private repo and set the PIP_GLOBAL_EXTRA_PACKAGES in the Dokerfile, but the created venv does not pick up this, even setting the extra package through the UI has no effect, the package is not in the list of the installed packages when setting up the venv.

[deepgarg760]

@vivafung, did you get a chance to look into datahub-actions pod logs as @treff7es suggested ?

[amidani-inovany]

As a work around I downgraded acryl-executor when building datahug-actions image, I added this in the Dockerfile:

RUN uv pip install acryl-executor==0.2.2