update task_resume_workflows to also resume processes in CREATED/RESUMED status #984

tjeerddie · 2025-07-02T14:56:10Z

improve start_process to include logic that happens in both executors.
move threadpool executor functions to its own file.

Related: #898

…MED status - improve start_process to include logic that happens in both executors. - move threadpool executor functions to its own file.

codspeed-hq · 2025-07-02T15:00:01Z

CodSpeed Performance Report

Merging #984 will not alter performance

_{Comparing 898-improve-worker-process (cd857b1) with main (a954ce7)}

Summary

✅ 12 untouched benchmarks

codecov · 2025-07-03T11:49:14Z

Codecov Report

Attention: Patch coverage is 65.47619% with 29 lines in your changes missing coverage. Please review.

Project coverage is 84.05%. Comparing base (3150434) to head (cd857b1).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
orchestrator/services/tasks.py	0.00%	13 Missing ⚠️
orchestrator/services/executors/celery.py	0.00%	7 Missing ⚠️
orchestrator/services/processes.py	44.44%	3 Missing and 2 partials ⚠️
orchestrator/services/executors/threadpool.py	91.30%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #984      +/-   ##
==========================================
- Coverage   84.37%   84.05%   -0.33%     
==========================================
  Files         213      214       +1     
  Lines       10275    10302      +27     
  Branches     1009     1011       +2     
==========================================
- Hits         8670     8659      -11     
- Misses       1337     1372      +35     
- Partials      268      271       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Mark90 · 2025-07-07T14:20:07Z

docs/reference-docs/app/celery-flow.png

With drawio you can export a .png file and embed the drawio diagram in the same file; you can use that here as a celery-flow.drawio.png

Mark90 · 2025-07-07T14:25:44Z

orchestrator/services/tasks.py

@@ -118,17 +95,15 @@ def resume_process(process_id: UUID, user_inputs: list[State] | None, user: str)

    celery_task = partial(celery.task, log=local_logger, serializer="orchestrator-json")

-    @celery_task(name=NEW_TASK)  # type: ignore
-    def new_task(process_id, workflow_key: str, user: str) -> UUID | None:
+    @celery_task(name=NEW_TASK, time_limit=1)  # type: ignore


1 second time limit?

woops

Suggested change

@celery_task(name=NEW_TASK, time_limit=1) # type: ignore

@celery_task(name=NEW_TASK) # type: ignore

Mark90 · 2025-07-08T08:30:30Z

docs/reference-docs/app/scaling.md

+
+### Celery Workflow/Task flow
+
+![Celery Workflow/Task flow](celery-flow.png)


Can you add a brief summary about the diagram? This helps providing documentation readers with some context, and allows people to find it through search keywords.

I think this is the intended situation after this PR is merged?

Mark90 · 2025-07-08T09:10:00Z

orchestrator/services/processes.py

@@ -460,19 +460,6 @@ def create_process(
    return pstat


-def thread_start_process(


Can you add a line to the PR description that these functions were moved from orchestrator.service.processes into the executor folders?

Mark90 · 2025-07-08T09:17:33Z

orchestrator/services/processes.py

@@ -567,6 +509,9 @@ def resume_process(
    """
    pstat = load_process(process)

+    if pstat.workflow == removed_workflow:
+        raise ValueError("This workflow cannot be resumed")


Suggested change

raise ValueError("This workflow cannot be resumed")

raise ValueError("This workflow cannot be resumed because it has been removed")

Mark90 · 2025-07-08T13:54:09Z

orchestrator/services/executors/threadpool.py

+    broadcast_func: BroadcastFunc | None = None,
+) -> UUID:
+    if pstat.workflow == removed_workflow:
+        raise ValueError("This workflow cannot be started")


Suggested change

raise ValueError("This workflow cannot be started")

raise ValueError("This workflow cannot be started because it has been removed")

Mark90 · 2025-07-10T06:34:20Z

orchestrator/services/tasks.py

        local_logger.info("Start task", process_id=process_id, workflow_key=workflow_key)
-        state = retrieve_input_state(process_id, "initial_state").input_state
-        return start_process(process_id, workflow_key, state=state, user=user)
+        return start_process(process_id, user=user)


The initial_state is never retrieved anymore. Is this intentional? I don't see where we are currently storing the user_inputs

Mark90 · 2025-07-10T09:02:05Z

orchestrator/services/executors/celery.py

@@ -80,7 +67,7 @@ def _celery_resume_process(
    user_inputs: list[State] | None = None,
    user: str | None = None,
    **kwargs: Any,
-) -> UUID:
+) -> bool:


Now that we're revisiting things; this function calls:
store_input_state(pstat.process_id, user_inputs, "user_input")

Could you move this to resume_process() in the processes.py file? I think it should be done there after the post_form validation, with a if user_inputs: check to prevent inserting empty input.

This would make it consistent with how it's done for new processes (start_process() calls create_process() which then does store_input_state(process_id, state | initial_state, "initial_state")).

Mark90 · 2025-07-10T09:20:33Z

orchestrator/workflows/tasks/resume_workflows.py

+    created_process_ids = get_process_ids_by_process_statuses([ProcessStatus.CREATED])
+    resumed_process_ids = get_process_ids_by_process_statuses([ProcessStatus.RESUMED])


I'm not completely sure if this is the way to go.

Imagine this scenario:

There are separate queues for 1) new workflows (status CREATED) 2) resumed workflows (status RESUMED)

Each queue has 50 workflows queued

For each queue there are 5 workers churning through queued workflows which they have put on status RUNNING

When task_resume_workflows runs it finds 45 processes on CREATED and 45 on RESUMED so it calls resume_process() for each of them

Unless I'm missing something, this would re-queue 90 processes that don't need to be re-queued.

In thread_start_process() you already changed it to set the workflow to RUNNING before executing any step. This means that chance of the celery worker going AWOL before setting it to RUNNING has become much smaller, so that is already much better. If a worker dies when a process is running, it can be resumed by an engine restart.

So, I don't think we should resume created/resumed workflows in this task.
What do you think?

Mark90 · 2025-07-10T09:36:53Z

orchestrator/services/processes.py

-    return thread_start_process(validation_workflow, user_inputs=json)
-
-
-THREADPOOL_EXECUTION_CONTEXT: dict[str, Callable] = {


Maybe the PR should mention that these dicts were moved. And that the signature of the functions has changed. Just in case someone is using them directly

update task_resume_workflows to also resume processes in CREATED/RESU…

1ac7b2d

…MED status - improve start_process to include logic that happens in both executors. - move threadpool executor functions to its own file.

tjeerddie force-pushed the 898-improve-worker-process branch from ff408fb to 1ac7b2d Compare July 2, 2025 14:57

Fix unit tests

cd857b1

kvklink approved these changes Jul 3, 2025

View reviewed changes

Mark90 self-requested a review July 9, 2025 08:28

Mark90 requested changes Jul 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

update task_resume_workflows to also resume processes in CREATED/RESUMED status #984

update task_resume_workflows to also resume processes in CREATED/RESUMED status #984

tjeerddie commented Jul 2, 2025

Uh oh!

codspeed-hq bot commented Jul 2, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 3, 2025

Uh oh!

Mark90 Jul 7, 2025

Uh oh!

Mark90 Jul 7, 2025

Uh oh!

tjeerddie Jul 10, 2025

Uh oh!

Mark90 Jul 8, 2025

Uh oh!

Mark90 Jul 8, 2025

Uh oh!

Mark90 Jul 8, 2025

Uh oh!

Mark90 Jul 8, 2025

Uh oh!

Mark90 Jul 10, 2025

Uh oh!

Mark90 Jul 10, 2025

Uh oh!

Mark90 Jul 10, 2025

Uh oh!

Mark90 Jul 10, 2025

Uh oh!

Uh oh!

	@celery_task(name=NEW_TASK, time_limit=1) # type: ignore
	@celery_task(name=NEW_TASK) # type: ignore


		### Celery Workflow/Task flow

		![Celery Workflow/Task flow](celery-flow.png)

		@@ -460,19 +460,6 @@ def create_process(
		return pstat


		def thread_start_process(

	raise ValueError("This workflow cannot be resumed")
	raise ValueError("This workflow cannot be resumed because it has been removed")

		created_process_ids = get_process_ids_by_process_statuses([ProcessStatus.CREATED])
		resumed_process_ids = get_process_ids_by_process_statuses([ProcessStatus.RESUMED])

		return thread_start_process(validation_workflow, user_inputs=json)


		THREADPOOL_EXECUTION_CONTEXT: dict[str, Callable] = {

update task_resume_workflows to also resume processes in CREATED/RESUMED status #984

Are you sure you want to change the base?

update task_resume_workflows to also resume processes in CREATED/RESUMED status #984

Conversation

tjeerddie commented Jul 2, 2025

Uh oh!

codspeed-hq bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #984 will not alter performance

Summary

Uh oh!

codecov bot commented Jul 3, 2025

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq bot commented Jul 2, 2025 •

edited

Loading