update task_resume_workflows to also resume processes in CREATED/RESUMED status

[tjeerddie]

• improve start_process to include logic that happens in both executors.
• Change process status updates to be done within a transaction to lock the process in DB in executor functions to prevent race conditions:
  • change _celery_set_process_status_resumed to a transaction to lock process in the DB, preventing more then one worker to pick up the process.
    • also use can_be_resumed to check that only correct statuses can be changed.
  • add _set_process_status_running to thread_start_process and thread_resume_process to update process status to RUNNING within a transaction and make sure that its status isn't RUNNING (picked up by another worker).
• move threadpool executor functions to its own file.
  • celery: orchestrator/services/celery.py -> orchestrator/services/executors/celery.py
  • thread: orchestrator/services/processes.py -> orchestrator/services/executors/threadpool.py
    • thread_start_process
    • thread_resume_process
    • thread_validate_workflow
    • THREADPOOL_EXECUTION_CONTEXT
• move can_be_resumed from api/processes.py to services/processes.py.
• retrieve input state in thread_start_process and thread_resume_process to be able to restart the processes when the process gets stuck in CREATED or RESUMED right after initial form or form step.
• improve workflow removed error messages.

updated testing:

• add unit tests for threadpool and celery executor functions.
• add unit tests for resume_process.
• improve unit tests for start_process.
• add 2 tests that validate the happy flow of start_process and resume_process.
• change api resume workflow with incorrect status test to check all incorrect statuses.
• update task_resume_workflow tests.

Related: #898

[codspeed-hq]

CodSpeed Performance Report

Merging #984 will not alter performance

_{Comparing 898-improve-worker-process (5f6e9bb) with main (0153fdb)}

Summary

✅ 12 untouched benchmarks

[codecov]

Codecov Report

❌ Patch coverage is 80.53691% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.90%. Comparing base (757fe18) to head (5f6e9bb).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
orchestrator/services/tasks.py	17.64%	14 Missing ⚠️
orchestrator/services/executors/celery.py	79.16%	3 Missing and 2 partials ⚠️
orchestrator/services/processes.py	82.60%	4 Missing ⚠️
orchestrator/workflows/tasks/resume_workflows.py	86.66%	3 Missing and 1 partial ⚠️
orchestrator/services/input_state.py	50.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #984      +/-   ##
==========================================
+ Coverage   84.42%   84.90%   +0.48%     
==========================================
  Files         213      214       +1     
  Lines       10354    10390      +36     
  Branches     1016     1020       +4     
==========================================
+ Hits         8741     8822      +81     
+ Misses       1344     1295      -49     
- Partials      269      273       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Mark90]

LGTM, 3 small nitpicks (sorry) and looks like the branch is out of date.

Great work! 😄