infra: propagate custom timeouts to Cloud Workflows preprocessing and postprocessing steps

[dwnoble]

This change exposes and propagates user-defined timeouts (ingestion_preprocessing_job_timeout and ingestion_postprocessing_job_timeout) from the stack infrastructure variables down to the Google Cloud Workflows orchestrator steps.

Problem

Large dataset ingestions can exceed the default 30-minute (1800s) Google Cloud Workflows connector polling limit for long-running operations (LROs). Although custom timeouts are defined in the stack configuration, they were not being forwarded to the workflow definition. Consequently, the workflow orchestrator prematurely timed out with a TimeoutError during operation status polling, even when the underlying Cloud Run jobs were running correctly.

Solution

• Variable Propagation: Passed the timeout parameters down from module.stack to module.ingestion_workflow.
• Template Mapping: Added timeout variables to the workflow YAML templatefile inside modules/ingestion/workflow/main.tf.
• Workflow Integration: Configured the timeout parameter inside the connector_params block for both run_cloud_run_job (preprocessing) and run_cloud_run_agg_job (postprocessing) steps in workflow.yaml.
• Increased Defaults: This changes the effective default workflow polling timeout from 30 minutes (1800s) to 6 hours (21600s), aligned with the default timeout variables defined in the stack configuration.

[gemini-code-assist]

Code Review

This pull request introduces configurable timeouts for preprocessing and postprocessing Cloud Run jobs within the ingestion workflow. It adds variables for these timeouts, integrates them into the workflow definition, and parses them in the stack module. The review feedback suggests wrapping the timeout parsing logic in try() blocks to prevent Terraform evaluation errors if the timeout values are null or empty.

[gemini-code-assist]

If preprocessing_job_timeout or postprocessing_job_timeout is null or empty (e.g., when ingestion is disabled or timeouts are omitted), replace() or tonumber() will throw a Terraform evaluation error.

Using try() provides a safe fallback to null, allowing the downstream workflow module to use its own default values.

  preprocessing_job_timeout      = try(tonumber(replace(var.ingestion_config.preprocessing_job_timeout, "s", "")), null)
  postprocessing_job_timeout     = try(tonumber(replace(var.ingestion_config.postprocessing_job_timeout, "s", "")), null)

References

1. When designing nested Terraform modules, set the top-level variable default to null and declare the attribute as optional in intermediate module configuration objects. This ensures that the default value defined in the innermost module is respected and not overridden by parent defaults.

[gmechali]

A small nit, I dont think you're propagating these to the actual pre/post processing jobs as the timeout, it's just the polling connection.

You also can pass in these timeouts in the cloud run job containers running these.

Maybe passing it in both the cloud run job + the polling connector is the right approach? Can you update the worfklow.yaml connector params on the preprocessing polling too? I hardcoded that value (3993c71)

[gmechali]

nit: remove extra lines