DAB: Ingestion pipeline, ignores field serverless: false on deployment

[PLsergent]

Describe the issue

When deploying an ingestion pipeline via DAB the field serverless is being ignored. Hence it's not possible to set serverless: false to run on classic compute.

It does work when creating one manually from the Web app.

Configuration

resources:
  pipelines:
    test_ingestion:
      name: Test ingestion
      serverless: false
      ingestion_definition:
        connection_name: connection_oracle
        source_type: ORACLE
      schema: oracle_ingestion
      channel: PREVIEW
      catalog: bronze

Steps to reproduce the behavior

1. Run databricks bundle deploy ...
2. Check deployment > serverless is missing from definition

Expected Behavior

Compute should not be serverless (example from manual creation)

Actual Behavior

Compute is set to serverless

OS and CLI version

Tested with following versions:

• v0.297.2
• v1.0.0
• v1.10.0

Is this a regression?

I don't know, it's our first time using ingestion pipeline.

Debug Logs

databricks validate --output json

The field is also ignored

test_ingestion": {
        "catalog": "catalog",
        "channel": "PREVIEW",
        "deployment": {
            {
              "table": {
                "destination_catalog": "catalog",
                "destination_schema": "schema",
                "source_catalog": "xxxx",
                "source_schema": "xxxx",
                "source_table": "xxxx",
                "table_configuration": {
                  "query_based_connector_config": {
                    "cursor_columns": [
                      "EXTRACTION_LEARN_DATE"
                    ]
                  }
                }
              }
            }
          ],
          "source_type": "ORACLE"
        }

[janniklasrose]

This is a known issue with the terraform provider. Please raise an issue in https://github.com/databricks/terraform-provider-databricks.

From a DABs perspective, I can recommend migrating to the direct engine (bundle.engine: direct) which sends the field correctly