Deploying agent engine with Secret environment variables from the Secret Manager

[lposilovic]

I'm trying to deploy the agent using Vertex AI Agent Engine using secrets as env_vars through python. The agent is deployed succesfully if I'm not using the secrets in the environment variables, but fails when using secrets. Deployment actually passes, but the agent won't start.

The agent is being deployed using github actions and a IAM service account. Service account used has Secret Manager Secret Accessor permission. I also set up the same permission in the secret configuration in the secret manager for this service account. The secrets and the agent are being deployed in the same gcp project. Error and logs are not very much useful. I guess there is some kind of access needed that I didn't yet set up?

Environment details

• OS type and version: Ubuntu 24.04.02 (github action runner)
• Python version: 3.12.3
• google-cloud-aiplatform version: 1.104.0

Steps to reproduce

1. Set up secrets in the secrets manager, give the service account permission (Secret Manager Secret Accessor) for this secret
2. Deploy the agent with code (as per documentation):

    remote_app = agent_engines.create(
        app,
        requirements=[agent_whl_file],
        extra_packages=[agent_whl_file],
        env_vars={
            "AWS_REGION": aws_region,
            "AWS_ACCESS_KEY_ID": {"secret": "AWS_ACCESS_KEY_ID", "version": "latest"},
            "AWS_SECRET_ACCESS_KEY": {
                "secret": "AWS_SECRET_ACCESS_KEY",
                "version": "latest",
            },
        },
    )

Stack trace

INFO:__main__:Initializing Vertex AI with the following configurations:
INFO:__main__:Project: ***
INFO:__main__:Location: ***
INFO:__main__:Staging Bucket: ***
ERROR:__main__:Agent 2544419440255041536 not found
INFO:__main__:Deploying app...
DEBUG:__main__:Deploying agent to agent engine:
INFO:vertexai.agent_engines:Identified the following requirements: ***'cloudpickle': '3.1.1', 'google-cloud-aiplatform': '1.104.0', 'pydantic': '2.11.7'***
Identified the following requirements: ***'cloudpickle': '3.1.1', 'google-cloud-aiplatform': '1.104.0', 'pydantic': '2.11.7'***
WARNING:vertexai.agent_engines:The following requirements are missing: ***'cloudpickle', 'google-cloud-aiplatform', 'pydantic'***
INFO:vertexai.agent_engines:The following requirements are appended: ***'cloudpickle==3.1.1', 'pydantic==2.11.7'***
INFO:vertexai.agent_engines:The final list of requirements: ['kitro_cs-0.1.0-py3-none-any.whl', 'cloudpickle==3.1.1', 'pydantic==2.11.7']
The following requirements are missing: ***'cloudpickle', 'google-cloud-aiplatform', 'pydantic'***
The following requirements are appended: ***'cloudpickle==3.1.1', 'pydantic==2.11.7'***
The final list of requirements: ['kitro_cs-0.1.0-py3-none-any.whl', 'cloudpickle==3.1.1', 'pydantic==2.11.7']
DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:urllib3.connectionpool:[https://oauth2.googleapis.com:443](https://oauth2.googleapis.com/) "POST /token HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): storage.googleapis.com:443
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "GET /storage/v1/b/***?projection=noAcl&prettyPrint=false HTTP/1.1" 200 757
INFO:vertexai.agent_engines:Using bucket ***
Using bucket ***
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=resumable HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "PUT /upload/storage/v1/b/***/o?uploadType=resumable&upload_id=ABgVH8_mG8tK5rSL9iQbaTBxxbtXznU8eQPV2Mgu2APU5lnTvWECe2RVzx58N7XSTFnKr98jChcJ0oHZ3MCILs5sIBRU25bw2_2AeF4zhgdv2yc HTTP/1.1" 200 869
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "GET /download/storage/v1/b/***/o/agent_engine%2Fagent_engine.pkl?alt=media HTTP/1.1" 206 13037
INFO:vertexai.agent_engines:Wrote to ***/agent_engine/agent_engine.pkl
Wrote to ***/agent_engine/agent_engine.pkl
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=multipart HTTP/1.1" 200 852
Writing to ***/agent_engine/requirements.txt
Creating in-memory tarfile of extra_packages
INFO:vertexai.agent_engines:Writing to ***/agent_engine/requirements.txt
INFO:vertexai.agent_engines:Creating in-memory tarfile of extra_packages
DEBUG:urllib3.connectionpool:[https://storage.googleapis.com:443](https://storage.googleapis.com/) "POST /upload/storage/v1/b/***/o?uploadType=multipart HTTP/1.1" 200 867
Writing to ***/agent_engine/dependencies.tar.gz
INFO:vertexai.agent_engines:Writing to ***/agent_engine/dependencies.tar.gz
INFO:vertexai.agent_engines:Creating AgentEngine
Creating AgentEngine
INFO:vertexai.agent_engines:Create AgentEngine backing LRO: projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848
INFO:vertexai.agent_engines:View progress and logs at https://console.cloud.google.com/logs/query?project=***
Create AgentEngine backing LRO: projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848
View progress and logs at https://console.cloud.google.com/logs/query?project=***
Traceback (most recent call last):
  File "/home/runner/work/***/***/agents/***/deployment/deploy.py", line 127, in <module>
    resource_name = deploy_agent(
                    ^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/deployment/deploy.py", line 77, in deploy_agent
    remote_app = agent_engines.create(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/vertexai/agent_engines/__init__.py", line 161, in create
    return AgentEngine.create(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/vertexai/agent_engines/_agent_engines.py", line 523, in create
    created_resource = operation_future.result()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/***/***/agents/***/.venv/lib/python3.12/site-packages/google/api_core/future/polling.py", line 261, in result
    raise self._exception
google.api_core.exceptions.InvalidArgument: 400 Reasoning Engine resource [projects/***/locations/***/reasoningEngines/388532224883949568] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips. 3: Reasoning Engine resource [projects/***/locations/***/reasoningEngines/388532224883949568] failed to start and cannot serve traffic. Please refer to our documentation (https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/troubleshooting/deploy) for checking logs and other troubleshooting tips.

In the Logs Explorer though no error is reported, this is the last log:

INFO 2025-07-25T08:18:00.840894547Z DONE
NOTICE 2025-07-25T08:18:33.616264Z [protoPayload.serviceName: aiplatform.googleapis.com] [protoPayload.methodName: google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine] [protoPayload.resourceName: projects/***/locations/us-central1] [protoPayload.authenticationInfo.principalEmail: ***@***.iam.gserviceaccount.com] audit_log, method: "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine", principal_email: "***@***.iam.gserviceaccount.com"
  {
    "protoPayload": {
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "status": {},
      "authenticationInfo": {
        "principalEmail": "***@***.iam.gserviceaccount.com",
        "principalSubject": "serviceAccount:***@***.iam.gserviceaccount.com"
      },
      "requestMetadata": {
        "callerIp": "64.236.177.0",
        "callerSuppliedUserAgent": "grpc-python/1.73.1 grpc-c/48.0.0 (linux; chttp2),gzip(gfe)",
        "requestAttributes": {
          "time": "2025-07-25T08:14:52.497012701Z",
          "auth": {}
        },
        "destinationAttributes": {}
      },
      "serviceName": "aiplatform.googleapis.com",
      "methodName": "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine",
      "authorizationInfo": [
        {
          "resource": "projects/***/locations/***",
          "permission": "aiplatform.reasoningEngines.create",
          "granted": true,
          "resourceAttributes": {},
          "permissionType": "ADMIN_WRITE"
        }
      ],
      "resourceName": "projects/***/locations/***"
    },
    "insertId": "pv6lnld16me",
    "resource": {
      "type": "audited_resource",
      "labels": {
        "project_id": "***",
        "method": "google.cloud.aiplatform.v1.ReasoningEngineService.CreateReasoningEngine",
        "service": "aiplatform.googleapis.com"
      }
    },
    "timestamp": "2025-07-25T08:18:33.616264Z",
    "severity": "NOTICE",
    "logName": "projects/***/logs/cloudaudit.googleapis.com%2Factivity",
    "operation": {
      "id": "projects/***/locations/***/reasoningEngines/388532224883949568/operations/8115121181423566848",
      "producer": "aiplatform.googleapis.com",
      "last": true
    },
    "receiveTimestamp": "2025-07-25T08:18:33.767350476Z"
  }

Thank you for your assistance!