Archetypes for AI/ML on GCP

[obriensystems]

review the following quickstarts

• https://console.cloud.google.com/products/solutions/details/aiml-image-processing-on-cloud-functions?hl=en&project=solutions-ml-image-proc-old

[obriensystems]

For
https://console.cloud.google.com/products/solutions/details/aiml-image-processing-on-cloud-functions?hl=en&project=solutions-ml-image-proc-old
https://github.com/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/tree/sic-jss/infra

make...
1084baf8a506: Pull complete 
Digest: sha256:8ebd79a373313ee94ccaba8d8962cb5bc300901aef1bea217f4c1f18d0204c15
Status: Downloaded newer image for gcr.io/cloud-foundation-cicd/cft/developer-tools:1.18
Loading /usr/local/bin/task_helper_functions.sh from ~/.bashrc
Invoking init_credentials from ~/.bashrc
Disable this behavior by setting CFT_DISABLE_INIT_CREDENTIALS=yes
Error: SERVICE_ACCOUNT_JSON must contain the JSON string (not the
file path) of the service account credentials.  For example:
export SERVICE_ACCOUNT_JSON=$(< ~/.credentials/my-sa-key.json)
[root@d330229787f4 workspace]# ls ~/.credentials

out of container - created json key and referenced
michael@cloudshell:~/wse_github/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/infra (solutions-ml-image-proc-old)$ export SERVICE_ACCOUNT_JSON=solutions-sa.json

michael@cloudshell:~/wse_github/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/infra (solutions-ml-image-proc-old)$ make
docker run --rm -it \
        -e SERVICE_ACCOUNT_JSON \
        -v "/home/michael/wse_github/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/infra":/workspace \
        gcr.io/cloud-foundation-cicd/cft/developer-tools:1.18 \
        /bin/bash
Loading /usr/local/bin/task_helper_functions.sh from ~/.bashrc
Invoking init_credentials from ~/.bashrc
Disable this behavior by setting CFT_DISABLE_INIT_CREDENTIALS=yes
ERROR: (gcloud.auth.activate-service-account) Missing required argument [ACCOUNT]: An account is required when using .p12 keys

direct terraform outside of the dev container

michael@cloudshell:~/wse_github/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/infra (solutions-ml-image-proc-old)$ terraform plan
var.project_id
  GCP project ID.

  Enter a value: solutions-ml-image-proc-old

data.google_project.project: Reading...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform planned the following actions, but then encountered a problem:

  # data.google_compute_default_service_account.default will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_compute_default_service_account" "default" {
      + display_name = (known after apply)
      + email        = (known after apply)
      + id           = (known after apply)
      + member       = (known after apply)
      + name         = (known after apply)
      + project      = (known after apply)
      + unique_id    = (known after apply)
    }

  # data.google_compute_zones.cz_available will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_compute_zones" "cz_available" {
      + id      = (known after apply)
      + names   = (known after apply)
      + project = "solutions-ml-image-proc-old"
      + region  = "us-west4"
    }

  # data.google_storage_project_service_account.gcs_account will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_storage_project_service_account" "gcs_account" {
      + email_address = (known after apply)
      + id            = (known after apply)
      + member        = (known after apply)
      + project       = (known after apply)
    }

  # null_resource.previous_time will be created
  + resource "null_resource" "previous_time" {
      + id = (known after apply)
    }

  # time_sleep.wait_for_apis will be created
  + resource "time_sleep" "wait_for_apis" {
      + create_duration = "30s"
      + id              = (known after apply)
    }

  # module.cloudfunctions.data.archive_file.functions will be read during apply
  # (config refers to values not yet known)
 <= data "archive_file" "functions" {
      + id                  = (known after apply)
      + output_base64sha256 = (known after apply)
      + output_base64sha512 = (known after apply)
      + output_md5          = (known after apply)
      + output_path         = (known after apply)
      + output_sha          = (known after apply)
      + output_sha256       = (known after apply)
      + output_sha512       = (known after apply)
      + output_size         = (known after apply)
      + source_dir          = "modules/cloudfunctions/../../../src/gcf/"
      + type                = "zip"
    }

  # module.cloudfunctions.data.google_project.project will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_project" "project" {
      + auto_create_network = (known after apply)
      + billing_account     = (known after apply)
      + folder_id           = (known after apply)
      + id                  = (known after apply)
      + labels              = (known after apply)
      + name                = (known after apply)
      + number              = (known after apply)
      + org_id              = (known after apply)
      + skip_delete         = (known after apply)
    }

  # module.cloudfunctions.data.google_storage_project_service_account.gcs_account will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_storage_project_service_account" "gcs_account" {
      + email_address = (known after apply)
      + id            = (known after apply)
      + member        = (known after apply)
      + project       = (known after apply)
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/artifactregistry.reader"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/artifactregistry.reader"
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/cloudfunctions.invoker"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/cloudfunctions.invoker"
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/eventarc.eventReceiver"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/eventarc.eventReceiver"
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/logging.logWriter"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/logging.logWriter"
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/run.invoker"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/run.invoker"
    }

  # module.cloudfunctions.google_project_iam_member.gcf_sa_roles["roles/storage.objectAdmin"] will be created
  + resource "google_project_iam_member" "gcf_sa_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/storage.objectAdmin"
    }

  # module.cloudfunctions.google_project_iam_member.gcs_pubsub_publishing will be created
  + resource "google_project_iam_member" "gcs_pubsub_publishing" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = (known after apply)
      + role    = "roles/pubsub.publisher"
    }

  # module.cloudfunctions.google_service_account.gcf_sa will be created
  + resource "google_service_account" "gcf_sa" {
      + account_id   = "gcf-sa"
      + disabled     = false
      + display_name = "Service Account - for cloud function and eventarc trigger."
      + email        = (known after apply)
      + id           = (known after apply)
      + member       = (known after apply)
      + name         = (known after apply)
      + project      = (known after apply)
      + unique_id    = (known after apply)
    }

  # module.cloudfunctions.google_storage_bucket.code_bucket will be created
  + resource "google_storage_bucket" "code_bucket" {
      + force_destroy               = true
      + id                          = (known after apply)
      + labels                      = {
          + "app" = "terraform-ml-image-annotation-gcf"
        }
      + location                    = "US-WEST4"
      + name                        = (known after apply)
      + project                     = (known after apply)
      + public_access_prevention    = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = true
      + url                         = (known after apply)
    }

  # module.cloudfunctions.google_storage_bucket_object.gcf_code will be created
  + resource "google_storage_bucket_object" "gcf_code" {
      + bucket         = (known after apply)
      + content        = (sensitive value)
      + content_type   = (known after apply)
      + crc32c         = (known after apply)
      + detect_md5hash = "different hash"
      + id             = (known after apply)
      + kms_key_name   = (known after apply)
      + md5hash        = (known after apply)
      + media_link     = (known after apply)
      + name           = (known after apply)
      + output_name    = (known after apply)
      + self_link      = (known after apply)
      + source         = (known after apply)
      + storage_class  = (known after apply)
    }

  # module.project-services.google_project_iam_member.project_service_identity_roles["eventarc.googleapis.com roles/eventarc.serviceAgent"] will be created
  + resource "google_project_iam_member" "project_service_identity_roles" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = "solutions-ml-image-proc-old"
      + role    = "roles/eventarc.serviceAgent"
    }

  # module.project-services.google_project_service.project_services["appengine.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "appengine.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["artifactregistry.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "artifactregistry.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["cloudbuild.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "cloudbuild.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["cloudfunctions.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "cloudfunctions.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["compute.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "compute.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["eventarc.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "eventarc.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["iam.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "iam.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["logging.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "logging.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["pubsub.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "pubsub.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["run.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "run.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["secretmanager.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "secretmanager.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["storage.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "storage.googleapis.com"
    }

  # module.project-services.google_project_service.project_services["vision.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "solutions-ml-image-proc-old"
      + service                    = "vision.googleapis.com"
    }

  # module.project-services.google_project_service_identity.project_service_identities["eventarc.googleapis.com"] will be created
  + resource "google_project_service_identity" "project_service_identities" {
      + email   = (known after apply)
      + id      = (known after apply)
      + project = "solutions-ml-image-proc-old"
      + service = "eventarc.googleapis.com"
    }

Plan: 27 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + neos_walkthrough_url = "https://console.cloud.google.com/products/solutions/deployments?walkthrough_id=panels--sic--image-processing-gcf_toc"
  + source_code_url      = (known after apply)
╷
│ Error: Error when reading or editing Project "solutions-ml-image-proc-old": googleapi: Error 403: Cloud Resource Manager API has not been used in project solutions-ml-image-proc-old before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=solutions-ml-image-proc-old then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "activationUrl": "https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=solutions-ml-image-proc-old",
│       "consumer": "projects/solutions-ml-image-proc-old",
│       "containerInfo": "solutions-ml-image-proc-old",
│       "service": "cloudresourcemanager.googleapis.com",
│       "serviceTitle": "Cloud Resource Manager API"
│     },
│     "reason": "SERVICE_DISABLED"
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.LocalizedMessage",
│     "locale": "en-US",
│     "message": "Cloud Resource Manager API has not been used in project solutions-ml-image-proc-old before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=solutions-ml-image-proc-old then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry."
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=solutions-ml-image-proc-old"
│       }
│     ]
│   }
│ ]
│ , accessNotConfigured
│ 
│   with data.google_project.project,
│   on main.tf line 17, in data "google_project" "project":
│   17: data "google_project" "project" {
│ 
╵

enable the api


wse_github/GoogleCloudPlatform/terraform-ml-image-annotation-gcf/infra (solutions-ml-image-proc-old)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-81630124114-895b9d31-7c12-4b26-b05a-2c808dc67234" finished successfully.

Plan: 32 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + annotate_gcs_function_name  = "annotate_gcs"
  + annotate_http_function_name = "annotate-http"
  + neos_walkthrough_url        = "https://console.cloud.google.com/products/solutions/deployments?walkthrough_id=panels--sic--image-processing-gcf_toc"
  + source_code_url             = (known after apply)
  + vision_annotations_gcs      = (known after apply)
  + vision_input_gcs            = (known after apply)
  + vision_prediction_url       = [
      + (known after apply),
      + "ingressIndex:0",
      + "ingressValue:ALLOW_ALL",
      + "isAuthenticated:false ",
    ]

[obriensystems]

apply

Plan: 32 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + annotate_gcs_function_name  = "annotate_gcs"
  + annotate_http_function_name = "annotate-http"
  + neos_walkthrough_url        = "https://console.cloud.google.com/products/solutions/deployments?walkthrough_id=panels--sic--image-processing-gcf_toc"
  + source_code_url             = (known after apply)
  + vision_annotations_gcs      = (known after apply)
  + vision_input_gcs            = (known after apply)
  + vision_prediction_url       = [
      + (known after apply),
      + "ingressIndex:0",
      + "ingressValue:ALLOW_ALL",
      + "isAuthenticated:false ",
    ]`


Apply complete! Resources: 32 added, 0 changed, 0 destroyed.

Outputs:

annotate_gcs_function_name = "annotate_gcs"
annotate_http_function_name = "annotate-http"
neos_walkthrough_url = "https://console.cloud.google.com/products/solutions/deployments?walkthrough_id=panels--sic--image-processing-gcf_toc"
source_code_url = "gs://vision-code-81630124114/gcf_code_c89fa53959021621b8dc29d169243db0..zip"
vision_annotations_gcs = "gs://vision-annotations-81630124114"
vision_input_gcs = "gs://vision-input-81630124114"
vision_prediction_url = [
  "https://annotate-http-fopy2ex6xq-wn.a.run.app",
  "ingressIndex:0",
  "ingressValue:ALLOW_ALL",
  "isAuthenticated:false ",
]
``