Skip to content

Latest commit

 

History

History
133 lines (98 loc) · 7.05 KB

destination_s3_glue.md

File metadata and controls

133 lines (98 loc) · 7.05 KB
page_title subcategory description
airbyte_destination_s3_glue Resource - terraform-provider-airbyte
DestinationS3Glue Resource

airbyte_destination_s3_glue (Resource)

DestinationS3Glue Resource

Example Usage

resource "airbyte_destination_s3_glue" "my_destination_s3glue" {
  configuration = {
    access_key_id     = "A012345678910EXAMPLE"
    file_name_pattern = "{date}"
    format = {
      json_lines_newline_delimited_json = {
        compression = {
          gzip = {
            compression_type = "GZIP"
          }
          no_compression = {
            compression_type = "No Compression"
          }
        }
        flattening  = "No flattening"
        format_type = "JSONL"
      }
    }
    glue_database              = "airbyte_database"
    glue_serialization_library = "org.openx.data.jsonserde.JsonSerDe"
    s3_bucket_name             = "airbyte_sync"
    s3_bucket_path             = "data_sync/test"
    s3_bucket_region           = "us-gov-west-1"
    s3_endpoint                = "http://localhost:9000"
    s3_path_format             = "${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_"
    secret_access_key          = "a012345678910ABCDEFGH/AbCdEfGhEXAMPLEKEY"
  }
  definition_id = "7a9c505b-1990-4b38-b243-1c254ff78c36"
  name          = "...my_name..."
  workspace_id  = "e7df4138-2176-42e6-aa11-ed72fe60b365"
}

Schema

Required

  • configuration (Attributes) (see below for nested schema)
  • name (String) Name of the destination e.g. dev-mysql-instance.
  • workspace_id (String)

Optional

  • definition_id (String) The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. Requires replacement if changed.

Read-Only

  • created_at (Number)
  • destination_id (String)
  • destination_type (String)

Nested Schema for configuration

Required:

  • format (Attributes) Format of the data output. See here for more details (see below for nested schema)
  • glue_database (String) Name of the glue database for creating the tables, leave blank if no integration
  • s3_bucket_name (String) The name of the S3 bucket. Read more here.
  • s3_bucket_path (String) Directory under the S3 bucket where data will be written. Read more here

Optional:

  • access_key_id (String, Sensitive) The access key ID to access the S3 bucket. Airbyte requires Read and Write permissions to the given bucket. Read more here.
  • file_name_pattern (String) The pattern allows you to set the file-name format for the S3 staging file(s)
  • glue_serialization_library (String) The library that your query engine will use for reading and writing data in your lake. Default: "org.openx.data.jsonserde.JsonSerDe"; must be one of ["org.openx.data.jsonserde.JsonSerDe", "org.apache.hive.hcatalog.data.JsonSerDe"]
  • s3_bucket_region (String) The region of the S3 bucket. See here for all region codes. Default: ""; must be one of ["", "af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-south-2", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ap-southeast-4", "ca-central-1", "ca-west-1", "cn-north-1", "cn-northwest-1", "eu-central-1", "eu-central-2", "eu-north-1", "eu-south-1", "eu-south-2", "eu-west-1", "eu-west-2", "eu-west-3", "il-central-1", "me-central-1", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"]
  • s3_endpoint (String) Your S3 endpoint url. Read more here. Default: ""
  • s3_path_format (String) Format string on how data will be organized inside the S3 bucket directory. Read more here
  • secret_access_key (String, Sensitive) The corresponding secret to the access key ID. Read more here

Nested Schema for configuration.format

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json

Optional:

  • compression (Attributes) Whether the output files should be compressed. If compression is selected, the output filename will have an extra extension (GZIP: ".jsonl.gz"). (see below for nested schema)
  • flattening (String) Whether the input json data should be normalized (flattened) in the output JSON Lines. Please refer to docs for details. Default: "Root level flattening"; must be one of ["No flattening", "Root level flattening"]
  • format_type (String) Default: "JSONL"; must be "JSONL"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression

Optional:

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.gzip

Optional:

  • compression_type (String) Default: "GZIP"; must be "GZIP"

Nested Schema for configuration.format.json_lines_newline_delimited_json.compression.no_compression

Optional:

  • compression_type (String) Default: "No Compression"; must be "No Compression"

Import

Import is supported using the following syntax:

terraform import airbyte_destination_s3_glue.my_airbyte_destination_s3_glue ""