| page_title | subcategory | description |
|---|---|---|
airbyte_destination_pinecone Resource - terraform-provider-airbyte |
DestinationPinecone Resource |
DestinationPinecone Resource
resource "airbyte_destination_pinecone" "my_destination_pinecone" {
configuration = {
embedding = {
azure_open_ai = {
api_base = "https://your-resource-name.openai.azure.com"
deployment = "your-resource-name"
openai_key = "...my_openai_key..."
}
cohere = {
cohere_key = "...my_cohere_key..."
}
fake = {
# ...
}
open_ai = {
openai_key = "...my_openai_key..."
}
open_ai_compatible = {
api_key = "...my_api_key..."
base_url = "https://your-service-name.com"
dimensions = 1536
model_name = "text-embedding-ada-002"
}
}
indexing = {
index = "...my_index..."
pinecone_environment = "us-west1-gcp"
pinecone_key = "...my_pinecone_key..."
}
omit_raw_text = true
processing = {
chunk_overlap = 4
chunk_size = 7074
field_name_mappings = [
{
from_field = "...my_from_field..."
to_field = "...my_to_field..."
}
]
metadata_fields = [
"..."
]
text_fields = [
"..."
]
text_splitter = {
by_markdown_header = {
split_level = 1
}
by_programming_language = {
language = "ruby"
}
by_separator = {
keep_separator = true
separators = [
"..."
]
}
}
}
}
definition_id = "ae1c7c9a-2b03-4eee-843a-7ed67978265e"
name = "...my_name..."
workspace_id = "a8bd2b0a-cdce-4816-9962-7547715c4567"
}configuration(Attributes) The configuration model for the Vector DB based destinations. This model is used to generate the UI for the destination configuration, as well as to provide type safety for the configuration passed to the destination.
The configuration model is composed of four parts:
- Processing configuration
- Embedding configuration
- Indexing configuration
- Advanced configuration
Processing, embedding and advanced configuration are provided by this base class, while the indexing configuration is provided by the destination connector in the sub class. (see below for nested schema)
name(String) Name of the destination e.g. dev-mysql-instance.workspace_id(String)
definition_id(String) The UUID of the connector definition. One of configuration.destinationType or definitionId must be provided. Requires replacement if changed.
created_at(Number)destination_id(String)destination_type(String)
Required:
embedding(Attributes) Embedding configuration (see below for nested schema)indexing(Attributes) Pinecone is a popular vector store that can be used to store and retrieve embeddings. (see below for nested schema)processing(Attributes) (see below for nested schema)
Optional:
omit_raw_text(Boolean) Do not store the text that gets embedded along with the vector and the metadata in the destination. If set to true, only the vector and the metadata will be stored - in this case raw text for LLM use cases needs to be retrieved from another source. Default: false
Optional:
azure_open_ai(Attributes) Use the Azure-hosted OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. (see below for nested schema)cohere(Attributes) Use the Cohere API to embed text. (see below for nested schema)fake(Attributes) Use a fake embedding made out of random vectors with 1536 embedding dimensions. This is useful for testing the data pipeline without incurring any costs. (see below for nested schema)open_ai(Attributes) Use the OpenAI API to embed text. This option is using the text-embedding-ada-002 model with 1536 embedding dimensions. (see below for nested schema)open_ai_compatible(Attributes) Use a service that's compatible with the OpenAI API to embed text. (see below for nested schema)
Required:
api_base(String) The base URL for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resourcedeployment(String) The deployment for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resourceopenai_key(String, Sensitive) The API key for your Azure OpenAI resource. You can find this in the Azure portal under your Azure OpenAI resource
Required:
cohere_key(String, Sensitive)
Required:
openai_key(String, Sensitive)
Required:
base_url(String) The base URL for your OpenAI-compatible servicedimensions(Number) The number of dimensions the embedding model is generating
Optional:
api_key(String, Sensitive) Default: ""model_name(String) The name of the model to use for embedding. Default: "text-embedding-ada-002"
Required:
index(String) Pinecone index in your project to load data intopinecone_environment(String) Pinecone Cloud environment to usepinecone_key(String, Sensitive) The Pinecone API key to use matching the environment (copy from Pinecone console)
Required:
chunk_size(Number) Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM)
Optional:
chunk_overlap(Number) Size of overlap between chunks in tokens to store in vector store to better capture relevant context. Default: 0field_name_mappings(Attributes List) List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. (see below for nested schema)metadata_fields(List of String) List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g.user.namewill access thenamefield in theuserobject. It's also possible to use wildcards to access all fields in an object, e.g.users.*.namewill access allnamesfields in all entries of theusersarray. When specifying nested paths, all matching values are flattened into an array set to a field named by the path.text_fields(List of String) List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g.user.namewill access thenamefield in theuserobject. It's also possible to use wildcards to access all fields in an object, e.g.users.*.namewill access allnamesfields in all entries of theusersarray.text_splitter(Attributes) Split text fields into chunks based on the specified method. (see below for nested schema)
Required:
from_field(String) The field name in the sourceto_field(String) The field name to use in the destination
Optional:
by_markdown_header(Attributes) Split the text by Markdown headers down to the specified header level. If the chunk size fits multiple sections, they will be combined into a single chunk. (see below for nested schema)by_programming_language(Attributes) Split the text by suitable delimiters based on the programming language. This is useful for splitting code into chunks. (see below for nested schema)by_separator(Attributes) Split the text by the list of separators until the chunk size is reached, using the earlier mentioned separators where possible. This is useful for splitting text fields by paragraphs, sentences, words, etc. (see below for nested schema)
Optional:
split_level(Number) Level of markdown headers to split text fields by. Headings down to the specified level will be used as split points. Default: 1
Required:
language(String) Split code in suitable places based on the programming language. must be one of ["cpp", "go", "java", "js", "php", "proto", "python", "rst", "ruby", "rust", "scala", "swift", "markdown", "latex", "html", "sol"]
Optional:
keep_separator(Boolean) Whether to keep the separator in the resulting chunks. Default: falseseparators(List of String) List of separator strings to split text fields by. The separator itself needs to be wrapped in double quotes, e.g. to split by the dot character, use ".". To split by a newline, use "\n".
Import is supported using the following syntax:
terraform import airbyte_destination_pinecone.my_airbyte_destination_pinecone ""