Support vector search endpoints and indexes in DABs with direct deployment engine #4816
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| bundle: | ||
| name: vector-search-$UNIQUE_NAME | ||
|
|
||
| workspace: | ||
| root_path: ~/.bundle/$UNIQUE_NAME | ||
|
|
||
| resources: | ||
|
Contributor
There was a problem hiding this comment. As a follow up, please add support for permissions for these resources |
||
| vector_search_endpoints: | ||
| test_endpoint: | ||
| name: $ENDPOINT_NAME | ||
| endpoint_type: STANDARD | ||
| vector_search_indexes: | ||
|
Contributor
There was a problem hiding this comment. I would assume customers want to use JSON data to add it to the index, so we need to likely test and implement this |
||
| test_index: | ||
| name: $INDEX_NAME | ||
| endpoint_name: ${resources.vector_search_endpoints.test_endpoint.name} | ||
| primary_key: my_pkey | ||
| index_type: DELTA_SYNC | ||
| delta_sync_index_spec: | ||
| source_table: my_table | ||
| pipeline_type: TRIGGERED | ||
|
|
||
| targets: | ||
| development: | ||
| default: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
|
|
||
| === Deploy bundle with vector search endpoint and index (index depends on endpoint) | ||
| >>> [CLI] bundle plan | ||
| create vector_search_endpoints.test_endpoint | ||
| create vector_search_indexes.test_index | ||
|
|
||
| Plan: 2 to add, 0 to change, 0 to delete, 0 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| === Assert the endpoint is created | ||
| >>> [CLI] vector-search-endpoints get-endpoint test-vector-search-endpoint-[UNIQUE_NAME] | ||
| { | ||
| "name": "test-vector-search-endpoint-[UNIQUE_NAME]", | ||
| "endpoint_type": "STANDARD" | ||
| } | ||
|
|
||
| === Assert the index is created | ||
| >>> [CLI] vector-search-indexes get-index workspace.default.test_vector_search_index_[UNIQUE_NAME] | ||
| { | ||
| "name": "workspace.default.test_vector_search_index_[UNIQUE_NAME]", | ||
| "endpoint_name": "test-vector-search-endpoint-[UNIQUE_NAME]", | ||
| "index_type": "DELTA_SYNC" | ||
| } | ||
|
|
||
| === Test cleanup | ||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.vector_search_endpoints.test_endpoint | ||
| delete resources.vector_search_indexes.test_index | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME] | ||
|
|
||
| Deleting files... | ||
| Destroy complete! | ||
|
|
||
| === Assert the endpoint is deleted | ||
| === Assert the index is deleted |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/bin/bash | ||
|
|
||
| export ENDPOINT_NAME="test-vector-search-endpoint-${UNIQUE_NAME}" | ||
| export INDEX_NAME="workspace.default.test_vector_search_index_${UNIQUE_NAME}" | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| cleanup() { | ||
| title "Test cleanup" | ||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| title "Assert the endpoint is deleted" | ||
| trace errcode $CLI vector-search-endpoints get-endpoint "${ENDPOINT_NAME}" 2>/dev/null | ||
| title "Assert the index is deleted" | ||
| trace errcode $CLI vector-search-indexes get-index "${INDEX_NAME}" 2>/dev/null | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| title "Deploy bundle with vector search endpoint and index (index depends on endpoint)" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Assert the endpoint is created" | ||
| trace $CLI vector-search-endpoints get-endpoint "${ENDPOINT_NAME}" | jq "{name, endpoint_type}" | ||
|
|
||
| title "Assert the index is created" | ||
| trace $CLI vector-search-indexes get-index "${INDEX_NAME}" | jq "{name, endpoint_name, index_type}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Tests endpoint + index with interdependence (index depends on endpoint). | ||
|
|
||
| Local = true | ||
| Cloud = true | ||
| RecordRequests = false | ||
|
|
||
| Ignore = [ | ||
| ".databricks", | ||
| "databricks.yml", | ||
| ] | ||
|
|
||
| [EnvMatrix] | ||
| DATABRICKS_BUNDLE_ENGINE = ["direct"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| package resources | ||
|
|
||
| import ( | ||
| "context" | ||
| "net/url" | ||
|
|
||
| "github.com/databricks/cli/libs/log" | ||
| "github.com/databricks/databricks-sdk-go" | ||
| "github.com/databricks/databricks-sdk-go/apierr" | ||
| "github.com/databricks/databricks-sdk-go/marshal" | ||
| "github.com/databricks/databricks-sdk-go/service/vectorsearch" | ||
| ) | ||
|
|
||
| type VectorSearchEndpoint struct { | ||
| BaseResource | ||
| vectorsearch.CreateEndpoint | ||
| } | ||
|
|
||
| func (e *VectorSearchEndpoint) UnmarshalJSON(b []byte) error { | ||
| return marshal.Unmarshal(b, e) | ||
| } | ||
|
|
||
| func (e VectorSearchEndpoint) MarshalJSON() ([]byte, error) { | ||
| return marshal.Marshal(e) | ||
| } | ||
|
|
||
| func (e *VectorSearchEndpoint) Exists(ctx context.Context, w *databricks.WorkspaceClient, name string) (bool, error) { | ||
| _, err := w.VectorSearchEndpoints.GetEndpoint(ctx, vectorsearch.GetEndpointRequest{ | ||
| EndpointName: name, | ||
| }) | ||
| if err != nil { | ||
| log.Debugf(ctx, "vector search endpoint %s does not exist: %v", name, err) | ||
| if apierr.IsMissing(err) { | ||
| return false, nil | ||
| } | ||
| return false, err | ||
| } | ||
| return true, nil | ||
| } | ||
|
|
||
| func (*VectorSearchEndpoint) ResourceDescription() ResourceDescription { | ||
| return ResourceDescription{ | ||
| SingularName: "vector_search_endpoint", | ||
| PluralName: "vector_search_endpoints", | ||
| SingularTitle: "Vector Search Endpoint", | ||
| PluralTitle: "Vector Search Endpoints", | ||
| } | ||
| } | ||
|
|
||
| func (e *VectorSearchEndpoint) InitializeURL(baseURL url.URL) { | ||
| if e.ID == "" { | ||
| return | ||
| } | ||
| baseURL.Path = "explore/vector-search/" + e.ID | ||
| e.URL = baseURL.String() | ||
| } | ||
|
|
||
| func (e *VectorSearchEndpoint) GetURL() string { | ||
| return e.URL | ||
| } | ||
|
|
||
| func (e *VectorSearchEndpoint) GetName() string { | ||
| return e.Name | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we merge budget_policy_id and effective_budget_policy_id? So that we detect drift there? OR are there cases where effective_budget_policy_id is different due to some external policy?