Using Vertex AI Connectors (e.g., Confluence) without the full Agentspace Product?

[this-dave]

Hi all,

I'm trying to figure out the best practice for connecting our Confluence instance to a custom agent built with the Google ADK.

The goal is simple: I want to use the Search tool in my ADK agent to query our internal Confluence knowledge base.

It seemed straightforward at first: I created a Data Store in Vertex AI and used the built-in Confluence connector. This worked perfectly and the data was ingested.

However, I've hit a roadblock trying to use this Data Store. To query it via the API, my agent needs a serving_config, which requires linking the Data Store to a "Search App". The standard "Custom Search" app explicitly rejects my connector-based Data Store. The only path the Google Cloud Console allows is to create a full "Agentspace" app, which is too expensive for our needs (we don't need the full Agentspace product, just the underlying search capability).

So my questions are:

Is there a way to use a Data Store created with a managed connector (like Confluence) for search without being forced to use the expensive Agentspace product tier?

If the answer is no, what is the recommended best practice for this? Is the only alternative to build our own RAG loading from scratch (scripting the Confluence export etc.) and then using the RAG-Engine?

It feels like there's a gap between the simple, managed connectors and the ability to use them in a cost-effective, developer-led way. Any clarification or guidance would be greatly appreciated.

Thanks!

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: You can use your Confluence-connected Data Store directly with the VertexAiSearchTool in the ADK without needing to use the full "Agentspace" product. The key is to provide the data_store_id to the tool.

---

Hi @this-dave,

Based on the ADK documentation, you should be able to use your Vertex AI Data Store created with the Confluence connector directly in your agent without being forced into the "Agentspace" product tier. The standard way to connect an ADK agent to a Vertex AI Search data store is by using the built-in VertexAiSearchTool.

This tool only requires the data_store_id for its configuration, and it handles the communication with the Vertex AI Search service for you, abstracting away the need to manually create a serving_config.

Here is an example of how you would configure your agent:

from google.adk.agents import Agent
from google.adk.tools import VertexAiSearchTool

# The full resource ID of your Vertex AI Search Data Store
# It should be in the format: projects/YOUR_PROJECT_ID/locations/global/collections/default_collection/dataStores/YOUR_DATASTORE_ID
YOUR_DATASTORE_ID = "projects/..." # Replace with your data store ID

# Instantiate the VertexAiSearchTool with your data store ID
vertex_search_tool = VertexAiSearchTool(data_store_id=YOUR_DATASTORE_ID)

# Create your agent and pass the tool
my_agent = Agent(
    name="confluence_agent",
    model="gemini-1.5-flash",
    instruction="Answer questions using our internal Confluence knowledge base.",
    tools=[vertex_search_tool]
)

This approach allows you to leverage the search capabilities of your connector-based data store in a cost-effective, developer-led way, which seems to be exactly what you're looking for.

For more details, you can refer to the official documentation on VertexAiSearchTool and grounding with Vertex AI Search. [7, 11]

I hope this helps clarify the best practice and provides a path forward for your project.

[this-dave]

HI @adk-bot

Thanks for your detailed response. I've been trying to implement your suggestion, but I'm running into a persistent issue and was hoping you might have some further insight.

The good news is that your initial advice helped us move past some early configuration problems. However, we're now consistently getting a 404 NOT_FOUND error when the agent tries to access the Vertex AI Data Store.

What We've Done

1. Agent Configuration: Our agent code is set up as you suggested, loading the full Data Store resource name from an environment variable.

   # agents/confluence_search_agent/agent.py
   import os
   from google.adk.agents import Agent
   from google.adk.tools import VertexAiSearchTool
   
   # The full resource ID, e.g., "projects/your-project-id/locations/eu/collections/default_collection/dataStores/your-datastore-id"
   DATA_STORE_ID = os.environ.get("DATA_STORE_ID")
   
   vertex_search_tool = VertexAiSearchTool(data_store_id=DATA_STORE_ID)
   
   root_agent = Agent(
       name="confluence_knowledge_agent",
       model="gemini-1.5-flash",
       instruction="Answer questions using our internal Confluence knowledge base.",
       tools=[vertex_search_tool],
   )

2. Environment: Our .env file defines the PROJECT_ID, LOCATION, and the full DATA_STORE_ID. Our Data Store was created in the eu multi-region.

   # .env file
   PROJECT_ID="your-gcp-project-id"
   LOCATION="europe-west3" # For the model endpoint
   DATA_STORE_ID="projects/your-gcp-project-id/locations/eu/collections/default_collection/dataStores/your-datastore-id"

3. Permissions: We've authenticated using gcloud auth application-default login with a user that has Owner rights on the project. We also tried setting up a dedicated service account with Vertex AI User and Discovery Engine Viewer roles, but the result is the same.

The Persistent Error

Despite the configuration, we consistently receive the following error when the agent runs:

google.genai.errors.ClientError: 404 NOT_FOUND. {'error': {'code': 404, 'message': 'DataStore projects/PROJECT_NUMBER/locations/eu/collections/default_collection/dataStores/confluence-vertexai_XXXX not found.', 'status': 'NOT_FOUND'}}

(Note: I've replaced our actual project number and data store ID with placeholders)

Our Questions

1. Data Store Location: We created our Data Store in the eu multi-region, but our model location is europe-west3. Could this mismatch be causing the issue, even though we provide the full resource name?
2. Resource Path: The error shows the correct project number (resolved from our project ID) and the correct data store ID. We have triple-checked these in the Google Cloud Console. Is there any other reason the resource path would be considered invalid?
3. "AI App" Requirement: To be absolutely certain, is it correct that we only need a "Data Store" and not a full "Search App" in Vertex AI to use the VertexAiSearchTool? The persistent 404 is making us wonder if the Data Store is not properly accessible via the API without being attached to a search app.

Any clarification on these points would be incredibly helpful. We feel like we're very close, but are missing one key piece of the puzzle.

Thanks again for your help!

[abhi-kandoi]

Can you reply to @this-dave, @adk-bot?

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: You should be able to use your Confluence-based Data Store directly with the ADK's VertexAiSearchTool without needing to create a "Search App" or "Agentspace" in the Google Cloud Console. The ADK simplifies this process for you.

---

Hi @this-dave,

I understand you're facing a challenge using a Vertex AI Data Store created with the Confluence connector in your custom ADK agent. The issue with serving_config and the expensive "Agentspace" tier is a valid concern.

The good news is that when using the Google ADK, you likely don't need to manually create a "Search App" or worry about the serving_config in the Google Cloud Console. The ADK provides a built-in tool, VertexAiSearchTool, that abstracts away this complexity.

You can directly use your Data Store's ID with this tool. Here is how you would configure it in your agent's code:

from google.adk.agents import Agent
from google.adk.tools import VertexAiSearchTool

# Replace with the full resource ID of your Confluence Data Store
DATASTORE_ID = "projects/YOUR_PROJECT_ID/locations/global/collections/default_collection/dataStores/YOUR_CONFLUENCE_DATASTORE_ID"

# Your agent definition
my_agent = Agent(
    # ... other agent parameters
    tools=[
        VertexAiSearchTool(data_store_id=DATASTORE_ID)
    ]
)

By providing the data_store_id to the VertexAiSearchTool, the ADK will handle the communication with the Vertex AI Search service for you. This should work with your Data Store created from the Confluence connector, allowing you to query your knowledge base without needing to upgrade to the "Agentspace" product.

For more details, you can refer to the official documentation on Vertex AI Search grounding [1] and the code example for using VertexAiSearchTool [2].

Regarding your second question, building a custom RAG pipeline is indeed a powerful alternative, but it might not be necessary in your case. The VertexAiSearchTool is designed for this exact purpose, to provide a simpler, more direct way to connect your agent to your data.

I hope this helps you move forward with your project!

[1] https://google.github.io/adk-docs/grounding/vertex_ai_search_grounding/
[2] https://github.com/google/adk-docs/blob/main/examples/python/snippets/tools/built-in-tools/vertexai_search.py