feat: Ask Sourcebot (#392)

Co-authored-by: msukkari <[email protected]>

Author: brendan-kellam

.github/workflows/_gcp-deploy.yml

@@ -60,6 +60,8 @@ jobs:
             NEXT_PUBLIC_SENTRY_ENVIRONMENT=${{ vars.NEXT_PUBLIC_SENTRY_ENVIRONMENT }}
             NEXT_PUBLIC_SENTRY_WEBAPP_DSN=${{ vars.NEXT_PUBLIC_SENTRY_WEBAPP_DSN }}
             NEXT_PUBLIC_SENTRY_BACKEND_DSN=${{ vars.NEXT_PUBLIC_SENTRY_BACKEND_DSN }}
+            NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY=${{ vars.NEXT_PUBLIC_LANGFUSE_PUBLIC_KEY }}
+            NEXT_PUBLIC_LANGFUSE_BASE_URL=${{ vars.NEXT_PUBLIC_LANGFUSE_BASE_URL }}
             SENTRY_SMUAT=${{ secrets.SENTRY_SMUAT }}
             SENTRY_ORG=${{ vars.SENTRY_ORG }}
             SENTRY_WEBAPP_PROJECT=${{ vars.SENTRY_WEBAPP_PROJECT }}

CHANGELOG.md

@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Introducing Ask Sourcebot - ask natural langauge about your codebase. Get back comprehensive Markdown responses with inline citations back to the code. Bring your own LLM api key. [#392](https://github.com/sourcebot-dev/sourcebot/pull/392) 
+
 ## [4.5.3] - 2025-07-20
 
 ### Changed

README.md

@@ -44,16 +44,20 @@
 
 # About
 
-Sourcebot lets you index all your repos and branches across multiple code hosts (GitHub, GitLab, Bitbucket, Gitea, or Gerrit) and search through them using a blazingly fast interface.
+Sourcebot is a self-hosted tool that helps you understand your codebase. 
 
-https://github.com/user-attachments/assets/ced355f3-967e-4f37-ae6e-74ab8c06b9ec
+- **Ask Sourcebot:** Ask questions about your codebase and have Sourcebot provide detailed answers grounded with inline citations.
+- **Code search:** Search and navigate across all your repos and branches, no matter where they’re hosted.
+
+https://github.com/user-attachments/assets/286ad97a-a543-4eef-a2f1-4fa31bea1b32
 
 
 ## Features
 - 💻 **One-command deployment**: Get started instantly using Docker on your own machine.
-- 🔍 **Multi-repo search**: Index and search through multiple public and private repositories and branches on GitHub, GitLab, Bitbucket, Gitea, or Gerrit.
-- ⚡**Lightning fast performance**: Built on top of the powerful [Zoekt](https://github.com/sourcegraph/zoekt) search engine.
-- 🎨 **Modern web app**: Enjoy a sleek interface with features like syntax highlighting, light/dark mode, and vim-style navigation 
+- 🤖 **Bring your own model**: Connect Sourcebot to any of the reasoning models you're already using.
+- 🔍 **Multi-repo support**: Index and search through multiple public and private repositories and branches on GitHub, GitLab, Bitbucket, Gitea, or Gerrit.
+- ⚡ **Lightning fast performance**: Built on top of the powerful [Zoekt](https://github.com/sourcegraph/zoekt) search engine.
+- 🎨 **Modern web app**: Enjoy a sleek interface with features like syntax highlighting, light/dark mode, and vim-style navigation.
 - 📂 **Full file visualization**: Instantly view the entire file when selecting any search result.
 
 You can try out our public hosted demo [here](https://demo.sourcebot.dev)!

docs/docs.json

@@ -28,13 +28,21 @@
                         "group": "Features",
                         "pages": [
                             {
-                                "group": "Search",
+                                "group": "Code Search",
                                 "pages": [
+                                    "docs/features/search/overview",
                                     "docs/features/search/syntax-reference",
                                     "docs/features/search/multi-branch-indexing",
                                     "docs/features/search/search-contexts"
                                 ]
                             },
+                            {
+                                "group": "Ask Sourcebot",
+                                "pages": [
+                                    "docs/features/ask/overview",
+                                    "docs/features/ask/add-model-providers"
+                                ]
+                            },
                             "docs/features/code-navigation",
                             "docs/features/analytics",
                             "docs/features/mcp-server",
@@ -51,6 +59,7 @@
                     {
                         "group": "Configuration",
                         "pages": [
+                            "docs/configuration/config-file",
                             {
                                 "group": "Indexing your code",
                                 "pages": [
@@ -66,8 +75,7 @@
                                     "docs/connections/request-new"
                                 ]
                             },
-                            "docs/license-key",
-                            "docs/configuration/environment-variables",
+                            "docs/configuration/language-model-providers",
                             {
                                 "group": "Authentication",
                                 "pages": [
@@ -78,6 +86,8 @@
                                     "docs/configuration/auth/faq"
                                 ]
                             },
+                            "docs/configuration/environment-variables",
+                            "docs/license-key",
                             "docs/configuration/transactional-emails",
                             "docs/configuration/structured-logging",
                             "docs/configuration/audit-logs"

docs/docs/configuration/config-file.mdx

@@ -0,0 +1,49 @@
+---
+title: Config File
+sidebarTitle: Config file
+---
+
+When self-hosting Sourcebot, you **must** provide it a config file. This is done by defining a config file in a volume that's mounted to Sourcebot, and providing the path to this
+file in the `CONFIG_PATH` environment variable. For example:
+
+```bash icon="terminal" Passing in a CONFIG_PATH to Sourcebot
+docker run \
+    -v $(pwd)/config.json:/data/config.json \
+    -e CONFIG_PATH=/data/config.json \
+    ... \ # other options
+    ghcr.io/sourcebot-dev/sourcebot:latest
+```
+
+The config file tells Sourcebot which repos to index, what language models to use, and various other settings as defined in the [schema](#config-file-schema).
+
+# Config File Schema
+
+The config file you provide Sourcebot must follow the [schema](https://github.com/sourcebot-dev/sourcebot/blob/main/schemas/v3/index.json). This schema consists of the following properties:
+
+- [Connections](/docs/connections/overview) (`connections`): Defines a set of connections that tell Sourcebot which repos to index and from where
+- [Language Models](/docs/configuration/language-model-providers) (`models`): Defines a set of language model providers for use with [Ask Sourcebot](/docs/features/ask)
+- [Settings](#settings) (`settings`): Additional settings to tweak your Sourcebot deployment
+- [Search Contexts](/docs/features/search/search-contexts) (`contexts`): Groupings of repos that you can search against
+
+# Config File Syncing
+
+Sourcebot syncs the config file on startup, and automatically whenever a change is detected.
+
+# Settings
+
+The following are settings that can be provided in your config file to modify Sourcebot's behavior
+
+| Setting                                   | Type    | Default    | Minimum | Description / Notes                                                                    |
+|-------------------------------------------|---------|------------|---------|----------------------------------------------------------------------------------------|
+| `maxFileSize`                             | number  | 2 MB       | 1       | Maximum size (bytes) of a file to index. Files exceeding this are skipped.             |
+| `maxTrigramCount`                         | number  | 20 000     | 1       | Maximum trigrams per document. Larger files are skipped.                               |
+| `reindexIntervalMs`                       | number  | 1 hour     | 1       | Interval at which all repositories are re‑indexed.                                     |
+| `resyncConnectionIntervalMs`              | number  | 24 hours   | 1       | Interval for checking connections that need re‑syncing.                                |
+| `resyncConnectionPollingIntervalMs`       | number  | 1 second   | 1       | DB polling rate for connections that need re‑syncing.                                  |
+| `reindexRepoPollingIntervalMs`            | number  | 1 second   | 1       | DB polling rate for repos that should be re‑indexed.                                   |
+| `maxConnectionSyncJobConcurrency`         | number  | 8          | 1       | Concurrent connection‑sync jobs.                                                       |
+| `maxRepoIndexingJobConcurrency`           | number  | 8          | 1       | Concurrent repo‑indexing jobs.                                                         |
+| `maxRepoGarbageCollectionJobConcurrency`  | number  | 8          | 1       | Concurrent repo‑garbage‑collection jobs.                                               |
+| `repoGarbageCollectionGracePeriodMs`      | number  | 10 seconds | 1       | Grace period to avoid deleting shards while loading.                                   |
+| `repoIndexTimeoutMs`                      | number  | 2 hours    | 1       | Timeout for a single repo‑indexing run.                                                |
+| `enablePublicAccess` **(deprecated)**     | boolean | false      | —       | Use the `FORCE_ENABLE_ANONYMOUS_ACCESS` environment variable instead.                  |

docs/docs/configuration/language-model-providers.mdx

@@ -0,0 +1,184 @@
+---
+title: Language Model Providers
+sidebarTitle: Language model providers
+---
+
+To use [Ask Sourcebot](/docs/features/ask) you must define at least one Language Model Provider. These providers are defined within the [config file](/docs/configuration/config-file) you
+provide Sourcebot.
+
+
+```json wrap icon="code" Example config with language model provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        // 1. Google Vertex config for Gemini 2.5 Pro
+        {
+            "provider": "google-vertex",
+            "model": "gemini-2.5-pro",
+            "displayName": "Gemini 2.5 Pro",
+            "project": "sourcebot",
+            "credentials": {
+                "env": "GOOGLE_APPLICATION_CREDENTIALS"
+            }
+        },
+        // 2. OpenAI config for o3 
+        {
+            "provider": "openai",
+            "model": "o3",
+            "displayName": "o3",
+            "token": {
+                "env": "OPENAI_API_KEY"
+            }
+        }
+    ]
+}
+```
+
+# Supported Providers
+
+Sourcebot uses the [Vercel AI SDK](https://ai-sdk.dev/docs/introduction), so it can integrate with any provider the SDK supports. If you don't see your provider below please submit
+a [feature request](https://github.com/sourcebot-dev/sourcebot/discussions/categories/feature-requests).
+
+For a detailed description of all the providers, please refer to the [schema](https://github.com/sourcebot-dev/sourcebot/blob/main/schemas/v3/languageModel.json).
+
+<Note>Any parameter defined using `env` will read the value from the corresponding environment variable you provide Sourcebot</Note>
+
+### Amazon Bedrock
+
+[Vercel AI SDK Amazon Bedrock Docs](https://ai-sdk.dev/providers/ai-sdk-providers/amazon-bedrock)
+
+```json wrap icon="code" Example config with Amazon Bedrock provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "amazon-bedrock",
+            "model": "YOUR_MODEL_HERE",
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "accessKeyId": {
+                "env": "AWS_ACCESS_KEY_ID"
+            },
+            "accessKeySecret": {
+                "env": "AWS_SECRET_ACCESS_KEY"
+            },
+            "region": "YOUR_REGION_HERE", // defaults to the AWS_REGION env var if not set
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```
+
+### Anthropic
+
+[Vercel AI SDK Anthropic Docs](https://ai-sdk.dev/providers/ai-sdk-providers/anthropic)
+
+```json wrap icon="code" Example config with Anthropic provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "anthropic",
+            "model": "YOUR_MODEL_HERE",
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "token": {
+                "env": "ANTHROPIC_API_KEY"
+            },
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```
+
+### Google Generative AI
+
+[Vercel AI SDK Google Generative AI Docs](https://ai-sdk.dev/providers/ai-sdk-providers/google-generative-ai)
+
+```json wrap icon="code" Example config with Google Generative AI provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "google-generative-ai",
+            "model": "YOUR_MODEL_HERE",
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "token": {
+                "env": "GOOGLE_GENERATIVE_AI_API_KEY"
+            },
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```
+
+### Google Vertex
+
+<Note>If you're using an Anthropic model on Google Vertex, you must define a [Google Vertex Anthropic](#google-vertex-anthropic) provider instead</Note>
+<Note>The `credentials` paramater here expects a **path** to a [credentials](https://console.cloud.google.com/apis/credentials) file. This file **must be in a volume mounted by Sourcebot** for it to be readable.</Note>
+
+[Vercel AI SDK Google Vertex AI Docs](https://ai-sdk.dev/providers/ai-sdk-providers/google-vertex)
+
+```json wrap icon="code" Example config with Google Vertex provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "google-vertex",
+            "model": "YOUR_MODEL_HERE", // e.g., "gemini-2.0-flash-exp", "gemini-1.5-pro", "gemini-1.5-flash"
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "project": "YOUR_PROJECT_ID", // defaults to the GOOGLE_VERTEX_PROJECT env var if not set
+            "region": "YOUR_REGION_HERE", // defaults to the GOOGLE_VERTEX_REGION env var if not set, e.g., "us-central1", "us-east1", "europe-west1"
+            "credentials": {
+                "env": "GOOGLE_APPLICATION_CREDENTIALS"
+            },
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```
+
+### Google Vertex Anthropic
+
+<Note>The `credentials` paramater here expects a **path** to a [credentials](https://console.cloud.google.com/apis/credentials) file. This file **must be in a volume mounted by Sourcebot** for it to be readable.</Note>
+
+
+[Vercel AI SDK Google Vertex Anthropic Docs](https://ai-sdk.dev/providers/ai-sdk-providers/google-vertex#google-vertex-anthropic-provider-usage)
+
+```json wrap icon="code" Example config with Google Vertex Anthropic provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "google-vertex-anthropic",
+            "model": "YOUR_MODEL_HERE", // e.g., "claude-sonnet-4"
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "project": "YOUR_PROJECT_ID", // defaults to the GOOGLE_VERTEX_PROJECT env var if not set
+            "region": "YOUR_REGION_HERE", // defaults to the GOOGLE_VERTEX_REGION env var if not set, e.g., "us-central1", "us-east1", "europe-west1"
+            "credentials": {
+                "env": "GOOGLE_APPLICATION_CREDENTIALS"
+            },
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```
+
+### OpenAI
+
+[Vercel AI SDK OpenAI Docs](https://ai-sdk.dev/providers/ai-sdk-providers/openai)
+
+```json wrap icon="code" Example config with OpenAI provider
+{
+    "$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
+    "models": [
+        {
+            "provider": "openai",
+            "model": "YOUR_MODEL_HERE", // e.g., "gpt-4.1", "o4-mini", "o3", "o3-deep-research"
+            "displayName": "OPTIONAL_DISPLAY_NAME",
+            "token": {
+                "env": "OPENAI_API_KEY"
+            },
+            "baseUrl": "OPTIONAL_BASE_URL"
+        }
+    ] 
+}
+```

docs/docs/connections/bitbucket-cloud.mdx

@@ -12,6 +12,8 @@ import BitbucketSchema from '/snippets/schemas/v3/bitbucket.schema.mdx'
 Looking for docs on Bitbucket Data Center? See [this doc](/docs/connections/bitbucket-data-center).
 </Note>
 
+If you're not familiar with Sourcebot [connections](/docs/connections/overview), please read that overview first.
+
 ## Examples
 
 <AccordionGroup>

docs/docs/connections/bitbucket-data-center.mdx

@@ -12,6 +12,8 @@ import BitbucketSchema from '/snippets/schemas/v3/bitbucket.schema.mdx'
 Looking for docs on Bitbucket Cloud? See [this doc](/docs/connections/bitbucket-cloud).
 </Note>
 
+If you're not familiar with Sourcebot [connections](/docs/connections/overview), please read that overview first.
+
 ## Examples
 
 <AccordionGroup>

docs/docs/connections/generic-git-host.mdx

@@ -7,6 +7,8 @@ import GenericGitHost from '/snippets/schemas/v3/genericGitHost.schema.mdx'
 
 Sourcebot can sync code from any Git host (by clone url). This is helpful when you want to search code that not in a [supported code host](/docs/connections/overview#supported-code-hosts).
 
+If you're not familiar with Sourcebot [connections](/docs/connections/overview), please read that overview first.
+
 ## Getting Started
 
 To connect to a Git host, create a new [connection](/docs/connections/overview) with type `git` and specify the clone url in the `url` property. For example:

docs/docs/connections/gerrit.mdx

@@ -10,6 +10,8 @@ import GerritSchema from '/snippets/schemas/v3/gerrit.schema.mdx'
 
 Sourcebot can sync code from self-hosted gerrit instances.
 
+If you're not familiar with Sourcebot [connections](/docs/connections/overview), please read that overview first.
+
 ## Connecting to a Gerrit instance
 
 To connect to a gerrit instance, provide the `url` property to your config: