Skip to content

Commit cfe8b8c

Browse files
brendan-kellambrendan-kellam
authored
Search contexts (#273)
1 parent c201a5e commit cfe8b8c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+1573
-677
lines changed

‎CHANGELOG.md

+6
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Added
11+
- [Sourcebot EE] Added search contexts, user-defined groupings of repositories that help focus searches on specific areas of a codebase. [#273](https://github.com/sourcebot-dev/sourcebot/pull/273)
12+
13+
814
## [3.0.4] - 2025-04-12
915

1016
### Fixes

‎LICENSE

+6-2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
1-
MIT License
1+
Copyright (c) 2025 Taqla Inc.
22

3-
Copyright (c) Taqla, Inc.
3+
Portions of this software are licensed as follows:
4+
5+
- All content that resides under the "ee/" and "packages/web/src/ee/" directories of this repository, if these directories exist, is licensed under the license defined in "ee/LICENSE".
6+
- All third party components incorporated into the Sourcebot Software are licensed under the original license provided by the owner of the applicable component.
7+
- Content outside of the above mentioned directories or restrictions above is available under the "MIT Expat" license as defined below.
48

59
Permission is hereby granted, free of charge, to any person obtaining a copy
610
of this software and associated documentation files (the "Software"), to deal

‎Makefile

+1
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ clean:
3333
soft-reset:
3434
rm -rf .sourcebot
3535
redis-cli FLUSHALL
36+
yarn dev:prisma:migrate:reset
3637

3738

3839
.PHONY: bin

‎docs/docs.json

+5-2
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
{
4040
"group": "More",
4141
"pages": [
42+
"docs/more/syntax-reference",
4243
"docs/more/roles-and-permissions"
4344
]
4445
}
@@ -52,7 +53,8 @@
5253
"group": "Getting Started",
5354
"pages": [
5455
"self-hosting/overview",
55-
"self-hosting/configuration"
56+
"self-hosting/configuration",
57+
"self-hosting/license-key"
5658
]
5759
},
5860
{
@@ -61,7 +63,8 @@
6163
"self-hosting/more/authentication",
6264
"self-hosting/more/tenancy",
6365
"self-hosting/more/transactional-emails",
64-
"self-hosting/more/declarative-config"
66+
"self-hosting/more/declarative-config",
67+
"self-hosting/more/search-contexts"
6568
]
6669
},
6770
{

‎docs/docs/more/syntax-reference.mdx

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: Writing search queries
3+
---
4+
5+
Sourcebot uses a powerful regex-based query language that enabled precise code search within large codebases.
6+
7+
8+
## Syntax reference guide
9+
10+
Queries consist of space-separated regular expressions. Wrapping expressions in `""` combines them. By default, a file must have at least one match for each expression to be included.
11+
12+
| Example | Explanation |
13+
| :--- | :--- |
14+
| `foo` | Match files with regex `/foo/` |
15+
| `foo bar` | Match files with regex `/foo/` **and** `/bar/` |
16+
| `"foo bar"` | Match files with regex `/foo bar/` |
17+
18+
Multiple expressions can be or'd together with `or`, negated with `-`, or grouped with `()`.
19+
20+
| Example | Explanation |
21+
| :--- | :--- |
22+
| `foo or bar` | Match files with regex `/foo/` **or** `/bar/` |
23+
| `foo -bar` | Match files with regex `/foo/` but **not** `/bar/` |
24+
| `foo (bar or baz)` | Match files with regex `/foo/` **and** either `/bar/` **or** `/baz/` |
25+
26+
Expressions can be prefixed with certain keywords to modify search behavior. Some keywords can be negated using the `-` prefix.
27+
28+
| Prefix | Description | Example |
29+
| :--- | :--- | :--- |
30+
| `file:` | Filter results from filepaths that match the regex. By default all files are searched. | `file:README` - Filter results to filepaths that match regex `/README/`<br/>`file:"my file"` - Filter results to filepaths that match regex `/my file/`<br/>`-file:test\.ts$` - Ignore results from filepaths match regex `/test\.ts$/` |
31+
| `repo:` | Filter results from repos that match the regex. By default all repos are searched. | `repo:linux` - Filter results to repos that match regex `/linux/`<br/>`-repo:^web/.*` - Ignore results from repos that match regex `/^web\/.*` |
32+
| `rev:` | Filter results from a specific branch or tag. By default **only** the default branch is searched. | `rev:beta` - Filter results to branches that match regex `/beta/` |
33+
| `lang:` | Filter results by language (as defined by [linguist](https://github.com/github-linguist/linguist/blob/main/lib/linguist/languages.yml)). By default all languages are searched. | `lang:TypeScript` - Filter results to TypeScript files<br/>`-lang:YAML` - Ignore results from YAML files |
34+
| `sym:` | Match symbol definitions created by [universal ctags](https://ctags.io/) at index time. | `sym:\bmain\b` - Filter results to symbols that match regex `/\bmain\b/` |
35+
| `context:` | Filter results to a predefined [search context](/self-hosting/more/search-contexts). | `context:web` - Filter results to the web context<br/>`-context:pipelines` - Ignore results from the pipelines context |

‎docs/images/search_contexts_example.png

65.8 KB
Loading

‎docs/self-hosting/license-key.mdx

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
title: License key
3+
sidebarTitle: License key
4+
---
5+
6+
All core Sourcebot features are available in Sourcebot OSS (MIT Licensed). Some additional features require a license key. See the [pricing page](https://www.sourcebot.dev/pricing) for more details.
7+
8+
9+
## Activating a license key
10+
11+
After purchasing a license key, you can activate it by setting the `SOURCEBOT_EE_LICENSE_KEY` environment variable.
12+
13+
```bash
14+
docker run \
15+
-e SOURCEBOT_EE_LICENSE_KEY=<your-license-key> \
16+
/* additional args */ \
17+
ghcr.io/sourcebot-dev/sourcebot:latest
18+
```
19+
20+
## Questions?
21+
22+
If you have any questions regarding licensing, please [contact us](mailto:[email protected]).

‎docs/self-hosting/more/declarative-config.mdx

+4
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@ title: Configuring Sourcebot from a file (declarative config)
33
sidebarTitle: Declarative config
44
---
55

6+
<Warning>
7+
Declaratively defining `connections` is not available when [multi-tenancy](/self-hosting/more/tenancy) is enabled.
8+
</Warning>
9+
610
Some teams require Sourcebot to be configured via a file (where it can be stored in version control, run through CI/CD pipelines, etc.) instead of a web UI. For more information on configuring connections, see this [overview](/docs/connections/overview).
711

812

‎docs/self-hosting/more/search-contexts.mdx

+153
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
title: Search contexts
3+
sidebarTitle: Search contexts (EE)
4+
---
5+
6+
<Note>
7+
This is only available in the Enterprise Edition. Please add your [license key](/self-hosting/license-key) to activate it.
8+
</Note>
9+
10+
A **search context** is a user-defined grouping of repositories that helps focus searches on specific areas of your codebase, like frontend, backend, or infrastructure code. Some example queries using search contexts:
11+
12+
- `context:data_engineering userId` - search for `userId` across all repos related to Data Engineering.
13+
- `context:k8s ingress` - search for anything related to ingresses in your k8's configs.
14+
- `( context:project1 or context:project2 ) logger\.debug` - search for debug log calls in project1 and project2
15+
16+
17+
Search contexts are defined in the `context` object inside of a [declarative config](/self-hosting/more/declarative-config). Repositories can be included / excluded from a search context by specifying the repo's URL in either the `include` array or `exclude` array. Glob patterns are supported.
18+
19+
## Example
20+
21+
Let's assume we have a GitLab instance hosted at `https://gitlab.example.com` with three top-level groups, `web`, `backend`, and `shared`:
22+
23+
```sh
24+
web/
25+
├─ admin_panel/
26+
├─ customer_portal/
27+
├─ pipelines/
28+
├─ ...
29+
backend/
30+
├─ billing_server/
31+
├─ auth_server/
32+
├─ db_migrations/
33+
├─ pipelines/
34+
├─ ...
35+
shared/
36+
├─ protobufs/
37+
├─ react/
38+
├─ pipelines/
39+
├─ ...
40+
```
41+
42+
To make searching easier, we can create three search contexts in our [config.json](/self-hosting/more/declarative-config):
43+
- `web`: For all frontend-related code
44+
- `backend`: For backend services and shared APIs
45+
- `pipelines`: For all CI/CD configurations
46+
47+
48+
```json
49+
{
50+
"$schema": "https://raw.githubusercontent.com/sourcebot-dev/sourcebot/main/schemas/v3/index.json",
51+
"contexts": {
52+
"web": {
53+
// To include repositories in a search context,
54+
// you can reference them...
55+
"include": [
56+
// ... individually by specifying the repo URL.
57+
"gitlab.example.com/web/admin_panel/core",
58+
59+
60+
// ... or as groups using glob patterns. This is
61+
// particularly useful for including entire "sub-folders"
62+
// of repositories in one go.
63+
"gitlab.example.com/web/customer_portal/**",
64+
"gitlab.example.com/shared/react/**",
65+
"gitlab.example.com/shared/protobufs/**"
66+
],
67+
68+
// Same with excluding repositories.
69+
"exclude": [
70+
"gitlab.example.com/web/customer_portal/pipelines",
71+
"gitlab.example.com/shared/react/hooks/**",
72+
],
73+
74+
// Optional description of the search context
75+
// that surfaces in the UI.
76+
"description": "Web related repos."
77+
},
78+
"backend": { /* ... specifies backend replated repos ... */},
79+
"pipelines": { /* ... specifies pipeline related repos ... */ }
80+
},
81+
"connections": {
82+
/* ...connection definitions... */
83+
}
84+
}
85+
```
86+
87+
<Accordion title="Repository URL details">
88+
Repo URLs are expected to be formatted without the leading http(s):// prefix. For example:
89+
- `github.com/sourcebot-dev/sourcebot` ([link](https://github.com/sourcebot-dev/sourcebot))
90+
- `gitlab.com/gitlab-org/gitlab` ([link](https://gitlab.com/gitlab-org/gitlab))
91+
- `chromium.googlesource.com/chromium` ([link](https://chromium-review.googlesource.com/admin/repos/chromium,general))
92+
</Accordion>
93+
94+
95+
Once configured, you can use these contexts in the search bar by prefixing your query with the context name. For example:
96+
- `context:web login form` searches for login form code in frontend repositories
97+
- `context:backend auth` searches for authentication code in backend services
98+
- `context:pipelines deploy` searches for deployment configurations
99+
100+
![Example](/images/search_contexts_example.png)
101+
102+
Like other prefixes, contexts can be negated using `-` or combined using `or`:
103+
- `-context:web` excludes frontend repositories from results
104+
- `( context:web or context:backend )` searches across both frontend and backend code
105+
106+
See [this doc](/docs/more/syntax-reference) for more details on the search query syntax.
107+
108+
## Schema reference
109+
110+
<Accordion title="Reference">
111+
```json
112+
{
113+
"type": "object",
114+
"description": "Search context",
115+
"properties": {
116+
"include": {
117+
"type": "array",
118+
"description": "List of repositories to include in the search context. Expected to be formatted as a URL without any leading http(s):// prefix (e.g., 'github.com/sourcebot-dev/sourcebot'). Glob patterns are supported.",
119+
"items": {
120+
"type": "string"
121+
},
122+
"examples": [
123+
[
124+
"github.com/sourcebot-dev/**",
125+
"gerrit.example.org/sub/path/**"
126+
]
127+
]
128+
},
129+
"exclude": {
130+
"type": "array",
131+
"description": "List of repositories to exclude from the search context. Expected to be formatted as a URL without any leading http(s):// prefix (e.g., 'github.com/sourcebot-dev/sourcebot'). Glob patterns are supported.",
132+
"items": {
133+
"type": "string"
134+
},
135+
"examples": [
136+
[
137+
"github.com/sourcebot-dev/sourcebot",
138+
"gerrit.example.org/sub/path/**"
139+
]
140+
]
141+
},
142+
"description": {
143+
"type": "string",
144+
"description": "Optional description of the search context that surfaces in the UI."
145+
}
146+
},
147+
"required": [
148+
"include"
149+
],
150+
"additionalProperties": false
151+
}
152+
```
153+
</Accordion>

‎ee/LICENSE

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Sourcebot Enterprise license (the “Enterprise License” or "EE license")
2+
Copyright (c) 2025 Taqla Inc.
3+
4+
With regard to the Sourcebot Enterprise Software:
5+
6+
This software and associated documentation files (the "Software") may only be used for
7+
internal business purposes if you (and any entity that you represent) are in compliance
8+
with an agreement governing the use of the Software, as agreed by you and Sourcebot, and otherwise
9+
have a valid Sourcebot Enterprise license for the correct number of user seats. Subject to the foregoing
10+
sentence, you are free to modify this Software and publish patches to the Software. You agree that Sourcebot
11+
and/or its licensors (as applicable) retain all right, title and interest in and to all such modifications
12+
and/or patches, and all such modifications and/or patches may only be used, copied, modified, displayed,
13+
distributed, or otherwise exploited with a valid Sourcebot Enterprise license for the correct number of user seats.
14+
Notwithstanding the foregoing, you may copy and modify the Software for non-production evaluation or internal
15+
experimentation purposes, without requiring a subscription. You agree that Sourcebot and/or
16+
its licensors (as applicable) retain all right, title and interest in and to all such modifications.
17+
You are not granted any other rights beyond what is expressly stated herein. Subject to the
18+
foregoing, it is forbidden to copy, merge, publish, distribute, sublicense, and/or sell the Software.
19+
20+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
21+
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
22+
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
23+
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
24+
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
25+
26+
For all third party components incorporated into the Sourcebot Software, those components are
27+
licensed under the original license provided by the owner of the applicable component.

‎package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
"packages/*"
55
],
66
"scripts": {
7-
"build": "cross-env SKIP_ENV_VALIDATION=1 yarn workspaces run build",
8-
"test": "yarn workspaces run test",
7+
"build": "cross-env SKIP_ENV_VALIDATION=1 yarn workspaces foreach -A run build",
8+
"test": "yarn workspaces foreach -A run test",
99
"dev": "yarn dev:prisma:migrate:dev && npm-run-all --print-label --parallel dev:zoekt dev:backend dev:web",
1010
"with-env": "cross-env PATH=\"$PWD/bin:$PATH\" dotenv -e .env.development -c --",
1111
"dev:zoekt": "yarn with-env zoekt-webserver -index .sourcebot/index -rpc",

‎packages/db/prisma/migrations/20250403044104_add_search_contexts/migration.sql

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
-- CreateTable
2+
CREATE TABLE "SearchContext" (
3+
"id" SERIAL NOT NULL,
4+
"name" TEXT NOT NULL,
5+
"description" TEXT,
6+
"orgId" INTEGER NOT NULL,
7+
8+
CONSTRAINT "SearchContext_pkey" PRIMARY KEY ("id")
9+
);
10+
11+
-- CreateTable
12+
CREATE TABLE "_RepoToSearchContext" (
13+
"A" INTEGER NOT NULL,
14+
"B" INTEGER NOT NULL,
15+
16+
CONSTRAINT "_RepoToSearchContext_AB_pkey" PRIMARY KEY ("A","B")
17+
);
18+
19+
-- CreateIndex
20+
CREATE UNIQUE INDEX "SearchContext_name_orgId_key" ON "SearchContext"("name", "orgId");
21+
22+
-- CreateIndex
23+
CREATE INDEX "_RepoToSearchContext_B_index" ON "_RepoToSearchContext"("B");
24+
25+
-- AddForeignKey
26+
ALTER TABLE "SearchContext" ADD CONSTRAINT "SearchContext_orgId_fkey" FOREIGN KEY ("orgId") REFERENCES "Org"("id") ON DELETE CASCADE ON UPDATE CASCADE;
27+
28+
-- AddForeignKey
29+
ALTER TABLE "_RepoToSearchContext" ADD CONSTRAINT "_RepoToSearchContext_A_fkey" FOREIGN KEY ("A") REFERENCES "Repo"("id") ON DELETE CASCADE ON UPDATE CASCADE;
30+
31+
-- AddForeignKey
32+
ALTER TABLE "_RepoToSearchContext" ADD CONSTRAINT "_RepoToSearchContext_B_fkey" FOREIGN KEY ("B") REFERENCES "SearchContext"("id") ON DELETE CASCADE ON UPDATE CASCADE;

‎packages/db/prisma/schema.prisma

+17
Original file line numberDiff line numberDiff line change
@@ -61,9 +61,24 @@ model Repo {
6161
org Org @relation(fields: [orgId], references: [id], onDelete: Cascade)
6262
orgId Int
6363
64+
searchContexts SearchContext[]
65+
6466
@@unique([external_id, external_codeHostUrl, orgId])
6567
}
6668

69+
model SearchContext {
70+
id Int @id @default(autoincrement())
71+
72+
name String
73+
description String?
74+
repos Repo[]
75+
76+
org Org @relation(fields: [orgId], references: [id], onDelete: Cascade)
77+
orgId Int
78+
79+
@@unique([name, orgId])
80+
}
81+
6782
model Connection {
6883
id Int @id @default(autoincrement())
6984
name String
@@ -138,6 +153,8 @@ model Org {
138153
139154
/// List of pending invites to this organization
140155
invites Invite[]
156+
157+
searchContexts SearchContext[]
141158
}
142159

143160
enum OrgRole {

0 commit comments

Comments
 (0)