Skip to content

Commit 5510b8a

Browse files
geroplgeropl
authored
[memory-bank] Update with more details (#20639)
* [memory-bank] Tell cline how to build components Tool: gitpod/catfood.gitpod.cloud * [memory-bank] Document API components as well Tool: gitpod/catfood.gitpod.cloud * [memory-bank] Document content-service-api, ide-metrics-api, ide-service-api Tool: gitpod/catfood.gitpod.cloud * [memory-bank] Document image-builder-api, local-app-api, registry-facade-api Tool: gitpod/catfood.gitpod.cloud * [memory-bank] Document supervisor-api, usage-api, ws-daemon-api Tool: gitpod/catfood.gitpod.cloud * [memory-bank] Document ws-manager-api, ws-manager-bridge-api Tool: gitpod/catfood.gitpod.cloud
1 parent 0cba458 commit 5510b8a

15 files changed

+1721
-23
lines changed

‎memory-bank/.clinerules

+46-10
Original file line numberDiff line numberDiff line change
@@ -26,19 +26,35 @@ This file captures important patterns, preferences, and project intelligence tha
2626

2727
## Development Workflow
2828

29-
- **Build System**: Use Leeway for building components
30-
- `leeway build` to build specific components
31-
- `leeway exec` to run commands across components
29+
- **Build Approaches**:
30+
- **In-tree builds** (primary for local development):
31+
- TypeScript components: Use `yarn` commands defined in package.json
32+
- `yarn build`: Compile the component
33+
- `yarn test`: Run tests
34+
- `yarn lint`: Check code style
35+
- `yarn watch`: Watch for changes and rebuild
36+
- Go components: Use standard Go tools
37+
- `go build ./...`: Build all packages
38+
- `go test ./...`: Test all packages
39+
- `go run main.go`: Build and run
40+
41+
- **Out-of-tree builds** (Leeway, primary for CI):
42+
- `leeway build components/component-name:app`: Build a specific component
43+
- `leeway build -D components/component-name:app`: Build with dependencies
44+
- `leeway exec --package components/component-name:app -- command`: Run a command for a package
3245

3346
- **Testing**:
3447
- Unit tests alongside code
3548
- Integration tests in separate directories
3649
- End-to-end tests in the `test/` directory
50+
- Component-specific test commands in package.json (for TypeScript)
51+
- Go tests use standard `go test` command
3752

3853
- **Local Development**:
3954
- Use Gitpod workspaces for development (dogfooding)
4055
- Components can be run individually for testing
4156
- Preview environments for testing changes
57+
- Use in-tree builds for rapid iteration during development
4258

4359
## Critical Implementation Paths
4460

@@ -76,9 +92,12 @@ This file captures important patterns, preferences, and project intelligence tha
7692
- **Memory Bank Documentation**:
7793
- Component-specific documentation is stored in `memory-bank/components/` directory
7894
- Each component gets its own markdown file with detailed information about its purpose, architecture, and implementation
79-
- API components (with "*-api" suffix) should NOT be documented separately as they are typically just interface definitions
80-
- Component documentation should focus on the main service components that implement business logic
95+
- Component documentation should focus on both service components that implement business logic and API components that define interfaces
8196
- Documentation follows a consistent structure with sections for Overview, Purpose, Architecture, Key Features, etc.
97+
- API component documentation should include a "Code Generation and Building" section that explains:
98+
- How to regenerate code from protobuf definitions
99+
- The implementation details of the generation process
100+
- How to build components that depend on the API after regeneration
82101

83102
## Evolution of Project Decisions
84103

@@ -92,16 +111,33 @@ This section will be updated as I learn about how and why certain architectural
92111
3. Make it easier to find information about specific components
93112
4. Enable incremental updates to component documentation without affecting other files
94113

95-
- **API Component Exclusion**: The decision to exclude "*-api" components from separate documentation was made because:
96-
1. API components primarily contain interface definitions, not implementation logic
97-
2. The interfaces are typically documented within the main service component that implements them
98-
3. Documenting API components separately would create redundancy
99-
4. The focus should be on documenting the service components that implement business logic
114+
- **API Component Documentation**: The decision to include "*-api" components in separate documentation was made because:
115+
1. API components define critical interfaces between services
116+
2. Understanding these interfaces is essential for developers working across components
117+
3. API documentation helps clarify system boundaries and communication patterns
118+
4. Separate documentation makes it easier to track API changes and versioning
100119

101120
## User Preferences
102121

103122
This section will be updated as I learn about specific user preferences for working with the codebase.
104123

124+
## Build and Test Information
125+
126+
When working with components, I should always document:
127+
128+
- **Build Commands**: Document any new or component-specific build commands I encounter
129+
- **Test Commands**: Document how to run tests for each component
130+
- **Dependencies**: Note any special dependencies required for building or testing
131+
- **Common Issues**: Document common build or test issues and their solutions
132+
- **Performance Considerations**: Note any performance considerations for builds
133+
134+
Whenever I encounter new build patterns or commands, I should update:
135+
1. The relevant component documentation in `memory-bank/components/`
136+
2. The `.clinerules` file with general patterns
137+
3. The `techContext.md` file if it represents a significant pattern
138+
139+
This information is critical for effective development work, as being able to build and test components is fundamental to making changes and verifying their correctness.
140+
105141
---
106142

107143
Note: This file will be continuously updated as I work with the Gitpod codebase and discover new patterns, preferences, and insights.

‎memory-bank/activeContext.md

+33-4
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Key areas of focus include:
1818
- Created detailed documentation for key components:
1919
- blobserve: Service that provides static assets from OCI images
2020
- content-service: Manages various types of content within the platform
21-
- dashboard: Web-based user interface for Gitpod
21+
- dashboard: Web-based user inSterface for Gitpod
2222
- ws-manager-mk2: Kubernetes controller for managing workspace lifecycle
2323
- supervisor: Init process that runs inside each workspace container
2424
- ws-daemon: Node-level daemon for workspace operations
@@ -49,6 +49,23 @@ Key areas of focus include:
4949
- openvsx-proxy: Caching proxy service for the OpenVSX registry
5050
- scheduler-extender: Extends Kubernetes scheduling capabilities for workspaces
5151
- ipfs: Provides distributed content-addressable storage for container images
52+
- Created documentation for API components:
53+
- content-service-api: Interfaces for managing workspace content, blobs, logs, and IDE plugins
54+
- ide-metrics-api: Interfaces for collecting metrics and error reports from IDE components
55+
- ide-service-api: Interfaces for managing IDE configurations and resolving workspace IDE requirements
56+
- image-builder-api: Interfaces for building Docker images for workspaces
57+
- local-app-api: Interfaces for communication between local machines and remote workspaces
58+
- registry-facade-api: Interfaces for dynamically assembling workspace container images
59+
- supervisor-api: Interfaces for workspace management, terminal handling, and port forwarding
60+
- usage-api: Interfaces for tracking, calculating, and managing workspace usage and billing
61+
- ws-daemon-api: Interfaces for workspace content management and container operations
62+
- ws-manager-api: Interfaces for managing the lifecycle of workspaces in Kubernetes clusters
63+
- ws-manager-bridge-api: Interfaces for dynamic management of workspace clusters
64+
- Enhanced API component documentation with code generation information:
65+
- Added details on how to regenerate code from protobuf definitions
66+
- Documented the implementation details of the generation process
67+
- Included instructions for building components after code regeneration
68+
- Updated .clinerules to standardize API documentation with code generation sections
5269

5370
As work progresses, this section will continue to be updated to reflect:
5471
- Additional component documentation
@@ -63,9 +80,11 @@ The immediate next steps are:
6380

6481
1. **Explore Component Interactions**: Understand how components interact with each other
6582
2. **Set Up Development Environment**: Configure a local development environment for effective testing
66-
3. **Identify Initial Tasks**: Determine specific tasks or improvements to focus on
67-
4. **Establish Testing Approach**: Define how changes will be tested and validated
68-
5. **Update Memory Bank**: Continue to refine and expand the memory bank as new information is discovered
83+
3. **Explore Build System**: Gain hands-on experience with both in-tree and Leeway builds
84+
4. **Test Component Builds**: Practice building and testing different component types
85+
5. **Identify Initial Tasks**: Determine specific tasks or improvements to focus on
86+
6. **Establish Testing Approach**: Define how changes will be tested and validated
87+
7. **Update Memory Bank**: Continue to refine and expand the memory bank as new information is discovered
6988

7089
## Active Decisions and Considerations
7190

@@ -115,5 +134,15 @@ Initial exploration of the Gitpod codebase has revealed:
115134
- Go services typically have a cmd/ directory with command implementations
116135
- TypeScript services use React and modern frontend practices
117136
- Most components have a clear separation between API definitions and implementations
137+
- **Build System Approaches**: Gitpod uses two primary approaches for building components:
138+
- **In-tree builds**: Using language-specific tools (yarn, go) directly in the workspace
139+
- Primary method for local development
140+
- TypeScript components use commands defined in package.json (yarn build, yarn test, etc.)
141+
- Go components use standard Go tools (go build, go test, etc.)
142+
- **Out-of-tree builds**: Using Leeway, a custom build tool
143+
- Primary method for CI to generate build artifacts
144+
- Works by copying relevant sources into a separate file tree
145+
- Can also be run from inside the workspace
146+
- Manages complex dependencies between components
118147

119148
This section will be continuously updated as new insights are gained through working with the system.

‎memory-bank/components/content-service-api.md

+127
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# Content Service API
2+
3+
## Overview
4+
The Content Service API defines the gRPC interfaces for the Content Service, which is responsible for managing various types of content within the Gitpod platform, including workspace files, blobs, logs, and IDE plugins.
5+
6+
## Purpose
7+
This API provides a standardized interface for other components to interact with the Content Service, enabling operations such as:
8+
- Managing workspace content (downloading, deleting, checking snapshots)
9+
- Handling blob storage (uploading, downloading, deleting)
10+
- Accessing headless logs
11+
- Managing IDE plugins
12+
- Defining workspace initialization methods
13+
14+
## Architecture
15+
The Content Service API is implemented as a set of gRPC services defined in Protocol Buffer files. These definitions are used to generate client and server code in various languages (Go, TypeScript) for use by other components in the system.
16+
17+
## Key Services
18+
19+
### ContentService
20+
Provides core content management functionality:
21+
- `DeleteUserContent`: Deletes all content associated with a user
22+
23+
### WorkspaceService
24+
Manages workspace content:
25+
- `WorkspaceDownloadURL`: Provides a URL from which workspace content can be downloaded
26+
- `DeleteWorkspace`: Deletes the content of a single workspace
27+
- `WorkspaceSnapshotExists`: Checks whether a workspace snapshot exists
28+
29+
### BlobService
30+
Handles blob storage operations:
31+
- `UploadUrl`: Provides a URL to which clients can upload content via HTTP PUT
32+
- `DownloadUrl`: Provides a URL from which clients can download content via HTTP GET
33+
- `Delete`: Deletes uploaded content
34+
35+
### HeadlessLogService
36+
Manages access to headless logs:
37+
- `LogDownloadURL`: Provides a URL from which logs can be downloaded
38+
- `ListLogs`: Returns a list of task IDs for a specified workspace instance
39+
40+
### IDEPluginService
41+
Manages IDE plugins:
42+
- `UploadURL`: Provides a URL for uploading plugin content
43+
- `DownloadURL`: Provides a URL for downloading plugin content
44+
- `PluginHash`: Provides a hash of a plugin
45+
46+
## Key Data Structures
47+
48+
### WorkspaceInitializer
49+
Defines how a workspace is initialized, with several initialization methods:
50+
- `EmptyInitializer`: Creates an empty workspace
51+
- `GitInitializer`: Initializes from a Git repository
52+
- `SnapshotInitializer`: Initializes from a snapshot
53+
- `PrebuildInitializer`: Combines snapshots with Git
54+
- `CompositeInitializer`: Uses multiple initializers in sequence
55+
- `FileDownloadInitializer`: Downloads files for workspace content
56+
- `FromBackupInitializer`: Initializes from a backup
57+
58+
### GitStatus
59+
Describes the current Git working copy status, similar to a combination of "git status" and "git branch".
60+
61+
## Communication Patterns
62+
- The API uses gRPC for efficient, typed communication between services
63+
- Many operations return URLs rather than content directly, allowing for efficient transfer of large files
64+
- Authentication information is passed in request messages
65+
66+
## Dependencies
67+
- Used by various components that need to interact with workspace content
68+
- Depends on underlying storage systems (not specified in the API)
69+
70+
## Usage Examples
71+
- Workspace Manager uses this API to initialize workspaces
72+
- Dashboard uses this API to provide download links for workspace content
73+
- IDE components use this API to access plugins
74+
75+
## Version Compatibility
76+
The API uses Protocol Buffers version 3 (proto3) syntax, which provides forward and backward compatibility features.
77+
78+
## Code Generation and Building
79+
80+
### Regenerating Code from Protobuf Definitions
81+
The Content Service API uses Protocol Buffers and gRPC for defining interfaces. When changes are made to the `.proto` files, the corresponding code in various languages needs to be regenerated.
82+
83+
To regenerate the code:
84+
85+
1. Navigate to the content-service-api directory:
86+
```bash
87+
cd components/content-service-api
88+
```
89+
90+
2. Run the generate script:
91+
```bash
92+
./generate.sh
93+
```
94+
95+
This script performs the following actions:
96+
- Installs necessary dependencies (protoc plugins)
97+
- Generates Go code using `protoc-gen-go` and `protoc-gen-go-grpc`
98+
- Generates TypeScript code
99+
- Updates license headers
100+
101+
### Implementation Details
102+
The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`, which provides common functionality for all API components:
103+
104+
- `install_dependencies`: Installs required protoc plugins
105+
- `protoc_buf_generate`: Uses the `buf` tool to generate code based on the configuration in `buf.gen.yaml`
106+
- `update_license`: Updates license headers in generated files
107+
108+
### Building After Code Generation
109+
After regenerating the code, you may need to rebuild components that depend on the Content Service API. This typically involves:
110+
111+
1. For Go components:
112+
```bash
113+
cd <component-directory>
114+
go build ./...
115+
```
116+
117+
2. For TypeScript components:
118+
```bash
119+
cd <component-directory>
120+
yarn install
121+
yarn build
122+
```
123+
124+
3. Using Leeway (for CI/CD):
125+
```bash
126+
leeway build -D components/<component-name>:app
127+
```

0 commit comments

Comments
 (0)