Update: Version 2.0 → Migrate to pygls + Microsoft's lsprotocol + Save Document

[hlouzada]

Overview

To align with Spyder IDE's remote development features (part of CZI 2022-316698 grant), the version 2.0 of the python's LSP server needs to support the latest LSP protocol version and allow to save document edditions on remote machines (enableling remote editing for Spyder). With this requirements, instead of relying on the current custom protocol implementation, it will be adopted Microsoft’s official lsprotocol package for Python. Also, the server will be refactored to use pygls for handling base TCP communications, features management and document management. This changes lays the foundation for a server more scalable, maintainable and robust, especially for remote environments.

Key Features

• Asyncio-based foundation with pygls
  The server leverages pygls as its base implementation, providing a robust asyncio-driven framework for building language servers in Python. This includes built-in support for feature and command registration, multiple connection modes (TCP, STDIO, WebSocket), and automatic handling of document lifecycle events (open, change, close).

• Standards-compliant typing with lsprotocol
  By adopting lsprotocol, the server uses officially maintained, up-to-date Python types that align directly with the evolving LSP specification. This ensures cleaner versioning, greater consistency, and reduced maintenance overhead compared to proprietary type definitions.

• Atomic document commit mechanism
  To enhance reliability in remote development workflows (such as Spyder), the server supports an opt-in atomic commit feature. Instead of persisting edits immediately, the server can hold them in memory until a custom command (e.g., spyder/commitEdits) is issued. This enables atomic persistence of edits, improving resilience against latency and partial failures.

Roadmap

1. Migrate to pygls

   • Replace the custom LSP scaffolding with pygls to serve as the event loop and request/notification dispatcher.
   • Ensure existing LSP features (completion, hover, diagnostics, etc.) are re-registered using pygls’s feature/command decorators with current plugin architecture.
   • Configure entry-points for TCP, STDIO, and WebSocket connection modes for compatibility with current server API.

2. Refactor configuration and plugin architecture

   • Adopt lsprotocol types across the plugins codebase for functions signatures ensuring alignment with the official LSP specification.
   • Type check all plugins to ensure proper specification.
   • Redesign configuration loading and plugin interfaces to extend the pygls base server, allowing external modules to register features more cleanly.

3. Implement atomic document commit support

   • Maintain in-memory buffers of document changes rather than writing them immediately to disk.
   • Introduce a custom command (e.g., server/commitEdits) to atomically persist pending edits when requested.

[ccordoba12]

@krassowski, perhaps you could be interested on this one since JupyterLab-LSP is the other main user of this project besides Spyder.

Although not mentioned by @hlouzada, we created an additional branch for this refactoring (pylsp-2.0). That way we'll be able to keep releasing fixes and small additions for the current implementation until everything is working fine at least on Spyder.