API endpoint to expose ASTs

[theogiraudet]

Hey!
Maybe this is a bit out-of-scope of Sypglass but I still suggest.
I have a lot of ideas of tools I want to do around datapacks. Still, my main limitation is always parsing/linking steps and co-evolving these parsers/linkers with new Minecraft versions. Especially when it comes to have them for the different datapack resources (functions, advancements, predicates, …) we want to link.
Currently, Spyglass is the reference concerning datapack tooling. It already has these parsers/linkers and numerous maintainers to follow the Minecraft versions. So I think it would be a lost of time to reinvent the wheel.
Do you think it would be possible for Spyglass to have a specific endpoint, out of the LSP protocol, we can query (or even subscribe) to get the object graph of a datapack (including ASTs and CSTs information)? The main idea is to still use Spyglass for all the frontend/LSP tooling with the capability to provide other tools not in the scope of LSP but by taking advantage of what Spyglass already does.

[misode]

An API seems like it would take a lot of server performance to supply everyone's needs. Additionally, sending whole data packs and AST results over the network seems like a waste and could get slow. I think if you could use the @spyglass/ npm packages yourself that would be a better solution.

[theogiraudet]

Thanks for your answer!
Perhaps I expressed myself badly, I'm sorry.
By API, I mean Spyglass exposes something, so through the VSCode extension or wherever we execute it.
In other words, Spyglass is working on a workspace and I want to query the object graph.
So not a public server you are hosting somewhere, and where we can send our documents.
Depending on Spyglass packages may be a solution, but I believe the parser/linker part is not entirely uncoupled from the LSP protocol. So the idea is to use Spyglass as a black-box model source, without having to parse and link again the same documents at the same time from the outside.

[misode]

@SPGoding I know there were plans a while ago to have some kind of extension API. Do you still think this is feasible, and how would we go about doing that?

[SPGoding]

I think last time I thought about this, instead of creating and maintaining some sort of public API, we would just allow Popular Projects (TM) to create contrib-xxx packages under the packages/ directory in this repo and add some config toggles for users to enable their functionalities. One major downside is that it kinda forces the use of TypeScript/JavaScript in those third party projects compared to some sort of language-agnostic form of inter-process communication API similar to LSP (although if we add a WebAssembly compatibility layer, some compiled languages might be used as well), but the upsides are that we don't have to worry about breaking changes and that through observing third party usage of the private APIs we may have a better idea of what people want and be able to provide a better public API if we ever do that in the future.