You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
about: "Lokesh T. R. is a Google Summer Of Code 2024 Student from Tamil Nadu, India, who worked on SourceKitLSP and the Swift Extension for Visual Studio Code. He has a wide range of interests including App & Web Development, Programming Languages & Developer Tools, AI & Machine Learning and many more."
479
+
about: "Lokesh T. R. is a Google Summer Of Code 2024 Student from Tamil Nadu, India, who worked on SourceKit-LSP and the Swift Extension for Visual Studio Code. He has a wide range of interests including App & Web Development, Programming Languages & Developer Tools, AI & Machine Learning and many more."
Copy file name to clipboardexpand all lines: _posts/2030-12-28-expansion-of-swift-macros-in-vscode.md
+32-58
Original file line number
Diff line number
Diff line change
@@ -35,9 +35,7 @@ And as a bonus, we also worked on supporting macro expansions in other LSP-based
35
35
36
36
### When can you start using this feature?
37
37
38
-
This will be available with SourceKit-LSP bundled with Swift 6.1 and the corresponding VS Code Swift Extension Release.
39
-
40
-
For the curious minds, This feature is available in the `main` branch of `sourcekit-lsp` and `vscode-swift` repositories, right now.
38
+
This will be available with SourceKit-LSP bundled with **Swift 6.1 and the corresponding VS Code Swift Extension Release**. For the curious minds, This feature is available in the `main` branch of `sourcekit-lsp` and `vscode-swift` repositories, right now.
41
39
42
40
## Implementation Details
43
41
@@ -52,7 +50,7 @@ Let's have a look at three key components that you use in your everyday life in
52
50
- communicates using Language Server Protocol (LSP)
53
51
3. SourceKitD (Background Service):
54
52
- provides the raw data and operations to SourceKit-LSP
55
-
-baked into the swift compiler
53
+
-shares implementation details with the Swift compiler
-`"workspace/peekDocuments"` allows the SourceKit-LSP Server to show the the contents stored in the `locations` inside the source file `uri` as a peek window
96
-
- Reference Document URL Scheme `sourcekit-lsp://` so that we can use it to encode the necessary data required to generate any form of content to which the URL corresponds to.
97
-
- We introduce the very first `document-type` of the Reference Document URL which is `swift-macro expansion`. This will encode all the necessary data required to generate the macro expansion contents.
98
-
-`"workspace/getReferenceDocument"` is introduced so that the Editor Client can make a request to the SourceKit-LSP Server with the Reference Document URL to fetch its contents.
93
+
#### Working of the "Expand Macro" Code Action
99
94
100
-
The way this works is that, we generate the Reference Document URLs from the macro expansions generated using sourcekitd and make a `"workspace/peekDocuments"` request to the Editor Client. In VS Code, this executes the `"editor.action.peekLocations"` command to present a peeked editor.
95
+
1. The `"workspace/peekDocuments"` request is sent from SourceKit-LSP to the editor and prompts it to show the the contents of `locations` inside the source file `uri` as a peek window at `position`. In VS Code, this executes the `"editor.action.peekLocations"` command to present a peeked editor.
96
+
2. Since the contents of the macro expansion are not represented by a file on disk, we introduce a custom `sourcekit-lsp://` URL scheme to represent the macro expansion buffers. In the future, this URL scheme can be represented for other use cases such as generated interfaces by using a `document-type` other than `swift-macro-expansion`.
97
+
macro expansion contents.
98
+
3. To fetch the contents of this custom URL scheme, the editor client sends a `"workspace/getReferenceDocument"` to SourceKit-LSP.
101
99
102
-

100
+

103
101
104
-
Since VS Code can't resolve the contents of Reference Document URL, it makes a `"workspace/getReferenceDocument"` request to the SourceKit-LSP Server, thereby retrieveing the contents and successfully displaying it in the peeked editor.
102
+
### Stretch Goals
105
103
106
-

104
+
#### Achieving Semantic Functionality (jump-to-definition, quick help on hover, syntax highlighting, etc.)
107
105
108
-
### Stretch Goals
106
+
By default, SourceKit-LSP and SourceKitD are not equipped to handle our newly introduced Reference Document URLs. To enable semantic functionality, such as jump-to-definition, quick help on hover, and syntax highlighting, the build arguments of the file are required. To address this, we utilised the source file's build arguments and applied them to the reference documents. This effectively tricks sourcekitd into providing semantic functionality for these reference documents.
109
107
110
-
1. Achieving Semantic Functionality (jump-to-definition, quick help on hover, syntax highlighting, etc.):
111
-
- SourceKit-LSP and SourceKitD by default doesn't know how to handle Reference Document URLs.
112
-
- The build arguments of a file is needed to provide semantic functionality.
113
-
- We used the source file's build arguments as build arguments of the reference documents to trick sourcekitd to provide Semantic Functionality for the reference documents.
114
-
2. Achieving Nested Macro Expansion:
115
-
- Due to the flexible nature of the Reference Document URLs, nested macro expansions becomes trivial.
116
-
- The beauty of the Reference Document URLs is that we can nest the Reference Document URLs.
117
-
- We set the `parent` parameter of the macro expansion reference document URL to the source file if it's a first level macro expansion or to the reference document from which the macro expansion originates if it was second or third or n-th level macro expansion. This allows us to expand nested macros efficiently.
108
+
#### Achieving Nested Macro Expansion
109
+
110
+
The flexible nature of Reference Document URLs simplifies the process of nested macro expansions. A key advantage of these URLs is their ability to allow nesting. To achieve nested macro expansion, we set the `parent` parameter of the macro expansion reference document URL to the source file if it represents a first-level macro expansion. For subsequent expansions, whether at the second, third, or n-th level, the `parent` parameter points to the reference document from which the macro expansion originates. This mechanism enables the efficient expansion of nested macros.
118
111
119
112
### Bonus
120
113
121
-
While the new LSP Extensions which we introduced, doesn't work out-of-the-box in other LSP-based editors and requires the extension / plugin developers of respective editors to make use of it, We worked on providing a basic form of first level macro expansion support using the standard LSP Requests to support other LSP-based editors.
114
+
While the new LSP extensions which we introduced, don't work out-of-the-box in other LSP-based editors and requires the extension / plugin developers of respective editors to make use of it, we worked on providing a basic form of first level macro expansion support using the standard LSP Requests to support other LSP-based editors.
122
115
123
116
This works as follows:
124
117
125
-
1.SourceKitLSP asks SourceKitD for macro expansions.
118
+
1.SourceKit-LSP asks sourcekitd for macro expansions.
126
119
2. It then stitches all the macro expansions together in a single file.
127
120
3. It stores the file in some temporary location in the disk.
128
121
4. It then makes a `ShowDocumentRequest` to open and show the file in the editor.
129
122
130
-
That wraps up the GSoC project successfully!
131
-
132
-
### What's left to do?
133
-
134
-
- I will be working on implementing a test case that encompasses all the semantic features in all nested macro levels in sourcekit-lsp.
135
-
- I will also implement some end-to-end test cases in the vscode-swift side which ensures that they really work as intended in a real world situation.
123
+
## Conclusion
136
124
137
-
Test cases for freestanding macros, attached macros and nested macros are already in place.
138
-
Code Documentation is also in place for everything that we have implemented so far.
125
+
That wraps up our Google Summer of Code project successfully! 🎉
139
126
140
127
### Future Directions
141
128
142
-
1. Feedback, Feedback, Feedback!
129
+
#### 1. Feedback, Feedback, Feedback
143
130
144
-
- We had put so much attention to detail and ensured that every decision made in the design process was thoughtful.
145
-
- But, you may have a better idea, or you may find some issues, and we would love to improve this feature.
146
-
- Please file an issue to suggest an idea or report bugs in the sourcekit-lsp or vscode-swift repository wherever you face the issue.
131
+
We put a great deal of attention to detail, ensuring that every decision in the design process was thoughtful and deliberate. However, we understand that you might have better ideas or come across issues we haven't anticipated, and we're always eager to improve this feature. If you have suggestions or encounter any bugs, we encourage you to file an issue in either the sourcekit-lsp or vscode-swift repository, depending on where you face the problem.
147
132
148
-
2.Migrating the non-standard `"workspace/getReferenceDocument"` to the new standard `"workspace/textDocumentContent"` Request
133
+
#### 2. Implementing Semantic Functionality and Nested Macro Expansions for all LSP-based editors
149
134
150
-
- We should be able to perform this migration when the specifications of LSP 3.18 gets finalised.
151
-
- With this, the custom request need not be handled by the extension / plugin developer of an LSP-based editor.
135
+
With LSP 3.18 nearing finalization, there should be focus on unifying how macro expansions and reference documents are handled across LSP-based editors to enable semantic functionality and nested macro expansions.
152
136
153
-
3. Migrate from generating temporary files in the Disk to Reference Document URLs for other LSP-based editors
137
+
One of the key steps in this process is migrating from the custom `"workspace/getReferenceDocument"` request to the upcoming standardized `"workspace/textDocumentContent"` request. This will simplify the integration, removing the need for extension / plugin developers to handle custom requests and aligning with LSP specifications. Additionally, once LSP 3.18 is implemented, we can eliminate the reliance on temporary file storage by replacing it with the generated on-the-fly Reference Document URLs.
154
138
155
-
- We are currently unable to generate macro expansions on-the-fly in other LSP-based editors since we don't have our LSP Extension for getting the contents of the reference document.
156
-
- Building on top of the previous idea, when LSP 3.18 gets finalised, we should be able to completely eliminate temporary file storage in favour of the standard `"workspace/textDocumentContent"` Request.
139
+
A significant challenge will involve handling positional shifts in line and character numbers caused by macro expansions. When macros expand, they often change the layout of the file, causing subsequent macros to shift from their original positions. Implementing semantic functionality and nested macro expansions in macro expansion documents for all LSP-based editors should account for these shifts, ensuring that editors can accurately track and display the correct positions of expanded macros. This adjustment is crucial to providing smooth, error-free navigation and proper context across all LSP-based editors.
157
140
158
-
4. Adding Semantic Functionality & Nested Macro Expansion support for other LSP-based editors
141
+
#### 3. Other Use Cases for the Reference Document URL
159
142
160
-
- This is tricky to implement since the temporary file (or reference document after LSP 3.18) will have all the macro expansions of a given macro in a single file.
161
-
- Although the same approach can be used such as passing the source file's build arguments and `parent` to be the macro's originating file, there will be line and character position shifts that should be taken into consideration.
162
-
- For example, the third macro expansion of a given attached macro will be expected to start at `0:0` but its actual location will be shifted by the first and second macro expansion's content length and three lines of comments that describes where the macro will be present in the original file
143
+
Reference Document URLs were built from the ground up to encode the data necessary to display any form of content. This functionality allows anyone to present any content of their choice, whether in a peeked editor or a fully open document, as long as it is generated during compile time. One such example we discussed today is the `swift-macro-expansion` document type.
163
144
164
-
5. Other Use Cases for the Reference Document URL
145
+
Also, For instance, we can migrate `OpenInterfaceRequest` to Reference Document URLs to display the generated `.swiftinterface` files of modules. Additionally, we can show implicitly generated constructors and synthesized code related to `Equatable`, `Hashable`, and `Codable` conformances. Another use case involves previewing or rendering generated HTML from the Mustache template engine by integrating its CLI.
165
146
166
-
- Reference Document URLs where built from the ground up to allow for encoding the data required to show any form of content, one such example which we discussed today is `swift-macro-expansion` document type.
167
-
- This should allow anyone to show any content of their choice, in a peeked editor or a fully open document, as long as its generated during compile time.
168
-
- The following use cases are not related to macro expansions but uses the Reference Document URL that we created to show other document types:
169
-
- Migrating `OpenInterfaceRequest` to Reference Document URLs to show Swift Generated Interfaces
170
-
- Showing Implicitly generated constructors and Synthesized code upon `Equatable`, `Hashable` and `Codable` Conformances.
171
-
- Showing a preview of generated HTML or rendering the generated HTML from the Mustache template engine by hooking up its CLI.
172
-
- These are just few examples, and the fact that you can show various document formats based on various code generation behaviours brings a wide range of possibilities, and hence, you can bring your own idea.
173
-
- And with LSP 3.18, this will be standardised across all editors, not just VS Code.
147
+
These are just a few examples. The ability to present various document formats based on different code generation behaviors opens up a wide range of possibilities, encouraging you to bring your own ideas to life. With LSP 3.18, this functionality will be standardized across all editors, not just VS Code.
174
148
175
-
## Thanks & Gratitude
149
+
###Thanks & Gratitude
176
150
177
151
I offer my deepest gratitude to my mentors, Alex Hoppen (@ahoppen) and Adam Fowler (@adam-fowler) without whom this journey is impossible. Google Summer of Code is not only the work of myself as a contributor but also the work of my mentors in guiding me and helping me out whenever possible.
178
152
179
153
Thanks to you both for accepting my project proposal, spending hours setting up my environment, getting me started with PRs, your wonderful ideas and feedback on my ideas, attending the weekly meetings, allowing me to be flexible, detailed PR reviews, giving me directions and all the immediate and quick responses. I hope I gave you both a wonderful GSoC experience.
180
154
181
-
If not for you two people, this project wouldn't be a success.
155
+
If not for you two people, this project wouldn't be a success. 💖
182
156
183
-
### Special Thanks
157
+
####Special Thanks ❤️
184
158
185
159
- Thanks to Fredrik Wieczerkowski (@fwcd) for his initial proof-of-concept on Expansion of Swift Macros in Visual Studio Code and also for his Pull Request that migrates `workspace/getReferenceDocument` to the upcoming `workspace/textDocumentContent`.
186
160
- Thanks to Paul LeMarquand (@plemarquand) for testing out my project in its very early stages.
187
161
- Thanks to Douglas Gregor (@douglas_gregor) and Rintaro Ishizaki (@rintaro) for coming up with their own use cases with the generic LSP Requests that also laid the foundation for macro expansions.
188
162
- Thanks to Mateusz Bąk (@Matejkob) and Paris Pittman (@parispittman) for fueling me with positivity to present my project wherever possible.
189
-
- Thanks to the entire Swift Community always being so welcoming and supportive.
163
+
- Thanks to the entire Swift Community for always being so welcoming and supportive.
0 commit comments