Merge pull request #2870 from ahoppen/document-environment-variables

Move contributor documentation into a dedicated folder

Author: ahoppen

CONTRIBUTING.md

@@ -1,5 +1,7 @@
 # Contributing
 
+This document contains notes about development and testing of swift-syntax, the [Contributor Documentation](Contributor%20Documentation) folder has some more in-depth documents.
+
 ## Building & Testing
 
 swift-syntax is a SwiftPM package, so you can build and test it using anything that supports packages - opening in Xcode, Visual Studio Code with [Swift for Visual Studio Code](https://github.com/swift-server/vscode-swift) installed, or through the command line using `swift build` and `swift test`.
@@ -13,7 +15,7 @@ swift-syntax is a SwiftPM package, so you can build and test it using anything t
 > - On the command line: Set the `SKIP_LONG_TESTS` environment variable to `1` when running tests, e.g by running `SKIP_LONG_TESTS=1 swift test`
 
 > [!NOTE]
-> If you are only modifying the `SwiftParser` module, you can run only the parser tests by selecting the `SwiftParserTest` target. 
+> If you are only modifying the `SwiftParser` module, you can run only the parser tests by selecting the `SwiftParserTest` target.
 > - In Xcode: Select the `SwiftParserTest` scheme. If you can’t find it in your Schemes, you need to manually add it using Product -> Scheme -> New Scheme…
 > - On the command line: Run `swift test --test-product SwiftParserTests`
 
@@ -30,7 +32,7 @@ It will build a local copy of swift-format from the `main` branch and format the
 Generated source code is not formatted to make it easier to spot changes when re-running code generation.
 
 > [!NOTE]
-> You can add a git hook to ensure all commits to the swift-syntax repository are correctly formatted. 
+> You can add a git hook to ensure all commits to the swift-syntax repository are correctly formatted.
 > 1. Save the following contents to `swift-syntax/.git/hooks/pre-commit`
 > ```bash
 > #!/usr/bin/env bash
@@ -39,7 +41,7 @@ Generated source code is not formatted to make it easier to spot changes when re
 > swift "$SOURCE_DIR/swift-syntax-dev-utils" format --lint
 > ```
 > 2. Mark the file as executable: `chmod a+x swift-syntax/.git/hooks/pre-commit`
-> 3. If you have global git hooks installed, be sure to call them at the end of the script with `path/to/global/hooks/pre-commit "$@"` 
+> 3. If you have global git hooks installed, be sure to call them at the end of the script with `path/to/global/hooks/pre-commit "$@"`
 
 
 ## Generating Source Code
@@ -54,7 +56,7 @@ Alternatively, you can generate the files from the command line by running the f
 
 ## Running Pre-PR Checks Script
 
-To ensure that your changes to the project are implemented correctly and do not introduce issues across the repository, a script has been provided to automate the necessary pre-PR checks. 
+To ensure that your changes to the project are implemented correctly and do not introduce issues across the repository, a script has been provided to automate the necessary pre-PR checks.
 
 ```bash
 ./swift-syntax-dev-utils local-pr-precheck
@@ -69,7 +71,7 @@ Prefer to squash the commits of your PR (*pull request*) and avoid adding commit
 
 We prefer to not squash commits when merging a PR because, especially for larger PRs, it sometimes makes sense to split the PR into multiple self-contained chunks of changes. For example, a PR might do a refactoring first before adding a new feature or fixing a bug. This separation is useful for two reasons:
 - During review, the commits can be reviewed individually, making each review chunk smaller
-- In case this PR introduced a bug that is identified later, it is possible to check if it resulted from the refactoring or the actual change, thereby making it easier find the lines that introduce the issue. 
+- In case this PR introduced a bug that is identified later, it is possible to check if it resulted from the refactoring or the actual change, thereby making it easier find the lines that introduce the issue.
 
 ## Opening a PR
 
@@ -92,17 +94,17 @@ See the [dedicated section][section] on the Swift project website.
 
 After you opened your PR, a maintainer will review it and test your changes in CI (*Continuous Integration*) by adding a `@swift-ci Please test` comment on the pull request. Once your PR is approved and CI has passed, the maintainer will merge your pull request.
 
-Only contributors with [commit access](https://www.swift.org/contributing/#commit-access) are able to approve pull requests and trigger CI. 
+Only contributors with [commit access](https://www.swift.org/contributing/#commit-access) are able to approve pull requests and trigger CI.
 
 ## Additional Verification
 
 swift-syntax has additional verification methods (see the sections below) that provide more extensive validation. They have a significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.
 
 ### RawSyntax Validation
 
-When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax, SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that 
+When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax, SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that
 1. every child of a syntax node has the correct kind, which should be guaranteed by the Swift type system in most cases
-2. each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package. 
+2. each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package.
 
 If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double-check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.
 

CodeGeneration/Sources/generate-swift-syntax/templates/swiftsyntax/SwiftSyntaxDoccIndex.swift

@@ -81,33 +81,6 @@ let nodesSections: String = {
   return result
 }()
 
-var contributingDocs: String = {
-  let contributingDocsFolder = URL(fileURLWithPath: #filePath)
-    .deletingLastPathComponent()
-    .deletingLastPathComponent()
-    .deletingLastPathComponent()
-    .deletingLastPathComponent()
-    .deletingLastPathComponent()
-    .deletingLastPathComponent()
-    .appendingPathComponent("Sources")
-    .appendingPathComponent("SwiftSyntax")
-    .appendingPathComponent("Documentation.docc")
-    .appendingPathComponent("Contributing")
-
-  let files =
-    (try? FileManager.default.contentsOfDirectory(at: contributingDocsFolder, includingPropertiesForKeys: nil)) ?? []
-
-  return files.compactMap { file in
-    if file.pathExtension != "md" {
-      return nil
-    }
-    let doccName = file.lastPathComponent
-      .replacingOccurrences(of: ".md", with: "")
-      .replacingOccurrences(of: " ", with: "-")
-    return "- <doc:\(doccName)>"
-  }.sorted().joined(separator: "\n")
-}()
-
 let swiftSyntaxDoccIndex: String = {
   let templateURL = URL(fileURLWithPath: #filePath)
     .deletingLastPathComponent()
@@ -117,5 +90,4 @@ let swiftSyntaxDoccIndex: String = {
   return
     template
     .replacingOccurrences(of: "{{Nodes}}", with: nodesSections)
-    .replacingOccurrences(of: "{{ContributingDocs}}", with: contributingDocs)
 }()

CodeGeneration/Sources/generate-swift-syntax/templates/swiftsyntax/SwiftSyntaxDoccIndexTemplate.md

@@ -25,12 +25,6 @@ allows Swift tools to parse, inspect, generate, and transform Swift source code.
 
 - <doc:Tutorial-Table-of-Contents>
 
-### Contributing
-
-These articles are intended for developers wishing to contribute to SwiftSyntax
-
-{{ContributingDocs}}
-
 ### Syntax
 
 - <doc:SwiftSyntax/Syntax>

Sources/SwiftSyntax/Documentation.docc/Contributing/ChangingSwiftSyntax.md renamed to Contributor Documentation/Changing Swift Syntax.md

@@ -6,12 +6,12 @@ Discover the steps necessary to add, remove, and update syntax nodes
 
 The surface area of the entire Swift programming language is represented in the
 Swift Syntax API and its associated data structures. In order to keep these
-structures in sync, this project generates code using the Swift package 
-`CodeGeneration`, located in the root of this repository. 
+structures in sync, this project generates code using the Swift package
+`CodeGeneration`, located in the root of this repository.
 
 ## Regenerating Files
 
-To re-generate the files after changing `CodeGeneration` run the `generate-swift-syntax` 
+To re-generate the files after changing `CodeGeneration` run the `generate-swift-syntax`
 target of `CodeGeneration`.
 
 On the command line, this would be
@@ -22,16 +22,16 @@ swift run --package-path CodeGeneration generate-swift-syntax
 ## Adding and Removing Syntax Nodes
 
 The files containing the definition of all of the syntax nodes are available
-under the [SyntaxSupport][SyntaxSupport] directory. These files
+under the [SyntaxSupport](https://github.com/swiftlang/swift-syntax/tree/main/CodeGeneration/Sources/SyntaxSupport) directory. These files
 are roughly divided according to broad syntactic categories in the Swift
 programming language. That is, the syntax nodes for classes, structs, and actors
-are defined in `DeclNodes.swift`, while the syntax nodes for string literals, 
+are defined in `DeclNodes.swift`, while the syntax nodes for string literals,
 arrays, and tuples is defined in `ExprNodes.swift`.
 
-To add a node to these files, it can be helpful to copy an existing node and 
+To add a node to these files, it can be helpful to copy an existing node and
 alter its definition to suit the needs of the new syntax being defined. A syntax
-node consists of a kind (which also defines the node’s name), a base kind, and a list of 
-child nodes. The base kind of a syntax node defines the class of syntax the node belongs to. 
+node consists of a kind (which also defines the node’s name), a base kind, and a list of
+child nodes. The base kind of a syntax node defines the class of syntax the node belongs to.
 All nodes are at least of the `Syntax`
 kind, though some nodes may have a more specific kind like `Stmt` for
 statements or `Expr` for expressions. The SwiftSyntax library expands these
@@ -63,20 +63,16 @@ Node(
 
 ## Committing Changes
 
-Added syntactic elements will require corresponding changes to the included 
-SwiftParser library. For an introduction on parsing Swift nodes, see 
-[the article on Parsing Basics][ParserBasics].
+Added syntactic elements will require corresponding changes to the included
+SwiftParser library. For an introduction on parsing Swift nodes, see
+[the article on Parsing Basics](https://github.com/swiftlang/swift-syntax/tree/main/Sources/SwiftParser/SwiftParser.docc/ParsingBasics.md).
 
 When updating nodes, certain clients of SwiftSyntax that are relying upon those
-nodes will need to be changed in tandem. For example, the 
-[swift-stress-tester][swift-stress-tester] uses SwiftSyntax, and the CI
+nodes will need to be changed in tandem. For example, the
+[swift-stress-tester](https://github.com/swiftlang/swift-stress-tester) uses SwiftSyntax, and the CI
 system will not allow changes to SwiftSyntax that break `swift-stress-tester`
 without a paired change to that repository.
 
-[SyntaxSupport]: https://github.com/swiftlang/swift-syntax/tree/main/CodeGeneration/Sources/SyntaxSupport
-[swift-stress-tester]: https://github.com/swiftlang/swift-stress-tester
-[ParserBasics]: https://github.com/swiftlang/swift-syntax/tree/main/Sources/SwiftParser/SwiftParser.docc/ParsingBasics.md
-
 @Comment {
   When docc resolves https://github.com/apple/swift-docc/issues/255, `ParserBasic` should be an inter-target link.
 }

Contributor Documentation/Environment variables.md

@@ -0,0 +1,15 @@
+# Environment variables
+
+The following environment variables can be used to control some behavior in swift-syntax.
+
+## Build time
+
+- `SWIFT_BUILD_SCRIPT_ENVIRONMENT`: Enable assertions in release builds and remove the dependency of swift-syntax on os_log.
+- `SWIFTCI_USE_LOCAL_DEPS`: Assume that all of SourceKit-LSP’s dependencies are checked out next to it and use those instead of cloning the repositories. Primarily intended for CI environments that check out related branches.
+- `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION`: Mutate the input of `assertParse` test cases. See [CONTRIBUTING](../CONTRIBUTING.md) for more information.
+- `SWIFTSYNTAX_BUILD_DYNAMIC_LIBRARIES`: Build all libraries as *dynamic* instead of *automatic*. This allows us to build swift-syntax as dynamic libraries, which in turn allows us to build SourceKit-LSP using SwiftPM on Windows. Linking swift-syntax statically into sourcekit-lsp exceeds the maximum number of exported symbols on Windows.
+- `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION`: Check that the layout of the syntax tree is correct when creating or modifying nodes. See [CONTRIBUTING](../CONTRIBUTING.md) for more information.
+
+## Testing
+
+- `SKIP_LONG_TESTS`: Skip tests that typically take more than 1s to execute.

Sources/SwiftParser/SwiftParser.docc/FilingBugReports.md renamed to Contributor Documentation/Filing Parser Bug Reports.md

@@ -0,0 +1,12 @@
+# Contributor Documentation
+
+The following documentation documents are primarily intended for developers of swift-syntax.
+
+- [Changing Swift Syntax](Changing%20Swift%20Syntax.md)
+- [Filing Parser Bug Reports](Filing%20Parser%20Bug%20Reports.md)
+- [Fixing Bugs](Fixing%20Bugs.md)
+- [Parser Design](Parser%20Design.md)
+- [Parser Recovery](Parser%20Recovery.md)
+- [Parsing Basics](Parsing%20Basics.md)
+- [When to use protocols in SwiftSyntax](When%20to%20use%20protocols%20in%20SwiftSyntax.md)
+- [`@_spi` Attribute](SPI%20Attribute.md)

Sources/SwiftParser/SwiftParser.docc/FixingBugs.md renamed to Contributor Documentation/Fixing Bugs.md

@@ -4,7 +4,7 @@ A parser for the Swift programming language.
 
 ## Overview
 
-The `SwiftParser` framework implements a parser that accepts Swift source text as input and produces a SwiftSyntax syntax tree. 
+The `SwiftParser` framework implements a parser that accepts Swift source text as input and produces a SwiftSyntax syntax tree.
 
 ## Quickstart
 
@@ -49,8 +49,3 @@ There are several design principles that govern the parser:
 - **Source-preserving**: SwiftSyntax is designed to maintain all “trivia” (including whitespace, comments, etc.) precisely as it occurs in the source text, so that a syntax tree can be rendered back into text that is byte-for-byte identical to the original source. The parser must maintain this property, regardless of whether the input text was well-formed Swift code.
 - **Minimal context**: The parser requires minimal context to parse Swift code, which consists of only those things required to handle a suitable Swift dialect, e.g., whether [regex literals](https://github.com/apple/swift-evolution/blob/main/proposals/0354-regex-literals.md) are supported. The parser can be invoked on any input source code, starting at any major production in the grammar (e.g., full source file, an individual type, an individual expression).
 - **Incremental**: A parse tree produced for a source file can be incrementally updated for a new version of that source file, reusing syntax nodes where possible to reduce computation overhead and memory.
-
-### Development
-
-- <doc:SwiftParser/FilingBugReports>
-- <doc:SwiftParser/FixingBugs>

Sources/SwiftParser/SwiftParser.docc/ParserDesign.md renamed to Contributor Documentation/Parser Design.md

@@ -25,15 +25,6 @@ allows Swift tools to parse, inspect, generate, and transform Swift source code.
 
 - <doc:Tutorial-Table-of-Contents>
 
-### Contributing
-
-These articles are intended for developers wishing to contribute to SwiftSyntax
-
-- <doc:ChangingSwiftSyntax>
-- <doc:Existentials>
-- <doc:SPI>
-- <doc:Swift-Version>
-
 ### Syntax
 
 - <doc:SwiftSyntax/Syntax>