Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove single-{ handling entirely - jinja across the board #1510

Merged
merged 3 commits into from
Jan 29, 2025

Conversation

mattgodbolt
Copy link
Member

There are several changes in this PR which you can look at individually:

  1. Work to add a ce_install config-dump command which dumps a sequence of JSON lines of the entire expanded configuration of everything.
  2. A change where (carefully) all the { were turned into jinja {{ and }} etc. At various stages of this change, and most importantly at the end, the output of ce_install config-dump was compared and no actual config changes occurred.
  3. The ability to use single { was removed from the config loader, and tests updated and added.

This should allow single quotes to be used as normal, and in some cases {% raw %} type things (though this is not perfect for that, we may need more work, I have ideas on how to fix that but let me know if this is enough to start with @partouf.

Some junky editor config (both from intellij and vscode) made it into the change too, I can extract if it's annoying.

Example of the first few lines of the config dump output:

{"name": "compilers/ada/gnat/arm 10.3.0-2", "type": "TarballInstallable(compilers/ada/gnat/arm 10.3.0-2, arm/gnat-arm-elf-linux64-10.3.0-2)", "config": {"check_file": "", "install_path": "arm/gnat-arm-elf-linux64-10.3.0-2", "language": "ada", "name": "compilers/ada/gnat/arm 10.3.0-2", "remove_older_pattern": "", "target_name": "10.3.0-2", "untar_path": "gnat-arm-elf-linux64-10.3.0-2", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-10.3.0-2/gnat-arm-elf-linux64-10.3.0-2.tar.gz"}}
{"name": "compilers/ada/gnat/arm 11.2.0-3", "type": "TarballInstallable(compilers/ada/gnat/arm 11.2.0-3, arm/gnat-arm-elf-linux64-11.2.0-3)", "config": {"check_file": "", "install_path": "arm/gnat-arm-elf-linux64-11.2.0-3", "language": "ada", "name": "compilers/ada/gnat/arm 11.2.0-3", "remove_older_pattern": "", "target_name": "11.2.0-3", "untar_path": "gnat-arm-elf-linux64-11.2.0-3", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-11.2.0-3/gnat-arm-elf-linux64-11.2.0-3.tar.gz"}}
{"name": "compilers/ada/gnat/riscv64 10.3.0-2", "type": "TarballInstallable(compilers/ada/gnat/riscv64 10.3.0-2, riscv64/gnat-riscv64-elf-linux64-10.3.0-2)", "config": {"check_file": "", "install_path": "riscv64/gnat-riscv64-elf-linux64-10.3.0-2", "language": "ada", "name": "compilers/ada/gnat/riscv64 10.3.0-2", "remove_older_pattern": "", "target_name": "10.3.0-2", "untar_path": "gnat-riscv64-elf-linux64-10.3.0-2", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-10.3.0-2/gnat-riscv64-elf-linux64-10.3.0-2.tar.gz"}}
{"name": "compilers/ada/gnat/riscv64 11.2.0-3", "type": "TarballInstallable(compilers/ada/gnat/riscv64 11.2.0-3, riscv64/gnat-riscv64-elf-linux64-11.2.0-3)", "config": {"check_file": "", "install_path": "riscv64/gnat-riscv64-elf-linux64-11.2.0-3", "language": "ada", "name": "compilers/ada/gnat/riscv64 11.2.0-3", "remove_older_pattern": "", "target_name": "11.2.0-3", "untar_path": "gnat-riscv64-elf-linux64-11.2.0-3", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-11.2.0-3/gnat-riscv64-elf-linux64-11.2.0-3.tar.gz"}}
{"name": "compilers/nightly/dex2oat dex2oat-latest", "type": "ScriptInstallable(compilers/nightly/dex2oat dex2oat-latest, dex2oat-latest)", "config": {"check_file": "x86_64/bin/dex2oat64", "install_path": "dex2oat-latest", "language": "nightly", "name": "compilers/nightly/dex2oat dex2oat-latest", "script": "mkdir dex2oat-latest\ncd dex2oat-latest\n/home/matthew/dev/ce/infra/bin/yaml/android-java/fetch_art_release_from_head.sh\nunzip art_release.zip\nrm art_release.zip\n/home/matthew/dev/ce/infra/bin/yaml/android-java/create_boot_images.sh", "target_name": "dex2oat-latest"}}
{"name": "compilers/android-d8 8.1.56", "type": "SingleFileInstallable(compilers/android-d8 8.1.56, r8-8.1.56)", "config": {"check_file": "", "filename": "r8-8.1.56.jar", "install_path": "r8-8.1.56", "language": "android-d8", "name": "compilers/android-d8 8.1.56", "target_name": "8.1.56", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.1.56/r8-8.1.56.jar"}}
{"name": "compilers/android-d8 8.1.72", "type": "SingleFileInstallable(compilers/android-d8 8.1.72, r8-8.1.72)", "config": {"check_file": "", "filename": "r8-8.1.72.jar", "install_path": "r8-8.1.72", "language": "android-d8", "name": "compilers/android-d8 8.1.72", "target_name": "8.1.72", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.1.72/r8-8.1.72.jar"}}
{"name": "compilers/android-d8 8.2.33", "type": "SingleFileInstallable(compilers/android-d8 8.2.33, r8-8.2.33)", "config": {"check_file": "", "filename": "r8-8.2.33.jar", "install_path": "r8-8.2.33", "language": "android-d8", "name": "compilers/android-d8 8.2.33", "target_name": "8.2.33", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.33/r8-8.2.33.jar"}}
{"name": "compilers/android-d8 8.2.42", "type": "SingleFileInstallable(compilers/android-d8 8.2.42, r8-8.2.42)", "config": {"check_file": "", "filename": "r8-8.2.42.jar", "install_path": "r8-8.2.42", "language": "android-d8", "name": "compilers/android-d8 8.2.42", "target_name": "8.2.42", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.42/r8-8.2.42.jar"}}
{"name": "compilers/android-d8 8.2.47", "type": "SingleFileInstallable(compilers/android-d8 8.2.47, r8-8.2.47)", "config": {"check_file": "", "filename": "r8-8.2.47.jar", "install_path": "r8-8.2.47", "language": "android-d8", "name": "compilers/android-d8 8.2.47", "target_name": "8.2.47", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.47/r8-8.2.47.jar"}}

To repro you can:

$ git checkout be49fae707c516553b108a72f2859ac3070a7927
$ ce_install config-dump /tmp/before
$ git checkout 50e3cc1b9fd8b7f02b73d0e50248de20d18a3a7e
$ ce_install config-dump /tmp/after
$ diff /tmp/before /tmp/after
$

@mattgodbolt mattgodbolt requested a review from partouf January 28, 2025 20:57
@mattgodbolt
Copy link
Member Author

See #1476

@mattgodbolt mattgodbolt merged commit 8c1545d into main Jan 29, 2025
5 checks passed
@mattgodbolt mattgodbolt deleted the mg/support_escapes branch January 29, 2025 16:00
mattgodbolt added a commit to steveschnepp/infra that referenced this pull request Jan 29, 2025
…r-explorer#1510)

- Work to add a `ce_install config-dump` command which dumps a sequence of JSON lines of the entire expanded configuration of everything.
-  A change where (carefully) all the `{` were turned into jinja `{{` and `}}` etc. At various stages of this change, and most importantly at the end, the output of `ce_install config-dump` was compared and no actual config changes occurred.
- The ability to use single `{` was removed from the config loader, and tests updated and added.

This should allow single quotes to be used as normal, and in some cases `{% raw %}` type things (though this is not perfect for that, we may need more work, I have ideas on how to fix that but let me know if this is enough to start with @partouf.

Some junky editor config (both from intellij and vscode) made it into the change too, I can extract if it's annoying.

Example of the first few lines of the config dump output:
```json
{"name": "compilers/ada/gnat/arm 10.3.0-2", "type": "TarballInstallable(compilers/ada/gnat/arm 10.3.0-2, arm/gnat-arm-elf-linux64-10.3.0-2)", "config": {"check_file": "", "install_path": "arm/gnat-arm-elf-linux64-10.3.0-2", "language": "ada", "name": "compilers/ada/gnat/arm 10.3.0-2", "remove_older_pattern": "", "target_name": "10.3.0-2", "untar_path": "gnat-arm-elf-linux64-10.3.0-2", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-10.3.0-2/gnat-arm-elf-linux64-10.3.0-2.tar.gz"}}
{"name": "compilers/ada/gnat/arm 11.2.0-3", "type": "TarballInstallable(compilers/ada/gnat/arm 11.2.0-3, arm/gnat-arm-elf-linux64-11.2.0-3)", "config": {"check_file": "", "install_path": "arm/gnat-arm-elf-linux64-11.2.0-3", "language": "ada", "name": "compilers/ada/gnat/arm 11.2.0-3", "remove_older_pattern": "", "target_name": "11.2.0-3", "untar_path": "gnat-arm-elf-linux64-11.2.0-3", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-11.2.0-3/gnat-arm-elf-linux64-11.2.0-3.tar.gz"}}
{"name": "compilers/ada/gnat/riscv64 10.3.0-2", "type": "TarballInstallable(compilers/ada/gnat/riscv64 10.3.0-2, riscv64/gnat-riscv64-elf-linux64-10.3.0-2)", "config": {"check_file": "", "install_path": "riscv64/gnat-riscv64-elf-linux64-10.3.0-2", "language": "ada", "name": "compilers/ada/gnat/riscv64 10.3.0-2", "remove_older_pattern": "", "target_name": "10.3.0-2", "untar_path": "gnat-riscv64-elf-linux64-10.3.0-2", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-10.3.0-2/gnat-riscv64-elf-linux64-10.3.0-2.tar.gz"}}
{"name": "compilers/ada/gnat/riscv64 11.2.0-3", "type": "TarballInstallable(compilers/ada/gnat/riscv64 11.2.0-3, riscv64/gnat-riscv64-elf-linux64-11.2.0-3)", "config": {"check_file": "", "install_path": "riscv64/gnat-riscv64-elf-linux64-11.2.0-3", "language": "ada", "name": "compilers/ada/gnat/riscv64 11.2.0-3", "remove_older_pattern": "", "target_name": "11.2.0-3", "untar_path": "gnat-riscv64-elf-linux64-11.2.0-3", "untar_to": ".", "url": "https://github.com/alire-project/GNAT-FSF-builds/releases/download/gnat-11.2.0-3/gnat-riscv64-elf-linux64-11.2.0-3.tar.gz"}}
{"name": "compilers/nightly/dex2oat dex2oat-latest", "type": "ScriptInstallable(compilers/nightly/dex2oat dex2oat-latest, dex2oat-latest)", "config": {"check_file": "x86_64/bin/dex2oat64", "install_path": "dex2oat-latest", "language": "nightly", "name": "compilers/nightly/dex2oat dex2oat-latest", "script": "mkdir dex2oat-latest\ncd dex2oat-latest\n/home/matthew/dev/ce/infra/bin/yaml/android-java/fetch_art_release_from_head.sh\nunzip art_release.zip\nrm art_release.zip\n/home/matthew/dev/ce/infra/bin/yaml/android-java/create_boot_images.sh", "target_name": "dex2oat-latest"}}
{"name": "compilers/android-d8 8.1.56", "type": "SingleFileInstallable(compilers/android-d8 8.1.56, r8-8.1.56)", "config": {"check_file": "", "filename": "r8-8.1.56.jar", "install_path": "r8-8.1.56", "language": "android-d8", "name": "compilers/android-d8 8.1.56", "target_name": "8.1.56", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.1.56/r8-8.1.56.jar"}}
{"name": "compilers/android-d8 8.1.72", "type": "SingleFileInstallable(compilers/android-d8 8.1.72, r8-8.1.72)", "config": {"check_file": "", "filename": "r8-8.1.72.jar", "install_path": "r8-8.1.72", "language": "android-d8", "name": "compilers/android-d8 8.1.72", "target_name": "8.1.72", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.1.72/r8-8.1.72.jar"}}
{"name": "compilers/android-d8 8.2.33", "type": "SingleFileInstallable(compilers/android-d8 8.2.33, r8-8.2.33)", "config": {"check_file": "", "filename": "r8-8.2.33.jar", "install_path": "r8-8.2.33", "language": "android-d8", "name": "compilers/android-d8 8.2.33", "target_name": "8.2.33", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.33/r8-8.2.33.jar"}}
{"name": "compilers/android-d8 8.2.42", "type": "SingleFileInstallable(compilers/android-d8 8.2.42, r8-8.2.42)", "config": {"check_file": "", "filename": "r8-8.2.42.jar", "install_path": "r8-8.2.42", "language": "android-d8", "name": "compilers/android-d8 8.2.42", "target_name": "8.2.42", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.42/r8-8.2.42.jar"}}
{"name": "compilers/android-d8 8.2.47", "type": "SingleFileInstallable(compilers/android-d8 8.2.47, r8-8.2.47)", "config": {"check_file": "", "filename": "r8-8.2.47.jar", "install_path": "r8-8.2.47", "language": "android-d8", "name": "compilers/android-d8 8.2.47", "target_name": "8.2.47", "url": "https://dl.google.com/android/maven2/com/android/tools/r8/8.2.47/r8-8.2.47.jar"}}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant