Create a new "bootstrap" toolchain and compile with 2.1.0 #1244

restingbull · 2024-12-14T03:02:50Z

Given that Kotlin 2.1.0 now has a version metadata check during compilation, the idea of bootstrapping kotlin by using the previous is pretty much dead. Probably not the worst thing, as bzlmod did not enjoy the shennanigans necessary to make it work.

This adds a new "cli" toolchain that invokes kotlinc via the established command line interface. Additionally, it sketches out a simpler, more contained toolchain api to build the more complex usecases (such as multiplex workers, jdeps, compiler plugins, etc.) on top of. the end goal is a more agile rule approach -- with the performance optimization (jdeps, class path reductions, etc) tucked behind the simple api.Compiler plugins, linting, and the like should be able to live comfortably in the "production" rule and toolchain combinations.

It also introduces new "core" rules, while limited in scope, should be IDE friendly and generally easier to work with than the previous bootstrapping effort -- and since they use the common providers, they will interoperate with the existing rules.

Changes:

Workaround change in reflection type resolution for jdk type aliases. see Tracking Issue: Kotlin 2.1.20 #1246
Update a few dependencies due to byte code shifts.

MODULE.bazel

WORKSPACE.dev.bazel

Given that Kotlin 2.1.0 now has a version metadata check during compilation, the idea of bootstrapping kotlin by using the previous is pretty much dead. Probably not the worst thing, as bzlmod did not enjoy the shennanigans necessary to make it work. This adds a new "cli" toolchain that invokes kotlinc via the established command line interface. Additionally, it sketches out a simpler, more contained toolchain api to build the more complex usecases (such as multiplex workers, jdeps, compiler plugins, etc.) on top of. the end goal is a more agile rule approach -- with the performance optimization (jdeps, class path reductions, etc) tucked behindthe simple api.Compiler plugins, linting, and the like should be able to live comfortably in the "production" rule and toolchain combinations. It also introduces new "core" rules, while limited in scope, should be IDE friendly and generally easier to work with than the previous bootstrapping effort -- and since they use the common providers, they will interoperate with the existing rules.

Still some outstanding issues iwth the core rules test.

Bencodes · 2024-12-23T18:47:34Z

.bazelci/presubmit.yml

@@ -31,8 +31,8 @@ tasks:
  integration_tests:
    name: "Integration Tests"
    platform: ${{ integration_platform }}
-    test_flags:
-      - "--config=rbe"
+#    test_flags:


Did you mean to leave this one commented out still?

Bencodes · 2024-12-23T18:57:56Z

MODULE.bazel

-)
-use_repo(
-    rules_kotlin_bootstrap_extensions,
-    "released_com_github_google_ksp",


Are these not being synced anymore for some reason?

I removed the build-kotlin-with kotlin, so we don't use the released rules anymore.

Bencodes · 2024-12-23T18:59:17Z

examples/deps/WORKSPACE

    ],
    repositories = [
        "https://maven.google.com",
        "https://repo1.maven.org/maven2",
    ],
 )

+register_toolchains("//bzl:experimental_toolchain")


Why does this need to be registered here in the example app? Is this requirement moving forward?

Defining a toolchain and not registering it means it will never be chosen.

App didn't compile without the toolchain -- complained about all sorts of things.

Bencodes · 2024-12-23T19:10:17Z

kotlin/internal/toolchains.bzl

-            doc = "the -jvm_target flag. This is only tested at 1.8.",
-            default = "1.8",
+            doc = "the -jvm_target flag.",
+            default = "11",


I'll do this against the main branch, but there are a few more JDK releases that we can add here.

Got tired of the warnings. Will revert.

Bencodes · 2024-12-23T19:10:52Z

kotlin/internal/defs.bzl

@@ -23,30 +29,16 @@ load(
 TOOLCHAIN_TYPE = "%s" % Label("//kotlin/internal:kt_toolchain_type")

 # Java toolchains
-JAVA_TOOLCHAIN_TYPE = "@bazel_tools//tools/jdk:toolchain_type"
-JAVA_RUNTIME_TOOLCHAIN_TYPE = "@bazel_tools//tools/jdk:runtime_toolchain_type"
+JAVA_TOOLCHAIN_TYPE = _JAVA_TOOLCHAIN_TYPE


Nice clean up here

Bencodes · 2024-12-23T19:13:48Z

src/main/starlark/core/compile/cli/toolchain.bzl

+        language_version = ".".join(ctx.attr.api_version.split(".")[:2]),
+        executable_zip = ctx.attr.zip[DefaultInfo].files_to_run,
+        kotlinc = ctx.attr.kotlinc[DefaultInfo].files_to_run,
+        compile_mnemonic = "CliKotlinc",


Testing this locally, i'm seeing a TON of console noise from this action:

logging: loading modules: [java.se, com.azul.tooling, jdk.accessibility, jdk.attach, jdk.compiler, jdk.dynalink, jdk.httpserver, jdk.jartool, jdk.javadoc, jdk.jconsole, jdk.jdi, jdk.jfr, jdk.jshell, jdk.jsobject, jdk.management, jdk.management.jfr, jdk.naming.ldap, jdk.net, jdk.scripting.nashorn, jdk.sctp, jdk.security.auth, jdk.security.jgss, jdk.unsupported, jdk.unsupported.desktop, jdk.xml.dom, java.base, java.compiler, java.datatransfer, java.desktop, java.xml, java.instrument, java.logging, java.management, java.management.rmi, java.rmi, java.naming, java.net.http, java.prefs, java.scripting, java.security.jgss, java.security.sasl, java.sql, java.transaction.xa, java.sql.rowset, java.xml.crypto, jdk.internal.jvmstat, jdk.management.agent, jdk.jdwp.agent, jdk.internal.ed, jdk.internal.le, jdk.internal.opt] INFO: From CliKotlinc src/main/kotlin/io/bazel/kotlin/builder/cmd/build_lib.jar: logging: using Kotlin home directory /private/var/tmp/_bazel_blee/705a39c26a4d07ab15f533d9e4d9df56/external/_main~rules_kotlin_extensions~com_github_jetbrains_kotlin_git logging: using JDK home directory external/rules_java~~toolchains~local_jdk logging: exception on loading scripting plugin: java.lang.ClassNotFoundException: org.jetbrains.kotlin.scripting.compiler.plugin.ScriptingCompilerConfigurationComponentRegistrar logging: using JVM IR backend logging: configuring the compilation environment logging: loading modules: [java.se, com.azul.tooling, jdk.accessibility, jdk.attach, jdk.compiler, jdk.dynalink, jdk.httpserver, jdk.jartool, jdk.javadoc, jdk.jconsole, jdk.jdi, jdk.jfr, jdk.jshell, jdk.jsobject, jdk.management, jdk.management.jfr, jdk.naming.ldap, jdk.net, jdk.scripting.nashorn, jdk.sctp, jdk.security.auth, jdk.security.jgss, jdk.unsupported, jdk.unsupported.desktop, jdk.xml.dom, java.base, java.compiler, java.datatransfer, java.desktop, java.xml, java.instrument, java.logging, java.management, java.management.rmi, java.rmi, java.naming, java.net.http, java.prefs, java.scripting, java.security.jgss, java.security.sasl, java.sql, java.transaction.xa, java.sql.rowset, java.xml.crypto, jdk.internal.jvmstat, jdk.management.agent, jdk.jdwp.agent, jdk.internal.ed, jdk.internal.le, jdk.internal.opt]

Ah, right verbose actually does something with K2.

Bencodes · 2024-12-23T20:39:24Z

src/main/starlark/core/compile/cli/BUILD.bazel

+    ],
+    kotlinc = ":kotlinc",
+    language_version = versions.KOTLIN_CURRENT_COMPILER_RELEASE.version,
+    zip = "@bazel_tools//tools/zip:zipper",


I'd just make this the toolchain default instead of making it customizable. Could even make it internal with _zip.

After getting kicked off using Bazel's jarjar, I'm leaving options open.

Bencodes · 2024-12-23T20:40:47Z

src/main/starlark/core/compile/cli/compile.bzl

+    )
+    needs_runfiles = "0" if java_bin_path.startswith("/") or (len(java_bin_path) > 2 and java_bin_path[1] == ":") else "1"
+
+    actions.expand_template(


This can probably be simplified a bunch by using java_binary here. Less re-creating the wheel with the launcher code.

Added for completeness of the toolchain. If you want complete freedom to play with the kotlin compiler, you could use this.

Basically, it's a thin wrapper around kotlinc -- it can't support persistent workers (at least I don't see an easy path on it) -- but it will always be forward compatible (until they change the cli interface... I give it two major versions). It's the "no warranty" rules and toolchain for the discerning kotlin developer. I've even added basic plugin support: #1251

Bencodes · 2024-12-23T22:31:10Z

...ain/starlark/core/repositories/kotlin/capabilities_2.1.bzl.com_github_jetbrains_kotlin.bazel

@@ -0,0 +1,16 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.


the options are missing from here and it needs to be registered in src/main/starlark/core/repositories/compiler.bzl, otherwise all of the 2.1 options won't be usable.

Need to finish automating the process. :p

Bencodes

👍 pending #1244 (comment)

Apparently, the reflection subtlely changed as well.

restingbull force-pushed the restingbull/bootstrap-again branch 2 times, most recently from bad7f65 to d54801b Compare December 17, 2024 05:42

Bencodes reviewed Dec 17, 2024

View reviewed changes

MODULE.bazel Outdated Show resolved Hide resolved

Bencodes reviewed Dec 17, 2024

View reviewed changes

WORKSPACE.dev.bazel Outdated Show resolved Hide resolved

restingbull force-pushed the restingbull/bootstrap-again branch 2 times, most recently from 49fb292 to aba42a9 Compare December 17, 2024 19:50

restingbull requested a review from Bencodes December 17, 2024 19:50

restingbull marked this pull request as ready for review December 17, 2024 23:44

restingbull requested review from ahumesky, cgruber and nkoroste as code owners December 17, 2024 23:44

restingbull force-pushed the restingbull/bootstrap-again branch 5 times, most recently from 17792ec to efec0d5 Compare December 21, 2024 01:32

restingbull added 14 commits December 20, 2024 20:04

Add neverlink and export support.

125ceb7

Able to compile with 2.1.0.

c66eed3

Still some outstanding issues iwth the core rules test.

Fix naming of outputs in test.

23ecc6d

Add cli toolchain to workspace

e0f20d5

Cleanup lint

c712839

Update maven deps

10b48d6

Workaround change in reflection resolution in 2.1.0. Fixed in 2.1.20

3b7bffd

Pass stdlib to clit compiler runtime.

3f80a00

Remove integration test as examples now covers all cases.

1a70251

Include coroutines, as the compiler needs them to run

b036cf2

Remove kotlin released references.

7a13ebc

Add 2.1 capabilities

e2c7fc0

Update docs, add provider to the release

2ea9a15

restingbull added 10 commits December 20, 2024 20:05

Update runner to use override_module when correct

5f09d1b

switch off rbe

79b04ce

Fix workspace cycles

fb7c090

Fix ordering on deps, update compose version

0b7da29

Map compse versions, fix deps toolchain version

ae72932

update jetpack again

b0b3abd

Finally get a working set of libraries

fbd257c

Add pnning mechanism and hard code workspace

aca7ee1

Remove stdlib. Should pulled from toolchain

ff12c2c

Work around bazelbuild/bazel#21225

c40d2e5

restingbull force-pushed the restingbull/bootstrap-again branch from efec0d5 to c40d2e5 Compare December 21, 2024 02:05

Bencodes reviewed Dec 23, 2024

View reviewed changes

Re-enable rbe for integration tests, turn off verbose

51b753b

Bencodes approved these changes Dec 23, 2024

View reviewed changes

restingbull added 2 commits December 24, 2024 09:57

Update options

e78858a

Apparently, the reflection subtlely changed as well.

revert jvm default for another pr

42c2f78

restingbull force-pushed the restingbull/bootstrap-again branch from 29ed523 to 42c2f78 Compare December 24, 2024 16:15

restingbull added 2 commits December 24, 2024 10:17

Add capabilities

9088ba7

Add capabilities and update docs

cd7ca3e

restingbull merged commit 7a988ee into master Dec 24, 2024
1 of 2 checks passed

restingbull deleted the restingbull/bootstrap-again branch December 24, 2024 17:00

erikkerber mentioned this pull request Jan 3, 2025

Bzlmod Workflow for developing against an updated released_ rules_kotlin? #1237

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a new "bootstrap" toolchain and compile with 2.1.0 #1244

Create a new "bootstrap" toolchain and compile with 2.1.0 #1244

restingbull commented Dec 14, 2024 •

edited

Loading

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes Dec 23, 2024

restingbull Dec 23, 2024

Bencodes left a comment

		@@ -0,0 +1,16 @@
		# Copyright 2024 The Bazel Authors. All rights reserved.

Create a new "bootstrap" toolchain and compile with 2.1.0 #1244

Create a new "bootstrap" toolchain and compile with 2.1.0 #1244

Conversation

restingbull commented Dec 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bencodes left a comment

Choose a reason for hiding this comment

restingbull commented Dec 14, 2024 •

edited

Loading