Compilation error only happening during Dokka build

[CLOVIS-AI]

I am facing a compilation error when generating the Dokka output. However, the problem does not appear when compiling the project normally (./gradlew check).

It seems Dokka cannot find the standard library, and complains about many symbols from it:

e: file:///…/src/commonMain/kotlin/Parameters.kt:46:31 Unresolved reference: HashMap
e: file:///…/src/commonMain/kotlin/Parameters.kt:120:3 Unresolved reference: JvmInline
e: file:///…/commonMain/kotlin/Parameters.kt:141:4 Unresolved reference: require
e: file:///…/src/commonMain/kotlin/Parameters.kt:141:17 Unresolved reference: isNotBlank

---

Reproduction:

• Clone the repository https://gitlab.com/opensavvy/spine.git
• Checkout commit a5dede751c40d143b671b12e33df2ac769e3fdfd
• Run ./gradlew check ← passes
• Run ./gradlew :dokkatooGeneratePublicationHtml ← fails with compilation error

The full output of the failed execution is available here.

[aSemy]

Thanks for the report! As we discussed I think this related to #165. I'm not sure though, so it needs some investigation.

[aSemy]

Thanks for the reproducer, I got the same problem. I added a workaround for #165, and tested your project with Dokkatoo 2.2.0-SNAPSHOT, and it solved the issue and Dokkatoo generation succeeded. I had to do some other fiddling around though, because the version of Dokkatoo was set in the remote convention plugins. Can you double check that 2.2.0-SNAPSHOT fixes the issue?

[CLOVIS-AI]

In which repository can I find 2.2.0-SNAPSHOT? My project isn't finding it.

The same problem has appeared in another of my projects, but this time it's complaining about Arrow, not the standard library; reproducer:

• https://gitlab.com/opensavvy/pedestal.git
• Commit c0e3ccea

I had to do some other fiddling around though, because the version of Dokkatoo was set in the remote convention plugins.

Yeah, the best way to test other versions is to includeBuild the repository https://gitlab.com/opensavvy/automation/gradle-conventions.

[aSemy]

In which repository can I find 2.2.0-SNAPSHOT? My project isn't finding it.

Maven Central snapshots

// settings.gradle.kts

pluginManagement {
  repositories {
    mavenCentral()
    gradlePluginPortal()

    // add Maven Central snapshot repository
    maven("https://s01.oss.sonatype.org/content/repositories/snapshots/") {
      name = "MavenCentralSnapshots"
      mavenContent { snapshotsOnly() }
    }
  }
}

[CLOVIS-AI]

It doesn't look like 2.2.0-SNAPSHOT fixes the problem for Pedestal :/

I need to make reproduction easier though, I had to add the repository in a bunch of places.

[CLOVIS-AI]

After updating to 2.2.0, I don't see the problem anymore on the Spine repository. However, I still see it on the Pedestal repository.

Reproduction:

• Clone https://gitlab.com/opensavvy/pedestal.git, branch tests
• In Pedestal's repository, run ./gradlew :dokkatooGeneratePublicationHtml

Note that ./gradlew allTests does work.

To switch the Dokkatoo version:

• Clone https://gitlab.com/opensavvy/automation/gradle-conventions.git
• In Pedestal's settings.gradle.kts, add includeBuild("../path-to-gradle-conventions")
• Edit Gradle Convention's gradle/libs.versions.toml
• If you need to add repositories, there is a section for this at the top of each settings.gradle.kts (no need to edit any build.gradle.kts file)

[CLOVIS-AI]

It's now propagating to all my projects one-by-one 😅 It's now visible on https://gitlab.com/opensavvy/ui/compose-material3-tailwind.git by merging mirroring-17639 into main.

[aSemy]

Thanks! I really appreciate you digging into it.

I can reproduce it but I have no idea where to start with debugging it :(

It also looks like #173 didn't help with #165.

The only idea I have is to majorly refactor how Dokkatoo works, which is something I had in mind anyway.

Currently in multi-subproject builds, Dokkatoo generates the 'partial' modules in each subproject using the Gradle Worker API. I think this is less performant, because each 'generateModule' task only sends one task to the Worker API, so it's missing out on parallelising work.

Instead, Dokkatoo could gather the components needed to build a module, send them to the aggregating project, which would then be able to use a single Task to generate all modules with the Worker API.

This approach would also help with sharing the Dokka parameters. Modules would be able to 'inherit' parameters defined in the aggregating project #111.

Unfortunately, sharing files between subprojects in Gradle is such a phenomenal ball ache, it's is easier said than done. There's just so many bugs, poorly designed and documented APIs, and mysterious behaviour. Gradle is absolutely infuriating. I experimented with it, but it's so overwhelming I gave up.

[CLOVIS-AI]

Unfortunately, sharing files between subprojects in Gradle is such a phenomenal ball ache, it's is easier said than done. There's just so many bugs, poorly designed and documented APIs, and mysterious behaviour. Gradle is absolutely infuriating.

Ahah, yeah, I discovered that when working on this. I'm glad the final result is relatively simple, it took so long to understand :/

Do you have an idea what causes the bug? From your comment, I mostly understand your goal is performance.

[aSemy]

Ah right, yeah, I missed the context. So...

Dokka Generator modes

There are basically three modes Dokka Generator works in:

1. A single subproject → a rendered publication (no further processing required)
2. A single subproject → a single 'partial' module (requires further processing)
3. Multiple 'partial' modules → an aggregated publication (no further processing required)

So, if you have a typical multi-module Gradle project, each subproject will generate a single 'partial' module, and then the aggregating project will combine all of those into a single rendered publication.

Aggregation Requirement: all-modues-page-plugin

The problem is that in order to aggregate modules, Dokka Generator requires an additional Dokka Plugin: org.jetbrains.dokka:all-modules-page-plugin.

In Dokka runners, and in earlier Dokkatoo versions, users had to manually specify this plugin #14. But this was annoying, and error prone, especially because when all-modules-page-plugin was missing Dokka Generator gave no clear indication, and happily ran, but didn't aggregate the modules.

Automatic all-modues-page-plugin

So, I tried implementing a workaround, so Dokkatoo adds all-modues-page-plugin automatically. However, it's not easy!

Gradle doesn't have a good option to have a dynamically conditional dependency (e.g. "if dependency FOO is present, then add dependency BAR").

The current workaround is as follows:

1. Dokkatoo has a Configuration that exposes all-modues-page-plugin as a dependency to consumers.
2. Aggregating Dokkatoo projects will be able to automatically resolve this Configuration, and automatically add all-modues-page-plugin to the Dokka Generator.

And, BAM, it worked! When aggregating, Dokkatoo will be able to automatically get the required dependencies. And I liked that it was flexible, so the Configuration didn't have to be strictly only for all-modues-page-plugin.

Except... because sharing files between projects in Gradle is a living nightmare very well written and not buggy at all, Gradle will ignore the attributes of Configurations in random situations, and it will pick up Dokkatoo's all-modues-page-plugin Configuration instead of the intended Configuration.

I tried fixing this issue in #173, by making Dokkatoo use some ✨special✨ Attributes, and some Attribute compatibility rules, so they wouldn't clash with 'regular' Configurations. Dokkatoo's Configurations would work in a 'parallel dimension'. But, it seems there's some other bug! And that's what you're experiencing.

Big plan

So, the idea I explained in my previous comment, is to basically stop sharing Dokka Plugins between subprojects. Dokkatoo should only share the raw materials used to construct a 'partial' module. It will then be easier to automatically add the all-modues-page-plugin as required, as well as some other benefits.

[CLOVIS-AI]

Thanks for the very detailed explanation!

To clarify, all-modules-page-plugin is a plugin for the Dokka generator, not a Gradle plugin, right? Which means all the files you mention are the JAR files for the plugin, that you inject into Dokka's classpath?

It does sound like Configurations are the right tool for the job, but it seems really strange that they would cause this problem. I'm glad I haven't found some problems in my own plugins 🥲.

[aSemy]

To clarify, all-modules-page-plugin is a plugin for the Dokka generator, not a Gradle plugin, right?

Correct! (The terminology is very confusing.)

There's more info about Dokka Plugins here: https://kotlin.github.io/dokka/1.9.20/developer_guide/plugin-development/introduction/

Which means all the files you mention are the JAR files for the plugin, that you inject into Dokka's classpath?

Again, correct ✅

[CLOVIS-AI]

Whenever you create the PR for the fixes you mentioned, don't hesitate to ping me. I doubt I'll be of much help, but now I'm curious :)

[CLOVIS-AI]

Another question, do you know how to reproduce the bug on purpose in a simple project? If I could reproduce it in my canary build (https://gitlab.com/opensavvy/automation/gradle-conventions/-/tree/main/examples/kotlin?ref_type=heads), it would make it much easier for me to wait for the proper fix as I could catch it before it randomly appears in real projects 😅

[aSemy]

@martinbonnin made a smaller reproducer, but I haven't retested it with 2.2.0

https://github.com/martinbonnin/reproducer-attributes

[aSemy]

hey @CLOVIS-AI, would you mind checking if Dokkatoo v2.3.0-SNAPSHOT makes any difference, when you have a chance? I'm wondering if #204 might help.

[CLOVIS-AI]

Hey! Sorry, I completely missed your message :/

I just saw 2.3.0 was released, I'm starting the migration on my side. If it all goes well, I should get back to you in less than a week :)

PS. I saw you featured Prepared in the "favorite showcases" on the new website <3

[aSemy]

No problem! I've done some experimenting of my own, and I don't think 2.3.0 helps unfortunately.

PS. I saw you featured Prepared in the "favorite showcases" on the new website <3

Nice! Yeah, I liked how your Dokka sites were laid out and fully documented, they look good!

[CLOVIS-AI]

Still broken with 2.3.0, indeed. Logs • Build scan

[CLOVIS-AI]

Don't ask me how, but Kotlin 2.0.0-RC1 seems to have fixed the Pedestal repository. It's still running under Dokkatoo 2.3.0, though I'm in the process of upgrading to 2.3.1.

[aSemy]

Don't ask me how, but Kotlin 2.0.0-RC1 seems to have fixed the Pedestal repository. It's still running under Dokkatoo 2.3.0, though I'm in the process of upgrading to 2.3.1.

Great news! Thanks for the update. Perhaps it was a problem with KGP.

[aSemy]

Are you going to try RC2 as well @CLOVIS-AI?

[CLOVIS-AI]

Are you going to try RC2 as well?

That's unclear at the moment, I'm affected by https://youtrack.jetbrains.com/issue/KT-67636. Maybe I'll have to wait for RC3.

[CLOVIS-AI]

Nope, just a fluke :/ it's back in https://gitlab.com/opensavvy/groundwork/pedestal/-/merge_requests/120