Add full configuration cache support#13
Add full configuration cache support#13Luna712 wants to merge 5 commits intorecloudstream:masterfrom
Conversation
This adds full configuration cache support for the last two remaining tasks, `compileResources`, and `make`. This took me like a year of trying *a lot* to finally figure out how to fix the `make` task for it. I finally figured out that the manifest generation was being done at execution time, not configuration time. This adds a new task for generating manifests in order to do it at configuration time and support configuration cache. That also means the Project helper, `makeManifest()` is no longer necessary/used and is now removed.
|
This seems to drastically speed up the build from about 6 minutes to about 2 minutes as well even without configuration cache. I am thinking as true configuration cache support will be a bit more difficult, we could mark it as not truly supporting configuration cache and merge this in the meantime while I work on true support. Thoughts @fire-light42 ? |
This is now solved! This PR should now have support in every single area/task now. |
|
Looks good, but a more comprehensive review is required and will take some time. |
Understandable, appreciate that. Just to note though I have now tested on the largest repos with many plugins and it worked. I also compared plugins.json for both including file size and it fully matching. Functionality seems to work. I havent fully tested I wish I could have split this PR more, but while there are some tiny cleanup to some lines that arent fully related most of this depends on the rest of it so I had to do just one larger PR this time unfortunately. |
Finally completes configuration cache compatibility for all remaining unsupported tasks:
compileResources,make,cleanCache, anddeployWithAdb, and fixes numerous issues with themakePluginsJsontask with it.compileResourcesThe task resolved the Android SDK directory (for
aapt2andandroid.jar) at execution time by queryingLibraryExtensionCompatdirectly insideexec(). These paths are now declared as@InputFileproperties (aaptExecutable,androidJar) and wired up at configuration time inTasks.kt.This also has massive performance benefits even without configuration cache because by wiring them into the task it prevents wiring the entire directory which for the SDK Directory can have tens of thousands of files in it which resulted in huge input size, and thus takes a lot longer than should be necessary. This can also be achieved by simply removing the
android-actions/setup-androidfrom building plugins also which is mostly unnecessary, and prevents having such a large directory passed to it, but it is still something good to properly handle as well.make/GenerateManifestTask(new)The root cause blocking configuration cache on
makewas that the plugin manifest was being built at execution time viaProject.makeManifest(), which accessed project state lazily. This is fixed by introducing a dedicatedGenerateManifestTaskthat produces the manifest JSON at configuration time using properly declared@Input/@OutputFileproperties.WriteCacheEntryTask(new)Plugin entry metadata (file size, SHA-256 hashes, repo URLs, etc.) was previously computed in a
doLastblock on themaketask and stored on the internal extension object, which is incompatible with configuration cache. This logic is extracted into a newWriteCacheEntryTaskthat runs aftermake, computes hashes and sizes from its declared@InputFilereferences, and writes aplugin-entry.json. ThemakePluginsJsontask now achieves full configuration cache support by depending onwriteCacheEntryand reading from these per-plugin entry files, rather than pulling data from the extension at execution time.cleanCache,deployWithAdb, andgenerateSourcesAll three tasks previously resolved inputs lazily at execution time. They now receive all inputs (
jarFile,adbPath,pluginFile) as configuration-time properties wired via providers.With
Project.makeManifest()andProject.makePluginEntry()both eliminated, and SHA-256 hashing now self-contained withinWriteCacheEntryTask,Utils.ktno longer has any remaining responsibilities and has been deleted entirely. This PR also entirely replacescreateProgressLogger()methods inDownloadUtilswith service injection, which is fully configuration-cache safe and modern practices.Finally,
ensureJarCompatibilityis also now automatically ran aftercompilePluginJarto ensure the JAR file is valid and to prevent the need for another separate task running.All necessary tasks for plugin builds can now be done with just a single task (
makePluginsJson) and nothing else. When running locally you can runmaketo avoid unnecessaryplugins.jsonwork, or alsodeployWithAdbwill runmakeand everything else that is necessary automatically as well. Tasks are now are more tightly coupled and the task graph now follows things more cleanly to prevent race conditions and other things.This is a little off-topic but kinda related as this PR sets up the foundation to support this as well. I just wanted to put this here so I don't forget. I also have some things planned for the future after this. To unify more logic all extensions use inside this plugin. I have a working PoC for this already, including the following:
But for the final task, a new task name would be used to avoid conflicts and maintain backwards compatibility.
BaseExtensionversion ofProject.androidwouldn't be added, so that it could just be fully removed when migrating to AGP 9.NOTE: This may still need more testing, however once the basic use cases are fully tested we could probably merge this, and depend on users reporting potential issues they find in more advanced usages.
UPDATE: i have fully tested the primary tasks (everything that runs to build plugins (
make,makePluginsJson, etc...) and it all works. I have also briefly tested, though may need a little more testing,cleanCacheandgenerateSources.deployWithAdbhas not been tested.UPDATE 2: I also just realized this fixes compatibility with Kotlin 2.1.0-2.3.0, fixing numerous errors
Type argument is not within its bounds: must be subtype of 'Any'.with now disallowed nullablePropertydeclarations, which are now proper@Optionaldeclarations. It was supposedly an error in 2.1.0, but was inconsistent until 2.3.0 where they fixed it.