Cargo-first build story with "pure" ESP-IDF tooling

[ivmarkov]

I was looking at what it would take to replace (or have it as an additional option) the existing PlatformIO-based cargo-first build with a "native" esp-idf-tooling based cargo-first build.

Scope & Requirements

This is for now only for "Cargo-first" builds. I.e.:

• The target audience consists of folks who would like to stay in "pure Rust" land
• Use only Rust crates (the ones we provide here + commonly available)
• Use Cargo as the one and only build tool (that's on the surface of course, but that's what they see and care about)
• The firmware should be built as a standard Rust binary crate (with a main() function et. all) and not as a static library which is a better option for CMake/PlatformIO-first builds which are out of scope here
• If - during the building of this esp-idf-sys crate (which is somehow the representation of ESP-IDF in Rust-land) - Cargo is invoking or even downloading other tools, like the Espressif toolchains, or CMake or Python - this is completely hidden from the user (unless the user wants to opt-in with their own ESP IDF SDK dir and/or .espressif toolchain dir).
• The compilation of the ESP-IDF SDK itself is also completely transparent
• No PATH alterations required before invoking Cargo
• No special environment variables required (e.g. ESP-IDF) before invoking Cargo
• Minimum prerequisites that the user has to install up-front (ideally, only Python3)

This story is currently implemented by utilizing the PlatformIO build capabilities.

Why ESP-IDF native toolchain?

• Possibility to build and use the ESP-IDF master branch (PlatformIO only supports the last stable one)
• Possibility to build and use older ESP-IDF releases (PlatformIO seems to support building e.g. ESP-IDF V4.2, but not sure for earlier ones - might require older versions of PlatformIO itself, whichis too big of a hack)
• Create an "ESP8266" story with the "ESP-IDF-like" ESP8266 SDK. Maybe even with libstd support. Not supported by PlatformIO.
• Faster build times (hopefully)

Assessment

The install.sh script available in the root of the ESP-IDF repository might be a very good start, fulfilling most (but maybe not all) of the above requirements:
(+) A thin wrapper over Python-based tooling in tools/
(+) Creates Python virtual envs and does not pollute the user's main Python environment (right?)
(+) Downloads all toolchains automatically
(+) Older and newer ESP-IDFs can co-exist (have separate python virtual envs)
(?) Can I use it on Windows without the "Windows installer" stuff?
(-) Some tooling besides Python still required on the PC of the user (cmake, ninja). PIO downloads this stuff too
(-) Grabbing the PATH requires running . ./export.sh. Can I do this with pure Python scripts?
(-) PowerShell necessary on Windows. Can I do everything with only Python scripts?

Harder stuff

How the esp-idf-sys works is that it essentially builds a hidden ESP-IDF "mini" project, which only contains a stubbed app_main function.

It then grabs all the artefacts of this build, namely:

• (a) All ESP-IDF include directories -> needed for Bindgen so that we get unsafe bindings to various ESP-IDF APIs which are not mapped in Rust libstd and which are used by type-safe crates like esp-idf-hal and esp-idf-svc
• (b) All ESP-IDF link artefacts: static libraries + linker scripts + linker flags and passes them up to the user's top-level binary crate, which is passing these to the Rust linker (which is whatever the rust target triple says - for our case - the linker from the relevant Espressif toolchain), together with the Rust libraries generated by pure Rust crates

So the question is: is there an easy way to "interrogate" the CMake build system, so that I can get (a) and (b)? With PlatformIO, I'm making use of Scons Construction Variables which provide all of this.

I'm sure there's a way to do this with CMake (or idf.py) too, I just don't know it yet. Any hints?

If we can solve (a) and (b), the rest is a breeze.

[ivmarkov]

@igrr @georgik @MabezDev ^^^

[ivmarkov]

Forgot to say: since we have the luxury of full control over the hidden mini ESP-IDF project, we can interrogate the native CMake build system using project-specific means, like custom CMake scripts that we inject in the project. This is what I'm actually doing for PIO here.

[N3xed]

I just created a draft pull request for this: #7.

[N3xed]

Closed by #7.