Invalidate core cache if external dependency was modified

[Bodmer]

Describe the request

Add an option to prevent caching or to clear the cache.

Describe the current behavior

Some libraries and board package use a build_opt.h file in the sketch to configure the board core or indeed a library.

See for example here:
https://github.com/stm32duino/wiki/wiki/Customize-build-options-using-build_opt.h

The problem is that swapping between sketches that use slightly different build options will fail to use the new options.

The crude work-around is to select a different processor, compile (even if this generates errors) so the cache is deleted/changed, then swap back to the correct processor and compile again. Clearly this is a rather clumsy way of forcing a cache clear.

If there is a neater solution then please post it. Otherwise please update the IDE with an option to use/not use caching. Clearly this option can default to the current arrangement.

Demo

$ arduino-cli version

arduino-cli.exe  Version: git-snapshot Commit: b82a519e Date: 2023-10-21T11:43:36Z

$ export ARDUINO_DIRECTORIES_DATA="/tmp/arduino-cli-directories/data"  # Use a throwaway directories.data for the demo

$ arduino-cli core install arduino:[email protected]  # Install an arbitrary platform for use in the demo

[...]

$ echo 'compiler.cpp.extra_flags="-I{build.source.path}"' > "${ARDUINO_DIRECTORIES_DATA}/packages/arduino/hardware/avr/1.8.6/platform.local.txt"  # Add the sketch folder to the compiler's search path

$ printf '#include <SomeSketchHeaderFile.h>\n#ifdef MACRO_FROM_SKETCH_HEADER_FILE\n#error "MACRO_FROM_SKETCH_HEADER_FILE is defined"\n#endif' > "${ARDUINO_DIRECTORIES_DATA}/packages/arduino/hardware/avr/1.8.6/cores/arduino/DependsOnSomeSketchHeaderFile.cpp"  # Add source file to the core that depends on a header file from the sketch

$ SKETCH_PATH="/tmp/FooSketch"

$ arduino-cli sketch new "$SKETCH_PATH"

$ touch "${SKETCH_PATH}/SomeSketchHeaderFile.h"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "$SKETCH_PATH" # Create core cache

[...]
Compiling core...
[...]
Archiving built core (caching) in: C:\Users\per\AppData\Local\Temp\arduino\cores\arduino_avr_uno_5958a1d0f1df880da5aa74c2f1a07415\core.a
[...]

$ echo "#define MACRO_FROM_SKETCH_HEADER_FILE" > "${SKETCH_PATH}/SomeSketchHeaderFile.h"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "$SKETCH_PATH"

[...]
Compiling core...
Using precompiled core: C:\Users\per\AppData\Local\Temp\arduino\cores\arduino_avr_uno_5958a1d0f1df880da5aa74c2f1a07415\core.a
[...]
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.

🙁 The cached core was used even though a dependency had changed.

The expected result is that the cache would be invalidated, which in this simplified demo would result in the compilation failing:

$ arduino-cli compile --clean --fqbn arduino:avr:uno --verbose "$SKETCH_PATH"

[...]
Compiling core...
[...]
C:\Users\per\AppData\Local\Temp\arduino-cli-directories\data\packages\arduino\hardware\avr\1.8.6\cores\arduino\DependsOnSomeSketchHeaderFile.cpp:3:2: error: #error "MACRO_FROM_SKETCH_HEADER_FILE is defined"
 #error "MACRO_FROM_SKETCH_HEADER_FILE is defined"
  ^~~~~
[...]

Arduino CLI version

Original report

0.27.1

Last verified with

b82a519

Operating system

Windows

Operating system version

10/11

Additional context

See:

https://github.com/stm32duino/wiki/wiki/Customize-build-options-using-build_opt.h

Note that the suggested approach of closing the IDE and opening again no longer works.

Related

• Provide an option for a complete recompile, including any board specific cached code arduino-ide#419
• https://forum.arduino.cc/t/cache-clear-option/1040106

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the nightly build
• My request contains all necessary details

[matthijskooijman]

I believe that arduino-cli compile has a --clean option that would achieve what you need.

However, I also think that the caching code should handle this automatically, e.g. by tracking the include paths used for compiling the core files and noticing that these change when compiling another sketch (this implies that sketches using build_opt.h or otherwise include sketch dir in the core include path can never share cached cores between different sketches, but that's just how it is, I guess).