Dynamic USB devices, revamped webassembly port, openamp, tls, vfs modules

[dpgeorge]

This release of MicroPython adds support for dynamic USB devices defined in Python, adds new openamp, tls and vfs modules, completely revamps the webassembly port to add proxying between JavaScript and Python, and implements significant code size optimisations for frozen modules. There are also many other enhancements and bug fixes. Keep reading for a more detailed summary of the main changes.

After a lot of design, development and testing, MicroPython now has full support for user-defined USB devices. The interface is via a new machine.USBDevice class, and this allows the user to specify the USB descriptors and implement Python callbacks for USB endpoint transfers. With this, any USB device can be implemented in pure Python. While the machine.USBDevice interface is low-level and complete, there is a higher-level USB library in micropython-lib that allows easier implementation of devices, with examples for keyboard, mouse, MIDI and serial CDC. This feature is currently available on rp2 and samd ports, and other ports will follow in the future.

Support for the OpenAMP (asymmetric multiprocessing) protocol has been added through the new openamp module. This allows MicroPython to control other CPU cores in the system, to load and start processes and communicate with them through endpoints. This feature is currently available on the mimxrt and stm32 ports.

Two other new modules have been added: vfs and tls. The vfs module contains all VFS (virtual filesystem) related functions and classes, such as mount, umount and VfsFat. These were originally in the os module but having them there is not compatible with CPython, so they have been moved to their own dedicated module. They still exist in the os module for now but will eventually be removed from there, so it's recommended to start using the vfs module from now on. Similarly the new tls module is an evolution of the ssl module, whereby all the existing functionality in ssl has been moved to the tls module. This is done because MicroPython's SSL interface is becoming increasingly different to CPython's and moving this SSL/TLS functionality to a new tls module gives it room to grow and obtain new features that are useful for embedded applications. And compatibility with normal Python is still retained via a pure Python implementation of the ssl module. One new feature in the new tls module is the ability to register a certificate verification callback.

Other additions include more methods on the deque object so it is doubly-ended and supports iteration, and support for half-float 'e' format in struct.pack/struct.unpack. Dynamic native modules have had some additional runtime methods exposed, and the .mpy sub-version has been increased from v6.2 to v6.3 (native code in .mpy files will need to be recompiled, but bytecode does not and is still compatible).

There has also been significant code size optimisations for frozen modules. A new internal mp_proto_fun_t type has been defined which allows the common case of bytecode functions (as opposed to native code) to be stored in frozen code without any additional overhead of the mp_raw_code_t descriptor structure. All firmware builds using frozen modules will see a significant decrease in size. Code size has also improved further for very small targets by adding an option to remove the qstr hash bytes.

Internally in the source code the "STATIC" macro definition has been removed. Code should now just use "static" instead. If you have C/C++ code that uses "STATIC" then either replace it with "static", or provide your own #define to define "STATIC" as "static". See commit decf8e6 for details.

Mbedtls has been updated to version 3.5.1. And network interface constants, such as IF_STA and SEC_WPA2, have now been consolidated across ports so they all live at the WLAN class level, for example network.WLAN.SEC_WPA2 (existing constants at the network module level have been retained for backwards compatibility, but the new ones should be preferred from now on).

The esp32 port has seen some important bug fixes in the BLE component, to deinitialise without crashing, and increase the BLE task stack size. This port also uses the new I2S IDF driver, and supports IDF 5.0.5 and 5.2. There is support to enter the bootloader via machine.bootloader() and a new esp32.mcu_temperature(), for ESP32-C3/S2/S3 devices.

The rp2 port has added the new machine.USBDevice dynamic USB driver, and has had firmware performance optimisations applied to critical parts of the runtime and VM, giving about a 10% performance boost. There is now direct memory access to PIO and SPI FIFOs via proxy arrays, and bugs have been fixed with threads, lightsleep and UART IRQ latency.

The stm32 port has integrated support for the new openamp module, which is enabled on all Arduino boards. And firmware for Arduino boards now freeze in additional Arduino-specific library code. There have been fixes for internal flash writes on STM32H5 and STM32H7 MCUs (bank selection and flashing of the last word in a buffer), an important fix to a SPI DMA caching bug, an I2C clock enable fix for I2C4 on STM32F7 MCUs, and a fix for the FDCAN source clock frequency on STM32G4 MCUs. Mboot has added support for a new raw filesystem, to allow simpler and more robust firmware updates.

The webassembly port has seen a significant overhaul in its structure, and is now built as a JavaScript .mjs module that exports a user-friendly JavaScript-level API which is inspired by the API provided by Pyodide (which is a version of CPython that runs in the browser). This change is motivated by the need to use MicroPython as an engine within Pyscript, which is a platform for Python in the browser. New features in the webassembly port include proxying of objects between Python and JavaScript, a js module to interface with the JavaScript namespace, a jsffi module for miscellaneous proxy helpers, top-level async code, JavaScript driven asyncio support, automatic growing of the Python heap, integration of JavaScript and Python finalisation for global memory management, more time module functions, and support for build variants (following the unix port).

The change in code size since the previous release for various ports is (absolute and percentage change in the text section):

   bare-arm:   -220  -0.383%
minimal x86:   -341  -0.184%
   unix x64: +20168  +2.527%
      stm32:  -1692  -0.430%
     cc3200:   +256  +0.139%
    esp8266:  -8880  -1.260%
      esp32:  -3328  -0.194%
     mimxrt:   -408  -0.112%
 renesas-ra:   -464  -0.074%
        nrf:   -640  -0.341%
        rp2:  +5380  +1.626%
       samd:  +3224  +1.229%

The leading causes of these changes in code size are:

• bare-arm, minimal: disabling qstr hashing
• unix: updating mbedtls to version 3.5.1
• stm32: optimising size of frozen modules
• cc3200: addition of new vfs module, and bug fixes in the VM and array type
• esp8266: disabling unused MICROPY_DEBUG_PRINTERS and optimising frozen modules
• esp32, mimxrt, renesas-ra, nrf: optimising frozen modules
• rp2: addition of machine.USBDevice, enabling -O2 optimisations
• samd: addition of machine.USBDevice

Performance is effectively unchanged since the previous release on all ports, except the rp2 port which sees a performance improvement of roughly 10%.

Thanks to everyone who contributed to this release: Amirreza Hamzavi, Andrew Leech, Angus Gratton, Brian Pugh, Carlosgg, Christian Walther, Damien George, Daniël van de Giessen, darc, Dash Peters, David Lechner, Felix Dörre, iabdalkader, IhorNehrutsa, Iksas, J. Neuschäfer, Jared Hancock, Jim Lipsey, Jim Mussared, Jochen Sprickerhof, Joey232, Jos Verlinde, Kwabena W. Agyeman, Maarten van der Schrieck, Matt Trentini, Matthias Urlichs, Michiel W. Beijen, MikeTeachman, Nicko van Someren, Olivier Lenoir, Phil Howard, Rick Sorensen, robert-hh, Sebastian Romero, Simon Wood, Stanislav Ponomarev, stijn, Takeo Takahashi, Trent Piepho, Trent Warlaven, Vonasmic, YAMAMOTO Takashi, Yoctopuce.

MicroPython is a global Open Source project, and contributions were made from the following timezones: -0800, -0700, -0600, -0500, -0300, +0000, +0100, +0200, +0330, +0400, +0900, +1000, +1100.

The work done in this release was funded in part through GitHub Sponsors, and in part by George Robotics, Espressif, Anaconda, Arduino, LEGO Education, OpenMV and Planet Innovation.

---

This discussion was created from the release Dynamic USB devices, revamped webassembly port, openamp, tls, vfs modules.

[dpgeorge]

This took a while, but it's finally out! Enjoy.

[shariltumin]

Thank you Damien. Thank you for all your hard work.

I have now compiled the latest MP for ESP32, ESP32C3, ESP32S2 and ESP32S3 using my custom configuration. All builds were successful, all test runs with WiFi were also successful.

I know you are a busy man, you are not obliged to answer. I have compiled these firmwares without THREAD and GIL. Do you see any problem with that in the long run?

[dpgeorge]

I have compiled these firmwares without THREAD and GIL. Do you see any problem with that in the long run?

I don't see any issue with disabling threading. Everything is supposed to run correctly with threading disabled. If you see any issues with that, please report them.

[Josverl]

Thanks all for the release and the many useful features included.
I notice that the v1.23.0 firmware files are not available on https://micropython.org/download.
Is this something that is still todo ? ( I could not find an obvious failed GHA)

[scruss]

You're right: the v1.23.0 builds aren't showing as releases.
Commit a61c446 is v1.23.0, but it hasn't got the full version applied

[dpgeorge]

The issue was that the auto build system (the tools/autobuild/autobuild.sh script that runs somewhere in the cloud) triggered on the push of the v1.23.0 commit, and started building too early, before the tag for v1.23.0 was pushed.

In the future I need to use git push --follow-tags to make the two pushes (commit and tag) atomic.

The system is currently rebuilding v1.23.0 with the tag pulled.

[mattytrentini]

....and it's finished! v1.23.0 builds are now available. Thanks @dpgeorge!

[shariltumin]

Espressif has just released esp-idf-v5.2.2

FYI - I just built MP (custom-ESP32) firmware, MicroPython v1.24.0-preview.10.gc1a6b95bf.dirty on 2024-06-04, without any problems. In fact MP with esp-idf-v5.2.2 is smaller in size than MP with esp-idf-v5.2.1

$ strings firmware.bin-BCK6 | grep IDF
[0;32mI (%lu) %s: ESP-IDF:          %s
IDF version : v5.2.1
MicroPython-1.24.0-preview-xtensa-IDFv5.2.1-with-newlib4.3.0

$ strings firmware.bin | grep IDF
[0;32mI (%lu) %s: ESP-IDF:          %s
IDF version : v5.2.2
MicroPython-1.24.0-preview-xtensa-IDFv5.2.2-with-newlib4.3.0

$ ls -l firmware.bin-BCK6 firmware.bin
-rw-rw-r-- 1 sharil sharil 1586384 Jun  4 17:07 firmware.bin
-rw-rw-r-- 1 sharil sharil 1639520 Jun  4 15:29 firmware.bin-BCK6

[cezar1]

This might be off-topic. But it seems it's not possible to buy pyboards from https://store.micropython.org/product/PYBv1.1 . Do you have some expectation of being back in stock?

[syedhuzaifa2119]

Hello @dpgeorge Thanks for the release - I am running script for establishing mqtt connection with hivemq server- I get the error related to USSL module not found - Would be great if you can guide on that to how to deal with that for the latest version of micropython and my script work on this version of micropython "[v1.21.0 (2023-10-05) .uf2]" but i think this version doesn't supports the certificate validation -Further can you also comment on whether the latest version supports the certificate validation to create a tls connection