MicroPython parser enhancements available

[yoctopuce]

Hello,

I am part of a small company that manufactures hardware devices, and we have recently integrated MicroPython into the firmware of a future product. We have made a number of small (and larger) enhancements to MicroPython to suit our needs, and I was looking for the best way we could contribute to the MicroPython project in return for the value it provides.

I have recently been working on some improvements to the micropython parser, mainly targeting mpy-cross, with the aim of reducing the footprint of compiled mpy files to save flash and RAM. I think this work is easy to share and might be useful for others as well, so I have written a small summary of what has been implemented below. If you think any of this might be of interest for integration into the official MicroPython project, let me know and I will try to do a clean git push with just those changes. Each feature can be enabled via an individual mpconfigport define to avoid increasing the footprint of the runtime. Obviously these enhancements make most sense in the mpy-cross compiler, but they could also be enabled in the runtime if someone wanted to.

The common idea behind all these new features is to extend the use of compile-time const() expressions to some unhandled cases that we found useful in the MicroPython implementation of our programming library.

1) MICROPY_COMP_FLOAT_CONST

Our library uses named constants to refer to specific cases, such as INVALID_MEASURE. For floating-point methods, this requires a definition like:

_INVALID_MEASURE = const(math.nan)

or

_INVALID_MEASURE = const(-math.inf)

However, this is not supported by the current implementation, because MICROPY_COMP_MODULE_CONST and MICROPY_COMP_CONST_FOLDING are restricted to integer constants.

So we have introduced a new MICROPY_COMP_FLOAT_CONST feature which leverages the code of MICROPY_COMP_CONST_FOLDING to also support folding of floating point constants, and to include math module constants when MICROPY_COMP_MODULE_CONST is defined. This makes it possible to use compile time math constants such as:

_DEG_TO_GRADIANT = const(math.pi/180)
_INVALID_VALUE = const(math.nan)

2) MICROPY_COMP_COMPAR_FOLDING and MICROPY_COMP_SYSNAME_CONST

We have been looking for a way to use a common code base for both CPython and Micropython versions of our programming library. Obviously, the constraints are not the same in the two environments, and we want to be able to exclude some pieces of code from the Micropython version, or provide lightweight versions. To this end, we were very pleased to discover that the MicroPython compiler is able to prune unreachable code in if statements when the expression is a constant boolean.

However this benefit is currently limited by the fact that comparison operators are not handled at compile time. With the current implementation in MicroPython, we had to rely on code like this at the beginning of each source file:

_IS_MICROPYTHON = (sys.implementation.name == 'micropython')
try:
    _IS_MICROPYTHON = const(True) # noqa
except NameError:
    pass

if not _IS_MICROPYTHON:
    from collections.abc import Callable, Awaitable, Coroutine
    def const(obj: Any) -> Any: return obj
    # etc.

However, this is a bit clumsy, and causes the try/except clause to be emitted in the final bytecode.

So we wrote two simple extensions:

• MICROPY_COMP_SYSNAME_CONST uses about 25 lines in parse.c to recognise the tokens <sys>.<implementation>.<name> in the parse tree and replace them with the constant string "micropython"
• MICROPY_COMP_COMPAR_FOLDING is about 50 lines in parse.c to fold relational comparison operators (<, >, ==, !=, <=, >=) in the same way that integer operations can already be folded with MICROPY_COMP_CONST_FOLDING.

With these two enhancement, it is possible to simply use

if sys.implementation.name != "micropython":
    # Extra stuff that you better avoid on micropython.
    # This will not result in any bytecode.
    pass

Comparison folding also makes it possible to use constructs like:

if TRACE_LEVEL > 1:
  # this code will not be included in production build.
  pass

3) MICROPY_COMP_CONST_MEMBERS

Our programming library makes extensive use of constants defined as static class members, as in

relay.set_state(YRelay.STATE_B)

This is a typical pattern used by many libraries. For example, aiohttp uses an enum-like class WSMsgType:

class WSMsgType:
    TEXT = 1
    BINARY = 2
    CLOSE = 8
    PING = 9
    PONG = 10

It is very important for us to be able to completely wipe out these symbols from the bytecode and resolve them at compile time, to reduce RAM and flash usage. So we implemented support for handling scoped const() expressions defined within a class, by adding about 40 lines in parse.c. It works by recognising when a const() definition is found within a class definition, and replacing the dotted expression with the specific value.

4) MICROPY_COMP_PREDEFINED_CONST

To make the most of compile-time constants defined from static class members, we have added support for importing a Python file to load constant definitions without actually generating bytecode (the file is parsed, but not even compiled). This ensures that static class members can be replaced by constants in user code without having to duplicate them in the user's source code.

    mpy-cross app.py -I aiohttp.py

We have also made it possible to inject constants directly from the mpy-cross command line, as is commonly done with a C compiler:

    mpy-cross ... -D TRACE_LEVEL=2

This code adds about 20 lines in parse.c, plus 50 lines in main.c of mpy-cross and an extra mp_obj_t field in mp_state_vm_t to keep track of predefined constants. If enabled in a micropython VM, this also adds a function in modmicropython, micropython.predef_const() which works in the same way as micropython.opt_level() but with a dictionary of predefined constants. However this is probably less useful than in mpy-cross.

That's about it. Let me know your thoughts and I will see what is the best way to go...

PS. We have also implemented a C native DAP endpoint to enable remote debugging using VSCode, but this is a bigger thing that will require more integration work to be pushed into the official tree. Let me know if this is something I should investigate or not.

[peterhinch]

@dpgeorge is the man to assess this.

[Josverl]

Seems like some good valuable pickings. Thanks for offering to share.