Recollect sliced JSON in large bytearray()

[nastiliano]

Hello everyone!

I'm using the "Bleak" Python Library to transmit data to my ESP32 over bluetooth BLE UART (GATT service) as client.
In ESP32 I'm using the "aioble" Micropython Library to receive the data as server.

Due the limitations of BLE UART GATT protocol, the "aioble" buffer is limited to 256 bytes and "Bleak" too.

I already solved this sending an sliced JSON from "Bleak" (PC) and deslicing data in EP32 usin bytearray().extend() method but the problem now is the max siz data of a bytearray in Micropython...

PC code:

def sliced(data: bytes, n: int) -> Iterator[bytes]:
    """ Slices *data* into chunks of size *n*. The last slice may be smaller than *n* """
    return takewhile(len , (data[i : i + n] for i in count(0, n)))            
         
    for s in sliced(json.dumps(data).encode() , rx_char.max_write_without_response_size):
        await client.write_gatt_char(rx_char , s , response = False)
        
    await client.write_gatt_char(rx_char ,  b"\r\n" , response = False)

    await client.disconnect()

ESP32 code:

while True:
    async with await aioble.advertise(self.ADV_INTERVAL_MS , name = "NAME" , services = [self.UART_UUID] , appearance = self.ADV_APPEARANCE_GENERIC_COMPUTER):
       data = None
       self.rx_data = bytearray()

       while data != b"\r\n":
           connection , data = await self.rx_characteristic.written(timeout_ms = None)

           if data != b"\r\n":
               self.rx_data.extend(data) # HERE THE CODE CRASHES

        print(json.loads(self.rx_data))

        await asyncio.sleep_ms(1000)

Is the a way to use a large bytearray() or use NVM memory or anather way?

Thank you very much!!

[mendenm]

If you have a known absolute upper limit on the size of the received packet, pre-allocate an array of bytes, and copy the slices into it. Otherwise, python has to keep making copies of the bytearray, which results in heap fragmentation, and impossibility of allocating the extended rray.

[mendenm]

Not quite:

self.rx_data_array = bytearray(46080)
self.rx_data = memoryview(self.rx_data_array)
self.offset = 0
while not_done:
    data=get_stuff()
    self.rx_data[offset:offset+len(data)] = data
    offset += len(data)
do_something_with_data(self.rx_data[:self.offset])

note that I haven't checked this on a live machine, so it is a skeleton, not running code.
Also, note that memoryview() allows you to take slices of otherwise non-sliceable arrays (anything supporting the buffer protocol), without creating copies, and allows these to appear on the right-hand side of an expression.
Further note that your get_stuff might have a read_into capability, in which case it might be able to read slices directly into the memoryview, without even allocating the slice.

[nastiliano]

ok, thanks

Tomorrow I'll try this in my code

[nastiliano]

Hello.

It works!

buff_BLE_UART , rx_data_mv , offset = [None , memoryview(bytearray(32768)) , 0]

    while buff_BLE_UART != b"\r\n":
        conn_BLE_UART , buff_BLE_UART = await self.rx_characteristic.written(timeout_ms = None)

                if buff_BLE_UART != b"\r\n":
                    rx_data_mv[offset : offset + len(buff_BLE_UART)] = buff_BLE_UART
                    offset += len(buff_BLE_UART)

            print(json.loads(bytes(rx_data_mv[ : offset]).decode()))

            await asyncio.sleep_ms(1000)

Better... It works at half :)

I can manage until 20kB but for higher amount of data when I want to get the data from "memoryview()" I have the same problem. The ESP32 can't manage this amount of data. I tried to put "rx_data_mv" directly in "NVS" memory but the table partitions are limited too... Really is hard to manage long data in ESP32

Any idea?

Thank you very very much.

[mendenm]

Where does it fail?

When you do the print(...) the conversion to bytes at least doubles the amount of memory needed, since it has to decode your string into another buffer. Then, the json convert needs yet another buffer! You may just be pushing beyond what this processor can handle.

You could try doing the decode to bytes one slice at a time, but you do risk a multibyte unicode getting split, resulting in a failure. If something assures you will never have a split sequence, this then results in only doubling the size of a single slice, rather than the whole buffer. You can then assemble the decoded bits into your memoryview.

Also, note that you are allocating a new bytearray for each new message. You probably want to create the bytearray right at the top of the whole module, so it is created before the heap gets fragmented. It can be re-used for many messages.

[nastiliano]

Hello!

Now I modified the code in this way:

buff_BLE_UART , rx_data_mv , offset = [None , memoryview(bytearray(65536)) , 0]

    while buff_BLE_UART != b"\r\n":
        conn_BLE_UART , buff_BLE_UART = await self.rx_characteristic.written(timeout_ms = None)

                if buff_BLE_UART != b"\r\n":
                    rx_data_mv[offset : offset + len(buff_BLE_UART)] = buff_BLE_UART
                    offset += len(buff_BLE_UART)

        a_list = json.loads(bytes(rx_data_mv[ : offset]).decode())

        print(a_list[0])

        await asyncio.sleep_ms(1000)

And this is the error:

>>> Traceback (most recent call last):
  File "<stdin>", line 147, in <module>
  File "asyncio/core.py", line 1, in run
  File "asyncio/core.py", line 1, in run_until_complete
  File "asyncio/core.py", line 1, in run_until_complete
  File "<stdin>", line 131, in main
  File "asyncio/funcs.py", line 1, in gather
  File "asyncio/core.py", line 1, in run_until_complete
  File "protocols_com.py", line 62, in connectar
MemoryError: memory allocation failed, allocating 4096 bytes

Is like I won't able to manage this data amount in anyway...

PD: I'm going to try to create the bytearray right at the top of the whole module