zephyr-cp: common_hal_wifi_scannednetworks_next() waits on k_poll(K_FOREVER) with no bound (no reproducible hang)

[mikeysklar]

Recovering from #58 by calling wifi.radio.stop_scanning_networks() hung the board hard.

Observed

Board associated, web workflow serving, radio stuck in the #58 state. Ran:

import wifi
r = wifi.radio
r.stop_scanning_networks()

After that:

• no serial output at all on the console UART
• web workflow gone, curl http://<ip>/cp/version.json returns nothing (curl exit 000)
• the J-Link still enumerates and commander adapter list still reports the board

Only commander device reset recovered it, about 19 seconds to come back.

Note

common_hal_wifi_radio_stop_scanning_networks() already carries this comment at Radio.c:272:

// Return early if self->current_scan is NULL to avoid hang
if (self->current_scan == NULL) {
    return;
}

So a hang in this function is already known, and the existing guard only covers the NULL case. Here current_scan was non-NULL but pointed at a scan that had been abandoned, and wifi_scannednetworks_deinit() was reached with the msgq, fifo and k_poll state left mid-scan.

I have not isolated which part of wifi_scannednetworks_deinit() blocks. Worth checking whether it waits on channel_done or drains the fifo without a timeout when no further scan results will ever arrive.

This matters more than the scan duplication in #57: it takes the whole board down, including the network stack, from ordinary Python.

Related: #58, #57.

[mikeysklar]

Correcting the diagnosis. stop_scanning_networks() is not where it hangs.

wifi_scannednetworks_deinit() (ScannedNetworks.c:118-128) is entirely non blocking. Every call in it is K_NO_WAIT:

while (!k_fifo_is_empty(&self->fifo)) {
    wifi_network_obj_t *entry = k_fifo_get(&self->fifo, K_NO_WAIT);
    ...
}

The blocking call is in the iterator, common_hal_wifi_scannednetworks_next() at ScannedNetworks.c:49:

while (k_fifo_is_empty(&self->fifo) && k_msgq_num_used_get(&self->msgq) == 0) {
    k_poll(self->events, ARRAY_SIZE(self->events), K_FOREVER);

K_FOREVER, with no timeout and no scan-done deadline. The wait ends only if a result arrives, channel_done is signalled, or mp_interrupt_sem fires. If the driver has stopped delivering, for example because a previous scan was abandoned and the NWP is not going to produce another NET_EVENT_WIFI_SCAN_RESULT or SCAN_DONE, none of those happen and the VM thread parks permanently.

That also explains losing the network, not just the REPL. There is no RUN_BACKGROUND_TASKS anywhere in that loop, so while it is parked the supervisor's background callbacks never run and the web workflow stops being serviced. The board stays alive at the debug level, which matches what was observed: commander adapter list still reported it while nothing answered on serial or HTTP.

So the real sequence was a fresh start_scanning_networks() after the recovery attempt, which then blocked in next() because no results ever came.

Two related things in the same file:

• wifi_scannednetworks_wait_for_scan() (ScannedNetworks.c:38-41) is now just return !mp_hal_is_interrupted(); and is never called. Dead leftover from the espressif copy, where it actually waited for the scan-done bit.
• The missing RUN_BACKGROUND_TASKS in the next() loop is the same defect class as the connect() and ping() loops.

Suggested: bound the k_poll with a timeout derived from the scan deadline rather than K_FOREVER, and run background tasks each pass so the workflow survives a scan.

[mikeysklar]

Fixed and verified on hardware.

common_hal_wifi_scannednetworks_next() now polls in 50 ms slices with RUN_BACKGROUND_TASKS between them, and gives up if nothing arrives for 10 s. That matches what the other ports already do: espressif runs background tasks in wifi_scannednetworks_wait_for_scan() (ports/espressif/common-hal/wifi/ScannedNetworks.c:39) and raspberrypi in its own next() loop (ports/raspberrypi/common-hal/wifi/ScannedNetworks.c:67). Neither blocks indefinitely.

Measured after the change

Four consecutive scans from code.py, while a host hit /cp/version.json every 2 s:

scan 0 results 57 uniq 9 secs 8.8
scan 1 results 53 uniq 9 secs 8.9
scan 2 results 51 uniq 10 secs 8.9
scan 3 results 53 uniq 8 secs 9.0

Web workflow: 12 requests during active scanning, 12 x HTTP 200, no failures. Previously the workflow stopped being served for the duration of a scan, because the VM thread was parked in k_poll and background callbacks never ran.

No hang across repeated scans. The 10 s budget never trips in normal operation, results arrive continuously; it exists only so a driver that goes quiet ends the scan instead of parking the VM.

Cost

168 bytes of flash. No RAM: the deadline is a local, which matters on a board with under 8 KB of CircuitPython heap.

The duplicate and dropped results visible above (51-57 raw for 8-10 unique) are #57 and are not addressed here.

[mikeysklar]

Confirmed independently on a second board, 440353787 on a different host.

Fourteen /cp/version.json requests at 3 s intervals, spanning roughly 42 s that included several active scans:

200 200 200 200 200 200 200 200 200 200 200 200 200 200

14 of 14, no failures, while code.py was scanning. Before the change the workflow was not served for the duration of a scan, because the VM thread was parked in k_poll(K_FOREVER) and the supervisor's background callbacks never ran.

The 10 s timeout did not trip on this board either. Results arrive continuously in normal operation.

[mikeysklar]

Upstream declined the fix for this. adafruit#11214 proposed bounding the wait and running background tasks; tannewt closed it:

I don't think we need this until we see a reason why. The real fix would be to fix any wifi driver that doesn't always get back to us. I don't believe we need run_background_tasks either because Zephyr should be migrated away from it in favor of Zephyr tasks.

He is right on the evidence. Testing on a SiWx917-DK2605A could not reproduce a hang, and the claim that the web workflow goes unserved during a scan was wrong: it stays served, with response time rising from about 0.13 s idle to about 0.4 s during a scan.

So this issue stands as written but has no demonstrated failure behind it. Two things follow:

• The fix belongs in the driver that stops reporting, not in a timeout in common_hal_wifi_scannednetworks_next().
• RUN_BACKGROUND_TASKS is the wrong direction for zephyr-cp regardless, since the port is meant to move to Zephyr tasks.

Leaving this open as a record. It needs a reproducible hang before it is actionable.

[mikeysklar]

Found the reason results stopped arriving, and it was ours, not the driver's.

NET_EVENT_WIFI_SCAN_RESULT was never in the mask passed to net_mgmt_init_event_callback(). The handler's case for it has existed since the port landed but never ran, so no result was ever queued. Filed as adafruit#11223 with a before/after on an in-tree Pico 2 W: 0 networks before, 204 after.

That changes the picture here in two ways.

The symptom on stock upstream is an empty scan, not a hang. next() sees channel_done and returns, so nothing parks. My comment above reasoned from "if the driver has stopped delivering" and treated the wait as the failure; the wait was never reached with a starved fifo on stock firmware.

The SiWx917 numbers in comments 2 and 3 (51-57 results per scan) came from a patched build that already carried the subscription fix, which is why that board saw results at all while stock boards see none. I did not make that distinction at the time.

What still stands: ScannedNetworks.c:49 really does wait on k_poll(..., K_FOREVER) with no deadline, and wifi_scannednetworks_wait_for_scan() is still dead code. No hang has been reproduced against it, and tannewt's position on adafruit#11214 is unchanged: the bound belongs in a driver that stops reporting, and RUN_BACKGROUND_TASKS is the wrong direction for a port moving to Zephyr tasks.

Retitled to match. Staying open as a record of the unbounded wait, still not actionable without a reproducible hang.