zephyr-cp: scan results lost and duplicated; two 10-deep queues drop, one silently

[mikeysklar]

Measured on hardware, board associated to an AP, build 10.3.0-alpha.4-57-gd96b766f32:

[connected] total: 61 uniq_bssid: 12

61 scan results for 12 real access points, about 5x duplication. Same SSID and RSSI repeat, for example six entries of one SSID all at -45 dBm on channel 6.

Cause

ports/zephyr-cp/common-hal/wifi/ScannedNetworks.c:83 sweeps channels one at a time, copied from the espressif port:

static uint8_t scan_pattern[] = {6, 1, 11, 3, 9, 13, 2, 4, 8, 12, 5, 7, 10, 14, 0};

Each entry issues its own NET_REQUEST_WIFI_SCAN. That works on espressif because esp_wifi_scan_start() honors the requested channel. It does not work here.

zephyr/drivers/wifi/siwx91x/siwx91x_wifi_scan.c:188 switches scan type once the radio is associated:

if (sidev->state == WIFI_STATE_COMPLETED) {
    siwx91x_configure_scan_dwell_time(SL_WIFI_SCAN_TYPE_ADV_SCAN, ...

and Silicon Labs documents that background scans ignore the channel selection (modules/hal/silabs/wiseconnect/components/protocol/wifi/inc/sl_wifi_types.h:274):

For background scans (SL_WIFI_SCAN_TYPE_ADV_SCAN): Scans the channels that were originally specified in channel_bitmap_2g4 during the first scan. The channel bitmap cannot be modified for background scans.

So while connected, every pass of the sweep re-reports the same APs. Nothing between the driver and Python filters them: wifi_scannednetworks_scan_result() puts every result straight on the msgq.

Knock-on

MAX_BUFFERED_SCAN_RESULTS is 10 (ScannedNetworks.h:17) and overflow is dropped with a Dropping scan result warning. Duplicates from a strong nearby AP can push genuine distant APs out of the results entirely, so this is not only cosmetic.

Not unique to this port

CircuitPython does not deduplicate scan results anywhere, on any port. espressif looks clean only because ESP-IDF returns a per-BSS deduplicated list. raspberrypi has the same visible symptom, reported by dhalbert in adafruit#7318, open since 2022 and labeled third-party.

MicroPython solves it for the same cyw43 chip in extmod/network_cyw43.c:159-169 by deduplicating on BSSID and keeping the strongest RSSI.

Options

1. Deduplicate on BSSID before queueing. Fixes the symptom regardless of scan type, and generalizes to Pico W: network scanning shows a lot of duplicates adafruit/circuitpython#7318.
2. Skip the per-channel sweep when the driver would use ADV_SCAN, since the channel restriction is ignored anyway.

These are complementary. 1 is the more useful upstream change.

[mikeysklar]

Correcting this. Dedup is not the right fix, and neither is dropping the channel sweep. Both were treating a symptom.

Measured, three runs each, same RF environment

build	results
baseline: per-channel sweep, no dedup	61 raw, 12 unique BSSIDs
single all-channel scan	5, 6, 6
sweep plus BSSID dedup	10, 10, 10

10 is exactly MAX_BUFFERED_SCAN_RESULTS, and three byte-identical histograms is a ceiling, not a survey.

What is actually happening

There are two 10-deep queues in series and both drop.

queue	depth	on overflow
CONFIG_NET_MGMT_EVENT_QUEUE_SIZE	10	drops after a 10 ms timeout; the NET_WARN is compiled out because CONFIG_NET_MGMT_EVENT_LOG_LEVEL=0
MAX_BUFFERED_SCAN_RESULTS	10	drops with a LOG_WRN that goes to RTT

Both values confirmed in the build's .config.

The driver emits the whole burst in one tight loop with no yield (zephyr/drivers/wifi/siwx91x/siwx91x_wifi_scan.c:90-92), and the net_mgmt thread runs at cooperative priority -1, above the WiseConnect event thread, so it drains straight into our handler while the VM thread cannot run.

So the per-channel sweep was accidentally rate limiting delivery, and the duplicates were acting as retries against a lossy queue. Removing either makes the result set worse rather than better. That is why the ugly original returned the most access points of the three.

Why espressif does not have this

espressif uses a pull model. It waits for WIFI_SCAN_DONE, then reads the whole list with esp_wifi_scan_get_ap_records() into a buffer grown with m_renew_maybe() to the exact reported count (ports/espressif/common-hal/wifi/ScannedNetworks.c:63-94). The producer has already finished before the consumer looks, so there is no race.

raspberrypi has the same defect as us, a 16 slot ring that drops with no log at all (ports/raspberrypi/common-hal/wifi/ScannedNetworks.c:27-33). That is adafruit#7318, open since 2022.

The fix

Zephyr has no bulk retrieval API, so espressif's mechanism cannot be copied directly. What can be copied is the gate: do not hand the first Network to Python until NET_EVENT_WIFI_SCAN_DONE for that pass. Drain into a CircuitPython side list first. Nothing about the current design requires streaming results while the driver is still producing them.

CONFIG_NET_MGMT_EVENT_QUEUE_SIZE also has to be raised in the board conf, otherwise events are dropped before our callback ever runs.

Dropping both patch branches. Neither addressed the cause.

[mikeysklar]

Status update. No fix filed, and the related upstream attempt is closed.

Two things learned since this was written:

Duplicate counts vary run to run on a SiWx917-DK2605A. Recent clean scans returned 73, 64 and 56 networks on the same board and network within minutes, so any measurement here needs several runs before it means anything. The earlier dedup and single-pass patches both made things worse than baseline and were discarded.

More useful, tannewt's position on the neighbouring issue (adafruit#11214) applies here too:

The real fix would be to fix any wifi driver that doesn't always get back to us.

This issue is about two ten-deep queues in series dropping results, one of them silently, which is driver and Kconfig territory rather than something to paper over in common-hal. That is the direction to take it.