mDNS responder stops answering after serial contact when an mdns.Server exists (web workflow enabled)

[mikeysklar]

With the web workflow enabled, serial/REPL contact kills the mDNS responder until reboot. HTTP and ICMP usually survive. No user code, no advertise_service(), and nothing running to interrupt.

The trigger is the presence of an mdns.Server object, which the web workflow always constructs. Boards without it are unaffected.

Evidence

Two-arm test, arms alternated, criterion declared before the first reset, both metrics on one socket, replies classified by record type and by destination address.

arm cyc | base_A post_A | post_P | http | KbdInt
A    1  | 3/4    4/4    | 0/1    | 200  | False
B    2  | 4/4    0/4    | 0/1    | 200  | False
A    3  | 4/4    3/4    | 0/1    | 200  | False
B    4  | 4/4    0/4    | 0/1    | 200  | False
A    5  | 3/4    4/4    | 0/1    | 200  | False
B    6  | 3/4    0/4    | 0/1    | 000  | False
A    7  | 4/4    4/4    | 0/1    | 200  | False
B    8  | 4/4    0/4    | 0/1    | 200  | False
A    9  | 4/4    3/4    | 0/1    | 200  | False
B   10  | 4/4    0/4    | 0/1    | 200  | False

arm A hostname post: [4,3,4,4,3]   5/5 valid
arm B hostname post: [0,0,0,0,0]   5/5 valid
all 55 answers multicast, zero unicast

KbdInt is False in every arm-B cycle. code.py was print("Hello World!"), which finishes before the Ctrl-C arrives, so there was no KeyboardInterrupt and no running program to interrupt. mDNS died anyway, five times out of five.

Cycle 6 also lost HTTP (000) while the other four arm-B cycles kept 200, so the effect is usually mDNS-only but not always. One occurrence in five; noted, not explained.

Why every earlier cross-board comparison was misleading

One board ran the web workflow and reproduced; the other had no CIRCUITPY_WEB_API_PASSWORD, therefore no web workflow, therefore no mdns.Server was ever constructed, and did not reproduce. Adding the password to the second board made it reproduce on demand.

That reconciles every result in this thread without appealing to silicon, access points, host OS, toolchain or the Zephyr pin — all of which were investigated and none of which was the difference.

Related, and visible throughout

post_P is 0/1 in all ten cycles, including every healthy control. Services never publish even when the responder is demonstrably answering hostname queries. That is the separate DNS-SD bug, and it is why a service-discovery probe cannot be used as an mDNS health check here.

Mechanism: still not established

Six candidate mechanisms have been proposed and retired by measurement rather than argument:

1. BLE adapter teardown
2. advertise_service()
3. The per-object record array in common-hal/mdns/Server.c
4. The Zephyr stack-allocated listener poll arrays (fixed upstream, tracked separately)
5. Interrupting a running program, as a sufficient condition
6. The upstream mDNS commit as the differentiator

What is established: with an mdns.Server present, serial contact kills mDNS responses until reboot; without one, it does not. The mdns_responder_set_ext_records() replace-not-append behaviour remains the most plausible surviving lead, since it is the one path that ties an mdns.Server object to the responder's record state.

[mikeysklar]

Confirmed on a second board (unique ID c09b9ec39910) against Linux avahi, a different resolver stack from the macOS mDNSResponder used in the original report. Independent agreement, not a repeat.

Widening the scope, which is why the title changed. The board does not appear under _circuitpython._tcp either, and that advertisement is not a user call — it is the web workflow's own, registered by firmware without anyone calling advertise_service(). So the responder answers hostname (A record) queries but publishes no PTR/SRV/TXT records at all. A fix aimed only at advertise_service() would leave the web workflow undiscoverable.

Positive control, from the same tool on the same LAN in the same second — another CircuitPython board publishing a fully resolvable service:

+ wlp2s0 IPv4 Adafruit Metro ESP32S3   _circuitpython._tcp   local
   hostname = [cpy-metro_esp32s3-14c19f2fad04.local]
   address  = [192.168.0.46]   port = [80]

That rules out avahi being broken, the LAN blocking mDNS multicast, and too short a browse window.

Our board, same moment:

advertise_service("_bravotest", "_tcp", 8080)  -> returns cleanly, no exception
avahi-browse -rt _bravotest._tcp  at t=0    -> nothing
avahi-browse -rt _bravotest._tcp  at t=45s  -> nothing
avahi-browse -at | grep -i 'c39910|siwx|dk2605'  -> absent from all service browse

The Server object stayed alive across gc.collect(), so garbage collection is not the cause.

Hostname resolution is unaffected throughout — cpy-c39910.local resolved before, during, at 45 s, and after. Constructing mdns.Server does not disturb it.

One difference from the first board: no hostname rename was observed here. This board was already cpy-c39910 before mdns.Server was constructed and never changed. That does not refute the rename seen on the other board, since this one may simply have started in the post-rename state. Untested rather than refuted.

[mikeysklar]

Retracting the previous comment's central claim, and replacing it with the actual root cause.

My earlier comment said the web workflow's own _circuitpython._tcp advertisement was also missing, and concluded DNS-SD publication was entirely absent. That was wrong. _circuitpython._tcp works fine on a freshly reset board and is live-resolvable, not merely cached:

= wlp2s0 IPv4 siwx917-dk2605a  _circuitpython._tcp  local
   hostname = [siwx917-dk2605a.local]  address = [192.168.0.133]  port = [80]

It had vanished in that run because the test itself broke it. It is also not a separate registration path: supervisor/shared/web_workflow/web_workflow.c:395 calls the same common_hal_mdns_server_advertise_service() that user code does. So that evidence was neither independent nor evidence, and the issue title has been corrected accordingly.

The actual bug

Creating an mdns.Server from Python and calling advertise_service() disables DNS-SD answering for the entire board, including the web workflow's own advertisement, until a hard reset.

state	_circuitpython._tcp	user service	_services._dns-sd._udp
after hard reset, no user Server	answers live	n/a	answers
after mdns.Server() + advertise_service()	dead	dead	dead
after another hard reset	answers live again	n/a	answers

Raw multicast queries with avahi bypassed, 15 s waits: zero packets from the board in the middle row. Fully reversible with a reset, reproduced twice.

Confirmed working before the break, live on the wire — so Zephyr's responder, DNS-SD, PTR/SRV/TXT generation and service-type enumeration are all fine, and this does not need fixing in Zephyr:

PTR  _circuitpython._tcp.local -> siwx917-dk2605a._circuitpython._tcp.local ttl=4500
TXT  siwx917-dk2605a._circuitpython._tcp.local
SRV  siwx917-dk2605a._circuitpython._tcp.local -> port=80
A    siwx917-dk2605a.local -> 192.168.0.133

Mechanism, verified in source

mdns_responder_set_ext_records() (zephyr subsys/net/lib/dns/mdns_responder.c:2465) replaces, it does not append:

external_records = records;
external_records_count = count;

ports/zephyr-cp/common-hal/mdns/Server.c:160 passes self->records, a per-object array. The web workflow owns one Server object; user code creating mdns.Server() gets another. Whoever advertises last wins, and everything registered by the other object stops being answered. Since the web workflow always creates one, any user calling mdns.Server() hits this.

The existing comment in Server.c about set_ext_records having no way to clear records is the same sharp edge seen from the other side: not merely a deinit gap, but a live correctness bug as soon as two objects exist.

Not yet isolated: why the user's own records also fail to answer. The PTR path (mdns_responder.c:777-790) iterates static DNS_SD_GET records then external_records; with the static count at zero, the user array is all that remains, so records failing dns_sd_rec_match() would produce exactly the observed total silence including enumeration. Every rec_is_valid() condition was checked by hand and all appear satisfied, so that part is inference, not proof. Pinning it needs CONFIG_NET_LOG for the "DNS SD record at %p is invalid" line, which costs ~9.5 KB and a reflash.

Ruled out by test: the dangling-Python-string theory (rec->service storing a borrowed pointer). Holding the strings in module globals and forcing gc.collect() changed nothing.

Also ruled out: CONFIG_MDNS_RESPONDER_PROBE. The non-PROBE path calls init_listener() directly and the responder demonstrably answers queries, so the listener is alive. Enabling it would pull in MDNS_RESOLVER/DNS_RESOLVER and push DNS_DISPATCHER_MAX_POLL to 9, requiring ZVFS_POLL_MAX >= 10 — a real risk to the fix in #44 for no benefit.

Where a fix belongs

CircuitPython, ports/zephyr-cp/common-hal/mdns/Server.c: one shared record array across all mdns_server_obj_t instances, so set_ext_records receives the union rather than each object clobbering the last.

Method traps for anyone reproducing

• avahi-browse -t exits on "cache exhausted" and misses this board. Drop -t and allow ~20 s.
• avahi's cache shows + lines for services that no longer answer. Only the = resolve line proves liveness.
• Raw probes must join 224.0.0.251 and send from port 5353. With avahi holding 5353, SO_REUSEPORT means unicast replies may land in avahi rather than your socket, so only multicast is reliably observable without sudo.

[mikeysklar]

Second correction, and this one replaces the mechanism as well as the location. The previous comment's conclusion that the root cause is in our port was wrong.

What was wrong

The table showing advertise_service() deterministically killing DNS-SD does not hold. On a later hard reset the board answered nothing at baseline, before any Python object existed:

baseline (fresh reset, no user Server) : 0 answers
after mdns.Server() construct only     : 0
after advertise_service()              : 0

then six more rounds over ~90 s with no Python touched, board pinging and serving /cp/version.json throughout: 0 answers every round. An earlier post-reset window on the same build and same board answered A, PTR and service-type enumeration correctly within ~4.3 s.

So advertise_service() was a correlation that got over-read. Same firmware, no user code, opposite behaviour.

The actual cause: a stack-use-after-return in Zephyr, already fixed upstream

Upstream ba0bd94d4e9 (Robert Lubos, 2026-08-06): "Use static storage for listener poll arrays... stored pointer dangled once the function returned, leading to a stack-use-after-return." It moves ipv4_fds[] and ipv6_fds[] out of init_listener() to file scope.

Our pinned Zephyr has the buggy form, verified line by line in subsys/net/lib/dns/mdns_responder.c:

1643:  struct zsock_pollfd ipv6_fds[MAX_IPV6_IFACE_COUNT];   automatic storage, inside init_listener()
1753:  struct zsock_pollfd ipv4_fds[MAX_IPV4_IFACE_COUNT];   same
1741:  register_dispatcher(&v6_ctx[i], &v6_svc, ..., ipv6_fds, ARRAY_SIZE(ipv6_fds));
1851:  register_dispatcher(&v4_ctx[i], &v4_svc, ..., ipv4_fds, ARRAY_SIZE(ipv4_fds));
1512:  ctx->dispatcher.fds = fds;      the pointer is retained, into static v4_ctx/v6_ctx

and the retained pointer is dereferenced after the frame dies, at dispatcher.c:398 plus ctx->fds[i].fd reads at dispatcher.c:278-288 and 335-349.

Subtlety worth recording, because it nearly cleared the bug wrongly: net_socket_service_register() itself copies (svc->pev[i].event = fds[i]). Looking only there, the code appears safe. The retention happens one level up, via ctx->dispatcher.fds.

The nondeterminism is the signature: fine while the freed frame happens to be intact, dead once it is reused.

What survives from the previous analysis

The mdns_responder_set_ext_records() replace-not-append observation still stands as a real latent bug — two mdns_server_obj_t instances clobber each other and the web workflow always creates one. But it is separate, smaller, and almost certainly not the cause of the total silence. Worth its own issue rather than a fix priority.

Fix

Cherry-pick ba0bd94d4e9 (14 lines, self-contained), or bump Zephyr. Not a patch to common-hal/mdns/.

Related commits also absent from our pin, in the same area: 748a0b1b023 (multi-homed A/AAAA and DNS-SD response interface selection — likely irrelevant here since the test board is single-homed, inferred from the commit description rather than tested), 19289bd7529, d7859a7117a.

Falsifiable prediction for after the pick: service publication should answer reliably across resets rather than intermittently. If it stays a coin flip, the dangling fds was not the whole story and the set_ext_records finding moves back up.

[mikeysklar]

Resolution: there were two independent bugs. Cherry-picking the upstream fix separates them, and #51 survives as a genuine port bug.

Applied ba0bd94d4e9 as a local test patch on the second board. .bss grew by exactly 32 bytes, independently confirming the arrays moved to file scope.

Declared before running: 5 hard-reset cycles, 3 probe rounds each ~15 s apart, no user mdns.Server; pass = every cycle answers in >=2 of 3 rounds.

cycle 1: 3/3   cycle 2: 3/3   cycle 3: 3/3   cycle 4: 3/3   cycle 5: 3/3   PASS

Not one silent round in fifteen. Pre-patch the same probe gave one healthy window then 0/6, with a fresh-reset baseline of 0. The intermittency is gone.

But advertise_service() still fails, and now deterministically:

fresh reset, then mdns.Server() + advertise_service("_picktest")
post-advertise round 1: 0 packets
post-advertise round 2: 0 packets
post-advertise round 3: 0 packets
avahi-browse _picktest._tcp: nothing

So the earlier advertise_service() table was half right. The effect was real; what was wrong was trusting the baseline, because the dangling-fds bug was randomising it underneath. With that confound removed the effect reproduces cleanly every time.

The two bugs:

1. Zephyr — stack-use-after-return on the listener poll arrays. Intermittent total mDNS death, no user code required. Fixed upstream by ba0bd94d4e9, absent from our pin. Tracked in Bump pinned Zephyr for ba0bd94d4e9: mDNS listener poll arrays are stack-allocated (intermittent total mDNS death) #52; needs a pin bump, not a port change.
2. This issue — advertise_service() publishes nothing and silences the responder. Untouched by the pick. Still points at common-hal/mdns/Server.c handing mdns_responder_set_ext_records() a per-object array, where it replaces rather than appends and the web workflow already owns one.

Poll budget re-verified after the pick, nothing regressed: getaddrinfo 80 ms and 98 ms, HTTP 200 by name, cpy-c39910.local 3/3, Wi-Fi and BLE both up.

[mikeysklar]

Retracting the mechanism in this issue. The premise is wrong: there are not two mdns_server_obj_t instances.

shared-bindings/mdns/Server.c, mdns_server_make_new():

#if CIRCUITPY_WEB_WORKFLOW
mdns_server_obj_t *web_workflow_mdns = supervisor_web_workflow_mdns(args[ARG_network_interface].u_obj);
if (web_workflow_mdns != NULL) {
    return web_workflow_mdns;
}
#endif

supervisor_web_workflow_mdns() returns the supervisor's file-scope object whenever the web workflow is up. So mdns.Server(wifi.radio) returns the web workflow's existing object. One object, one records[] array, one instance_name. common_hal_mdns_server_construct() is never called in that path.

Confirmed on hardware rather than by reading alone: a freshly constructed object would have instance_name = "", and the board reports INSTANCE_NAME_AT_CONSTRUCT: 'siwx917-dk2605a', which is the web workflow's name.

Consequences:

• Nothing clobbers anything. advertise_service() already appends into the shared array (slot search by service type, num_services++), and set_ext_records already receives the union.
• The "whoever advertises last wins" story in the earlier comment was wrong.
• MDNS_MAX_SERVICES is already 4 with a loud RuntimeError on overflow, and dedup by service type already exists.
• The deinit comment about being unable to clear records remains a real latent gap, but it is single-object teardown, not union membership.

Further, the observations behind this issue are confounded. Every run showing "silent after advertise_service()", including the _picktest result, interrupted code.py with Ctrl-C first. A separate run on the patched build showed a silent baseline before any Python ran at all. So the trigger attributed to advertise_service() may be something else entirely, and the mechanism for this issue is not established.

Next step is a two-arm experiment holding everything else identical — reset, wait, probe, with and without a code.py interrupt, five cycles each — to determine whether the interrupt is the trigger. This issue should not be acted on until that returns.