Skip to content

archinstall-ci/qemu may be unable to report success — the gate may be inert and the guest may have no network #4866

Closed

Description

@xander-lin

The status has never gone green: of the 57 currently open PRs, 0 have a success,
24 are failure, 6 are stuck pending, 1 is error, and 26 never ran.

That leaves reviewers without a signal. On #4865 you wrote that #4857 might be the
more reasonable one to merge — that is the judgement the CI is supposed to make.
So rather than report "it fails", here are four places in the code that look wrong,
and the evidence I have for each. Two are readable from the source alone; two I
reproduced. I have not seen your CI host, so item 3 is the one I am least sure of.

# Where Suspected problem
1 test_tooling/ci/monitor.py:175-185 count_running_workers() filters --state=running, but worker units are Type=oneshot, so it may always return 0
2 test_tooling/ci/monitor.py:188-206 start_worker() calls blocking systemctl start with timeout=10, so any long run may be recorded as error: Failed to start CI worker
3 tests/qemu/machines.py:49 -netdev tap,...,script=no, with nothing in worker.py / run_test.py configuring the guest — unless the host has a bridge with DHCP, the guest has no network
4 tests/qemu/runners.py:26 waits for b'Set/Modify the below options', which no longer exists in the tree

1. The concurrency gate may never engage

# monitor.py:175-185
["systemctl", "list-units", "--type=service", "--state=running",
 "--no-legend", "archinstall-ci-worker@*"]

archinstall-ci-worker@.service:8 is Type=oneshot with RemainAfterExit=yes.
A oneshot that is still executing is ActiveState=activating, not running.

$ systemctl show -p ActiveState -p SubState t@ccc.service     # oneshot, sleeping 600s
ActiveState=activating
SubState=start

$ systemctl list-units --type=service --state=running --no-legend 't@*' | wc -l
0
$ systemctl list-units --type=service --state=running,activating --no-legend 't@*' | wc -l
1

If that holds, if running >= max_concurrent: break never fires and max_concurrent
does nothing. Consistent with what the timestamps show: ten PRs dispatched inside
145 s on 2026-10-28 (08:50:29–08:52:54), each dying 12–43 s later, and none of them
ever reaching Running QEMU test... — i.e. all ten died in the ISO build stage,
concurrent mkarchiso runs on one host.

2. Dispatch may be recorded as failure while the unit is fine

# monitor.py:188-206
result = subprocess.run(["systemctl", "start", unit], ..., timeout=10)

systemctl start on a oneshot blocks until the unit exits. The worker has a 45 min
cap, so this would time out on essentially every real run.

$ timeout 10 systemctl start t2@x1.service
rc=124, elapsed 10s
$ systemctl is-active t2@x1.service
activating                                  # it is running
$ systemctl start --no-block t2@x2.service
rc=0, elapsed 0s

This would explain error: Failed to start CI worker landing seconds before the
worker's own (also red) status, e.g. #4865 at 08:50:40 followed by 08:50:52.

3. The guest may have no network, and archinstall exits before the TUI

This is the part I can only half-verify, because I do not know what sits behind
your tap. What the code shows: script=no means QEMU runs no setup script, and
neither worker.py nor run_test.py sets an address, a route or a resolver inside
the guest. test_tooling/qemu/qemu.py does have --tap / --bridge handling, but
the CI path goes through run_test.py + machines.py, not qemu.py.

If nothing else provides DHCP, the guest has no network, and then:

# archinstall/main.py:152-153
if not _fetch_arch_db():
    return 1

_fetch_arch_db() runs Pacman.run('-Sy'). In that state, on a local copy of the
harness, the guest printed:

Fetching Arch Linux package database...
Failed to sync Arch Linux package database.
Most likely due to a missing network connection or DNS issue.

module.main() at main.py:171 is then never reached and the TUI is never built,
so from outside it looks like "the TUI flashes and exits, nothing in the log":
serial has no TUI frames at all, /var/log/archinstall/install.log has no
Error while running main app, and the last line is No wifi interface found.

Two things can hide this:

  • _check_online() (main.py:38-48) returns True unconditionally when there is no
    wifi device, because wifi_handler is None skips the whole branch, and the ping()
    return value is discarded (ping() returns -1 on timeout, it does not raise).
  • ICMP may be dropped while TCP works: ping 1.1.1.1 showed 100% loss while
    curl https://geo.mirror.pkgbuild.com/... returned 200.

Giving the guest an address, a route and a resolver on my copy was enough to make
the TUI appear and the install proceed through pacstrap and bootloader:

Installing packages: ['base', 'sudo', 'linux-firmware', 'mkinitcpio', 'linux']
Installing bootloader Systemd-boot to /dev/sda1
Installing packages: ['efibootmgr']

Same ISO, same QEMU arguments, only the guest network differed — which is why I
suspect this is the reason the test cannot pass, but it is also the item that
depends on your host's setup.

4. The test body waits for a string that is gone

# runners.py:26
if b'Set/Modify the below options' in self.buffer and self.running_test is False:
$ grep -rn "Set/Modify" archinstall/*.py archinstall/lib/ archinstall/tui/
(no results — it survives only as a msgid in archinstall/locales/*/LC_MESSAGES/base.po)

So the sequence may never start. Related, in the same loop (run_test.py:259-266):
milestones are matched against a single recv() chunk rather than accumulated
output, and the chain is elif, so if the login banner and archiso login: arrive
in the same 8 KiB chunk only the first matches and archinstall start point never
fires.

Separately, the key sequence itself is tied to a menu layout. The current
GlobalMenu has 20 entries and in my run the blind arrow-key counts selected
Keyboard layout: mac-dvorak and Locale language: en_ZW where they intended
sv-latin1 and Sweden. That is a maintenance property of the approach, not a bug
in any one line.


What I could and could not check

  • Reproduced here: the two systemd behaviours (items 1, 2); the missing anchor string
    (item 4); the guest-with-no-network → pacman -Sy → early return path, and that
    adding a network fixes it (item 3).
  • Not verified: what your CI host actually provides to the tap; LVM and btrfs; and the
    stage after pacstrap. Nothing here is a statement about any of those.

If these hold

--state=running,activating, systemctl start --no-block, something that gives the
guest DHCP (a bridge, or -netdev user), and an anchor string that exists. I can
send patches for any of these against test-cases — but I would rather you confirm
items 1–3 on the host first, since I am inferring them from the code and from a
local copy of the harness.

Activity

  1. h8d13 commented on Oct 28, 2026

    @h8d13
    Contributor

    Considering there hasn't been a single commit in a month CI might be the least of issues. And might also be due to how the repo was configured for CI runs.

    Also, as much as the effort is appreciated, these long prose of AI text most developers will not want to read them (between the lines)

    https://github.com/archlinux/archinstall/blob/master/AI_POLICY.md

  2. xander-lin commented on Oct 29, 2026

    @xander-lin
    Author

    Thanks for the feedback. I’ll change how I do this.

  3. h8d13 commented on Oct 29, 2026

    @h8d13
    Contributor

    You're welcome, usually the same is true to how much code you change in a PR.

    The smaller the patch (+ repro) the more likely it'll be accepted and easy to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions