Skip to content

[Fix] Isolate conduct.sh runs: own port window, own logs_dir and databases - #88

Open
aaronsmulktis wants to merge 5 commits into
mainfrom
aaronsmulktis/conduct-port-windows
Open

aaronsmulktis wants to merge 5 commits into
mainfrom
aaronsmulktis/conduct-port-windows

Conversation

@aaronsmulktis

@aaronsmulktis aaronsmulktis commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Why this PR?

scripts/conduct.sh runs up to MAX_PARALLEL (default 4) launch_agent.py processes at once on one machine, either locally or on a node via conduct_slurm.sh. It never isolated them from each other, in two ways.

Ports. It never passed a job_id, so every run in the pool searched for a free port starting at 5001. AgentLauncher.pick_empty_port tests a port by binding it and releasing it immediately. The web app doesn't claim the port until its subprocess starts, a few seconds to half a minute later. In that gap two runs can pick the same port. The loser's web app exits with address already in use, and its health check finds the winner's app answering instead. From a conduct.sh pool on a cluster node:

Web app is not running on http://localhost:5004 as expected: <urlopen error [Errno 111] Connection refused>
Web app on http://localhost:5004 is serving a DIFFERENT job's config (...). Another job's process must own this port.
...
RuntimeError: Web app process exited before becoming ready; see .../webapp.log

Databases. Under mode=local, logs_dir is log_outputs/<timestamp to the second>-$USER, and the SQLite databases_dir lives inside it. Runs started in the same second shared both: in a 10-run pool, 4 runs shared one directory and 2 shared another. Their web apps read and wrote the same database files. One run's reseed on startup reset another mid-episode, and each agent's edits showed up in its neighbours' reward diffs. The sibling-job guard couldn't catch it, because it tells runs apart by databases_dir. Any conduct.sh result with MAX_PARALLEL > 1 may be affected.

What changes

Each run in the pool now gets:

  • ++job_id=$((i % 500)), which selects the 100-port window AgentLauncher._job_port_offset already reserves per job (5001 + job_id*100).
    • ++ because job_id isn't declared in config/config.yaml; a plain job_id= is rejected by Hydra.
    • % 500 keeps the top of the window under 65535 for any COUNT.
  • logs_dir=<batch>/run-NNN, so each run has its own config, webapp.log and databases. Those sit next to the per-run stdout log that was already in <batch>/.
  • record_video_dir=<batch>/recordings, so a batch's videos still land in one folder. The _job<N> suffix keeps their names apart.

launch_parallel_agents.py already did both (job_id = i, logs_dir/<i>). conduct.sh was the only launcher that didn't. Forwarded overrides come last, so passing your own logs_dir= or record_video_dir= still wins.

Test plan

  • Hydra compose: ++job_id=3 resolves to 3; job_id=3 is rejected. A repeated value override resolves to the last one, so forwarded args win.
  • bash -n scripts/conduct.sh.
  • Ports, macOS: AGENTS=dummy, 10 runs, pool of 4. All completed. Each run picked a port in its own window (5101 … 5901), with no DIFFERENT job's config lines. Run 0 found 5001–5003 held by web servers leaked from earlier runs (fixed separately in the process-group cleanup PR) and moved to 5004, still inside its own window.
  • Databases, macOS: AGENTS=dummy COUNT=4 MAX_PARALLEL=4, all four started together. Each got its own run-00N/ with its own databases/ and its own port (5001, 5101, 5201, 5301). No shared log_outputs/<timestamp> directory was created.

Until this merges, MAX_PARALLEL=1 avoids both problems.

conduct.sh never passed job_id, so every run in the pool scanned for a port from 5001 at once. Two runs could pick the same free port before either web app bound it; the loser exited with 'address already in use' while its health check found the winner's app. Pass ++job_id=$((i % 500)) so each run gets the 100-port window AgentLauncher already reserves per job.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 14, 2026
@aaronsmulktis aaronsmulktis changed the title Conduct port windows [Fix] Give each conduct.sh run its own port window Oct 14, 2026
@aaronsmulktis aaronsmulktis changed the title [Fix] Give each conduct.sh run its own port window [Fix] Give each conduct.sh run its own (offset) port Oct 14, 2026
@aaronsmulktis aaronsmulktis self-assigned this Oct 15, 2026
logs_dir defaults to log_outputs/<now to the second>, and databases_dir lives inside it, so pool runs started in the same second shared both: in a 10-run pool, 4 runs shared one directory. Their web apps read and wrote the same SQLite files, reseeding each other mid-episode and leaking edits into each other's rewards, and the sibling-job guard could not tell them apart because it keys on databases_dir. Pass logs_dir=<batch>/run-NNN per run, and point record_video_dir at one batch folder.
@aaronsmulktis aaronsmulktis changed the title [Fix] Give each conduct.sh run its own (offset) port [Fix] Isolate conduct.sh runs: own port window, own logs_dir and databases Oct 15, 2026
@aaronsmulktis
aaronsmulktis marked this pull request as ready for review October 16, 2026 14:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant