Repository navigation
[Fix] Isolate conduct.sh runs: own port window, own logs_dir and databases - #88
Open
aaronsmulktis wants to merge 5 commits into
Open
aaronsmulktis wants to merge 5 commits into
aaronsmulktis wants to merge 5 commits into
Conversation
conduct.sh never passed job_id, so every run in the pool scanned for a port from 5001 at once. Two runs could pick the same free port before either web app bound it; the loser exited with 'address already in use' while its health check found the winner's app. Pass ++job_id=$((i % 500)) so each run gets the 100-port window AgentLauncher already reserves per job.
5 tasks done
logs_dir defaults to log_outputs/<now to the second>, and databases_dir lives inside it, so pool runs started in the same second shared both: in a 10-run pool, 4 runs shared one directory. Their web apps read and wrote the same SQLite files, reseeding each other mid-episode and leaking edits into each other's rewards, and the sibling-job guard could not tell them apart because it keys on databases_dir. Pass logs_dir=<batch>/run-NNN per run, and point record_video_dir at one batch folder.
aaronsmulktis
marked this pull request as ready for review
October 16, 2026 14:38
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this PR?
scripts/conduct.shruns up toMAX_PARALLEL(default 4)launch_agent.pyprocesses at once on one machine, either locally or on a node viaconduct_slurm.sh. It never isolated them from each other, in two ways.Ports. It never passed a
job_id, so every run in the pool searched for a free port starting at 5001.AgentLauncher.pick_empty_porttests a port by binding it and releasing it immediately. The web app doesn't claim the port until its subprocess starts, a few seconds to half a minute later. In that gap two runs can pick the same port. The loser's web app exits withaddress already in use, and its health check finds the winner's app answering instead. From aconduct.shpool on a cluster node:Databases. Under
mode=local,logs_dirislog_outputs/<timestamp to the second>-$USER, and the SQLitedatabases_dirlives inside it. Runs started in the same second shared both: in a 10-run pool, 4 runs shared one directory and 2 shared another. Their web apps read and wrote the same database files. One run's reseed on startup reset another mid-episode, and each agent's edits showed up in its neighbours' reward diffs. The sibling-job guard couldn't catch it, because it tells runs apart bydatabases_dir. Anyconduct.shresult withMAX_PARALLEL > 1may be affected.What changes
Each run in the pool now gets:
++job_id=$((i % 500)), which selects the 100-port windowAgentLauncher._job_port_offsetalready reserves per job (5001 + job_id*100).++becausejob_idisn't declared inconfig/config.yaml; a plainjob_id=is rejected by Hydra.% 500keeps the top of the window under 65535 for anyCOUNT.logs_dir=<batch>/run-NNN, so each run has its own config,webapp.logand databases. Those sit next to the per-run stdout log that was already in<batch>/.record_video_dir=<batch>/recordings, so a batch's videos still land in one folder. The_job<N>suffix keeps their names apart.launch_parallel_agents.pyalready did both (job_id = i,logs_dir/<i>).conduct.shwas the only launcher that didn't. Forwarded overrides come last, so passing your ownlogs_dir=orrecord_video_dir=still wins.Test plan
++job_id=3resolves to3;job_id=3is rejected. A repeated value override resolves to the last one, so forwarded args win.bash -n scripts/conduct.sh.AGENTS=dummy, 10 runs, pool of 4. All completed. Each run picked a port in its own window (5101 … 5901), with noDIFFERENT job's configlines. Run 0 found 5001–5003 held by web servers leaked from earlier runs (fixed separately in the process-group cleanup PR) and moved to 5004, still inside its own window.AGENTS=dummy COUNT=4 MAX_PARALLEL=4, all four started together. Each got its ownrun-00N/with its owndatabases/and its own port (5001, 5101, 5201, 5301). No sharedlog_outputs/<timestamp>directory was created.Until this merges,
MAX_PARALLEL=1avoids both problems.