Repository navigation
fix(indexer): let box-deploy recover a box whose serving container is down - #130
Merged
Merged
Conversation
… down The #129 deploy went out as an in-place image swap on atp-indexer-si-v2. Ponder fingerprints the whole src/ tree, not just ponder.config.ts, so the new image refused to start on a schema an older build created. box-deploy.sh then exited without restoring the previous image, which left the prod API returning 502. Redeploying with a fresh schema could not fix it either. The si-v3 rebuild completed, but the script finds the old schema by asking the serving container, and that container was crash-looping. CURRENT came back empty, psql ran against "".atp_position, the failing command substitution tripped set -e, and the script died before the flip. Three changes: - current_schema falls back to docker-compose.override.yml, which this script writes and which records what the box was serving, whenever the container can't answer. - A failed /ready after an image swap restores the previous image instead of leaving the box down. - psql_count tolerates a schema that doesn't exist. The parity check then compares against 0 rather than killing the script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prod
api.stake.aztec.networkis down (502). The #129 image swap onatp-indexer-si-v2failed, because Ponder fingerprints all ofsrc/and refuses to start on the old schema. The script then left the broken container serving.A redeploy to
atp-indexer-si-v3rebuilt fine, but died before the flip. The script reads the old schema name from the serving container, which is down. With an empty name the paritypsqlfailed underset -e.This PR:
docker-compose.override.ymlwhen the container can't answer;/readycheck fails;psql_counttolerate a missing schema.After merge, rerun
deploy-indexer.yamlfor prod frommainwithnew_schema=atp-indexer-si-v3. The script reuses the si-v3 rebuild that already finished, andatp-indexer-si-v2stays in Postgres as the rollback.