Postgres migration orchestration

[olimart]

I have a very simple setup. One server, 1 db accessory for PG.
I'm trying to evaluate options to migration pg 16 to pg 17.

So far, here is what I'm thinking:

1. Put the app in maintenance mode
   From Kamal docs https://kamal-deploy.org/docs/upgrading/proxy-changes/ maintenance mode coming soon, therefore assuming not ready yet

which I think is summarized here #114 (comment) functionality exists in the proxy but not exposed to kamal yet.

2. Stop/Pause traffic
   Still from Db migrations during changeover from blue to green #114 (comment) it looks like we can wrap
   kamal-proxy pause my-service & kamal-proxy resume my-service
   Therefore I'm thinking about creating a deploy hook for this.

3. Run db data migration
   Create .kamal/hooks/pre-deploy with:

kamal-proxy pause my-service # (does it wait for existing requests to complete? (therefore, can't really predict how long the hook will run))
pg_upgrade ...
kamal-proxy resume my-service

Would that work?

I have no urgency to migrate right now so, if you tell me there is an easier approach coming later, I will gladly wait 😄

cc. @kevinmcconnell 🙏

[kevinmcconnell]

Hey @olimart,

You're on the right track here I think. As you say, there are proxy features for pausing and maintenance mode, but we haven't exposed them in Kamal yet. You can use them, you just need to run the proxy commands directly. You can do that from a deploy hook, or you can put them in a script and run the script yourself (assuming the PG upgrade is not tied to another code change that you're deploying).

kamal-proxy pause does wait for existing requests to complete, so that you can avoid causing them to fail. But it accepts a --drain-timeout option that sets the maximum time for it to wait on the requests (30s by default, but can be whatever you want), and if that elapses it'll cut them off. So you can predict how long that step will take to run (in the worst case; obviously it'll be faster if all the requests finish quickly).

If you think your upgrade is likely to take less than, say, 30s, then pausing can be a good option. If you think you'll need much longer, then kamal-proxy stop is how to trigger maintenance mode. The difference is that with pausing, clients will see slower requests while waiting for the upgrade to complete, and with maintenance mode they'll see a 503 page.

It if were me, I'd probably test the upgrade on a backup to see how long it's likely to take. And if it's pretty quick, I'd write a pause script like your pre-deploy example, but just run it directly in the proxy container (rather than as part of a deploy). But there's lots of options here, so it's totally up to you :)

Lastly, we do plan to make this a bit easier in the future by exposing this functionality in Kamal. But I don't have an ETA for that yet, so I wouldn't necessarily suggest waiting.

[olimart]

just run it directly in the proxy container (rather than as part of a deploy)

thanks.