Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve is_closed effectiveness #41

Open
wants to merge 3 commits into
base: neon
Choose a base branch
from
Open

Conversation

conradludgate
Copy link

@ololobus had some issues in compute_ctl with is_closed() not detecting postgres connections being closed. There seemed to be two general problems. More details in sfackler#1229 but this is an early patch in before we can get it in upstream.

@conradludgate conradludgate requested a review from ololobus March 26, 2025 13:10
@conradludgate
Copy link
Author

To fix clippy: #42

@ololobus
Copy link
Member

Thanks for looking into it!

@ololobus
Copy link
Member

LGTM, but I don't know the protocol that well, so cc @hlinnaka @knizhnik just in case

Copy link

@hlinnaka hlinnaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Between compute_ctl and postgres, a heartbeat-like message seems unnecessary. Can you merely check if the socket is still open? Within the same host, you shouldn't get into a situation where the client thinks that the connection is still open even though the server has already closed it.

Comment on lines +492 to +496
/// Check the connection is alive and wait for the confirmation.
pub async fn check_connection(&self) -> Result<(), Error> {
// sync is a very quick message to test the connection health.
query::sync(self.inner()).await
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Sync message affects the protocol state, it's not a simple "no-op" message. Per docs:

At completion of each series of extended-query messages, the frontend should issue a Sync message. This parameterless message causes the backend to close the current transaction if it's not inside a BEGIN/COMMIT transaction block (“close” meaning to commit if no error, or roll back if error). Then a ReadyForQuery response is issued. The purpose of Sync is to provide a resynchronization point for error recovery.

So we'll have to be careful to only call it when not in the middle of processing a query.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true, fortunately the query encoding on the client is always atomic

@ololobus
Copy link
Member

Can you merely check if the socket is still open? Within the same host, you shouldn't get into a situation where the client thinks that the connection is still open even though the server has already closed it.

It reproduces with my new test locally if I comment this line out https://github.com/neondatabase/neon/pull/11346/files#diff-31e2bc8fcc7da75eeda79b330b11c902f0031b77ba28df20874a45b71a53f541R121. I think that we currently do not reconnect at all if connection to postgres gets closed

I was pretty sure that it's some rust-postgres issue, i.e. it tries to write to the closed socket, but I can double-check the sockets state later this week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants