Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] spk publish (debug build) hangs when erroring #990

Closed
dcookspi opened this issue Feb 29, 2024 · 5 comments · Fixed by #991
Closed

[BUG] spk publish (debug build) hangs when erroring #990

dcookspi opened this issue Feb 29, 2024 · 5 comments · Fixed by #991
Assignees
Labels
bug Something isn't working SPI AOI Area of interest for SPI SPI-0.41

Comments

@dcookspi
Copy link
Collaborator

dcookspi commented Feb 29, 2024

spk publish from a debug builds hangs when erroring instead of outputting the error and exiting.

To Reproduce
Steps to reproduce the behavior:

  1. Have a copy of a local package that is also already published in the origin repo. I ran spk convert pip ... using an existing package/version, but stopped it from publishing thebuild, e.g. spk convert pip -- --python-version 3.7 --python-abi cp37m colorama==0.4.6 and answer 'N' when asked.
  2. Checkout the main branch
  3. Build the debug version of spk
  4. Run target/debug/spk publish -r origin python-colorama/0.4.6
  5. It should error and exit, but it will hang after outputting just:
 INFO loading recipe: python-colorama/0.4.6
 INFO publishing recipe: python-colorama/0.4.6

  1. But if you build the release verison of spk and try the same publish command, it operate normally, won't hang, and will output the correct error message (...Version exists) and exit.

Expected behavior

> target/debug/spk publish -r origin python-colorama/0.4.6
 INFO loading recipe: python-colorama/0.4.6
 INFO publishing recipe: python-colorama/0.4.6
Failed to publish recipe python-colorama/0.4.6: Version exists

Additional context
The debug build does not hang if you downgrade the sentry and sentry-tracing dependencies in the Cargo.toml from 0.32.2 to the previous version spk used: 0.27.0

Added a branch and PR with the sentry rollback that seems to fix it: #991

@dcookspi dcookspi added bug Something isn't working SPI AOI Area of interest for SPI SPI-0.41 labels Feb 29, 2024
@jrray jrray changed the title [BUG] spk publish (bebug build) hangs when erroring [BUG] spk publish (debug build) hangs when erroring Feb 29, 2024
@dcookspi dcookspi self-assigned this Mar 1, 2024
@dcookspi
Copy link
Collaborator Author

dcookspi commented Mar 1, 2024

We decided to go with rolling back to the earlier sentry version.

@jrray
Copy link
Collaborator

jrray commented Mar 1, 2024

Using git bisect on the sentry-rust repo I tracked the hanging down to this commit.

Then, going back to the "good" version 0.27.0 but enabling the debug-images feature at that version causes the same hang. So the problem isn't new in the latest sentry release but from enabling this feature. Sure enough, using 0.32.2 with --no-default-features doesn't hang.

I'm about to start looking into what that feature does.

BTW I found a simple way to reproduce the hang by simply doing publish -r non-existent-repo ... to cause a sentry error to be produced from the repo being unknown.

@jrray
Copy link
Collaborator

jrray commented Mar 1, 2024

Attaching a debugger to the hanging spk process shows something similar to what we saw in the past:

  1. some error causing sentry to process an event
  2. a lazy init static field DEBUG_META is attempted to be initialized for the first time
  3. [inside lazy init code] something panics
  4. [inside lazy init code] sentry panic handler tries to generate a new event
  5. [inside lazy init code] same DEBUG_META is attempted to be initialized for the first time
  6. deadlock

@jrray
Copy link
Collaborator

jrray commented Mar 1, 2024

I found an issue that is effectively about our problem: getsentry/sentry-rust#358

jrray pushed a commit that referenced this issue Mar 2, 2024
Avoid enabling the default feature `debug-images` which was causing the
deadlock on centos 7.

`cargo tree -e features -i sentry` is useful for verifying the list of
enabled features.

More details can be found in issue #990.

Signed-off-by: J Robert Ray <[email protected]>
@onelson
Copy link

onelson commented Mar 2, 2024

I found an issue that is effectively about our problem: getsentry/sentry-rust#358

This was 2 employers and 3 years ago but I do seem to remember the defect I reported to sentry was not present when we tested on a later Linux kernel. Nobody was thrilled with the idea of deviating from the kernel Centos 7 brought by default. I'm sorry to hear this is still something to worry about 😭

Nice job narrowing things to the debug-images feature 👏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working SPI AOI Area of interest for SPI SPI-0.41
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants