Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/airflow] error - runtime/cgo: pthread_create failed: Operation not permitted #63489

Closed
alekseyolg opened this issue Mar 1, 2024 · 3 comments
Closed

[bitnami/airflow] error - runtime/cgo: pthread_create failed: Operation not permitted #63489

alekseyolg opened this issue Mar 1, 2024 · 3 comments
Assignees
@juan131
Labels
airflow-scheduler airflow-worker solved tech-issues The user has a technical issue about an application

Comments

@alekseyolg
Copy link
Contributor

alekseyolg commented Mar 1, 2024 •
edited by carrodher
Loading

Name and Version

bitnami/airflow-[scheduler,worker]:2.8.2-debian-12-r0

What architecture are you using?

amd64

What steps will reproduce the bug?

extraEnvVars:
  - name: AIRFLOW__API__AUTH_BACKENDS
    value: "airflow.api.auth.backend.basic_auth"
  - name: AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS
    value: "True"

serviceAccount:
  create: true
  name: ""
  automountServiceAccountToken: true
  annotations: {}
rbac:
  create: true
executor: KubernetesExecutor
worker:
  automountServiceAccountToken: true
  livenessProbe:
    enabled: true
    initialDelaySeconds: 180
    periodSeconds: 86400
    timeoutSeconds: 5
    failureThreshold: 1
    successThreshold: 1
  readinessProbe:
    enabled: true
    initialDelaySeconds: 86400
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 1
    successThreshold: 1
web:
  automountServiceAccountToken: true
  resources:
    requests:
      cpu: 1
      memory: 1Gi
    limits:
      cpu: 2
      memory: 4Gi
scheduler:
  automountServiceAccountToken: true
  resources:
    requests:
      cpu: 1
      memory: 1Gi
    limits:
      cpu: 4
      memory: 8Gi
  podSecurityContext:
    enabled: false
    fsGroup: 1001
externalDatabase:
  host: pre-postgres-airflow
  port: 5432
  user: bn_airflow
  database: bitnami_airflow
  password: ""
  existingSecret: "postgresql"
  existingSecretPasswordKey: ""

What is the expected behavior?

Correct operation of all modules.

What do you see instead?

werf helm upgrade --install airflow-dwh-gb . --version 0.0.1-4R --create-namespace --namespace airflow-dwh-gb -f values-preprod.yaml --atomic --timeout 5m --status-progress-period=300
Release "airflow-dwh-gb" does not exist. Installing it now.

┌ Waiting for resources to become ready
│ ┌ deploy/airflow-dwh-gb-scheduler po/airflow-dwh-gb-scheduler-846879b6fd-dj5fs container/clone-repositories logs
│ │  07:11:59.86 INFO  ==>
│ │  07:11:59.86 INFO  ==> Welcome to the Bitnami git container
│ │  07:11:59.86 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
│ │  07:11:59.87 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
│ │  07:11:59.87 INFO  ==>
│ │ Cloning into '/dags/dags'...
│ └ deploy/airflow-dwh-gb-scheduler po/airflow-dwh-gb-scheduler-846879b6fd-dj5fs container/clone-repositories logs
│
│ ┌ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/clone-repositories logs
│ │  07:11:59.91 INFO  ==>
│ │  07:11:59.91 INFO  ==> Welcome to the Bitnami git container
│ │  07:11:59.91 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
│ │  07:11:59.91 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
│ │  07:11:59.91 INFO  ==>
│ │ Cloning into '/dags/dags'...
│ └ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/clone-repositories logs
│
│ ┌ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/airflow-web logs
│ │ airflow 07:12:00.77 INFO  ==>
│ │ airflow 07:12:00.77 INFO  ==> Welcome to the Bitnami airflow container
│ │ airflow 07:12:00.77 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
│ │ airflow 07:12:00.77 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
│ │ airflow 07:12:00.78 INFO  ==>
│ │ airflow 07:12:00.78 INFO  ==> Enabling non-root system user with nss_wrapper
│ │ airflow 07:12:00.78 INFO  ==> ** Starting Airflow setup **
│ │ airflow 07:12:00.79 INFO  ==> Validating settings in POSTGRESQL_CLIENT_* env vars
│ │ airflow 07:12:00.81 INFO  ==> Initializing Airflow ...
│ │ airflow 07:12:00.82 INFO  ==> No injected configuration file found. Creating default config file
│ │ airflow 07:12:01.22 INFO  ==> Configuring Airflow webserver authentication
│ │ airflow 07:12:01.22 INFO  ==> Enabling LDAP authentication
│ │ airflow 07:12:01.26 INFO  ==> Configuring Airflow database
│ │ airflow 07:12:01.28 INFO  ==> Trying to connect to the database server
│ │ airflow 07:12:05.47 INFO  ==> Upgrading database schema
│ └ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/airflow-web logs
│
│ ┌ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/sync-repositories logs
│ │  07:12:00.61 INFO  ==>
│ │  07:12:00.61 INFO  ==> Welcome to the Bitnami git container
│ │  07:12:00.61 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
│ │  07:12:00.61 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
│ │  07:12:00.61 INFO  ==>
│ │ From bitbucket.playteam.ru:scm/analytics/airflow-dwh-dags
│ │  * branch            develop    -> FETCH_HEAD
│ │ Already up to date.
│ └ deploy/airflow-dwh-gb-web po/airflow-dwh-gb-web-6465489d5f-znzz9 container/sync-repositories logs
│
│ ┌ deploy/airflow-dwh-gb-scheduler po/airflow-dwh-gb-scheduler-846879b6fd-dj5fs container/airflow-scheduler logs
│ │ airflow-scheduler 07:12:01.65 INFO  ==>
│ │ airflow-scheduler 07:12:01.65 INFO  ==> Welcome to the Bitnami airflow-scheduler container
│ │ airflow-scheduler 07:12:01.65 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
│ │ airflow-scheduler 07:12:01.65 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
│ │ airflow-scheduler 07:12:01.66 INFO  ==>
│ │ airflow-scheduler 07:12:01.66 INFO  ==> Enabling non-root system user with nss_wrapper
│ │ airflow-scheduler 07:12:01.66 INFO  ==> ** Starting Airflow setup **
│ │ airflow-scheduler 07:12:01.69 INFO  ==> No injected configuration file found. Creating default config file
│ │ airflow-scheduler 07:12:03.53 INFO  ==> Configuring Airflow database
│ │ airflow-scheduler 07:12:03.55 INFO  ==> Trying to connect to the database server
│ │ airflow-scheduler 07:12:05.65 INFO  ==> Waiting for Airflow Webserver to be up
│ │ runtime/cgo: pthread_create failed: Operation not permitted
│ │ SIGABRT: abort
│ │ PC=0x7f17cc0ede2c m=0 sigcode=18446744073709551610
│ │
│ │ goroutine 0 [idle]:
│ │ runtime: g 0: unknown pc 0x7f17cc0ede2c
│ │ stack: frame={sp:0x7ffd6a7dc7d0, fp:0x0} stack=[0x7ffd69fddc50,0x7ffd6a7dcc60)
│ │ 0x00007ffd6a7dc6d0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc6e0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc6f0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc700:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc710:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc720:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc730:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc740:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc750:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc760:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc770:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc780:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc790:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7a0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7b0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7c0:  0x0000000000000000  0x00007f17cc0ede1e
│ │ 0x00007ffd6a7dc7d0: <0x0000000000000000  0xab0857554963b900
│ │ 0x00007ffd6a7dc7e0:  0x0000000000000006  0x00007f17cc060740
│ │ 0x00007ffd6a7dc7f0:  0x00007ffd6a7dcaa0  0x0000000000000000
│ │ 0x00007ffd6a7dc800:  0x00000000006079e0  0x00007f17cc09efb2
│ │ 0x00007ffd6a7dc810:  0x00007f17cc236e70  0x00007f17cc089472
│ │ 0x00007ffd6a7dc820:  0x0000000000000020  0x0000000000000000
│ │ 0x00007ffd6a7dc830:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc840:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc850:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc860:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc870:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc880:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc890:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc8a0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc8b0:  0x0000000000000000  0xab0857554963b900
│ │ 0x00007ffd6a7dc8c0:  0x00007f17cc236840  0x00007f17cc236840
│ │ runtime: g 0: unknown pc 0x7f17cc0ede2c
│ │ stack: frame={sp:0x7ffd6a7dc7d0, fp:0x0} stack=[0x7ffd69fddc50,0x7ffd6a7dcc60)
│ │ 0x00007ffd6a7dc6d0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc6e0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc6f0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc700:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc710:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc720:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc730:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc740:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc750:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc760:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc770:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc780:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc790:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7a0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7b0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc7c0:  0x0000000000000000  0x00007f17cc0ede1e
│ │ 0x00007ffd6a7dc7d0: <0x0000000000000000  0xab0857554963b900
│ │ 0x00007ffd6a7dc7e0:  0x0000000000000006  0x00007f17cc060740
│ │ 0x00007ffd6a7dc7f0:  0x00007ffd6a7dcaa0  0x0000000000000000
│ │ 0x00007ffd6a7dc800:  0x00000000006079e0  0x00007f17cc09efb2
│ │ 0x00007ffd6a7dc810:  0x00007f17cc236e70  0x00007f17cc089472
│ │ 0x00007ffd6a7dc820:  0x0000000000000020  0x0000000000000000
│ │ 0x00007ffd6a7dc830:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc840:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc850:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc860:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc870:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc880:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc890:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc8a0:  0x0000000000000000  0x0000000000000000
│ │ 0x00007ffd6a7dc8b0:  0x0000000000000000  0xab0857554963b900
│ │ 0x00007ffd6a7dc8c0:  0x00007f17cc236840  0x00007f17cc236840
│ │
│ │ goroutine 1 [running]:
│ │ runtime.systemstack_switch()
│ │     /opt/bitnami/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00005c740 sp=0xc00005c730 pc=0x464ba8
│ │ runtime.main()
│ │     /opt/bitnami/go/src/runtime/proc.go:169 +0x6d fp=0xc00005c7e0 sp=0xc00005c740 pc=0x438b8d
│ │ runtime.goexit()
│ │     /opt/bitnami/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x466b81
│ │
│ │ rax    0x0
│ │ rbx    0x43
│ │ rcx    0xffffffffffffffff
│ │ rdx    0x6
│ │ rdi    0x43
│ │ rsi    0x43
│ │ rbp    0x7f17cc060740
│ │ rsp    0x7ffd6a7dc7d0
│ │ r8     0x0
│ │ r9     0x73
│ │ r10    0x8
│ │ r11    0x246
│ │ r12    0x6
│ │ r13    0x0
│ │ r14    0x6079e0
│ │ r15    0x6
│ │ rip    0x7f17cc0ede2c
│ │ rflags 0x246
│ │ cs     0x33
│ │ fs     0x0
│ │ gs     0x0
│ └ deploy/airflow-dwh-gb-scheduler po/airflow-dwh-gb-scheduler-846879b6fd-dj5fs container/airflow-scheduler logs

Additional information

When working on images based on Debian 11 there are no problems, when using images based on Debian 12 I get the errors described above, they appear in all containers, the errors are the same everywhere.
kubernetes version 1.27.2
containerd://1.6.14
@alekseyolg alekseyolg added the tech-issues The user has a technical issue about an application label Mar 1, 2024
@github-actions github-actions bot added the triage Triage is needed label Mar 1, 2024
@carrodher
Copy link
Member

Are you able to reproduce the issue when installing the Helm chart by simply running helm install airflow oci://registry-1.docker.io/bitnamicharts/airflow? If that deployment succeeds, what happens when you use your custom values helm install airflow oci://registry-1.docker.io/bitnamicharts/airflow -f values-preprod.yaml?

@alekseyolg
Copy link
Contributor Author

@carrodher i use: When using standard `helm install airflow oci://registry-1.docker.io/bitnamicharts/airflow' values, redis and Postgres do not start for me:
Redis

1:C 04 Mar 2024 15:38:31.451 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 04 Mar 2024 15:38:31.451 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 04 Mar 2024 15:38:31.451 * Configuration loaded
1:M 04 Mar 2024 15:38:31.451 * monotonic clock: POSIX clock_gettime
1:M 04 Mar 2024 15:38:31.452 * Running mode=standalone, port=6379.
1:M 04 Mar 2024 15:38:31.452 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 04 Mar 2024 15:38:31.452 # Fatal: Can't initialize Background Jobs. Error message: Operation not permitted

Postgres

postgresql 15:38:39.46 INFO  ==> 
postgresql 15:38:39.46 INFO  ==> Welcome to the Bitnami postgresql container
postgresql 15:38:39.47 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 15:38:39.47 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 15:38:39.47 INFO  ==> 
postgresql 15:38:39.48 INFO  ==> ** Starting PostgreSQL setup **
postgresql 15:38:39.50 INFO  ==> Validating settings in POSTGRESQL_* env vars..
postgresql 15:38:39.51 INFO  ==> Loading custom pre-init scripts...
postgresql 15:38:39.51 INFO  ==> Initializing PostgreSQL database...
postgresql 15:38:39.52 INFO  ==> pg_hba.conf file not detected. Generating it...
postgresql 15:38:39.52 INFO  ==> Generating local authentication configuration
postgresql 15:39:12.47 INFO  ==>

and reebot.
I used the latest images for this:

bitnami/postgresql:16.2.0-debian-12-r5
bitnami/redis:7.2.4-debian-12-r9

If I replace the Redis and Postgres images, I get the same error from airflow as above (attached in the screenshot).
image

When I use the command helm install airflow oci://registry-1.docker.io/bitnamicharts/airflow -f values-preprod.yaml
I get exactly the same answer as at the beginning of the conversation.

@github-actions github-actions bot removed the triage Triage is needed label Mar 6, 2024
@github-actions github-actions bot assigned juan131 and unassigned carrodher Mar 6, 2024
@juan131
Copy link
Contributor

juan131 commented Mar 6, 2024

Hi @alekseyolg

This is an issue already reported already in Docker library and it's related to libseccomp, please check the response below:

Please note container images rely on the host's kernel, so you'll have to adapt your cluster' kernel in order to use Debian 12 images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juan131 juan131

Labels
airflow-scheduler airflow-worker solved tech-issues The user has a technical issue about an application
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@juan131 @carrodher @alekseyolg