[BUG] INTERNAL: ERROR_CODE_SCAN due to client cancel in PG

[Disimasa]

Describe the bug
Hi! We use self-hosted Permify in production of our service. And I'm getting an unstable error on production - INTERNAL: ERROR_CODE_SCAN (for example when calling lookupEntity). This error raises randomly, most of the time it works right.

Permify logs:

permify-1  | time=2026-01-12T10:24:28.481Z level=ERROR msg="An operational error occurred" error="ERROR: canceling statement due to user request (SQLSTATE 57014)"
permify-1  | time=2026-01-12T10:24:28.481Z level=ERROR msg=ERROR_CODE_SCAN
permify-1  | time=2026-01-12T10:24:28.481Z level=ERROR msg="finished call" protocol=grpc grpc.component=server grpc.service=base.v1.Permission grpc.method=LookupEntity grpc.method_type=unary peer.address=172.17.2.1:59954 grpc.start_time=2026-01-12T10:24:28Z grpc.code=Internal grpc.error="rpc error: code = Internal desc = ERROR_CODE_SCAN" grpc.time_ms=5.532

Also I found logs in PostgreSQL connected to this Permify error:

{
backend_type: "client backend",
command_tag: "BIND",
connection_from: "localhost:35662",
context: "",
database_name: "permify",
detail: "",
error_severity: "ERROR",
hint: "",
internal_query: "",
internal_query_pos: 0,
leader_pid: 0,
location: "",
message: "canceling statement due to user request",
process_id: 9039,
query: "SELECT entity_type, entity_id, relation, subject_type, subject_id, subject_relation FROM relation_tuples WHERE tenant_id = $1 AND entity_type = $2 AND relation = $3 AND subject_id = $4 AND subject_type = $5 AND (pg_visible_in_snapshot(created_tx_id, (select snapshot from transactions where id = $6::xid8)) = true OR created_tx_id = $7::xid8) AND ((pg_visible_in_snapshot(expired_tx_id, (select snapshot from transactions where id = $8::xid8)) = false OR expired_tx_id = $9::xid8) AND expired_tx_id <> $10::xid8)",
query_id: 0,
query_pos: 0,
session_id: "6964cae2.234f",
session_line_num: 1,
session_start_time: "2026-01-12T13:20:18+03:00",
sql_state_code: 57014,
transaction_id: 0,
}

To Reproduce

I call lookupEntity and other methods by permify-node client from my SvelteKit app like that.

export const schemaVersion = await permify.schema.list({ tenantId }).then(r => r.head)

const response = await permify.permission.lookupEntity({
						tenantId: 't1',
						metadata: { schemaVersion, depth: 40 },
						entityType: target.entity.type,
						permission: target.permission,
						subject: target.subject,
					})

I have such permify config.yaml settings:

server:
  rate_limit: 500
  http:
    enabled: true
    port: 3476
  grpc:
    port: 3478

authn:
  enabled: true
  method: preshared
  preshared:
    keys: ...

service:
  circuit_breaker: false
  watch:
    enabled: false
  schema:
    cache:
      number_of_counters: 1_000
      max_cost: 10MiB
  permission:
    bulk_limit: 100
    concurrency_limit: 100
    cache:
      number_of_counters: 10_000
      max_cost: 10MiB

database:
  engine: postgres
  uri: ...
  auto_migrate: true
  max_open_connections: 100
  max_idle_connections: 10
  max_connection_lifetime: 300s
  max_connection_idle_time: 60s
  garbage_collection:
    enabled: true
    interval: 200h
    window: 200h
    timeout: 5m

My Permify schema - schema.txt

Expected behavior

I checked my Postgres settings (statement_timeout, idle_in_transaction_session_timeout, lock_timeout) and it doesn't have limits. So I guess permify service closes connection.

Environment

• Permify - v1.4.4
• @permify/permify-node - 1.1.4

[polarnik]

One of root causes is a connection limit:

• postgresql server connection limit
• and postgresql database connection limit

We can see this value

max_open_connections: 100

And we can check a server connection limit

show all

and a database connection limit

SELECT datname, datconnlimit FROM pg_database

some workarounds

max_open_connections >= datconnlimit;

max_open_connections=100; datconnlimit=100

-- update:
ALTER DATABASE "permify-tenants" CONNECTION LIMIT 400;
-- and check
SELECT datname, datconnlimit FROM pg_database

or

max_open_connections=50

max_open_connections >= max_connections;

max_open_connections=100; max_connections=100

https://www.postgresql.org/docs/current/runtime-config-connection.html

max_connections (integer)
Determines the maximum number of concurrent connections to the database server. The default is typically 100 connections, but might be less if your kernel settings will not support it (as determined during initdb). This parameter can only be set at server start.

# update postgresql.conf
# increase max_connections from 100 to 1000

# restart the postgresql

-- check the settings
show all

or

max_open_connections=50