Supabase Async client does not close httpx client leading to `OSError: [Errno 24] Too many open files`

[crro]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

Creating a new Supabase async client per request in FastAPI causes 'Too many open files (OSError: [Errno 24])'

To Reproduce

1. Start a FastAPI app with the following code:
   I used poetry for the project, this is what my pyproject looks like:

[tool.poetry]
name = "test-supabase"
version = "0.1.0"
description = ""
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
fastapi = "^0.115.11"
supabase = "^2.13.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This is the code to repro:

    from fastapi import FastAPI, Depends
    from supabase._async.client import AsyncClient as Client, create_client
    
    SUPABASE_URL = "SUPABASE_URL"
    SUPABASE_ANON_KEY = "SUPABASE_ANON_KEY"
    
    app = FastAPI()
    
    # Dependency that creates a new Supabase client per request
    async def get_supabase_client():
        supabase = await create_client(SUPABASE_URL, SUPABASE_ANON_KEY)  # Creates a new client per request
        return supabase
    
    @app.get("/")
    async def test_supabase(supabase: Client = Depends(get_supabase_client)):
        """A route that uses Supabase."""
        res = await supabase.table("sample_table").select("*").limit(1).execute()
        return res

2. Run the FastAPI app:

   uvicorn main:app --reload

3. Send concurrent requests using wrk:

   ab -n 5000 -c 100 http://127.0.0.1:8000/

4. Observe the error in logs:

   OSError: [Errno 24] Too many open files
   

INFO:     127.0.0.1:59916 - "GET / HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
    solved_result = await solve_dependencies(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 638, in solve_dependencies
  File "/Users/crro/dev/test-supabase/test_supabase/main.py", line 15, in get_supabase_client
    supabase = await create_client(SUPABASE_URL, SUPABASE_ANON_KEY)  # Creates a new client per request
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/supabase/_async/client.py", line 337, in create_client
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/supabase/_async/client.py", line 103, in create
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/supabase/_async/client.py", line 81, in __init__
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/supabase/_async/client.py", line 246, in _init_supabase_auth_client
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/supabase/_async/auth_client.py", line 47, in __init__
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/gotrue/_async/gotrue_client.py", line 101, in __init__
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/gotrue/_async/gotrue_base_api.py", line 28, in __init__
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1402, in __init__
    self._transport = self._init_transport(
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/httpx/_client.py", line 1445, in _init_transport
    return AsyncHTTPTransport(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/httpx/_transports/default.py", line 297, in __init__
    ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crro/Library/Caches/pypoetry/virtualenvs/test-supabase-bTT9SAhI-py3.12/lib/python3.12/site-packages/httpx/_config.py", line 40, in create_ssl_context
  File "/opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 707, in create_default_context
OSError: [Errno 24] Too many open files

Expected behavior

The supabase client should clean up resources after itself to prevent the leaking of file descriptors. I think the fix is to add support so that we can create an async supabase client with async with so that the get_supbase_client dependency looks like this:

async def get_supabase_client():
    async with await create_client(SUPABASE_URL, SUPABASE_ANON_KEY) as supabase:
        yield supabase

I think the fix is to:

1. Implement the __aenter__ and __aexit__ methods in the AsyncSupabaseAuthClient class to ensure the HTTP client is properly closed.
2. Implement the __aenter__ and __aexit__ methods in the AsyncClient class to manage the lifecycle of the auth and realtime clients.

Changes to AsyncSupabaseAuthClient

Add the following methods to AsyncSupabaseAuthClient:

class AsyncSupabaseAuthClient(AsyncGoTrueClient):
    # Existing __init__ method

    async def __aenter__(self):
        return self

    # since https://github.com/supabase/supabase-py/blob/main/supabase/_async/auth_client.py#L9
    # https://github.com/supabase/auth-py/blob/4194347d2c6891fae1e5f7eb6cdbebc1054064cf/supabase_auth/http_clients.py#L3
    # https://github.com/encode/httpx/blob/9e8ab40369bd3ec2cc8bff37ab79bf5769c8b00f/httpx/_client.py#L2008
    async def __aexit__(self, exc_type, exc_value, traceback):
        if not self._http_client_provided and self.http_client:
            await self.http_client.__aexit__(exc_type, exc_value, traceback)

Changes to AsyncClient

Add the following methods to AsyncClient:

class AsyncClient:
    # Existing __init__ method

    async def __aenter__(self):
        await self.auth.__aenter__()
        return self

    async def __aexit__(self, exc_type, exc_value, traceback):
        await self.auth.__aexit__(exc_type, exc_value, traceback)
        await self.realtime.close() #https://github.com/supabase/realtime-py/blob/628f1e688dd2339efff560a314cbef9472df363c/realtime/_async/client.py#L199

Screenshots

It first runs out of file descriptors

then it just fails on subsequent connections

System information

• OS: macOS
• Browser (if applies) N/A
• Version of supabase-py: ^2.13.0
• Version of Node.js: v20.3.1

Additional context

Happy to contribute this fix but don't know if I'm missing something

[linear]

CLIBS-132 Supabase Async client does not close httpx client leading to `OSError: [Errno 24] Too many open files`

[crro]

Tried this instead to try to get the GC to clean it up but it still fails:

# Dependency that creates a new Supabase client per request
async def get_supabase_client():
    supabase = await create_client(SUPABASE_URL, SUPABASE_ANON_KEY)
    try:
        yield supabase  # Inject client
    finally:
        supabase = None  # Attempted cleanup

[grdsdev]

Hi @crro thanks for the detailed issue description.

I'm not sure how FastAPI works, but can't you reuse the client between requests?

[crro]

Hey @grdsdev, we can and it works great when we use the service_role key that bypasses RLS. But when we want to enforce RLS by setting the session on the python client is when this becomes a problem.

In this scenario imagine user A and user B making concurrent requests, user A request gets there first but halfway through completion user B request arrives. If we re-use the client we would swap the session while user A still needs to use it to finish some of it's action and those will fail because now it's user B. So ideally we should be able to create one client per request like we can with the js sdk and just ensure that everything is cleaned correctly.

[crro]

Update, by manually closing the resources I do get up to 100 concurrent requests without error:

# Dependency that creates a new Supabase client per request
async def get_supabase_client():
    supabase = await create_client(SUPABASE_URL, SUPABASE_ANON_KEY)  # Creates a new client per request
    try:
        yield supabase
    finally:
        
        if supabase.realtime is not None:
            await supabase.realtime.close()

        if supabase._postgrest is not None:
            await supabase._postgrest.aclose()
            await supabase._postgrest.session.aclose()

        if supabase.auth is not None:
            await supabase.auth._http_client.aclose()
        supabase = None

Concurrency Level:      100
Time taken for tests:   87.846 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      840000 bytes
HTML transferred:       120000 bytes
Requests per second:    56.92 [#/sec] (mean)
Time per request:       1756.919 [ms] (mean)
Time per request:       17.569 [ms] (mean, across all concurrent requests)
Transfer rate:          9.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.0      1      25
Processing:   188 1749 319.5   1690    3101
Waiting:      186 1712 321.1   1667    3054
Total:        188 1750 319.5   1691    3102

Percentage of the requests served within a certain time (ms)
  50%   1691
  66%   1942
  75%   2022
  80%   2052
  90%   2154
  95%   2203
  98%   2257
  99%   2304
 100%   3102 (longest request)

[silentworks]

I've tested this locally and haven't ran into this issue. I tried both ab and oha for load testing and both seem to not get the error thrown. Can you provide some more information about your system spec as maybe my system has enough memory to open many files without any errors.

[silentworks]

Also something else to note, you are running uvicorn in dev mode as you have the --reload flag turned on. I just did another test with uvicorn in production mode with 4 workers and still getting successful responses.

uvicorn main:app --workers 4

[silentworks]

I've setup an example repo with the code I used to test this out. https://github.com/silentworks/sb-python-issues/tree/main/supabase_py_1075

[crro]

Thanks @silentworks! I really need to migrate to uv because when i run your repo with it I don't see it even with the --reload flag on and 4 workers.

When I run it without uv, just uvicorn main:app --workers 4 and without the --reload flag I see it.

It seems like the resource leak still happens, but uv is more efficient at cleaning up quickly?

I have a macbook pro with m1 chip and 16GB of ram.

[silentworks]

@crro you are correct indeed. I will look into this a bit more now as running it with only uvicorn does result in an error for every request made. Thanks again for providing this much information.

[silentworks]

Sorry my mistake, I didn't have my Supabase instance running so it was giving me errors about that. I've now gotten my instance running and haven't gotten any errors since then. I'm on a Lenovo Thinkpad Intel i7-8550U 4GHz with 16 GB of ram and running on Ubuntu 24.04.2. All my requests passed without any errors. I've attached a video showing this.

uvicorn_oha_benchmark.mp4