Memory not being released by httpx

[splotsh]

I've been spending a bit of time this week hunting down a memory leak. I'm able to reliably produce this now with a few lines of code, and it seems to be related to httpx. I've run some tests using memory_profiler comparing aiohttp and httpx to ensure that what I was expecting to see is valid. aiohttp does not exhibit the same problem.

The code (comment/uncomment each section to compare):

import httpx
import asyncio
from memory_profiler import profile
import aiohttp


@profile(precision=4)
async def memory_test(url):
	
	'''
        async with aiohttp.ClientSession() as session:
		async with session.get(url) as response:
			html = await response.text()	
			print(f'Length of response is: {len(html)}')
	'''
	async with httpx.AsyncClient(http2=True) as client:
		
		html = await client.get(url, follow_redirects=True)
		print(f'Length of response is: {len(html.text)}')	
		
	del html	
	return None

async def main():
	url = 'https://www.autoscout24.fr/offres/bmw-320-serie-3-touring-e91-touring-163ch-pack-m-sport-diesel-bleu-671904de-6139-4061-a451-f63bdb61de2b'
	result = await memory_test(url)

if __name__ == "__main__":
    asyncio.run(main())    

For the version using httpx, memory profiler shows me:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10  84.7266 MiB  84.7266 MiB           1   @profile(precision=4)
    11                                         async def memory_test(url):	
    12                                         		
    13                                         	'''
    14                                         	async with aiohttp.ClientSession() as session:
    15                                         		async with session.get(url) as response:
    16                                         			html = await response.text()	
    17                                         			print(f'Length of response is: {len(html)}')
    18                                         	
    19                                         	'''
    20  89.1055 MiB   1.8125 MiB           4   	async with httpx.AsyncClient(http2=True) as client:
    21                                         		
    22  88.2461 MiB   1.7070 MiB          91   		html = await client.get(url, follow_redirects=True)
    23  89.1055 MiB   0.8594 MiB           1   		print(f'Length of response is: {len(html.text)}')	
    24                                         	
    25                                         	
    26  89.1055 MiB   0.0000 MiB           1   	del html	
    27  89.1055 MiB   0.0000 MiB           1   	return None

The memory for the variable html holding the response is not freed.
Whereas the results for the same code with aiohttp:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    10  84.6523 MiB  84.6523 MiB           1   @profile(precision=4)
    11                                         async def memory_test(url):	
    12                                         		
    13                                         	
    14  88.2812 MiB   0.0000 MiB           3   	async with aiohttp.ClientSession() as session:
    15  88.2812 MiB   2.2344 MiB           7   		async with session.get(url) as response:
    16  88.2812 MiB   1.3945 MiB           3   			html = await response.text()	
    17  88.2812 MiB   0.0000 MiB           1   			print(f'Length of response is: {len(html)}')
    18                                         	
    19  88.2812 MiB   0.0000 MiB           1   	'''
    20                                         	async with httpx.AsyncClient(http2=True) as client:
    21                                         		
    22                                         		html = await client.get(url, follow_redirects=True)
    23                                         		print(f'Length of response is: {len(html.text)}')	
    24                                         	'''
    25                                         	
    26  87.6484 MiB  -0.6328 MiB           1   	del html	
    27  87.6484 MiB   0.0000 MiB           1   	return None

Maybe I'm doing something wrong?
Any help would be much appreciated.

Edit:
Running gc.collect() does seem to free additional memory in the httpx version of the above code, but it's still less than what the other library releases and I'm not sure if this is the solution.

[splotsh]

Digging further and extending my test, I see that over time memory does get freed and there is no memory leak here (It levels out after just a few requests as can also be seen here #1893)
The test below requests a 100kb file, 300 times and then measuring the amount of memory used.
After 300 requests, there is an overall memory increase of only ~5mb in my tests.

Code below:

import httpx
import asyncio
from memory_profiler import profile
import gc

RUNS = 150

def getCurrentMemoryUsage() -> int:    
    """Return memory usage in kB
    """
    with open('/proc/self/status') as f:
        memusage = f.read().split('VmRSS:')[1].split('\n')[0][:-3]

    return int(memusage.strip())

#@profile(precision=4)
async def memory_test(client,url):			
			
	html = await client.get(url, follow_redirects=True)		
	return html

async def main():
	url = 'https://httpbin.org/bytes/100000'
	init_mem_usage = getCurrentMemoryUsage()

	async with httpx.AsyncClient(http2=True) as client:
		for i in range(RUNS):
			result = await memory_test(client,url)
			print(f'Current memory usage is: {getCurrentMemoryUsage()/1000}mb')	

	gc.collect()
	print(f'Final memory usage: {getCurrentMemoryUsage()/1000}mb')
	print(f'Additional memory used after {RUNS} requests: {(getCurrentMemoryUsage()-init_mem_usage)/1000}mb')

if __name__ == "__main__":
    asyncio.run(main())    

[lovelydinosaur]

Ace. Thanks for the update.