Feature request: Pass certificates in string / bytes representation instead of file path

[philipptrenz]

TL;DR: Requesting a way to pass certificate's content as in-memory data to the Client object of httpx.

I'm using the certificate based client authentication with the clients cert parameter. Also, I'm on a distributed cloud environment, where file systems don't make much sense. Therefore the certificates are only available in memory.

As far as I could see, there is no way to pass the certificates content as str, bytes or io.BytesIO object.
Having to write the files to the file system first and then read them via the file path involves a high overhead due to the IO operations ... Passing the data directly does make a lot of sense and is well supported in other libraries, like cryptography.

Thanks!

[lovelydinosaur]

I'm on a distributed cloud environment, where file systems don't make much sense. Therefore the certificates are only available in memory.

Hiya @philipptrenz - that's an interesting case you've highlighted there!

What would your suggestion be on how we'd change the API to accommodate this? Currently the cert parameter accepts a str and treats that as the filename to the certificate. I wonder how we could best adapt things to also support this use-case without adding confusion around that? (Incidentally I think the interface for SSL configuration generally has room for improvement, but that'd a bit of a different story.)

I'm also interested to know if this issue has been raised in either the urllib3, requests, or aiohttp projects - since there could be some useful considerations that their teams have made here.

[philipptrenz]

Hi @tomchristie, thanks for the quick supply.

I just traced the cert parameter down to the SSLContext.load_cert_chain method of the ssl module: https://docs.python.org/3/library/ssl.html#ssl.SSLContext.load_cert_chain
That method also only accepts file paths as str or bytes, so I think that's what the current interface design comes from.

By digging deeper, I figured out that the ssl package underlying OpenSSL c library provides methods to pass certificate data from memory, it's just not implemented within ssl. This feature has been heavily discussed, though: https://bugs.python.org/issue16487

BTW: The pyOpenSSL package does provide the functionality to load certificates from memory buffer

[philipptrenz]

For anyone stumbling over this thread, my current workaround looks like this:

from tempfile import NamedTemporaryFile

with NamedTemporaryFile(mode='w+b') as cert_file, NamedTemporaryFile(mode='w+b') as key_file:
  cert_file.write(cert_str.encode('utf-8')); cert_file.seek(0)
  key_file.write(priv_key_str.encode('utf-8'));  key_file.seek(0)

  cert_tuple = (f"{ cert_file.name }", f"{ key_file.name }", priv_key_secret)
  client = httpx.Client(cert=cert_tuple, ... )

[lovelydinosaur]

Great - thanks for the update.

[philipptrenz]

No worries!

I overlooked to answer your question: The interface I prefer would be a class type (similar to httpx.Limits) that accepts both file paths and also file-like objects.

[lovelydinosaur]

See also #924 (comment)

[whiteowl3]

for the wandering googler:

from urllib3.contrib.pyopenssl import inject_into_urllib3
from urllib3 import PoolManager
from OpenSSL import SSL

#cert, key = generate with pyopenssl, or convert from cryptography, generate with certauth or other tools
#cert is a SSL.X509, key is a SSL.PKey

ctx = SSL.Context()
ctx.use_cert(cert)
ctx.use_privatekey(key)

inject_into_urllib3()

with PoolManager(ssl_context=ctx) as pool:
	pool.request(...)

currently unaware of an async solution.

[lovelydinosaur]

Thanks @whiteowl3 - good to know that this is actually possible. (With pyopenssl and urllib3, even if not with httpx yet.)

Perhaps it's worth escalating this discussion into an issue? I'd be happy with someone doing that if wanted.

[Wyko]

I would find some value in this as well. I am getting a certificate passed as an environment variable from a pipeline, and it's impractical to create a file for httpx to read from every time it needs the cert.

[lovelydinosaur]

Thanks for the bump. See #2114 for the status on this.

[mrakh]

If you're happy with a Linux-specific solution to this, you can use memfd to accomplish this cleanly, with no additional dependencies, no need to write to any filesystem, and no need to read from any filesystem except procfs (which will always be mounted and available in any practical deployment):

import os
import ssl

def create_context_from_data(cert_data: bytes, key_data: bytes, cert_auth_data: bytes) -> ssl.SSLContext:
	def _memfd_with_contents(contents: bytes) -> int:
		fd = os.memfd_create('')
		os.pwrite(fd, contents, 0)
		return fd
	ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
	ctx.minimum_version = ssl.TLSVersion.TLSv1_2
	ctx.maximum_version = ssl.TLSVersion.MAXIMUM_SUPPORTED
	ctx.load_verify_locations(cadata=cert_auth_data)
	tmp_cert_fd: int | None = None
	tmp_key_fd: int | None = None
	try:
		tmp_cert_fd = _memfd_with_contents(cert_data)
		tmp_key_fd = _memfd_with_contents(key_data)
		ctx.load_cert_chain(
			certfile=f"/proc/self/fd/{tmp_cert_fd}",
			keyfile=f"/proc/self/fd/{tmp_key_fd}",
		)
	finally:
		if tmp_key_file is not None:
			os.close(tmp_key_fd)
		if tmp_cert_file is not None:
			os.close(tmp_cert_fd)
	return ctx