Absolute URLs identified as relative

[nuno-andre]

Hi!

I'm trying to implement a file:// transport and I've just found an unexpected behavior in the URL parsing.

import httpx

url = httpx.URL('file:///etc/fstab')
print(url.is_relative_url)

#> True

This is indeed not the case and causes BaseClient._merge_url to remove the URL's scheme.

Related to this, there's also an improper error handling for HTTP URLs without host:

import httpx

client = httpx.Client()
client.get('https:///index.html')

# raises httpx.UnsupportedProtocol: Request URL missing either an 'http://' or 'https://' protocol.
# instead of httpx.LocalProtocolError: Missing hostname in URL.

I agree with the rationale in URL.is_absolute_url but a URL with a scheme or a host should not be considered as relative

httpx/httpx/_models.py

Lines 390 to 400 in e1abaf1

	@property
	def is_absolute_url(self) -> bool:
	"""
	Return `True` for absolute URLs such as 'http://example.com/path',
	and `False` for relative URLs such as '/path'.
	"""
	# We don't use `.is_absolute` from `rfc3986` because it treats
	# URLs with a fragment portion as not absolute.
	# What we actually care about is if the URL provides
	# a scheme and hostname to which connections should be made.
	return bool(self._uri_reference.scheme and self._uri_reference.host)

I think this could be fixed with very little change and without collateral damage:

# _models.py

class URL:
    ...

    @property
    def is_relative_url(self):
        return not (self._uri_reference.scheme or self._uri_reference.host)

    @property
    def is_absolute_url(self):
        return not self.is_relative_url

import httpx

url = httpx.URL('file:///etc/fstab')
print(url.is_relative_url)

#> False

client = httpx.Client()
client.get('https:///index.html')

# raises httpx.LocalProtocolError: Missing hostname in URL.

If you consider it appropriate I'd be happy to open an issue and make a PR.

Cheers!

[nuno-andre]

It may also be a good idea to expose the mounting mechanism to allow easy addition or modification of transports to instances.

# _client.py

class Client(BaseClient):
    ...

    def mount(self, pattern: str, transport: BaseTransport) -> None:
        self._mounts.update({URLPattern(pattern): transport})

import httpx

client = httpx.Client()
client.mount('http://*.onion', httpx.BaseTransport())

print(client._mounts)
#> {<httpx._utils.URLPattern object at ...>: <httpx.BaseTransport object at ...>}

And maybe a __repr__ method in URLPattern to show something like URLPattern('http://*.onion')

[stijn-devriendt]

I'm running into the same problem (with the same use-case).

+1 for making an issue for this.

Update: I gave your patch a shot, but the test suite fails in unexpected ways and it's turtles all the way down. Ultimately, I ended up finding trouble in httpcore/trio.

The better fix (that doesn't break all sorts of expectations from the rest of httpx) is:

    @property
    def is_absolute_url(self) -> bool:
        """
        Return `True` for absolute URLs such as 'http://example.com/path',
        and `False` for relative URLs such as '/path'.
        """
        # We work around `.is_absolute` from `rfc3986` because it treats
        # URLs with a fragment portion as not absolute.
        # What we actually care about is if the URL provides
        # a scheme and hostname to which connections should be made.
        return self._uri_reference.copy_with(fragment=None).is_absolute()

    @property
    def is_relative_url(self) -> bool:
        """
        Return `False` for absolute URLs such as 'http://example.com/path',
        and `True` for relative URLs such as '/path'.
        """
        return not self.is_absolute_url

I've got a patch ready at https://github.com/stijn-devriendt/httpx/tree/fix-incorrect-relative-url that keeps things running.

[stijn-devriendt]

Is anything happening with this?

[devenjarvis]

Unfortunately I am hitting the same issue above for the same usecase and would prefer not to patch these functions within httpx. Wanted to bump this discussion and see if there were any plans to fix this behavior in a future release, or other workarounds that could be used in the meantime?