uri resolution notes

[karenetheridge]

Not intended for merging in this form.

Notes on my initial thoughts on how each uri-reference in the OAD should resolve.

See #4925 (reply in thread).

[baywet]

Sharing some notes from the meeting: This is a working document from @karenetheridge with her understanding of the spec today. If some of that is "wrong" it probably means that we need to make the specification clearer. @handrews to provide some feedback to help get to a common understanding and identify the potential for improvements in the specification.

[handrews]

This is one of several spots where I would not use "retrieval URI." I tried to make this more clear in 3.2, so it would be good to understand what is not coming across. I would probably use a phrase like "against the document's base URI," although I don't think that that phrase is used prominently in 3.2. Maybe that is part of the lack of clarity?

Here's the wording for $self:

This string MUST be in the form of a URI reference as defined by [RFC3986] Section 4.1. The $self field provides the self-assigned URI of this document, which also serves as its base URI in accordance with [RFC3986] Section 5.1.1. Implementations MUST support identifying the targets of API description URIs using the URI defined by this field when it is present. See Establishing the Base URI for the base URI behavior when $self is absent or relative, and see Appendix F for examples of using $self to resolve references.

Perhaps we could change "See Establishing the Base URI for the base URI behavior when $self is absent or relative" to "When $self is relative, it is resolved against the document's base URI"?

In section 4.1.2.2.1 Establishing the Base URI, the third paragraph starts:

The most common base URI source that is used in the event of a missing or relative $self (in the OpenAPI Object) and (for Schema Object) $id is the retrieval URI.

My intention with that phrasing was both to note the common behavior and to emphasize that the base URI is not always the retrieval URI. Does it seem clear enough if the $self wording is changed, or is it still unclear?

[karenetheridge]

the base URI is not always the retrieval URI

I'm not clear on the difference. Isn't the retrieval URI already defined as "the initial URI we assign to the document when we start reading from it"? I recall we used the retrieval URI phrasing even if the document is never actually reachable as an URL on the network, but just a thing in application state.

It sounds like we need to come up with a clearer name for whatever this thing is, and use it throughout.

[handrews]

@karenetheridge the clear name is "base URI." The less-clear part is that there are several possible base URIs that are searched in order, and if any of them are relative, then they are resolved against the next possible base URI. Since $self is the first place to search, it both is a base URI, and potentially needs to be resolved against the next base URI in the list.

But all of this is about base URIs. It is not about retrieval URLs/URIs, except by coincidence. They happen to be one of four possible base URI sources (and one of three after $self). But it's always about base URIs, and we should always talk in terms of base URIs.

This is for OAD URIs, though, not API URLs, where the situation is different.

[karenetheridge]

Well yes, the key is "which base URI do we use", and that's the thing that needs to be stated clearly. :p

[handrews]

@karenetheridge then I do not understand your question about a clearer name. "Base URI" is the name. RFC3986 gives names for each of the four possible sources. "Retrieval URI" is incorrect except in specific circumstances, being one of the four sources. So what sort of clearer name are you looking for? It's a multi-step look-up, so there's no one name other than "Base URI".

[handrews]

Is this "HTTP request uri" the request URI of the OAD document? This is a place where our URI/URL distinction can get a bit muddled.

In this case, I would call this a URL matters because the document's identity (URI) might be set (by $self) to something other than the document's location (which is generally it's retrieval URL).

The tricky part here is that if we are simulating the document's retrieval URL (e.g. because you are testing things and don't want to actually deploy to production, but do want to test as if it were deployed), then it's arguably being a bit more URI-ish than URL-ish? But I still lean towards calling that a URL because the whole point is that you are simulating the location, and want it to behave as a location. You are not simulating it to separate identity from location (which is what $self does). You are simulating it to get location-based (not identity-based) behavior.

This simulation use case is discussed in the third paragraph of 4.1.2.2.1 Establishing the Base URI, although that's not linked from the Server Object so maybe that's a concern.

Does all of this make sense, and if so do you have any ideas on how we can clarify it? I moved the "Relative References in API URLs" section under the Server Object, but might have accidentally dropped explanatory text elsehwere in the process.

[karenetheridge]

Is this "HTTP request uri" the request URI of the OAD document?

No, whenever I refer to an HTTP request it is in the server context -- it is a request for one of the APIs described by the document (or perhaps not matching any API, but there is still an attempt to match one of the path-items to it).

When I try to match server urls to the HTTP request, I resolve the url against the retrieval uri (not $self), and then again against the HTTP request's URI, if the retrieval URI was relative. (We have to have a URI with host and scheme in order to match it against the HTTP request URI, and having a relative retrieval URI would prevent that.

I believe this is correct?

[handrews]

@karenetheridge I think I am confused. How can you resolve a relative server URL against an API request, when you need the server URL to make an API request? That sounds like server-side matching which is, to me, a different concern from URL construction, although they are definitely related.

I'm glad you are calling attention to server matching. It had never occurred to me to think through that process, so please bear with me as a try to wrap my head around it.

By URI/URL construction, I mean "I'm parsing this OAD and I found some part of an OAD URI or API URL and need to construct the full URI/URL before I can use it."

By server-side matching, I mean "I am handling an API request and I am trying to figure out which possible URL construction (as in the previous definition) matches this URL, after which I need to split it back into parts to extract server variables, path template parameters, and handle the query string."

Most of your comments here seem to be about construction, which is why I am confused. I think it is important to get the construction parts clear and separate from the matching guidance. Because if we aren't constructing correctly, we'll end up attempting to match to an incorrect set of potential URLs.

[karenetheridge]

How can you resolve a relative server URL against an API request, when you need the server URL to make an API request?

Those are on different systems though, so we get at the information in a different way.

Most of your comments here seem to be about construction, which is why I am confused.

I'm confused too, as I'm not sure what has given you that impression.

[handrews]

How can you resolve a relative server URL against an API request, when you need the server URL to make an API request?

Those are on different systems though, so we get at the information in a different way.

I don't follow, can you elaborate on this?

Regarding construction, perhaps that is my assumption. What are you trying to describe here? Construction, matching, or both? To me, successful matching requires successful construction first, otherwise you don't know what you are matching. Does that make sense? If not, how do you think of it?

Again, I have not thought about this much from the matching side, so I might be getting things really wrong. I'm trying to understand the perspective so I can understand what your goals are here.

[karenetheridge]

Constructing the request for the client, and parsing the request on the server, are two totally separate things done by separate systems and generally on different machines. What I've been talking about regarding resolving the server url against "something" and then matching it plus the path template against the request is all a server-side activity.

The client could construct the request in a number of ways. I have imagined that a client would construct the request based on some static configurations that it has been provided.. basically: "this is the base URL of the server you are talking to, so given the API operation you want to use, find the appropriate server url that matches that (with base URI resolution if needed), append the appropriate path template for this operation to that, and replace the template variables with the values appropriate to this particular request."

But perhaps it constructs the request through its own means and then evaluates the request against the OAD before sending it just to make sure that it's valid, in which case the behaviour would be just like receiving the request on the server side and using the OAD to parse and validate it before further processing.

(Likewise, a server could either construct a response by its own means and then validate it against the OAD, or use the OAD itself as a guide for constructing it, before sending the response back to the client.)

So really, there are just two mechanisms at play, each of which could be running on either a client or server system:

• construct a request URI using information from the path template and server url, providing server and path variables from an outside source, e.g. config files and the application
• parse a request URI against an operation's path template and server url (which might require iterating through a number of path-items if we don't yet know which one matches) to extract its server variables and path template variables (path parameter values).

I've implemented code for the second of these, but not the first.

[handrews]

FWIW, I had to think about whether Info/License/Contact should be OAD URI-based or API URL-based. Taking a look at these fields, we do say "URI", and that's what I was expecting to find. I wonder if we should make "URI" a link to 4.1.2.2 Relative References in API Description URIs for added clarity?

[handrews]

There's an interesting wrinkle here that this is a URL (including possibly a relative URL-reference), but the resolved server URL is expected to be used as a prefix (with resolved Path Templates) rather than a normal base URL. I'm not sure how much this matters as both behaviors are well-defined, but it means that, given a server URL of https://example.com/api, a Path Template of /foo, and a security URL of /foo, the resulting API endpoint is https://example.com/api/foo but the resulting security URL is https://example.com/foo (no /api).

Which leads to another observation: The Paths Object requires Path Templates to start with /, but the Server Object does not forbid URLs from ending with /. I'm not quite sure what to do with that 😵‍💫

[karenetheridge]

... but the resulting security URL is https://example.com/foo (no /api)

Yes, that was my observation as well. I assume that this is the best we can do, and it doesn't really make sense for the api portion to appear in the resolved form of the server url we use for matching.

It's even more clear when we change the retrieval URL to ".../api.json". That's clearly a filename, not a directory, and we don't expect to see that in any request URLs that we're trying to match.

The Paths Object requires Path Templates to start with /, but the Server Object does not forbid URLs from ending with /. I'm not quite sure what to do with that.

Yes. I played around with trying to resolve the path template against the server url, instead of appending, but the results were less favourable.

We could say something like "the templated URI resulting from the concatenation of the server url and path template portions should then be normalized", which would also allow for constructs like ../ to be removed, but I feel like there would be a lot of edge cases and surprises here in practice.

The alternative is: you end up with a uri template that contains //, which is likely not ever going to match. So don't do that.

[handrews]

it doesn't really make sense for the api portion to appear in the resolved form of the server url we use for matching.

In my example, the resolved API endpoint does include /api so I am confused as to this statement. Only the auth endpoint does not include it. This might be more of me not quite wrapping my head around the server matching use cases.

We could say something like "the templated URI resulting from the concatenation of the server url and path template portions should then be normalized", which would also allow for constructs like ../ to be removed, but I feel like there would be a lot of edge cases and surprises here in practice.

My sense is that URL normalization is something that does or doesn't happen independently. I would tend to normalize URLs as much as possible before attempting to compare, athough as you note that could get surprising. (For those unfamiliar, normalization and comparison is a tricky topic addressed in-depth by RFC3986 §6.)

The alternative is: you end up with a uri template that contains //, which is likely not ever going to match. So don't do that.

I want to say that // anywhere other than as the authority prefixe (as in https://example.com) gets normalized to /, but I can't find a reference for that so it might just be a thing that web servers tend to do? I'm really not sure.