urljoin works incorrectly for two path-relative URLs involving . and ..

[andersk]

Bug report

urllib.parse.urljoin is usually used to join a normalized absolute URL with a relative URL, and it generally works for that purpose. But if it’s used to join two path-relative URLs, it produces incorrect results in many cases when . or .. is involved.

>>> from urllib.parse import urljoin
>>> urljoin('a', 'b')  # ok
'b'
>>> urljoin('a/', 'b')  # ok
'a/b'
>>> urljoin('a', '.')  # expected . or ./
'/'
>>> urljoin('a', '..')  # expected .. or ../
'/'
>>> urljoin('..', 'b')  # expected ../b
'b'
>>> urljoin('../a', 'b')  # expected ../b
'b'
>>> urljoin('a', '../b')  # expected ../b
'b'
>>> urljoin('../a', '../b')  # expected ../../b
'b'
>>> urljoin('a/..', 'b')  # expected b
'a/b'

There are also some problems when the base is a non-normalized absolute URL:

>>> urljoin('http://host/a/..', 'b')  # expected http://host/b
'http://host/a/b'

Your environment

• CPython versions tested on: 3.10.5, 3.11.0rc1
• Operating system and architecture: NixOS 21.11 amd64

[6t8k]

According to the docs on urllib.urlparse.urljoin and the docstring in its code, the function is meant to construct an absolute URL, so I think that's what should be expected from its return value.

At least as far as RFC 3986 is concerned, which is mentioned in the linked docs, the "Base URI" is also expected to be absolute. Maybe the docs could be a bit clearer.

[andersk]

Even that excuse doesn’t apply to urljoin('http://host/a/..', 'b').

[6t8k]

That indeed appears to be a more subtle case. Some clues, without a conclusion from me:

• RFC 3986 sections 3.3 and 6.2.2.3 say dot-segments are intended for use at the beginning of a relative-path reference
• 5.2.1 says normalization of the base URI, as described in 6.2.2 and 6.2.3, is optional (for historical/compatibility reasons, I'd assume)
• In urllib.urlparse.urljoin, the last segment of the base URI, here a dot-segment, is removed here (this is not using the remove_dot_segments algorithm from 5.2.4 – effectively the segment is ignored)
• The URL API, in major web browsers, does produce http://host/b (but for relative bases, it throws errors). It seems that they opt to not skip normailzation of the base URI; the new URL(url, base) constructor's, and following this, basic URL parser's path state specs seem to indicate this:

Firefox 107.0

let baseUrl1 = "http://host/a/..";
undefined
let A = new URL("b", baseUrl1);
undefined
A.href
"http://host/b"
let baseUrl2 = "../a";
undefined
let B = new URL("b", baseUrl2);
Uncaught TypeError: URL constructor: ../a is not a valid URL.
    <anonymous> debugger eval code:1
debugger eval code:1:9 

Chromium 107.0.5304.121

let baseUrl1 = "http://host/a/..";
undefined
let A = new URL("b", baseUrl1);
undefined
A.href
'http://host/b'
let baseUrl2 = "../a";
undefined
let B = new URL("b", baseUrl2);
VM520:1 Uncaught TypeError: Failed to construct 'URL': Invalid base URL
    at <anonymous>:1:15
(anonymous) @ VM520:1

---

The question then is, I suppose, whether this behavior is unexpected/hard to work around enough that fixing it would be worthwhile to the point of making the risk of breaking existing code acceptable. So far I'm not aware of any standards-breaking behavior, at least.

[andersk]

The expected semantics are obvious. These URLs are equivalent:

"http://host/a/.." ~ "http://host/"

so we expect these URLs to be equvalent:

urljoin("http://host/a/..", "b") ~ urljoin("http://host/", "b").

In general, we always expect:

• if a ~ a1 and b ~ b1, then urljoin(a, b) ~ urljoin(a1, b1)

(where a ~ a1 is the equivalence relation normalize(a) == normalize(a1)).

Any time equivalent inputs lead to nonequivalent outputs is an opportunity for bugs. If the standard makes it optional to produce an expected result for legacy reasons, that doesn’t make it a good idea to produce an unexpected result.

---

Another law we always expect is

• urljoin(a, urljoin(b, c)) ~ urljoin(urljoin(a, b), c)

which makes it obvious how urljoin(b, c) is expected to behave for relative b (if it’s not going to throw an error). For example, we should have

urljoin("http://host/w/x/y/z", urljoin("../b", "../c"))
~ urljoin(urljoin("http://host/w/x/y/z", "../b"), "../c")
== urljoin("http://host/w/x/b", "../c")
== "http://host/w/c"
== urljoin("http://host/w/x/y/z", "../../c")

which can only work if urljoin("../b", "../c") ~ "../../c".