René's URL Explorer Experiment

Title: Don't choke on (legitimately) invalidly encoded Unicode paths by nvie · Pull Request #467 · gitpython-developers/GitPython · GitHub

Open Graph Title: Don't choke on (legitimately) invalidly encoded Unicode paths by nvie · Pull Request #467 · gitpython-developers/GitPython

X Title: Don't choke on (legitimately) invalidly encoded Unicode paths by nvie · Pull Request #467 · gitpython-developers/GitPython

Description: We've come across path names that contain bytes that are invalid in UTF-8 encoded strings, even though they're very rare. My assumption here is these commits have been created by an old (buggy?) version of Git, and now live in the tree objects with this data. Since we return only unicode strings for the a_path and b_path properties, we're not able to decode this string and thus choke when asking for the diff. This PR fixes that by using "replace" semantics when decoding. This will effectively replace the illegal bytes \200 (or \x80) by \ufffd (= �). Follow-up discussion However, this also means that if you would want to git-blame this file, there's no good way of referencing this path, since it's inherently a bytes path. Normally, when we pass unicode paths to git-blame via GitPython's blame API, the paths get converted to UTF-8 right before issuing the external command. But there's no way of getting the original bytes back after the "replace" operation happened. Example: The input path is b'illegal-\x80.txt' (containing illegal byte \x80) When decoded to UTF-8 with characters replaced, we get the unicode string u'illegal-\ufffd.txt' (= "illegal-�.txt") When encoding that in UTF-8, we find b'illegal-\xef\xbf\xbd.txt' When we next pass illegal-\xef\xbf\xbd.txt to git-blame, it will not be able to find this path. Perhaps it would be a good idea to not only return the decoded path strings, but also provide access to the raw bytes found, i.e. by exposing a_rawpath and b_rawpath, which would always be bytes? That way, you could still have the friendly "unicode paths" for most use cases, but use bytes if you need to speak the language of Git more accurately.

Open Graph Description: We've come across path names that contain bytes that are invalid in UTF-8 encoded strings, even though they're very rare. My assumption here is these commits have been created by an old (bu...

X Description: We've come across path names that contain bytes that are invalid in UTF-8 encoded strings, even though they're very rare. My assumption here is these commits have been created by an...

Opengraph URL: https://github.com/gitpython-developers/GitPython/pull/467

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width