René's URL Explorer Experiment

Title: gh-66428 Stop including all bidirectional "B" characters in line breakers by LoicGrobol · Pull Request #132369 · python/cpython · GitHub

Open Graph Title: gh-66428 Stop including all bidirectional "B" characters in line breakers by LoicGrobol · Pull Request #132369 · python/cpython

X Title: gh-66428 Stop including all bidirectional "B" characters in line breakers by LoicGrobol · Pull Request #132369 · python/cpython

Description: This is not a direct fix to the issue (as it is not clear how/if it should be fixed at this point), but to this comment: I couldn't find a trace of bidirectional B characters being non-tailorable line breaks (classes BK, CR, LF, NL), neither in the line breaking not in the bidirectional algorithm specification. What I think the latter means is that B characters actually only “terminate paragraphs” in the sense of “terminating the effect of directional formatting characters”. Now it is true that \N{PARAGRAPH SEPARATOR} is both B and BK, but does it means that all B characters should be line breaks? In particular what about \N{FILE SEPARATOR}, \N{GROUP SEPARATOR} and \N{RECORD SEPARATOR}, who are all caught by str.splitlines, but have the Combining Mark property (see e.g. the list of line breaks), which should in fact forbid a line break? And the answer by @malemburg At the time this code was written (in 2000), Unicode 3.0 was the current version and in those days, bidirectional B code points were listed as "Paragraph Separator", in particular the important ones CR, LF. See https://www.unicode.org/Public/3.0-Update/UnicodeData-3.0.0.html and https://www.unicode.org/Public/3.0-Update/UnicodeData-3.0.0.txt for details. I don't remember, but it's well possible that I wasn't aware of the CM LineBreak property at the time. I guess the _PyUnicode_IsLinebreak() function should really use the LineBreak.txt file as basis and only include code points which are listed as "Non-tailorable" in that file: https://www.unicode.org/Public/UCD/latest/ucd/LineBreak.txt Perhaps it's time to fix this after 25 years 😄 Patches are welcome ! It turns out that LineBreak.txt is already used to gather the line break categories, but that B was still kept as indicating a line breaker. Looking at the current state of Unicode (16.0) All the characters in B that are non-tailorable line breakers are in the BK, CR, LF or NL categories, so they were already captured. The other characters in B (U+001C, U+001D, U+001E) are all combining marks and shouldn't break lines. So I just removed the bidirectional == "B" condition in the code generating the list of line breakers and that should be it. I think from an API perspective, this only affects str.splitlines(), which at this point is only tested for behaviour against CR, LF and CR+LF and no other line breaker, so I didn't add any test, but I can if it seems useful. In general, I don't expect this to be a huge compatility-breaking change given the conversation in #66428, but don't really know how to check for that apart from searching for the codepoints (in U+ and \x forms) on Github, which didn't return any Python code that would be broken. This is my first PR, so it's very likely I missed something, please let me know! Issue: gh-66428 📚 Documentation preview 📚: https://cpython-previews--132369.org.readthedocs.build/

Open Graph Description: This is not a direct fix to the issue (as it is not clear how/if it should be fixed at this point), but to this comment: I couldn't find a trace of bidirectional B characters being non-tailora...

X Description: This is not a direct fix to the issue (as it is not clear how/if it should be fixed at this point), but to this comment: I couldn't find a trace of bidirectional B characters being non-tai...

Opengraph URL: https://github.com/python/cpython/pull/132369

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width