idnorm: a prefix is ASCII and the text it is cut from is not

`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: the DOI is printed that way in a real book, because
the typesetter marked where the line could break.

Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.

And an invisible character is not part of an identifier. Zero-width spaces,
joiners, the soft hyphen, a byte-order mark left at the front of a fragment:
text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI
carrying one does not resolve. They are dropped before anything is read.
This commit is contained in:
funkpower
2026-09-20 14:10:18 -04:00
parent 249edfe294
commit 1bfca45b67
3 changed files with 70 additions and 4 deletions
+9 -1
View File
@@ -37,7 +37,15 @@ pub fn parse(text: &str) -> Option<String> {
let text = text.trim();
let mut without = text;
for prefix in PREFIXES {
if without.len() >= prefix.len() && without[..prefix.len()].eq_ignore_ascii_case(prefix) {
// `get` rather than a slice: the prefixes are ASCII and the text is
// not, so cutting at the prefix's byte length can land inside a
// character. A DOI copied out of a PDF arrives full of zero-width
// spaces, and `10.1093/\u{200b}oso/` is exactly long enough to put
// one where `https://doi.org/` ends.
if without
.get(..prefix.len())
.is_some_and(|head| head.eq_ignore_ascii_case(prefix))
{
without = &without[prefix.len()..];
break;
}