mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 11:17:13 +00:00
idnorm: a prefix is ASCII and the text it is cut from is not
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of `https://doi.org/`, so testing that prefix against it sliced into the middle of the third zero-width space and brought the process down. Found by running citefind over the corpus: the DOI is printed that way in a real book, because the typesetter marked where the line could break. Both prefix tests — DOI and arXiv — now ask for the head rather than cutting one, so a byte length that is not a character boundary is an answer of no rather than a panic. And an invisible character is not part of an identifier. Zero-width spaces, joiners, the soft hyphen, a byte-order mark left at the front of a fragment: text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI carrying one does not resolve. They are dropped before anything is read.
This commit is contained in:
@@ -37,7 +37,15 @@ pub fn parse(text: &str) -> Option<String> {
|
||||
let text = text.trim();
|
||||
let mut without = text;
|
||||
for prefix in PREFIXES {
|
||||
if without.len() >= prefix.len() && without[..prefix.len()].eq_ignore_ascii_case(prefix) {
|
||||
// `get` rather than a slice: the prefixes are ASCII and the text is
|
||||
// not, so cutting at the prefix's byte length can land inside a
|
||||
// character. A DOI copied out of a PDF arrives full of zero-width
|
||||
// spaces, and `10.1093/\u{200b}oso/` is exactly long enough to put
|
||||
// one where `https://doi.org/` ends.
|
||||
if without
|
||||
.get(..prefix.len())
|
||||
.is_some_and(|head| head.eq_ignore_ascii_case(prefix))
|
||||
{
|
||||
without = &without[prefix.len()..];
|
||||
break;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user