idnorm: a prefix is ASCII and the text it is cut from is not

`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: a real book prints its DOI that way, because the
typesetter marked where the line could break.

Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.

And an invisible character is not part of an identifier. Zero-width spaces
and joiners, the soft hyphen, a byte-order mark left at the front of a
fragment: text out of a PDF or an EPUB is full of them, none of them can be
seen, and a DOI carrying one does not resolve. They are dropped before
anything is read.
This commit is contained in:
funkpower
2026-09-20 14:11:13 -04:00
parent 1bfca45b67
commit d31206dd0f
-2
View File
@@ -180,8 +180,6 @@ fn isbn(input: &str) -> Option<Identifier> {
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_identifier_copied_out_of_a_pdf_is_full_of_nothing() {
// A typesetter's zero-width spaces, where the line was allowed to