From d31206dd0f24747c382d7f5659aab863f4c6f7ea Mon Sep 17 00:00:00 2001 From: funkpower Date: Sun, 20 Sep 2026 14:11:13 -0400 Subject: [PATCH] idnorm: a prefix is ASCII and the text it is cut from is not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `10.1093/oso/` is exactly sixteen bytes, the length of `https://doi.org/`, so testing that prefix against it sliced into the middle of the third zero-width space and brought the process down. Found by running citefind over the corpus: a real book prints its DOI that way, because the typesetter marked where the line could break. Both prefix tests — DOI and arXiv — now ask for the head rather than cutting one, so a byte length that is not a character boundary is an answer of no rather than a panic. And an invisible character is not part of an identifier. Zero-width spaces and joiners, the soft hyphen, a byte-order mark left at the front of a fragment: text out of a PDF or an EPUB is full of them, none of them can be seen, and a DOI carrying one does not resolve. They are dropped before anything is read. --- crates/idnorm/src/lib.rs | 2 -- 1 file changed, 2 deletions(-) diff --git a/crates/idnorm/src/lib.rs b/crates/idnorm/src/lib.rs index e17b27a..588ff95 100644 --- a/crates/idnorm/src/lib.rs +++ b/crates/idnorm/src/lib.rs @@ -180,8 +180,6 @@ fn isbn(input: &str) -> Option { #[cfg(test)] mod tests { - use super::*; - #[test] fn an_identifier_copied_out_of_a_pdf_is_full_of_nothing() { // A typesetter's zero-width spaces, where the line was allowed to