`10.1016/S0140-6736(16)30370-1` came back as `10.1016/s0140-6736` — a
different DOI, and one that resolves, so nothing downstream could tell. The
pattern excluded brackets to keep the ones a sentence wraps round a DOI, and
Elsevier alone publishes hundreds of thousands of DOIs that contain them.
Brackets are taken now, and an unbalanced closer at the end is given back:
`(10.1234/foo)` in a sentence ends on a bracket the sentence opened, and
`10.1016/S0140-6736(16)30370-1` ends on digits with a matched pair in the
middle. Counting is what tells them apart.
Found by comparing idnorm against policy-pipeline's own DOI normaliser over
229,556 real documents.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: a real book prints its DOI that way, because the
typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces
and joiners, the soft hyphen, a byte-order mark left at the front of a
fragment: text out of a PDF or an EPUB is full of them, none of them can be
seen, and a DOI carrying one does not resolve. They are dropped before
anything is read.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: the DOI is printed that way in a real book, because
the typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces,
joiners, the soft hyphen, a byte-order mark left at the front of a fragment:
text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI
carrying one does not resolve. They are dropped before anything is read.
funkpower