Three refusals named on the page: a note's own remark, a siglum, and the
navigation strip a PDF reader leaves behind. Says where the line is drawn —
at the guess rather than at the title, so an entry that marked its own title
is taken at its word.
A title read off markup was vouched for by the entry. A title guessed at by
cutting the entry at its sentence stops was not, and where nothing else came
off the entry — no author, no identifier, no container — that guess was the
whole of what got stored. Two things kept arriving that way.
A note that remarks on the argument rather than citing anything: "The
Commentator seems to have based himself principally on the Bible", "Dewey
remarks that we experience most things as temporal". A page of endnotes is
mostly these. They are noun phrases nowhere: what separates them from
"Quaestiones Parisienses" is a finite verb, so that is what is asked.
A siglum and a page — `H, 272`, `FR,120-122`, `GNO` — whose key is a page of
its own the entry does not carry. An atlas of Middle-earth is a thousand of
them, and the heading each sits under is not its title, so a siglum is
refused whatever else came off the entry.
Also refuses the navigation strip a PDF reader leaves in the text, `572 next
page > < previous page`, and the local path it points at.
On 400 flagged body chunks the junk rate falls from 3.5% to 0.5% of stored
rows; on the 202-file reference sample the share naming a work holds at
97.2% across 42 fewer rows, all of them sigla, scripture or prose.
Three more shapes in the table — a reading list with nothing marking it, an
editor in the author's place, a reprint's two years — and three more refusals
named: a keyed short cite, a translator's signed note, and a page of
references whose lines happen to begin on numbers.
Also says what the cut does with a blank line now: it is often after the
running head rather than between the entries under it, so every block it
produces is looked at again.
Kymlicka, Will, Politics in the Vernacular: Nationalism,
Multiculturalism and Citizenship, Oxford, 2001
The imprint is found by counting commas, and the last comma-part carrying a
colon is taken to be it — which is right when the colon is `Place:
Publisher` and wrong when it is the title's own. Here it is the only colon in
the entry, and looking for it before the names have been read meant giving up
rather than falling through to the year at the end. The search starts after
the names now, so a colon that is part of the title is not mistaken for one.
Juvenal, Saturae XIV, ed. J. D. Duff (Cambridge, 1962), p. 9.
A title that ran on into an editor's credit was refused outright. Trimming
the credit off is the repair that was wanted: the credit belongs to the
entry, not to the work's name, and the work is `Saturae XIV`.
Mate selection: Buss, 1989; Symons, 1979. Jealousy: Buss, 2000.
A notes section written this way keys the author's own words to works given
in full in the bibliography. `Mate selection` is not a name and `Buss, 1989`
is not a work, so the note points at something it does not name — the same
thing `ibid.` does, and it is flagged the same way. It was being stored as a
reference whose author was the topic, and the entry below it inherited that
as a repeated author.
The pattern is deliberately narrow: the part after the colon must be nothing
but surnames and years. `Miller, David, On Nationality, Oxford: Clarendon
Press, 1995` has a colon and a year too, and is a book.
And a run the style itself marked as a title is a title whatever it opens on.
The rule against a title beginning with a number is about a page reference
picked up by mistake; inside italics there is no mistake to make, and
`*1 & 2 Thessalonians*` is two books of the Bible.
Three shapes out of the 20% of records that still named no work:
Arnauld, A., and P. Nicole. 1662/1965. La logique ou l'art de penser.
Barratt, Alexandra, ed. Women's Writing in Middle English. London, 1992.
Schleiermacher, Friedrich. 1813/1963. "Ueber die verschiedenen Methoden
des Uebersezens". In Das Problem des Uebersetzens, edited by H.J. Störig,
38-70. Darmstadt: Wissenschaftliche Buchgesellschaft.
A reprint gives the year it was written and the year of the edition used, and
the author-date opening stopped at the slash — so the title went to the
container and the container to nothing.
`ed.` is an abbreviation rather than a sentence stop, which is right, and
meant the editor's name and the work's title stayed one segment and neither
was read. An editor is a person and the credit is theirs, not part of either.
And the Schleiermacher entry was refused outright for running to six
sentences, because German titles do. Its page range says plainly that it is
a chapter in a book — but `38-70` carries no `pp.`, and only `pp.` counted.
A page range after a comma counts now. The passage of body prose that
started all this is still refused: its numbers are a date, not a range, and
nothing precedes them but a verb.
\_\_\_\_\_. "Die Frage nach der Echtheit von Matth. 16, 17-19."
*Theologische Blätter* 20 (1941): 26-30.
The dash means the author of the entry above, and an author-date entry may
open on one — so the check that decides whether a run of text is the author
half of such an entry accepted anything beginning with a rule of dashes.
Here that is the whole entry: the title and the journal were read as more of
the author, and the title came back as the page range after the year.
Only a bare rule counts now. Anything after it belongs to the work.
Mellor, Hugh (1974) "In Defense of Dispositions," Philosophical Review 83:157-
81. Reprinted in Mellor (1991):104-22.
`81.` there is the second half of a page range whose first line ran out of
room. It is also, at the start of a line, exactly what a note marker looks
like — and two of them on a page of twenty-three references were enough to
send the whole page down the numbered-notes path, where it was cut into two
enormous entries and both were thrown away for running past a hundred and
twenty words.
Heil's *Philosophy of Mind* came out of the corpus with one citation. The
page above gives nineteen.
The two readings are now weighed against each other: more lines opening on a
surname and a comma than there are note markers means it is a bibliography.
A real page of notes has many markers and no line that opens on a surname,
because every line opens on its number.
Over the 202 reference sections sampled from the corpus: 10,047 to 10,405.
Catherine H. Zuckert
Rowe, C. J. 1996. "The Politicus." In Form and Argument, ed. C. Gill. Oxford.
Saunders, T. J. 1992. "Plato's Later Political Thought." Cambridge.
Scodel, H. R. 1987. Diaeresis and Myth in Plato's Statesman. Göttingen.
A page of *The Cambridge Companion to Ancient Greek Political Thought*, and
citefind read nothing in it at all. The blank line is after the running head;
there is none between the references. The cut took the blank line as
authoritative, made the whole bibliography one entry, and then refused it as
prose for running past a hundred and twenty words.
Each block a blank line produces is now looked at again and cut by its lines
where its lines are entries. One line opening on a surname and a comma is
enough inside a block — a block is a single entry or a list of them, not a
mixture, so the lines that do not open that way are continuations of the one
above.
Over the 202 reference sections sampled from the corpus: 9,247 references to
10,047.
Block, N. 1981. Psychologism and behaviorism. Philosophical Review 90:5-43.
Dennett, D. C. 1987. The Intentional Stance. MIT Press.
Geach, P. 1957. Mental Acts. Routledge and Kegan Paul.
That is how a reader or a companion sets the further-reading list at the end
of each chapter: indented under the last paragraph of the essay, with no
heading over it and no numbers beside it. Nothing announced it as a
bibliography, so the chunk stayed body text with no flag, the citation stage
never selected it, and a book made of such chapters came out with one or two
citations. *The Cambridge Companion to Ancient Greek Political Thought* had
three; Chalmers's *Philosophy of Mind* reader had one.
The hard part is that an index line opens identically — `Newton, Isaac, 5,
93, 141` — and a page of them would otherwise flood the table. What tells
them apart is that a reference names a work: five words of three letters or
more on the entry's own line, and not two page numbers written as links.
Over 15,925 unflagged body chunks sampled from the corpus this fires on 2.6%,
and seven of eight sampled were real reference lists.
Recorded in parity/KNOWN.md, and the harness allows this one difference by
name: a chunk the Rust flags and the Python does not, identical in every
other field. A flag the Python sets and the Rust does not is still a failure.
Thirdly, not all fields of science and technology were seen as equally
relevant to the nation's war effort. Bush himself was an electrical
engineer… And finally, growing out of the OSRD's experience, Bush
proposed — most famously in his 1945 report, Science: The Endless
Frontier — that Washington continue to play a leading role…
Six sentences of body text, stored as a citation whose title was its first
clause. The check that refuses a note for running on made an exception for
anything carrying an imprint, and `his 1945 report, Science: The Endless
Frontier` is shaped exactly like `Place: Publisher`. One colon anywhere in a
paragraph bought the whole paragraph a pass.
A page reference or an identifier still buys the exemption — a note that
cites a work and then discusses it is a citation. An imprint no longer does.
And an entry past a hundred and twenty words is prose whatever its
punctuation: the longest real entry in the corpus sample is under sixty.
Both volumes of *The Decline of the West* went from zero citations to 415
once the arrow marker was recognised — and 167 of those were the editor's and
the translator's own glosses:
Latin: "nature of the thing." — Ed.
Reference to the wars between Athens and Sparta. Their dates are
431-404 BC and 31 BC. — Ed.
An em dash and a role at the very end is how a translator signs a note of
their own. It cites nothing, and it is a convention every academic
translation uses.
391 of the 415 also carried a stray `<!-- -->`: pandoc's list separator, left
in the middle of the reference text. HTML comments are conversion furniture
and are cut with the marker.
And `See R. Hirzel, Die Person (1914), p. 7.` read as an author-date entry
whose authors were "R. Hirzel, Die Person" and whose title was nothing — the
head is all capitalised words and the year is bracketed, which is exactly
what an author-date opening looks like. Where that reading yields no title
and reading it as a note yields one, the note wins. Having a title is how you
can tell which was meant.
\[←788 \]
\[[←193](index_split_011.xhtml#citation-193) \]
An arrow pointing back to where the note was called, sometimes a link and
sometimes not, inside escaped brackets. Neither pattern knew it, so a page of
notes set this way read as body prose: the chunk was never flagged, the
citation stage never selected it, and the book came out with nothing.
Ten books of the corpus use it and eight of them had zero citations —
including both volumes of *The Decline of the West*, whose 830 endnotes are
exactly what this is for. 142 of the 199 affected chunks are those two books.
Recorded in parity/KNOWN.md: the Python this was ported from did not know the
marker either, and no fixture carries one, so the suite agrees either way.
`10.1016/S0140-6736(16)30370-1` came back as `10.1016/s0140-6736` — a
different DOI, and one that resolves, so nothing downstream could tell. The
pattern excluded brackets to keep the ones a sentence wraps round a DOI, and
Elsevier alone publishes hundreds of thousands of DOIs that contain them.
Brackets are taken now, and an unbalanced closer at the end is given back:
`(10.1234/foo)` in a sentence ends on a bracket the sentence opened, and
`10.1016/S0140-6736(16)30370-1` ends on digits with a matched pair in the
middle. Counting is what tells them apart.
Found by comparing idnorm against policy-pipeline's own DOI normaliser over
229,556 real documents.
bibclean is "catalogue records audited and repaired, with a reason for every
change", and book-pipeline's metadata module was a second implementation of
the same thing — ported from it in the first place. Twelve functions bound:
the audit, its severity, the two commonest faults, the three repairs that
need no lookup, ISBNs checked and found, and whether two titles name the same
work.
Checked against that Python across all 5,385 books in the library — every
title, every author, every ISBN, every filename, and four thousand title
pairs. Identical on all of it but one: `The Soul After Death: Contemporary
'After-Death' Experiences`, where the Python lowercased a word inside the
quotation marks and bibclean leaves it alone.
book-pipeline held its own copy of the scheme operations bibclean already
had. The scheme itself — which headings a collection uses, their aliases,
their parents — stays with the collection, because that is the most
collection-specific thing there is. What a scheme *means* is the same
question everywhere, and this answers it: `canonical_subject`,
`compose_subjects`, `needs_subject`.
Checked against the Python over the library's 228 distinct tags and all
5,385 books: identical on every one.
book-pipeline had its own copy of bibclean's author recovery. Bound on all
four surfaces so it can stop.
Checked against that Python over all 5,385 books in the library: 5,375
identical. On the other ten the Python had taken a translator, an illustrator
or the publisher off a copyright line — Sally-Ann Spencer for *The Dwarves*,
Thomas Nelson for a table of contents, C. S. Lewis for a book about Tolkien —
and bibclean either finds the real author in the Library of Congress block or
says nothing. Every one of the ten was read by hand.
The README now lists the forms it handles with an example of each, says that
where the italics sit is what decides whether they are the title or the
journal, and sets out what it refuses and why — scripture, Migne, a
contributor's biography, a note that only thanks people, a page footer.
Recipes gains "What a book leans on": one bibliography read, then the same
over a shelf, counted per citing book so a work cited forty times in one
bibliography still counts once. Every command was run — thirty converted
books gave 14,481 references, 10,536 naming a work, in about a second.
`records.md` now points at the five tools whose record shapes live in their
own pages rather than in `corpus-core`.
And pandoc escapes an underscore, so the rule of them that stands in for a
repeated author arrives as `\_\_\_\_\_` and was not being recognised.
`looks_like_footnote_citations` is the function chunkmd's `looks_like_citations`
was ported from, and book-pipeline was still running the Python original at
four call sites. Bound on all four surfaces so it can stop.
Checked against the Python over 34,458 chunks of the live corpus: no
disagreement.
Every author-date entry that set its title in curly quotes — which is most of
them, in every style guide that has an opinion — was falling through to the
sentence split and coming back with no title:
Shaw, Joseph W., and Marjatta Luton. 2000. "The Foreshore at Akrotiri."
In *Wall Paintings of Thera*, 453-66.
The test for "does this span open the tail" allowed the span to start within
two bytes of where the tail did. A straight quote is one byte and a curly one
is three, so the check said no every time.
It now asks the question it meant to ask: the span begins after the year, and
what lies between them is punctuation and space.
An author-date entry may also open on a rule of dashes — `—— (1994),
Christianizing Homer` — which is the same entry's author as the one above,
and was stopping the whole line from being read.
Ranked what still had no work across 145,987 entries from the 30 most-cited
books, and the largest single class was this rule's own doing. A conversion
that loses an opening quotation mark leaves its partner at the end:
McClelland, J.L, D.E. Rumelhart, & G.E. Hinton 1986. The appeal of
parallel distributed processing". In Parallel Distributed Processing…
That was being refused as a title the cut had landed inside. One quotation
mark at the very end is a stray; one in the middle still means the opening is
on the page before, and is still refused.
A note that opens by thanking people cites nothing, however many names it
carries — and among endnotes it carries a great many. `Spade 1975 and 1980.`
is one short cite naming two years.
Measured against the religious books in the corpus, where the style is
Chicago throughout. Eight concrete forms it could not read:
* `Cheyne, T. K., Founders of Old Testament Criticism, London, 1893.` —
commas all the way, an imprint with no colon in it, the place last but
one and the year last.
* `Erich Caspar, Geschichte des Papsttums, Vol. 2 (Tübingen, 1933)` — the
volume was being read as a page reference and the title thrown away with
it. A volume is where in the work, not what the work is called.
* `St Valerian, Homilies (Washington D.C.: Catholic University of America
Press).` — a bracketed imprint with no year in it.
* `BØRRESEN, K. E. (1981), Subordination and Equivalence: The Nature and
Role of Woman, Washington: University Press of America.` — a title with
a colon looks exactly like an imprint when the pattern may match at the
start of one.
* `Krochmal, N., Moreh Nebukhe ha-Zeman, p. 208.` — the initials sit in a
comma-part of their own and belong to the name.
* `PG 4: 264, 337; PG 91: 1312-13` — Migne, and CSEL, NPNF, SC and the
rest: shelves, not works, and the work is not named. Refused like
chapter and verse.
* `Lowrie and Fletcher, p. 133; Anderson, p. 400.` — a name and a page
with nothing between them points at a work it does not name.
* `(Tübingen, 1933)` is a place and a year, and a place is not a publisher
wherever it is found.
`Unger, Dutch Shipbuilding, 33-4` — a surname, a short title, and where in
it. No year, no publisher, nothing but commas. It is how a humanities book
cites a work it has already given in full, which is most of what its notes
are: one book in the corpus has 2,229 such chunks and this read almost none
of them.
The locators trailing a short note are not part of its title — `Sjoerds,
Algemene Beschryvinge, i, f I, 501-2` cites the Beschryvinge, not the
Beschryvinge-comma-i-comma-f-I. An entry ending on a year is an imprint and
goes on being read as one.
Also: a work after a semicolon may open on a quotation mark, and the marks a
style puts round a short title belong to the style.
626 more references over the 202 reference sections sampled from the corpus.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: a real book prints its DOI that way, because the
typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces
and joiners, the soft hyphen, a byte-order mark left at the front of a
fragment: text out of a PDF or an EPUB is full of them, none of them can be
seen, and a DOI carrying one does not resolve. They are dropped before
anything is read.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: the DOI is printed that way in a real book, because
the typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces,
joiners, the soft hyphen, a byte-order mark left at the front of a fragment:
text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI
carrying one does not resolve. They are dropped before anything is read.
The largest book in the corpus is a collection of printed web pages, and
every page carries what the browser put at its foot:
http://www.u.arizona.edu/~chalmers/biblio/1.html (54 of 100) [4/7/2002 1:48:08 PM]
That was a reference with a URL and nothing else, one per page, and it is why
that book wanted a model call on all 1,114 of its chunks. An address followed
by which-page-of-how-many and a print time is furniture, not a citation.
`Spade 1980, 12.` and `Trentman 1970; Spade 1980.` are short cites: they point
at a full entry elsewhere in the same bibliography the way `ibid.` points at
the note above. The author and year they do give are kept and the title is not
asked after, because no reading of that line will produce one.
Put a real model behind the entries this could not read and looked at what it
gave back. Almost none of it was knowledge — it was punctuation already on the
page that the parser was walking past:
Goodwin, F.K. and Jamison, K.R. (1990). *Manicdepressive illness* Oxford
University Press, New York.
The title is in asterisks. It was being read as the journal, because the rule
said italics after an author-date opening are the journal — true of `Peirce
(1878). Plain title. *Popular Science Monthly*`, and false here. What decides
it is where the italics sit: at the head of the tail they are the work, after
the title they are what it appeared in.
Five more the model was covering for, all of them on the page:
* `de la Torre, Carlos` opens on a particle and is a name.
* `Tinbergen, n. 1953` has an initial the scan read in lower case.
* `Theories of Nationalism: A Critical Introduction, Basingstoke: Palgrave`
has two colons and the imprint is the last one, not the first.
* `36Armaud Puig i Tarrech` is a note number run into the name.
* `S. Marc Cohen is Professor of Philosophy at...` opens exactly as a
Chicago note does and is a contributor's biography, not a reference.
A title may quote something — `What's the Meaning of "This"?` — so only an
unbalanced quotation mark means the cut landed inside one. A question mark
ends a sentence, which is how that title was being lost. A trailing backslash
is a line break conversion left behind.
`Altmann, M. 1951, A Study of Behaviour. Sociometry 14, 351-354.` names an
author, a year, a journal and a page range, and has no bracketed year, no
`p.`, no `ed.` and no `Press,` in it — so the test chunkmd uses to spot a page
of notes says it cites nothing, and it was being dropped before it was read.
citefind adds its own test alongside, for the same reason it added its own
marker: chunkmd is deciding whether a chunk is a page of notes and can afford
to miss one entry in thirty, and this is deciding whether a reference exists.
Coverage of the model's citations rises to 84% on bibliographies.
The verse pattern now covers the forms a real page of them uses: a chapter and
verse with no book named, a note number glued into the middle of the run, a
chapter range, and a semicolon inside a verse list.
Found by putting a real model at the end of the chain: a page of notes from a
theological book composed a prompt asking who wrote Matthew 4:17, and the
model rightly answered nothing for all twenty-three entries.
Those are scripture references, which is most of what the notes of such a book
are, and they are now not references at all — no record, and no question put
to anybody about them. A journal and its volume look similar on the page and
are still read as what they are.
Also: the stop that belonged to a marker no longer opens the entry after it.
Tried against three of the largest books in the corpus — 2,700 chunks, 44,000
citations the model had already extracted — and two things showed up that the
smaller sample had not.
A chunk that begins part-way through an entry leaves a closing quotation mark
with no opening one, and the title read off it starts mid-word and runs to the
end of the line. A title carrying a stray quote, an escape, or an editor's
credit is now no title, and `unsettled` says so.
The optional fields are written as null rather than left out. A key that comes
and goes is a key every consumer has to guard, and the documented Python use
raised KeyError on the first reference without a page.
Measured against the citations an LLM had already extracted from 40 books of
the same corpus — 19,340 of them over 3,424 chunks — and improved where the
comparison showed a real gap rather than a difference of task.
On pages of notes, where most citations live: year agreement with the model
44% -> 56%, title 18% -> 23%, author 16% -> 24%.
What changed:
- A note listing several works, semicolon-separated, becomes one record per
work. The note number is kept on all of them.
- A full stop is not a separator. It parts the author from the title in every
bibliography entry there is.
- A page of body text with notes at its foot no longer reads the body as the
first note.
- A run carrying a page reference or a semicolon is not reported as a title.
- Six sentences with no imprint, no page and no identifier is a remark about
the argument, not a reference.
- A translator standing where the journal goes is a credit, not a container,
and `(London, 1973)` is a place, not a publisher.
The model is not ground truth and the comparison is not a score: on several
sampled disagreements it named the journal where the article was wanted, or
the collected works instead of the essay.
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.
Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.
`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.
chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
The hashed bag of words asks only whether two stretches repeat the same
words, and cannot tell that ships and vessels are one subject. An embedding
can. segment_with takes one vector per window and falls back to the words
when it is given none, or the wrong number of them — a caller that embedded
some other recording's windows is asking about that recording.
windows_of gives the text to embed, in the order the vectors come back in,
so nobody has to reimplement the windowing pass to use this.
Bound in all three languages, and on the command line as --windows and
--embeddings, which pipe through emb. The C ABI version goes to 2:
catena_segment_cues gained an argument.
funkpower