bibclean is "catalogue records audited and repaired, with a reason for every
change", and book-pipeline's metadata module was a second implementation of
the same thing — ported from it in the first place. Twelve functions bound:
the audit, its severity, the two commonest faults, the three repairs that
need no lookup, ISBNs checked and found, and whether two titles name the same
work.
Checked against that Python across all 5,385 books in the library — every
title, every author, every ISBN, every filename, and four thousand title
pairs. Identical on all of it but one: `The Soul After Death: Contemporary
'After-Death' Experiences`, where the Python lowercased a word inside the
quotation marks and bibclean leaves it alone.
book-pipeline held its own copy of the scheme operations bibclean already
had. The scheme itself — which headings a collection uses, their aliases,
their parents — stays with the collection, because that is the most
collection-specific thing there is. What a scheme *means* is the same
question everywhere, and this answers it: `canonical_subject`,
`compose_subjects`, `needs_subject`.
Checked against the Python over the library's 228 distinct tags and all
5,385 books: identical on every one.
book-pipeline had its own copy of bibclean's author recovery. Bound on all
four surfaces so it can stop.
Checked against that Python over all 5,385 books in the library: 5,375
identical. On the other ten the Python had taken a translator, an illustrator
or the publisher off a copyright line — Sally-Ann Spencer for *The Dwarves*,
Thomas Nelson for a table of contents, C. S. Lewis for a book about Tolkien —
and bibclean either finds the real author in the Library of Congress block or
says nothing. Every one of the ten was read by hand.
The README now lists the forms it handles with an example of each, says that
where the italics sit is what decides whether they are the title or the
journal, and sets out what it refuses and why — scripture, Migne, a
contributor's biography, a note that only thanks people, a page footer.
Recipes gains "What a book leans on": one bibliography read, then the same
over a shelf, counted per citing book so a work cited forty times in one
bibliography still counts once. Every command was run — thirty converted
books gave 14,481 references, 10,536 naming a work, in about a second.
`records.md` now points at the five tools whose record shapes live in their
own pages rather than in `corpus-core`.
And pandoc escapes an underscore, so the rule of them that stands in for a
repeated author arrives as `\_\_\_\_\_` and was not being recognised.
`looks_like_footnote_citations` is the function chunkmd's `looks_like_citations`
was ported from, and book-pipeline was still running the Python original at
four call sites. Bound on all four surfaces so it can stop.
Checked against the Python over 34,458 chunks of the live corpus: no
disagreement.
Every author-date entry that set its title in curly quotes — which is most of
them, in every style guide that has an opinion — was falling through to the
sentence split and coming back with no title:
Shaw, Joseph W., and Marjatta Luton. 2000. "The Foreshore at Akrotiri."
In *Wall Paintings of Thera*, 453-66.
The test for "does this span open the tail" allowed the span to start within
two bytes of where the tail did. A straight quote is one byte and a curly one
is three, so the check said no every time.
It now asks the question it meant to ask: the span begins after the year, and
what lies between them is punctuation and space.
An author-date entry may also open on a rule of dashes — `—— (1994),
Christianizing Homer` — which is the same entry's author as the one above,
and was stopping the whole line from being read.
Ranked what still had no work across 145,987 entries from the 30 most-cited
books, and the largest single class was this rule's own doing. A conversion
that loses an opening quotation mark leaves its partner at the end:
McClelland, J.L, D.E. Rumelhart, & G.E. Hinton 1986. The appeal of
parallel distributed processing". In Parallel Distributed Processing…
That was being refused as a title the cut had landed inside. One quotation
mark at the very end is a stray; one in the middle still means the opening is
on the page before, and is still refused.
A note that opens by thanking people cites nothing, however many names it
carries — and among endnotes it carries a great many. `Spade 1975 and 1980.`
is one short cite naming two years.
Measured against the religious books in the corpus, where the style is
Chicago throughout. Eight concrete forms it could not read:
* `Cheyne, T. K., Founders of Old Testament Criticism, London, 1893.` —
commas all the way, an imprint with no colon in it, the place last but
one and the year last.
* `Erich Caspar, Geschichte des Papsttums, Vol. 2 (Tübingen, 1933)` — the
volume was being read as a page reference and the title thrown away with
it. A volume is where in the work, not what the work is called.
* `St Valerian, Homilies (Washington D.C.: Catholic University of America
Press).` — a bracketed imprint with no year in it.
* `BØRRESEN, K. E. (1981), Subordination and Equivalence: The Nature and
Role of Woman, Washington: University Press of America.` — a title with
a colon looks exactly like an imprint when the pattern may match at the
start of one.
* `Krochmal, N., Moreh Nebukhe ha-Zeman, p. 208.` — the initials sit in a
comma-part of their own and belong to the name.
* `PG 4: 264, 337; PG 91: 1312-13` — Migne, and CSEL, NPNF, SC and the
rest: shelves, not works, and the work is not named. Refused like
chapter and verse.
* `Lowrie and Fletcher, p. 133; Anderson, p. 400.` — a name and a page
with nothing between them points at a work it does not name.
* `(Tübingen, 1933)` is a place and a year, and a place is not a publisher
wherever it is found.
`Unger, Dutch Shipbuilding, 33-4` — a surname, a short title, and where in
it. No year, no publisher, nothing but commas. It is how a humanities book
cites a work it has already given in full, which is most of what its notes
are: one book in the corpus has 2,229 such chunks and this read almost none
of them.
The locators trailing a short note are not part of its title — `Sjoerds,
Algemene Beschryvinge, i, f I, 501-2` cites the Beschryvinge, not the
Beschryvinge-comma-i-comma-f-I. An entry ending on a year is an imprint and
goes on being read as one.
Also: a work after a semicolon may open on a quotation mark, and the marks a
style puts round a short title belong to the style.
626 more references over the 202 reference sections sampled from the corpus.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: a real book prints its DOI that way, because the
typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces
and joiners, the soft hyphen, a byte-order mark left at the front of a
fragment: text out of a PDF or an EPUB is full of them, none of them can be
seen, and a DOI carrying one does not resolve. They are dropped before
anything is read.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: the DOI is printed that way in a real book, because
the typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces,
joiners, the soft hyphen, a byte-order mark left at the front of a fragment:
text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI
carrying one does not resolve. They are dropped before anything is read.
The largest book in the corpus is a collection of printed web pages, and
every page carries what the browser put at its foot:
http://www.u.arizona.edu/~chalmers/biblio/1.html (54 of 100) [4/7/2002 1:48:08 PM]
That was a reference with a URL and nothing else, one per page, and it is why
that book wanted a model call on all 1,114 of its chunks. An address followed
by which-page-of-how-many and a print time is furniture, not a citation.
`Spade 1980, 12.` and `Trentman 1970; Spade 1980.` are short cites: they point
at a full entry elsewhere in the same bibliography the way `ibid.` points at
the note above. The author and year they do give are kept and the title is not
asked after, because no reading of that line will produce one.
Put a real model behind the entries this could not read and looked at what it
gave back. Almost none of it was knowledge — it was punctuation already on the
page that the parser was walking past:
Goodwin, F.K. and Jamison, K.R. (1990). *Manicdepressive illness* Oxford
University Press, New York.
The title is in asterisks. It was being read as the journal, because the rule
said italics after an author-date opening are the journal — true of `Peirce
(1878). Plain title. *Popular Science Monthly*`, and false here. What decides
it is where the italics sit: at the head of the tail they are the work, after
the title they are what it appeared in.
Five more the model was covering for, all of them on the page:
* `de la Torre, Carlos` opens on a particle and is a name.
* `Tinbergen, n. 1953` has an initial the scan read in lower case.
* `Theories of Nationalism: A Critical Introduction, Basingstoke: Palgrave`
has two colons and the imprint is the last one, not the first.
* `36Armaud Puig i Tarrech` is a note number run into the name.
* `S. Marc Cohen is Professor of Philosophy at...` opens exactly as a
Chicago note does and is a contributor's biography, not a reference.
A title may quote something — `What's the Meaning of "This"?` — so only an
unbalanced quotation mark means the cut landed inside one. A question mark
ends a sentence, which is how that title was being lost. A trailing backslash
is a line break conversion left behind.
`Altmann, M. 1951, A Study of Behaviour. Sociometry 14, 351-354.` names an
author, a year, a journal and a page range, and has no bracketed year, no
`p.`, no `ed.` and no `Press,` in it — so the test chunkmd uses to spot a page
of notes says it cites nothing, and it was being dropped before it was read.
citefind adds its own test alongside, for the same reason it added its own
marker: chunkmd is deciding whether a chunk is a page of notes and can afford
to miss one entry in thirty, and this is deciding whether a reference exists.
Coverage of the model's citations rises to 84% on bibliographies.
The verse pattern now covers the forms a real page of them uses: a chapter and
verse with no book named, a note number glued into the middle of the run, a
chapter range, and a semicolon inside a verse list.
Found by putting a real model at the end of the chain: a page of notes from a
theological book composed a prompt asking who wrote Matthew 4:17, and the
model rightly answered nothing for all twenty-three entries.
Those are scripture references, which is most of what the notes of such a book
are, and they are now not references at all — no record, and no question put
to anybody about them. A journal and its volume look similar on the page and
are still read as what they are.
Also: the stop that belonged to a marker no longer opens the entry after it.
Tried against three of the largest books in the corpus — 2,700 chunks, 44,000
citations the model had already extracted — and two things showed up that the
smaller sample had not.
A chunk that begins part-way through an entry leaves a closing quotation mark
with no opening one, and the title read off it starts mid-word and runs to the
end of the line. A title carrying a stray quote, an escape, or an editor's
credit is now no title, and `unsettled` says so.
The optional fields are written as null rather than left out. A key that comes
and goes is a key every consumer has to guard, and the documented Python use
raised KeyError on the first reference without a page.
Measured against the citations an LLM had already extracted from 40 books of
the same corpus — 19,340 of them over 3,424 chunks — and improved where the
comparison showed a real gap rather than a difference of task.
On pages of notes, where most citations live: year agreement with the model
44% -> 56%, title 18% -> 23%, author 16% -> 24%.
What changed:
- A note listing several works, semicolon-separated, becomes one record per
work. The note number is kept on all of them.
- A full stop is not a separator. It parts the author from the title in every
bibliography entry there is.
- A page of body text with notes at its foot no longer reads the body as the
first note.
- A run carrying a page reference or a semicolon is not reported as a title.
- Six sentences with no imprint, no page and no identifier is a remark about
the argument, not a reference.
- A translator standing where the journal goes is a credit, not a container,
and `(London, 1973)` is a place, not a publisher.
The model is not ground truth and the comparison is not a score: on several
sampled disagreements it named the journal where the article was wanted, or
the collected works instead of the essay.
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.
Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.
`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.
chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
The hashed bag of words asks only whether two stretches repeat the same
words, and cannot tell that ships and vessels are one subject. An embedding
can. segment_with takes one vector per window and falls back to the words
when it is given none, or the wrong number of them — a caller that embedded
some other recording's windows is asking about that recording.
windows_of gives the text to embed, in the order the vectors come back in,
so nobody has to reimplement the windowing pass to use this.
Bound in all three languages, and on the command line as --windows and
--embeddings, which pipe through emb. The C ABI version goes to 2:
catena_segment_cues gained an argument.
funkpower