The commit that reordered the chain did not carry the version bump. The
bump was chained after check.sh with &&, the commit after a semicolon, so
a failing check skipped the bump and let the commit through. The tag then
pointed at a tree still calling itself 0.3.0, which the release workflow
asserts against and would have refused.
check.sh was failing because a test names which register is asked first
about an ISBN, which is exactly what changed. It now names the index, for
the reason the reorder happened.
The chain asked OpenLibrary's edition record first and its search index
second. The edition record is the slow one: measured against this
library it takes five to twenty-eight seconds, and it names its authors
by key, so a second request was needed anyway. The index, asked for its
fields by name, returns the title, the authors, the publisher, the
language, a page count and the subjects together, in about a quarter of
a second.
Reordering them turns three requests per book into one, and a pass over
five thousand books from fifty-three hours into well under one. The
edition record is still there for a subtitle or a publisher the index
did not give.
The chain asked three registers and returned whichever answered first
with a title and an author, discarding the rest. The registers are
complementary rather than redundant: OpenLibrary's edition record
carries the publisher, the pagination and the language and names its
authors by key, while its search index carries the authors and not the
publisher. One book, two half-answers, and the whole of it only if they
are put together.
They are merged now, first answer winning each field, so the order of
the chain is the order of trust. The record says which registers spoke.
It also carries what was being thrown away: subjects, language and a
page count, from whichever register has them. The index will not return
those unless asked for by name, which is how a language it held went
unread. Subjects are capped at twelve — the index lists every heading
every edition of a work has ever been given, which on a well-known book
is hundreds.
An edition record keeps its subtitle in a separate field, so a title
came back without it.
Read out of the review queue a real run produced, where each of these was
sitting as a proposed title waiting for a person to say no:
1. General Remarks a numbered heading
2 James W. McAllister a numbered heading
5222 What Is This a folio that ran into the line
827 North Fourth Street an address
Page xxxvi / Chapter I a marker
title a form label, printed literally
Sidebottom, Harry. Ancient a bibliography entry
To Mackenzie, a dedication
The numbers are the delicate part. A leading number that could be a year
is left alone, because 1984 Revisited is a book and 5222 What Is This is
a folio, and nothing else distinguishes them. A dedication has to end in
a comma, so To Kill a Mockingbird survives.
The copyright notices in the same pile are already fixed: those
proposals were written before 0.2.2 and are stale rather than wrong.
title_case counted small words from the start of the whole string, so
every subtitle that opened with one was set in lower case: 'Aesthetic in
Kant: a Critique', 'Born Fighting: the Scots-Irish'. A colon starts a
title of its own and the first word of one is capitalised however small
it is.
Found by book-pipeline, which had been proposing a title page's capitals
verbatim into a catalogue and needed this to stop.
'(c) Alfredo Ferrarin 2004' was read as a title. The notice pattern
wanted the year to follow the symbol directly, and a copyright line puts
the holder's name in between, which is the ordinary way to set one.
read_colophon is several milliseconds of computation over a few hundred
kilobytes, and it held the GIL for all of them. A caller reading a
library through asyncio.to_thread therefore got no overlap whatsoever:
six books in flight, one book's worth of work at a time.
Measured on this library, that was most of why a concurrent pass still
paced at four and a half hours.
colophon has composed the question about what its pages did not settle
since it was written, and merged a reply back into the record it read.
Both were reachable only from a terminal, so a pipeline wanting the same
fallback would have had to write its own prompt — a second one, drifting
from the first, asking for a shape merge does not accept.
merge fills only the unsettled fields, so a model cannot overwrite what a
book printed about itself. That is the property that makes asking one
safe, and it belongs to colophon rather than to any caller.
Read against five thousand real books, the title page is often not the
first thing in the file, and what is there instead has shape. Six kinds
of it were being read as titles:
vi CONTENTS a folioed heading
Copyright (c) 2013 by Barry Davies, BEM a notice
Translated from the French by Willard R. Trask a credit
Vol. 11. HENRY JAMES, JR. a collected edition's list
[PART ONE: a part heading
Piper Verlag GmbH, Munich<span style=... a converter's leavings
Rejecting one line only moves the misread down one, because a page that
converted badly converted badly throughout, so the whole shape goes:
markdown links, stylesheet declarations, HTML entities, epub filenames,
and the sentences a copyright page sets.
The notice pattern wants the symbol or the year, never the word alone.
The first draft rejected 'Copyright and Its Discontents', which its own
test caught.
On sixty books this drops five claimed titles, all of them wrong, and
fixes six that were reading one homily's heading for every other
homily's title.
book-pipeline reaches for colophon and bibfetch on every book it repairs.
Shelling out would mean a process per book per tool across five thousand
books, and parsing back the JSON it is about to parse anyway.
bibfetch's fetch loop lived in main.rs, so binding it meant either
lifting it or writing a second one. A second implementation of 'which
register answers about this kind of identifier, and in what order' would
drift from the first, and the copy nobody runs from a terminal would
drift silently. It is now bibfetch::ask, and the command line calls it
too — verified by running the CLI before and after and diffing.
fetch_identifier releases the GIL around the request. It is the only
function here that opens a socket, and holding the interpreter for the
length of an HTTP call would stop every other thread in it.
bibfetch is optional and only the python feature turns it on: its HTTP
client does not build for wasm32, and an unconditional dependency would
have taken the JavaScript package down with it.
A cataloguing block states subjects in Library of Congress form, which
is precise, deep and no use for browsing. Real blocks from this library
read 'Political science – Early works to 1800' and 'English
literature—Middle English, 1100–1500—History and criticism'. Nobody has
a shelf called that.
--shelf takes the closed list the library is actually browsed by and
files the printed headings onto it, longest term first so a period of
thought does not collapse into its parent. Matching is on whole words
after folding punctuation away, because a cataloguing block separates
with em dashes, en dashes, full stops and plain hyphens, often three in
one heading, and OCR turns any of those into any other.
What matches nothing comes back under unshelved rather than being
dropped: a subject silently discarded is indistinguishable from a book
that printed none, and the difference is the whole value of reading the
block.
An ISBN is not a DOI. The body that assigns a DOI is the only one that has
it, so one register is the whole answer. A book's identifier is assigned by
its publisher and recorded by whoever gets round to it, and the record that
comes back is as often as not missing the thing the caller wanted: asked
about 9780199205592, OpenLibrary's edition record names *The First World
War* and the press that printed it, and nobody at all as its author, because
an edition record names its authors by key.
So an ISBN now has a chain rather than a register. The edition record first,
then OpenLibrary's own index — which answers with `author_name` filled in
and costs nothing — and Google Books last, because it answers anonymous
callers only until a daily quota runs out, after which it replies 429 to
everyone behind the same address, as it is doing here today.
The first complete answer wins, an incomplete one is kept in case nothing
better comes, and a register being down is a failure only when every
register is. `source` on the record says which one answered.
colophon comes with it: the imprint reader was taking `Oxford is a
registered trade mark of Oxford University Press` for a publisher and
`Oxford N`, off `Oxford New York`, for a place. A legal notice mentioning a
press is not the press, and a city is the city and not the first letter of
the next one.
A catalogue's worst records are the ones whose file says nothing true about
itself: a PDF titled `Untitled` by an author called `Oxford`, which is a
publisher, from a filename that was never anything but a filename. What
those books do have is pages. The title page carries the title as the
publisher set it; the copyright page carries the imprint, the year, the
ISBN, and in most English books since the 1970s the cataloguing block, which
is a catalogue record a librarian already made, subject headings and all.
So this reads the pages. Every field names the page it came from, and a book
that printed nothing comes back empty with its fields in `unsettled` rather
than filled with something plausible — an extraction that starts at chapter
one has no front matter in it, and saying so is the answer.
Two pages look like a title page and are not, and both are how a catalogue
comes to name the wrong book. A series advertisement lists other titles in
the shape a title page uses, over as many as four pages with only the first
announcing itself. A translation's cataloguing block prints the original
title in brackets above the real one. Both are found and refused.
Three things the pages give that no file does: the subject headings a
librarian assigned, an ISBN that exists only in print — which turns a book
with no identifier into one `bibfetch` can look up — and a title in the case
the publisher meant, because a half title prints unbroken what the title
page breaks across two lines and sets in capitals.
`--prompt` and `--merge` are citefind's: a question about exactly the fields
the pages left unsettled, composed here and answered wherever the caller
keeps a model, with the advertising taken out of what the model is shown.
A field a model answered is marked `model`, so a record always says which of
it was read and which was inferred.
Measured against 600 books of the home corpus: the cataloguing block
answered for 21% of them and agreed with the catalogue 89% of the time, most
of the residue being the catalogue's error rather than this one's. Title
pages answered for another 41%. The remaining 38% print nothing readable,
and are what the prompt is for.
Both registries already have a `catena`: an HPE blockchain client on PyPI,
and a JavaScript OOP wrapper at 2.2.0 on npm. Neither is going anywhere, so
the distributions take a name that is free on both.
The Python import is unaffected — a distribution name and a module name are
independent there, so `pip install catena-corpus` still gives `import
catena`, and nothing in the examples or the bindings changes. On npm the
package name is the import specifier, so that one does change: `import
{ chunkMarkdown } from "catena-corpus"`.
The crates keep their own names. `catena` is free on crates.io.
Two things between a converted EPUB and its chapter titles.
The heading level taken as the chapter map was the shallowest present. Pandoc
writes the book's title as the only `#` and every real chapter as a `##`, so
the shallowest level had one heading in it and the whole book came back as a
single chapter. The dividing level is now the shallowest one used more than
once — a level used once is a title.
And an empty bracketed span survived the heading cleaner, because the pattern
wanted at least one character between the brackets. Pandoc writes an anchor
as exactly that: `## []{#link2H_INTR} Introduction`.
Gibbon's first volume, converted from the Project Gutenberg EPUB: 2 distinct
chapter titles before, 61 after, and none of them carrying an element id.
The chapter a passage carries was resolved at the offset of the page it came
from, not the offset of the passage. A document with no page markers is a
single page beginning at zero, so every chunk in such a book answered with
whichever heading came first in the file — and took the role classification
with it, since that reads the chapter title.
Measured over a corpus of 5,386 books: all 1,232 unpaginated books with
twenty or more chunks carried exactly one chapter title from cover to cover,
as did 73% of the paginated ones. A bibliography chapter was therefore
invisible to anything selecting on the chapter title, which is the first of
the three ways the citation stage finds one.
The packer now hands back each passage with the offset it began at, and the
page's own offset is the true start of its text rather than the position of
the marker before it.
The harness compared these tools against the code they replaced — Python
imported from book-pipeline, podcast-pipeline and knowledge-mcp, and
JavaScript copied out of the podcast media server and indexer, each pinned to
a commit. All of it has since been deleted, so the live half cannot run and
the golden half pins answers against a source nobody can consult.
What was worth keeping was never the comparison. `KNOWN.md` argued, at
length, why a dozen of these tools deliberately do something a reader would
not expect: why pdfgutter measures the crop box throughout, why avplan finds
the encoder rather than reading one out of the environment, why mdscrub will
not manufacture a page boundary. Those arguments are about this code and
stand on their own, so they are now `docs/decisions.md`, rewritten to say
what each tool does and why rather than what it does differently.
Binding coverage is unaffected: the bindings carry 24 tests of their own, a
node smoke test against the built package, and a typecheck.
An entry opening was a surname at the start of a line, and a converted
bibliography is often a bulleted list — so the bullet sat exactly where the
surname was looked for and not one entry was seen:
• Skorupski, J. 1995. 'Agent-Neutrality, Consequentialism,
Utilitarianism', Utilitas, 7: 49-54.
The whole page went unflagged, no chunk of it was ever selected, and books
whose only bibliography arrived this way came back with no citations at all.
A bullet may now stand where the surname would.
Measured against the corpus: of 300 sampled body chunks that open lines on a
bullet and an inverted name, 0 were flagged before and 223 after. Across
3,000 randomly sampled unflagged body chunks, the change fires on none — it
buys the bulleted list and nothing else.
The bar a guessed title has to clear was waived when the entry also gave up
a container, on the reasoning that a container vouches for the rest of the
reading. Auditing the corpus rather than the reference chapters showed that
is wrong, and wrong on the commonest note there is:
As Lewis White Beck has rightly put it. See his Early German
Philosophy (Cambridge, Mass., 1969), p. 439.
The remark opens, the citation follows, and the second sentence becomes the
container — so the first was stored as the work and the book named in it was
lost. Neither a siglum nor a remark is the name of a work now, whatever else
came off the entry.
Measured where these notes actually live, which is flagged body chunks
rather than bibliography chapters: on the corpus the class ran at 1.43% of
rows naming a work. The cost is 46 rows in 34,359 on the reference holdout,
where the share naming an author rises from 88.1% to 88.2%.
The stop after `P.` is an initial rather than a sentence stop, so `Hoyt,
Edwin P. The Supersalesmen. New York: World Publishing, 1962.` never got cut
there. The name came back as `Hoyt, Edwin P. The Supersalesmen` and the next
segment along — the imprint — became the work. Both fields wrong from one
missing cut.
An inverted name whose forenames end in an initial, with two or more words
after it, now ends at that initial. Two words, because one is a forename:
`Price, M. Philips` and `Drexler, K. Eric` are whole names and must not be
cut.
And the segment the reader settles on for a title may not be an imprint —
which is what the function has said it did since it was written, in a comment
above a predicate that never checked. Asked narrowly enough that a title with
a colon in it survives.
On 199 reference chapters never used for tuning, imprints stored as the work
fall from 376 to 55, and the share of rows naming an author rises from 86.1%
to 88.1%.
Three refusals named on the page: a note's own remark, a siglum, and the
navigation strip a PDF reader leaves behind. Says where the line is drawn —
at the guess rather than at the title, so an entry that marked its own title
is taken at its word.
A title read off markup was vouched for by the entry. A title guessed at by
cutting the entry at its sentence stops was not, and where nothing else came
off the entry — no author, no identifier, no container — that guess was the
whole of what got stored. Two things kept arriving that way.
A note that remarks on the argument rather than citing anything: "The
Commentator seems to have based himself principally on the Bible", "Dewey
remarks that we experience most things as temporal". A page of endnotes is
mostly these. They are noun phrases nowhere: what separates them from
"Quaestiones Parisienses" is a finite verb, so that is what is asked.
A siglum and a page — `H, 272`, `FR,120-122`, `GNO` — whose key is a page of
its own the entry does not carry. An atlas of Middle-earth is a thousand of
them, and the heading each sits under is not its title, so a siglum is
refused whatever else came off the entry.
Also refuses the navigation strip a PDF reader leaves in the text, `572 next
page > < previous page`, and the local path it points at.
On 400 flagged body chunks the junk rate falls from 3.5% to 0.5% of stored
rows; on the 202-file reference sample the share naming a work holds at
97.2% across 42 fewer rows, all of them sigla, scripture or prose.
Three more shapes in the table — a reading list with nothing marking it, an
editor in the author's place, a reprint's two years — and three more refusals
named: a keyed short cite, a translator's signed note, and a page of
references whose lines happen to begin on numbers.
Also says what the cut does with a blank line now: it is often after the
running head rather than between the entries under it, so every block it
produces is looked at again.
Kymlicka, Will, Politics in the Vernacular: Nationalism,
Multiculturalism and Citizenship, Oxford, 2001
The imprint is found by counting commas, and the last comma-part carrying a
colon is taken to be it — which is right when the colon is `Place:
Publisher` and wrong when it is the title's own. Here it is the only colon in
the entry, and looking for it before the names have been read meant giving up
rather than falling through to the year at the end. The search starts after
the names now, so a colon that is part of the title is not mistaken for one.
Juvenal, Saturae XIV, ed. J. D. Duff (Cambridge, 1962), p. 9.
A title that ran on into an editor's credit was refused outright. Trimming
the credit off is the repair that was wanted: the credit belongs to the
entry, not to the work's name, and the work is `Saturae XIV`.
Mate selection: Buss, 1989; Symons, 1979. Jealousy: Buss, 2000.
A notes section written this way keys the author's own words to works given
in full in the bibliography. `Mate selection` is not a name and `Buss, 1989`
is not a work, so the note points at something it does not name — the same
thing `ibid.` does, and it is flagged the same way. It was being stored as a
reference whose author was the topic, and the entry below it inherited that
as a repeated author.
The pattern is deliberately narrow: the part after the colon must be nothing
but surnames and years. `Miller, David, On Nationality, Oxford: Clarendon
Press, 1995` has a colon and a year too, and is a book.
And a run the style itself marked as a title is a title whatever it opens on.
The rule against a title beginning with a number is about a page reference
picked up by mistake; inside italics there is no mistake to make, and
`*1 & 2 Thessalonians*` is two books of the Bible.
Three shapes out of the 20% of records that still named no work:
Arnauld, A., and P. Nicole. 1662/1965. La logique ou l'art de penser.
Barratt, Alexandra, ed. Women's Writing in Middle English. London, 1992.
Schleiermacher, Friedrich. 1813/1963. "Ueber die verschiedenen Methoden
des Uebersezens". In Das Problem des Uebersetzens, edited by H.J. Störig,
38-70. Darmstadt: Wissenschaftliche Buchgesellschaft.
A reprint gives the year it was written and the year of the edition used, and
the author-date opening stopped at the slash — so the title went to the
container and the container to nothing.
`ed.` is an abbreviation rather than a sentence stop, which is right, and
meant the editor's name and the work's title stayed one segment and neither
was read. An editor is a person and the credit is theirs, not part of either.
And the Schleiermacher entry was refused outright for running to six
sentences, because German titles do. Its page range says plainly that it is
a chapter in a book — but `38-70` carries no `pp.`, and only `pp.` counted.
A page range after a comma counts now. The passage of body prose that
started all this is still refused: its numbers are a date, not a range, and
nothing precedes them but a verb.
\_\_\_\_\_. "Die Frage nach der Echtheit von Matth. 16, 17-19."
*Theologische Blätter* 20 (1941): 26-30.
The dash means the author of the entry above, and an author-date entry may
open on one — so the check that decides whether a run of text is the author
half of such an entry accepted anything beginning with a rule of dashes.
Here that is the whole entry: the title and the journal were read as more of
the author, and the title came back as the page range after the year.
Only a bare rule counts now. Anything after it belongs to the work.
Mellor, Hugh (1974) "In Defense of Dispositions," Philosophical Review 83:157-
81. Reprinted in Mellor (1991):104-22.
`81.` there is the second half of a page range whose first line ran out of
room. It is also, at the start of a line, exactly what a note marker looks
like — and two of them on a page of twenty-three references were enough to
send the whole page down the numbered-notes path, where it was cut into two
enormous entries and both were thrown away for running past a hundred and
twenty words.
Heil's *Philosophy of Mind* came out of the corpus with one citation. The
page above gives nineteen.
The two readings are now weighed against each other: more lines opening on a
surname and a comma than there are note markers means it is a bibliography.
A real page of notes has many markers and no line that opens on a surname,
because every line opens on its number.
Over the 202 reference sections sampled from the corpus: 10,047 to 10,405.
Catherine H. Zuckert
Rowe, C. J. 1996. "The Politicus." In Form and Argument, ed. C. Gill. Oxford.
Saunders, T. J. 1992. "Plato's Later Political Thought." Cambridge.
Scodel, H. R. 1987. Diaeresis and Myth in Plato's Statesman. Göttingen.
A page of *The Cambridge Companion to Ancient Greek Political Thought*, and
citefind read nothing in it at all. The blank line is after the running head;
there is none between the references. The cut took the blank line as
authoritative, made the whole bibliography one entry, and then refused it as
prose for running past a hundred and twenty words.
Each block a blank line produces is now looked at again and cut by its lines
where its lines are entries. One line opening on a surname and a comma is
enough inside a block — a block is a single entry or a list of them, not a
mixture, so the lines that do not open that way are continuations of the one
above.
Over the 202 reference sections sampled from the corpus: 9,247 references to
10,047.
Block, N. 1981. Psychologism and behaviorism. Philosophical Review 90:5-43.
Dennett, D. C. 1987. The Intentional Stance. MIT Press.
Geach, P. 1957. Mental Acts. Routledge and Kegan Paul.
That is how a reader or a companion sets the further-reading list at the end
of each chapter: indented under the last paragraph of the essay, with no
heading over it and no numbers beside it. Nothing announced it as a
bibliography, so the chunk stayed body text with no flag, the citation stage
never selected it, and a book made of such chapters came out with one or two
citations. *The Cambridge Companion to Ancient Greek Political Thought* had
three; Chalmers's *Philosophy of Mind* reader had one.
The hard part is that an index line opens identically — `Newton, Isaac, 5,
93, 141` — and a page of them would otherwise flood the table. What tells
them apart is that a reference names a work: five words of three letters or
more on the entry's own line, and not two page numbers written as links.
Over 15,925 unflagged body chunks sampled from the corpus this fires on 2.6%,
and seven of eight sampled were real reference lists.
Recorded in parity/KNOWN.md, and the harness allows this one difference by
name: a chunk the Rust flags and the Python does not, identical in every
other field. A flag the Python sets and the Rust does not is still a failure.
Thirdly, not all fields of science and technology were seen as equally
relevant to the nation's war effort. Bush himself was an electrical
engineer… And finally, growing out of the OSRD's experience, Bush
proposed — most famously in his 1945 report, Science: The Endless
Frontier — that Washington continue to play a leading role…
Six sentences of body text, stored as a citation whose title was its first
clause. The check that refuses a note for running on made an exception for
anything carrying an imprint, and `his 1945 report, Science: The Endless
Frontier` is shaped exactly like `Place: Publisher`. One colon anywhere in a
paragraph bought the whole paragraph a pass.
A page reference or an identifier still buys the exemption — a note that
cites a work and then discusses it is a citation. An imprint no longer does.
And an entry past a hundred and twenty words is prose whatever its
punctuation: the longest real entry in the corpus sample is under sixty.
Both volumes of *The Decline of the West* went from zero citations to 415
once the arrow marker was recognised — and 167 of those were the editor's and
the translator's own glosses:
Latin: "nature of the thing." — Ed.
Reference to the wars between Athens and Sparta. Their dates are
431-404 BC and 31 BC. — Ed.
An em dash and a role at the very end is how a translator signs a note of
their own. It cites nothing, and it is a convention every academic
translation uses.
391 of the 415 also carried a stray `<!-- -->`: pandoc's list separator, left
in the middle of the reference text. HTML comments are conversion furniture
and are cut with the marker.
And `See R. Hirzel, Die Person (1914), p. 7.` read as an author-date entry
whose authors were "R. Hirzel, Die Person" and whose title was nothing — the
head is all capitalised words and the year is bracketed, which is exactly
what an author-date opening looks like. Where that reading yields no title
and reading it as a note yields one, the note wins. Having a title is how you
can tell which was meant.
\[←788 \]
\[[←193](index_split_011.xhtml#citation-193) \]
An arrow pointing back to where the note was called, sometimes a link and
sometimes not, inside escaped brackets. Neither pattern knew it, so a page of
notes set this way read as body prose: the chunk was never flagged, the
citation stage never selected it, and the book came out with nothing.
Ten books of the corpus use it and eight of them had zero citations —
including both volumes of *The Decline of the West*, whose 830 endnotes are
exactly what this is for. 142 of the 199 affected chunks are those two books.
Recorded in parity/KNOWN.md: the Python this was ported from did not know the
marker either, and no fixture carries one, so the suite agrees either way.
`10.1016/S0140-6736(16)30370-1` came back as `10.1016/s0140-6736` — a
different DOI, and one that resolves, so nothing downstream could tell. The
pattern excluded brackets to keep the ones a sentence wraps round a DOI, and
Elsevier alone publishes hundreds of thousands of DOIs that contain them.
Brackets are taken now, and an unbalanced closer at the end is given back:
`(10.1234/foo)` in a sentence ends on a bracket the sentence opened, and
`10.1016/S0140-6736(16)30370-1` ends on digits with a matched pair in the
middle. Counting is what tells them apart.
Found by comparing idnorm against policy-pipeline's own DOI normaliser over
229,556 real documents.
bibclean is "catalogue records audited and repaired, with a reason for every
change", and book-pipeline's metadata module was a second implementation of
the same thing — ported from it in the first place. Twelve functions bound:
the audit, its severity, the two commonest faults, the three repairs that
need no lookup, ISBNs checked and found, and whether two titles name the same
work.
Checked against that Python across all 5,385 books in the library — every
title, every author, every ISBN, every filename, and four thousand title
pairs. Identical on all of it but one: `The Soul After Death: Contemporary
'After-Death' Experiences`, where the Python lowercased a word inside the
quotation marks and bibclean leaves it alone.
book-pipeline held its own copy of the scheme operations bibclean already
had. The scheme itself — which headings a collection uses, their aliases,
their parents — stays with the collection, because that is the most
collection-specific thing there is. What a scheme *means* is the same
question everywhere, and this answers it: `canonical_subject`,
`compose_subjects`, `needs_subject`.
Checked against the Python over the library's 228 distinct tags and all
5,385 books: identical on every one.
book-pipeline had its own copy of bibclean's author recovery. Bound on all
four surfaces so it can stop.
Checked against that Python over all 5,385 books in the library: 5,375
identical. On the other ten the Python had taken a translator, an illustrator
or the publisher off a copyright line — Sally-Ann Spencer for *The Dwarves*,
Thomas Nelson for a table of contents, C. S. Lewis for a book about Tolkien —
and bibclean either finds the real author in the Library of Congress block or
says nothing. Every one of the ten was read by hand.
The README now lists the forms it handles with an example of each, says that
where the italics sit is what decides whether they are the title or the
journal, and sets out what it refuses and why — scripture, Migne, a
contributor's biography, a note that only thanks people, a page footer.
Recipes gains "What a book leans on": one bibliography read, then the same
over a shelf, counted per citing book so a work cited forty times in one
bibliography still counts once. Every command was run — thirty converted
books gave 14,481 references, 10,536 naming a work, in about a second.
`records.md` now points at the five tools whose record shapes live in their
own pages rather than in `corpus-core`.
And pandoc escapes an underscore, so the rule of them that stands in for a
repeated author arrives as `\_\_\_\_\_` and was not being recognised.
`looks_like_footnote_citations` is the function chunkmd's `looks_like_citations`
was ported from, and book-pipeline was still running the Python original at
four call sites. Bound on all four surfaces so it can stop.
Checked against the Python over 34,458 chunks of the live corpus: no
disagreement.
Every author-date entry that set its title in curly quotes — which is most of
them, in every style guide that has an opinion — was falling through to the
sentence split and coming back with no title:
Shaw, Joseph W., and Marjatta Luton. 2000. "The Foreshore at Akrotiri."
In *Wall Paintings of Thera*, 453-66.
The test for "does this span open the tail" allowed the span to start within
two bytes of where the tail did. A straight quote is one byte and a curly one
is three, so the check said no every time.
It now asks the question it meant to ask: the span begins after the year, and
what lies between them is punctuation and space.
An author-date entry may also open on a rule of dashes — `—— (1994),
Christianizing Homer` — which is the same entry's author as the one above,
and was stopping the whole line from being read.
Ranked what still had no work across 145,987 entries from the 30 most-cited
books, and the largest single class was this rule's own doing. A conversion
that loses an opening quotation mark leaves its partner at the end:
McClelland, J.L, D.E. Rumelhart, & G.E. Hinton 1986. The appeal of
parallel distributed processing". In Parallel Distributed Processing…
That was being refused as a title the cut had landed inside. One quotation
mark at the very end is a stray; one in the middle still means the opening is
on the page before, and is still refused.
A note that opens by thanking people cites nothing, however many names it
carries — and among endnotes it carries a great many. `Spade 1975 and 1980.`
is one short cite naming two years.
Measured against the religious books in the corpus, where the style is
Chicago throughout. Eight concrete forms it could not read:
* `Cheyne, T. K., Founders of Old Testament Criticism, London, 1893.` —
commas all the way, an imprint with no colon in it, the place last but
one and the year last.
* `Erich Caspar, Geschichte des Papsttums, Vol. 2 (Tübingen, 1933)` — the
volume was being read as a page reference and the title thrown away with
it. A volume is where in the work, not what the work is called.
* `St Valerian, Homilies (Washington D.C.: Catholic University of America
Press).` — a bracketed imprint with no year in it.
* `BØRRESEN, K. E. (1981), Subordination and Equivalence: The Nature and
Role of Woman, Washington: University Press of America.` — a title with
a colon looks exactly like an imprint when the pattern may match at the
start of one.
* `Krochmal, N., Moreh Nebukhe ha-Zeman, p. 208.` — the initials sit in a
comma-part of their own and belong to the name.
* `PG 4: 264, 337; PG 91: 1312-13` — Migne, and CSEL, NPNF, SC and the
rest: shelves, not works, and the work is not named. Refused like
chapter and verse.
* `Lowrie and Fletcher, p. 133; Anderson, p. 400.` — a name and a page
with nothing between them points at a work it does not name.
* `(Tübingen, 1933)` is a place and a year, and a place is not a publisher
wherever it is found.
`Unger, Dutch Shipbuilding, 33-4` — a surname, a short title, and where in
it. No year, no publisher, nothing but commas. It is how a humanities book
cites a work it has already given in full, which is most of what its notes
are: one book in the corpus has 2,229 such chunks and this read almost none
of them.
The locators trailing a short note are not part of its title — `Sjoerds,
Algemene Beschryvinge, i, f I, 501-2` cites the Beschryvinge, not the
Beschryvinge-comma-i-comma-f-I. An entry ending on a year is an imprint and
goes on being read as one.
Also: a work after a semicolon may open on a quotation mark, and the marks a
style puts round a short title belong to the style.
626 more references over the 202 reference sections sampled from the corpus.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: a real book prints its DOI that way, because the
typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces
and joiners, the soft hyphen, a byte-order mark left at the front of a
fragment: text out of a PDF or an EPUB is full of them, none of them can be
seen, and a DOI carrying one does not resolve. They are dropped before
anything is read.
`10.1093/<zwsp>oso/<zwsp>` is exactly sixteen bytes, which is the length of
`https://doi.org/`, so testing that prefix against it sliced into the middle
of the third zero-width space and brought the process down. Found by running
citefind over the corpus: the DOI is printed that way in a real book, because
the typesetter marked where the line could break.
Both prefix tests — DOI and arXiv — now ask for the head rather than cutting
one, so a byte length that is not a character boundary is an answer of no
rather than a panic.
And an invisible character is not part of an identifier. Zero-width spaces,
joiners, the soft hyphen, a byte-order mark left at the front of a fragment:
text out of a PDF or an EPUB is full of them, they cannot be seen, and a DOI
carrying one does not resolve. They are dropped before anything is read.
The largest book in the corpus is a collection of printed web pages, and
every page carries what the browser put at its foot:
http://www.u.arizona.edu/~chalmers/biblio/1.html (54 of 100) [4/7/2002 1:48:08 PM]
That was a reference with a URL and nothing else, one per page, and it is why
that book wanted a model call on all 1,114 of its chunks. An address followed
by which-page-of-how-many and a print time is furniture, not a citation.
`Spade 1980, 12.` and `Trentman 1970; Spade 1980.` are short cites: they point
at a full entry elsewhere in the same bibliography the way `ibid.` points at
the note above. The author and year they do give are kept and the title is not
asked after, because no reading of that line will produce one.
Put a real model behind the entries this could not read and looked at what it
gave back. Almost none of it was knowledge — it was punctuation already on the
page that the parser was walking past:
Goodwin, F.K. and Jamison, K.R. (1990). *Manicdepressive illness* Oxford
University Press, New York.
The title is in asterisks. It was being read as the journal, because the rule
said italics after an author-date opening are the journal — true of `Peirce
(1878). Plain title. *Popular Science Monthly*`, and false here. What decides
it is where the italics sit: at the head of the tail they are the work, after
the title they are what it appeared in.
Five more the model was covering for, all of them on the page:
* `de la Torre, Carlos` opens on a particle and is a name.
* `Tinbergen, n. 1953` has an initial the scan read in lower case.
* `Theories of Nationalism: A Critical Introduction, Basingstoke: Palgrave`
has two colons and the imprint is the last one, not the first.
* `36Armaud Puig i Tarrech` is a note number run into the name.
* `S. Marc Cohen is Professor of Philosophy at...` opens exactly as a
Chicago note does and is a contributor's biography, not a reference.
A title may quote something — `What's the Meaning of "This"?` — so only an
unbalanced quotation mark means the cut landed inside one. A question mark
ends a sentence, which is how that title was being lost. A trailing backslash
is a line break conversion left behind.
`Altmann, M. 1951, A Study of Behaviour. Sociometry 14, 351-354.` names an
author, a year, a journal and a page range, and has no bracketed year, no
`p.`, no `ed.` and no `Press,` in it — so the test chunkmd uses to spot a page
of notes says it cites nothing, and it was being dropped before it was read.
citefind adds its own test alongside, for the same reason it added its own
marker: chunkmd is deciding whether a chunk is a page of notes and can afford
to miss one entry in thirty, and this is deciding whether a reference exists.
Coverage of the model's citations rises to 84% on bibliographies.
The verse pattern now covers the forms a real page of them uses: a chapter and
verse with no book named, a note number glued into the middle of the run, a
chapter range, and a semicolon inside a verse list.
funkpower