read_colophon is several milliseconds of computation over a few hundred
kilobytes, and it held the GIL for all of them. A caller reading a
library through asyncio.to_thread therefore got no overlap whatsoever:
six books in flight, one book's worth of work at a time.
Measured on this library, that was most of why a concurrent pass still
paced at four and a half hours.
colophon has composed the question about what its pages did not settle
since it was written, and merged a reply back into the record it read.
Both were reachable only from a terminal, so a pipeline wanting the same
fallback would have had to write its own prompt — a second one, drifting
from the first, asking for a shape merge does not accept.
merge fills only the unsettled fields, so a model cannot overwrite what a
book printed about itself. That is the property that makes asking one
safe, and it belongs to colophon rather than to any caller.
book-pipeline reaches for colophon and bibfetch on every book it repairs.
Shelling out would mean a process per book per tool across five thousand
books, and parsing back the JSON it is about to parse anyway.
bibfetch's fetch loop lived in main.rs, so binding it meant either
lifting it or writing a second one. A second implementation of 'which
register answers about this kind of identifier, and in what order' would
drift from the first, and the copy nobody runs from a terminal would
drift silently. It is now bibfetch::ask, and the command line calls it
too — verified by running the CLI before and after and diffing.
fetch_identifier releases the GIL around the request. It is the only
function here that opens a socket, and holding the interpreter for the
length of an HTTP call would stop every other thread in it.
bibfetch is optional and only the python feature turns it on: its HTTP
client does not build for wasm32, and an unconditional dependency would
have taken the JavaScript package down with it.
bibclean is "catalogue records audited and repaired, with a reason for every
change", and book-pipeline's metadata module was a second implementation of
the same thing — ported from it in the first place. Twelve functions bound:
the audit, its severity, the two commonest faults, the three repairs that
need no lookup, ISBNs checked and found, and whether two titles name the same
work.
Checked against that Python across all 5,385 books in the library — every
title, every author, every ISBN, every filename, and four thousand title
pairs. Identical on all of it but one: `The Soul After Death: Contemporary
'After-Death' Experiences`, where the Python lowercased a word inside the
quotation marks and bibclean leaves it alone.
book-pipeline held its own copy of the scheme operations bibclean already
had. The scheme itself — which headings a collection uses, their aliases,
their parents — stays with the collection, because that is the most
collection-specific thing there is. What a scheme *means* is the same
question everywhere, and this answers it: `canonical_subject`,
`compose_subjects`, `needs_subject`.
Checked against the Python over the library's 228 distinct tags and all
5,385 books: identical on every one.
book-pipeline had its own copy of bibclean's author recovery. Bound on all
four surfaces so it can stop.
Checked against that Python over all 5,385 books in the library: 5,375
identical. On the other ten the Python had taken a translator, an illustrator
or the publisher off a copyright line — Sally-Ann Spencer for *The Dwarves*,
Thomas Nelson for a table of contents, C. S. Lewis for a book about Tolkien —
and bibclean either finds the real author in the Library of Congress block or
says nothing. Every one of the ten was read by hand.
`looks_like_footnote_citations` is the function chunkmd's `looks_like_citations`
was ported from, and book-pipeline was still running the Python original at
four call sites. Bound on all four surfaces so it can stop.
Checked against the Python over 34,458 chunks of the live corpus: no
disagreement.
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.
Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.
`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.
chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
The hashed bag of words asks only whether two stretches repeat the same
words, and cannot tell that ships and vessels are one subject. An embedding
can. segment_with takes one vector per window and falls back to the words
when it is given none, or the wrong number of them — a caller that embedded
some other recording's windows is asking about that recording.
windows_of gives the text to embed, in the order the vectors come back in,
so nobody has to reimplement the windowing pass to use this.
Bound in all three languages, and on the command line as --windows and
--embeddings, which pipe through emb. The C ABI version goes to 2:
catena_segment_cues gained an argument.
funkpower