Measured against the citations an LLM had already extracted from 40 books of
the same corpus — 19,340 of them over 3,424 chunks — and improved where the
comparison showed a real gap rather than a difference of task.
On pages of notes, where most citations live: year agreement with the model
44% -> 56%, title 18% -> 23%, author 16% -> 24%.
What changed:
- A note listing several works, semicolon-separated, becomes one record per
work. The note number is kept on all of them.
- A full stop is not a separator. It parts the author from the title in every
bibliography entry there is.
- A page of body text with notes at its foot no longer reads the body as the
first note.
- A run carrying a page reference or a semicolon is not reported as a title.
- Six sentences with no imprint, no page and no identifier is a remark about
the argument, not a reference.
- A translator standing where the journal goes is a credit, not a container,
and `(London, 1973)` is a place, not a publisher.
The model is not ground truth and the comparison is not a score: on several
sampled disagreements it named the journal where the article was wanted, or
the collected works instead of the essay.
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.
Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.
`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.
chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
The hashed bag of words asks only whether two stretches repeat the same
words, and cannot tell that ships and vessels are one subject. An embedding
can. segment_with takes one vector per window and falls back to the words
when it is given none, or the wrong number of them — a caller that embedded
some other recording's windows is asking about that recording.
windows_of gives the text to embed, in the order the vectors come back in,
so nobody has to reimplement the windowing pass to use this.
Bound in all three languages, and on the command line as --windows and
--embeddings, which pipe through emb. The C ABI version goes to 2:
catena_segment_cues gained an argument.
funkpower