Commit Graph
12 Commits
Author SHA1 Message Date
funkpower 1387b1a62b chunkmd: a reading list with nothing marking it is citation material
Block, N. 1981. Psychologism and behaviorism. Philosophical Review 90:5-43.
    Dennett, D. C. 1987. The Intentional Stance. MIT Press.
    Geach, P. 1957. Mental Acts. Routledge and Kegan Paul.

That is how a reader or a companion sets the further-reading list at the end
of each chapter: indented under the last paragraph of the essay, with no
heading over it and no numbers beside it. Nothing announced it as a
bibliography, so the chunk stayed body text with no flag, the citation stage
never selected it, and a book made of such chapters came out with one or two
citations. *The Cambridge Companion to Ancient Greek Political Thought* had
three; Chalmers's *Philosophy of Mind* reader had one.

The hard part is that an index line opens identically — `Newton, Isaac, 5,
93, 141` — and a page of them would otherwise flood the table. What tells
them apart is that a reference names a work: five words of three letters or
more on the entry's own line, and not two page numbers written as links.

Over 15,925 unflagged body chunks sampled from the corpus this fires on 2.6%,
and seven of eight sampled were real reference lists.

Recorded in parity/KNOWN.md, and the harness allows this one difference by
name: a chunk the Rust flags and the Python does not, identical in every
other field. A flag the Python sets and the Rust does not is still a failure.
2026-09-20 16:58:07 -04:00
funkpower 1284136f69 chunkmd, citefind: the back-reference marker a whole class of EPUBs leaves
\[←788 \]
    \[[←193](index_split_011.xhtml#citation-193) \]

An arrow pointing back to where the note was called, sometimes a link and
sometimes not, inside escaped brackets. Neither pattern knew it, so a page of
notes set this way read as body prose: the chunk was never flagged, the
citation stage never selected it, and the book came out with nothing.

Ten books of the corpus use it and eight of them had zero citations —
including both volumes of *The Decline of the West*, whose 830 endnotes are
exactly what this is for. 142 of the 199 affected chunks are those two books.

Recorded in parity/KNOWN.md: the Python this was ported from did not know the
marker either, and no fixture carries one, so the suite agrees either way.
2026-09-20 16:06:22 -04:00
funkpower c071f84829 citefind: the references in a bibliography, and what could not be read off them
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.

Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.

`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.

chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
2026-09-20 10:38:48 -04:00
funkpower f9f809f701 chunkmd: the half of a word boundary that applies before a digit 2026-09-20 07:18:34 -04:00
funkpower a80f8a024d chunkmd: say rfind, which is what this was doing 2026-09-19 17:36:18 -04:00
funkpower 9fd654ad5f docs: the manual-page flag, and the records that belong to one tool 2026-09-19 17:06:17 -04:00
funkpower ea0a3694b1 mdsplit: the works inside one file, and the books inside scripture 2026-09-19 14:58:54 -04:00
funkpower 7fa610c578 each crate carries its own readme, and the contract describes rather than instructs 2026-09-19 14:15:01 -04:00
funkpower 1071f0bed3 booksniff, and one place for the parts every tool shares 2026-09-19 12:40:20 -04:00
funkpower 339f402610 ported patterns mean in rust what they mean in python 2026-09-19 12:26:41 -04:00
funkpower 1b3c002b3d mdscrub: residual html, control characters, headings, page markers 2026-09-19 11:30:05 -04:00
funkpower e0473fb8ae chunkmd: page and chapter aware chunking, roles, footnote flag 2026-09-19 11:22:56 -04:00