docs: what citefind reads, and a citation graph built out of the books

The README now lists the forms it handles with an example of each, says that where the italics sit is what decides whether they are the title or the journal, and sets out what it refuses and why — scripture, Migne, a contributor's biography, a note that only thanks people, a page footer. Recipes gains "What a book leans on": one bibliography read, then the same over a shelf, counted per citing book so a work cited forty times in one bibliography still counts once. Every command was run — thirty converted books gave 14,481 references, 10,536 naming a work, in about a second. `records.md` now points at the five tools whose record shapes live in their own pages rather than in `corpus-core`. And pandoc escapes an underscore, so the rule of them that stands in for a repeated author arrives as `\_\_\_\_\_` and was not being recognised.
2026-10-22 18:57:14 +00:00 · 2026-09-20 14:56:13 -04:00
parent fbdf8f6333
commit 2dfa370cd7
4 changed files with 175 additions and 25 deletions
@@ -139,18 +139,14 @@ interpretation of it and somebody will want to check.
 ## How it cuts

 **A page of notes** is cut at its note numbers, in any of the shapes
-conversion leaves them: `12.`, `[12]`, `fn39.`, a superscript, or a
-back-reference link. The pattern is the one
+conversion leaves them: `12.`, `[12]`, `fn39.`, a superscript, a number run
+straight into the text, or a back-reference link. The pattern is the one
 [`chunkmd`](../chunkmd/README.md) uses to recognise a page of notes, so the
-two cannot drift apart, with one form added — a link pointing at another file
-rather than at a fragment. `chunkmd` is deciding whether a chunk is a page of
-notes and can afford to miss one marker in thirty; here a missed marker runs
-two references together.
-
-A note that cites nothing is dropped. A page of notes mixes bibliographic
-notes with discursive ones, and a remark about the argument is not a
-reference — and one that runs on for six sentences with no imprint, no page
-reference and no identifier is a remark whatever else it mentions.
+two cannot drift apart, with two forms added — a link pointing at another
+file rather than at a fragment, and an author-date entry carrying none of the
+punctuation `chunkmd` looks for. `chunkmd` is deciding whether a chunk is a
+page of notes and can afford to miss one marker in thirty; here a missed
+marker runs two references together.

 A note that cites **several works** becomes several records, cut at the
 semicolons that separate them, each keeping the note's number. A full stop
@@ -159,7 +155,8 @@ from the title in every bibliography entry ever written.

 Where the first note number sits a long way into the block, what precedes it
 is the body of the page rather than a note that lost its number, and it is
-left out.
+left out. The copyright line a digitised book prints at the foot of every
+page is cut off the end of whatever note it landed on.

 **A bibliography** numbers nothing, so the signal is layout: a blank line
 between entries, a hanging indent under each, or the surname-comma opening
@@ -168,32 +165,61 @@ is the one used. When it carries none of them the fallback is one entry per
 line, which is right for a list that was never wrapped and visibly wrong for
 one that was.

+## The forms it reads
+
+| Form | Example |
+|---|---|
+| Bibliography | `Smith, John. *The Making of the Second Estate*. Oxford: Clarendon Press, 1998.` |
+| Author-date | `Peirce, C. S. (1878). How to make our ideas clear. *Popular Science Monthly*, 12, 286-302.` |
+| Author-date, no brackets | `Altmann, M. 1951, A Study of Behaviour. Sociometry 14, 351-354.` |
+| Chicago note | `John MacGowan, The Imperial History of China (1897), p. 154.` |
+| Chicago short note | `Unger, Dutch Shipbuilding, 33-4` |
+| Commas only | `Miller, David, On Nationality, Oxford: Clarendon Press, 1995` |
+| Commas and a place | `Cheyne, T. K., Founders of Old Testament Criticism, London, 1893.` |
+| Chapter in a book | `Brown, Peter. "Sorcery…" In *Witchcraft Confessions*, ed. Mary Douglas, 17-45. London: Tavistock, 1970.` |
+
+**Where the italics sit decides what they are.** In `Peirce (1878). Plain
+title. *Popular Science Monthly*` they come after the title and name the
+journal; in `Goffman (1963). *Stigma* Penguin` they open the tail and are the
+title. The same goes for quotation marks.
+
+A rule of dashes where the author should be means the author of the entry
+above — the style says so, and the entry it refers to is the one immediately
+before — so it is filled in.
+
 ## What it will not do

 **Resolve `ibid.`** A continuation refers to whatever was cited before it,
 which is a fact about the surrounding page and not about the entry. It is
-flagged `continuation` and left alone, and it is left out of `--prompt`
-entirely — there is nothing in it to ask about.
+flagged `continuation` and left alone, and left out of `--prompt` entirely.
+
+The same goes for a **short cite**: `Spade 1980, 12.` and `Lowrie and
+Fletcher, p. 133` point at a full entry elsewhere in the same bibliography.
+The author and year they do give are kept; no reading of that line will
+produce a title.

 **Report a title it did not find.** An entry held together by commas alone,
 with no names to anchor it, yields no title rather than its own text echoed
 back. Nor does a run carrying a page reference or a semicolon count as one:
-that is two entries run together, or a reference to a work rather than the
-name of one.
+that is two entries run together. A quotation mark with no partner means the
+cut landed inside a title — unless it is the last character, which is a
+stray an opening mark was lost from.

 **Take chapter and verse for a work.** `Matthew 4:17` and `Cf. Deuteronomy
-18:15-19; Isaiah 53` are scripture references, which is most of what the notes
-of a theological book are. Nobody is looking for the publisher of Galatians,
-so they are not references and no record is written for them.
+18:15-19; Isaiah 53` are scripture references, which is most of what the
+notes of a theological book are. So is `PG 4: 264` — Migne, and CSEL, NPNF,
+SC and the rest are shelves, not works, and the work is not named there.

 **Name a place of publication as the publisher.** `(London, 1973)` says where
 a book was published and not by whom, so `container` stays empty. The place
 is still in `text`.

-A rule of dashes where the author should be is the exception, and a
-deliberate one: `———. *The Second Book*.` means the author of the entry above,
-the style says so, and the entry it refers to is the one immediately before.
-That is read and filled in.
+**Mistake the back matter for the bibliography.** A contributor's biography
+opens exactly as a Chicago note does — `S. Marc Cohen is Professor of…` — and
+a note that opens by thanking people cites nothing however many names it
+carries. Neither is a reference. Nor is a page footer a browser printed, nor
+a note that runs on for six sentences with no imprint, no page reference and
+no identifier.

 ## As a library

@@ -322,8 +322,11 @@ static CITATION_SIGNAL: LazyLock<Regex> = LazyLock::new(|| {

 /// A rule of dashes standing in for the author of the entry above, which is
 /// how every style avoids printing the same name forty times running.
-static REPEATED_AUTHOR: LazyLock<Regex> =
-    LazyLock::new(|| Regex::new(r"^\s*(?:[—–]+|-{3,}|_{3,})").expect("a valid pattern"));
+static REPEATED_AUTHOR: LazyLock<Regex> = LazyLock::new(|| {
+    // Pandoc escapes an underscore, so a rule of them arrives as
+    // `\_\_\_\_\_` rather than `_____`.
+    Regex::new(r"^\s*(?:[—–]+|-{3,}|(?:\\?_){3,})").expect("a valid pattern")
+});

 /// Whether an entry defers to the one above it for its author.
 pub fn repeats_author(text: &str) -> bool {
@@ -1688,6 +1691,16 @@ mod tests {
        assert!(several.continuation);
        assert_eq!(several.authors, ["Trentman", "Spade"]);

+        // Pandoc escapes an underscore, so a rule of them arrives escaped.
+        let escaped = read(
+            "\\_\\_\\_\\_\\_. *The Community of the Beloved Disciple*. New York: Paulist, 1979.",
+        );
+        assert_eq!(
+            escaped.title.as_deref(),
+            Some("The Community of the Beloved Disciple")
+        );
+        assert!(repeats_author(&escaped.text));
+
        // One name, two years: `Spade 1975 and 1980.`
        let years = read("Spade 1975 and 1980.");
        assert!(years.continuation);
@@ -214,6 +214,109 @@ $ pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd --min-tokens 10
 Every chunk carries the page it came from and the chapter it sits under, which
 is what lets a search result cite itself.

+## What a book leans on
+
+The bibliography is the one part of a book that is already structured. Every
+entry names a work, and most name it in one of a handful of shapes that have
+not changed in a century — so `citefind` reads it without asking anybody
+anything.
+
+```
+$ citefind bibliography.md | head -1
+{"v":1,"idx":0,"text":"Abogunrin, S. O. “The Three Variant Accounts of Peter’s Call: A Critical and Theological Examination of the Texts.” *New Testament Studies* 31 (1985): 587–602.","marker":null,"authors":["Abogunrin, S. O."],"year":1985,"title":"The Three Variant Accounts of Peter’s Call: A Critical and Theological Examination of the Texts","container":"New Testament Studies","pages":"587–602","ids":[],"kind":"article","continuation":false,"unsettled":[]}
+```
+
+Whose work a book rests on, counted:
+
+```sh
+citefind bibliography.md \
+  | jq -r 'select(.authors | length > 0) | .authors[0]' \
+  | sort | uniq -c | sort -rn | head
+```
+
+```
+  64 Yates
+  26 Quigley
+  23 Bulgakov, Sergius
+  17 Lorenz, K.
+```
+
+What it reads, and what it reaches for:
+
+```sh
+citefind bibliography.md | jq -r .kind | sort | uniq -c | sort -rn
+```
+
+```
+4200 other
+3356 book
+ 808 chapter
+ 700 article
+ 300 web
+```
+
+The identifiers it finds, checked and resolved against a register:
+
+```sh
+citefind bibliography.md \
+  | jq -r '.ids[]? | select(.kind == "doi") | .id' \
+  | idnorm | bibfetch
+```
+
+### Across a whole shelf
+
+Every book's bibliography, cut out of the converted markdown and read:
+
+```sh
+for f in converted/*.md; do
+  awk '/^#+ *(Bibliography|References|Works Cited|Notes)/{on=1} on' "$f" \
+    | citefind 2>/dev/null \
+    | jq -c --arg from "$(basename "$f" .md)" '. + {citing: $from}'
+done > citations.ndjson
+```
+
+Over thirty converted books that was 14,481 references, 10,536 of them naming
+a work, in about a second. A book whose heading matched but whose section was
+empty exits `1` and says so on stderr, which is why the loop sends it
+nowhere.
+
+Which works the shelf agrees on — cited by the most different books:
+
+```sh
+jq -r 'select(.title) | [.citing, (.title | ascii_downcase)] | @tsv' citations.ndjson \
+  | sort -u | cut -f2 | uniq -c | sort -rn | head
+```
+
+Counted per citing book rather than per entry, so a work cited forty times in
+one bibliography still counts once. On a shelf whose books share a subject
+this is the spine of the field; on one that does not, it is mostly ones.
+
+That is a citation graph, built out of the books themselves, with nothing
+running but a parser.
+
+### What it could not read
+
+Every record names the fields it could not settle. A bibliography set in a
+style the parser does not know, or a chunk that began part-way through an
+entry, shows up here rather than as a confident wrong answer:
+
+```sh
+citefind bibliography.md | jq -r 'select(.unsettled | length > 0) | .text'
+```
+
+`--prompt` writes a question about exactly those entries and no others, for a
+model of your choosing. Nothing is sent from here:
+
+```sh
+citefind --prompt bibliography.md | ollama run qwen3:8b > reply.json
+citefind --merge reply.json bibliography.md
+```
+
+A reply fills in only the fields that were asked about; a model that
+volunteers a better title for an entry whose title was read off the page is
+overruled. Most bibliographies have nothing to ask about, and `--prompt`
+exits `1` and writes nothing when that is so.
+
 ## A collected edition into one record per work

 Forty books in one file, indexed whole, means a query for any of them returns a
@@ -111,6 +111,14 @@ has a shape of its own, described where that tool is:
  and the call that would do it.
 - [`pdfgutter`](../crates/pdfgutter/README.md) — whether a scan is two pages
  to a sheet, and what each sampled page measured.
+- [`citefind`](../crates/citefind/README.md) — one reference out of a
+  bibliography, and the fields it could not read off the entry.
+- [`bookmeta`](../crates/bookmeta/README.md) — what a file says about itself.
+- [`tocfind`](../crates/tocfind/README.md) — one entry of a table of contents.
+- [`seriesof`](../crates/seriesof/README.md) — what a title says about its
+  series.
+- [`idnorm`](../crates/idnorm/README.md) — an identifier as it arrived and as
+  it should be stored.

 ## Getting the schemas