diff --git a/crates/citefind/README.md b/crates/citefind/README.md index db55c09..b5b5fad 100644 --- a/crates/citefind/README.md +++ b/crates/citefind/README.md @@ -139,18 +139,14 @@ interpretation of it and somebody will want to check. ## How it cuts **A page of notes** is cut at its note numbers, in any of the shapes -conversion leaves them: `12.`, `[12]`, `fn39.`, a superscript, or a -back-reference link. The pattern is the one +conversion leaves them: `12.`, `[12]`, `fn39.`, a superscript, a number run +straight into the text, or a back-reference link. The pattern is the one [`chunkmd`](../chunkmd/README.md) uses to recognise a page of notes, so the -two cannot drift apart, with one form added — a link pointing at another file -rather than at a fragment. `chunkmd` is deciding whether a chunk is a page of -notes and can afford to miss one marker in thirty; here a missed marker runs -two references together. - -A note that cites nothing is dropped. A page of notes mixes bibliographic -notes with discursive ones, and a remark about the argument is not a -reference — and one that runs on for six sentences with no imprint, no page -reference and no identifier is a remark whatever else it mentions. +two cannot drift apart, with two forms added — a link pointing at another +file rather than at a fragment, and an author-date entry carrying none of the +punctuation `chunkmd` looks for. `chunkmd` is deciding whether a chunk is a +page of notes and can afford to miss one marker in thirty; here a missed +marker runs two references together. A note that cites **several works** becomes several records, cut at the semicolons that separate them, each keeping the note's number. A full stop @@ -159,7 +155,8 @@ from the title in every bibliography entry ever written. Where the first note number sits a long way into the block, what precedes it is the body of the page rather than a note that lost its number, and it is -left out. +left out. The copyright line a digitised book prints at the foot of every +page is cut off the end of whatever note it landed on. **A bibliography** numbers nothing, so the signal is layout: a blank line between entries, a hanging indent under each, or the surname-comma opening @@ -168,32 +165,61 @@ is the one used. When it carries none of them the fallback is one entry per line, which is right for a list that was never wrapped and visibly wrong for one that was. +## The forms it reads + +| Form | Example | +|---|---| +| Bibliography | `Smith, John. *The Making of the Second Estate*. Oxford: Clarendon Press, 1998.` | +| Author-date | `Peirce, C. S. (1878). How to make our ideas clear. *Popular Science Monthly*, 12, 286-302.` | +| Author-date, no brackets | `Altmann, M. 1951, A Study of Behaviour. Sociometry 14, 351-354.` | +| Chicago note | `John MacGowan, The Imperial History of China (1897), p. 154.` | +| Chicago short note | `Unger, Dutch Shipbuilding, 33-4` | +| Commas only | `Miller, David, On Nationality, Oxford: Clarendon Press, 1995` | +| Commas and a place | `Cheyne, T. K., Founders of Old Testament Criticism, London, 1893.` | +| Chapter in a book | `Brown, Peter. "Sorcery…" In *Witchcraft Confessions*, ed. Mary Douglas, 17-45. London: Tavistock, 1970.` | + +**Where the italics sit decides what they are.** In `Peirce (1878). Plain +title. *Popular Science Monthly*` they come after the title and name the +journal; in `Goffman (1963). *Stigma* Penguin` they open the tail and are the +title. The same goes for quotation marks. + +A rule of dashes where the author should be means the author of the entry +above — the style says so, and the entry it refers to is the one immediately +before — so it is filled in. + ## What it will not do **Resolve `ibid.`** A continuation refers to whatever was cited before it, which is a fact about the surrounding page and not about the entry. It is -flagged `continuation` and left alone, and it is left out of `--prompt` -entirely — there is nothing in it to ask about. +flagged `continuation` and left alone, and left out of `--prompt` entirely. + +The same goes for a **short cite**: `Spade 1980, 12.` and `Lowrie and +Fletcher, p. 133` point at a full entry elsewhere in the same bibliography. +The author and year they do give are kept; no reading of that line will +produce a title. **Report a title it did not find.** An entry held together by commas alone, with no names to anchor it, yields no title rather than its own text echoed back. Nor does a run carrying a page reference or a semicolon count as one: -that is two entries run together, or a reference to a work rather than the -name of one. +that is two entries run together. A quotation mark with no partner means the +cut landed inside a title — unless it is the last character, which is a +stray an opening mark was lost from. **Take chapter and verse for a work.** `Matthew 4:17` and `Cf. Deuteronomy -18:15-19; Isaiah 53` are scripture references, which is most of what the notes -of a theological book are. Nobody is looking for the publisher of Galatians, -so they are not references and no record is written for them. +18:15-19; Isaiah 53` are scripture references, which is most of what the +notes of a theological book are. So is `PG 4: 264` — Migne, and CSEL, NPNF, +SC and the rest are shelves, not works, and the work is not named there. **Name a place of publication as the publisher.** `(London, 1973)` says where a book was published and not by whom, so `container` stays empty. The place is still in `text`. -A rule of dashes where the author should be is the exception, and a -deliberate one: `———. *The Second Book*.` means the author of the entry above, -the style says so, and the entry it refers to is the one immediately before. -That is read and filled in. +**Mistake the back matter for the bibliography.** A contributor's biography +opens exactly as a Chicago note does — `S. Marc Cohen is Professor of…` — and +a note that opens by thanking people cites nothing however many names it +carries. Neither is a reference. Nor is a page footer a browser printed, nor +a note that runs on for six sentences with no imprint, no page reference and +no identifier. ## As a library diff --git a/crates/citefind/src/parse.rs b/crates/citefind/src/parse.rs index f334f48..be7c6b7 100644 --- a/crates/citefind/src/parse.rs +++ b/crates/citefind/src/parse.rs @@ -322,8 +322,11 @@ static CITATION_SIGNAL: LazyLock = LazyLock::new(|| { /// A rule of dashes standing in for the author of the entry above, which is /// how every style avoids printing the same name forty times running. -static REPEATED_AUTHOR: LazyLock = - LazyLock::new(|| Regex::new(r"^\s*(?:[—–]+|-{3,}|_{3,})").expect("a valid pattern")); +static REPEATED_AUTHOR: LazyLock = LazyLock::new(|| { + // Pandoc escapes an underscore, so a rule of them arrives as + // `\_\_\_\_\_` rather than `_____`. + Regex::new(r"^\s*(?:[—–]+|-{3,}|(?:\\?_){3,})").expect("a valid pattern") +}); /// Whether an entry defers to the one above it for its author. pub fn repeats_author(text: &str) -> bool { @@ -1688,6 +1691,16 @@ mod tests { assert!(several.continuation); assert_eq!(several.authors, ["Trentman", "Spade"]); + // Pandoc escapes an underscore, so a rule of them arrives escaped. + let escaped = read( + "\\_\\_\\_\\_\\_. *The Community of the Beloved Disciple*. New York: Paulist, 1979.", + ); + assert_eq!( + escaped.title.as_deref(), + Some("The Community of the Beloved Disciple") + ); + assert!(repeats_author(&escaped.text)); + // One name, two years: `Spade 1975 and 1980.` let years = read("Spade 1975 and 1980."); assert!(years.continuation); diff --git a/docs/recipes.md b/docs/recipes.md index b990ebe..e85a7df 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -214,6 +214,109 @@ $ pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd --min-tokens 10 Every chunk carries the page it came from and the chapter it sits under, which is what lets a search result cite itself. +## What a book leans on + +The bibliography is the one part of a book that is already structured. Every +entry names a work, and most name it in one of a handful of shapes that have +not changed in a century — so `citefind` reads it without asking anybody +anything. + +``` +$ citefind bibliography.md | head -1 +{"v":1,"idx":0,"text":"Abogunrin, S. O. “The Three Variant Accounts of Peter’s Call: A Critical and Theological Examination of the Texts.” *New Testament Studies* 31 (1985): 587–602.","marker":null,"authors":["Abogunrin, S. O."],"year":1985,"title":"The Three Variant Accounts of Peter’s Call: A Critical and Theological Examination of the Texts","container":"New Testament Studies","pages":"587–602","ids":[],"kind":"article","continuation":false,"unsettled":[]} +``` + +Whose work a book rests on, counted: + +```sh +citefind bibliography.md \ + | jq -r 'select(.authors | length > 0) | .authors[0]' \ + | sort | uniq -c | sort -rn | head +``` + +``` + 64 Yates + 26 Quigley + 23 Bulgakov, Sergius + 17 Lorenz, K. +``` + +What it reads, and what it reaches for: + +```sh +citefind bibliography.md | jq -r .kind | sort | uniq -c | sort -rn +``` + +``` +4200 other +3356 book + 808 chapter + 700 article + 300 web +``` + +The identifiers it finds, checked and resolved against a register: + +```sh +citefind bibliography.md \ + | jq -r '.ids[]? | select(.kind == "doi") | .id' \ + | idnorm | bibfetch +``` + +### Across a whole shelf + +Every book's bibliography, cut out of the converted markdown and read: + +```sh +for f in converted/*.md; do + awk '/^#+ *(Bibliography|References|Works Cited|Notes)/{on=1} on' "$f" \ + | citefind 2>/dev/null \ + | jq -c --arg from "$(basename "$f" .md)" '. + {citing: $from}' +done > citations.ndjson +``` + +Over thirty converted books that was 14,481 references, 10,536 of them naming +a work, in about a second. A book whose heading matched but whose section was +empty exits `1` and says so on stderr, which is why the loop sends it +nowhere. + +Which works the shelf agrees on — cited by the most different books: + +```sh +jq -r 'select(.title) | [.citing, (.title | ascii_downcase)] | @tsv' citations.ndjson \ + | sort -u | cut -f2 | uniq -c | sort -rn | head +``` + +Counted per citing book rather than per entry, so a work cited forty times in +one bibliography still counts once. On a shelf whose books share a subject +this is the spine of the field; on one that does not, it is mostly ones. + +That is a citation graph, built out of the books themselves, with nothing +running but a parser. + +### What it could not read + +Every record names the fields it could not settle. A bibliography set in a +style the parser does not know, or a chunk that began part-way through an +entry, shows up here rather than as a confident wrong answer: + +```sh +citefind bibliography.md | jq -r 'select(.unsettled | length > 0) | .text' +``` + +`--prompt` writes a question about exactly those entries and no others, for a +model of your choosing. Nothing is sent from here: + +```sh +citefind --prompt bibliography.md | ollama run qwen3:8b > reply.json +citefind --merge reply.json bibliography.md +``` + +A reply fills in only the fields that were asked about; a model that +volunteers a better title for an entry whose title was read off the page is +overruled. Most bibliographies have nothing to ask about, and `--prompt` +exits `1` and writes nothing when that is so. + ## A collected edition into one record per work Forty books in one file, indexed whole, means a query for any of them returns a diff --git a/docs/records.md b/docs/records.md index e150c54..e84f722 100644 --- a/docs/records.md +++ b/docs/records.md @@ -111,6 +111,14 @@ has a shape of its own, described where that tool is: and the call that would do it. - [`pdfgutter`](../crates/pdfgutter/README.md) — whether a scan is two pages to a sheet, and what each sampled page measured. +- [`citefind`](../crates/citefind/README.md) — one reference out of a + bibliography, and the fields it could not read off the entry. +- [`bookmeta`](../crates/bookmeta/README.md) — what a file says about itself. +- [`tocfind`](../crates/tocfind/README.md) — one entry of a table of contents. +- [`seriesof`](../crates/seriesof/README.md) — what a title says about its + series. +- [`idnorm`](../crates/idnorm/README.md) — an identifier as it arrived and as + it should be stored. ## Getting the schemas