recipes, examples: shelving a library by what its books printed

The suite could read a cataloguing block and could ask a register, and
nothing showed the two together — which is the only way either is worth
acting on. A title page alone gives up a running head when the scan
starts late; an ISBN alone returns a confidently wrong book when the
catalogue has been carrying the wrong one for years.

The example's title comparison folds to letters and digits rather than
comparing strings. A register and a title page differ over case, over a
curly apostrophe against a straight one, and over whether the subtitle
counts as part of the title. An example that called those disagreements
would teach the wrong lesson, and did until it was fixed.

It runs on this library and finds both of the cases worth finding: a file
called Document5 by -ducati996 that both sources agree is Russell's
Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book
about sculpture.

docs/recipes.md

@@ -6,6 +6,7 @@ this suite, because a real pipeline has a database at the end of it.
 
 - [A shelf of files, sorted before anything else touches it](#a-shelf-of-files-sorted-before-anything-else-touches-it)
 - [A catalogue out of what the files already say](#a-catalogue-out-of-what-the-files-already-say)
+- [Shelving a library by what its books printed](#shelving-a-library-by-what-its-books-printed)
 - [Asking a model, without this suite calling one](#asking-a-model-without-this-suite-calling-one)
 - [A scanned book into searchable passages](#a-scanned-book-into-searchable-passages)
 - [A collected edition into one record per work](#a-collected-edition-into-one-record-per-work)
@@ -98,6 +99,97 @@ with:
 langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv'
 ```
 
+## Shelving a library by what its books printed
+
+A catalogue's subjects come from wherever the file came from, which is to say
+from nowhere in particular. A book's own cataloguing block is the one place its
+subjects are stated rather than guessed — and stated in Library of Congress
+form, which is precise and useless for browsing:
+
+```
+$ colophon converted/hobbes.md | jq -r '.subjects[]'
+Hobbes, Thomas, 1588–1679. Leviathan
+Political science – Early works to 1800
+```
+
+Nobody has a shelf called that. Give `colophon` the list you actually file by
+and it files them for you:
+
+```
+$ cat shelf.txt
+Political Science = political science, politics
+Byzantine History = byzantium, byzantine, constantinople
+Theology = theology, christianity
+Ethics
+
+$ colophon --shelf shelf.txt converted/*.md | jq -c '{title, shelved, unshelved}'
+{"title":"Leviathan","shelved":["Political Science"],"unshelved":["Hobbes, Thomas, 1588–1679. Leviathan"]}
+{"title":"Christianity and Classical Culture","shelved":["Theology"],"unshelved":[]}
+```
+
+`unshelved` is the point of the exercise. A subject quietly dropped reads
+exactly like a book that printed none, and the difference decides whether you
+go looking for a heading your shelf is missing.
+
+**The title page is not evidence on its own.** A scan that begins mid-frontmatter
+gives up a running head, a copyright line, or the author's name where the title
+should be. What makes a printed record trustworthy is a second one agreeing,
+and the ISBN in the same block is how you get one:
+
+```sh
+colophon --shelf shelf.txt converted/*.md > printed.ndjson
+
+jq -r 'select(.isbns != null) | .isbns[0]' printed.ndjson \
+  | idnorm --quiet \
+  | bibfetch --polite you@example.org > register.ndjson
+```
+
+Now two sources that cannot have copied each other have spoken, and you can
+say which books are settled and which need a person:
+
+```sh
+jq -s -r '
+  (.[1] | map({key: .id, value: .}) | from_entries) as $reg
+  | .[0][]
+  | . as $printed
+  | ($reg[$printed.isbns[0] // ""] // {}) as $found
+  | [ $printed.title,
+      $found.title,
+      (if $printed.title and $found.title then "corroborated" else "one source" end)
+    ] | @tsv
+' printed.ndjson register.ndjson
+```
+
+```
+The First World War    The First World War: A Very Short Introduction   corroborated
+Document5              The Problems of Philosophy                       corroborated
+Born Fighting          —                                                one source
+```
+
+The second row is the case worth the whole recipe: a file named `Document5`
+whose author reads `-ducati996`, which the book itself and OpenLibrary agree
+is Russell's *The Problems of Philosophy*. The third is the honest failure —
+one source said something, nothing confirmed it, and it goes to a human.
+
+In process, where a pipeline actually lives:
+
+```python
+import catena
+
+shelf = open("shelf.txt").read()
+printed = catena.read_colophon(markdown, shelf=shelf)
+
+found = None
+if printed["isbns"]:
+    found = catena.fetch_identifier(printed["isbns"][0], polite="you@example.org")
+
+settled = bool(printed.get("title") and found and found.get("title"))
+```
+
+`catena.shelve(shelf, subjects)` files headings a catalogue already holds, by
+the same rules, so a record's existing subjects and its book's printed ones are
+judged alike.
+
 ## Asking a model, without this suite calling one
 
 A tool here may compose a prompt. It does not send one. That keeps every

examples/README.md

@@ -15,6 +15,7 @@ NASA broadcast, which is a work of the US government.
 | [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
 | [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
 | [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
+| [`python/shelve_a_library.py`](python/shelve_a_library.py) | What a book says about itself, shelved and checked against a register |
 | [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
 | [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
 | [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
@@ -83,6 +84,20 @@ python examples/python/book_to_citations.py
 pip install catena-corpus psycopg
 DATABASE_URL=postgresql://user@host/db \
     python examples/python/search_and_rerank.py "why did the empire decline"
+
+`shelve_a_library.py` is the metadata half: it reads the cataloguing block a
+book prints in its own front matter, files those subjects onto a closed list
+you supply, and asks the register that assigned the ISBN whether it agrees.
+
+```sh
+python examples/python/shelve_a_library.py converted/*.md
+```
+
+Neither source is worth acting on alone — a scan that starts mid-frontmatter
+gives up a running head where the title should be, and an ISBN a catalogue has
+been carrying wrongly returns a confidently wrong book. The example prints
+`corroborated` only when both said the same thing, and the disagreements are
+the interesting output.
 ```
 
 ## TypeScript

examples/python/shelve_a_library.py

@@ -0,0 +1,110 @@
+"""What a book says about itself, checked against what a register says back.
+
+A catalogue's subjects and titles come from wherever its files came from. Two
+things can correct that without a model: the cataloguing block the book prints
+in its own front matter, and the register that assigned its ISBN. Neither is
+trustworthy alone — a scan that starts mid-frontmatter yields a running head
+where the title should be, and an ISBN a catalogue has been carrying wrongly
+returns a confidently wrong book. Together they are worth acting on.
+
+    python examples/python/shelve_a_library.py converted/*.md
+
+With no arguments it reads a book from Project Gutenberg, which prints no
+cataloguing block at all — the honest common case, and worth seeing.
+"""
+
+import pathlib
+import re
+import sys
+import urllib.request
+
+import catena
+
+# The closed list this library is browsed by. `Heading = term, term` gives the
+# words that file a book under it; a heading on its own matches its own name.
+SHELF = """
+Philosophy = philosophy, philosophical
+Ancient Philosophy = ancient philosophy, plato, aristotle
+Political Science = political science, politics, government
+Theology = theology, christianity, religion
+Church History = church history, ecclesiastical history
+Byzantine History = byzantium, byzantine, constantinople
+History = history, historiography
+Literature = literature, fiction, poetry
+Psychology = psychology
+Ethics
+Logic
+"""
+
+GUTENBERG = "https://www.gutenberg.org/cache/epub/5827/pg5827.txt"
+
+
+def sample(into: pathlib.Path) -> str:
+    """Locke's *Essay*, which Gutenberg strips the front matter from."""
+    book = into / "locke.txt"
+    if not book.exists():
+        into.mkdir(parents=True, exist_ok=True)
+        print(f"fetching {GUTENBERG}", file=sys.stderr)
+        with urllib.request.urlopen(GUTENBERG, timeout=120) as reply:  # noqa: S310
+            book.write_bytes(reply.read())
+    return book.read_text(encoding="utf-8", errors="replace")
+
+
+def same(a: str, b: str) -> bool:
+    """Whether two printings of one title are the same title.
+
+    Compared on letters and digits alone. A register and a title page differ
+    over case, over a curly apostrophe against a straight one, and over
+    whether the subtitle is part of the title — none of which is a
+    disagreement, and all of which a plain string comparison calls one.
+    """
+    fold = lambda s: re.sub(r"[^a-z0-9]+", "", (s or "").lower())
+    a, b = fold(a), fold(b)
+    return bool(a) and bool(b) and (a in b or b in a)
+
+
+def look(text: str, name: str) -> None:
+    printed = catena.read_colophon(text, shelf=SHELF)
+
+    title = printed.get("title")
+    isbns = printed.get("isbns") or []
+    print(f"\n{name}")
+    print(f"  printed title   {title or '— (no title page in these pages)'}")
+    if printed.get("authors"):
+        print(f"  printed authors {', '.join(printed['authors'])}")
+    if printed.get("shelved"):
+        print(f"  shelved under   {', '.join(printed['shelved'])}")
+    for raw in printed.get("unshelved", []):
+        # Not a failure: a heading the shelf has no place for is how you find
+        # out the shelf is missing one.
+        print(f"  no shelf for    {raw}")
+    if not printed.get("subjects"):
+        print("  subjects        — (no cataloguing block printed)")
+
+    if not isbns:
+        print("  register        — (nothing to look up; one source only)")
+        return
+
+    # The only call here that opens a socket, and it blocks.
+    found = catena.fetch_identifier(isbns[0], polite="you@example.org")
+    if not found or not found.get("found"):
+        print(f"  register        {isbns[0]} is unknown to every register asked")
+        return
+
+    print(f"  register says   {found['title']} / {', '.join(found.get('authors') or [])}")
+    agrees = bool(title) and same(title, found["title"])
+    print(f"  verdict         {'corroborated' if agrees else 'two sources disagree — ask a person'}")
+
+
+def main() -> int:
+    paths = [pathlib.Path(p) for p in sys.argv[1:]]
+    if not paths:
+        look(sample(pathlib.Path("/tmp/catena-example")), "Locke, from Gutenberg")
+        return 0
+    for path in paths:
+        look(path.read_text(encoding="utf-8", errors="replace"), path.name)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())