diff --git a/docs/recipes.md b/docs/recipes.md index e85a7df..9abf456 100644 --- a/docs/recipes.md +++ b/docs/recipes.md @@ -6,6 +6,7 @@ this suite, because a real pipeline has a database at the end of it. - [A shelf of files, sorted before anything else touches it](#a-shelf-of-files-sorted-before-anything-else-touches-it) - [A catalogue out of what the files already say](#a-catalogue-out-of-what-the-files-already-say) +- [Shelving a library by what its books printed](#shelving-a-library-by-what-its-books-printed) - [Asking a model, without this suite calling one](#asking-a-model-without-this-suite-calling-one) - [A scanned book into searchable passages](#a-scanned-book-into-searchable-passages) - [A collected edition into one record per work](#a-collected-edition-into-one-record-per-work) @@ -98,6 +99,97 @@ with: langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv' ``` +## Shelving a library by what its books printed + +A catalogue's subjects come from wherever the file came from, which is to say +from nowhere in particular. A book's own cataloguing block is the one place its +subjects are stated rather than guessed — and stated in Library of Congress +form, which is precise and useless for browsing: + +``` +$ colophon converted/hobbes.md | jq -r '.subjects[]' +Hobbes, Thomas, 1588–1679. Leviathan +Political science – Early works to 1800 +``` + +Nobody has a shelf called that. Give `colophon` the list you actually file by +and it files them for you: + +``` +$ cat shelf.txt +Political Science = political science, politics +Byzantine History = byzantium, byzantine, constantinople +Theology = theology, christianity +Ethics + +$ colophon --shelf shelf.txt converted/*.md | jq -c '{title, shelved, unshelved}' +{"title":"Leviathan","shelved":["Political Science"],"unshelved":["Hobbes, Thomas, 1588–1679. Leviathan"]} +{"title":"Christianity and Classical Culture","shelved":["Theology"],"unshelved":[]} +``` + +`unshelved` is the point of the exercise. A subject quietly dropped reads +exactly like a book that printed none, and the difference decides whether you +go looking for a heading your shelf is missing. + +**The title page is not evidence on its own.** A scan that begins mid-frontmatter +gives up a running head, a copyright line, or the author's name where the title +should be. What makes a printed record trustworthy is a second one agreeing, +and the ISBN in the same block is how you get one: + +```sh +colophon --shelf shelf.txt converted/*.md > printed.ndjson + +jq -r 'select(.isbns != null) | .isbns[0]' printed.ndjson \ + | idnorm --quiet \ + | bibfetch --polite you@example.org > register.ndjson +``` + +Now two sources that cannot have copied each other have spoken, and you can +say which books are settled and which need a person: + +```sh +jq -s -r ' + (.[1] | map({key: .id, value: .}) | from_entries) as $reg + | .[0][] + | . as $printed + | ($reg[$printed.isbns[0] // ""] // {}) as $found + | [ $printed.title, + $found.title, + (if $printed.title and $found.title then "corroborated" else "one source" end) + ] | @tsv +' printed.ndjson register.ndjson +``` + +``` +The First World War The First World War: A Very Short Introduction corroborated +Document5 The Problems of Philosophy corroborated +Born Fighting — one source +``` + +The second row is the case worth the whole recipe: a file named `Document5` +whose author reads `-ducati996`, which the book itself and OpenLibrary agree +is Russell's *The Problems of Philosophy*. The third is the honest failure — +one source said something, nothing confirmed it, and it goes to a human. + +In process, where a pipeline actually lives: + +```python +import catena + +shelf = open("shelf.txt").read() +printed = catena.read_colophon(markdown, shelf=shelf) + +found = None +if printed["isbns"]: + found = catena.fetch_identifier(printed["isbns"][0], polite="you@example.org") + +settled = bool(printed.get("title") and found and found.get("title")) +``` + +`catena.shelve(shelf, subjects)` files headings a catalogue already holds, by +the same rules, so a record's existing subjects and its book's printed ones are +judged alike. + ## Asking a model, without this suite calling one A tool here may compose a prompt. It does not send one. That keeps every diff --git a/examples/README.md b/examples/README.md index 2f6c825..45a56f7 100644 --- a/examples/README.md +++ b/examples/README.md @@ -15,6 +15,7 @@ NASA broadcast, which is a work of the US government. | [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles | | [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere | | [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take | +| [`python/shelve_a_library.py`](python/shelve_a_library.py) | What a book says about itself, shelved and checked against a register | | [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in | | [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player | | [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means | @@ -83,6 +84,20 @@ python examples/python/book_to_citations.py pip install catena-corpus psycopg DATABASE_URL=postgresql://user@host/db \ python examples/python/search_and_rerank.py "why did the empire decline" + +`shelve_a_library.py` is the metadata half: it reads the cataloguing block a +book prints in its own front matter, files those subjects onto a closed list +you supply, and asks the register that assigned the ISBN whether it agrees. + +```sh +python examples/python/shelve_a_library.py converted/*.md +``` + +Neither source is worth acting on alone — a scan that starts mid-frontmatter +gives up a running head where the title should be, and an ISBN a catalogue has +been carrying wrongly returns a confidently wrong book. The example prints +`corroborated` only when both said the same thing, and the disagreements are +the interesting output. ``` ## TypeScript diff --git a/examples/python/shelve_a_library.py b/examples/python/shelve_a_library.py new file mode 100644 index 0000000..677a13b --- /dev/null +++ b/examples/python/shelve_a_library.py @@ -0,0 +1,110 @@ +"""What a book says about itself, checked against what a register says back. + +A catalogue's subjects and titles come from wherever its files came from. Two +things can correct that without a model: the cataloguing block the book prints +in its own front matter, and the register that assigned its ISBN. Neither is +trustworthy alone — a scan that starts mid-frontmatter yields a running head +where the title should be, and an ISBN a catalogue has been carrying wrongly +returns a confidently wrong book. Together they are worth acting on. + + python examples/python/shelve_a_library.py converted/*.md + +With no arguments it reads a book from Project Gutenberg, which prints no +cataloguing block at all — the honest common case, and worth seeing. +""" + +import pathlib +import re +import sys +import urllib.request + +import catena + +# The closed list this library is browsed by. `Heading = term, term` gives the +# words that file a book under it; a heading on its own matches its own name. +SHELF = """ +Philosophy = philosophy, philosophical +Ancient Philosophy = ancient philosophy, plato, aristotle +Political Science = political science, politics, government +Theology = theology, christianity, religion +Church History = church history, ecclesiastical history +Byzantine History = byzantium, byzantine, constantinople +History = history, historiography +Literature = literature, fiction, poetry +Psychology = psychology +Ethics +Logic +""" + +GUTENBERG = "https://www.gutenberg.org/cache/epub/5827/pg5827.txt" + + +def sample(into: pathlib.Path) -> str: + """Locke's *Essay*, which Gutenberg strips the front matter from.""" + book = into / "locke.txt" + if not book.exists(): + into.mkdir(parents=True, exist_ok=True) + print(f"fetching {GUTENBERG}", file=sys.stderr) + with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310 + book.write_bytes(reply.read()) + return book.read_text(encoding="utf-8", errors="replace") + + +def same(a: str, b: str) -> bool: + """Whether two printings of one title are the same title. + + Compared on letters and digits alone. A register and a title page differ + over case, over a curly apostrophe against a straight one, and over + whether the subtitle is part of the title — none of which is a + disagreement, and all of which a plain string comparison calls one. + """ + fold = lambda s: re.sub(r"[^a-z0-9]+", "", (s or "").lower()) + a, b = fold(a), fold(b) + return bool(a) and bool(b) and (a in b or b in a) + + +def look(text: str, name: str) -> None: + printed = catena.read_colophon(text, shelf=SHELF) + + title = printed.get("title") + isbns = printed.get("isbns") or [] + print(f"\n{name}") + print(f" printed title {title or '— (no title page in these pages)'}") + if printed.get("authors"): + print(f" printed authors {', '.join(printed['authors'])}") + if printed.get("shelved"): + print(f" shelved under {', '.join(printed['shelved'])}") + for raw in printed.get("unshelved", []): + # Not a failure: a heading the shelf has no place for is how you find + # out the shelf is missing one. + print(f" no shelf for {raw}") + if not printed.get("subjects"): + print(" subjects — (no cataloguing block printed)") + + if not isbns: + print(" register — (nothing to look up; one source only)") + return + + # The only call here that opens a socket, and it blocks. + found = catena.fetch_identifier(isbns[0], polite="you@example.org") + if not found or not found.get("found"): + print(f" register {isbns[0]} is unknown to every register asked") + return + + print(f" register says {found['title']} / {', '.join(found.get('authors') or [])}") + agrees = bool(title) and same(title, found["title"]) + print(f" verdict {'corroborated' if agrees else 'two sources disagree — ask a person'}") + + +def main() -> int: + paths = [pathlib.Path(p) for p in sys.argv[1:]] + if not paths: + look(sample(pathlib.Path("/tmp/catena-example")), "Locke, from Gutenberg") + return 0 + for path in paths: + look(path.read_text(encoding="utf-8", errors="replace"), path.name) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())