mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 11:17:13 +00:00
recipes, examples: shelving a library by what its books printed
The suite could read a cataloguing block and could ask a register, and nothing showed the two together — which is the only way either is worth acting on. A title page alone gives up a running head when the scan starts late; an ISBN alone returns a confidently wrong book when the catalogue has been carrying the wrong one for years. The example's title comparison folds to letters and digits rather than comparing strings. A register and a title page differ over case, over a curly apostrophe against a straight one, and over whether the subtitle counts as part of the title. An example that called those disagreements would teach the wrong lesson, and did until it was fixed. It runs on this library and finds both of the cases worth finding: a file called Document5 by -ducati996 that both sources agree is Russell's Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book about sculpture.
This commit is contained in:
@@ -6,6 +6,7 @@ this suite, because a real pipeline has a database at the end of it.
|
||||
|
||||
- [A shelf of files, sorted before anything else touches it](#a-shelf-of-files-sorted-before-anything-else-touches-it)
|
||||
- [A catalogue out of what the files already say](#a-catalogue-out-of-what-the-files-already-say)
|
||||
- [Shelving a library by what its books printed](#shelving-a-library-by-what-its-books-printed)
|
||||
- [Asking a model, without this suite calling one](#asking-a-model-without-this-suite-calling-one)
|
||||
- [A scanned book into searchable passages](#a-scanned-book-into-searchable-passages)
|
||||
- [A collected edition into one record per work](#a-collected-edition-into-one-record-per-work)
|
||||
@@ -98,6 +99,97 @@ with:
|
||||
langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv'
|
||||
```
|
||||
|
||||
## Shelving a library by what its books printed
|
||||
|
||||
A catalogue's subjects come from wherever the file came from, which is to say
|
||||
from nowhere in particular. A book's own cataloguing block is the one place its
|
||||
subjects are stated rather than guessed — and stated in Library of Congress
|
||||
form, which is precise and useless for browsing:
|
||||
|
||||
```
|
||||
$ colophon converted/hobbes.md | jq -r '.subjects[]'
|
||||
Hobbes, Thomas, 1588–1679. Leviathan
|
||||
Political science – Early works to 1800
|
||||
```
|
||||
|
||||
Nobody has a shelf called that. Give `colophon` the list you actually file by
|
||||
and it files them for you:
|
||||
|
||||
```
|
||||
$ cat shelf.txt
|
||||
Political Science = political science, politics
|
||||
Byzantine History = byzantium, byzantine, constantinople
|
||||
Theology = theology, christianity
|
||||
Ethics
|
||||
|
||||
$ colophon --shelf shelf.txt converted/*.md | jq -c '{title, shelved, unshelved}'
|
||||
{"title":"Leviathan","shelved":["Political Science"],"unshelved":["Hobbes, Thomas, 1588–1679. Leviathan"]}
|
||||
{"title":"Christianity and Classical Culture","shelved":["Theology"],"unshelved":[]}
|
||||
```
|
||||
|
||||
`unshelved` is the point of the exercise. A subject quietly dropped reads
|
||||
exactly like a book that printed none, and the difference decides whether you
|
||||
go looking for a heading your shelf is missing.
|
||||
|
||||
**The title page is not evidence on its own.** A scan that begins mid-frontmatter
|
||||
gives up a running head, a copyright line, or the author's name where the title
|
||||
should be. What makes a printed record trustworthy is a second one agreeing,
|
||||
and the ISBN in the same block is how you get one:
|
||||
|
||||
```sh
|
||||
colophon --shelf shelf.txt converted/*.md > printed.ndjson
|
||||
|
||||
jq -r 'select(.isbns != null) | .isbns[0]' printed.ndjson \
|
||||
| idnorm --quiet \
|
||||
| bibfetch --polite you@example.org > register.ndjson
|
||||
```
|
||||
|
||||
Now two sources that cannot have copied each other have spoken, and you can
|
||||
say which books are settled and which need a person:
|
||||
|
||||
```sh
|
||||
jq -s -r '
|
||||
(.[1] | map({key: .id, value: .}) | from_entries) as $reg
|
||||
| .[0][]
|
||||
| . as $printed
|
||||
| ($reg[$printed.isbns[0] // ""] // {}) as $found
|
||||
| [ $printed.title,
|
||||
$found.title,
|
||||
(if $printed.title and $found.title then "corroborated" else "one source" end)
|
||||
] | @tsv
|
||||
' printed.ndjson register.ndjson
|
||||
```
|
||||
|
||||
```
|
||||
The First World War The First World War: A Very Short Introduction corroborated
|
||||
Document5 The Problems of Philosophy corroborated
|
||||
Born Fighting — one source
|
||||
```
|
||||
|
||||
The second row is the case worth the whole recipe: a file named `Document5`
|
||||
whose author reads `-ducati996`, which the book itself and OpenLibrary agree
|
||||
is Russell's *The Problems of Philosophy*. The third is the honest failure —
|
||||
one source said something, nothing confirmed it, and it goes to a human.
|
||||
|
||||
In process, where a pipeline actually lives:
|
||||
|
||||
```python
|
||||
import catena
|
||||
|
||||
shelf = open("shelf.txt").read()
|
||||
printed = catena.read_colophon(markdown, shelf=shelf)
|
||||
|
||||
found = None
|
||||
if printed["isbns"]:
|
||||
found = catena.fetch_identifier(printed["isbns"][0], polite="you@example.org")
|
||||
|
||||
settled = bool(printed.get("title") and found and found.get("title"))
|
||||
```
|
||||
|
||||
`catena.shelve(shelf, subjects)` files headings a catalogue already holds, by
|
||||
the same rules, so a record's existing subjects and its book's printed ones are
|
||||
judged alike.
|
||||
|
||||
## Asking a model, without this suite calling one
|
||||
|
||||
A tool here may compose a prompt. It does not send one. That keeps every
|
||||
|
||||
Reference in New Issue
Block a user