idnorm and bibfetch: an identifier read, and a register asked

This commit is contained in:
funkpower
2026-09-19 19:19:54 -04:00
parent 4d14544409
commit d47881ef3b
29 changed files with 1920 additions and 9 deletions
+2 -1
View File
@@ -61,9 +61,10 @@ same as any other filter.
## Nothing reaches outside the process
No network, no database, no configuration file, no state kept between runs.
Four exceptions, each declared in `--help`:
Five exceptions, each declared in `--help`:
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
- [`bibfetch`](../crates/bibfetch/README.md) asks a register about an identifier, likewise.
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`.
- [`pdfgutter`](../crates/pdfgutter/README.md) runs `pdftoppm` to look at a page.
- [`avplan`](../crates/avplan/README.md) runs `ffprobe` to read a file's streams, and
+42
View File
@@ -12,6 +12,7 @@ this suite, because a real pipeline has a database at the end of it.
- [Passages into pgvector](#passages-into-pgvector)
- [Hybrid search, fused](#hybrid-search-fused)
- [A catalogue that has drifted](#a-catalogue-that-has-drifted)
- [A bibliography that has been copied between formats](#a-bibliography-that-has-been-copied-between-formats)
- [A video library that plays on the thing you carry](#a-video-library-that-plays-on-the-thing-you-carry)
- [Notes on composing these](#notes-on-composing-these)
@@ -267,6 +268,14 @@ vocabmerge --kind author authors.ndjson > merges.ndjson
vocabmerge --kind tag tags.ndjson >> merges.ndjson
```
Series and volume, which most catalogues have on almost nothing:
```sh
seriesof catalogue.ndjson \
| jq -r 'select(.volume) | "UPDATE book SET series = \(.series|@json), series_index = \(.volume) WHERE title = \(.title|@json);"' \
| psql
```
Apply only what you believe:
```sh
@@ -275,6 +284,39 @@ jq -r 'select(.action == "merge" and .confidence >= 0.9)
merges.ndjson | psql
```
## A bibliography that has been copied between formats
Identifiers arrive wrapped in whatever the last tool left on them, and a
catalogue that stores them as they came cannot tell two spellings of one DOI
apart.
```sh
idnorm --quiet refs.txt > ids.ndjson
```
The ones that cannot be right — an ISBN or an ISSN whose check digit fails:
```sh
jq -r 'select(.checksum == false) | .input' ids.ndjson
```
The ones that are well formed and still wrong, which needs a register:
```sh
bibfetch --polite you@example.org ids.ndjson \
| jq -r 'select(.found | not) | "missing: \(.id)"'
```
That second question is a different one, which is why it is a different tool:
`idnorm` never touches the network and `bibfetch` does nothing else.
Filling in what you only had the identifier for:
```sh
idnorm --quiet refs.txt | bibfetch \
| jq -r 'select(.found) | "\(.year)\t\(.authors | join(", "))\t\(.title)"'
```
## A video library that plays on the thing you carry
```sh