funkpower
f4d936e620
The suite could read a cataloguing block and could ask a register, and nothing showed the two together — which is the only way either is worth acting on. A title page alone gives up a running head when the scan starts late; an ISBN alone returns a confidently wrong book when the catalogue has been carrying the wrong one for years. The example's title comparison folds to letters and digits rather than comparing strings. A register and a title page differ over case, over a curly apostrophe against a straight one, and over whether the subtitle counts as part of the title. An example that called those disagreements would teach the wrong lesson, and did until it was fixed. It runs on this library and finds both of the cases worth finding: a file called Document5 by -ducati996 that both sources agree is Russell's Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book about sculpture.
Examples
Working programs that put these tools together, in the four ways you would actually reach for them: as filters in a shell script, and as libraries from Rust, Python and JavaScript.
Every one of them runs. Where an example needs a book it fetches a public domain one from Project Gutenberg; where it needs a recording it fetches a NASA broadcast, which is a work of the US government.
shell/01-book-to-passages.sh |
A book file to passages worth embedding, and what its notes cite |
shell/02-video-to-chapters.sh |
A recording to chapter marks, citable passages, and a copy that plays on a phone |
shell/03-search-and-rerank.sh |
Passages into pgvector, then a search that asks twice and reconciles |
rust/ |
The same book walked through in one process, with no pipe anywhere |
python/book_to_citations.py |
A book to rows a database can take |
python/shelve_a_library.py |
What a book says about itself, shelved and checked against a register |
python/search_and_rerank.py |
The pgvector search, in the language a pipeline is usually written in |
typescript/src/episode-to-chapters.ts |
Captions into chapter marks and deep links for a player |
typescript/src/search-and-rerank.ts |
Fusing two rankings app-side, and what re-ranking means |
Shell
The tools are filters first: bytes in, records out, and jq between them.
Nothing is installed but the commands themselves.
cargo install --git https://git.pumpin.online/funkpower/catena \
booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
cuenorm cueseg cuepack avplan emb rrf
examples/shell/01-book-to-passages.sh ./work
01 fetches Gibbon's first volume, works out what the file is, reads the
metadata and the contents the publisher declared, converts and scrubs and
chunks it, and reads the footnotes. Needs pandoc and jq.
02 takes a recording's captions and produces chapter marks, passages that
carry the second they were said at, and — since a recording that is indexed
but unplayable is half a job — the one ffmpeg call that makes it play on a
phone. Needs jq, ffmpeg, and yt-dlp with impersonation support, which
YouTube now requires before it will hand over captions:
pipx install "yt-dlp[default,curl-cffi]"
03 loads the passages 01 wrote into Postgres with a vector column and a
text index, searches both ways, and fuses the two rankings. Needs psql, a
Postgres with pgvector, and an embedding server:
DATABASE_URL=postgresql://user@host/db \
EMB_MODEL=nomic-embed-text \
examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
Rust
Every tool is a library with a thin command on top, so anything a command does can be done in process instead. This walks a file from "what even is it" to passages and references without serialising anything.
cd examples/rust
cargo run # the built-in sample
cargo run -- book.md # a real one
Python
The shape a pipeline stage ends up as once the reading is a library call: convert, then loop, and what is left is the part that is actually about your schema.
pip install catena-corpus
python examples/python/book_to_citations.py
pip install catena-corpus psycopg
DATABASE_URL=postgresql://user@host/db \
python examples/python/search_and_rerank.py "why did the empire decline"
`shelve_a_library.py` is the metadata half: it reads the cataloguing block a
book prints in its own front matter, files those subjects onto a closed list
you supply, and asks the register that assigned the ISBN whether it agrees.
```sh
python examples/python/shelve_a_library.py converted/*.md
Neither source is worth acting on alone — a scan that starts mid-frontmatter
gives up a running head where the title should be, and an ISBN a catalogue has
been carrying wrongly returns a confidently wrong book. The example prints
corroborated only when both said the same thing, and the disagreements are
the interesting output.
## TypeScript
The bindings are WebAssembly, so they run in Node and in a browser with the
same API.
```sh
cd examples/typescript
npm install
npm run chapters # captions to chapter marks
npm run search # fusing two rankings
On re-ranking
Three of these examples end with the same explanation, because it is the thing most often got wrong.
Retrieval uses a bi-encoder: the passage and the query are embedded separately. That independence is what lets passage vectors be computed once and indexed, and it is also the limit — nothing in a passage's vector knows what was asked.
Fusion takes several ranked lists and makes one. Reciprocal rank fusion
scores a hit by where it placed rather than by what it scored: 1/(k + rank)
from each list that found it, added up. It is used because the scores are not
comparable — a cosine distance and a ts_rank_cd measure different things,
and no scaling makes them the same — so the ranks are kept and the scores
thrown away. A passage that both searches liked beats one that either loved.
That is rrf, and it needs no model.
A cross-encoder reads the query and the passage together and scores the pair. Far more accurate, and far too slow to run over a corpus, so it runs over the twenty or fifty candidates retrieval already found. In a pipeline the two stack: retrieve both ways, fuse the lists, then hand the top of the fused list to a cross-encoder for the final order.
Both get called re-ranking. Nothing here loads a model, so the last step is yours — the fused list is what you would hand it.