catena/examples/README.md

Examples

Working programs that put these tools together, in the four ways you would actually reach for them: as filters in a shell script, and as libraries from Rust, Python and JavaScript.

Every one of them runs. Where an example needs a book it fetches a public domain one from Project Gutenberg; where it needs a recording it fetches a NASA broadcast, which is a work of the US government.

shell/01-book-to-passages.sh	A book file to passages worth embedding, and what its notes cite
shell/02-video-to-chapters.sh	A recording to chapter marks, citable passages, and a copy that plays on a phone
shell/03-search-and-rerank.sh	Passages into pgvector, then a search that asks twice and reconciles
rust/	The same book walked through in one process, with no pipe anywhere
python/book_to_citations.py	A book to rows a database can take
python/search_and_rerank.py	The pgvector search, in the language a pipeline is usually written in
typescript/src/episode-to-chapters.ts	Captions into chapter marks and deep links for a player
typescript/src/search-and-rerank.ts	Fusing two rankings app-side, and what re-ranking means

Shell

The tools are filters first: bytes in, records out, and jq between them. Nothing is installed but the commands themselves.

cargo install --git https://git.pumpin.online/funkpower/catena \
    booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
    cuenorm cueseg cuepack avplan emb rrf

examples/shell/01-book-to-passages.sh ./work

01 fetches Gibbon's first volume, works out what the file is, reads the metadata and the contents the publisher declared, converts and scrubs and chunks it, and reads the footnotes. Needs pandoc and jq.

02 takes a recording's captions and produces chapter marks, passages that carry the second they were said at, and — since a recording that is indexed but unplayable is half a job — the one ffmpeg call that makes it play on a phone. Needs jq, ffmpeg, and yt-dlp with impersonation support, which YouTube now requires before it will hand over captions:

pipx install "yt-dlp[default,curl-cffi]"

03 loads the passages 01 wrote into Postgres with a vector column and a text index, searches both ways, and fuses the two rankings. Needs psql, a Postgres with pgvector, and an embedding server:

DATABASE_URL=postgresql://user@host/db \
EMB_MODEL=nomic-embed-text \
    examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work

Rust

Every tool is a library with a thin command on top, so anything a command does can be done in process instead. This walks a file from "what even is it" to passages and references without serialising anything.

cd examples/rust
cargo run                 # the built-in sample
cargo run -- book.md      # a real one

Python

The shape a pipeline stage ends up as once the reading is a library call: convert, then loop, and what is left is the part that is actually about your schema.

pip install catena-corpus
python examples/python/book_to_citations.py

pip install catena-corpus psycopg
DATABASE_URL=postgresql://user@host/db \
    python examples/python/search_and_rerank.py "why did the empire decline"

TypeScript

The bindings are WebAssembly, so they run in Node and in a browser with the same API.

cd examples/typescript
npm install
npm run chapters          # captions to chapter marks
npm run search            # fusing two rankings

On re-ranking

Three of these examples end with the same explanation, because it is the thing most often got wrong.

Retrieval uses a bi-encoder: the passage and the query are embedded separately. That independence is what lets passage vectors be computed once and indexed, and it is also the limit — nothing in a passage's vector knows what was asked.

Fusion takes several ranked lists and makes one. Reciprocal rank fusion scores a hit by where it placed rather than by what it scored: 1/(k + rank) from each list that found it, added up. It is used because the scores are not comparable — a cosine distance and a ts_rank_cd measure different things, and no scaling makes them the same — so the ranks are kept and the scores thrown away. A passage that both searches liked beats one that either loved. That is rrf, and it needs no model.

A cross-encoder reads the query and the passage together and scores the pair. Far more accurate, and far too slow to run over a corpus, so it runs over the twenty or fifty candidates retrieval already found. In a pipeline the two stack: retrieve both ways, fuse the lists, then hand the top of the fused list to a cross-encoder for the final order.

Both get called re-ranking. Nothing here loads a model, so the last step is yours — the fused list is what you would hand it.