mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 03:07:13 +00:00
funkpower
58be244478
Eight of them, each doing something a pipeline actually has to do, and each starting from something anyone can fetch — a public domain book from Project Gutenberg, a NASA broadcast, or a sample carried in the file. Shell, because the tools are filters first: a book to passages and what its notes cite; a recording to chapter marks, citable passages and the one ffmpeg call that makes it play on a phone; passages into pgvector and a search that asks twice and reconciles. Rust, for the case where the caller already holds the text and a subprocess would be serialising for nothing. Python, for the shape a pipeline stage takes once the reading is a library call. TypeScript, for the app side — chapter marks and deep links, and fusing two rankings in process. Three of them end with what re-ranking means, because it is two different things under one name: fusing ranked lists, which is what `rrf` does and needs no model, and a cross-encoder scoring query and passage together, which is accurate, slow, and runs over what retrieval already found.
126 lines
5.1 KiB
Markdown
126 lines
5.1 KiB
Markdown
# Examples
|
|
|
|
Working programs that put these tools together, in the four ways you would
|
|
actually reach for them: as filters in a shell script, and as libraries from
|
|
Rust, Python and JavaScript.
|
|
|
|
Every one of them runs. Where an example needs a book it fetches a public
|
|
domain one from Project Gutenberg; where it needs a recording it fetches a
|
|
NASA broadcast, which is a work of the US government.
|
|
|
|
| | |
|
|
|---|---|
|
|
| [`shell/01-book-to-passages.sh`](shell/01-book-to-passages.sh) | A book file to passages worth embedding, and what its notes cite |
|
|
| [`shell/02-video-to-chapters.sh`](shell/02-video-to-chapters.sh) | A recording to chapter marks, citable passages, and a copy that plays on a phone |
|
|
| [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
|
|
| [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
|
|
| [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
|
|
| [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
|
|
| [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
|
|
| [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
|
|
|
|
## Shell
|
|
|
|
The tools are filters first: bytes in, records out, and `jq` between them.
|
|
Nothing is installed but the commands themselves.
|
|
|
|
```sh
|
|
cargo install --git https://git.pumpin.online/funkpower/catena \
|
|
booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
|
|
cuenorm cueseg cuepack avplan emb rrf
|
|
|
|
examples/shell/01-book-to-passages.sh ./work
|
|
```
|
|
|
|
`01` fetches Gibbon's first volume, works out what the file is, reads the
|
|
metadata and the contents the publisher declared, converts and scrubs and
|
|
chunks it, and reads the footnotes. Needs `pandoc` and `jq`.
|
|
|
|
`02` takes a recording's captions and produces chapter marks, passages that
|
|
carry the second they were said at, and — since a recording that is indexed
|
|
but unplayable is half a job — the one `ffmpeg` call that makes it play on a
|
|
phone. Needs `jq`, `ffmpeg`, and yt-dlp with impersonation support, which
|
|
YouTube now requires before it will hand over captions:
|
|
|
|
```sh
|
|
pipx install "yt-dlp[default,curl-cffi]"
|
|
```
|
|
|
|
`03` loads the passages `01` wrote into Postgres with a vector column and a
|
|
text index, searches both ways, and fuses the two rankings. Needs `psql`, a
|
|
Postgres with pgvector, and an embedding server:
|
|
|
|
```sh
|
|
DATABASE_URL=postgresql://user@host/db \
|
|
EMB_MODEL=nomic-embed-text \
|
|
examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
|
|
```
|
|
|
|
## Rust
|
|
|
|
Every tool is a library with a thin command on top, so anything a command
|
|
does can be done in process instead. This walks a file from "what even is it"
|
|
to passages and references without serialising anything.
|
|
|
|
```sh
|
|
cd examples/rust
|
|
cargo run # the built-in sample
|
|
cargo run -- book.md # a real one
|
|
```
|
|
|
|
## Python
|
|
|
|
The shape a pipeline stage ends up as once the reading is a library call:
|
|
convert, then loop, and what is left is the part that is actually about your
|
|
schema.
|
|
|
|
```sh
|
|
pip install catena
|
|
python examples/python/book_to_citations.py
|
|
```
|
|
|
|
```sh
|
|
pip install catena psycopg
|
|
DATABASE_URL=postgresql://user@host/db \
|
|
python examples/python/search_and_rerank.py "why did the empire decline"
|
|
```
|
|
|
|
## TypeScript
|
|
|
|
The bindings are WebAssembly, so they run in Node and in a browser with the
|
|
same API.
|
|
|
|
```sh
|
|
cd examples/typescript
|
|
npm install
|
|
npm run chapters # captions to chapter marks
|
|
npm run search # fusing two rankings
|
|
```
|
|
|
|
## On re-ranking
|
|
|
|
Three of these examples end with the same explanation, because it is the
|
|
thing most often got wrong.
|
|
|
|
Retrieval uses a **bi-encoder**: the passage and the query are embedded
|
|
separately. That independence is what lets passage vectors be computed once
|
|
and indexed, and it is also the limit — nothing in a passage's vector knows
|
|
what was asked.
|
|
|
|
**Fusion** takes several ranked lists and makes one. Reciprocal rank fusion
|
|
scores a hit by where it placed rather than by what it scored: `1/(k + rank)`
|
|
from each list that found it, added up. It is used because the scores are not
|
|
comparable — a cosine distance and a `ts_rank_cd` measure different things,
|
|
and no scaling makes them the same — so the ranks are kept and the scores
|
|
thrown away. A passage that both searches liked beats one that either loved.
|
|
That is [`rrf`](../crates/rrf/README.md), and it needs no model.
|
|
|
|
A **cross-encoder** reads the query and the passage together and scores the
|
|
pair. Far more accurate, and far too slow to run over a corpus, so it runs
|
|
over the twenty or fifty candidates retrieval already found. In a pipeline
|
|
the two stack: retrieve both ways, fuse the lists, then hand the top of the
|
|
fused list to a cross-encoder for the final order.
|
|
|
|
Both get called re-ranking. Nothing here loads a model, so the last step is
|
|
yours — the fused list is what you would hand it.
|