mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
funkpower
f4d936e620
The suite could read a cataloguing block and could ask a register, and nothing showed the two together — which is the only way either is worth acting on. A title page alone gives up a running head when the scan starts late; an ISBN alone returns a confidently wrong book when the catalogue has been carrying the wrong one for years. The example's title comparison folds to letters and digits rather than comparing strings. A register and a title page differ over case, over a curly apostrophe against a straight one, and over whether the subtitle counts as part of the title. An example that called those disagreements would teach the wrong lesson, and did until it was fixed. It runs on this library and finds both of the cases worth finding: a file called Document5 by -ducati996 that both sources agree is Russell's Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book about sculpture.
141 lines
5.8 KiB
Markdown
141 lines
5.8 KiB
Markdown
# Examples
|
|
|
|
Working programs that put these tools together, in the four ways you would
|
|
actually reach for them: as filters in a shell script, and as libraries from
|
|
Rust, Python and JavaScript.
|
|
|
|
Every one of them runs. Where an example needs a book it fetches a public
|
|
domain one from Project Gutenberg; where it needs a recording it fetches a
|
|
NASA broadcast, which is a work of the US government.
|
|
|
|
| | |
|
|
|---|---|
|
|
| [`shell/01-book-to-passages.sh`](shell/01-book-to-passages.sh) | A book file to passages worth embedding, and what its notes cite |
|
|
| [`shell/02-video-to-chapters.sh`](shell/02-video-to-chapters.sh) | A recording to chapter marks, citable passages, and a copy that plays on a phone |
|
|
| [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
|
|
| [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
|
|
| [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
|
|
| [`python/shelve_a_library.py`](python/shelve_a_library.py) | What a book says about itself, shelved and checked against a register |
|
|
| [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
|
|
| [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
|
|
| [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
|
|
|
|
## Shell
|
|
|
|
The tools are filters first: bytes in, records out, and `jq` between them.
|
|
Nothing is installed but the commands themselves.
|
|
|
|
```sh
|
|
cargo install --git https://git.pumpin.online/funkpower/catena \
|
|
booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
|
|
cuenorm cueseg cuepack avplan emb rrf
|
|
|
|
examples/shell/01-book-to-passages.sh ./work
|
|
```
|
|
|
|
`01` fetches Gibbon's first volume, works out what the file is, reads the
|
|
metadata and the contents the publisher declared, converts and scrubs and
|
|
chunks it, and reads the footnotes. Needs `pandoc` and `jq`.
|
|
|
|
`02` takes a recording's captions and produces chapter marks, passages that
|
|
carry the second they were said at, and — since a recording that is indexed
|
|
but unplayable is half a job — the one `ffmpeg` call that makes it play on a
|
|
phone. Needs `jq`, `ffmpeg`, and yt-dlp with impersonation support, which
|
|
YouTube now requires before it will hand over captions:
|
|
|
|
```sh
|
|
pipx install "yt-dlp[default,curl-cffi]"
|
|
```
|
|
|
|
`03` loads the passages `01` wrote into Postgres with a vector column and a
|
|
text index, searches both ways, and fuses the two rankings. Needs `psql`, a
|
|
Postgres with pgvector, and an embedding server:
|
|
|
|
```sh
|
|
DATABASE_URL=postgresql://user@host/db \
|
|
EMB_MODEL=nomic-embed-text \
|
|
examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
|
|
```
|
|
|
|
## Rust
|
|
|
|
Every tool is a library with a thin command on top, so anything a command
|
|
does can be done in process instead. This walks a file from "what even is it"
|
|
to passages and references without serialising anything.
|
|
|
|
```sh
|
|
cd examples/rust
|
|
cargo run # the built-in sample
|
|
cargo run -- book.md # a real one
|
|
```
|
|
|
|
## Python
|
|
|
|
The shape a pipeline stage ends up as once the reading is a library call:
|
|
convert, then loop, and what is left is the part that is actually about your
|
|
schema.
|
|
|
|
```sh
|
|
pip install catena-corpus
|
|
python examples/python/book_to_citations.py
|
|
```
|
|
|
|
```sh
|
|
pip install catena-corpus psycopg
|
|
DATABASE_URL=postgresql://user@host/db \
|
|
python examples/python/search_and_rerank.py "why did the empire decline"
|
|
|
|
`shelve_a_library.py` is the metadata half: it reads the cataloguing block a
|
|
book prints in its own front matter, files those subjects onto a closed list
|
|
you supply, and asks the register that assigned the ISBN whether it agrees.
|
|
|
|
```sh
|
|
python examples/python/shelve_a_library.py converted/*.md
|
|
```
|
|
|
|
Neither source is worth acting on alone — a scan that starts mid-frontmatter
|
|
gives up a running head where the title should be, and an ISBN a catalogue has
|
|
been carrying wrongly returns a confidently wrong book. The example prints
|
|
`corroborated` only when both said the same thing, and the disagreements are
|
|
the interesting output.
|
|
```
|
|
|
|
## TypeScript
|
|
|
|
The bindings are WebAssembly, so they run in Node and in a browser with the
|
|
same API.
|
|
|
|
```sh
|
|
cd examples/typescript
|
|
npm install
|
|
npm run chapters # captions to chapter marks
|
|
npm run search # fusing two rankings
|
|
```
|
|
|
|
## On re-ranking
|
|
|
|
Three of these examples end with the same explanation, because it is the
|
|
thing most often got wrong.
|
|
|
|
Retrieval uses a **bi-encoder**: the passage and the query are embedded
|
|
separately. That independence is what lets passage vectors be computed once
|
|
and indexed, and it is also the limit — nothing in a passage's vector knows
|
|
what was asked.
|
|
|
|
**Fusion** takes several ranked lists and makes one. Reciprocal rank fusion
|
|
scores a hit by where it placed rather than by what it scored: `1/(k + rank)`
|
|
from each list that found it, added up. It is used because the scores are not
|
|
comparable — a cosine distance and a `ts_rank_cd` measure different things,
|
|
and no scaling makes them the same — so the ranks are kept and the scores
|
|
thrown away. A passage that both searches liked beats one that either loved.
|
|
That is [`rrf`](../crates/rrf/README.md), and it needs no model.
|
|
|
|
A **cross-encoder** reads the query and the passage together and scores the
|
|
pair. Far more accurate, and far too slow to run over a corpus, so it runs
|
|
over the twenty or fifty candidates retrieval already found. In a pipeline
|
|
the two stack: retrieve both ways, fuse the lists, then hand the top of the
|
|
fused list to a cross-encoder for the final order.
|
|
|
|
Both get called re-ranking. Nothing here loads a model, so the last step is
|
|
yours — the fused list is what you would hand it.
|