funkpower/catena
1
0
Fork 0
mirror of https://git.pumpin.online/funkpower/catena.git synced 2026-10-22 18:57:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

examples: programs that run, in four languages

Browse Source 
Eight of them, each doing something a pipeline actually has to do, and each
starting from something anyone can fetch — a public domain book from Project
Gutenberg, a NASA broadcast, or a sample carried in the file.

Shell, because the tools are filters first: a book to passages and what its
notes cite; a recording to chapter marks, citable passages and the one
ffmpeg call that makes it play on a phone; passages into pgvector and a
search that asks twice and reconciles.

Rust, for the case where the caller already holds the text and a subprocess
would be serialising for nothing. Python, for the shape a pipeline stage
takes once the reading is a library call. TypeScript, for the app side —
chapter marks and deep links, and fusing two rankings in process.

Three of them end with what re-ranking means, because it is two different
things under one name: fusing ranked lists, which is what `rrf` does and
needs no model, and a cross-encoder scoring query and passage together,
which is accurate, slow, and runs over what retrieval already found.
This commit is contained in:
funkpower
2026-09-21 08:12:13 -04:00
parent cfa86f778f
commit 58be244478
14 changed files with 1114 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4
View File
@@ -48,6 +48,10 @@ Each tool installs by name, so you can take one without taking the rest.
Each links to its own page: what it does, when you would use it, worked
examples, options and exit codes.
**→ [Examples](examples/README.md) — programs that run.** A book to
passages and references, a recording to chapter marks and a copy that plays
on a phone, and a hybrid search fused, in shell, Rust, Python and TypeScript.
**→ [Recipes](docs/recipes.md) — using these tools together.** A scanned book
to a searchable passage, a transcript to a citation with a timestamp, vectors
into pgvector, and hybrid search fused on top.
examples/.gitignore
+3
View File
@@ -0,0 +1,3 @@
work/
typescript/node_modules/
typescript/package-lock.json
examples/README.md
+125
View File
@@ -0,0 +1,125 @@
# Examples
Working programs that put these tools together, in the four ways you would
actually reach for them: as filters in a shell script, and as libraries from
Rust, Python and JavaScript.
Every one of them runs. Where an example needs a book it fetches a public
domain one from Project Gutenberg; where it needs a recording it fetches a
NASA broadcast, which is a work of the US government.
| | |
|---|---|
| [`shell/01-book-to-passages.sh`](shell/01-book-to-passages.sh) | A book file to passages worth embedding, and what its notes cite |
| [`shell/02-video-to-chapters.sh`](shell/02-video-to-chapters.sh) | A recording to chapter marks, citable passages, and a copy that plays on a phone |
| [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
| [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
| [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
| [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
| [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
| [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
## Shell
The tools are filters first: bytes in, records out, and `jq` between them.
Nothing is installed but the commands themselves.
```sh
cargo install --git https://git.pumpin.online/funkpower/catena \
booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
cuenorm cueseg cuepack avplan emb rrf
examples/shell/01-book-to-passages.sh ./work
```
`01` fetches Gibbon's first volume, works out what the file is, reads the
metadata and the contents the publisher declared, converts and scrubs and
chunks it, and reads the footnotes. Needs `pandoc` and `jq`.
`02` takes a recording's captions and produces chapter marks, passages that
carry the second they were said at, and — since a recording that is indexed
but unplayable is half a job — the one `ffmpeg` call that makes it play on a
phone. Needs `jq`, `ffmpeg`, and yt-dlp with impersonation support, which
YouTube now requires before it will hand over captions:
```sh
pipx install "yt-dlp[default,curl-cffi]"
```
`03` loads the passages `01` wrote into Postgres with a vector column and a
text index, searches both ways, and fuses the two rankings. Needs `psql`, a
Postgres with pgvector, and an embedding server:
```sh
DATABASE_URL=postgresql://user@host/db \
EMB_MODEL=nomic-embed-text \
examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
```
## Rust
Every tool is a library with a thin command on top, so anything a command
does can be done in process instead. This walks a file from "what even is it"
to passages and references without serialising anything.
```sh
cd examples/rust
cargo run # the built-in sample
cargo run -- book.md # a real one
```
## Python
The shape a pipeline stage ends up as once the reading is a library call:
convert, then loop, and what is left is the part that is actually about your
schema.
```sh
pip install catena
python examples/python/book_to_citations.py
```
```sh
pip install catena psycopg
DATABASE_URL=postgresql://user@host/db \
python examples/python/search_and_rerank.py "why did the empire decline"
```
## TypeScript
The bindings are WebAssembly, so they run in Node and in a browser with the
same API.
```sh
cd examples/typescript
npm install
npm run chapters # captions to chapter marks
npm run search # fusing two rankings
```
## On re-ranking
Three of these examples end with the same explanation, because it is the
thing most often got wrong.
Retrieval uses a **bi-encoder**: the passage and the query are embedded
separately. That independence is what lets passage vectors be computed once
and indexed, and it is also the limit — nothing in a passage's vector knows
what was asked.
**Fusion** takes several ranked lists and makes one. Reciprocal rank fusion
scores a hit by where it placed rather than by what it scored: `1/(k + rank)`
from each list that found it, added up. It is used because the scores are not
comparable — a cosine distance and a `ts_rank_cd` measure different things,
and no scaling makes them the same — so the ranks are kept and the scores
thrown away. A passage that both searches liked beats one that either loved.
That is [`rrf`](../crates/rrf/README.md), and it needs no model.
A **cross-encoder** reads the query and the passage together and scores the
pair. Far more accurate, and far too slow to run over a corpus, so it runs
over the twenty or fifty candidates retrieval already found. In a pipeline
the two stack: retrieve both ways, fuse the lists, then hand the top of the
fused list to a cross-encoder for the final order.
Both get called re-ranking. Nothing here loads a model, so the last step is
yours — the fused list is what you would hand it.
examples/python/book_to_citations.py
+128
View File
@@ -0,0 +1,128 @@
"""A book to rows a database can take, without a subprocess in sight.
This is the shape a pipeline stage ends up as once the reading is a library
call: convert, then loop, and what is left is the part that is actually about
your schema.
python examples/python/book_to_citations.py path/to/book.epub
With no argument it fetches Gibbon's first volume from Project Gutenberg,
which is public domain and has a few hundred pages of footnotes.
"""
import pathlib
import subprocess
import sys
import urllib.request
import catena
GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub"
def fetch(into: pathlib.Path) -> pathlib.Path:
"""The sample book, downloaded once."""
book = into / "gibbon.epub"
if not book.exists():
into.mkdir(parents=True, exist_ok=True)
print(f"fetching {GUTENBERG}")
with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310
book.write_bytes(reply.read())
return book
def to_markdown(book: pathlib.Path) -> str:
"""Pandoc is the one part of this that is somebody else's program.
`commonmark` rather than `commonmark_x`: the extended flavour writes
anchors as bracketed spans, and they end up inside the passages.
"""
done = subprocess.run(
["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)],
capture_output=True,
text=True,
check=True,
)
return done.stdout
def rows(markdown: str) -> list[dict]:
"""Passages in, citation rows out.
Two library calls do the work. `chunk_markdown` cuts the book into
passages and flags the ones carrying citation material — which is the
whole reason to ask it rather than looking for a heading, since a page of
notes at the foot of a chapter has no heading over it. `find_citations`
then reads each of those.
"""
out = []
for chunk in catena.chunk_markdown(catena.scrub(markdown)):
if "footnote_citations" not in chunk["flags"] and chunk["role"] != "bibliography":
continue
for found in catena.find_citations(chunk["text"]):
# An `ibid.` or a short cite points at an entry above rather than
# naming a work of its own, so there is nothing to store.
if found["continuation"]:
continue
if not (found["title"] or found["authors"] or found["ids"]):
continue
out.append(
{
"page": chunk["page"],
"chapter": chunk["chapter"],
"citation_text": found["text"],
"referenced_work": found["title"],
"referenced_author": "; ".join(found["authors"]) or None,
"referenced_year": found["year"],
"referenced_container": found["container"],
"kind": found["kind"],
"unsettled": found["unsettled"],
}
)
return out
def main() -> int:
where = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else fetch(pathlib.Path("work"))
print(f"== {where.name}")
identified = catena.identify(where.name, where.read_bytes())
print(f" a {identified['kind']}, {identified['bytes']} bytes")
markdown = to_markdown(where)
passages = catena.chunk_markdown(catena.scrub(markdown))
carrying = [c for c in passages if "footnote_citations" in c["flags"]]
print(f" {len(passages)} passages, {len(carrying)} carrying citation material")
found = rows(markdown)
named = [r for r in found if r["referenced_work"]]
print(f"\n== {len(found)} rows, {len(named)} naming a work")
# What a stage like this would do next is its own business: one executemany
# against your own table. The point is that everything above it is two
# function calls rather than a parser you have to keep.
print("\n INSERT INTO citations (citation_text, referenced_work, referenced_author, …)")
whole = [r for r in named if r["referenced_author"]] or named
for row in whole[:5]:
work = (row["referenced_work"] or "")[:40]
who = (row["referenced_author"] or "—")[:22]
print(f" (…, {work!r}, {who!r}, {row['referenced_year']})")
# Nothing is invented. Where a field could not be read the record says so
# rather than guessing, and a caller that wants a model to fill the gaps
# has `citation_prompt` and `merge_citation_reply` for exactly that.
# Gibbon cites in the classical short form — `Tacit. Annal. xiii. 5.` —
# so most of his notes name a passage rather than a book. Nothing is
# invented to cover that: the record says which fields it could not read,
# and `citation_prompt` and `merge_citation_reply` are there for a caller
# that wants a model to finish them.
unsettled = [r for r in found if r["unsettled"]]
print(f"\n {len(unsettled)} rows name something they could not finish reading:")
for row in unsettled[:3]:
print(f" {row['citation_text'][:56]} → {', '.join(row['unsettled'])}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
examples/python/search_and_rerank.py
+192
View File
@@ -0,0 +1,192 @@
"""Passages into pgvector, then a search that asks twice and reconciles.
Loads the passages of a book into Postgres with both a vector column and a
text index, searches both ways, and fuses the two rankings into one.
DATABASE_URL=postgresql://user@host/db \
EMB_MODEL=nomic-embed-text \
python examples/python/search_and_rerank.py "why did the empire decline"
Needs psycopg, a Postgres with pgvector, an embedding server with Ollama's
API, and a book — the sample is fetched if none is given.
"""
import json
import os
import pathlib
import subprocess
import sys
import urllib.request
import catena
import psycopg
EMB_HOST = os.environ.get("EMB_HOST", "http://localhost:11434")
EMB_MODEL = os.environ.get("EMB_MODEL", "nomic-embed-text")
GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub"
def embed(texts: list[str]) -> list[list[float]]:
"""Ask the embedding server, in batches it will accept.
The one thing here that touches the network. No model is loaded in this
process: it belongs behind a server, where it is shared and where its
GPU is somebody else's problem.
"""
out: list[list[float]] = []
for at in range(0, len(texts), 32):
batch = texts[at : at + 32]
body = json.dumps({"model": EMB_MODEL, "input": batch}).encode()
request = urllib.request.Request(
f"{EMB_HOST}/api/embed", data=body, headers={"content-type": "application/json"}
)
with urllib.request.urlopen(request, timeout=300) as reply: # noqa: S310
out.extend(json.load(reply)["embeddings"])
print(f" embedded {len(out)}/{len(texts)}", end="\r", file=sys.stderr)
return out
def passages(book: pathlib.Path) -> list[dict]:
"""The book, cut into passages worth embedding."""
markdown = subprocess.run(
["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)],
capture_output=True, text=True, check=True,
).stdout
chunks = catena.chunk_markdown(catena.scrub(markdown))
# A table of contents embeds as well as anything and retrieves for every
# query, which is worse than not having it.
return [c for c in chunks if c["role"] == "body" and c["tokens"] > 40]
def load(conn, rows: list[dict], vectors: list[list[float]]) -> None:
width = len(vectors[0])
with conn.cursor() as cur:
cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
cur.execute("DROP TABLE IF EXISTS catena_passages")
# The tsvector is generated from the body rather than written
# alongside it, so the two cannot drift apart.
cur.execute(f"""
CREATE TABLE catena_passages (
idx integer PRIMARY KEY,
chapter text,
body text NOT NULL,
fts tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
vec vector({width})
)""")
with cur.copy("COPY catena_passages (idx, chapter, body, vec) FROM STDIN") as copy:
for chunk, vector in zip(rows, vectors):
copy.write_row((chunk["idx"], chunk["chapter"], chunk["text"], str(vector)))
cur.execute("CREATE INDEX ON catena_passages USING gin (fts)")
cur.execute("CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops)")
conn.commit()
def by_meaning(conn, vector: list[float], limit: int = 10) -> list[dict]:
"""Nearest in the embedding space.
`<=>` is cosine distance, so smaller is nearer. This finds a passage
about the fall of Rome for a query about decline, sharing no words.
"""
with conn.cursor() as cur:
cur.execute(
"""SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g')
FROM catena_passages ORDER BY vec <=> %s::vector LIMIT %s""",
(str(vector), limit),
)
return [
{"v": 1, "id": str(idx), "channel": "vec", "rank": at, "score": 0.0, "text": body}
for at, (idx, body) in enumerate(cur.fetchall(), start=1)
]
def by_word(conn, query: str, limit: int = 10) -> list[dict]:
"""Nearest by the words themselves.
Finds the passage that actually says the word where the vector search
drifted to a paraphrase, and finds nothing at all where the words are
not in the book — which is a useful thing to be told.
"""
with conn.cursor() as cur:
cur.execute(
"""SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g')
FROM catena_passages
WHERE fts @@ websearch_to_tsquery('english', %(q)s)
ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', %(q)s)) DESC
LIMIT %(n)s""",
{"q": query, "n": limit},
)
return [
{"v": 1, "id": str(idx), "channel": "fts", "rank": at, "score": 0.0, "text": body}
for at, (idx, body) in enumerate(cur.fetchall(), start=1)
]
def main() -> int:
query = sys.argv[1] if len(sys.argv) > 1 else "why did the roman empire decline"
url = os.environ.get("DATABASE_URL")
if not url:
print("set DATABASE_URL", file=sys.stderr)
return 64
work = pathlib.Path("work")
work.mkdir(exist_ok=True)
book = work / "gibbon.epub"
if not book.exists():
print(f"fetching {GUTENBERG}")
with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310
book.write_bytes(reply.read())
rows = passages(book)
print(f"== {len(rows)} passages")
vectors = embed([" ".join(chunk["text"].split()) for chunk in rows])
print(f" {EMB_MODEL} returns {len(vectors[0])} numbers per passage")
with psycopg.connect(url) as conn:
load(conn, rows, vectors)
print(" loaded into catena_passages")
print(f"\n== query: {query}")
hits = by_meaning(conn, embed([query])[0]) + by_word(conn, query)
for channel, label in (("vec", "by meaning"), ("fts", "by word")):
print(f"\n {label}:")
for hit in [h for h in hits if h["channel"] == channel][:4]:
print(f" {hit['rank']}. [{hit['id']}] {hit['text']}")
# Fusion is a library call rather than a subprocess: by the time a hit
# reaches here it is already yours, and serialising it to hand to a
# program and back is work for nothing.
print("\n== fused")
text_of = {hit["id"]: hit["text"] for hit in hits}
for hit in catena.fuse_hits(hits, limit=5):
print(f" {hit['rank']}. [{hit['id']}] score {hit['score']:.3f} {text_of[hit['id']]}")
print(RERANKING)
return 0
RERANKING = """
Reciprocal rank fusion scores a passage by where it placed, not by what
it scored: 1/(k + rank) from each list that found it, added up. The two
searches return numbers that mean different things — a cosine distance
and a ts_rank_cd — and no scaling makes them comparable, so their ranks
are used and their scores thrown away. A passage both searches liked
beats one that either loved, which is the behaviour worth having.
This is fusion, and it is often called re-ranking, but a second thing
wears the same name. Retrieval uses a bi-encoder: passage and query are
embedded separately, which is what lets the passage vectors be computed
once and indexed, and is also what limits it — nothing in a passage's
vector knows what was asked.
A cross-encoder reads the query and the passage together and scores the
pair. It is far more accurate and far too slow to run over a corpus, so
it runs over the twenty or fifty candidates retrieval already found. The
two stack: retrieve both ways, fuse the lists, then hand the top of the
fused list to a cross-encoder for the final order. Nothing here loads a
model, so that step is yours — the fused list is what you would hand it.
"""
if __name__ == "__main__":
raise SystemExit(main())
examples/rust/.gitignore
+1
View File
@@ -0,0 +1 @@
target/
examples/rust/src/main.rs
+60 -9
View File
@@ -8,6 +8,13 @@
//!
//! cargo run -- book.md
//! cargo run # reads the sample below
//!
//! For a real book, convert one first — Gibbon's first volume is public
//! domain and has a few hundred pages of footnotes:
//!
//! curl -sL -o gibbon.epub https://www.gutenberg.org/cache/epub/731/pg731.epub
//! pandoc -f epub -t commonmark --wrap=none gibbon.epub -o gibbon.md
//! cargo run -- gibbon.md
use std::path::Path;
@@ -74,8 +81,8 @@ fn main() {
},
};
let chunks = chunkmd::chunk(&clean, &budgets);
println!("\n{} passage(s):", chunks.len());
for chunk in &chunks {
println!("\n{} passage(s), the first few:", chunks.len());
for chunk in chunks.iter().take(6) {
let chapter = chunk.chapter.as_deref().unwrap_or("—");
let opening: String = chunk.text.chars().take(64).collect();
println!(
@@ -87,18 +94,50 @@ fn main() {
);
}
// The references at the back. A passage whose role is `bibliography` is
// where they live; a note at the foot of a page is the other place, and
// `chunkmd` flags those so this does not have to guess.
let references: Vec<citefind::Reference> = chunks
// The references. A passage whose role is `bibliography` is one place
// they live; a page of notes at the foot of a chapter is the other, and
// it has no heading over it to find it by — which is why `chunkmd`
// flags those, so this does not have to guess.
let carrying: Vec<&corpus_core::Chunk> = chunks
.iter()
.filter(|chunk| {
chunk.role == corpus_core::Role::Bibliography
|| chunk
.flags
.iter()
.any(|flag| flag == corpus_core::chunk::flags::FOOTNOTE_CITATIONS)
})
.collect();
let references: Vec<citefind::Reference> = carrying
.iter()
.filter(|chunk| chunk.role == corpus_core::Role::Bibliography)
.flat_map(|chunk| citefind::read(&chunk.text))
.filter(|found| !found.continuation)
.collect();
println!("\n{} reference(s):", references.len());
for found in &references {
println!(
"\n{} passage(s) carry citation material, giving {} reference(s):",
carrying.len(),
references.len()
);
// The ones that named something, which is not all of them: a book whose
// notes cite in the classical short form — `Tacit. Annal. xiii. 5.` —
// gives mostly passages rather than works, and the record says as much
// rather than inventing a title to fill the field.
let with_work = references
.iter()
.filter(|found| found.title.is_some())
.count();
let named: Vec<&citefind::Reference> = references
.iter()
.filter(|found| !found.authors.is_empty())
.collect();
println!(
"{with_work} name a work and {} name an author; those:",
named.len()
);
for found in named.iter().take(6) {
let who = match found.authors.is_empty() {
true => "—".to_string(),
false => found.authors.join("; "),
@@ -116,4 +155,16 @@ fn main() {
println!(" could not read: {}", found.unsettled.join(", "));
}
}
// Worth saying plainly, because the numbers above look poor on a book
// like Gibbon and the reason is not the parser having failed quietly.
// His notes cite in the classical short form — `Tacit. Annal. xiii. 5.`
// — which names a passage rather than a work, so there is no title and
// no year to read and the record says which fields it could not fill
// rather than inventing them. `citation_prompt` and `merge_citation_reply`
// are there for a caller that wants a model to finish the job.
if references.len() > 20 && named.len() * 10 < references.len() {
println!("\n most of these are classical short cites — a passage");
println!(" rather than a work — there is nothing more to read in them.");
}
}
examples/shell/01-book-to-passages.sh
Executable
+93
View File
@@ -0,0 +1,93 @@
#!/bin/sh
# A book file to passages worth embedding, and what its notes cite.
#
# The starting point is Gibbon's first volume from Project Gutenberg: public
# domain, a real table of contents, and a few hundred pages of footnotes.
#
# examples/shell/01-book-to-passages.sh [work-dir]
#
# Needs pandoc and jq, and the catena commands on PATH.
set -eu
work="${1:-./work}"
mkdir -p "$work"
book="$work/gibbon.epub"
if [ ! -s "$book" ]; then
echo "fetching Gibbon vol. 1 from Project Gutenberg"
curl -sSL --max-time 120 -o "$book" \
"https://www.gutenberg.org/cache/epub/731/pg731.epub"
fi
# What the file actually is. Extensions lie, and a `.epub` that is really a
# zip of page scans fails four stages later, in a way that looks like a bug
# in the converter.
echo
echo "== what it is"
booksniff "$book" | jq -r '" \(.kind), \(.bytes) bytes"'
# A PDF gets one more question first: exit 0 from pdftriage means the text
# layer is not worth reading and the file has to be OCRed before any of this.
if [ "$(booksniff "$book" | jq -r .kind)" = "pdf" ] && pdftriage --quiet "$book"; then
echo " the text layer is garbled; OCR it before going on"
exit 0
fi
# What the file already says about itself — the publisher's own metadata,
# which beats anything guessed from a filename, and costs no lookup.
echo
echo "== what it says about itself"
bookmeta "$book" | jq -r '" \(.title)\n \(.authors | join("; "))\n \(.language // "?") · \(.date // "no date")"'
# The chapters the book declares, rather than any inferred from the text.
echo
echo "== the contents it declares"
tocfind "$book" --outline | sed -n '6,10p' | sed 's/^/ /'
echo " … $(tocfind "$book" | wc -l | tr -d ' ') entries"
# Convert, scrub, chunk. `commonmark` rather than `commonmark_x`, because the
# extended flavour writes anchors as bracketed spans and they end up in the
# passages. The scrub is what takes out the rest of the converter's leavings.
echo
echo "== passages"
pandoc -f epub -t commonmark --wrap=none "$book" -o "$work/book.md"
mdscrub < "$work/book.md" > "$work/clean.md"
chunkmd < "$work/clean.md" > "$work/chunks.ndjson"
jq -s -r '
" \(length) passages over \([.[].chapter] | unique | length) chapters",
" roles: \([.[].role] | group_by(.) | map("\(length) \(.[0])") | join(", "))",
" \([.[] | select(.flags | index("footnote_citations"))] | length) carry citation material"
' "$work/chunks.ndjson"
echo
echo " a passage, with where it came from:"
jq -s -r '[.[] | select(.role == "body" and .chapter != null)][40]
| " [\(.idx)] \(.tokens) tokens · \(.chapter)\n \(.text[0:150])…"' \
"$work/chunks.ndjson"
# The notes. A passage flagged `footnote_citations` is a page of them, and
# each is read on its own — the flag says which passages are worth the ask,
# which is the whole reason chunkmd sets it.
echo
echo "== what the notes cite"
: > "$work/references.ndjson"
jq -c 'select(.flags | index("footnote_citations")) | .text' "$work/chunks.ndjson" \
| while IFS= read -r line; do
printf '%s' "$line" | jq -r . | citefind >> "$work/references.ndjson" || true
done
jq -s -r '
" \(length) entries read",
" \([.[] | select(.title)] | length) name a work, \([.[] | select(.authors | length > 0)] | length) name an author",
" \([.[] | select(.unsettled | length > 0)] | length) came back with something unread"
' "$work/references.ndjson"
echo
echo " Gibbon cites in the classical short form — \"Tacit. Annal. xiii. 5.\" —"
echo " so most entries name the passage rather than a book, and say so:"
jq -r 'select(.unsettled | length > 0) | " \(.text[0:58]) → unread: \(.unsettled | join(", "))"' \
"$work/references.ndjson" | head -4
echo
echo "written to $work/"
examples/shell/02-video-to-chapters.sh
Executable
+109
View File
@@ -0,0 +1,109 @@
#!/bin/sh
# A recording to chapter marks, citable passages, and a copy that plays.
#
# Takes the captions a video already carries, normalises them to one cue
# stream, and turns that into chapter marks for a player and passages worth
# embedding — each keeping the second it was said at. Then asks whether the
# file itself would play on a phone, and prints the one ffmpeg call that
# fixes it.
#
# examples/shell/02-video-to-chapters.sh [url] [work-dir]
#
# The default is a NASA upload: a work of the US government, so public
# domain, and captioned by a person rather than by a model. Any URL yt-dlp
# can reach works, and any local subtitle file works too — skip to the
# cuenorm line and feed it your own.
#
# Needs jq, ffmpeg, and yt-dlp with impersonation support, which YouTube now
# requires before it will hand over captions:
#
# pipx install "yt-dlp[default,curl-cffi]"
#
# and the catena commands on PATH.
set -eu
url="${1:-https://www.youtube.com/watch?v=h0wLZJeYGxw}"
work="${2:-./work}"
mkdir -p "$work"
# Captions written by a person where there are any, and a model's otherwise.
# Automatic captions roll — each line is re-sent as it scrolls up the screen —
# so the same sentence arrives three times and every passage says it thrice.
if [ -z "$(find "$work" -name '*.vtt' -print -quit 2>/dev/null)" ]; then
echo "fetching captions"
yt-dlp --skip-download --write-subs --sub-langs 'en.*' --sub-format vtt \
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 ||
yt-dlp --skip-download --write-auto-subs --sub-langs 'en.*' --sub-format vtt \
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null
fi
# Where several tracks came back, the smallest is the written one: a rolling
# track repeats every line and is four or five times the size.
subs="$(find "$work" -name '*.vtt' -exec ls -S {} + | tail -1)"
[ -n "$subs" ] || { echo "no captions on that video; try another URL" >&2; exit 1; }
echo " captions: $(basename "$subs")"
echo
echo "== cues"
cuenorm "$subs" > "$work/cues.ndjson"
jq -s -r '" \(length) cues over \((map(.end) | max) / 60 | floor)m\((map(.end) | max) % 60 | floor)s"' "$work/cues.ndjson"
jq -r '" [\(.start | floor)s] \(.text[0:62])"' "$work/cues.ndjson" | head -3
# Where the subject changes, found from the words alone — no model, no
# network. The titles come out of each chapter's own first sentence, and are
# meant to be replaced by something better later.
echo
echo "== chapters"
cueseg < "$work/cues.ndjson" > "$work/chapters.ndjson"
jq -r '" \(.start | floor)s–\(.end | floor)s \(.title)"' "$work/chapters.ndjson"
# Cues packed to a size worth embedding, each carrying the seconds it spans,
# so a search hit can be played back at the moment it was said.
echo
echo "== passages"
cuepack < "$work/cues.ndjson" > "$work/passages.ndjson"
jq -s -r '" \(length) passages, \(map(.tokens) | add / length | floor) tokens on average"' "$work/passages.ndjson"
jq -s -r '.[0] | " [\(.start_time | floor)s–\(.end_time | floor)s] \(.text[0:150])…"' "$work/passages.ndjson"
echo
echo "== chapter marks, as a player would take them"
jq -r '"\(.start | floor) \(.title)"' "$work/chapters.ndjson" |
awk '{ t = $1; $1 = ""; printf " %02d:%02d:%02d%s\n", t/3600, (t%3600)/60, t%60, $0 }'
# And the file itself. A recording indexed but unplayable is half a job: a
# phone will not open VP9 in a WebM container, whatever the transcript says.
# Twenty seconds of it is enough to judge the streams, and spares the
# download.
echo
echo "== would it play on a phone"
video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
if [ -z "$video" ]; then
echo " fetching twenty seconds of it"
yt-dlp -f 'worstvideo[ext=webm]+worstaudio[ext=webm]/worst' \
--download-sections '*0-20' --force-keyframes-at-cuts \
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 || true
video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
fi
[ -n "$video" ] || { echo " nothing to judge"; exit 0; }
avplan --json "$video" | jq -r '
" \(.streams.container): \(.streams.video) video, \(.streams.audio // "no") audio",
(if .playable then " plays as it stands" else (.reasons[] | " · \(.)") end)'
# Exit 0 means there is a plan to print; 1 means the file already plays. The
# encoder is whatever this machine actually has — videotoolbox on a Mac,
# NVENC or VAAPI on a Linux box, libx264 where there is no card. Nothing is
# run: converting is expensive and irreversible enough to read first.
echo
if plan="$(avplan --output "$work/for-phone.mp4" "$video")"; then
echo " the one call that fixes it:"
echo
printf ' %s\n' "$plan"
echo
echo " run it with: eval \"\$(avplan --output out.mp4 FILE)\""
else
echo " nothing to convert"
fi
echo
echo "written to $work/"
examples/shell/03-search-and-rerank.sh
Executable
+178
View File
@@ -0,0 +1,178 @@
#!/bin/sh
# Passages into pgvector, then a search that asks twice and reconciles.
#
# Takes the passages example 01 wrote, embeds them, loads them into Postgres
# with a vector column and a text index, then searches both ways and fuses
# the two lists into one.
#
# examples/shell/01-book-to-passages.sh ./work
# examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
#
# Needs psql and jq, a Postgres with pgvector, an embedding server, and the
# catena commands on PATH:
#
# DATABASE_URL=postgresql://user@host/db
# EMB_HOST=http://localhost:11434 # ollama, or anything with its API
# EMB_MODEL=nomic-embed-text # 768 wide; bge-m3 is 1024
set -eu
query="${1:-why did the roman empire decline}"
work="${2:-./work}"
: "${DATABASE_URL:?set DATABASE_URL}"
host="${EMB_HOST:-http://localhost:11434}"
model="${EMB_MODEL:-nomic-embed-text}"
[ -s "$work/chunks.ndjson" ] || { echo "run 01-book-to-passages.sh first" >&2; exit 1; }
psql="psql -X -q -v ON_ERROR_STOP=1 $DATABASE_URL"
tab=$(printf '\t')
# How wide the model's vectors are. Asked rather than assumed, because the
# column is typed and a mismatch is a wall of errors at insert time.
dim=$(printf 'how wide is this\n' | emb --host "$host" --model "$model" | jq 'length')
echo "== $model returns $dim numbers per passage"
# One table. The vector is what the model said; the tsvector is Postgres's
# own, generated from the text so it cannot drift out of step with it.
$psql <<SQL
CREATE EXTENSION IF NOT EXISTS vector;
DROP TABLE IF EXISTS catena_passages;
CREATE TABLE catena_passages (
idx integer PRIMARY KEY,
chapter text,
body text NOT NULL,
tokens integer NOT NULL,
fts tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
vec vector($dim)
);
CREATE INDEX ON catena_passages USING gin (fts);
SQL
# Body passages only: a table of contents embeds as well as anything and
# retrieves for every query, which is worse than not having it.
jq -c 'select(.role == "body" and .tokens > 40)' "$work/chunks.ndjson" > "$work/body.ndjson"
echo "== embedding $(wc -l < "$work/body.ndjson" | tr -d ' ') passages"
# The newlines have to go before the text is a line: emb is line-oriented,
# so a passage containing one would arrive as two.
jq -r '.text | gsub("\\s+"; " ")' "$work/body.ndjson" > "$work/lines.txt"
emb --host "$host" --model "$model" --dim "$dim" < "$work/lines.txt" > "$work/vectors.ndjson"
# Line for line, which is emb's promise, so paste is a safe join.
jq -c '[.idx, .chapter, (.text | gsub("\\s+"; " ")), .tokens]' "$work/body.ndjson" > "$work/meta.ndjson"
paste -d '\t' "$work/meta.ndjson" "$work/vectors.ndjson" |
python3 -c '
import csv, json, sys
out = csv.writer(sys.stdout, delimiter="\t", quoting=csv.QUOTE_NONE, escapechar="\\")
for line in sys.stdin:
meta, vector = line.rstrip("\n").split("\t")
idx, chapter, body, tokens = json.loads(meta)
out.writerow([idx, chapter or "", body, tokens, vector])
' > "$work/rows.tsv"
$psql -c "\copy catena_passages (idx, chapter, body, tokens, vec) FROM '$work/rows.tsv'"
$psql -c "CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops);"
$psql -tAc "SELECT ' ' || count(*) || ' passages loaded' FROM catena_passages"
# psql's rows into hit records. `rrf` wants an id, which channel found it,
# and what rank that channel gave it; the text is carried along so the
# output is readable. Written in Python because the passages contain
# quotation marks and a shell printf would not escape them.
hits() {
python3 -c '
import json, sys
channel = sys.argv[1]
for line in sys.stdin:
line = line.rstrip("\n")
if not line.strip():
continue
ident, rank, text = line.split("\t", 2)
print(json.dumps({"v": 1, "id": ident, "channel": channel,
"rank": int(rank), "score": 0.0, "text": text}))
' "$1"
}
# ---------------------------------------------------------------------------
# Two searches. They disagree, and that is the point.
# ---------------------------------------------------------------------------
echo
echo "== query: $query"
printf '%s\n' "$query" | emb --host "$host" --model "$model" --dim "$dim" | tr -d '\n' > "$work/query.vec"
qvec=$(cat "$work/query.vec")
# Nearest by meaning. `<=>` is cosine distance, so smaller is closer, and a
# passage about "the fall of Rome" ranks even though it shares no words with
# the query.
$psql -tAF"$tab" -c "
SELECT idx, row_number() OVER (ORDER BY vec <=> '$qvec'::vector),
regexp_replace(left(body, 68), '\s+', ' ', 'g')
FROM catena_passages
ORDER BY vec <=> '$qvec'::vector
LIMIT 10" | hits vec > "$work/hits.ndjson"
# Nearest by word. Finds the passage that says "decline" where the vector
# search drifted to a paraphrase, and finds nothing at all for a query whose
# words are not in the book.
$psql -tAF"$tab" -c "
SELECT idx,
row_number() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', \$\$$query\$\$)) DESC),
regexp_replace(left(body, 68), '\s+', ' ', 'g')
FROM catena_passages
WHERE fts @@ websearch_to_tsquery('english', \$\$$query\$\$)
ORDER BY 2
LIMIT 10" | hits fts >> "$work/hits.ndjson"
echo
echo " by meaning:"
jq -r 'select(.channel == "vec") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
echo
echo " by word:"
jq -r 'select(.channel == "fts") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
# ---------------------------------------------------------------------------
# Reconciling them.
# ---------------------------------------------------------------------------
echo
echo "== fused"
rrf --limit 5 < "$work/hits.ndjson" > "$work/fused.ndjson"
jq -s -r --slurpfile hits "$work/hits.ndjson" '
($hits | INDEX(.id)) as $by_id
| .[] | " \(.rank). [\(.id)] score \(.score * 1000 | floor / 1000) \($by_id[.id].text)"
' "$work/fused.ndjson"
cat <<'NOTE'
Reciprocal rank fusion scores a passage by where it placed, not by what
it scored: 1/(k + rank) from each list that found it, added up. The two
searches return numbers that mean different things — a cosine distance
and a ts_rank_cd — and no amount of scaling makes them comparable, so
their ranks are used and their scores thrown away. A passage both
searches liked beats one that either loved, which is the behaviour worth
having: agreement between two different ways of looking is evidence, and
a high score from one is only ever evidence about that one.
The run above shows it happening. The word search's first hit is the
front matter, which says "Roman Empire" more often than any chapter of
the book does; the vector search never offers it. Fusion pushes it down
and lifts the one passage both searches found.
The constant k, 60 by default, is how much a first place is worth over a
second. Raise it and the lists flatten toward equal weight; lower it and
the top of each list dominates. `--weight vec=2` trusts one channel more
than the other, which is the knob to reach for when one of them is
consistently better on your corpus.
This is fusion, and it is often called re-ranking, but there is a second
thing wearing the same name. Retrieval uses a bi-encoder: the passage
and the query are embedded separately, so the passage vectors can be
computed once and indexed, and a search is one vector comparison against
an index. That independence is what makes it fast and is also what
limits it — nothing in the passage's vector knows what was asked.
A cross-encoder reads the query and the passage together and scores the
pair. It is far more accurate and far too slow to run over a corpus, so
it runs over the twenty or fifty candidates retrieval already found. In
a pipeline the two stack: retrieve both ways, fuse the lists as above,
then hand the top of the fused list to a cross-encoder for the final
order. Nothing here does that step — it needs a model, and these tools
do not load models — but the fused list is what you would hand it.
NOTE
examples/typescript/package.json
+15
View File
@@ -0,0 +1,15 @@
{
"name": "catena-example",
"private": true,
"type": "module",
"scripts": {
"chapters": "node --experimental-strip-types src/episode-to-chapters.ts",
"search": "node --experimental-strip-types src/search-and-rerank.ts"
},
"dependencies": {
"catena": "file:../../crates/bindings/npm/dist"
},
"devDependencies": {
"@types/node": "^22"
}
}
examples/typescript/src/episode-to-chapters.ts
+95
View File
@@ -0,0 +1,95 @@
// A recording's captions into chapter marks and citable passages.
//
// What a player needs from an episode: somewhere to jump to, and passages
// that carry the second they were said at so a search result can be played
// rather than just quoted.
//
// npm install
// npm run chapters -- path/to/captions.vtt
//
// With no argument it reads the sample below, which is a few cues of a NASA
// broadcast — a work of the US government, so public domain.
import { readFileSync } from "node:fs";
import { parseSubtitles, segmentCues, packCues, countTokens } from "catena";
const SAMPLE = `WEBVTT
00:00:00.000 --> 00:00:04.200
Hey there, and welcome to NASA's Curious Universe.
00:00:04.200 --> 00:00:08.900
Our universe is a wild and wonderful place, and in this
show, NASA is your tour guide.
00:00:08.900 --> 00:00:15.100
Today we are going to talk about how we find planets
around other stars, which is harder than it sounds.
00:00:15.100 --> 00:00:21.800
A planet is small and dark and its star is enormous and
bright, so you almost never see the planet itself.
00:00:21.800 --> 00:00:28.400
Instead you watch the star, and you wait for it to dim
by about one part in ten thousand.
00:00:28.400 --> 00:00:34.000
That dimming is a transit, and it is how most of the
planets we know about were found.
`;
const path = process.argv[2];
const content = path ? readFileSync(path, "utf8") : SAMPLE;
// One cue stream, whatever the format was. `keepRepeats` is off, which is
// what throws away the re-sent lines of a rolling automatic caption track —
// leave it on and every passage says each sentence three times.
const cues = parseSubtitles(content);
const runsTo = Math.max(...cues.map((cue) => cue.end));
console.log(`== ${cues.length} cues over ${Math.round(runsTo)}s`);
for (const cue of cues.slice(0, 3)) {
console.log(` [${cue.start.toFixed(1)}s] ${cue.text.slice(0, 58)}`);
}
// Where the subject changes, from the words alone — no model, no network.
// The titles come out of each chapter's own first sentence and are meant to
// be replaced by something better later.
const chapters = segmentCues(cues);
console.log(`\n== ${chapters.length} chapter(s)`);
for (const chapter of chapters) {
console.log(` ${stamp(chapter.start)} ${chapter.title}`);
}
// Cues packed to a size worth embedding, each keeping the seconds it spans.
const passages = packCues(cues);
console.log(`\n== ${passages.length} passage(s)`);
for (const passage of passages) {
console.log(
` [${stamp(passage.start_time ?? 0)}–${stamp(passage.end_time ?? 0)}] ` +
`${passage.tokens} tokens`,
);
console.log(` ${passage.text.slice(0, 92)}…`);
}
// What a player would take: a cue sheet, and a deep link per passage.
console.log("\n== what the player gets");
for (const chapter of chapters) {
console.log(` ${stamp(chapter.start)} ${chapter.title}`);
}
console.log("\n== what a search result would link to");
for (const passage of passages.slice(0, 2)) {
const at = Math.floor(passage.start_time ?? 0);
console.log(` ?t=${at} ${passage.text.slice(0, 64)}…`);
}
// The same token count the chunker used, so a budget worked out here and a
// budget enforced there agree.
console.log(`\n ${countTokens(cues.map((c) => c.text).join(" "))} tokens of transcript in all`);
function stamp(seconds: number): string {
const whole = Math.floor(seconds);
const mm = String(Math.floor(whole / 60)).padStart(2, "0");
const ss = String(whole % 60).padStart(2, "0");
return `${String(Math.floor(whole / 3600)).padStart(2, "0")}:${mm}:${ss}`;
}
examples/typescript/src/search-and-rerank.ts
+96
View File
@@ -0,0 +1,96 @@
// Two searches, reconciled — the app-side half of hybrid retrieval.
//
// Fusion is the one thing in this suite worth calling in process rather than
// as a subprocess: by the time a hit reaches your code it is already yours,
// and serialising it to hand to a program and back is work for nothing.
//
// npm install
// npm run search
//
// The hits below stand in for two queries you would run against your own
// store — pgvector and a text index, Elasticsearch and a KNN service, or a
// vector database and whatever else. What they came from does not matter
// here; only where each of them placed does.
import { fuseHits, type Hit } from "catena";
// `payload` is whatever you want carried alongside — rrf hands it back
// untouched, so a hit can arrive here with the row it came from attached.
const hit = (id: string, channel: string, rank: number, score: number): Hit => ({
v: 1,
id,
channel,
rank,
score,
payload: {},
});
// Nearest in the embedding space. Finds the passage about the fall of Rome
// for a query about decline, sharing not one word with it.
const byMeaning: Hit[] = [
hit("p39", "vec", 1, 0.21),
hit("p127", "vec", 2, 0.24),
hit("p462", "vec", 3, 0.26),
hit("p177", "vec", 4, 0.29),
];
// Nearest by the words themselves. Finds the passage that says the word
// where the vectors drifted to a paraphrase — and finds the front matter,
// which says "Roman Empire" more often than any chapter of the book does.
const byWord: Hit[] = [
hit("p0", "fts", 1, 0.68),
hit("p462", "fts", 2, 0.51),
hit("p8", "fts", 3, 0.44),
hit("p71", "fts", 4, 0.4),
];
const text: Record<string, string> = {
p0: "The Project Gutenberg eBook of History of the Decline and Fall…",
p8: "In imitation of his classical models, Gibbon places Rome as the…",
p39: "In the second century of the Christian Æra, the empire of Rome…",
p71: "Tacit. Annal. iv. 5. Those who fix a regular proportion…",
p127: "It was by such institutions that the nations of the empire…",
p177: "Among many other passages, see Pliny on the extent of the…",
p462: "The received calculation of Varro assigns to the foundation…",
};
for (const [label, hits] of [["by meaning", byMeaning], ["by word", byWord]] as const) {
console.log(`\n== ${label}`);
for (const hit of hits) {
console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(2)} ${text[hit.id]}`);
}
}
// The two score columns above mean different things — a cosine distance and
// a ts_rank_cd — and no scaling makes them comparable. So the ranks are used
// and the scores thrown away.
console.log("\n== fused");
for (const hit of fuseHits([...byMeaning, ...byWord], undefined, undefined, 5)) {
console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)} ${text[hit.id]}`);
}
// One channel can be trusted more than the other where a corpus warrants it.
console.log("\n== fused, trusting the vectors twice as much");
for (const hit of fuseHits([...byMeaning, ...byWord], undefined, { vec: 2 }, 5)) {
console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)} ${text[hit.id]}`);
}
console.log(`
Reciprocal rank fusion scores a hit by where it placed, not by what it
scored: 1/(k + rank) from each list that found it, added up. p462 is
third in one list and second in the other and wins, because agreement
between two different ways of looking is evidence; p0 is first in one
and absent from the other, and drops.
This is fusion, and it is often called re-ranking, but a second thing
wears the same name. Retrieval uses a bi-encoder: the passage and the
query are embedded separately, which is what lets passage vectors be
computed once and indexed — and is also what limits it, since nothing in
a passage's vector knows what was asked.
A cross-encoder reads the query and the passage together and scores the
pair. Far more accurate, far too slow to run over a corpus, so it runs
over the twenty or fifty candidates retrieval already found. The two
stack: retrieve both ways, fuse as above, then hand the top of the fused
list to a cross-encoder for the final order. Nothing here loads a model,
so that step is yours — the fused list is what you would hand it.`);
examples/typescript/tsconfig.json
+15
View File
@@ -0,0 +1,15 @@
{
"compilerOptions": {
"target": "es2022",
"module": "nodenext",
"moduleResolution": "nodenext",
"lib": ["es2023"],
"types": ["node"],
"strict": true,
"noEmit": true,
"allowImportingTsExtensions": true,
"rewriteRelativeImportExtensions": true,
"verbatimModuleSyntax": true
},
"include": ["src"]
}