mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
examples: programs that run, in four languages
Eight of them, each doing something a pipeline actually has to do, and each starting from something anyone can fetch — a public domain book from Project Gutenberg, a NASA broadcast, or a sample carried in the file. Shell, because the tools are filters first: a book to passages and what its notes cite; a recording to chapter marks, citable passages and the one ffmpeg call that makes it play on a phone; passages into pgvector and a search that asks twice and reconciles. Rust, for the case where the caller already holds the text and a subprocess would be serialising for nothing. Python, for the shape a pipeline stage takes once the reading is a library call. TypeScript, for the app side — chapter marks and deep links, and fusing two rankings in process. Three of them end with what re-ranking means, because it is two different things under one name: fusing ranked lists, which is what `rrf` does and needs no model, and a cross-encoder scoring query and passage together, which is accurate, slow, and runs over what retrieval already found.
This commit is contained in:
funkpower
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/bin/sh
|
||||
# A book file to passages worth embedding, and what its notes cite.
|
||||
#
|
||||
# The starting point is Gibbon's first volume from Project Gutenberg: public
|
||||
# domain, a real table of contents, and a few hundred pages of footnotes.
|
||||
#
|
||||
# examples/shell/01-book-to-passages.sh [work-dir]
|
||||
#
|
||||
# Needs pandoc and jq, and the catena commands on PATH.
|
||||
set -eu
|
||||
|
||||
work="${1:-./work}"
|
||||
mkdir -p "$work"
|
||||
book="$work/gibbon.epub"
|
||||
|
||||
if [ ! -s "$book" ]; then
|
||||
echo "fetching Gibbon vol. 1 from Project Gutenberg"
|
||||
curl -sSL --max-time 120 -o "$book" \
|
||||
"https://www.gutenberg.org/cache/epub/731/pg731.epub"
|
||||
fi
|
||||
|
||||
# What the file actually is. Extensions lie, and a `.epub` that is really a
|
||||
# zip of page scans fails four stages later, in a way that looks like a bug
|
||||
# in the converter.
|
||||
echo
|
||||
echo "== what it is"
|
||||
booksniff "$book" | jq -r '" \(.kind), \(.bytes) bytes"'
|
||||
|
||||
# A PDF gets one more question first: exit 0 from pdftriage means the text
|
||||
# layer is not worth reading and the file has to be OCRed before any of this.
|
||||
if [ "$(booksniff "$book" | jq -r .kind)" = "pdf" ] && pdftriage --quiet "$book"; then
|
||||
echo " the text layer is garbled; OCR it before going on"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# What the file already says about itself — the publisher's own metadata,
|
||||
# which beats anything guessed from a filename, and costs no lookup.
|
||||
echo
|
||||
echo "== what it says about itself"
|
||||
bookmeta "$book" | jq -r '" \(.title)\n \(.authors | join("; "))\n \(.language // "?") · \(.date // "no date")"'
|
||||
|
||||
# The chapters the book declares, rather than any inferred from the text.
|
||||
echo
|
||||
echo "== the contents it declares"
|
||||
tocfind "$book" --outline | sed -n '6,10p' | sed 's/^/ /'
|
||||
echo " … $(tocfind "$book" | wc -l | tr -d ' ') entries"
|
||||
|
||||
# Convert, scrub, chunk. `commonmark` rather than `commonmark_x`, because the
|
||||
# extended flavour writes anchors as bracketed spans and they end up in the
|
||||
# passages. The scrub is what takes out the rest of the converter's leavings.
|
||||
echo
|
||||
echo "== passages"
|
||||
pandoc -f epub -t commonmark --wrap=none "$book" -o "$work/book.md"
|
||||
mdscrub < "$work/book.md" > "$work/clean.md"
|
||||
chunkmd < "$work/clean.md" > "$work/chunks.ndjson"
|
||||
|
||||
jq -s -r '
|
||||
" \(length) passages over \([.[].chapter] | unique | length) chapters",
|
||||
" roles: \([.[].role] | group_by(.) | map("\(length) \(.[0])") | join(", "))",
|
||||
" \([.[] | select(.flags | index("footnote_citations"))] | length) carry citation material"
|
||||
' "$work/chunks.ndjson"
|
||||
|
||||
echo
|
||||
echo " a passage, with where it came from:"
|
||||
jq -s -r '[.[] | select(.role == "body" and .chapter != null)][40]
|
||||
| " [\(.idx)] \(.tokens) tokens · \(.chapter)\n \(.text[0:150])…"' \
|
||||
"$work/chunks.ndjson"
|
||||
|
||||
# The notes. A passage flagged `footnote_citations` is a page of them, and
|
||||
# each is read on its own — the flag says which passages are worth the ask,
|
||||
# which is the whole reason chunkmd sets it.
|
||||
echo
|
||||
echo "== what the notes cite"
|
||||
: > "$work/references.ndjson"
|
||||
jq -c 'select(.flags | index("footnote_citations")) | .text' "$work/chunks.ndjson" \
|
||||
| while IFS= read -r line; do
|
||||
printf '%s' "$line" | jq -r . | citefind >> "$work/references.ndjson" || true
|
||||
done
|
||||
|
||||
jq -s -r '
|
||||
" \(length) entries read",
|
||||
" \([.[] | select(.title)] | length) name a work, \([.[] | select(.authors | length > 0)] | length) name an author",
|
||||
" \([.[] | select(.unsettled | length > 0)] | length) came back with something unread"
|
||||
' "$work/references.ndjson"
|
||||
|
||||
echo
|
||||
echo " Gibbon cites in the classical short form — \"Tacit. Annal. xiii. 5.\" —"
|
||||
echo " so most entries name the passage rather than a book, and say so:"
|
||||
jq -r 'select(.unsettled | length > 0) | " \(.text[0:58]) → unread: \(.unsettled | join(", "))"' \
|
||||
"$work/references.ndjson" | head -4
|
||||
|
||||
echo
|
||||
echo "written to $work/"
|
||||
Executable
+109
@@ -0,0 +1,109 @@
|
||||
#!/bin/sh
|
||||
# A recording to chapter marks, citable passages, and a copy that plays.
|
||||
#
|
||||
# Takes the captions a video already carries, normalises them to one cue
|
||||
# stream, and turns that into chapter marks for a player and passages worth
|
||||
# embedding — each keeping the second it was said at. Then asks whether the
|
||||
# file itself would play on a phone, and prints the one ffmpeg call that
|
||||
# fixes it.
|
||||
#
|
||||
# examples/shell/02-video-to-chapters.sh [url] [work-dir]
|
||||
#
|
||||
# The default is a NASA upload: a work of the US government, so public
|
||||
# domain, and captioned by a person rather than by a model. Any URL yt-dlp
|
||||
# can reach works, and any local subtitle file works too — skip to the
|
||||
# cuenorm line and feed it your own.
|
||||
#
|
||||
# Needs jq, ffmpeg, and yt-dlp with impersonation support, which YouTube now
|
||||
# requires before it will hand over captions:
|
||||
#
|
||||
# pipx install "yt-dlp[default,curl-cffi]"
|
||||
#
|
||||
# and the catena commands on PATH.
|
||||
set -eu
|
||||
|
||||
url="${1:-https://www.youtube.com/watch?v=h0wLZJeYGxw}"
|
||||
work="${2:-./work}"
|
||||
mkdir -p "$work"
|
||||
|
||||
# Captions written by a person where there are any, and a model's otherwise.
|
||||
# Automatic captions roll — each line is re-sent as it scrolls up the screen —
|
||||
# so the same sentence arrives three times and every passage says it thrice.
|
||||
if [ -z "$(find "$work" -name '*.vtt' -print -quit 2>/dev/null)" ]; then
|
||||
echo "fetching captions"
|
||||
yt-dlp --skip-download --write-subs --sub-langs 'en.*' --sub-format vtt \
|
||||
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 ||
|
||||
yt-dlp --skip-download --write-auto-subs --sub-langs 'en.*' --sub-format vtt \
|
||||
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null
|
||||
fi
|
||||
|
||||
# Where several tracks came back, the smallest is the written one: a rolling
|
||||
# track repeats every line and is four or five times the size.
|
||||
subs="$(find "$work" -name '*.vtt' -exec ls -S {} + | tail -1)"
|
||||
[ -n "$subs" ] || { echo "no captions on that video; try another URL" >&2; exit 1; }
|
||||
echo " captions: $(basename "$subs")"
|
||||
|
||||
echo
|
||||
echo "== cues"
|
||||
cuenorm "$subs" > "$work/cues.ndjson"
|
||||
jq -s -r '" \(length) cues over \((map(.end) | max) / 60 | floor)m\((map(.end) | max) % 60 | floor)s"' "$work/cues.ndjson"
|
||||
jq -r '" [\(.start | floor)s] \(.text[0:62])"' "$work/cues.ndjson" | head -3
|
||||
|
||||
# Where the subject changes, found from the words alone — no model, no
|
||||
# network. The titles come out of each chapter's own first sentence, and are
|
||||
# meant to be replaced by something better later.
|
||||
echo
|
||||
echo "== chapters"
|
||||
cueseg < "$work/cues.ndjson" > "$work/chapters.ndjson"
|
||||
jq -r '" \(.start | floor)s–\(.end | floor)s \(.title)"' "$work/chapters.ndjson"
|
||||
|
||||
# Cues packed to a size worth embedding, each carrying the seconds it spans,
|
||||
# so a search hit can be played back at the moment it was said.
|
||||
echo
|
||||
echo "== passages"
|
||||
cuepack < "$work/cues.ndjson" > "$work/passages.ndjson"
|
||||
jq -s -r '" \(length) passages, \(map(.tokens) | add / length | floor) tokens on average"' "$work/passages.ndjson"
|
||||
jq -s -r '.[0] | " [\(.start_time | floor)s–\(.end_time | floor)s] \(.text[0:150])…"' "$work/passages.ndjson"
|
||||
|
||||
echo
|
||||
echo "== chapter marks, as a player would take them"
|
||||
jq -r '"\(.start | floor) \(.title)"' "$work/chapters.ndjson" |
|
||||
awk '{ t = $1; $1 = ""; printf " %02d:%02d:%02d%s\n", t/3600, (t%3600)/60, t%60, $0 }'
|
||||
|
||||
# And the file itself. A recording indexed but unplayable is half a job: a
|
||||
# phone will not open VP9 in a WebM container, whatever the transcript says.
|
||||
# Twenty seconds of it is enough to judge the streams, and spares the
|
||||
# download.
|
||||
echo
|
||||
echo "== would it play on a phone"
|
||||
video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
|
||||
if [ -z "$video" ]; then
|
||||
echo " fetching twenty seconds of it"
|
||||
yt-dlp -f 'worstvideo[ext=webm]+worstaudio[ext=webm]/worst' \
|
||||
--download-sections '*0-20' --force-keyframes-at-cuts \
|
||||
--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 || true
|
||||
video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
|
||||
fi
|
||||
[ -n "$video" ] || { echo " nothing to judge"; exit 0; }
|
||||
|
||||
avplan --json "$video" | jq -r '
|
||||
" \(.streams.container): \(.streams.video) video, \(.streams.audio // "no") audio",
|
||||
(if .playable then " plays as it stands" else (.reasons[] | " · \(.)") end)'
|
||||
|
||||
# Exit 0 means there is a plan to print; 1 means the file already plays. The
|
||||
# encoder is whatever this machine actually has — videotoolbox on a Mac,
|
||||
# NVENC or VAAPI on a Linux box, libx264 where there is no card. Nothing is
|
||||
# run: converting is expensive and irreversible enough to read first.
|
||||
echo
|
||||
if plan="$(avplan --output "$work/for-phone.mp4" "$video")"; then
|
||||
echo " the one call that fixes it:"
|
||||
echo
|
||||
printf ' %s\n' "$plan"
|
||||
echo
|
||||
echo " run it with: eval \"\$(avplan --output out.mp4 FILE)\""
|
||||
else
|
||||
echo " nothing to convert"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "written to $work/"
|
||||
Executable
+178
@@ -0,0 +1,178 @@
|
||||
#!/bin/sh
|
||||
# Passages into pgvector, then a search that asks twice and reconciles.
|
||||
#
|
||||
# Takes the passages example 01 wrote, embeds them, loads them into Postgres
|
||||
# with a vector column and a text index, then searches both ways and fuses
|
||||
# the two lists into one.
|
||||
#
|
||||
# examples/shell/01-book-to-passages.sh ./work
|
||||
# examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
|
||||
#
|
||||
# Needs psql and jq, a Postgres with pgvector, an embedding server, and the
|
||||
# catena commands on PATH:
|
||||
#
|
||||
# DATABASE_URL=postgresql://user@host/db
|
||||
# EMB_HOST=http://localhost:11434 # ollama, or anything with its API
|
||||
# EMB_MODEL=nomic-embed-text # 768 wide; bge-m3 is 1024
|
||||
set -eu
|
||||
|
||||
query="${1:-why did the roman empire decline}"
|
||||
work="${2:-./work}"
|
||||
: "${DATABASE_URL:?set DATABASE_URL}"
|
||||
host="${EMB_HOST:-http://localhost:11434}"
|
||||
model="${EMB_MODEL:-nomic-embed-text}"
|
||||
|
||||
[ -s "$work/chunks.ndjson" ] || { echo "run 01-book-to-passages.sh first" >&2; exit 1; }
|
||||
psql="psql -X -q -v ON_ERROR_STOP=1 $DATABASE_URL"
|
||||
tab=$(printf '\t')
|
||||
|
||||
# How wide the model's vectors are. Asked rather than assumed, because the
|
||||
# column is typed and a mismatch is a wall of errors at insert time.
|
||||
dim=$(printf 'how wide is this\n' | emb --host "$host" --model "$model" | jq 'length')
|
||||
echo "== $model returns $dim numbers per passage"
|
||||
|
||||
# One table. The vector is what the model said; the tsvector is Postgres's
|
||||
# own, generated from the text so it cannot drift out of step with it.
|
||||
$psql <<SQL
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
DROP TABLE IF EXISTS catena_passages;
|
||||
CREATE TABLE catena_passages (
|
||||
idx integer PRIMARY KEY,
|
||||
chapter text,
|
||||
body text NOT NULL,
|
||||
tokens integer NOT NULL,
|
||||
fts tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
|
||||
vec vector($dim)
|
||||
);
|
||||
CREATE INDEX ON catena_passages USING gin (fts);
|
||||
SQL
|
||||
|
||||
# Body passages only: a table of contents embeds as well as anything and
|
||||
# retrieves for every query, which is worse than not having it.
|
||||
jq -c 'select(.role == "body" and .tokens > 40)' "$work/chunks.ndjson" > "$work/body.ndjson"
|
||||
echo "== embedding $(wc -l < "$work/body.ndjson" | tr -d ' ') passages"
|
||||
|
||||
# The newlines have to go before the text is a line: emb is line-oriented,
|
||||
# so a passage containing one would arrive as two.
|
||||
jq -r '.text | gsub("\\s+"; " ")' "$work/body.ndjson" > "$work/lines.txt"
|
||||
emb --host "$host" --model "$model" --dim "$dim" < "$work/lines.txt" > "$work/vectors.ndjson"
|
||||
|
||||
# Line for line, which is emb's promise, so paste is a safe join.
|
||||
jq -c '[.idx, .chapter, (.text | gsub("\\s+"; " ")), .tokens]' "$work/body.ndjson" > "$work/meta.ndjson"
|
||||
paste -d '\t' "$work/meta.ndjson" "$work/vectors.ndjson" |
|
||||
python3 -c '
|
||||
import csv, json, sys
|
||||
out = csv.writer(sys.stdout, delimiter="\t", quoting=csv.QUOTE_NONE, escapechar="\\")
|
||||
for line in sys.stdin:
|
||||
meta, vector = line.rstrip("\n").split("\t")
|
||||
idx, chapter, body, tokens = json.loads(meta)
|
||||
out.writerow([idx, chapter or "", body, tokens, vector])
|
||||
' > "$work/rows.tsv"
|
||||
|
||||
$psql -c "\copy catena_passages (idx, chapter, body, tokens, vec) FROM '$work/rows.tsv'"
|
||||
$psql -c "CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops);"
|
||||
$psql -tAc "SELECT ' ' || count(*) || ' passages loaded' FROM catena_passages"
|
||||
|
||||
# psql's rows into hit records. `rrf` wants an id, which channel found it,
|
||||
# and what rank that channel gave it; the text is carried along so the
|
||||
# output is readable. Written in Python because the passages contain
|
||||
# quotation marks and a shell printf would not escape them.
|
||||
hits() {
|
||||
python3 -c '
|
||||
import json, sys
|
||||
channel = sys.argv[1]
|
||||
for line in sys.stdin:
|
||||
line = line.rstrip("\n")
|
||||
if not line.strip():
|
||||
continue
|
||||
ident, rank, text = line.split("\t", 2)
|
||||
print(json.dumps({"v": 1, "id": ident, "channel": channel,
|
||||
"rank": int(rank), "score": 0.0, "text": text}))
|
||||
' "$1"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Two searches. They disagree, and that is the point.
|
||||
# ---------------------------------------------------------------------------
|
||||
echo
|
||||
echo "== query: $query"
|
||||
printf '%s\n' "$query" | emb --host "$host" --model "$model" --dim "$dim" | tr -d '\n' > "$work/query.vec"
|
||||
qvec=$(cat "$work/query.vec")
|
||||
|
||||
# Nearest by meaning. `<=>` is cosine distance, so smaller is closer, and a
|
||||
# passage about "the fall of Rome" ranks even though it shares no words with
|
||||
# the query.
|
||||
$psql -tAF"$tab" -c "
|
||||
SELECT idx, row_number() OVER (ORDER BY vec <=> '$qvec'::vector),
|
||||
regexp_replace(left(body, 68), '\s+', ' ', 'g')
|
||||
FROM catena_passages
|
||||
ORDER BY vec <=> '$qvec'::vector
|
||||
LIMIT 10" | hits vec > "$work/hits.ndjson"
|
||||
|
||||
# Nearest by word. Finds the passage that says "decline" where the vector
|
||||
# search drifted to a paraphrase, and finds nothing at all for a query whose
|
||||
# words are not in the book.
|
||||
$psql -tAF"$tab" -c "
|
||||
SELECT idx,
|
||||
row_number() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', \$\$$query\$\$)) DESC),
|
||||
regexp_replace(left(body, 68), '\s+', ' ', 'g')
|
||||
FROM catena_passages
|
||||
WHERE fts @@ websearch_to_tsquery('english', \$\$$query\$\$)
|
||||
ORDER BY 2
|
||||
LIMIT 10" | hits fts >> "$work/hits.ndjson"
|
||||
|
||||
echo
|
||||
echo " by meaning:"
|
||||
jq -r 'select(.channel == "vec") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
|
||||
echo
|
||||
echo " by word:"
|
||||
jq -r 'select(.channel == "fts") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reconciling them.
|
||||
# ---------------------------------------------------------------------------
|
||||
echo
|
||||
echo "== fused"
|
||||
rrf --limit 5 < "$work/hits.ndjson" > "$work/fused.ndjson"
|
||||
jq -s -r --slurpfile hits "$work/hits.ndjson" '
|
||||
($hits | INDEX(.id)) as $by_id
|
||||
| .[] | " \(.rank). [\(.id)] score \(.score * 1000 | floor / 1000) \($by_id[.id].text)"
|
||||
' "$work/fused.ndjson"
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
Reciprocal rank fusion scores a passage by where it placed, not by what
|
||||
it scored: 1/(k + rank) from each list that found it, added up. The two
|
||||
searches return numbers that mean different things — a cosine distance
|
||||
and a ts_rank_cd — and no amount of scaling makes them comparable, so
|
||||
their ranks are used and their scores thrown away. A passage both
|
||||
searches liked beats one that either loved, which is the behaviour worth
|
||||
having: agreement between two different ways of looking is evidence, and
|
||||
a high score from one is only ever evidence about that one.
|
||||
|
||||
The run above shows it happening. The word search's first hit is the
|
||||
front matter, which says "Roman Empire" more often than any chapter of
|
||||
the book does; the vector search never offers it. Fusion pushes it down
|
||||
and lifts the one passage both searches found.
|
||||
|
||||
The constant k, 60 by default, is how much a first place is worth over a
|
||||
second. Raise it and the lists flatten toward equal weight; lower it and
|
||||
the top of each list dominates. `--weight vec=2` trusts one channel more
|
||||
than the other, which is the knob to reach for when one of them is
|
||||
consistently better on your corpus.
|
||||
|
||||
This is fusion, and it is often called re-ranking, but there is a second
|
||||
thing wearing the same name. Retrieval uses a bi-encoder: the passage
|
||||
and the query are embedded separately, so the passage vectors can be
|
||||
computed once and indexed, and a search is one vector comparison against
|
||||
an index. That independence is what makes it fast and is also what
|
||||
limits it — nothing in the passage's vector knows what was asked.
|
||||
|
||||
A cross-encoder reads the query and the passage together and scores the
|
||||
pair. It is far more accurate and far too slow to run over a corpus, so
|
||||
it runs over the twenty or fifty candidates retrieval already found. In
|
||||
a pipeline the two stack: retrieve both ways, fuse the lists as above,
|
||||
then hand the top of the fused list to a cross-encoder for the final
|
||||
order. Nothing here does that step — it needs a model, and these tools
|
||||
do not load models — but the fused list is what you would hand it.
|
||||
NOTE
|
||||
Reference in New Issue
Block a user