From 58be244478ffc109730f87a835351ece8ca22b30 Mon Sep 17 00:00:00 2001 From: funkpower Date: Mon, 21 Sep 2026 08:12:13 -0400 Subject: [PATCH] examples: programs that run, in four languages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight of them, each doing something a pipeline actually has to do, and each starting from something anyone can fetch — a public domain book from Project Gutenberg, a NASA broadcast, or a sample carried in the file. Shell, because the tools are filters first: a book to passages and what its notes cite; a recording to chapter marks, citable passages and the one ffmpeg call that makes it play on a phone; passages into pgvector and a search that asks twice and reconciles. Rust, for the case where the caller already holds the text and a subprocess would be serialising for nothing. Python, for the shape a pipeline stage takes once the reading is a library call. TypeScript, for the app side — chapter marks and deep links, and fusing two rankings in process. Three of them end with what re-ranking means, because it is two different things under one name: fusing ranked lists, which is what `rrf` does and needs no model, and a cross-encoder scoring query and passage together, which is accurate, slow, and runs over what retrieval already found. --- README.md | 4 + examples/.gitignore | 3 + examples/README.md | 125 ++++++++++++ examples/python/book_to_citations.py | 128 ++++++++++++ examples/python/search_and_rerank.py | 192 ++++++++++++++++++ examples/rust/.gitignore | 1 + examples/rust/src/main.rs | 69 ++++++- examples/shell/01-book-to-passages.sh | 93 +++++++++ examples/shell/02-video-to-chapters.sh | 109 ++++++++++ examples/shell/03-search-and-rerank.sh | 178 ++++++++++++++++ examples/typescript/package.json | 15 ++ .../typescript/src/episode-to-chapters.ts | 95 +++++++++ examples/typescript/src/search-and-rerank.ts | 96 +++++++++ examples/typescript/tsconfig.json | 15 ++ 14 files changed, 1114 insertions(+), 9 deletions(-) create mode 100644 examples/.gitignore create mode 100644 examples/README.md create mode 100644 examples/python/book_to_citations.py create mode 100644 examples/python/search_and_rerank.py create mode 100644 examples/rust/.gitignore create mode 100755 examples/shell/01-book-to-passages.sh create mode 100755 examples/shell/02-video-to-chapters.sh create mode 100755 examples/shell/03-search-and-rerank.sh create mode 100644 examples/typescript/package.json create mode 100644 examples/typescript/src/episode-to-chapters.ts create mode 100644 examples/typescript/src/search-and-rerank.ts create mode 100644 examples/typescript/tsconfig.json diff --git a/README.md b/README.md index 7175e69..09ed412 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,10 @@ Each tool installs by name, so you can take one without taking the rest. Each links to its own page: what it does, when you would use it, worked examples, options and exit codes. +**→ [Examples](examples/README.md) — programs that run.** A book to +passages and references, a recording to chapter marks and a copy that plays +on a phone, and a hybrid search fused, in shell, Rust, Python and TypeScript. + **→ [Recipes](docs/recipes.md) — using these tools together.** A scanned book to a searchable passage, a transcript to a citation with a timestamp, vectors into pgvector, and hybrid search fused on top. diff --git a/examples/.gitignore b/examples/.gitignore new file mode 100644 index 0000000..1fcb137 --- /dev/null +++ b/examples/.gitignore @@ -0,0 +1,3 @@ +work/ +typescript/node_modules/ +typescript/package-lock.json diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..24e7388 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,125 @@ +# Examples + +Working programs that put these tools together, in the four ways you would +actually reach for them: as filters in a shell script, and as libraries from +Rust, Python and JavaScript. + +Every one of them runs. Where an example needs a book it fetches a public +domain one from Project Gutenberg; where it needs a recording it fetches a +NASA broadcast, which is a work of the US government. + +| | | +|---|---| +| [`shell/01-book-to-passages.sh`](shell/01-book-to-passages.sh) | A book file to passages worth embedding, and what its notes cite | +| [`shell/02-video-to-chapters.sh`](shell/02-video-to-chapters.sh) | A recording to chapter marks, citable passages, and a copy that plays on a phone | +| [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles | +| [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere | +| [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take | +| [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in | +| [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player | +| [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means | + +## Shell + +The tools are filters first: bytes in, records out, and `jq` between them. +Nothing is installed but the commands themselves. + +```sh +cargo install --git https://git.pumpin.online/funkpower/catena \ + booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \ + cuenorm cueseg cuepack avplan emb rrf + +examples/shell/01-book-to-passages.sh ./work +``` + +`01` fetches Gibbon's first volume, works out what the file is, reads the +metadata and the contents the publisher declared, converts and scrubs and +chunks it, and reads the footnotes. Needs `pandoc` and `jq`. + +`02` takes a recording's captions and produces chapter marks, passages that +carry the second they were said at, and — since a recording that is indexed +but unplayable is half a job — the one `ffmpeg` call that makes it play on a +phone. Needs `jq`, `ffmpeg`, and yt-dlp with impersonation support, which +YouTube now requires before it will hand over captions: + +```sh +pipx install "yt-dlp[default,curl-cffi]" +``` + +`03` loads the passages `01` wrote into Postgres with a vector column and a +text index, searches both ways, and fuses the two rankings. Needs `psql`, a +Postgres with pgvector, and an embedding server: + +```sh +DATABASE_URL=postgresql://user@host/db \ +EMB_MODEL=nomic-embed-text \ + examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work +``` + +## Rust + +Every tool is a library with a thin command on top, so anything a command +does can be done in process instead. This walks a file from "what even is it" +to passages and references without serialising anything. + +```sh +cd examples/rust +cargo run # the built-in sample +cargo run -- book.md # a real one +``` + +## Python + +The shape a pipeline stage ends up as once the reading is a library call: +convert, then loop, and what is left is the part that is actually about your +schema. + +```sh +pip install catena +python examples/python/book_to_citations.py +``` + +```sh +pip install catena psycopg +DATABASE_URL=postgresql://user@host/db \ + python examples/python/search_and_rerank.py "why did the empire decline" +``` + +## TypeScript + +The bindings are WebAssembly, so they run in Node and in a browser with the +same API. + +```sh +cd examples/typescript +npm install +npm run chapters # captions to chapter marks +npm run search # fusing two rankings +``` + +## On re-ranking + +Three of these examples end with the same explanation, because it is the +thing most often got wrong. + +Retrieval uses a **bi-encoder**: the passage and the query are embedded +separately. That independence is what lets passage vectors be computed once +and indexed, and it is also the limit — nothing in a passage's vector knows +what was asked. + +**Fusion** takes several ranked lists and makes one. Reciprocal rank fusion +scores a hit by where it placed rather than by what it scored: `1/(k + rank)` +from each list that found it, added up. It is used because the scores are not +comparable — a cosine distance and a `ts_rank_cd` measure different things, +and no scaling makes them the same — so the ranks are kept and the scores +thrown away. A passage that both searches liked beats one that either loved. +That is [`rrf`](../crates/rrf/README.md), and it needs no model. + +A **cross-encoder** reads the query and the passage together and scores the +pair. Far more accurate, and far too slow to run over a corpus, so it runs +over the twenty or fifty candidates retrieval already found. In a pipeline +the two stack: retrieve both ways, fuse the lists, then hand the top of the +fused list to a cross-encoder for the final order. + +Both get called re-ranking. Nothing here loads a model, so the last step is +yours — the fused list is what you would hand it. diff --git a/examples/python/book_to_citations.py b/examples/python/book_to_citations.py new file mode 100644 index 0000000..445d7a9 --- /dev/null +++ b/examples/python/book_to_citations.py @@ -0,0 +1,128 @@ +"""A book to rows a database can take, without a subprocess in sight. + +This is the shape a pipeline stage ends up as once the reading is a library +call: convert, then loop, and what is left is the part that is actually about +your schema. + + python examples/python/book_to_citations.py path/to/book.epub + +With no argument it fetches Gibbon's first volume from Project Gutenberg, +which is public domain and has a few hundred pages of footnotes. +""" + +import pathlib +import subprocess +import sys +import urllib.request + +import catena + +GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub" + + +def fetch(into: pathlib.Path) -> pathlib.Path: + """The sample book, downloaded once.""" + book = into / "gibbon.epub" + if not book.exists(): + into.mkdir(parents=True, exist_ok=True) + print(f"fetching {GUTENBERG}") + with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310 + book.write_bytes(reply.read()) + return book + + +def to_markdown(book: pathlib.Path) -> str: + """Pandoc is the one part of this that is somebody else's program. + + `commonmark` rather than `commonmark_x`: the extended flavour writes + anchors as bracketed spans, and they end up inside the passages. + """ + done = subprocess.run( + ["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)], + capture_output=True, + text=True, + check=True, + ) + return done.stdout + + +def rows(markdown: str) -> list[dict]: + """Passages in, citation rows out. + + Two library calls do the work. `chunk_markdown` cuts the book into + passages and flags the ones carrying citation material — which is the + whole reason to ask it rather than looking for a heading, since a page of + notes at the foot of a chapter has no heading over it. `find_citations` + then reads each of those. + """ + out = [] + for chunk in catena.chunk_markdown(catena.scrub(markdown)): + if "footnote_citations" not in chunk["flags"] and chunk["role"] != "bibliography": + continue + for found in catena.find_citations(chunk["text"]): + # An `ibid.` or a short cite points at an entry above rather than + # naming a work of its own, so there is nothing to store. + if found["continuation"]: + continue + if not (found["title"] or found["authors"] or found["ids"]): + continue + out.append( + { + "page": chunk["page"], + "chapter": chunk["chapter"], + "citation_text": found["text"], + "referenced_work": found["title"], + "referenced_author": "; ".join(found["authors"]) or None, + "referenced_year": found["year"], + "referenced_container": found["container"], + "kind": found["kind"], + "unsettled": found["unsettled"], + } + ) + return out + + +def main() -> int: + where = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else fetch(pathlib.Path("work")) + + print(f"== {where.name}") + identified = catena.identify(where.name, where.read_bytes()) + print(f" a {identified['kind']}, {identified['bytes']} bytes") + + markdown = to_markdown(where) + passages = catena.chunk_markdown(catena.scrub(markdown)) + carrying = [c for c in passages if "footnote_citations" in c["flags"]] + print(f" {len(passages)} passages, {len(carrying)} carrying citation material") + + found = rows(markdown) + named = [r for r in found if r["referenced_work"]] + print(f"\n== {len(found)} rows, {len(named)} naming a work") + + # What a stage like this would do next is its own business: one executemany + # against your own table. The point is that everything above it is two + # function calls rather than a parser you have to keep. + print("\n INSERT INTO citations (citation_text, referenced_work, referenced_author, …)") + whole = [r for r in named if r["referenced_author"]] or named + for row in whole[:5]: + work = (row["referenced_work"] or "")[:40] + who = (row["referenced_author"] or "—")[:22] + print(f" (…, {work!r}, {who!r}, {row['referenced_year']})") + + # Nothing is invented. Where a field could not be read the record says so + # rather than guessing, and a caller that wants a model to fill the gaps + # has `citation_prompt` and `merge_citation_reply` for exactly that. + # Gibbon cites in the classical short form — `Tacit. Annal. xiii. 5.` — + # so most of his notes name a passage rather than a book. Nothing is + # invented to cover that: the record says which fields it could not read, + # and `citation_prompt` and `merge_citation_reply` are there for a caller + # that wants a model to finish them. + unsettled = [r for r in found if r["unsettled"]] + print(f"\n {len(unsettled)} rows name something they could not finish reading:") + for row in unsettled[:3]: + print(f" {row['citation_text'][:56]} → {', '.join(row['unsettled'])}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/python/search_and_rerank.py b/examples/python/search_and_rerank.py new file mode 100644 index 0000000..49133c2 --- /dev/null +++ b/examples/python/search_and_rerank.py @@ -0,0 +1,192 @@ +"""Passages into pgvector, then a search that asks twice and reconciles. + +Loads the passages of a book into Postgres with both a vector column and a +text index, searches both ways, and fuses the two rankings into one. + + DATABASE_URL=postgresql://user@host/db \ + EMB_MODEL=nomic-embed-text \ + python examples/python/search_and_rerank.py "why did the empire decline" + +Needs psycopg, a Postgres with pgvector, an embedding server with Ollama's +API, and a book — the sample is fetched if none is given. +""" + +import json +import os +import pathlib +import subprocess +import sys +import urllib.request + +import catena +import psycopg + +EMB_HOST = os.environ.get("EMB_HOST", "http://localhost:11434") +EMB_MODEL = os.environ.get("EMB_MODEL", "nomic-embed-text") +GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub" + + +def embed(texts: list[str]) -> list[list[float]]: + """Ask the embedding server, in batches it will accept. + + The one thing here that touches the network. No model is loaded in this + process: it belongs behind a server, where it is shared and where its + GPU is somebody else's problem. + """ + out: list[list[float]] = [] + for at in range(0, len(texts), 32): + batch = texts[at : at + 32] + body = json.dumps({"model": EMB_MODEL, "input": batch}).encode() + request = urllib.request.Request( + f"{EMB_HOST}/api/embed", data=body, headers={"content-type": "application/json"} + ) + with urllib.request.urlopen(request, timeout=300) as reply: # noqa: S310 + out.extend(json.load(reply)["embeddings"]) + print(f" embedded {len(out)}/{len(texts)}", end="\r", file=sys.stderr) + return out + + +def passages(book: pathlib.Path) -> list[dict]: + """The book, cut into passages worth embedding.""" + markdown = subprocess.run( + ["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)], + capture_output=True, text=True, check=True, + ).stdout + chunks = catena.chunk_markdown(catena.scrub(markdown)) + # A table of contents embeds as well as anything and retrieves for every + # query, which is worse than not having it. + return [c for c in chunks if c["role"] == "body" and c["tokens"] > 40] + + +def load(conn, rows: list[dict], vectors: list[list[float]]) -> None: + width = len(vectors[0]) + with conn.cursor() as cur: + cur.execute("CREATE EXTENSION IF NOT EXISTS vector") + cur.execute("DROP TABLE IF EXISTS catena_passages") + # The tsvector is generated from the body rather than written + # alongside it, so the two cannot drift apart. + cur.execute(f""" + CREATE TABLE catena_passages ( + idx integer PRIMARY KEY, + chapter text, + body text NOT NULL, + fts tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED, + vec vector({width}) + )""") + with cur.copy("COPY catena_passages (idx, chapter, body, vec) FROM STDIN") as copy: + for chunk, vector in zip(rows, vectors): + copy.write_row((chunk["idx"], chunk["chapter"], chunk["text"], str(vector))) + cur.execute("CREATE INDEX ON catena_passages USING gin (fts)") + cur.execute("CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops)") + conn.commit() + + +def by_meaning(conn, vector: list[float], limit: int = 10) -> list[dict]: + """Nearest in the embedding space. + + `<=>` is cosine distance, so smaller is nearer. This finds a passage + about the fall of Rome for a query about decline, sharing no words. + """ + with conn.cursor() as cur: + cur.execute( + """SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g') + FROM catena_passages ORDER BY vec <=> %s::vector LIMIT %s""", + (str(vector), limit), + ) + return [ + {"v": 1, "id": str(idx), "channel": "vec", "rank": at, "score": 0.0, "text": body} + for at, (idx, body) in enumerate(cur.fetchall(), start=1) + ] + + +def by_word(conn, query: str, limit: int = 10) -> list[dict]: + """Nearest by the words themselves. + + Finds the passage that actually says the word where the vector search + drifted to a paraphrase, and finds nothing at all where the words are + not in the book — which is a useful thing to be told. + """ + with conn.cursor() as cur: + cur.execute( + """SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g') + FROM catena_passages + WHERE fts @@ websearch_to_tsquery('english', %(q)s) + ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', %(q)s)) DESC + LIMIT %(n)s""", + {"q": query, "n": limit}, + ) + return [ + {"v": 1, "id": str(idx), "channel": "fts", "rank": at, "score": 0.0, "text": body} + for at, (idx, body) in enumerate(cur.fetchall(), start=1) + ] + + +def main() -> int: + query = sys.argv[1] if len(sys.argv) > 1 else "why did the roman empire decline" + url = os.environ.get("DATABASE_URL") + if not url: + print("set DATABASE_URL", file=sys.stderr) + return 64 + + work = pathlib.Path("work") + work.mkdir(exist_ok=True) + book = work / "gibbon.epub" + if not book.exists(): + print(f"fetching {GUTENBERG}") + with urllib.request.urlopen(GUTENBERG, timeout=120) as reply: # noqa: S310 + book.write_bytes(reply.read()) + + rows = passages(book) + print(f"== {len(rows)} passages") + vectors = embed([" ".join(chunk["text"].split()) for chunk in rows]) + print(f" {EMB_MODEL} returns {len(vectors[0])} numbers per passage") + + with psycopg.connect(url) as conn: + load(conn, rows, vectors) + print(" loaded into catena_passages") + + print(f"\n== query: {query}") + hits = by_meaning(conn, embed([query])[0]) + by_word(conn, query) + + for channel, label in (("vec", "by meaning"), ("fts", "by word")): + print(f"\n {label}:") + for hit in [h for h in hits if h["channel"] == channel][:4]: + print(f" {hit['rank']}. [{hit['id']}] {hit['text']}") + + # Fusion is a library call rather than a subprocess: by the time a hit + # reaches here it is already yours, and serialising it to hand to a + # program and back is work for nothing. + print("\n== fused") + text_of = {hit["id"]: hit["text"] for hit in hits} + for hit in catena.fuse_hits(hits, limit=5): + print(f" {hit['rank']}. [{hit['id']}] score {hit['score']:.3f} {text_of[hit['id']]}") + + print(RERANKING) + return 0 + + +RERANKING = """ + Reciprocal rank fusion scores a passage by where it placed, not by what + it scored: 1/(k + rank) from each list that found it, added up. The two + searches return numbers that mean different things — a cosine distance + and a ts_rank_cd — and no scaling makes them comparable, so their ranks + are used and their scores thrown away. A passage both searches liked + beats one that either loved, which is the behaviour worth having. + + This is fusion, and it is often called re-ranking, but a second thing + wears the same name. Retrieval uses a bi-encoder: passage and query are + embedded separately, which is what lets the passage vectors be computed + once and indexed, and is also what limits it — nothing in a passage's + vector knows what was asked. + + A cross-encoder reads the query and the passage together and scores the + pair. It is far more accurate and far too slow to run over a corpus, so + it runs over the twenty or fifty candidates retrieval already found. The + two stack: retrieve both ways, fuse the lists, then hand the top of the + fused list to a cross-encoder for the final order. Nothing here loads a + model, so that step is yours — the fused list is what you would hand it. +""" + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/rust/.gitignore b/examples/rust/.gitignore new file mode 100644 index 0000000..2f7896d --- /dev/null +++ b/examples/rust/.gitignore @@ -0,0 +1 @@ +target/ diff --git a/examples/rust/src/main.rs b/examples/rust/src/main.rs index 3150570..eb5fe4c 100644 --- a/examples/rust/src/main.rs +++ b/examples/rust/src/main.rs @@ -8,6 +8,13 @@ //! //! cargo run -- book.md //! cargo run # reads the sample below +//! +//! For a real book, convert one first — Gibbon's first volume is public +//! domain and has a few hundred pages of footnotes: +//! +//! curl -sL -o gibbon.epub https://www.gutenberg.org/cache/epub/731/pg731.epub +//! pandoc -f epub -t commonmark --wrap=none gibbon.epub -o gibbon.md +//! cargo run -- gibbon.md use std::path::Path; @@ -74,8 +81,8 @@ fn main() { }, }; let chunks = chunkmd::chunk(&clean, &budgets); - println!("\n{} passage(s):", chunks.len()); - for chunk in &chunks { + println!("\n{} passage(s), the first few:", chunks.len()); + for chunk in chunks.iter().take(6) { let chapter = chunk.chapter.as_deref().unwrap_or("—"); let opening: String = chunk.text.chars().take(64).collect(); println!( @@ -87,18 +94,50 @@ fn main() { ); } - // The references at the back. A passage whose role is `bibliography` is - // where they live; a note at the foot of a page is the other place, and - // `chunkmd` flags those so this does not have to guess. - let references: Vec = chunks + // The references. A passage whose role is `bibliography` is one place + // they live; a page of notes at the foot of a chapter is the other, and + // it has no heading over it to find it by — which is why `chunkmd` + // flags those, so this does not have to guess. + let carrying: Vec<&corpus_core::Chunk> = chunks + .iter() + .filter(|chunk| { + chunk.role == corpus_core::Role::Bibliography + || chunk + .flags + .iter() + .any(|flag| flag == corpus_core::chunk::flags::FOOTNOTE_CITATIONS) + }) + .collect(); + + let references: Vec = carrying .iter() - .filter(|chunk| chunk.role == corpus_core::Role::Bibliography) .flat_map(|chunk| citefind::read(&chunk.text)) .filter(|found| !found.continuation) .collect(); - println!("\n{} reference(s):", references.len()); - for found in &references { + println!( + "\n{} passage(s) carry citation material, giving {} reference(s):", + carrying.len(), + references.len() + ); + // The ones that named something, which is not all of them: a book whose + // notes cite in the classical short form — `Tacit. Annal. xiii. 5.` — + // gives mostly passages rather than works, and the record says as much + // rather than inventing a title to fill the field. + let with_work = references + .iter() + .filter(|found| found.title.is_some()) + .count(); + let named: Vec<&citefind::Reference> = references + .iter() + .filter(|found| !found.authors.is_empty()) + .collect(); + println!( + "{with_work} name a work and {} name an author; those:", + named.len() + ); + + for found in named.iter().take(6) { let who = match found.authors.is_empty() { true => "—".to_string(), false => found.authors.join("; "), @@ -116,4 +155,16 @@ fn main() { println!(" could not read: {}", found.unsettled.join(", ")); } } + + // Worth saying plainly, because the numbers above look poor on a book + // like Gibbon and the reason is not the parser having failed quietly. + // His notes cite in the classical short form — `Tacit. Annal. xiii. 5.` + // — which names a passage rather than a work, so there is no title and + // no year to read and the record says which fields it could not fill + // rather than inventing them. `citation_prompt` and `merge_citation_reply` + // are there for a caller that wants a model to finish the job. + if references.len() > 20 && named.len() * 10 < references.len() { + println!("\n most of these are classical short cites — a passage"); + println!(" rather than a work — there is nothing more to read in them."); + } } diff --git a/examples/shell/01-book-to-passages.sh b/examples/shell/01-book-to-passages.sh new file mode 100755 index 0000000..998baa4 --- /dev/null +++ b/examples/shell/01-book-to-passages.sh @@ -0,0 +1,93 @@ +#!/bin/sh +# A book file to passages worth embedding, and what its notes cite. +# +# The starting point is Gibbon's first volume from Project Gutenberg: public +# domain, a real table of contents, and a few hundred pages of footnotes. +# +# examples/shell/01-book-to-passages.sh [work-dir] +# +# Needs pandoc and jq, and the catena commands on PATH. +set -eu + +work="${1:-./work}" +mkdir -p "$work" +book="$work/gibbon.epub" + +if [ ! -s "$book" ]; then + echo "fetching Gibbon vol. 1 from Project Gutenberg" + curl -sSL --max-time 120 -o "$book" \ + "https://www.gutenberg.org/cache/epub/731/pg731.epub" +fi + +# What the file actually is. Extensions lie, and a `.epub` that is really a +# zip of page scans fails four stages later, in a way that looks like a bug +# in the converter. +echo +echo "== what it is" +booksniff "$book" | jq -r '" \(.kind), \(.bytes) bytes"' + +# A PDF gets one more question first: exit 0 from pdftriage means the text +# layer is not worth reading and the file has to be OCRed before any of this. +if [ "$(booksniff "$book" | jq -r .kind)" = "pdf" ] && pdftriage --quiet "$book"; then + echo " the text layer is garbled; OCR it before going on" + exit 0 +fi + +# What the file already says about itself — the publisher's own metadata, +# which beats anything guessed from a filename, and costs no lookup. +echo +echo "== what it says about itself" +bookmeta "$book" | jq -r '" \(.title)\n \(.authors | join("; "))\n \(.language // "?") · \(.date // "no date")"' + +# The chapters the book declares, rather than any inferred from the text. +echo +echo "== the contents it declares" +tocfind "$book" --outline | sed -n '6,10p' | sed 's/^/ /' +echo " … $(tocfind "$book" | wc -l | tr -d ' ') entries" + +# Convert, scrub, chunk. `commonmark` rather than `commonmark_x`, because the +# extended flavour writes anchors as bracketed spans and they end up in the +# passages. The scrub is what takes out the rest of the converter's leavings. +echo +echo "== passages" +pandoc -f epub -t commonmark --wrap=none "$book" -o "$work/book.md" +mdscrub < "$work/book.md" > "$work/clean.md" +chunkmd < "$work/clean.md" > "$work/chunks.ndjson" + +jq -s -r ' + " \(length) passages over \([.[].chapter] | unique | length) chapters", + " roles: \([.[].role] | group_by(.) | map("\(length) \(.[0])") | join(", "))", + " \([.[] | select(.flags | index("footnote_citations"))] | length) carry citation material" +' "$work/chunks.ndjson" + +echo +echo " a passage, with where it came from:" +jq -s -r '[.[] | select(.role == "body" and .chapter != null)][40] + | " [\(.idx)] \(.tokens) tokens · \(.chapter)\n \(.text[0:150])…"' \ + "$work/chunks.ndjson" + +# The notes. A passage flagged `footnote_citations` is a page of them, and +# each is read on its own — the flag says which passages are worth the ask, +# which is the whole reason chunkmd sets it. +echo +echo "== what the notes cite" +: > "$work/references.ndjson" +jq -c 'select(.flags | index("footnote_citations")) | .text' "$work/chunks.ndjson" \ + | while IFS= read -r line; do + printf '%s' "$line" | jq -r . | citefind >> "$work/references.ndjson" || true + done + +jq -s -r ' + " \(length) entries read", + " \([.[] | select(.title)] | length) name a work, \([.[] | select(.authors | length > 0)] | length) name an author", + " \([.[] | select(.unsettled | length > 0)] | length) came back with something unread" +' "$work/references.ndjson" + +echo +echo " Gibbon cites in the classical short form — \"Tacit. Annal. xiii. 5.\" —" +echo " so most entries name the passage rather than a book, and say so:" +jq -r 'select(.unsettled | length > 0) | " \(.text[0:58]) → unread: \(.unsettled | join(", "))"' \ + "$work/references.ndjson" | head -4 + +echo +echo "written to $work/" diff --git a/examples/shell/02-video-to-chapters.sh b/examples/shell/02-video-to-chapters.sh new file mode 100755 index 0000000..2f689ef --- /dev/null +++ b/examples/shell/02-video-to-chapters.sh @@ -0,0 +1,109 @@ +#!/bin/sh +# A recording to chapter marks, citable passages, and a copy that plays. +# +# Takes the captions a video already carries, normalises them to one cue +# stream, and turns that into chapter marks for a player and passages worth +# embedding — each keeping the second it was said at. Then asks whether the +# file itself would play on a phone, and prints the one ffmpeg call that +# fixes it. +# +# examples/shell/02-video-to-chapters.sh [url] [work-dir] +# +# The default is a NASA upload: a work of the US government, so public +# domain, and captioned by a person rather than by a model. Any URL yt-dlp +# can reach works, and any local subtitle file works too — skip to the +# cuenorm line and feed it your own. +# +# Needs jq, ffmpeg, and yt-dlp with impersonation support, which YouTube now +# requires before it will hand over captions: +# +# pipx install "yt-dlp[default,curl-cffi]" +# +# and the catena commands on PATH. +set -eu + +url="${1:-https://www.youtube.com/watch?v=h0wLZJeYGxw}" +work="${2:-./work}" +mkdir -p "$work" + +# Captions written by a person where there are any, and a model's otherwise. +# Automatic captions roll — each line is re-sent as it scrolls up the screen — +# so the same sentence arrives three times and every passage says it thrice. +if [ -z "$(find "$work" -name '*.vtt' -print -quit 2>/dev/null)" ]; then + echo "fetching captions" + yt-dlp --skip-download --write-subs --sub-langs 'en.*' --sub-format vtt \ + --output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 || + yt-dlp --skip-download --write-auto-subs --sub-langs 'en.*' --sub-format vtt \ + --output "$work/%(id)s.%(ext)s" "$url" >/dev/null +fi + +# Where several tracks came back, the smallest is the written one: a rolling +# track repeats every line and is four or five times the size. +subs="$(find "$work" -name '*.vtt' -exec ls -S {} + | tail -1)" +[ -n "$subs" ] || { echo "no captions on that video; try another URL" >&2; exit 1; } +echo " captions: $(basename "$subs")" + +echo +echo "== cues" +cuenorm "$subs" > "$work/cues.ndjson" +jq -s -r '" \(length) cues over \((map(.end) | max) / 60 | floor)m\((map(.end) | max) % 60 | floor)s"' "$work/cues.ndjson" +jq -r '" [\(.start | floor)s] \(.text[0:62])"' "$work/cues.ndjson" | head -3 + +# Where the subject changes, found from the words alone — no model, no +# network. The titles come out of each chapter's own first sentence, and are +# meant to be replaced by something better later. +echo +echo "== chapters" +cueseg < "$work/cues.ndjson" > "$work/chapters.ndjson" +jq -r '" \(.start | floor)s–\(.end | floor)s \(.title)"' "$work/chapters.ndjson" + +# Cues packed to a size worth embedding, each carrying the seconds it spans, +# so a search hit can be played back at the moment it was said. +echo +echo "== passages" +cuepack < "$work/cues.ndjson" > "$work/passages.ndjson" +jq -s -r '" \(length) passages, \(map(.tokens) | add / length | floor) tokens on average"' "$work/passages.ndjson" +jq -s -r '.[0] | " [\(.start_time | floor)s–\(.end_time | floor)s] \(.text[0:150])…"' "$work/passages.ndjson" + +echo +echo "== chapter marks, as a player would take them" +jq -r '"\(.start | floor) \(.title)"' "$work/chapters.ndjson" | + awk '{ t = $1; $1 = ""; printf " %02d:%02d:%02d%s\n", t/3600, (t%3600)/60, t%60, $0 }' + +# And the file itself. A recording indexed but unplayable is half a job: a +# phone will not open VP9 in a WebM container, whatever the transcript says. +# Twenty seconds of it is enough to judge the streams, and spares the +# download. +echo +echo "== would it play on a phone" +video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)" +if [ -z "$video" ]; then + echo " fetching twenty seconds of it" + yt-dlp -f 'worstvideo[ext=webm]+worstaudio[ext=webm]/worst' \ + --download-sections '*0-20' --force-keyframes-at-cuts \ + --output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 || true + video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)" +fi +[ -n "$video" ] || { echo " nothing to judge"; exit 0; } + +avplan --json "$video" | jq -r ' + " \(.streams.container): \(.streams.video) video, \(.streams.audio // "no") audio", + (if .playable then " plays as it stands" else (.reasons[] | " · \(.)") end)' + +# Exit 0 means there is a plan to print; 1 means the file already plays. The +# encoder is whatever this machine actually has — videotoolbox on a Mac, +# NVENC or VAAPI on a Linux box, libx264 where there is no card. Nothing is +# run: converting is expensive and irreversible enough to read first. +echo +if plan="$(avplan --output "$work/for-phone.mp4" "$video")"; then + echo " the one call that fixes it:" + echo + printf ' %s\n' "$plan" + echo + echo " run it with: eval \"\$(avplan --output out.mp4 FILE)\"" +else + echo " nothing to convert" +fi + +echo +echo "written to $work/" diff --git a/examples/shell/03-search-and-rerank.sh b/examples/shell/03-search-and-rerank.sh new file mode 100755 index 0000000..31c1519 --- /dev/null +++ b/examples/shell/03-search-and-rerank.sh @@ -0,0 +1,178 @@ +#!/bin/sh +# Passages into pgvector, then a search that asks twice and reconciles. +# +# Takes the passages example 01 wrote, embeds them, loads them into Postgres +# with a vector column and a text index, then searches both ways and fuses +# the two lists into one. +# +# examples/shell/01-book-to-passages.sh ./work +# examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work +# +# Needs psql and jq, a Postgres with pgvector, an embedding server, and the +# catena commands on PATH: +# +# DATABASE_URL=postgresql://user@host/db +# EMB_HOST=http://localhost:11434 # ollama, or anything with its API +# EMB_MODEL=nomic-embed-text # 768 wide; bge-m3 is 1024 +set -eu + +query="${1:-why did the roman empire decline}" +work="${2:-./work}" +: "${DATABASE_URL:?set DATABASE_URL}" +host="${EMB_HOST:-http://localhost:11434}" +model="${EMB_MODEL:-nomic-embed-text}" + +[ -s "$work/chunks.ndjson" ] || { echo "run 01-book-to-passages.sh first" >&2; exit 1; } +psql="psql -X -q -v ON_ERROR_STOP=1 $DATABASE_URL" +tab=$(printf '\t') + +# How wide the model's vectors are. Asked rather than assumed, because the +# column is typed and a mismatch is a wall of errors at insert time. +dim=$(printf 'how wide is this\n' | emb --host "$host" --model "$model" | jq 'length') +echo "== $model returns $dim numbers per passage" + +# One table. The vector is what the model said; the tsvector is Postgres's +# own, generated from the text so it cannot drift out of step with it. +$psql < 40)' "$work/chunks.ndjson" > "$work/body.ndjson" +echo "== embedding $(wc -l < "$work/body.ndjson" | tr -d ' ') passages" + +# The newlines have to go before the text is a line: emb is line-oriented, +# so a passage containing one would arrive as two. +jq -r '.text | gsub("\\s+"; " ")' "$work/body.ndjson" > "$work/lines.txt" +emb --host "$host" --model "$model" --dim "$dim" < "$work/lines.txt" > "$work/vectors.ndjson" + +# Line for line, which is emb's promise, so paste is a safe join. +jq -c '[.idx, .chapter, (.text | gsub("\\s+"; " ")), .tokens]' "$work/body.ndjson" > "$work/meta.ndjson" +paste -d '\t' "$work/meta.ndjson" "$work/vectors.ndjson" | + python3 -c ' +import csv, json, sys +out = csv.writer(sys.stdout, delimiter="\t", quoting=csv.QUOTE_NONE, escapechar="\\") +for line in sys.stdin: + meta, vector = line.rstrip("\n").split("\t") + idx, chapter, body, tokens = json.loads(meta) + out.writerow([idx, chapter or "", body, tokens, vector]) +' > "$work/rows.tsv" + +$psql -c "\copy catena_passages (idx, chapter, body, tokens, vec) FROM '$work/rows.tsv'" +$psql -c "CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops);" +$psql -tAc "SELECT ' ' || count(*) || ' passages loaded' FROM catena_passages" + +# psql's rows into hit records. `rrf` wants an id, which channel found it, +# and what rank that channel gave it; the text is carried along so the +# output is readable. Written in Python because the passages contain +# quotation marks and a shell printf would not escape them. +hits() { + python3 -c ' +import json, sys +channel = sys.argv[1] +for line in sys.stdin: + line = line.rstrip("\n") + if not line.strip(): + continue + ident, rank, text = line.split("\t", 2) + print(json.dumps({"v": 1, "id": ident, "channel": channel, + "rank": int(rank), "score": 0.0, "text": text})) +' "$1" +} + +# --------------------------------------------------------------------------- +# Two searches. They disagree, and that is the point. +# --------------------------------------------------------------------------- +echo +echo "== query: $query" +printf '%s\n' "$query" | emb --host "$host" --model "$model" --dim "$dim" | tr -d '\n' > "$work/query.vec" +qvec=$(cat "$work/query.vec") + +# Nearest by meaning. `<=>` is cosine distance, so smaller is closer, and a +# passage about "the fall of Rome" ranks even though it shares no words with +# the query. +$psql -tAF"$tab" -c " + SELECT idx, row_number() OVER (ORDER BY vec <=> '$qvec'::vector), + regexp_replace(left(body, 68), '\s+', ' ', 'g') + FROM catena_passages + ORDER BY vec <=> '$qvec'::vector + LIMIT 10" | hits vec > "$work/hits.ndjson" + +# Nearest by word. Finds the passage that says "decline" where the vector +# search drifted to a paraphrase, and finds nothing at all for a query whose +# words are not in the book. +$psql -tAF"$tab" -c " + SELECT idx, + row_number() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', \$\$$query\$\$)) DESC), + regexp_replace(left(body, 68), '\s+', ' ', 'g') + FROM catena_passages + WHERE fts @@ websearch_to_tsquery('english', \$\$$query\$\$) + ORDER BY 2 + LIMIT 10" | hits fts >> "$work/hits.ndjson" + +echo +echo " by meaning:" +jq -r 'select(.channel == "vec") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4 +echo +echo " by word:" +jq -r 'select(.channel == "fts") | " \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4 + +# --------------------------------------------------------------------------- +# Reconciling them. +# --------------------------------------------------------------------------- +echo +echo "== fused" +rrf --limit 5 < "$work/hits.ndjson" > "$work/fused.ndjson" +jq -s -r --slurpfile hits "$work/hits.ndjson" ' + ($hits | INDEX(.id)) as $by_id + | .[] | " \(.rank). [\(.id)] score \(.score * 1000 | floor / 1000) \($by_id[.id].text)" +' "$work/fused.ndjson" + +cat <<'NOTE' + + Reciprocal rank fusion scores a passage by where it placed, not by what + it scored: 1/(k + rank) from each list that found it, added up. The two + searches return numbers that mean different things — a cosine distance + and a ts_rank_cd — and no amount of scaling makes them comparable, so + their ranks are used and their scores thrown away. A passage both + searches liked beats one that either loved, which is the behaviour worth + having: agreement between two different ways of looking is evidence, and + a high score from one is only ever evidence about that one. + + The run above shows it happening. The word search's first hit is the + front matter, which says "Roman Empire" more often than any chapter of + the book does; the vector search never offers it. Fusion pushes it down + and lifts the one passage both searches found. + + The constant k, 60 by default, is how much a first place is worth over a + second. Raise it and the lists flatten toward equal weight; lower it and + the top of each list dominates. `--weight vec=2` trusts one channel more + than the other, which is the knob to reach for when one of them is + consistently better on your corpus. + + This is fusion, and it is often called re-ranking, but there is a second + thing wearing the same name. Retrieval uses a bi-encoder: the passage + and the query are embedded separately, so the passage vectors can be + computed once and indexed, and a search is one vector comparison against + an index. That independence is what makes it fast and is also what + limits it — nothing in the passage's vector knows what was asked. + + A cross-encoder reads the query and the passage together and scores the + pair. It is far more accurate and far too slow to run over a corpus, so + it runs over the twenty or fifty candidates retrieval already found. In + a pipeline the two stack: retrieve both ways, fuse the lists as above, + then hand the top of the fused list to a cross-encoder for the final + order. Nothing here does that step — it needs a model, and these tools + do not load models — but the fused list is what you would hand it. +NOTE diff --git a/examples/typescript/package.json b/examples/typescript/package.json new file mode 100644 index 0000000..b5e8fbc --- /dev/null +++ b/examples/typescript/package.json @@ -0,0 +1,15 @@ +{ + "name": "catena-example", + "private": true, + "type": "module", + "scripts": { + "chapters": "node --experimental-strip-types src/episode-to-chapters.ts", + "search": "node --experimental-strip-types src/search-and-rerank.ts" + }, + "dependencies": { + "catena": "file:../../crates/bindings/npm/dist" + }, + "devDependencies": { + "@types/node": "^22" + } +} diff --git a/examples/typescript/src/episode-to-chapters.ts b/examples/typescript/src/episode-to-chapters.ts new file mode 100644 index 0000000..1088b04 --- /dev/null +++ b/examples/typescript/src/episode-to-chapters.ts @@ -0,0 +1,95 @@ +// A recording's captions into chapter marks and citable passages. +// +// What a player needs from an episode: somewhere to jump to, and passages +// that carry the second they were said at so a search result can be played +// rather than just quoted. +// +// npm install +// npm run chapters -- path/to/captions.vtt +// +// With no argument it reads the sample below, which is a few cues of a NASA +// broadcast — a work of the US government, so public domain. + +import { readFileSync } from "node:fs"; +import { parseSubtitles, segmentCues, packCues, countTokens } from "catena"; + +const SAMPLE = `WEBVTT + +00:00:00.000 --> 00:00:04.200 +Hey there, and welcome to NASA's Curious Universe. + +00:00:04.200 --> 00:00:08.900 +Our universe is a wild and wonderful place, and in this +show, NASA is your tour guide. + +00:00:08.900 --> 00:00:15.100 +Today we are going to talk about how we find planets +around other stars, which is harder than it sounds. + +00:00:15.100 --> 00:00:21.800 +A planet is small and dark and its star is enormous and +bright, so you almost never see the planet itself. + +00:00:21.800 --> 00:00:28.400 +Instead you watch the star, and you wait for it to dim +by about one part in ten thousand. + +00:00:28.400 --> 00:00:34.000 +That dimming is a transit, and it is how most of the +planets we know about were found. +`; + +const path = process.argv[2]; +const content = path ? readFileSync(path, "utf8") : SAMPLE; + +// One cue stream, whatever the format was. `keepRepeats` is off, which is +// what throws away the re-sent lines of a rolling automatic caption track — +// leave it on and every passage says each sentence three times. +const cues = parseSubtitles(content); +const runsTo = Math.max(...cues.map((cue) => cue.end)); +console.log(`== ${cues.length} cues over ${Math.round(runsTo)}s`); +for (const cue of cues.slice(0, 3)) { + console.log(` [${cue.start.toFixed(1)}s] ${cue.text.slice(0, 58)}`); +} + +// Where the subject changes, from the words alone — no model, no network. +// The titles come out of each chapter's own first sentence and are meant to +// be replaced by something better later. +const chapters = segmentCues(cues); +console.log(`\n== ${chapters.length} chapter(s)`); +for (const chapter of chapters) { + console.log(` ${stamp(chapter.start)} ${chapter.title}`); +} + +// Cues packed to a size worth embedding, each keeping the seconds it spans. +const passages = packCues(cues); +console.log(`\n== ${passages.length} passage(s)`); +for (const passage of passages) { + console.log( + ` [${stamp(passage.start_time ?? 0)}–${stamp(passage.end_time ?? 0)}] ` + + `${passage.tokens} tokens`, + ); + console.log(` ${passage.text.slice(0, 92)}…`); +} + +// What a player would take: a cue sheet, and a deep link per passage. +console.log("\n== what the player gets"); +for (const chapter of chapters) { + console.log(` ${stamp(chapter.start)} ${chapter.title}`); +} +console.log("\n== what a search result would link to"); +for (const passage of passages.slice(0, 2)) { + const at = Math.floor(passage.start_time ?? 0); + console.log(` ?t=${at} ${passage.text.slice(0, 64)}…`); +} + +// The same token count the chunker used, so a budget worked out here and a +// budget enforced there agree. +console.log(`\n ${countTokens(cues.map((c) => c.text).join(" "))} tokens of transcript in all`); + +function stamp(seconds: number): string { + const whole = Math.floor(seconds); + const mm = String(Math.floor(whole / 60)).padStart(2, "0"); + const ss = String(whole % 60).padStart(2, "0"); + return `${String(Math.floor(whole / 3600)).padStart(2, "0")}:${mm}:${ss}`; +} diff --git a/examples/typescript/src/search-and-rerank.ts b/examples/typescript/src/search-and-rerank.ts new file mode 100644 index 0000000..ced2643 --- /dev/null +++ b/examples/typescript/src/search-and-rerank.ts @@ -0,0 +1,96 @@ +// Two searches, reconciled — the app-side half of hybrid retrieval. +// +// Fusion is the one thing in this suite worth calling in process rather than +// as a subprocess: by the time a hit reaches your code it is already yours, +// and serialising it to hand to a program and back is work for nothing. +// +// npm install +// npm run search +// +// The hits below stand in for two queries you would run against your own +// store — pgvector and a text index, Elasticsearch and a KNN service, or a +// vector database and whatever else. What they came from does not matter +// here; only where each of them placed does. + +import { fuseHits, type Hit } from "catena"; + +// `payload` is whatever you want carried alongside — rrf hands it back +// untouched, so a hit can arrive here with the row it came from attached. +const hit = (id: string, channel: string, rank: number, score: number): Hit => ({ + v: 1, + id, + channel, + rank, + score, + payload: {}, +}); + +// Nearest in the embedding space. Finds the passage about the fall of Rome +// for a query about decline, sharing not one word with it. +const byMeaning: Hit[] = [ + hit("p39", "vec", 1, 0.21), + hit("p127", "vec", 2, 0.24), + hit("p462", "vec", 3, 0.26), + hit("p177", "vec", 4, 0.29), +]; + +// Nearest by the words themselves. Finds the passage that says the word +// where the vectors drifted to a paraphrase — and finds the front matter, +// which says "Roman Empire" more often than any chapter of the book does. +const byWord: Hit[] = [ + hit("p0", "fts", 1, 0.68), + hit("p462", "fts", 2, 0.51), + hit("p8", "fts", 3, 0.44), + hit("p71", "fts", 4, 0.4), +]; + +const text: Record = { + p0: "The Project Gutenberg eBook of History of the Decline and Fall…", + p8: "In imitation of his classical models, Gibbon places Rome as the…", + p39: "In the second century of the Christian Æra, the empire of Rome…", + p71: "Tacit. Annal. iv. 5. Those who fix a regular proportion…", + p127: "It was by such institutions that the nations of the empire…", + p177: "Among many other passages, see Pliny on the extent of the…", + p462: "The received calculation of Varro assigns to the foundation…", +}; + +for (const [label, hits] of [["by meaning", byMeaning], ["by word", byWord]] as const) { + console.log(`\n== ${label}`); + for (const hit of hits) { + console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(2)} ${text[hit.id]}`); + } +} + +// The two score columns above mean different things — a cosine distance and +// a ts_rank_cd — and no scaling makes them comparable. So the ranks are used +// and the scores thrown away. +console.log("\n== fused"); +for (const hit of fuseHits([...byMeaning, ...byWord], undefined, undefined, 5)) { + console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)} ${text[hit.id]}`); +} + +// One channel can be trusted more than the other where a corpus warrants it. +console.log("\n== fused, trusting the vectors twice as much"); +for (const hit of fuseHits([...byMeaning, ...byWord], undefined, { vec: 2 }, 5)) { + console.log(` ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)} ${text[hit.id]}`); +} + +console.log(` + Reciprocal rank fusion scores a hit by where it placed, not by what it + scored: 1/(k + rank) from each list that found it, added up. p462 is + third in one list and second in the other and wins, because agreement + between two different ways of looking is evidence; p0 is first in one + and absent from the other, and drops. + + This is fusion, and it is often called re-ranking, but a second thing + wears the same name. Retrieval uses a bi-encoder: the passage and the + query are embedded separately, which is what lets passage vectors be + computed once and indexed — and is also what limits it, since nothing in + a passage's vector knows what was asked. + + A cross-encoder reads the query and the passage together and scores the + pair. Far more accurate, far too slow to run over a corpus, so it runs + over the twenty or fifty candidates retrieval already found. The two + stack: retrieve both ways, fuse as above, then hand the top of the fused + list to a cross-encoder for the final order. Nothing here loads a model, + so that step is yours — the fused list is what you would hand it.`); diff --git a/examples/typescript/tsconfig.json b/examples/typescript/tsconfig.json new file mode 100644 index 0000000..c2804c2 --- /dev/null +++ b/examples/typescript/tsconfig.json @@ -0,0 +1,15 @@ +{ + "compilerOptions": { + "target": "es2022", + "module": "nodenext", + "moduleResolution": "nodenext", + "lib": ["es2023"], + "types": ["node"], + "strict": true, + "noEmit": true, + "allowImportingTsExtensions": true, + "rewriteRelativeImportExtensions": true, + "verbatimModuleSyntax": true + }, + "include": ["src"] +}