examples: programs that run, in four languages

Eight of them, each doing something a pipeline actually has to do, and each starting from something anyone can fetch — a public domain book from Project Gutenberg, a NASA broadcast, or a sample carried in the file. Shell, because the tools are filters first: a book to passages and what its notes cite; a recording to chapter marks, citable passages and the one ffmpeg call that makes it play on a phone; passages into pgvector and a search that asks twice and reconciles. Rust, for the case where the caller already holds the text and a subprocess would be serialising for nothing. Python, for the shape a pipeline stage takes once the reading is a library call. TypeScript, for the app side — chapter marks and deep links, and fusing two rankings in process. Three of them end with what re-ranking means, because it is two different things under one name: fusing ranked lists, which is what `rrf` does and needs no model, and a cross-encoder scoring query and passage together, which is accurate, slow, and runs over what retrieval already found.
2026-10-22 18:57:14 +00:00 · 2026-09-21 08:12:13 -04:00
parent cfa86f778f
commit 58be244478
14 changed files with 1114 additions and 9 deletions
@@ -48,6 +48,10 @@ Each tool installs by name, so you can take one without taking the rest.
 Each links to its own page: what it does, when you would use it, worked
 examples, options and exit codes.
 **→ [Examples](examples/README.md) — programs that run.** A book to
 passages and references, a recording to chapter marks and a copy that plays
 on a phone, and a hybrid search fused, in shell, Rust, Python and TypeScript.
 **→ [Recipes](docs/recipes.md) — using these tools together.** A scanned book
 to a searchable passage, a transcript to a citation with a timestamp, vectors
 into pgvector, and hybrid search fused on top.
@@ -0,0 +1,3 @@
 work/
 typescript/node_modules/
 typescript/package-lock.json
@@ -0,0 +1,125 @@
 # Examples
 Working programs that put these tools together, in the four ways you would
 actually reach for them: as filters in a shell script, and as libraries from
 Rust, Python and JavaScript.
 Every one of them runs. Where an example needs a book it fetches a public
 domain one from Project Gutenberg; where it needs a recording it fetches a
 NASA broadcast, which is a work of the US government.
 | | |
 |---|---|
 | [`shell/01-book-to-passages.sh`](shell/01-book-to-passages.sh) | A book file to passages worth embedding, and what its notes cite |
 | [`shell/02-video-to-chapters.sh`](shell/02-video-to-chapters.sh) | A recording to chapter marks, citable passages, and a copy that plays on a phone |
 | [`shell/03-search-and-rerank.sh`](shell/03-search-and-rerank.sh) | Passages into pgvector, then a search that asks twice and reconciles |
 | [`rust/`](rust/src/main.rs) | The same book walked through in one process, with no pipe anywhere |
 | [`python/book_to_citations.py`](python/book_to_citations.py) | A book to rows a database can take |
 | [`python/search_and_rerank.py`](python/search_and_rerank.py) | The pgvector search, in the language a pipeline is usually written in |
 | [`typescript/src/episode-to-chapters.ts`](typescript/src/episode-to-chapters.ts) | Captions into chapter marks and deep links for a player |
 | [`typescript/src/search-and-rerank.ts`](typescript/src/search-and-rerank.ts) | Fusing two rankings app-side, and what re-ranking means |
 ## Shell
 The tools are filters first: bytes in, records out, and `jq` between them.
 Nothing is installed but the commands themselves.
 ```sh
 cargo install --git https://git.pumpin.online/funkpower/catena \
    booksniff bookmeta tocfind pdftriage mdscrub chunkmd citefind \
    cuenorm cueseg cuepack avplan emb rrf
 examples/shell/01-book-to-passages.sh ./work
 ```
 `01` fetches Gibbon's first volume, works out what the file is, reads the
 metadata and the contents the publisher declared, converts and scrubs and
 chunks it, and reads the footnotes. Needs `pandoc` and `jq`.
 `02` takes a recording's captions and produces chapter marks, passages that
 carry the second they were said at, and — since a recording that is indexed
 but unplayable is half a job — the one `ffmpeg` call that makes it play on a
 phone. Needs `jq`, `ffmpeg`, and yt-dlp with impersonation support, which
 YouTube now requires before it will hand over captions:
 ```sh
 pipx install "yt-dlp[default,curl-cffi]"
 ```
 `03` loads the passages `01` wrote into Postgres with a vector column and a
 text index, searches both ways, and fuses the two rankings. Needs `psql`, a
 Postgres with pgvector, and an embedding server:
 ```sh
 DATABASE_URL=postgresql://user@host/db \
 EMB_MODEL=nomic-embed-text \
    examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
 ```
 ## Rust
 Every tool is a library with a thin command on top, so anything a command
 does can be done in process instead. This walks a file from "what even is it"
 to passages and references without serialising anything.
 ```sh
 cd examples/rust
 cargo run                 # the built-in sample
 cargo run -- book.md      # a real one
 ```
 ## Python
 The shape a pipeline stage ends up as once the reading is a library call:
 convert, then loop, and what is left is the part that is actually about your
 schema.
 ```sh
 pip install catena
 python examples/python/book_to_citations.py
 ```
 ```sh
 pip install catena psycopg
 DATABASE_URL=postgresql://user@host/db \
    python examples/python/search_and_rerank.py "why did the empire decline"
 ```
 ## TypeScript
 The bindings are WebAssembly, so they run in Node and in a browser with the
 same API.
 ```sh
 cd examples/typescript
 npm install
 npm run chapters          # captions to chapter marks
 npm run search            # fusing two rankings
 ```
 ## On re-ranking
 Three of these examples end with the same explanation, because it is the
 thing most often got wrong.
 Retrieval uses a **bi-encoder**: the passage and the query are embedded
 separately. That independence is what lets passage vectors be computed once
 and indexed, and it is also the limit — nothing in a passage's vector knows
 what was asked.
 **Fusion** takes several ranked lists and makes one. Reciprocal rank fusion
 scores a hit by where it placed rather than by what it scored: `1/(k + rank)`
 from each list that found it, added up. It is used because the scores are not
 comparable — a cosine distance and a `ts_rank_cd` measure different things,
 and no scaling makes them the same — so the ranks are kept and the scores
 thrown away. A passage that both searches liked beats one that either loved.
 That is [`rrf`](../crates/rrf/README.md), and it needs no model.
 A **cross-encoder** reads the query and the passage together and scores the
 pair. Far more accurate, and far too slow to run over a corpus, so it runs
 over the twenty or fifty candidates retrieval already found. In a pipeline
 the two stack: retrieve both ways, fuse the lists, then hand the top of the
 fused list to a cross-encoder for the final order.
 Both get called re-ranking. Nothing here loads a model, so the last step is
 yours — the fused list is what you would hand it.
@@ -0,0 +1,128 @@
 """A book to rows a database can take, without a subprocess in sight.
 This is the shape a pipeline stage ends up as once the reading is a library
 call: convert, then loop, and what is left is the part that is actually about
 your schema.
    python examples/python/book_to_citations.py path/to/book.epub
 With no argument it fetches Gibbon's first volume from Project Gutenberg,
 which is public domain and has a few hundred pages of footnotes.
 """
 import pathlib
 import subprocess
 import sys
 import urllib.request
 import catena
 GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub"
 def fetch(into: pathlib.Path) -> pathlib.Path:
    """The sample book, downloaded once."""
    book = into / "gibbon.epub"
    if not book.exists():
        into.mkdir(parents=True, exist_ok=True)
        print(f"fetching {GUTENBERG}")
        with urllib.request.urlopen(GUTENBERG, timeout=120) as reply:  # noqa: S310
            book.write_bytes(reply.read())
    return book
 def to_markdown(book: pathlib.Path) -> str:
    """Pandoc is the one part of this that is somebody else's program.
    `commonmark` rather than `commonmark_x`: the extended flavour writes
    anchors as bracketed spans, and they end up inside the passages.
    """
    done = subprocess.run(
        ["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)],
        capture_output=True,
        text=True,
        check=True,
    )
    return done.stdout
 def rows(markdown: str) -> list[dict]:
    """Passages in, citation rows out.
    Two library calls do the work. `chunk_markdown` cuts the book into
    passages and flags the ones carrying citation material — which is the
    whole reason to ask it rather than looking for a heading, since a page of
    notes at the foot of a chapter has no heading over it. `find_citations`
    then reads each of those.
    """
    out = []
    for chunk in catena.chunk_markdown(catena.scrub(markdown)):
        if "footnote_citations" not in chunk["flags"] and chunk["role"] != "bibliography":
            continue
        for found in catena.find_citations(chunk["text"]):
            # An `ibid.` or a short cite points at an entry above rather than
            # naming a work of its own, so there is nothing to store.
            if found["continuation"]:
                continue
            if not (found["title"] or found["authors"] or found["ids"]):
                continue
            out.append(
                {
                    "page": chunk["page"],
                    "chapter": chunk["chapter"],
                    "citation_text": found["text"],
                    "referenced_work": found["title"],
                    "referenced_author": "; ".join(found["authors"]) or None,
                    "referenced_year": found["year"],
                    "referenced_container": found["container"],
                    "kind": found["kind"],
                    "unsettled": found["unsettled"],
                }
            )
    return out
 def main() -> int:
    where = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else fetch(pathlib.Path("work"))
    print(f"== {where.name}")
    identified = catena.identify(where.name, where.read_bytes())
    print(f"   a {identified['kind']}, {identified['bytes']} bytes")
    markdown = to_markdown(where)
    passages = catena.chunk_markdown(catena.scrub(markdown))
    carrying = [c for c in passages if "footnote_citations" in c["flags"]]
    print(f"   {len(passages)} passages, {len(carrying)} carrying citation material")
    found = rows(markdown)
    named = [r for r in found if r["referenced_work"]]
    print(f"\n== {len(found)} rows, {len(named)} naming a work")
    # What a stage like this would do next is its own business: one executemany
    # against your own table. The point is that everything above it is two
    # function calls rather than a parser you have to keep.
    print("\n   INSERT INTO citations (citation_text, referenced_work, referenced_author, …)")
    whole = [r for r in named if r["referenced_author"]] or named
    for row in whole[:5]:
        work = (row["referenced_work"] or "")[:40]
        who = (row["referenced_author"] or "—")[:22]
        print(f"     (…, {work!r}, {who!r}, {row['referenced_year']})")
    # Nothing is invented. Where a field could not be read the record says so
    # rather than guessing, and a caller that wants a model to fill the gaps
    # has `citation_prompt` and `merge_citation_reply` for exactly that.
    # Gibbon cites in the classical short form — `Tacit. Annal. xiii. 5.` —
    # so most of his notes name a passage rather than a book. Nothing is
    # invented to cover that: the record says which fields it could not read,
    # and `citation_prompt` and `merge_citation_reply` are there for a caller
    # that wants a model to finish them.
    unsettled = [r for r in found if r["unsettled"]]
    print(f"\n   {len(unsettled)} rows name something they could not finish reading:")
    for row in unsettled[:3]:
        print(f"     {row['citation_text'][:56]} → {', '.join(row['unsettled'])}")
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1,192 @@
 """Passages into pgvector, then a search that asks twice and reconciles.
 Loads the passages of a book into Postgres with both a vector column and a
 text index, searches both ways, and fuses the two rankings into one.
    DATABASE_URL=postgresql://user@host/db \
    EMB_MODEL=nomic-embed-text \
        python examples/python/search_and_rerank.py "why did the empire decline"
 Needs psycopg, a Postgres with pgvector, an embedding server with Ollama's
 API, and a book — the sample is fetched if none is given.
 """
 import json
 import os
 import pathlib
 import subprocess
 import sys
 import urllib.request
 import catena
 import psycopg
 EMB_HOST = os.environ.get("EMB_HOST", "http://localhost:11434")
 EMB_MODEL = os.environ.get("EMB_MODEL", "nomic-embed-text")
 GUTENBERG = "https://www.gutenberg.org/cache/epub/731/pg731.epub"
 def embed(texts: list[str]) -> list[list[float]]:
    """Ask the embedding server, in batches it will accept.
    The one thing here that touches the network. No model is loaded in this
    process: it belongs behind a server, where it is shared and where its
    GPU is somebody else's problem.
    """
    out: list[list[float]] = []
    for at in range(0, len(texts), 32):
        batch = texts[at : at + 32]
        body = json.dumps({"model": EMB_MODEL, "input": batch}).encode()
        request = urllib.request.Request(
            f"{EMB_HOST}/api/embed", data=body, headers={"content-type": "application/json"}
        )
        with urllib.request.urlopen(request, timeout=300) as reply:  # noqa: S310
            out.extend(json.load(reply)["embeddings"])
        print(f"   embedded {len(out)}/{len(texts)}", end="\r", file=sys.stderr)
    return out
 def passages(book: pathlib.Path) -> list[dict]:
    """The book, cut into passages worth embedding."""
    markdown = subprocess.run(
        ["pandoc", "-f", "epub", "-t", "commonmark", "--wrap=none", str(book)],
        capture_output=True, text=True, check=True,
    ).stdout
    chunks = catena.chunk_markdown(catena.scrub(markdown))
    # A table of contents embeds as well as anything and retrieves for every
    # query, which is worse than not having it.
    return [c for c in chunks if c["role"] == "body" and c["tokens"] > 40]
 def load(conn, rows: list[dict], vectors: list[list[float]]) -> None:
    width = len(vectors[0])
    with conn.cursor() as cur:
        cur.execute("CREATE EXTENSION IF NOT EXISTS vector")
        cur.execute("DROP TABLE IF EXISTS catena_passages")
        # The tsvector is generated from the body rather than written
        # alongside it, so the two cannot drift apart.
        cur.execute(f"""
            CREATE TABLE catena_passages (
                idx     integer PRIMARY KEY,
                chapter text,
                body    text NOT NULL,
                fts     tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
                vec     vector({width})
            )""")
        with cur.copy("COPY catena_passages (idx, chapter, body, vec) FROM STDIN") as copy:
            for chunk, vector in zip(rows, vectors):
                copy.write_row((chunk["idx"], chunk["chapter"], chunk["text"], str(vector)))
        cur.execute("CREATE INDEX ON catena_passages USING gin (fts)")
        cur.execute("CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops)")
    conn.commit()
 def by_meaning(conn, vector: list[float], limit: int = 10) -> list[dict]:
    """Nearest in the embedding space.
    `<=>` is cosine distance, so smaller is nearer. This finds a passage
    about the fall of Rome for a query about decline, sharing no words.
    """
    with conn.cursor() as cur:
        cur.execute(
            """SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g')
                 FROM catena_passages ORDER BY vec <=> %s::vector LIMIT %s""",
            (str(vector), limit),
        )
        return [
            {"v": 1, "id": str(idx), "channel": "vec", "rank": at, "score": 0.0, "text": body}
            for at, (idx, body) in enumerate(cur.fetchall(), start=1)
        ]
 def by_word(conn, query: str, limit: int = 10) -> list[dict]:
    """Nearest by the words themselves.
    Finds the passage that actually says the word where the vector search
    drifted to a paraphrase, and finds nothing at all where the words are
    not in the book — which is a useful thing to be told.
    """
    with conn.cursor() as cur:
        cur.execute(
            """SELECT idx, regexp_replace(left(body, 68), '\\s+', ' ', 'g')
                 FROM catena_passages
                WHERE fts @@ websearch_to_tsquery('english', %(q)s)
                ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', %(q)s)) DESC
                LIMIT %(n)s""",
            {"q": query, "n": limit},
        )
        return [
            {"v": 1, "id": str(idx), "channel": "fts", "rank": at, "score": 0.0, "text": body}
            for at, (idx, body) in enumerate(cur.fetchall(), start=1)
        ]
 def main() -> int:
    query = sys.argv[1] if len(sys.argv) > 1 else "why did the roman empire decline"
    url = os.environ.get("DATABASE_URL")
    if not url:
        print("set DATABASE_URL", file=sys.stderr)
        return 64
    work = pathlib.Path("work")
    work.mkdir(exist_ok=True)
    book = work / "gibbon.epub"
    if not book.exists():
        print(f"fetching {GUTENBERG}")
        with urllib.request.urlopen(GUTENBERG, timeout=120) as reply:  # noqa: S310
            book.write_bytes(reply.read())
    rows = passages(book)
    print(f"== {len(rows)} passages")
    vectors = embed([" ".join(chunk["text"].split()) for chunk in rows])
    print(f"   {EMB_MODEL} returns {len(vectors[0])} numbers per passage")
    with psycopg.connect(url) as conn:
        load(conn, rows, vectors)
        print("   loaded into catena_passages")
        print(f"\n== query: {query}")
        hits = by_meaning(conn, embed([query])[0]) + by_word(conn, query)
    for channel, label in (("vec", "by meaning"), ("fts", "by word")):
        print(f"\n   {label}:")
        for hit in [h for h in hits if h["channel"] == channel][:4]:
            print(f"   {hit['rank']}. [{hit['id']}] {hit['text']}")
    # Fusion is a library call rather than a subprocess: by the time a hit
    # reaches here it is already yours, and serialising it to hand to a
    # program and back is work for nothing.
    print("\n== fused")
    text_of = {hit["id"]: hit["text"] for hit in hits}
    for hit in catena.fuse_hits(hits, limit=5):
        print(f"   {hit['rank']}. [{hit['id']}] score {hit['score']:.3f}  {text_of[hit['id']]}")
    print(RERANKING)
    return 0
 RERANKING = """
   Reciprocal rank fusion scores a passage by where it placed, not by what
   it scored: 1/(k + rank) from each list that found it, added up. The two
   searches return numbers that mean different things — a cosine distance
   and a ts_rank_cd — and no scaling makes them comparable, so their ranks
   are used and their scores thrown away. A passage both searches liked
   beats one that either loved, which is the behaviour worth having.
   This is fusion, and it is often called re-ranking, but a second thing
   wears the same name. Retrieval uses a bi-encoder: passage and query are
   embedded separately, which is what lets the passage vectors be computed
   once and indexed, and is also what limits it — nothing in a passage's
   vector knows what was asked.
   A cross-encoder reads the query and the passage together and scores the
   pair. It is far more accurate and far too slow to run over a corpus, so
   it runs over the twenty or fifty candidates retrieval already found. The
   two stack: retrieve both ways, fuse the lists, then hand the top of the
   fused list to a cross-encoder for the final order. Nothing here loads a
   model, so that step is yours — the fused list is what you would hand it.
 """
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -0,0 +1 @@
 target/
@@ -8,6 +8,13 @@
 //!
 //!     cargo run -- book.md
 //!     cargo run          # reads the sample below
 //!
 //! For a real book, convert one first — Gibbon's first volume is public
 //! domain and has a few hundred pages of footnotes:
 //!
 //!     curl -sL -o gibbon.epub https://www.gutenberg.org/cache/epub/731/pg731.epub
 //!     pandoc -f epub -t commonmark --wrap=none gibbon.epub -o gibbon.md
 //!     cargo run -- gibbon.md
 use std::path::Path;
@@ -74,8 +81,8 @@ fn main() {
        },
    };
    let chunks = chunkmd::chunk(&clean, &budgets);
-    println!("\n{} passage(s):", chunks.len());
+    println!("\n{} passage(s), the first few:", chunks.len());
-    for chunk in &chunks {
+    for chunk in chunks.iter().take(6) {
        let chapter = chunk.chapter.as_deref().unwrap_or("—");
        let opening: String = chunk.text.chars().take(64).collect();
        println!(
@@ -87,18 +94,50 @@ fn main() {
        );
    }
-    // The references at the back. A passage whose role is `bibliography` is
+    // The references. A passage whose role is `bibliography` is one place
-    // where they live; a note at the foot of a page is the other place, and
+    // they live; a page of notes at the foot of a chapter is the other, and
-    // `chunkmd` flags those so this does not have to guess.
+    // it has no heading over it to find it by — which is why `chunkmd`
-    let references: Vec<citefind::Reference> = chunks
+    // flags those, so this does not have to guess.
    let carrying: Vec<&corpus_core::Chunk> = chunks
        .iter()
        .filter(|chunk| {
            chunk.role == corpus_core::Role::Bibliography
                || chunk
                    .flags
                    .iter()
                    .any(|flag| flag == corpus_core::chunk::flags::FOOTNOTE_CITATIONS)
        })
        .collect();
    let references: Vec<citefind::Reference> = carrying
        .iter()
        .filter(|chunk| chunk.role == corpus_core::Role::Bibliography)
        .flat_map(|chunk| citefind::read(&chunk.text))
        .filter(|found| !found.continuation)
        .collect();
-    println!("\n{} reference(s):", references.len());
+    println!(
-    for found in &references {
+        "\n{} passage(s) carry citation material, giving {} reference(s):",
        carrying.len(),
        references.len()
    );
    // The ones that named something, which is not all of them: a book whose
    // notes cite in the classical short form — `Tacit. Annal. xiii. 5.` —
    // gives mostly passages rather than works, and the record says as much
    // rather than inventing a title to fill the field.
    let with_work = references
        .iter()
        .filter(|found| found.title.is_some())
        .count();
    let named: Vec<&citefind::Reference> = references
        .iter()
        .filter(|found| !found.authors.is_empty())
        .collect();
    println!(
        "{with_work} name a work and {} name an author; those:",
        named.len()
    );
    for found in named.iter().take(6) {
        let who = match found.authors.is_empty() {
            true => "—".to_string(),
            false => found.authors.join("; "),
@@ -116,4 +155,16 @@ fn main() {
            println!("    could not read: {}", found.unsettled.join(", "));
        }
    }
    // Worth saying plainly, because the numbers above look poor on a book
    // like Gibbon and the reason is not the parser having failed quietly.
    // His notes cite in the classical short form — `Tacit. Annal. xiii. 5.`
    // — which names a passage rather than a work, so there is no title and
    // no year to read and the record says which fields it could not fill
    // rather than inventing them. `citation_prompt` and `merge_citation_reply`
    // are there for a caller that wants a model to finish the job.
    if references.len() > 20 && named.len() * 10 < references.len() {
        println!("\n    most of these are classical short cites — a passage");
        println!("    rather than a work — there is nothing more to read in them.");
    }
 }
@@ -0,0 +1,93 @@
 #!/bin/sh
 # A book file to passages worth embedding, and what its notes cite.
 #
 # The starting point is Gibbon's first volume from Project Gutenberg: public
 # domain, a real table of contents, and a few hundred pages of footnotes.
 #
 #     examples/shell/01-book-to-passages.sh [work-dir]
 #
 # Needs pandoc and jq, and the catena commands on PATH.
 set -eu
 work="${1:-./work}"
 mkdir -p "$work"
 book="$work/gibbon.epub"
 if [ ! -s "$book" ]; then
 	echo "fetching Gibbon vol. 1 from Project Gutenberg"
 	curl -sSL --max-time 120 -o "$book" \
 		"https://www.gutenberg.org/cache/epub/731/pg731.epub"
 fi
 # What the file actually is. Extensions lie, and a `.epub` that is really a
 # zip of page scans fails four stages later, in a way that looks like a bug
 # in the converter.
 echo
 echo "== what it is"
 booksniff "$book" | jq -r '"   \(.kind), \(.bytes) bytes"'
 # A PDF gets one more question first: exit 0 from pdftriage means the text
 # layer is not worth reading and the file has to be OCRed before any of this.
 if [ "$(booksniff "$book" | jq -r .kind)" = "pdf" ] && pdftriage --quiet "$book"; then
 	echo "   the text layer is garbled; OCR it before going on"
 	exit 0
 fi
 # What the file already says about itself — the publisher's own metadata,
 # which beats anything guessed from a filename, and costs no lookup.
 echo
 echo "== what it says about itself"
 bookmeta "$book" | jq -r '"   \(.title)\n   \(.authors | join("; "))\n   \(.language // "?") · \(.date // "no date")"'
 # The chapters the book declares, rather than any inferred from the text.
 echo
 echo "== the contents it declares"
 tocfind "$book" --outline | sed -n '6,10p' | sed 's/^/   /'
 echo "   … $(tocfind "$book" | wc -l | tr -d ' ') entries"
 # Convert, scrub, chunk. `commonmark` rather than `commonmark_x`, because the
 # extended flavour writes anchors as bracketed spans and they end up in the
 # passages. The scrub is what takes out the rest of the converter's leavings.
 echo
 echo "== passages"
 pandoc -f epub -t commonmark --wrap=none "$book" -o "$work/book.md"
 mdscrub < "$work/book.md" > "$work/clean.md"
 chunkmd < "$work/clean.md" > "$work/chunks.ndjson"
 jq -s -r '
 	"   \(length) passages over \([.[].chapter] | unique | length) chapters",
 	"   roles: \([.[].role] | group_by(.) | map("\(length) \(.[0])") | join(", "))",
 	"   \([.[] | select(.flags | index("footnote_citations"))] | length) carry citation material"
 ' "$work/chunks.ndjson"
 echo
 echo "   a passage, with where it came from:"
 jq -s -r '[.[] | select(.role == "body" and .chapter != null)][40]
 	| "   [\(.idx)] \(.tokens) tokens · \(.chapter)\n   \(.text[0:150])…"' \
 	"$work/chunks.ndjson"
 # The notes. A passage flagged `footnote_citations` is a page of them, and
 # each is read on its own — the flag says which passages are worth the ask,
 # which is the whole reason chunkmd sets it.
 echo
 echo "== what the notes cite"
 : > "$work/references.ndjson"
 jq -c 'select(.flags | index("footnote_citations")) | .text' "$work/chunks.ndjson" \
 	| while IFS= read -r line; do
 		printf '%s' "$line" | jq -r . | citefind >> "$work/references.ndjson" || true
 	done
 jq -s -r '
 	"   \(length) entries read",
 	"   \([.[] | select(.title)] | length) name a work, \([.[] | select(.authors | length > 0)] | length) name an author",
 	"   \([.[] | select(.unsettled | length > 0)] | length) came back with something unread"
 ' "$work/references.ndjson"
 echo
 echo "   Gibbon cites in the classical short form — \"Tacit. Annal. xiii. 5.\" —"
 echo "   so most entries name the passage rather than a book, and say so:"
 jq -r 'select(.unsettled | length > 0) | "   \(.text[0:58]) → unread: \(.unsettled | join(", "))"' \
 	"$work/references.ndjson" | head -4
 echo
 echo "written to $work/"
@@ -0,0 +1,109 @@
 #!/bin/sh
 # A recording to chapter marks, citable passages, and a copy that plays.
 #
 # Takes the captions a video already carries, normalises them to one cue
 # stream, and turns that into chapter marks for a player and passages worth
 # embedding — each keeping the second it was said at. Then asks whether the
 # file itself would play on a phone, and prints the one ffmpeg call that
 # fixes it.
 #
 #     examples/shell/02-video-to-chapters.sh [url] [work-dir]
 #
 # The default is a NASA upload: a work of the US government, so public
 # domain, and captioned by a person rather than by a model. Any URL yt-dlp
 # can reach works, and any local subtitle file works too — skip to the
 # cuenorm line and feed it your own.
 #
 # Needs jq, ffmpeg, and yt-dlp with impersonation support, which YouTube now
 # requires before it will hand over captions:
 #
 #     pipx install "yt-dlp[default,curl-cffi]"
 #
 # and the catena commands on PATH.
 set -eu
 url="${1:-https://www.youtube.com/watch?v=h0wLZJeYGxw}"
 work="${2:-./work}"
 mkdir -p "$work"
 # Captions written by a person where there are any, and a model's otherwise.
 # Automatic captions roll — each line is re-sent as it scrolls up the screen —
 # so the same sentence arrives three times and every passage says it thrice.
 if [ -z "$(find "$work" -name '*.vtt' -print -quit 2>/dev/null)" ]; then
 	echo "fetching captions"
 	yt-dlp --skip-download --write-subs --sub-langs 'en.*' --sub-format vtt \
 		--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 ||
 		yt-dlp --skip-download --write-auto-subs --sub-langs 'en.*' --sub-format vtt \
 			--output "$work/%(id)s.%(ext)s" "$url" >/dev/null
 fi
 # Where several tracks came back, the smallest is the written one: a rolling
 # track repeats every line and is four or five times the size.
 subs="$(find "$work" -name '*.vtt' -exec ls -S {} + | tail -1)"
 [ -n "$subs" ] || { echo "no captions on that video; try another URL" >&2; exit 1; }
 echo "   captions: $(basename "$subs")"
 echo
 echo "== cues"
 cuenorm "$subs" > "$work/cues.ndjson"
 jq -s -r '"   \(length) cues over \((map(.end) | max) / 60 | floor)m\((map(.end) | max) % 60 | floor)s"' "$work/cues.ndjson"
 jq -r '"   [\(.start | floor)s] \(.text[0:62])"' "$work/cues.ndjson" | head -3
 # Where the subject changes, found from the words alone — no model, no
 # network. The titles come out of each chapter's own first sentence, and are
 # meant to be replaced by something better later.
 echo
 echo "== chapters"
 cueseg < "$work/cues.ndjson" > "$work/chapters.ndjson"
 jq -r '"   \(.start | floor)s–\(.end | floor)s  \(.title)"' "$work/chapters.ndjson"
 # Cues packed to a size worth embedding, each carrying the seconds it spans,
 # so a search hit can be played back at the moment it was said.
 echo
 echo "== passages"
 cuepack < "$work/cues.ndjson" > "$work/passages.ndjson"
 jq -s -r '"   \(length) passages, \(map(.tokens) | add / length | floor) tokens on average"' "$work/passages.ndjson"
 jq -s -r '.[0] | "   [\(.start_time | floor)s–\(.end_time | floor)s] \(.text[0:150])…"' "$work/passages.ndjson"
 echo
 echo "== chapter marks, as a player would take them"
 jq -r '"\(.start | floor) \(.title)"' "$work/chapters.ndjson" |
 	awk '{ t = $1; $1 = ""; printf "   %02d:%02d:%02d%s\n", t/3600, (t%3600)/60, t%60, $0 }'
 # And the file itself. A recording indexed but unplayable is half a job: a
 # phone will not open VP9 in a WebM container, whatever the transcript says.
 # Twenty seconds of it is enough to judge the streams, and spares the
 # download.
 echo
 echo "== would it play on a phone"
 video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
 if [ -z "$video" ]; then
 	echo "   fetching twenty seconds of it"
 	yt-dlp -f 'worstvideo[ext=webm]+worstaudio[ext=webm]/worst' \
 		--download-sections '*0-20' --force-keyframes-at-cuts \
 		--output "$work/%(id)s.%(ext)s" "$url" >/dev/null 2>&1 || true
 	video="$(find "$work" -type f \( -name '*.webm' -o -name '*.mkv' -o -name '*.mp4' \) -print -quit)"
 fi
 [ -n "$video" ] || { echo "   nothing to judge"; exit 0; }
 avplan --json "$video" | jq -r '
 	"   \(.streams.container): \(.streams.video) video, \(.streams.audio // "no") audio",
 	(if .playable then "   plays as it stands" else (.reasons[] | "   · \(.)") end)'
 # Exit 0 means there is a plan to print; 1 means the file already plays. The
 # encoder is whatever this machine actually has — videotoolbox on a Mac,
 # NVENC or VAAPI on a Linux box, libx264 where there is no card. Nothing is
 # run: converting is expensive and irreversible enough to read first.
 echo
 if plan="$(avplan --output "$work/for-phone.mp4" "$video")"; then
 	echo "   the one call that fixes it:"
 	echo
 	printf '   %s\n' "$plan"
 	echo
 	echo "   run it with:  eval \"\$(avplan --output out.mp4 FILE)\""
 else
 	echo "   nothing to convert"
 fi
 echo
 echo "written to $work/"
@@ -0,0 +1,178 @@
 #!/bin/sh
 # Passages into pgvector, then a search that asks twice and reconciles.
 #
 # Takes the passages example 01 wrote, embeds them, loads them into Postgres
 # with a vector column and a text index, then searches both ways and fuses
 # the two lists into one.
 #
 #     examples/shell/01-book-to-passages.sh ./work
 #     examples/shell/03-search-and-rerank.sh "why did the empire decline" ./work
 #
 # Needs psql and jq, a Postgres with pgvector, an embedding server, and the
 # catena commands on PATH:
 #
 #     DATABASE_URL=postgresql://user@host/db
 #     EMB_HOST=http://localhost:11434     # ollama, or anything with its API
 #     EMB_MODEL=nomic-embed-text          # 768 wide; bge-m3 is 1024
 set -eu
 query="${1:-why did the roman empire decline}"
 work="${2:-./work}"
 : "${DATABASE_URL:?set DATABASE_URL}"
 host="${EMB_HOST:-http://localhost:11434}"
 model="${EMB_MODEL:-nomic-embed-text}"
 [ -s "$work/chunks.ndjson" ] || { echo "run 01-book-to-passages.sh first" >&2; exit 1; }
 psql="psql -X -q -v ON_ERROR_STOP=1 $DATABASE_URL"
 tab=$(printf '\t')
 # How wide the model's vectors are. Asked rather than assumed, because the
 # column is typed and a mismatch is a wall of errors at insert time.
 dim=$(printf 'how wide is this\n' | emb --host "$host" --model "$model" | jq 'length')
 echo "== $model returns $dim numbers per passage"
 # One table. The vector is what the model said; the tsvector is Postgres's
 # own, generated from the text so it cannot drift out of step with it.
 $psql <<SQL
 CREATE EXTENSION IF NOT EXISTS vector;
 DROP TABLE IF EXISTS catena_passages;
 CREATE TABLE catena_passages (
    idx      integer PRIMARY KEY,
    chapter  text,
    body     text NOT NULL,
    tokens   integer NOT NULL,
    fts      tsvector GENERATED ALWAYS AS (to_tsvector('english', body)) STORED,
    vec      vector($dim)
 );
 CREATE INDEX ON catena_passages USING gin (fts);
 SQL
 # Body passages only: a table of contents embeds as well as anything and
 # retrieves for every query, which is worse than not having it.
 jq -c 'select(.role == "body" and .tokens > 40)' "$work/chunks.ndjson" > "$work/body.ndjson"
 echo "== embedding $(wc -l < "$work/body.ndjson" | tr -d ' ') passages"
 # The newlines have to go before the text is a line: emb is line-oriented,
 # so a passage containing one would arrive as two.
 jq -r '.text | gsub("\\s+"; " ")' "$work/body.ndjson" > "$work/lines.txt"
 emb --host "$host" --model "$model" --dim "$dim" < "$work/lines.txt" > "$work/vectors.ndjson"
 # Line for line, which is emb's promise, so paste is a safe join.
 jq -c '[.idx, .chapter, (.text | gsub("\\s+"; " ")), .tokens]' "$work/body.ndjson" > "$work/meta.ndjson"
 paste -d '\t' "$work/meta.ndjson" "$work/vectors.ndjson" |
 	python3 -c '
 import csv, json, sys
 out = csv.writer(sys.stdout, delimiter="\t", quoting=csv.QUOTE_NONE, escapechar="\\")
 for line in sys.stdin:
    meta, vector = line.rstrip("\n").split("\t")
    idx, chapter, body, tokens = json.loads(meta)
    out.writerow([idx, chapter or "", body, tokens, vector])
 ' > "$work/rows.tsv"
 $psql -c "\copy catena_passages (idx, chapter, body, tokens, vec) FROM '$work/rows.tsv'"
 $psql -c "CREATE INDEX ON catena_passages USING hnsw (vec vector_cosine_ops);"
 $psql -tAc "SELECT '   ' || count(*) || ' passages loaded' FROM catena_passages"
 # psql's rows into hit records. `rrf` wants an id, which channel found it,
 # and what rank that channel gave it; the text is carried along so the
 # output is readable. Written in Python because the passages contain
 # quotation marks and a shell printf would not escape them.
 hits() {
 	python3 -c '
 import json, sys
 channel = sys.argv[1]
 for line in sys.stdin:
    line = line.rstrip("\n")
    if not line.strip():
        continue
    ident, rank, text = line.split("\t", 2)
    print(json.dumps({"v": 1, "id": ident, "channel": channel,
                      "rank": int(rank), "score": 0.0, "text": text}))
 ' "$1"
 }
 # ---------------------------------------------------------------------------
 # Two searches. They disagree, and that is the point.
 # ---------------------------------------------------------------------------
 echo
 echo "== query: $query"
 printf '%s\n' "$query" | emb --host "$host" --model "$model" --dim "$dim" | tr -d '\n' > "$work/query.vec"
 qvec=$(cat "$work/query.vec")
 # Nearest by meaning. `<=>` is cosine distance, so smaller is closer, and a
 # passage about "the fall of Rome" ranks even though it shares no words with
 # the query.
 $psql -tAF"$tab" -c "
  SELECT idx, row_number() OVER (ORDER BY vec <=> '$qvec'::vector),
         regexp_replace(left(body, 68), '\s+', ' ', 'g')
    FROM catena_passages
   ORDER BY vec <=> '$qvec'::vector
   LIMIT 10" | hits vec > "$work/hits.ndjson"
 # Nearest by word. Finds the passage that says "decline" where the vector
 # search drifted to a paraphrase, and finds nothing at all for a query whose
 # words are not in the book.
 $psql -tAF"$tab" -c "
  SELECT idx,
         row_number() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', \$\$$query\$\$)) DESC),
         regexp_replace(left(body, 68), '\s+', ' ', 'g')
    FROM catena_passages
   WHERE fts @@ websearch_to_tsquery('english', \$\$$query\$\$)
   ORDER BY 2
   LIMIT 10" | hits fts >> "$work/hits.ndjson"
 echo
 echo "   by meaning:"
 jq -r 'select(.channel == "vec") | "   \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
 echo
 echo "   by word:"
 jq -r 'select(.channel == "fts") | "   \(.rank). [\(.id)] \(.text)"' "$work/hits.ndjson" | head -4
 # ---------------------------------------------------------------------------
 # Reconciling them.
 # ---------------------------------------------------------------------------
 echo
 echo "== fused"
 rrf --limit 5 < "$work/hits.ndjson" > "$work/fused.ndjson"
 jq -s -r --slurpfile hits "$work/hits.ndjson" '
 	($hits | INDEX(.id)) as $by_id
 	| .[] | "   \(.rank). [\(.id)] score \(.score * 1000 | floor / 1000)  \($by_id[.id].text)"
 ' "$work/fused.ndjson"
 cat <<'NOTE'
   Reciprocal rank fusion scores a passage by where it placed, not by what
   it scored: 1/(k + rank) from each list that found it, added up. The two
   searches return numbers that mean different things — a cosine distance
   and a ts_rank_cd — and no amount of scaling makes them comparable, so
   their ranks are used and their scores thrown away. A passage both
   searches liked beats one that either loved, which is the behaviour worth
   having: agreement between two different ways of looking is evidence, and
   a high score from one is only ever evidence about that one.
   The run above shows it happening. The word search's first hit is the
   front matter, which says "Roman Empire" more often than any chapter of
   the book does; the vector search never offers it. Fusion pushes it down
   and lifts the one passage both searches found.
   The constant k, 60 by default, is how much a first place is worth over a
   second. Raise it and the lists flatten toward equal weight; lower it and
   the top of each list dominates. `--weight vec=2` trusts one channel more
   than the other, which is the knob to reach for when one of them is
   consistently better on your corpus.
   This is fusion, and it is often called re-ranking, but there is a second
   thing wearing the same name. Retrieval uses a bi-encoder: the passage
   and the query are embedded separately, so the passage vectors can be
   computed once and indexed, and a search is one vector comparison against
   an index. That independence is what makes it fast and is also what
   limits it — nothing in the passage's vector knows what was asked.
   A cross-encoder reads the query and the passage together and scores the
   pair. It is far more accurate and far too slow to run over a corpus, so
   it runs over the twenty or fifty candidates retrieval already found. In
   a pipeline the two stack: retrieve both ways, fuse the lists as above,
   then hand the top of the fused list to a cross-encoder for the final
   order. Nothing here does that step — it needs a model, and these tools
   do not load models — but the fused list is what you would hand it.
 NOTE
@@ -0,0 +1,15 @@
 {
  "name": "catena-example",
  "private": true,
  "type": "module",
  "scripts": {
    "chapters": "node --experimental-strip-types src/episode-to-chapters.ts",
    "search": "node --experimental-strip-types src/search-and-rerank.ts"
  },
  "dependencies": {
    "catena": "file:../../crates/bindings/npm/dist"
  },
  "devDependencies": {
    "@types/node": "^22"
  }
 }
@@ -0,0 +1,95 @@
 // A recording's captions into chapter marks and citable passages.
 //
 // What a player needs from an episode: somewhere to jump to, and passages
 // that carry the second they were said at so a search result can be played
 // rather than just quoted.
 //
 //     npm install
 //     npm run chapters -- path/to/captions.vtt
 //
 // With no argument it reads the sample below, which is a few cues of a NASA
 // broadcast — a work of the US government, so public domain.
 import { readFileSync } from "node:fs";
 import { parseSubtitles, segmentCues, packCues, countTokens } from "catena";
 const SAMPLE = `WEBVTT
 00:00:00.000 --> 00:00:04.200
 Hey there, and welcome to NASA's Curious Universe.
 00:00:04.200 --> 00:00:08.900
 Our universe is a wild and wonderful place, and in this
 show, NASA is your tour guide.
 00:00:08.900 --> 00:00:15.100
 Today we are going to talk about how we find planets
 around other stars, which is harder than it sounds.
 00:00:15.100 --> 00:00:21.800
 A planet is small and dark and its star is enormous and
 bright, so you almost never see the planet itself.
 00:00:21.800 --> 00:00:28.400
 Instead you watch the star, and you wait for it to dim
 by about one part in ten thousand.
 00:00:28.400 --> 00:00:34.000
 That dimming is a transit, and it is how most of the
 planets we know about were found.
 `;
 const path = process.argv[2];
 const content = path ? readFileSync(path, "utf8") : SAMPLE;
 // One cue stream, whatever the format was. `keepRepeats` is off, which is
 // what throws away the re-sent lines of a rolling automatic caption track —
 // leave it on and every passage says each sentence three times.
 const cues = parseSubtitles(content);
 const runsTo = Math.max(...cues.map((cue) => cue.end));
 console.log(`== ${cues.length} cues over ${Math.round(runsTo)}s`);
 for (const cue of cues.slice(0, 3)) {
  console.log(`   [${cue.start.toFixed(1)}s] ${cue.text.slice(0, 58)}`);
 }
 // Where the subject changes, from the words alone — no model, no network.
 // The titles come out of each chapter's own first sentence and are meant to
 // be replaced by something better later.
 const chapters = segmentCues(cues);
 console.log(`\n== ${chapters.length} chapter(s)`);
 for (const chapter of chapters) {
  console.log(`   ${stamp(chapter.start)}  ${chapter.title}`);
 }
 // Cues packed to a size worth embedding, each keeping the seconds it spans.
 const passages = packCues(cues);
 console.log(`\n== ${passages.length} passage(s)`);
 for (const passage of passages) {
  console.log(
    `   [${stamp(passage.start_time ?? 0)}–${stamp(passage.end_time ?? 0)}] ` +
      `${passage.tokens} tokens`,
  );
  console.log(`      ${passage.text.slice(0, 92)}…`);
 }
 // What a player would take: a cue sheet, and a deep link per passage.
 console.log("\n== what the player gets");
 for (const chapter of chapters) {
  console.log(`   ${stamp(chapter.start)} ${chapter.title}`);
 }
 console.log("\n== what a search result would link to");
 for (const passage of passages.slice(0, 2)) {
  const at = Math.floor(passage.start_time ?? 0);
  console.log(`   ?t=${at}  ${passage.text.slice(0, 64)}…`);
 }
 // The same token count the chunker used, so a budget worked out here and a
 // budget enforced there agree.
 console.log(`\n   ${countTokens(cues.map((c) => c.text).join(" "))} tokens of transcript in all`);
 function stamp(seconds: number): string {
  const whole = Math.floor(seconds);
  const mm = String(Math.floor(whole / 60)).padStart(2, "0");
  const ss = String(whole % 60).padStart(2, "0");
  return `${String(Math.floor(whole / 3600)).padStart(2, "0")}:${mm}:${ss}`;
 }
@@ -0,0 +1,96 @@
 // Two searches, reconciled — the app-side half of hybrid retrieval.
 //
 // Fusion is the one thing in this suite worth calling in process rather than
 // as a subprocess: by the time a hit reaches your code it is already yours,
 // and serialising it to hand to a program and back is work for nothing.
 //
 //     npm install
 //     npm run search
 //
 // The hits below stand in for two queries you would run against your own
 // store — pgvector and a text index, Elasticsearch and a KNN service, or a
 // vector database and whatever else. What they came from does not matter
 // here; only where each of them placed does.
 import { fuseHits, type Hit } from "catena";
 // `payload` is whatever you want carried alongside — rrf hands it back
 // untouched, so a hit can arrive here with the row it came from attached.
 const hit = (id: string, channel: string, rank: number, score: number): Hit => ({
  v: 1,
  id,
  channel,
  rank,
  score,
  payload: {},
 });
 // Nearest in the embedding space. Finds the passage about the fall of Rome
 // for a query about decline, sharing not one word with it.
 const byMeaning: Hit[] = [
  hit("p39", "vec", 1, 0.21),
  hit("p127", "vec", 2, 0.24),
  hit("p462", "vec", 3, 0.26),
  hit("p177", "vec", 4, 0.29),
 ];
 // Nearest by the words themselves. Finds the passage that says the word
 // where the vectors drifted to a paraphrase — and finds the front matter,
 // which says "Roman Empire" more often than any chapter of the book does.
 const byWord: Hit[] = [
  hit("p0", "fts", 1, 0.68),
  hit("p462", "fts", 2, 0.51),
  hit("p8", "fts", 3, 0.44),
  hit("p71", "fts", 4, 0.4),
 ];
 const text: Record<string, string> = {
  p0: "The Project Gutenberg eBook of History of the Decline and Fall…",
  p8: "In imitation of his classical models, Gibbon places Rome as the…",
  p39: "In the second century of the Christian Æra, the empire of Rome…",
  p71: "Tacit. Annal. iv. 5. Those who fix a regular proportion…",
  p127: "It was by such institutions that the nations of the empire…",
  p177: "Among many other passages, see Pliny on the extent of the…",
  p462: "The received calculation of Varro assigns to the foundation…",
 };
 for (const [label, hits] of [["by meaning", byMeaning], ["by word", byWord]] as const) {
  console.log(`\n== ${label}`);
  for (const hit of hits) {
    console.log(`   ${hit.rank}. [${hit.id}] ${hit.score.toFixed(2)}  ${text[hit.id]}`);
  }
 }
 // The two score columns above mean different things — a cosine distance and
 // a ts_rank_cd — and no scaling makes them comparable. So the ranks are used
 // and the scores thrown away.
 console.log("\n== fused");
 for (const hit of fuseHits([...byMeaning, ...byWord], undefined, undefined, 5)) {
  console.log(`   ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)}  ${text[hit.id]}`);
 }
 // One channel can be trusted more than the other where a corpus warrants it.
 console.log("\n== fused, trusting the vectors twice as much");
 for (const hit of fuseHits([...byMeaning, ...byWord], undefined, { vec: 2 }, 5)) {
  console.log(`   ${hit.rank}. [${hit.id}] ${hit.score.toFixed(4)}  ${text[hit.id]}`);
 }
 console.log(`
   Reciprocal rank fusion scores a hit by where it placed, not by what it
   scored: 1/(k + rank) from each list that found it, added up. p462 is
   third in one list and second in the other and wins, because agreement
   between two different ways of looking is evidence; p0 is first in one
   and absent from the other, and drops.
   This is fusion, and it is often called re-ranking, but a second thing
   wears the same name. Retrieval uses a bi-encoder: the passage and the
   query are embedded separately, which is what lets passage vectors be
   computed once and indexed — and is also what limits it, since nothing in
   a passage's vector knows what was asked.
   A cross-encoder reads the query and the passage together and scores the
   pair. Far more accurate, far too slow to run over a corpus, so it runs
   over the twenty or fifty candidates retrieval already found. The two
   stack: retrieve both ways, fuse as above, then hand the top of the fused
   list to a cross-encoder for the final order. Nothing here loads a model,
   so that step is yours — the fused list is what you would hand it.`);
@@ -0,0 +1,15 @@
 {
  "compilerOptions": {
    "target": "es2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "lib": ["es2023"],
    "types": ["node"],
    "strict": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "rewriteRelativeImportExtensions": true,
    "verbatimModuleSyntax": true
  },
  "include": ["src"]
 }