Both registries already have a `catena`: an HPE blockchain client on PyPI,
and a JavaScript OOP wrapper at 2.2.0 on npm. Neither is going anywhere, so
the distributions take a name that is free on both.
The Python import is unaffected — a distribution name and a module name are
independent there, so `pip install catena-corpus` still gives `import
catena`, and nothing in the examples or the bindings changes. On npm the
package name is the import specifier, so that one does change: `import
{ chunkMarkdown } from "catena-corpus"`.
The crates keep their own names. `catena` is free on crates.io.
Eight of them, each doing something a pipeline actually has to do, and each
starting from something anyone can fetch — a public domain book from Project
Gutenberg, a NASA broadcast, or a sample carried in the file.
Shell, because the tools are filters first: a book to passages and what its
notes cite; a recording to chapter marks, citable passages and the one
ffmpeg call that makes it play on a phone; passages into pgvector and a
search that asks twice and reconciles.
Rust, for the case where the caller already holds the text and a subprocess
would be serialising for nothing. Python, for the shape a pipeline stage
takes once the reading is a library call. TypeScript, for the app side —
chapter marks and deep links, and fusing two rankings in process.
Three of them end with what re-ranking means, because it is two different
things under one name: fusing ranked lists, which is what `rrf` does and
needs no model, and a cross-encoder scoring query and passage together,
which is accurate, slow, and runs over what retrieval already found.
Cuts a reference list into entries — at the note numbers where there are any,
at the layout where there are not — and reads each as far as its punctuation
allows. Fields that would not parse are named in `unsettled` rather than
guessed at, and `--prompt` asks a model about exactly those and no others.
Over 202 real reference sections from the corpus: 8,566 references, 95% with
a title, 82% with a year, 46% settling every field with no model involved.
A title is refused where the entry was never divided, so no record reports
its own text back as the work it cites.
`ibid.` is flagged and left alone — it points at a work, and the entry does
not say which. A rule of dashes is filled in, because the style defines it as
the author above.
chunkmd gains `note_starts` and `cites_something`: recognising a page of
notes and splitting one are the same knowledge, and two copies of that
pattern would drift.
funkpower