The suite could read a cataloguing block and could ask a register, and
nothing showed the two together — which is the only way either is worth
acting on. A title page alone gives up a running head when the scan
starts late; an ISBN alone returns a confidently wrong book when the
catalogue has been carrying the wrong one for years.
The example's title comparison folds to letters and digits rather than
comparing strings. A register and a title page differ over case, over a
curly apostrophe against a straight one, and over whether the subtitle
counts as part of the title. An example that called those disagreements
would teach the wrong lesson, and did until it was fixed.
It runs on this library and finds both of the cases worth finding: a file
called Document5 by -ducati996 that both sources agree is Russell's
Problems of Philosophy, and a Pelikan whose stored ISBN belongs to a book
about sculpture.
Both registries already have a `catena`: an HPE blockchain client on PyPI,
and a JavaScript OOP wrapper at 2.2.0 on npm. Neither is going anywhere, so
the distributions take a name that is free on both.
The Python import is unaffected — a distribution name and a module name are
independent there, so `pip install catena-corpus` still gives `import
catena`, and nothing in the examples or the bindings changes. On npm the
package name is the import specifier, so that one does change: `import
{ chunkMarkdown } from "catena-corpus"`.
The crates keep their own names. `catena` is free on crates.io.
Eight of them, each doing something a pipeline actually has to do, and each
starting from something anyone can fetch — a public domain book from Project
Gutenberg, a NASA broadcast, or a sample carried in the file.
Shell, because the tools are filters first: a book to passages and what its
notes cite; a recording to chapter marks, citable passages and the one
ffmpeg call that makes it play on a phone; passages into pgvector and a
search that asks twice and reconciles.
Rust, for the case where the caller already holds the text and a subprocess
would be serialising for nothing. Python, for the shape a pipeline stage
takes once the reading is a library call. TypeScript, for the app side —
chapter marks and deep links, and fusing two rankings in process.
Three of them end with what re-ranking means, because it is two different
things under one name: fusing ranked lists, which is what `rrf` does and
needs no model, and a cross-encoder scoring query and passage together,
which is accurate, slow, and runs over what retrieval already found.
Two things between a converted EPUB and its chapter titles.
The heading level taken as the chapter map was the shallowest present. Pandoc
writes the book's title as the only `#` and every real chapter as a `##`, so
the shallowest level had one heading in it and the whole book came back as a
single chapter. The dividing level is now the shallowest one used more than
once — a level used once is a title.
And an empty bracketed span survived the heading cleaner, because the pattern
wanted at least one character between the brackets. Pandoc writes an anchor
as exactly that: `## []{#link2H_INTR} Introduction`.
Gibbon's first volume, converted from the Project Gutenberg EPUB: 2 distinct
chapter titles before, 61 after, and none of them carrying an element id.
funkpower