Every one of them is a filter: bytes in, records out, diagnostics on stderr, and an exit code that means something. None of them needs a database, a GPU, a queue or a config file.

The tools

Working out what you have


`booksniff`	What a file actually is, whatever its name claims
`pdftriage`	Whether a PDF's text layer is worth reading, or whether it has to be OCRed
`bibclean`	Catalogue records audited and repaired, with a reason for every change

Turning documents into passages


`mdscrub`	Converted markdown in, clean markdown out
`chunkmd`	Markdown into passages that carry their page, chapter and role

Turning recordings into passages


`cuenorm`	Any subtitle format in, one normalised cue stream out
`cuepack`	Cues into passages worth embedding, keeping the times they were said at
`cueseg`	A timed transcript into topic chapters

Searching


`emb`	Lines into vectors, against an embedding server
`rrf`	Several ranked lists into one, without the database

The libraries underneath

Every tool is a library crate with a thin command on top, so anything a command does can be done in process instead.


`corpus-core`	The record types and the stream plumbing everything shares
`textkit`	Token counting, sentence and paragraph splitting
`cuekit`	The cue model, and the three subtitle formats it is read from
`catena-cli`	The parts of a filter every command does the same way

Reference

The records — the shapes that travel between the tools.
The contract — what every one of these programs promises.

A worked pipeline

A book, from a converted chapter to passages an index can hold:

$ mdscrub chapter.md | chunkmd --min-tokens 20
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}

A recording, through the same shape:

$ cuenorm episode.vtt | cuepack --seconds 10
{"v":1,"idx":0,"text":"The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest.","tokens":27,"page":null,"chapter":null,"role":"body","start_time":1.0,"end_time":8.0,"flags":[]}
{"v":1,"idx":1,"text":"Its wheel was undershot, which was wasteful,","tokens":11,"page":null,"chapter":null,"role":"body","start_time":11.0,"end_time":19.0,"flags":[]}

Both produce the same record, so one index answers across a book and a lecture and returns both with a location you can cite — a page number for one, a timestamp for the other.