From 7fa610c578e0098038f0612926385c30586aa838 Mon Sep 17 00:00:00 2001 From: funkpower Date: Sat, 19 Sep 2026 14:15:01 -0400 Subject: [PATCH] each crate carries its own readme, and the contract describes rather than instructs --- README.md | 79 ++++++++------- docs/bibclean.md => crates/bibclean/README.md | 6 +- .../booksniff/README.md | 8 +- .../catena-cli/README.md | 8 +- docs/chunkmd.md => crates/chunkmd/README.md | 12 +-- .../corpus-core/README.md | 18 ++-- docs/cuekit.md => crates/cuekit/README.md | 14 +-- docs/cuenorm.md => crates/cuenorm/README.md | 12 +-- docs/cuepack.md => crates/cuepack/README.md | 14 +-- docs/cueseg.md => crates/cueseg/README.md | 10 +- docs/emb.md => crates/emb/README.md | 6 +- docs/mdscrub.md => crates/mdscrub/README.md | 6 +- .../pdftriage/README.md | 10 +- docs/rrf.md => crates/rrf/README.md | 10 +- docs/textkit.md => crates/textkit/README.md | 8 +- docs/README.md | 79 --------------- docs/contract.md | 97 +++++++++++-------- docs/records.md | 18 ++-- 18 files changed, 180 insertions(+), 235 deletions(-) rename docs/bibclean.md => crates/bibclean/README.md (96%) rename docs/booksniff.md => crates/booksniff/README.md (92%) rename docs/catena-cli.md => crates/catena-cli/README.md (92%) rename docs/chunkmd.md => crates/chunkmd/README.md (91%) rename docs/corpus-core.md => crates/corpus-core/README.md (86%) rename docs/cuekit.md => crates/cuekit/README.md (85%) rename docs/cuenorm.md => crates/cuenorm/README.md (90%) rename docs/cuepack.md => crates/cuepack/README.md (86%) rename docs/cueseg.md => crates/cueseg/README.md (93%) rename docs/emb.md => crates/emb/README.md (94%) rename docs/mdscrub.md => crates/mdscrub/README.md (95%) rename docs/pdftriage.md => crates/pdftriage/README.md (91%) rename docs/rrf.md => crates/rrf/README.md (90%) rename docs/textkit.md => crates/textkit/README.md (94%) delete mode 100644 docs/README.md diff --git a/README.md b/README.md index 1af00b4..a3040a7 100644 --- a/README.md +++ b/README.md @@ -23,19 +23,31 @@ Each tool installs by name, so you can take one without taking the rest. | | | |---|---| -| [`booksniff`](docs/booksniff.md) | What a file actually is, whatever its name claims | -| [`pdftriage`](docs/pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed | -| [`bibclean`](docs/bibclean.md) | Catalogue records audited and repaired, with a reason for every change | -| [`mdscrub`](docs/mdscrub.md) | Converted markdown in, clean markdown out | -| [`chunkmd`](docs/chunkmd.md) | Markdown into passages that carry their page, chapter and role | -| [`cuenorm`](docs/cuenorm.md) | Any subtitle format in, one normalised cue stream out | -| [`cuepack`](docs/cuepack.md) | Cues into passages worth embedding, keeping the times they were said at | -| [`cueseg`](docs/cueseg.md) | A timed transcript into topic chapters | -| [`emb`](docs/emb.md) | Lines into vectors, against an embedding server | -| [`rrf`](docs/rrf.md) | Several ranked lists into one, without the database | +| [`booksniff`](crates/booksniff/README.md) | What a file actually is, whatever its name claims | +| [`pdftriage`](crates/pdftriage/README.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed | +| [`bibclean`](crates/bibclean/README.md) | Catalogue records audited and repaired, with a reason for every change | +| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out | +| [`chunkmd`](crates/chunkmd/README.md) | Markdown into passages that carry their page, chapter and role | +| [`cuenorm`](crates/cuenorm/README.md) | Any subtitle format in, one normalised cue stream out | +| [`cuepack`](crates/cuepack/README.md) | Cues into passages worth embedding, keeping the times they were said at | +| [`cueseg`](crates/cueseg/README.md) | A timed transcript into topic chapters | +| [`emb`](crates/emb/README.md) | Lines into vectors, against an embedding server | +| [`rrf`](crates/rrf/README.md) | Several ranked lists into one, without the database | -Full documentation, including the libraries underneath, is in -**[docs/](docs/README.md)**. +Each links to its own page: what it does, when you would use it, worked +examples, options and exit codes. + +## The libraries underneath + +Every tool is a library crate with a thin command on top, so anything a +command does can be done in process instead. + +| | | +|---|---| +| [`corpus-core`](crates/corpus-core/README.md) | The record types and the stream plumbing everything shares | +| [`textkit`](crates/textkit/README.md) | Token counting, sentence and paragraph splitting | +| [`cuekit`](crates/cuekit/README.md) | The cue model, and the three subtitle formats it is read from | +| [`catena-cli`](crates/catena-cli/README.md) | The parts of a filter every command does the same way | ## A worked pipeline @@ -67,32 +79,33 @@ garbled, needs OCR $ for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done ``` -## The contract +## What they promise -Every binary in this workspace obeys all of this, or it does not ship. The -whole of it is in [docs/contract.md](docs/contract.md). +Eight things hold for all of them, and together they are what lets any of +these go into a pipeline without first reading how it works: -1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two - tools. -2. **stdin to stdout.** Input on stdin or a path argument, result on stdout, - diagnostics on stderr only. -3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per - line, flushed as it is produced. -4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell; - failures use the `sysexits` range, so "no" and "broke" are never the same - number. -5. **No network, no database, no config file, no state**, with two declared - exceptions. -6. **No ambient knowledge.** No hostnames, no local paths, no private - vocabularies compiled in. -7. **Deterministic.** Same bytes in, same bytes out, across runs and machines. -8. **`--json` everywhere.** The human-readable default is for a person at a - terminal; `--json` is the contract. +- Each does a single job, with no mode switch that changes what it is. +- Input on stdin, results on stdout, diagnostics on stderr — never mixed. +- Anything list-shaped is NDJSON, one record per line, flushed as produced. +- Exit codes carry the answer: `0` yes, `1` no, `2` could not tell, and + failures in the `sysexits` range, so "no" and "broke" are never the same + number. +- Nothing reaches the network, a database, a config file or any stored state, + bar two exceptions that say so in `--help`. +- Nothing is assumed about where it is running: no hostnames, no paths, no + private vocabularies built in. +- The same input gives the same output, on every run and every machine. +- `--json` is the stable form, and does not change shape without a major + version. + +The whole of it, with the reasoning: [what these programs +promise](docs/contract.md). ## The records -Three shapes travel between the tools, versioned, defined in `corpus-core` and -described in [docs/records.md](docs/records.md). +Five shapes travel between the tools, versioned, defined in +[`corpus-core`](crates/corpus-core/README.md) and described in [the +records](docs/records.md). ``` $ cargo run -p corpus-core -- schema all diff --git a/docs/bibclean.md b/crates/bibclean/README.md similarity index 96% rename from docs/bibclean.md rename to crates/bibclean/README.md index 0d99705..e41c6e4 100644 --- a/docs/bibclean.md +++ b/crates/bibclean/README.md @@ -2,7 +2,7 @@ Catalogue records audited and repaired, with a reason for every change. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -150,5 +150,5 @@ description to another. ## See also -- [`booksniff`](booksniff.md) — what the file behind the record actually is. -- [The records](records.md) — the shapes the rest of the suite uses. +- [`booksniff`](../booksniff/README.md) — what the file behind the record actually is. +- [The records](../../docs/records.md) — the shapes the rest of the suite uses. diff --git a/docs/booksniff.md b/crates/booksniff/README.md similarity index 92% rename from docs/booksniff.md rename to crates/booksniff/README.md index 27e458e..845665f 100644 --- a/docs/booksniff.md +++ b/crates/booksniff/README.md @@ -2,7 +2,7 @@ What a file actually is, whatever its name claims. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -11,7 +11,7 @@ and says what format it really is. It never reads the whole file, so a directory of nine-hundred-megabyte scans costs the same as a directory of anything else. -Writes one [`doc` record](records.md#doc--what-a-file-is) per file, with a +Writes one [`doc` record](../../docs/records.md#doc--what-a-file-is) per file, with a note for anything worth remarking on: a name that disagrees with the bytes, a file too small to be a book, a name that says "sample". @@ -113,6 +113,6 @@ the bytes need not write them to disk first. ## See also -- [`pdftriage`](pdftriage.md) — once you know it is a PDF, whether its text is +- [`pdftriage`](../pdftriage/README.md) — once you know it is a PDF, whether its text is worth reading. -- [The records](records.md) — the shape this writes. +- [The records](../../docs/records.md) — the shape this writes. diff --git a/docs/catena-cli.md b/crates/catena-cli/README.md similarity index 92% rename from docs/catena-cli.md rename to crates/catena-cli/README.md index 9c0f4f3..7fe625d 100644 --- a/docs/catena-cli.md +++ b/crates/catena-cli/README.md @@ -2,7 +2,7 @@ The parts of a filter every command does the same way. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it is @@ -16,7 +16,7 @@ the same at the edges. ## When you would use it - **Writing a filter that joins this pipeline**, and wanting it to behave like - the others without rereading [the contract](contract.md). + the others without rereading [the contract](../../docs/contract.md). - **Anywhere a small Unix filter is wanted**, in or out of this workspace. ## Parsing a command line @@ -117,5 +117,5 @@ code, and should stay under a hundred and fifty lines. ## See also -- [The contract](contract.md) — what these functions are enforcing. -- [`corpus-core`](corpus-core.md) — the records and the exit-code constants. +- [The contract](../../docs/contract.md) — what these functions are enforcing. +- [`corpus-core`](../corpus-core/README.md) — the records and the exit-code constants. diff --git a/docs/chunkmd.md b/crates/chunkmd/README.md similarity index 91% rename from docs/chunkmd.md rename to crates/chunkmd/README.md index a8185fd..1949c40 100644 --- a/docs/chunkmd.md +++ b/crates/chunkmd/README.md @@ -2,7 +2,7 @@ Markdown into passages that carry their page, chapter and role. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -10,7 +10,7 @@ Cuts a document into pieces of about the size an embedding model wants, and puts on each one the page it came from, the chapter it is in, and what part of the work it belongs to. -Writes [`chunk` records](records.md#chunk--a-retrievable-passage), one per +Writes [`chunk` records](../../docs/records.md#chunk--a-retrievable-passage), one per line. ## When you would use it @@ -125,7 +125,7 @@ for chunk in chunkmd::chunk(&markdown, &Budgets::default()) { ## See also -- [`mdscrub`](mdscrub.md) — what to run before this. -- [`cuepack`](cuepack.md) — the same record, from a recording. -- [`emb`](emb.md) — turning the text into vectors. -- [The records](records.md) — the shape this writes. +- [`mdscrub`](../mdscrub/README.md) — what to run before this. +- [`cuepack`](../cuepack/README.md) — the same record, from a recording. +- [`emb`](../emb/README.md) — turning the text into vectors. +- [The records](../../docs/records.md) — the shape this writes. diff --git a/docs/corpus-core.md b/crates/corpus-core/README.md similarity index 86% rename from docs/corpus-core.md rename to crates/corpus-core/README.md index 3e403ac..f3444ec 100644 --- a/docs/corpus-core.md +++ b/crates/corpus-core/README.md @@ -2,12 +2,12 @@ The record types and the stream plumbing everything shares. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it is A library, not a command — though it carries a small one for printing -schemas. It holds the three [record types](records.md), the NDJSON reader and +schemas. It holds the three [record types](../../docs/records.md), the NDJSON reader and writer, and the exit-code conventions. Everything else in the workspace depends on it, which is what makes the tools @@ -23,9 +23,9 @@ else. ## The records -[`Doc`](records.md#doc--what-a-file-is), -[`Chunk`](records.md#chunk--a-retrievable-passage) and -[`Hit`](records.md#hit--one-scored-candidate). Each derives `Serialize`, +[`Doc`](../../docs/records.md#doc--what-a-file-is), +[`Chunk`](../../docs/records.md#chunk--a-retrievable-passage) and +[`Hit`](../../docs/records.md#hit--one-scored-candidate). Each derives `Serialize`, `Deserialize` and `JsonSchema`, and implements `Record`, which is how a reader knows what version it is looking at. @@ -122,7 +122,7 @@ let answer = Verdict::from(needs_ocr); std::process::ExitCode::from(answer) ``` -See [the contract](contract.md#predicates-answer-with-exit-codes) for the +See [the contract](../../docs/contract.md#exit-codes-carry-the-answer) for the whole table. ## The schemas @@ -137,6 +137,6 @@ it can be committed and diffed. ## See also -- [The records](records.md) — the shapes, described. -- [`catena-cli`](catena-cli.md) — the rest of what a filter needs. -- [`textkit`](textkit.md) — counting and splitting text. +- [The records](../../docs/records.md) — the shapes, described. +- [`catena-cli`](../catena-cli/README.md) — the rest of what a filter needs. +- [`textkit`](../textkit/README.md) — counting and splitting text. diff --git a/docs/cuekit.md b/crates/cuekit/README.md similarity index 85% rename from docs/cuekit.md rename to crates/cuekit/README.md index f27b66e..3a6b186 100644 --- a/docs/cuekit.md +++ b/crates/cuekit/README.md @@ -2,16 +2,16 @@ The cue model, and the three subtitle formats it is read from. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it is A library. Three formats go in — SubRip, WebVTT and SubStation Alpha — and -one [`cue`](records.md#cue--one-line-of-a-transcript) stream comes out. +one [`cue`](../../docs/records.md#cue--one-line-of-a-transcript) stream comes out. Everything downstream works on cues and never has to know which of the three a recording arrived with. -[`cuenorm`](cuenorm.md) is a command over this. +[`cuenorm`](../cuenorm/README.md) is a command over this. ## When you would use it @@ -49,7 +49,7 @@ let cue = Cue::new(12.5, 15.0, "the mill stood at the northern end"); assert_eq!(cue.seconds(), 2.5); ``` -Deliberately smaller than a [chunk](records.md#chunk--a-retrievable-passage). +Deliberately smaller than a [chunk](../../docs/records.md#chunk--a-retrievable-passage). A cue is what a subtitle file holds — a few seconds, often half a sentence — and turning cues into passages worth retrieving is a later stage's job. @@ -99,6 +99,6 @@ makes a malformed file drop the same cues here as there. ## See also -- [`cuenorm`](cuenorm.md) — the command over this. -- [`cuepack`](cuepack.md) and [`cueseg`](cueseg.md) — what reads cues. -- [The records](records.md) — the cue shape on the wire. +- [`cuenorm`](../cuenorm/README.md) — the command over this. +- [`cuepack`](../cuepack/README.md) and [`cueseg`](../cueseg/README.md) — what reads cues. +- [The records](../../docs/records.md) — the cue shape on the wire. diff --git a/docs/cuenorm.md b/crates/cuenorm/README.md similarity index 90% rename from docs/cuenorm.md rename to crates/cuenorm/README.md index d23f852..eddd8ab 100644 --- a/docs/cuenorm.md +++ b/crates/cuenorm/README.md @@ -2,12 +2,12 @@ Any subtitle format in, one normalised cue stream out. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does Reads a subtitle file — SubRip, WebVTT or SubStation Alpha — and writes one -[`cue` record](records.md#cue--one-line-of-a-transcript) per line: a start, an +[`cue` record](../../docs/records.md#cue--one-line-of-a-transcript) per line: a start, an end, and what was said. The format is worked out from the content, not from the name, because a @@ -123,10 +123,10 @@ for cue in &cues { } ``` -See [`cuekit`](cuekit.md) for the parsers one at a time. +See [`cuekit`](../cuekit/README.md) for the parsers one at a time. ## See also -- [`cuepack`](cuepack.md) — cues into passages worth embedding. -- [`cueseg`](cueseg.md) — cues into topic chapters. -- [`cuekit`](cuekit.md) — the library this is a command over. +- [`cuepack`](../cuepack/README.md) — cues into passages worth embedding. +- [`cueseg`](../cueseg/README.md) — cues into topic chapters. +- [`cuekit`](../cuekit/README.md) — the library this is a command over. diff --git a/docs/cuepack.md b/crates/cuepack/README.md similarity index 86% rename from docs/cuepack.md rename to crates/cuepack/README.md index dc5391e..83afd74 100644 --- a/docs/cuepack.md +++ b/crates/cuepack/README.md @@ -2,13 +2,13 @@ Cues into passages worth embedding, keeping the times they were said at. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does Packs consecutive cues into passages of about a minute of continuous speech, and writes each one as a [`chunk` -record](records.md#chunk--a-retrievable-passage) carrying the start of its +record](../../docs/records.md#chunk--a-retrievable-passage) carrying the start of its first cue and the end of its last. ## When you would use it @@ -20,7 +20,7 @@ first cue and the end of its last. first cue's start and the last cue's end, a hit can be cited to the second and a player can jump to it. - **To index recordings and books together.** The record is the same one - [`chunkmd`](chunkmd.md) writes, so one index answers across both and returns + [`chunkmd`](../chunkmd/README.md) writes, so one index answers across both and returns each with a location: a page number for one, a timestamp for the other. ## Install @@ -84,7 +84,7 @@ with its own echo embeds as something nobody said. ## Input -Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file. A +Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file. A cue stream begins with a JSON object and a subtitle file does not, which is enough to tell them apart. @@ -100,6 +100,6 @@ for passage in cuepack::pack(&cues, &Budgets::default()) { ## See also -- [`cuenorm`](cuenorm.md) — what to run before this. -- [`cueseg`](cueseg.md) — the other thing to do with a cue stream. -- [`chunkmd`](chunkmd.md) — the same record, from a book. +- [`cuenorm`](../cuenorm/README.md) — what to run before this. +- [`cueseg`](../cueseg/README.md) — the other thing to do with a cue stream. +- [`chunkmd`](../chunkmd/README.md) — the same record, from a book. diff --git a/docs/cueseg.md b/crates/cueseg/README.md similarity index 93% rename from docs/cueseg.md rename to crates/cueseg/README.md index f087bd1..fa1d4c5 100644 --- a/docs/cueseg.md +++ b/crates/cueseg/README.md @@ -2,12 +2,12 @@ A timed transcript into topic chapters. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does Finds where a recording changes subject, and writes one [`segment` -record](records.md#segment--one-topic-chapter) per chapter: a start, an end, a +record](../../docs/records.md#segment--one-topic-chapter) per chapter: a start, an end, a title taken from the chapter's own first sentence, and an excerpt. No model and no network. The text is hashed into a small lexical vector a @@ -103,7 +103,7 @@ than the end of the file is a mark that lands in the wrong place. ## Input -Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file. +Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file. ## As a library @@ -115,5 +115,5 @@ for chapter in cueseg::segment(&cues, 7245.0) { ## See also -- [`cuenorm`](cuenorm.md) — what to run before this. -- [`cuepack`](cuepack.md) — the other thing to do with a cue stream. +- [`cuenorm`](../cuenorm/README.md) — what to run before this. +- [`cuepack`](../cuepack/README.md) — the other thing to do with a cue stream. diff --git a/docs/emb.md b/crates/emb/README.md similarity index 94% rename from docs/emb.md rename to crates/emb/README.md index bf85d78..d943730 100644 --- a/docs/emb.md +++ b/crates/emb/README.md @@ -2,7 +2,7 @@ Lines into vectors, against an embedding server. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -124,6 +124,6 @@ let vectors = emb::embed_batch(&agent, &endpoint, &texts)?; ## See also -- [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md) — where the text comes +- [`chunkmd`](../chunkmd/README.md) and [`cuepack`](../cuepack/README.md) — where the text comes from. -- [`rrf`](rrf.md) — what to do with the results of searching those vectors. +- [`rrf`](../rrf/README.md) — what to do with the results of searching those vectors. diff --git a/docs/mdscrub.md b/crates/mdscrub/README.md similarity index 95% rename from docs/mdscrub.md rename to crates/mdscrub/README.md index b4389b9..23c184a 100644 --- a/docs/mdscrub.md +++ b/crates/mdscrub/README.md @@ -2,7 +2,7 @@ Converted markdown in, clean markdown out. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -140,5 +140,5 @@ wants only one of them. ## See also -- [`chunkmd`](chunkmd.md) — what to do with the markdown once it is clean. -- [`pdftriage`](pdftriage.md) — whether the text was worth extracting at all. +- [`chunkmd`](../chunkmd/README.md) — what to do with the markdown once it is clean. +- [`pdftriage`](../pdftriage/README.md) — whether the text was worth extracting at all. diff --git a/docs/pdftriage.md b/crates/pdftriage/README.md similarity index 91% rename from docs/pdftriage.md rename to crates/pdftriage/README.md index b2b9b15..13d6337 100644 --- a/docs/pdftriage.md +++ b/crates/pdftriage/README.md @@ -2,7 +2,7 @@ Whether a PDF's text layer is worth reading, or whether it has to be OCRed. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does @@ -79,7 +79,7 @@ $ pdftriage --json book.pdf | Option | Effect | |---|---| -| `--json` | Emit a [`doc` record](records.md#doc--what-a-file-is) instead of a readable verdict | +| `--json` | Emit a [`doc` record](../../docs/records.md#doc--what-a-file-is) instead of a readable verdict | | `-q`, `--quiet` | Write no verdict; answer with the exit code alone. Diagnostics still go to stderr, because a silent failure is worse than a quiet one | ## Exit codes @@ -131,6 +131,6 @@ Running `pdftotext` is the command's job, not the library's. ## See also -- [`booksniff`](booksniff.md) — whether it is a PDF at all. -- [`mdscrub`](mdscrub.md) — cleaning up the text once it has been extracted. -- [The records](records.md) — the shape `--json` writes. +- [`booksniff`](../booksniff/README.md) — whether it is a PDF at all. +- [`mdscrub`](../mdscrub/README.md) — cleaning up the text once it has been extracted. +- [The records](../../docs/records.md) — the shape `--json` writes. diff --git a/docs/rrf.md b/crates/rrf/README.md similarity index 90% rename from docs/rrf.md rename to crates/rrf/README.md index 9ee37aa..35b7b8b 100644 --- a/docs/rrf.md +++ b/crates/rrf/README.md @@ -2,12 +2,12 @@ Several ranked lists into one, without the database. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it does Reciprocal rank fusion. Reads [`hit` -records](records.md#hit--one-scored-candidate) from any number of retrieval +records](../../docs/records.md#hit--one-scored-candidate) from any number of retrieval channels and writes one ranked list, best first. ## When you would use it @@ -88,7 +88,7 @@ it. ## Input and output -One [`hit`](records.md#hit--one-scored-candidate) per line, in any order and +One [`hit`](../../docs/records.md#hit--one-scored-candidate) per line, in any order and interleaved if you like — each carries the rank its own channel gave it, so the stream does not have to be grouped. @@ -110,5 +110,5 @@ let fused = rrf::fuse(&hits, &rrf::Weights::default()); ## See also -- [`emb`](emb.md) — producing the vectors one of the channels searches. -- [The records](records.md) — the shape this reads and writes. +- [`emb`](../emb/README.md) — producing the vectors one of the channels searches. +- [The records](../../docs/records.md) — the shape this reads and writes. diff --git a/docs/textkit.md b/crates/textkit/README.md similarity index 94% rename from docs/textkit.md rename to crates/textkit/README.md index ca2bf77..49dcb6d 100644 --- a/docs/textkit.md +++ b/crates/textkit/README.md @@ -2,7 +2,7 @@ Token counting, sentence and paragraph splitting. -Back to [the index](README.md). +Part of [catena](../../README.md). ## What it is @@ -14,7 +14,7 @@ and cutting text into sentences and paragraphs. - **Deciding how much text fits** in a model's context, before sending it. - **Writing your own chunker** that has to agree with - [`chunkmd`](chunkmd.md) about where a boundary is. + [`chunkmd`](../chunkmd/README.md) about where a boundary is. - **Porting anything from Python** where the exact behaviour of `\s`, `\w` or `\b` matters more than it looks like it should. @@ -117,5 +117,5 @@ that has them. ## See also -- [`corpus-core`](corpus-core.md) — the records these help produce. -- [`chunkmd`](chunkmd.md) — the chunker built on this. +- [`corpus-core`](../corpus-core/README.md) — the records these help produce. +- [`chunkmd`](../chunkmd/README.md) — the chunker built on this. diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index e470620..0000000 --- a/docs/README.md +++ /dev/null @@ -1,79 +0,0 @@ -# catena - -Small programs that each do one part of turning a pile of files into a -searchable corpus, and hand the next one text on stdout. - -Every one of them is a filter: bytes in, records out, diagnostics on stderr, -and an exit code that means something. None of them needs a database, a GPU, a -queue or a config file. - -## The tools - -### Working out what you have - -| | | -|---|---| -| [`booksniff`](booksniff.md) | What a file actually is, whatever its name claims | -| [`pdftriage`](pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed | -| [`bibclean`](bibclean.md) | Catalogue records audited and repaired, with a reason for every change | - -### Turning documents into passages - -| | | -|---|---| -| [`mdscrub`](mdscrub.md) | Converted markdown in, clean markdown out | -| [`chunkmd`](chunkmd.md) | Markdown into passages that carry their page, chapter and role | - -### Turning recordings into passages - -| | | -|---|---| -| [`cuenorm`](cuenorm.md) | Any subtitle format in, one normalised cue stream out | -| [`cuepack`](cuepack.md) | Cues into passages worth embedding, keeping the times they were said at | -| [`cueseg`](cueseg.md) | A timed transcript into topic chapters | - -### Searching - -| | | -|---|---| -| [`emb`](emb.md) | Lines into vectors, against an embedding server | -| [`rrf`](rrf.md) | Several ranked lists into one, without the database | - -## The libraries underneath - -Every tool is a library crate with a thin command on top, so anything a -command does can be done in process instead. - -| | | -|---|---| -| [`corpus-core`](corpus-core.md) | The record types and the stream plumbing everything shares | -| [`textkit`](textkit.md) | Token counting, sentence and paragraph splitting | -| [`cuekit`](cuekit.md) | The cue model, and the three subtitle formats it is read from | -| [`catena-cli`](catena-cli.md) | The parts of a filter every command does the same way | - -## Reference - -- [The records](records.md) — the shapes that travel between the tools. -- [The contract](contract.md) — what every one of these programs promises. - -## A worked pipeline - -A book, from a converted chapter to passages an index can hold: - -``` -$ mdscrub chapter.md | chunkmd --min-tokens 20 -{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]} -{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]} -``` - -A recording, through the same shape: - -``` -$ cuenorm episode.vtt | cuepack --seconds 10 -{"v":1,"idx":0,"text":"The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest.","tokens":27,"page":null,"chapter":null,"role":"body","start_time":1.0,"end_time":8.0,"flags":[]} -{"v":1,"idx":1,"text":"Its wheel was undershot, which was wasteful,","tokens":11,"page":null,"chapter":null,"role":"body","start_time":11.0,"end_time":19.0,"flags":[]} -``` - -Both produce the same record, so one index answers across a book and a lecture -and returns both with a location you can cite — a page number for one, a -timestamp for the other. diff --git a/docs/contract.md b/docs/contract.md index aaf36ca..2500fdd 100644 --- a/docs/contract.md +++ b/docs/contract.md @@ -1,33 +1,41 @@ -# The contract +# What these programs promise -Every program here obeys all of this, or it does not ship. It is worth reading -once, because it is what lets these be used together without reading any of -them. +Eight things hold for every program here. Together they are what lets a person +put any of them in a pipeline without first reading how it works. -Back to [the index](README.md). +Part of [catena](../README.md). -## One job +## Each one does a single job -A tool that needs a `--mode` flag to change what it *is* is two tools. +No program here has a mode switch that changes what it is. Where two jobs +would otherwise share a name, there are two programs — which is why +[`cuenorm`](../crates/cuenorm/README.md), [`cuepack`](../crates/cuepack/README.md) and [`cueseg`](../crates/cueseg/README.md) are +separate commands rather than one command with a flag. -## stdin to stdout +## Input on stdin, results on stdout, diagnostics on stderr -Input on stdin or a path argument, result on stdout, diagnostics on stderr -only — never content and chatter on the same stream. A tool that takes a path -instead of a stream says why in its `--help`; [`pdftriage`](pdftriage.md) does, -because it reads two specific page ranges rather than the whole file. +Output and commentary never share a stream, so a result can always be piped +somewhere without filtering anything out of it. -## Streams, not documents +Almost all of them read stdin or a path. Where one takes a path only, it says +so in `--help` and says why: [`pdftriage`](../crates/pdftriage/README.md) reads two specific +page ranges rather than the whole file, which a stream cannot offer. -Anything list-shaped is NDJSON: one [record](records.md) per line, flushed as -it is produced, so a slow stage never blocks the pipe behind it. +## Anything list-shaped is NDJSON -The reader is strict on purpose. It refuses a record whose version it does not -know, it stops at the first bad line rather than reading past a disagreement -about the format, and it bounds how much one line may cost — a filter pointed -at a binary file should fail, not allocate the file. +One [record](records.md) per line, flushed as it is produced — so a slow stage +downstream sees each record as it appears rather than when the producer +finishes. -## Predicates answer with exit codes +Readers are strict, and each strictness prevents a failure that would +otherwise be silent. A record from a version the reader does not know is +refused rather than parsed into whatever lines up. A bad line ends the stream +instead of being skipped, because it means the two ends disagree about the +format. Lines are bounded, so a program pointed at a binary file fails instead +of allocating it. A byte-order mark and blank lines are ignored, so a stream +that was concatenated or written on Windows still reads. + +## Exit codes carry the answer | Code | Meaning | |---|---| @@ -35,44 +43,47 @@ at a binary file should fail, not allocate the file. | `1` | No | | `2` | Could not tell | | `64` | The command line was wrong | -| `65` | The input was not what the tool can work on | +| `65` | The input was not what the program can work on | | `69` | A program it needs is not installed | -| `70` | The tool's own fault | +| `70` | Its own fault | | `74` | Reading or writing failed | -The failure codes start where `sysexits.h` puts them so that "no" and "broke" -are never the same number. That is what makes this work: +Failures start where `sysexits.h` puts them, so "no" and "broke" are never the +same number. That is what makes this work: ```sh for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done ``` -A consumer closing the pipe is not a failure. `… | head -1` exits `0`, like -every other filter. +A consumer closing the pipe is not a failure. `… | head -1` exits `0`, the +same as any other filter. -## No network, no database, no config file, no state +## Nothing reaches outside the process +No network, no database, no configuration file, no state kept between runs. Two exceptions, both declared in `--help`: -- [`emb`](emb.md), whose whole job is an HTTP call. -- [`pdftriage`](pdftriage.md), which shells out to `pdftotext` and `pdfinfo` - and exits `69` when either is missing. +- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does. +- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`, and exits `69` + when either is missing rather than guessing. -## No ambient knowledge +## Nothing is assumed about where it is running -No hostnames, no local paths, no private vocabularies compiled in. General -scheme data — ISBN check digits, the HTML character references — may be; -anything local is a `--rules` file. +No hostnames, no paths, no private vocabularies are built in. General +reference data is — ISBN check digits, the HTML character references, the +names of well-known publishers — and anything specific to one collection is +supplied in a file rather than compiled in. -## Deterministic +## The same input gives the same output -Same bytes in, same bytes out, across runs and machines. No hash-order -dependence, no clock, no locale reads. Floats that JSON cannot carry are -refused where they are produced rather than written as `null` and lost. +On every run and on every machine. No dependence on hash ordering, no reading +of the clock or the locale. Numbers that JSON cannot represent are refused +where they are produced rather than written as `null` and lost somewhere +downstream. -## `--json` everywhere +## `--json` is the stable form -The human-readable default is for a person at a terminal; `--json` is the -contract, and it does not change shape without a major version. Tools whose -only sensible output is a stream of records write that stream and have no -`--json` to add. +Where a program prints something for a person to read, `--json` prints the +same answer as a record, and that record does not change shape without a major +version. Programs whose only sensible output is a stream of records write that +stream and have no `--json` to add. diff --git a/docs/records.md b/docs/records.md index 3135df3..a124020 100644 --- a/docs/records.md +++ b/docs/records.md @@ -4,10 +4,10 @@ Five shapes travel between these tools. Each carries a version field `v`, so a reader can refuse a stream it does not understand rather than parsing it into whatever happens to line up. -Every list-shaped stream is [NDJSON](contract.md#streams-not-documents): one +Every list-shaped stream is [NDJSON](contract.md#anything-list-shaped-is-ndjson): one record per line, flushed as it is produced. -Back to [the index](README.md). +Part of [catena](../README.md). ## `doc` — what a file is @@ -31,7 +31,7 @@ Back to [the index](README.md). A tool fills what it can and leaves the rest null: a format sniffer fills `kind` and leaves `text_layer` empty, a triage tool does the reverse. -Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md). +Written by [`booksniff`](../crates/booksniff/README.md) and [`pdftriage`](../crates/pdftriage/README.md). ## `chunk` — a retrievable passage @@ -55,7 +55,7 @@ Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md). recordings. One shape either way, which is what lets a single index serve a book and a lecture and cite both. -Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md). +Written by [`chunkmd`](../crates/chunkmd/README.md) and [`cuepack`](../crates/cuepack/README.md). ## `hit` — one scored candidate @@ -74,7 +74,7 @@ Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md). `rank` is what fusion consumes. `score` is carried along because it is useful to a person reading a trace and useless for comparing across channels. -Read and written by [`rrf`](rrf.md). +Read and written by [`rrf`](../crates/rrf/README.md). ## `cue` — one line of a transcript @@ -85,8 +85,8 @@ Read and written by [`rrf`](rrf.md). Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few seconds and often half a sentence. -Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and -[`cueseg`](cueseg.md). +Written by [`cuenorm`](../crates/cuenorm/README.md), read by [`cuepack`](../crates/cuepack/README.md) and +[`cueseg`](../crates/cueseg/README.md). ## `segment` — one topic chapter @@ -98,7 +98,7 @@ Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and The marks a scrubber needs, and the spans a summariser should be given one at a time. -Written by [`cueseg`](cueseg.md). +Written by [`cueseg`](../crates/cueseg/README.md). ## Getting the schemas @@ -113,4 +113,4 @@ $ cargo run -p corpus-core -- schema chunk } ``` -See [`corpus-core`](corpus-core.md). +See [`corpus-core`](../crates/corpus-core/README.md).