each crate carries its own readme, and the contract describes rather than instructs

README.md

@@ -23,19 +23,31 @@ Each tool installs by name, so you can take one without taking the rest.
 
 | | |
 |---|---|
-| [`booksniff`](docs/booksniff.md) | What a file actually is, whatever its name claims |
+| [`booksniff`](crates/booksniff/README.md) | What a file actually is, whatever its name claims |
-| [`pdftriage`](docs/pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
+| [`pdftriage`](crates/pdftriage/README.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
-| [`bibclean`](docs/bibclean.md) | Catalogue records audited and repaired, with a reason for every change |
+| [`bibclean`](crates/bibclean/README.md) | Catalogue records audited and repaired, with a reason for every change |
-| [`mdscrub`](docs/mdscrub.md) | Converted markdown in, clean markdown out |
+| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
-| [`chunkmd`](docs/chunkmd.md) | Markdown into passages that carry their page, chapter and role |
+| [`chunkmd`](crates/chunkmd/README.md) | Markdown into passages that carry their page, chapter and role |
-| [`cuenorm`](docs/cuenorm.md) | Any subtitle format in, one normalised cue stream out |
+| [`cuenorm`](crates/cuenorm/README.md) | Any subtitle format in, one normalised cue stream out |
-| [`cuepack`](docs/cuepack.md) | Cues into passages worth embedding, keeping the times they were said at |
+| [`cuepack`](crates/cuepack/README.md) | Cues into passages worth embedding, keeping the times they were said at |
-| [`cueseg`](docs/cueseg.md) | A timed transcript into topic chapters |
+| [`cueseg`](crates/cueseg/README.md) | A timed transcript into topic chapters |
-| [`emb`](docs/emb.md) | Lines into vectors, against an embedding server |
+| [`emb`](crates/emb/README.md) | Lines into vectors, against an embedding server |
-| [`rrf`](docs/rrf.md) | Several ranked lists into one, without the database |
+| [`rrf`](crates/rrf/README.md) | Several ranked lists into one, without the database |
 
-Full documentation, including the libraries underneath, is in
+Each links to its own page: what it does, when you would use it, worked
-**[docs/](docs/README.md)**.
+examples, options and exit codes.
+
+## The libraries underneath
+
+Every tool is a library crate with a thin command on top, so anything a
+command does can be done in process instead.
+
+| | |
+|---|---|
+| [`corpus-core`](crates/corpus-core/README.md) | The record types and the stream plumbing everything shares |
+| [`textkit`](crates/textkit/README.md) | Token counting, sentence and paragraph splitting |
+| [`cuekit`](crates/cuekit/README.md) | The cue model, and the three subtitle formats it is read from |
+| [`catena-cli`](crates/catena-cli/README.md) | The parts of a filter every command does the same way |
 
 ## A worked pipeline
 
@@ -67,32 +79,33 @@ garbled, needs OCR
 $ for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
 ```
 
-## The contract
+## What they promise
 
-Every binary in this workspace obeys all of this, or it does not ship. The
+Eight things hold for all of them, and together they are what lets any of
-whole of it is in [docs/contract.md](docs/contract.md).
+these go into a pipeline without first reading how it works:
 
-1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
+- Each does a single job, with no mode switch that changes what it is.
-   tools.
+- Input on stdin, results on stdout, diagnostics on stderr — never mixed.
-2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
+- Anything list-shaped is NDJSON, one record per line, flushed as produced.
-   diagnostics on stderr only.
+- Exit codes carry the answer: `0` yes, `1` no, `2` could not tell, and
-3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
+  failures in the `sysexits` range, so "no" and "broke" are never the same
-   line, flushed as it is produced.
+  number.
-4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell;
+- Nothing reaches the network, a database, a config file or any stored state,
-   failures use the `sysexits` range, so "no" and "broke" are never the same
+  bar two exceptions that say so in `--help`.
-   number.
+- Nothing is assumed about where it is running: no hostnames, no paths, no
-5. **No network, no database, no config file, no state**, with two declared
+  private vocabularies built in.
-   exceptions.
+- The same input gives the same output, on every run and every machine.
-6. **No ambient knowledge.** No hostnames, no local paths, no private
+- `--json` is the stable form, and does not change shape without a major
-   vocabularies compiled in.
+  version.
-7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
+
-8. **`--json` everywhere.** The human-readable default is for a person at a
+The whole of it, with the reasoning: [what these programs
-   terminal; `--json` is the contract.
+promise](docs/contract.md).
 
 ## The records
 
-Three shapes travel between the tools, versioned, defined in `corpus-core` and
+Five shapes travel between the tools, versioned, defined in
-described in [docs/records.md](docs/records.md).
+[`corpus-core`](crates/corpus-core/README.md) and described in [the
+records](docs/records.md).
 
 ```
 $ cargo run -p corpus-core -- schema all

crates/bibclean/README.md

@@ -2,7 +2,7 @@
 
 Catalogue records audited and repaired, with a reason for every change.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -150,5 +150,5 @@ description to another.
 
 ## See also
 
-- [`booksniff`](booksniff.md) — what the file behind the record actually is.
+- [`booksniff`](../booksniff/README.md) — what the file behind the record actually is.
-- [The records](records.md) — the shapes the rest of the suite uses.
+- [The records](../../docs/records.md) — the shapes the rest of the suite uses.

crates/booksniff/README.md

@@ -2,7 +2,7 @@
 
 What a file actually is, whatever its name claims.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -11,7 +11,7 @@ and says what format it really is. It never reads the whole file, so a
 directory of nine-hundred-megabyte scans costs the same as a directory of
 anything else.
 
-Writes one [`doc` record](records.md#doc--what-a-file-is) per file, with a
+Writes one [`doc` record](../../docs/records.md#doc--what-a-file-is) per file, with a
 note for anything worth remarking on: a name that disagrees with the bytes, a
 file too small to be a book, a name that says "sample".
 
@@ -113,6 +113,6 @@ the bytes need not write them to disk first.
 
 ## See also
 
-- [`pdftriage`](pdftriage.md) — once you know it is a PDF, whether its text is
+- [`pdftriage`](../pdftriage/README.md) — once you know it is a PDF, whether its text is
   worth reading.
-- [The records](records.md) — the shape this writes.
+- [The records](../../docs/records.md) — the shape this writes.

crates/catena-cli/README.md

@@ -2,7 +2,7 @@
 
 The parts of a filter every command does the same way.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it is
 
@@ -16,7 +16,7 @@ the same at the edges.
 ## When you would use it
 
 - **Writing a filter that joins this pipeline**, and wanting it to behave like
-  the others without rereading [the contract](contract.md).
+  the others without rereading [the contract](../../docs/contract.md).
 - **Anywhere a small Unix filter is wanted**, in or out of this workspace.
 
 ## Parsing a command line
@@ -117,5 +117,5 @@ code, and should stay under a hundred and fifty lines.
 
 ## See also
 
-- [The contract](contract.md) — what these functions are enforcing.
+- [The contract](../../docs/contract.md) — what these functions are enforcing.
-- [`corpus-core`](corpus-core.md) — the records and the exit-code constants.
+- [`corpus-core`](../corpus-core/README.md) — the records and the exit-code constants.

crates/chunkmd/README.md

@@ -2,7 +2,7 @@
 
 Markdown into passages that carry their page, chapter and role.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -10,7 +10,7 @@ Cuts a document into pieces of about the size an embedding model wants, and
 puts on each one the page it came from, the chapter it is in, and what part of
 the work it belongs to.
 
-Writes [`chunk` records](records.md#chunk--a-retrievable-passage), one per
+Writes [`chunk` records](../../docs/records.md#chunk--a-retrievable-passage), one per
 line.
 
 ## When you would use it
@@ -125,7 +125,7 @@ for chunk in chunkmd::chunk(&markdown, &Budgets::default()) {
 
 ## See also
 
-- [`mdscrub`](mdscrub.md) — what to run before this.
+- [`mdscrub`](../mdscrub/README.md) — what to run before this.
-- [`cuepack`](cuepack.md) — the same record, from a recording.
+- [`cuepack`](../cuepack/README.md) — the same record, from a recording.
-- [`emb`](emb.md) — turning the text into vectors.
+- [`emb`](../emb/README.md) — turning the text into vectors.
-- [The records](records.md) — the shape this writes.
+- [The records](../../docs/records.md) — the shape this writes.

crates/corpus-core/README.md

@@ -2,12 +2,12 @@
 
 The record types and the stream plumbing everything shares.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it is
 
 A library, not a command — though it carries a small one for printing
-schemas. It holds the three [record types](records.md), the NDJSON reader and
+schemas. It holds the three [record types](../../docs/records.md), the NDJSON reader and
 writer, and the exit-code conventions.
 
 Everything else in the workspace depends on it, which is what makes the tools
@@ -23,9 +23,9 @@ else.
 
 ## The records
 
-[`Doc`](records.md#doc--what-a-file-is),
+[`Doc`](../../docs/records.md#doc--what-a-file-is),
-[`Chunk`](records.md#chunk--a-retrievable-passage) and
+[`Chunk`](../../docs/records.md#chunk--a-retrievable-passage) and
-[`Hit`](records.md#hit--one-scored-candidate). Each derives `Serialize`,
+[`Hit`](../../docs/records.md#hit--one-scored-candidate). Each derives `Serialize`,
 `Deserialize` and `JsonSchema`, and implements `Record`, which is how a reader
 knows what version it is looking at.
 
@@ -122,7 +122,7 @@ let answer = Verdict::from(needs_ocr);
 std::process::ExitCode::from(answer)
 ```
 
-See [the contract](contract.md#predicates-answer-with-exit-codes) for the
+See [the contract](../../docs/contract.md#exit-codes-carry-the-answer) for the
 whole table.
 
 ## The schemas
@@ -137,6 +137,6 @@ it can be committed and diffed.
 
 ## See also
 
-- [The records](records.md) — the shapes, described.
+- [The records](../../docs/records.md) — the shapes, described.
-- [`catena-cli`](catena-cli.md) — the rest of what a filter needs.
+- [`catena-cli`](../catena-cli/README.md) — the rest of what a filter needs.
-- [`textkit`](textkit.md) — counting and splitting text.
+- [`textkit`](../textkit/README.md) — counting and splitting text.

crates/cuekit/README.md

@@ -2,16 +2,16 @@
 
 The cue model, and the three subtitle formats it is read from.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it is
 
 A library. Three formats go in — SubRip, WebVTT and SubStation Alpha — and
-one [`cue`](records.md#cue--one-line-of-a-transcript) stream comes out.
+one [`cue`](../../docs/records.md#cue--one-line-of-a-transcript) stream comes out.
 Everything downstream works on cues and never has to know which of the three a
 recording arrived with.
 
-[`cuenorm`](cuenorm.md) is a command over this.
+[`cuenorm`](../cuenorm/README.md) is a command over this.
 
 ## When you would use it
 
@@ -49,7 +49,7 @@ let cue = Cue::new(12.5, 15.0, "the mill stood at the northern end");
 assert_eq!(cue.seconds(), 2.5);
 ```
 
-Deliberately smaller than a [chunk](records.md#chunk--a-retrievable-passage).
+Deliberately smaller than a [chunk](../../docs/records.md#chunk--a-retrievable-passage).
 A cue is what a subtitle file holds — a few seconds, often half a sentence —
 and turning cues into passages worth retrieving is a later stage's job.
 
@@ -99,6 +99,6 @@ makes a malformed file drop the same cues here as there.
 
 ## See also
 
-- [`cuenorm`](cuenorm.md) — the command over this.
+- [`cuenorm`](../cuenorm/README.md) — the command over this.
-- [`cuepack`](cuepack.md) and [`cueseg`](cueseg.md) — what reads cues.
+- [`cuepack`](../cuepack/README.md) and [`cueseg`](../cueseg/README.md) — what reads cues.
-- [The records](records.md) — the cue shape on the wire.
+- [The records](../../docs/records.md) — the cue shape on the wire.

crates/cuenorm/README.md

@@ -2,12 +2,12 @@
 
 Any subtitle format in, one normalised cue stream out.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
 Reads a subtitle file — SubRip, WebVTT or SubStation Alpha — and writes one
-[`cue` record](records.md#cue--one-line-of-a-transcript) per line: a start, an
+[`cue` record](../../docs/records.md#cue--one-line-of-a-transcript) per line: a start, an
 end, and what was said.
 
 The format is worked out from the content, not from the name, because a
@@ -123,10 +123,10 @@ for cue in &cues {
 }
 ```
 
-See [`cuekit`](cuekit.md) for the parsers one at a time.
+See [`cuekit`](../cuekit/README.md) for the parsers one at a time.
 
 ## See also
 
-- [`cuepack`](cuepack.md) — cues into passages worth embedding.
+- [`cuepack`](../cuepack/README.md) — cues into passages worth embedding.
-- [`cueseg`](cueseg.md) — cues into topic chapters.
+- [`cueseg`](../cueseg/README.md) — cues into topic chapters.
-- [`cuekit`](cuekit.md) — the library this is a command over.
+- [`cuekit`](../cuekit/README.md) — the library this is a command over.

crates/cuepack/README.md

@@ -2,13 +2,13 @@
 
 Cues into passages worth embedding, keeping the times they were said at.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
 Packs consecutive cues into passages of about a minute of continuous speech,
 and writes each one as a [`chunk`
-record](records.md#chunk--a-retrievable-passage) carrying the start of its
+record](../../docs/records.md#chunk--a-retrievable-passage) carrying the start of its
 first cue and the end of its last.
 
 ## When you would use it
@@ -20,7 +20,7 @@ first cue and the end of its last.
   first cue's start and the last cue's end, a hit can be cited to the second
   and a player can jump to it.
 - **To index recordings and books together.** The record is the same one
-  [`chunkmd`](chunkmd.md) writes, so one index answers across both and returns
+  [`chunkmd`](../chunkmd/README.md) writes, so one index answers across both and returns
   each with a location: a page number for one, a timestamp for the other.
 
 ## Install
@@ -84,7 +84,7 @@ with its own echo embeds as something nobody said.
 
 ## Input
 
-Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file. A
+Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file. A
 cue stream begins with a JSON object and a subtitle file does not, which is
 enough to tell them apart.
 
@@ -100,6 +100,6 @@ for passage in cuepack::pack(&cues, &Budgets::default()) {
 
 ## See also
 
-- [`cuenorm`](cuenorm.md) — what to run before this.
+- [`cuenorm`](../cuenorm/README.md) — what to run before this.
-- [`cueseg`](cueseg.md) — the other thing to do with a cue stream.
+- [`cueseg`](../cueseg/README.md) — the other thing to do with a cue stream.
-- [`chunkmd`](chunkmd.md) — the same record, from a book.
+- [`chunkmd`](../chunkmd/README.md) — the same record, from a book.

crates/cueseg/README.md

@@ -2,12 +2,12 @@
 
 A timed transcript into topic chapters.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
 Finds where a recording changes subject, and writes one [`segment`
-record](records.md#segment--one-topic-chapter) per chapter: a start, an end, a
+record](../../docs/records.md#segment--one-topic-chapter) per chapter: a start, an end, a
 title taken from the chapter's own first sentence, and an excerpt.
 
 No model and no network. The text is hashed into a small lexical vector a
@@ -103,7 +103,7 @@ than the end of the file is a mark that lands in the wrong place.
 
 ## Input
 
-Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file.
+Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file.
 
 ## As a library
 
@@ -115,5 +115,5 @@ for chapter in cueseg::segment(&cues, 7245.0) {
 
 ## See also
 
-- [`cuenorm`](cuenorm.md) — what to run before this.
+- [`cuenorm`](../cuenorm/README.md) — what to run before this.
-- [`cuepack`](cuepack.md) — the other thing to do with a cue stream.
+- [`cuepack`](../cuepack/README.md) — the other thing to do with a cue stream.

crates/emb/README.md

@@ -2,7 +2,7 @@
 
 Lines into vectors, against an embedding server.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -124,6 +124,6 @@ let vectors = emb::embed_batch(&agent, &endpoint, &texts)?;
 
 ## See also
 
-- [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md) — where the text comes
+- [`chunkmd`](../chunkmd/README.md) and [`cuepack`](../cuepack/README.md) — where the text comes
   from.
-- [`rrf`](rrf.md) — what to do with the results of searching those vectors.
+- [`rrf`](../rrf/README.md) — what to do with the results of searching those vectors.

crates/mdscrub/README.md

@@ -2,7 +2,7 @@
 
 Converted markdown in, clean markdown out.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -140,5 +140,5 @@ wants only one of them.
 
 ## See also
 
-- [`chunkmd`](chunkmd.md) — what to do with the markdown once it is clean.
+- [`chunkmd`](../chunkmd/README.md) — what to do with the markdown once it is clean.
-- [`pdftriage`](pdftriage.md) — whether the text was worth extracting at all.
+- [`pdftriage`](../pdftriage/README.md) — whether the text was worth extracting at all.

crates/pdftriage/README.md

@@ -2,7 +2,7 @@
 
 Whether a PDF's text layer is worth reading, or whether it has to be OCRed.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
@@ -79,7 +79,7 @@ $ pdftriage --json book.pdf
 
 | Option | Effect |
 |---|---|
-| `--json` | Emit a [`doc` record](records.md#doc--what-a-file-is) instead of a readable verdict |
+| `--json` | Emit a [`doc` record](../../docs/records.md#doc--what-a-file-is) instead of a readable verdict |
 | `-q`, `--quiet` | Write no verdict; answer with the exit code alone. Diagnostics still go to stderr, because a silent failure is worse than a quiet one |
 
 ## Exit codes
@@ -131,6 +131,6 @@ Running `pdftotext` is the command's job, not the library's.
 
 ## See also
 
-- [`booksniff`](booksniff.md) — whether it is a PDF at all.
+- [`booksniff`](../booksniff/README.md) — whether it is a PDF at all.
-- [`mdscrub`](mdscrub.md) — cleaning up the text once it has been extracted.
+- [`mdscrub`](../mdscrub/README.md) — cleaning up the text once it has been extracted.
-- [The records](records.md) — the shape `--json` writes.
+- [The records](../../docs/records.md) — the shape `--json` writes.

crates/rrf/README.md

@@ -2,12 +2,12 @@
 
 Several ranked lists into one, without the database.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it does
 
 Reciprocal rank fusion. Reads [`hit`
-records](records.md#hit--one-scored-candidate) from any number of retrieval
+records](../../docs/records.md#hit--one-scored-candidate) from any number of retrieval
 channels and writes one ranked list, best first.
 
 ## When you would use it
@@ -88,7 +88,7 @@ it.
 
 ## Input and output
 
-One [`hit`](records.md#hit--one-scored-candidate) per line, in any order and
+One [`hit`](../../docs/records.md#hit--one-scored-candidate) per line, in any order and
 interleaved if you like — each carries the rank its own channel gave it, so
 the stream does not have to be grouped.
 
@@ -110,5 +110,5 @@ let fused = rrf::fuse(&hits, &rrf::Weights::default());
 
 ## See also
 
-- [`emb`](emb.md) — producing the vectors one of the channels searches.
+- [`emb`](../emb/README.md) — producing the vectors one of the channels searches.
-- [The records](records.md) — the shape this reads and writes.
+- [The records](../../docs/records.md) — the shape this reads and writes.

crates/textkit/README.md

@@ -2,7 +2,7 @@
 
 Token counting, sentence and paragraph splitting.
 
-Back to [the index](README.md).
+Part of [catena](../../README.md).
 
 ## What it is
 
@@ -14,7 +14,7 @@ and cutting text into sentences and paragraphs.
 
 - **Deciding how much text fits** in a model's context, before sending it.
 - **Writing your own chunker** that has to agree with
-  [`chunkmd`](chunkmd.md) about where a boundary is.
+  [`chunkmd`](../chunkmd/README.md) about where a boundary is.
 - **Porting anything from Python** where the exact behaviour of `\s`, `\w` or
   `\b` matters more than it looks like it should.
 
@@ -117,5 +117,5 @@ that has them.
 
 ## See also
 
-- [`corpus-core`](corpus-core.md) — the records these help produce.
+- [`corpus-core`](../corpus-core/README.md) — the records these help produce.
-- [`chunkmd`](chunkmd.md) — the chunker built on this.
+- [`chunkmd`](../chunkmd/README.md) — the chunker built on this.

docs/README.md

@@ -1,79 +0,0 @@
-# catena
-
-Small programs that each do one part of turning a pile of files into a
-searchable corpus, and hand the next one text on stdout.
-
-Every one of them is a filter: bytes in, records out, diagnostics on stderr,
-and an exit code that means something. None of them needs a database, a GPU, a
-queue or a config file.
-
-## The tools
-
-### Working out what you have
-
-| | |
-|---|---|
-| [`booksniff`](booksniff.md) | What a file actually is, whatever its name claims |
-| [`pdftriage`](pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
-| [`bibclean`](bibclean.md) | Catalogue records audited and repaired, with a reason for every change |
-
-### Turning documents into passages
-
-| | |
-|---|---|
-| [`mdscrub`](mdscrub.md) | Converted markdown in, clean markdown out |
-| [`chunkmd`](chunkmd.md) | Markdown into passages that carry their page, chapter and role |
-
-### Turning recordings into passages
-
-| | |
-|---|---|
-| [`cuenorm`](cuenorm.md) | Any subtitle format in, one normalised cue stream out |
-| [`cuepack`](cuepack.md) | Cues into passages worth embedding, keeping the times they were said at |
-| [`cueseg`](cueseg.md) | A timed transcript into topic chapters |
-
-### Searching
-
-| | |
-|---|---|
-| [`emb`](emb.md) | Lines into vectors, against an embedding server |
-| [`rrf`](rrf.md) | Several ranked lists into one, without the database |
-
-## The libraries underneath
-
-Every tool is a library crate with a thin command on top, so anything a
-command does can be done in process instead.
-
-| | |
-|---|---|
-| [`corpus-core`](corpus-core.md) | The record types and the stream plumbing everything shares |
-| [`textkit`](textkit.md) | Token counting, sentence and paragraph splitting |
-| [`cuekit`](cuekit.md) | The cue model, and the three subtitle formats it is read from |
-| [`catena-cli`](catena-cli.md) | The parts of a filter every command does the same way |
-
-## Reference
-
-- [The records](records.md) — the shapes that travel between the tools.
-- [The contract](contract.md) — what every one of these programs promises.
-
-## A worked pipeline
-
-A book, from a converted chapter to passages an index can hold:
-
-```
-$ mdscrub chapter.md | chunkmd --min-tokens 20
-{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
-{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
-```
-
-A recording, through the same shape:
-
-```
-$ cuenorm episode.vtt | cuepack --seconds 10
-{"v":1,"idx":0,"text":"The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest.","tokens":27,"page":null,"chapter":null,"role":"body","start_time":1.0,"end_time":8.0,"flags":[]}
-{"v":1,"idx":1,"text":"Its wheel was undershot, which was wasteful,","tokens":11,"page":null,"chapter":null,"role":"body","start_time":11.0,"end_time":19.0,"flags":[]}
-```
-
-Both produce the same record, so one index answers across a book and a lecture
-and returns both with a location you can cite — a page number for one, a
-timestamp for the other.

docs/contract.md

@@ -1,33 +1,41 @@
-# The contract
+# What these programs promise
 
-Every program here obeys all of this, or it does not ship. It is worth reading
+Eight things hold for every program here. Together they are what lets a person
-once, because it is what lets these be used together without reading any of
+put any of them in a pipeline without first reading how it works.
-them.
 
-Back to [the index](README.md).
+Part of [catena](../README.md).
 
-## One job
+## Each one does a single job
 
-A tool that needs a `--mode` flag to change what it *is* is two tools.
+No program here has a mode switch that changes what it is. Where two jobs
+would otherwise share a name, there are two programs — which is why
+[`cuenorm`](../crates/cuenorm/README.md), [`cuepack`](../crates/cuepack/README.md) and [`cueseg`](../crates/cueseg/README.md) are
+separate commands rather than one command with a flag.
 
-## stdin to stdout
+## Input on stdin, results on stdout, diagnostics on stderr
 
-Input on stdin or a path argument, result on stdout, diagnostics on stderr
+Output and commentary never share a stream, so a result can always be piped
-only — never content and chatter on the same stream. A tool that takes a path
+somewhere without filtering anything out of it.
-instead of a stream says why in its `--help`; [`pdftriage`](pdftriage.md) does,
-because it reads two specific page ranges rather than the whole file.
 
-## Streams, not documents
+Almost all of them read stdin or a path. Where one takes a path only, it says
+so in `--help` and says why: [`pdftriage`](../crates/pdftriage/README.md) reads two specific
+page ranges rather than the whole file, which a stream cannot offer.
 
-Anything list-shaped is NDJSON: one [record](records.md) per line, flushed as
+## Anything list-shaped is NDJSON
-it is produced, so a slow stage never blocks the pipe behind it.
 
-The reader is strict on purpose. It refuses a record whose version it does not
+One [record](records.md) per line, flushed as it is produced — so a slow stage
-know, it stops at the first bad line rather than reading past a disagreement
+downstream sees each record as it appears rather than when the producer
-about the format, and it bounds how much one line may cost — a filter pointed
+finishes.
-at a binary file should fail, not allocate the file.
 
-## Predicates answer with exit codes
+Readers are strict, and each strictness prevents a failure that would
+otherwise be silent. A record from a version the reader does not know is
+refused rather than parsed into whatever lines up. A bad line ends the stream
+instead of being skipped, because it means the two ends disagree about the
+format. Lines are bounded, so a program pointed at a binary file fails instead
+of allocating it. A byte-order mark and blank lines are ignored, so a stream
+that was concatenated or written on Windows still reads.
+
+## Exit codes carry the answer
 
 | Code | Meaning |
 |---|---|
@@ -35,44 +43,47 @@ at a binary file should fail, not allocate the file.
 | `1` | No |
 | `2` | Could not tell |
 | `64` | The command line was wrong |
-| `65` | The input was not what the tool can work on |
+| `65` | The input was not what the program can work on |
 | `69` | A program it needs is not installed |
-| `70` | The tool's own fault |
+| `70` | Its own fault |
 | `74` | Reading or writing failed |
 
-The failure codes start where `sysexits.h` puts them so that "no" and "broke"
+Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
-are never the same number. That is what makes this work:
+same number. That is what makes this work:
 
 ```sh
 for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
 ```
 
-A consumer closing the pipe is not a failure. `… | head -1` exits `0`, like
+A consumer closing the pipe is not a failure. `… | head -1` exits `0`, the
-every other filter.
+same as any other filter.
 
-## No network, no database, no config file, no state
+## Nothing reaches outside the process
 
+No network, no database, no configuration file, no state kept between runs.
 Two exceptions, both declared in `--help`:
 
-- [`emb`](emb.md), whose whole job is an HTTP call.
+- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
-- [`pdftriage`](pdftriage.md), which shells out to `pdftotext` and `pdfinfo`
+- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`, and exits `69`
-  and exits `69` when either is missing.
+  when either is missing rather than guessing.
 
-## No ambient knowledge
+## Nothing is assumed about where it is running
 
-No hostnames, no local paths, no private vocabularies compiled in. General
+No hostnames, no paths, no private vocabularies are built in. General
-scheme data — ISBN check digits, the HTML character references — may be;
+reference data is — ISBN check digits, the HTML character references, the
-anything local is a `--rules` file.
+names of well-known publishers — and anything specific to one collection is
+supplied in a file rather than compiled in.
 
-## Deterministic
+## The same input gives the same output
 
-Same bytes in, same bytes out, across runs and machines. No hash-order
+On every run and on every machine. No dependence on hash ordering, no reading
-dependence, no clock, no locale reads. Floats that JSON cannot carry are
+of the clock or the locale. Numbers that JSON cannot represent are refused
-refused where they are produced rather than written as `null` and lost.
+where they are produced rather than written as `null` and lost somewhere
+downstream.
 
-## `--json` everywhere
+## `--json` is the stable form
 
-The human-readable default is for a person at a terminal; `--json` is the
+Where a program prints something for a person to read, `--json` prints the
-contract, and it does not change shape without a major version. Tools whose
+same answer as a record, and that record does not change shape without a major
-only sensible output is a stream of records write that stream and have no
+version. Programs whose only sensible output is a stream of records write that
-`--json` to add.
+stream and have no `--json` to add.

docs/records.md

@@ -4,10 +4,10 @@ Five shapes travel between these tools. Each carries a version field `v`, so a
 reader can refuse a stream it does not understand rather than parsing it into
 whatever happens to line up.
 
-Every list-shaped stream is [NDJSON](contract.md#streams-not-documents): one
+Every list-shaped stream is [NDJSON](contract.md#anything-list-shaped-is-ndjson): one
 record per line, flushed as it is produced.
 
-Back to [the index](README.md).
+Part of [catena](../README.md).
 
 ## `doc` — what a file is
 
@@ -31,7 +31,7 @@ Back to [the index](README.md).
 A tool fills what it can and leaves the rest null: a format sniffer fills
 `kind` and leaves `text_layer` empty, a triage tool does the reverse.
 
-Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md).
+Written by [`booksniff`](../crates/booksniff/README.md) and [`pdftriage`](../crates/pdftriage/README.md).
 
 ## `chunk` — a retrievable passage
 
@@ -55,7 +55,7 @@ Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md).
 recordings. One shape either way, which is what lets a single index serve a
 book and a lecture and cite both.
 
-Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md).
+Written by [`chunkmd`](../crates/chunkmd/README.md) and [`cuepack`](../crates/cuepack/README.md).
 
 ## `hit` — one scored candidate
 
@@ -74,7 +74,7 @@ Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md).
 `rank` is what fusion consumes. `score` is carried along because it is useful
 to a person reading a trace and useless for comparing across channels.
 
-Read and written by [`rrf`](rrf.md).
+Read and written by [`rrf`](../crates/rrf/README.md).
 
 ## `cue` — one line of a transcript
 
@@ -85,8 +85,8 @@ Read and written by [`rrf`](rrf.md).
 Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few
 seconds and often half a sentence.
 
-Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and
+Written by [`cuenorm`](../crates/cuenorm/README.md), read by [`cuepack`](../crates/cuepack/README.md) and
-[`cueseg`](cueseg.md).
+[`cueseg`](../crates/cueseg/README.md).
 
 ## `segment` — one topic chapter
 
@@ -98,7 +98,7 @@ Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and
 The marks a scrubber needs, and the spans a summariser should be given one at
 a time.
 
-Written by [`cueseg`](cueseg.md).
+Written by [`cueseg`](../crates/cueseg/README.md).
 
 ## Getting the schemas
 
@@ -113,4 +113,4 @@ $ cargo run -p corpus-core -- schema chunk
 }
 ```
 
-See [`corpus-core`](corpus-core.md).
+See [`corpus-core`](../crates/corpus-core/README.md).