mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
each crate carries its own readme, and the contract describes rather than instructs
This commit is contained in:
funkpower
@@ -23,19 +23,31 @@ Each tool installs by name, so you can take one without taking the rest.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`booksniff`](docs/booksniff.md) | What a file actually is, whatever its name claims |
|
||||
| [`pdftriage`](docs/pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
|
||||
| [`bibclean`](docs/bibclean.md) | Catalogue records audited and repaired, with a reason for every change |
|
||||
| [`mdscrub`](docs/mdscrub.md) | Converted markdown in, clean markdown out |
|
||||
| [`chunkmd`](docs/chunkmd.md) | Markdown into passages that carry their page, chapter and role |
|
||||
| [`cuenorm`](docs/cuenorm.md) | Any subtitle format in, one normalised cue stream out |
|
||||
| [`cuepack`](docs/cuepack.md) | Cues into passages worth embedding, keeping the times they were said at |
|
||||
| [`cueseg`](docs/cueseg.md) | A timed transcript into topic chapters |
|
||||
| [`emb`](docs/emb.md) | Lines into vectors, against an embedding server |
|
||||
| [`rrf`](docs/rrf.md) | Several ranked lists into one, without the database |
|
||||
| [`booksniff`](crates/booksniff/README.md) | What a file actually is, whatever its name claims |
|
||||
| [`pdftriage`](crates/pdftriage/README.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
|
||||
| [`bibclean`](crates/bibclean/README.md) | Catalogue records audited and repaired, with a reason for every change |
|
||||
| [`mdscrub`](crates/mdscrub/README.md) | Converted markdown in, clean markdown out |
|
||||
| [`chunkmd`](crates/chunkmd/README.md) | Markdown into passages that carry their page, chapter and role |
|
||||
| [`cuenorm`](crates/cuenorm/README.md) | Any subtitle format in, one normalised cue stream out |
|
||||
| [`cuepack`](crates/cuepack/README.md) | Cues into passages worth embedding, keeping the times they were said at |
|
||||
| [`cueseg`](crates/cueseg/README.md) | A timed transcript into topic chapters |
|
||||
| [`emb`](crates/emb/README.md) | Lines into vectors, against an embedding server |
|
||||
| [`rrf`](crates/rrf/README.md) | Several ranked lists into one, without the database |
|
||||
|
||||
Full documentation, including the libraries underneath, is in
|
||||
**[docs/](docs/README.md)**.
|
||||
Each links to its own page: what it does, when you would use it, worked
|
||||
examples, options and exit codes.
|
||||
|
||||
## The libraries underneath
|
||||
|
||||
Every tool is a library crate with a thin command on top, so anything a
|
||||
command does can be done in process instead.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`corpus-core`](crates/corpus-core/README.md) | The record types and the stream plumbing everything shares |
|
||||
| [`textkit`](crates/textkit/README.md) | Token counting, sentence and paragraph splitting |
|
||||
| [`cuekit`](crates/cuekit/README.md) | The cue model, and the three subtitle formats it is read from |
|
||||
| [`catena-cli`](crates/catena-cli/README.md) | The parts of a filter every command does the same way |
|
||||
|
||||
## A worked pipeline
|
||||
|
||||
@@ -67,32 +79,33 @@ garbled, needs OCR
|
||||
$ for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
|
||||
```
|
||||
|
||||
## The contract
|
||||
## What they promise
|
||||
|
||||
Every binary in this workspace obeys all of this, or it does not ship. The
|
||||
whole of it is in [docs/contract.md](docs/contract.md).
|
||||
Eight things hold for all of them, and together they are what lets any of
|
||||
these go into a pipeline without first reading how it works:
|
||||
|
||||
1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
|
||||
tools.
|
||||
2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
|
||||
diagnostics on stderr only.
|
||||
3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
|
||||
line, flushed as it is produced.
|
||||
4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell;
|
||||
failures use the `sysexits` range, so "no" and "broke" are never the same
|
||||
number.
|
||||
5. **No network, no database, no config file, no state**, with two declared
|
||||
exceptions.
|
||||
6. **No ambient knowledge.** No hostnames, no local paths, no private
|
||||
vocabularies compiled in.
|
||||
7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
|
||||
8. **`--json` everywhere.** The human-readable default is for a person at a
|
||||
terminal; `--json` is the contract.
|
||||
- Each does a single job, with no mode switch that changes what it is.
|
||||
- Input on stdin, results on stdout, diagnostics on stderr — never mixed.
|
||||
- Anything list-shaped is NDJSON, one record per line, flushed as produced.
|
||||
- Exit codes carry the answer: `0` yes, `1` no, `2` could not tell, and
|
||||
failures in the `sysexits` range, so "no" and "broke" are never the same
|
||||
number.
|
||||
- Nothing reaches the network, a database, a config file or any stored state,
|
||||
bar two exceptions that say so in `--help`.
|
||||
- Nothing is assumed about where it is running: no hostnames, no paths, no
|
||||
private vocabularies built in.
|
||||
- The same input gives the same output, on every run and every machine.
|
||||
- `--json` is the stable form, and does not change shape without a major
|
||||
version.
|
||||
|
||||
The whole of it, with the reasoning: [what these programs
|
||||
promise](docs/contract.md).
|
||||
|
||||
## The records
|
||||
|
||||
Three shapes travel between the tools, versioned, defined in `corpus-core` and
|
||||
described in [docs/records.md](docs/records.md).
|
||||
Five shapes travel between the tools, versioned, defined in
|
||||
[`corpus-core`](crates/corpus-core/README.md) and described in [the
|
||||
records](docs/records.md).
|
||||
|
||||
```
|
||||
$ cargo run -p corpus-core -- schema all
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Catalogue records audited and repaired, with a reason for every change.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -150,5 +150,5 @@ description to another.
|
||||
|
||||
## See also
|
||||
|
||||
- [`booksniff`](booksniff.md) — what the file behind the record actually is.
|
||||
- [The records](records.md) — the shapes the rest of the suite uses.
|
||||
- [`booksniff`](../booksniff/README.md) — what the file behind the record actually is.
|
||||
- [The records](../../docs/records.md) — the shapes the rest of the suite uses.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
What a file actually is, whatever its name claims.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -11,7 +11,7 @@ and says what format it really is. It never reads the whole file, so a
|
||||
directory of nine-hundred-megabyte scans costs the same as a directory of
|
||||
anything else.
|
||||
|
||||
Writes one [`doc` record](records.md#doc--what-a-file-is) per file, with a
|
||||
Writes one [`doc` record](../../docs/records.md#doc--what-a-file-is) per file, with a
|
||||
note for anything worth remarking on: a name that disagrees with the bytes, a
|
||||
file too small to be a book, a name that says "sample".
|
||||
|
||||
@@ -113,6 +113,6 @@ the bytes need not write them to disk first.
|
||||
|
||||
## See also
|
||||
|
||||
- [`pdftriage`](pdftriage.md) — once you know it is a PDF, whether its text is
|
||||
- [`pdftriage`](../pdftriage/README.md) — once you know it is a PDF, whether its text is
|
||||
worth reading.
|
||||
- [The records](records.md) — the shape this writes.
|
||||
- [The records](../../docs/records.md) — the shape this writes.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The parts of a filter every command does the same way.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it is
|
||||
|
||||
@@ -16,7 +16,7 @@ the same at the edges.
|
||||
## When you would use it
|
||||
|
||||
- **Writing a filter that joins this pipeline**, and wanting it to behave like
|
||||
the others without rereading [the contract](contract.md).
|
||||
the others without rereading [the contract](../../docs/contract.md).
|
||||
- **Anywhere a small Unix filter is wanted**, in or out of this workspace.
|
||||
|
||||
## Parsing a command line
|
||||
@@ -117,5 +117,5 @@ code, and should stay under a hundred and fifty lines.
|
||||
|
||||
## See also
|
||||
|
||||
- [The contract](contract.md) — what these functions are enforcing.
|
||||
- [`corpus-core`](corpus-core.md) — the records and the exit-code constants.
|
||||
- [The contract](../../docs/contract.md) — what these functions are enforcing.
|
||||
- [`corpus-core`](../corpus-core/README.md) — the records and the exit-code constants.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Markdown into passages that carry their page, chapter and role.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -10,7 +10,7 @@ Cuts a document into pieces of about the size an embedding model wants, and
|
||||
puts on each one the page it came from, the chapter it is in, and what part of
|
||||
the work it belongs to.
|
||||
|
||||
Writes [`chunk` records](records.md#chunk--a-retrievable-passage), one per
|
||||
Writes [`chunk` records](../../docs/records.md#chunk--a-retrievable-passage), one per
|
||||
line.
|
||||
|
||||
## When you would use it
|
||||
@@ -125,7 +125,7 @@ for chunk in chunkmd::chunk(&markdown, &Budgets::default()) {
|
||||
|
||||
## See also
|
||||
|
||||
- [`mdscrub`](mdscrub.md) — what to run before this.
|
||||
- [`cuepack`](cuepack.md) — the same record, from a recording.
|
||||
- [`emb`](emb.md) — turning the text into vectors.
|
||||
- [The records](records.md) — the shape this writes.
|
||||
- [`mdscrub`](../mdscrub/README.md) — what to run before this.
|
||||
- [`cuepack`](../cuepack/README.md) — the same record, from a recording.
|
||||
- [`emb`](../emb/README.md) — turning the text into vectors.
|
||||
- [The records](../../docs/records.md) — the shape this writes.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
The record types and the stream plumbing everything shares.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it is
|
||||
|
||||
A library, not a command — though it carries a small one for printing
|
||||
schemas. It holds the three [record types](records.md), the NDJSON reader and
|
||||
schemas. It holds the three [record types](../../docs/records.md), the NDJSON reader and
|
||||
writer, and the exit-code conventions.
|
||||
|
||||
Everything else in the workspace depends on it, which is what makes the tools
|
||||
@@ -23,9 +23,9 @@ else.
|
||||
|
||||
## The records
|
||||
|
||||
[`Doc`](records.md#doc--what-a-file-is),
|
||||
[`Chunk`](records.md#chunk--a-retrievable-passage) and
|
||||
[`Hit`](records.md#hit--one-scored-candidate). Each derives `Serialize`,
|
||||
[`Doc`](../../docs/records.md#doc--what-a-file-is),
|
||||
[`Chunk`](../../docs/records.md#chunk--a-retrievable-passage) and
|
||||
[`Hit`](../../docs/records.md#hit--one-scored-candidate). Each derives `Serialize`,
|
||||
`Deserialize` and `JsonSchema`, and implements `Record`, which is how a reader
|
||||
knows what version it is looking at.
|
||||
|
||||
@@ -122,7 +122,7 @@ let answer = Verdict::from(needs_ocr);
|
||||
std::process::ExitCode::from(answer)
|
||||
```
|
||||
|
||||
See [the contract](contract.md#predicates-answer-with-exit-codes) for the
|
||||
See [the contract](../../docs/contract.md#exit-codes-carry-the-answer) for the
|
||||
whole table.
|
||||
|
||||
## The schemas
|
||||
@@ -137,6 +137,6 @@ it can be committed and diffed.
|
||||
|
||||
## See also
|
||||
|
||||
- [The records](records.md) — the shapes, described.
|
||||
- [`catena-cli`](catena-cli.md) — the rest of what a filter needs.
|
||||
- [`textkit`](textkit.md) — counting and splitting text.
|
||||
- [The records](../../docs/records.md) — the shapes, described.
|
||||
- [`catena-cli`](../catena-cli/README.md) — the rest of what a filter needs.
|
||||
- [`textkit`](../textkit/README.md) — counting and splitting text.
|
||||
@@ -2,16 +2,16 @@
|
||||
|
||||
The cue model, and the three subtitle formats it is read from.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it is
|
||||
|
||||
A library. Three formats go in — SubRip, WebVTT and SubStation Alpha — and
|
||||
one [`cue`](records.md#cue--one-line-of-a-transcript) stream comes out.
|
||||
one [`cue`](../../docs/records.md#cue--one-line-of-a-transcript) stream comes out.
|
||||
Everything downstream works on cues and never has to know which of the three a
|
||||
recording arrived with.
|
||||
|
||||
[`cuenorm`](cuenorm.md) is a command over this.
|
||||
[`cuenorm`](../cuenorm/README.md) is a command over this.
|
||||
|
||||
## When you would use it
|
||||
|
||||
@@ -49,7 +49,7 @@ let cue = Cue::new(12.5, 15.0, "the mill stood at the northern end");
|
||||
assert_eq!(cue.seconds(), 2.5);
|
||||
```
|
||||
|
||||
Deliberately smaller than a [chunk](records.md#chunk--a-retrievable-passage).
|
||||
Deliberately smaller than a [chunk](../../docs/records.md#chunk--a-retrievable-passage).
|
||||
A cue is what a subtitle file holds — a few seconds, often half a sentence —
|
||||
and turning cues into passages worth retrieving is a later stage's job.
|
||||
|
||||
@@ -99,6 +99,6 @@ makes a malformed file drop the same cues here as there.
|
||||
|
||||
## See also
|
||||
|
||||
- [`cuenorm`](cuenorm.md) — the command over this.
|
||||
- [`cuepack`](cuepack.md) and [`cueseg`](cueseg.md) — what reads cues.
|
||||
- [The records](records.md) — the cue shape on the wire.
|
||||
- [`cuenorm`](../cuenorm/README.md) — the command over this.
|
||||
- [`cuepack`](../cuepack/README.md) and [`cueseg`](../cueseg/README.md) — what reads cues.
|
||||
- [The records](../../docs/records.md) — the cue shape on the wire.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
Any subtitle format in, one normalised cue stream out.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
Reads a subtitle file — SubRip, WebVTT or SubStation Alpha — and writes one
|
||||
[`cue` record](records.md#cue--one-line-of-a-transcript) per line: a start, an
|
||||
[`cue` record](../../docs/records.md#cue--one-line-of-a-transcript) per line: a start, an
|
||||
end, and what was said.
|
||||
|
||||
The format is worked out from the content, not from the name, because a
|
||||
@@ -123,10 +123,10 @@ for cue in &cues {
|
||||
}
|
||||
```
|
||||
|
||||
See [`cuekit`](cuekit.md) for the parsers one at a time.
|
||||
See [`cuekit`](../cuekit/README.md) for the parsers one at a time.
|
||||
|
||||
## See also
|
||||
|
||||
- [`cuepack`](cuepack.md) — cues into passages worth embedding.
|
||||
- [`cueseg`](cueseg.md) — cues into topic chapters.
|
||||
- [`cuekit`](cuekit.md) — the library this is a command over.
|
||||
- [`cuepack`](../cuepack/README.md) — cues into passages worth embedding.
|
||||
- [`cueseg`](../cueseg/README.md) — cues into topic chapters.
|
||||
- [`cuekit`](../cuekit/README.md) — the library this is a command over.
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
Cues into passages worth embedding, keeping the times they were said at.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
Packs consecutive cues into passages of about a minute of continuous speech,
|
||||
and writes each one as a [`chunk`
|
||||
record](records.md#chunk--a-retrievable-passage) carrying the start of its
|
||||
record](../../docs/records.md#chunk--a-retrievable-passage) carrying the start of its
|
||||
first cue and the end of its last.
|
||||
|
||||
## When you would use it
|
||||
@@ -20,7 +20,7 @@ first cue and the end of its last.
|
||||
first cue's start and the last cue's end, a hit can be cited to the second
|
||||
and a player can jump to it.
|
||||
- **To index recordings and books together.** The record is the same one
|
||||
[`chunkmd`](chunkmd.md) writes, so one index answers across both and returns
|
||||
[`chunkmd`](../chunkmd/README.md) writes, so one index answers across both and returns
|
||||
each with a location: a page number for one, a timestamp for the other.
|
||||
|
||||
## Install
|
||||
@@ -84,7 +84,7 @@ with its own echo embeds as something nobody said.
|
||||
|
||||
## Input
|
||||
|
||||
Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file. A
|
||||
Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file. A
|
||||
cue stream begins with a JSON object and a subtitle file does not, which is
|
||||
enough to tell them apart.
|
||||
|
||||
@@ -100,6 +100,6 @@ for passage in cuepack::pack(&cues, &Budgets::default()) {
|
||||
|
||||
## See also
|
||||
|
||||
- [`cuenorm`](cuenorm.md) — what to run before this.
|
||||
- [`cueseg`](cueseg.md) — the other thing to do with a cue stream.
|
||||
- [`chunkmd`](chunkmd.md) — the same record, from a book.
|
||||
- [`cuenorm`](../cuenorm/README.md) — what to run before this.
|
||||
- [`cueseg`](../cueseg/README.md) — the other thing to do with a cue stream.
|
||||
- [`chunkmd`](../chunkmd/README.md) — the same record, from a book.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
A timed transcript into topic chapters.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
Finds where a recording changes subject, and writes one [`segment`
|
||||
record](records.md#segment--one-topic-chapter) per chapter: a start, an end, a
|
||||
record](../../docs/records.md#segment--one-topic-chapter) per chapter: a start, an end, a
|
||||
title taken from the chapter's own first sentence, and an excerpt.
|
||||
|
||||
No model and no network. The text is hashed into a small lexical vector a
|
||||
@@ -103,7 +103,7 @@ than the end of the file is a mark that lands in the wrong place.
|
||||
|
||||
## Input
|
||||
|
||||
Either the cue stream [`cuenorm`](cuenorm.md) writes, or a subtitle file.
|
||||
Either the cue stream [`cuenorm`](../cuenorm/README.md) writes, or a subtitle file.
|
||||
|
||||
## As a library
|
||||
|
||||
@@ -115,5 +115,5 @@ for chapter in cueseg::segment(&cues, 7245.0) {
|
||||
|
||||
## See also
|
||||
|
||||
- [`cuenorm`](cuenorm.md) — what to run before this.
|
||||
- [`cuepack`](cuepack.md) — the other thing to do with a cue stream.
|
||||
- [`cuenorm`](../cuenorm/README.md) — what to run before this.
|
||||
- [`cuepack`](../cuepack/README.md) — the other thing to do with a cue stream.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Lines into vectors, against an embedding server.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -124,6 +124,6 @@ let vectors = emb::embed_batch(&agent, &endpoint, &texts)?;
|
||||
|
||||
## See also
|
||||
|
||||
- [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md) — where the text comes
|
||||
- [`chunkmd`](../chunkmd/README.md) and [`cuepack`](../cuepack/README.md) — where the text comes
|
||||
from.
|
||||
- [`rrf`](rrf.md) — what to do with the results of searching those vectors.
|
||||
- [`rrf`](../rrf/README.md) — what to do with the results of searching those vectors.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Converted markdown in, clean markdown out.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -140,5 +140,5 @@ wants only one of them.
|
||||
|
||||
## See also
|
||||
|
||||
- [`chunkmd`](chunkmd.md) — what to do with the markdown once it is clean.
|
||||
- [`pdftriage`](pdftriage.md) — whether the text was worth extracting at all.
|
||||
- [`chunkmd`](../chunkmd/README.md) — what to do with the markdown once it is clean.
|
||||
- [`pdftriage`](../pdftriage/README.md) — whether the text was worth extracting at all.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Whether a PDF's text layer is worth reading, or whether it has to be OCRed.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
@@ -79,7 +79,7 @@ $ pdftriage --json book.pdf
|
||||
|
||||
| Option | Effect |
|
||||
|---|---|
|
||||
| `--json` | Emit a [`doc` record](records.md#doc--what-a-file-is) instead of a readable verdict |
|
||||
| `--json` | Emit a [`doc` record](../../docs/records.md#doc--what-a-file-is) instead of a readable verdict |
|
||||
| `-q`, `--quiet` | Write no verdict; answer with the exit code alone. Diagnostics still go to stderr, because a silent failure is worse than a quiet one |
|
||||
|
||||
## Exit codes
|
||||
@@ -131,6 +131,6 @@ Running `pdftotext` is the command's job, not the library's.
|
||||
|
||||
## See also
|
||||
|
||||
- [`booksniff`](booksniff.md) — whether it is a PDF at all.
|
||||
- [`mdscrub`](mdscrub.md) — cleaning up the text once it has been extracted.
|
||||
- [The records](records.md) — the shape `--json` writes.
|
||||
- [`booksniff`](../booksniff/README.md) — whether it is a PDF at all.
|
||||
- [`mdscrub`](../mdscrub/README.md) — cleaning up the text once it has been extracted.
|
||||
- [The records](../../docs/records.md) — the shape `--json` writes.
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
Several ranked lists into one, without the database.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it does
|
||||
|
||||
Reciprocal rank fusion. Reads [`hit`
|
||||
records](records.md#hit--one-scored-candidate) from any number of retrieval
|
||||
records](../../docs/records.md#hit--one-scored-candidate) from any number of retrieval
|
||||
channels and writes one ranked list, best first.
|
||||
|
||||
## When you would use it
|
||||
@@ -88,7 +88,7 @@ it.
|
||||
|
||||
## Input and output
|
||||
|
||||
One [`hit`](records.md#hit--one-scored-candidate) per line, in any order and
|
||||
One [`hit`](../../docs/records.md#hit--one-scored-candidate) per line, in any order and
|
||||
interleaved if you like — each carries the rank its own channel gave it, so
|
||||
the stream does not have to be grouped.
|
||||
|
||||
@@ -110,5 +110,5 @@ let fused = rrf::fuse(&hits, &rrf::Weights::default());
|
||||
|
||||
## See also
|
||||
|
||||
- [`emb`](emb.md) — producing the vectors one of the channels searches.
|
||||
- [The records](records.md) — the shape this reads and writes.
|
||||
- [`emb`](../emb/README.md) — producing the vectors one of the channels searches.
|
||||
- [The records](../../docs/records.md) — the shape this reads and writes.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Token counting, sentence and paragraph splitting.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../../README.md).
|
||||
|
||||
## What it is
|
||||
|
||||
@@ -14,7 +14,7 @@ and cutting text into sentences and paragraphs.
|
||||
|
||||
- **Deciding how much text fits** in a model's context, before sending it.
|
||||
- **Writing your own chunker** that has to agree with
|
||||
[`chunkmd`](chunkmd.md) about where a boundary is.
|
||||
[`chunkmd`](../chunkmd/README.md) about where a boundary is.
|
||||
- **Porting anything from Python** where the exact behaviour of `\s`, `\w` or
|
||||
`\b` matters more than it looks like it should.
|
||||
|
||||
@@ -117,5 +117,5 @@ that has them.
|
||||
|
||||
## See also
|
||||
|
||||
- [`corpus-core`](corpus-core.md) — the records these help produce.
|
||||
- [`chunkmd`](chunkmd.md) — the chunker built on this.
|
||||
- [`corpus-core`](../corpus-core/README.md) — the records these help produce.
|
||||
- [`chunkmd`](../chunkmd/README.md) — the chunker built on this.
|
||||
@@ -1,79 +0,0 @@
|
||||
# catena
|
||||
|
||||
Small programs that each do one part of turning a pile of files into a
|
||||
searchable corpus, and hand the next one text on stdout.
|
||||
|
||||
Every one of them is a filter: bytes in, records out, diagnostics on stderr,
|
||||
and an exit code that means something. None of them needs a database, a GPU, a
|
||||
queue or a config file.
|
||||
|
||||
## The tools
|
||||
|
||||
### Working out what you have
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`booksniff`](booksniff.md) | What a file actually is, whatever its name claims |
|
||||
| [`pdftriage`](pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
|
||||
| [`bibclean`](bibclean.md) | Catalogue records audited and repaired, with a reason for every change |
|
||||
|
||||
### Turning documents into passages
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`mdscrub`](mdscrub.md) | Converted markdown in, clean markdown out |
|
||||
| [`chunkmd`](chunkmd.md) | Markdown into passages that carry their page, chapter and role |
|
||||
|
||||
### Turning recordings into passages
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`cuenorm`](cuenorm.md) | Any subtitle format in, one normalised cue stream out |
|
||||
| [`cuepack`](cuepack.md) | Cues into passages worth embedding, keeping the times they were said at |
|
||||
| [`cueseg`](cueseg.md) | A timed transcript into topic chapters |
|
||||
|
||||
### Searching
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`emb`](emb.md) | Lines into vectors, against an embedding server |
|
||||
| [`rrf`](rrf.md) | Several ranked lists into one, without the database |
|
||||
|
||||
## The libraries underneath
|
||||
|
||||
Every tool is a library crate with a thin command on top, so anything a
|
||||
command does can be done in process instead.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| [`corpus-core`](corpus-core.md) | The record types and the stream plumbing everything shares |
|
||||
| [`textkit`](textkit.md) | Token counting, sentence and paragraph splitting |
|
||||
| [`cuekit`](cuekit.md) | The cue model, and the three subtitle formats it is read from |
|
||||
| [`catena-cli`](catena-cli.md) | The parts of a filter every command does the same way |
|
||||
|
||||
## Reference
|
||||
|
||||
- [The records](records.md) — the shapes that travel between the tools.
|
||||
- [The contract](contract.md) — what every one of these programs promises.
|
||||
|
||||
## A worked pipeline
|
||||
|
||||
A book, from a converted chapter to passages an index can hold:
|
||||
|
||||
```
|
||||
$ mdscrub chapter.md | chunkmd --min-tokens 20
|
||||
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
|
||||
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
|
||||
```
|
||||
|
||||
A recording, through the same shape:
|
||||
|
||||
```
|
||||
$ cuenorm episode.vtt | cuepack --seconds 10
|
||||
{"v":1,"idx":0,"text":"The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest.","tokens":27,"page":null,"chapter":null,"role":"body","start_time":1.0,"end_time":8.0,"flags":[]}
|
||||
{"v":1,"idx":1,"text":"Its wheel was undershot, which was wasteful,","tokens":11,"page":null,"chapter":null,"role":"body","start_time":11.0,"end_time":19.0,"flags":[]}
|
||||
```
|
||||
|
||||
Both produce the same record, so one index answers across a book and a lecture
|
||||
and returns both with a location you can cite — a page number for one, a
|
||||
timestamp for the other.
|
||||
+54
-43
@@ -1,33 +1,41 @@
|
||||
# The contract
|
||||
# What these programs promise
|
||||
|
||||
Every program here obeys all of this, or it does not ship. It is worth reading
|
||||
once, because it is what lets these be used together without reading any of
|
||||
them.
|
||||
Eight things hold for every program here. Together they are what lets a person
|
||||
put any of them in a pipeline without first reading how it works.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../README.md).
|
||||
|
||||
## One job
|
||||
## Each one does a single job
|
||||
|
||||
A tool that needs a `--mode` flag to change what it *is* is two tools.
|
||||
No program here has a mode switch that changes what it is. Where two jobs
|
||||
would otherwise share a name, there are two programs — which is why
|
||||
[`cuenorm`](../crates/cuenorm/README.md), [`cuepack`](../crates/cuepack/README.md) and [`cueseg`](../crates/cueseg/README.md) are
|
||||
separate commands rather than one command with a flag.
|
||||
|
||||
## stdin to stdout
|
||||
## Input on stdin, results on stdout, diagnostics on stderr
|
||||
|
||||
Input on stdin or a path argument, result on stdout, diagnostics on stderr
|
||||
only — never content and chatter on the same stream. A tool that takes a path
|
||||
instead of a stream says why in its `--help`; [`pdftriage`](pdftriage.md) does,
|
||||
because it reads two specific page ranges rather than the whole file.
|
||||
Output and commentary never share a stream, so a result can always be piped
|
||||
somewhere without filtering anything out of it.
|
||||
|
||||
## Streams, not documents
|
||||
Almost all of them read stdin or a path. Where one takes a path only, it says
|
||||
so in `--help` and says why: [`pdftriage`](../crates/pdftriage/README.md) reads two specific
|
||||
page ranges rather than the whole file, which a stream cannot offer.
|
||||
|
||||
Anything list-shaped is NDJSON: one [record](records.md) per line, flushed as
|
||||
it is produced, so a slow stage never blocks the pipe behind it.
|
||||
## Anything list-shaped is NDJSON
|
||||
|
||||
The reader is strict on purpose. It refuses a record whose version it does not
|
||||
know, it stops at the first bad line rather than reading past a disagreement
|
||||
about the format, and it bounds how much one line may cost — a filter pointed
|
||||
at a binary file should fail, not allocate the file.
|
||||
One [record](records.md) per line, flushed as it is produced — so a slow stage
|
||||
downstream sees each record as it appears rather than when the producer
|
||||
finishes.
|
||||
|
||||
## Predicates answer with exit codes
|
||||
Readers are strict, and each strictness prevents a failure that would
|
||||
otherwise be silent. A record from a version the reader does not know is
|
||||
refused rather than parsed into whatever lines up. A bad line ends the stream
|
||||
instead of being skipped, because it means the two ends disagree about the
|
||||
format. Lines are bounded, so a program pointed at a binary file fails instead
|
||||
of allocating it. A byte-order mark and blank lines are ignored, so a stream
|
||||
that was concatenated or written on Windows still reads.
|
||||
|
||||
## Exit codes carry the answer
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
@@ -35,44 +43,47 @@ at a binary file should fail, not allocate the file.
|
||||
| `1` | No |
|
||||
| `2` | Could not tell |
|
||||
| `64` | The command line was wrong |
|
||||
| `65` | The input was not what the tool can work on |
|
||||
| `65` | The input was not what the program can work on |
|
||||
| `69` | A program it needs is not installed |
|
||||
| `70` | The tool's own fault |
|
||||
| `70` | Its own fault |
|
||||
| `74` | Reading or writing failed |
|
||||
|
||||
The failure codes start where `sysexits.h` puts them so that "no" and "broke"
|
||||
are never the same number. That is what makes this work:
|
||||
Failures start where `sysexits.h` puts them, so "no" and "broke" are never the
|
||||
same number. That is what makes this work:
|
||||
|
||||
```sh
|
||||
for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
|
||||
```
|
||||
|
||||
A consumer closing the pipe is not a failure. `… | head -1` exits `0`, like
|
||||
every other filter.
|
||||
A consumer closing the pipe is not a failure. `… | head -1` exits `0`, the
|
||||
same as any other filter.
|
||||
|
||||
## No network, no database, no config file, no state
|
||||
## Nothing reaches outside the process
|
||||
|
||||
No network, no database, no configuration file, no state kept between runs.
|
||||
Two exceptions, both declared in `--help`:
|
||||
|
||||
- [`emb`](emb.md), whose whole job is an HTTP call.
|
||||
- [`pdftriage`](pdftriage.md), which shells out to `pdftotext` and `pdfinfo`
|
||||
and exits `69` when either is missing.
|
||||
- [`emb`](../crates/emb/README.md) makes an HTTP call, which is the whole of what it does.
|
||||
- [`pdftriage`](../crates/pdftriage/README.md) runs `pdftotext` and `pdfinfo`, and exits `69`
|
||||
when either is missing rather than guessing.
|
||||
|
||||
## No ambient knowledge
|
||||
## Nothing is assumed about where it is running
|
||||
|
||||
No hostnames, no local paths, no private vocabularies compiled in. General
|
||||
scheme data — ISBN check digits, the HTML character references — may be;
|
||||
anything local is a `--rules` file.
|
||||
No hostnames, no paths, no private vocabularies are built in. General
|
||||
reference data is — ISBN check digits, the HTML character references, the
|
||||
names of well-known publishers — and anything specific to one collection is
|
||||
supplied in a file rather than compiled in.
|
||||
|
||||
## Deterministic
|
||||
## The same input gives the same output
|
||||
|
||||
Same bytes in, same bytes out, across runs and machines. No hash-order
|
||||
dependence, no clock, no locale reads. Floats that JSON cannot carry are
|
||||
refused where they are produced rather than written as `null` and lost.
|
||||
On every run and on every machine. No dependence on hash ordering, no reading
|
||||
of the clock or the locale. Numbers that JSON cannot represent are refused
|
||||
where they are produced rather than written as `null` and lost somewhere
|
||||
downstream.
|
||||
|
||||
## `--json` everywhere
|
||||
## `--json` is the stable form
|
||||
|
||||
The human-readable default is for a person at a terminal; `--json` is the
|
||||
contract, and it does not change shape without a major version. Tools whose
|
||||
only sensible output is a stream of records write that stream and have no
|
||||
`--json` to add.
|
||||
Where a program prints something for a person to read, `--json` prints the
|
||||
same answer as a record, and that record does not change shape without a major
|
||||
version. Programs whose only sensible output is a stream of records write that
|
||||
stream and have no `--json` to add.
|
||||
|
||||
+9
-9
@@ -4,10 +4,10 @@ Five shapes travel between these tools. Each carries a version field `v`, so a
|
||||
reader can refuse a stream it does not understand rather than parsing it into
|
||||
whatever happens to line up.
|
||||
|
||||
Every list-shaped stream is [NDJSON](contract.md#streams-not-documents): one
|
||||
Every list-shaped stream is [NDJSON](contract.md#anything-list-shaped-is-ndjson): one
|
||||
record per line, flushed as it is produced.
|
||||
|
||||
Back to [the index](README.md).
|
||||
Part of [catena](../README.md).
|
||||
|
||||
## `doc` — what a file is
|
||||
|
||||
@@ -31,7 +31,7 @@ Back to [the index](README.md).
|
||||
A tool fills what it can and leaves the rest null: a format sniffer fills
|
||||
`kind` and leaves `text_layer` empty, a triage tool does the reverse.
|
||||
|
||||
Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md).
|
||||
Written by [`booksniff`](../crates/booksniff/README.md) and [`pdftriage`](../crates/pdftriage/README.md).
|
||||
|
||||
## `chunk` — a retrievable passage
|
||||
|
||||
@@ -55,7 +55,7 @@ Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md).
|
||||
recordings. One shape either way, which is what lets a single index serve a
|
||||
book and a lecture and cite both.
|
||||
|
||||
Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md).
|
||||
Written by [`chunkmd`](../crates/chunkmd/README.md) and [`cuepack`](../crates/cuepack/README.md).
|
||||
|
||||
## `hit` — one scored candidate
|
||||
|
||||
@@ -74,7 +74,7 @@ Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md).
|
||||
`rank` is what fusion consumes. `score` is carried along because it is useful
|
||||
to a person reading a trace and useless for comparing across channels.
|
||||
|
||||
Read and written by [`rrf`](rrf.md).
|
||||
Read and written by [`rrf`](../crates/rrf/README.md).
|
||||
|
||||
## `cue` — one line of a transcript
|
||||
|
||||
@@ -85,8 +85,8 @@ Read and written by [`rrf`](rrf.md).
|
||||
Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few
|
||||
seconds and often half a sentence.
|
||||
|
||||
Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and
|
||||
[`cueseg`](cueseg.md).
|
||||
Written by [`cuenorm`](../crates/cuenorm/README.md), read by [`cuepack`](../crates/cuepack/README.md) and
|
||||
[`cueseg`](../crates/cueseg/README.md).
|
||||
|
||||
## `segment` — one topic chapter
|
||||
|
||||
@@ -98,7 +98,7 @@ Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and
|
||||
The marks a scrubber needs, and the spans a summariser should be given one at
|
||||
a time.
|
||||
|
||||
Written by [`cueseg`](cueseg.md).
|
||||
Written by [`cueseg`](../crates/cueseg/README.md).
|
||||
|
||||
## Getting the schemas
|
||||
|
||||
@@ -113,4 +113,4 @@ $ cargo run -p corpus-core -- schema chunk
|
||||
}
|
||||
```
|
||||
|
||||
See [`corpus-core`](corpus-core.md).
|
||||
See [`corpus-core`](../crates/corpus-core/README.md).
|
||||
|
||||
Reference in New Issue
Block a user