funkpower/catena
1
0
Fork 0
mirror of https://git.pumpin.online/funkpower/catena.git synced 2026-10-22 18:57:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: a page for every tool and every library, with real output

Browse Source
This commit is contained in:
funkpower
2026-09-19 13:09:20 -04:00
parent b0743d5bf7
commit 2e2988b1b1
20 changed files with 2162 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+38 -66
View File
@@ -19,20 +19,38 @@ cargo install --git https://git.pumpin.online/funkpower/catena <tool>
Each tool installs by name, so you can take one without taking the rest.
## The tools
| | |
|---|---|
| [`booksniff`](docs/booksniff.md) | What a file actually is, whatever its name claims |
| [`pdftriage`](docs/pdftriage.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
| [`bibclean`](docs/bibclean.md) | Catalogue records audited and repaired, with a reason for every change |
| [`mdscrub`](docs/mdscrub.md) | Converted markdown in, clean markdown out |
| [`chunkmd`](docs/chunkmd.md) | Markdown into passages that carry their page, chapter and role |
| [`cuenorm`](docs/cuenorm.md) | Any subtitle format in, one normalised cue stream out |
| [`cuepack`](docs/cuepack.md) | Cues into passages worth embedding, keeping the times they were said at |
| [`cueseg`](docs/cueseg.md) | A timed transcript into topic chapters |
| [`emb`](docs/emb.md) | Lines into vectors, against an embedding server |
| [`rrf`](docs/rrf.md) | Several ranked lists into one, without the database |
Full documentation, including the libraries underneath, is in
**[docs/](docs/README.md)**.
## A worked pipeline
Converted markdown in, retrievable passages out:
```
$ pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd --min-tokens 20
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly\nwhenever they were asked.4 A lord who could not\nprotect was a lord who could not command…","tokens":57,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested…","tokens":41,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly\nwhenever they were asked.4 A lord who could not\nprotect was a lord who could not command…","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
```
A recording instead of a book, with chapter marks a scrubber can use:
```
$ cuenorm episode.vtt | cueseg --duration 7245
$ cuenorm episode.vtt | cueseg --duration 7200
{"v":1,"start":0.0,"end":1190.6,"title":"The Mill Stood At The Northern End","excerpt":"the mill stood at the northern end of the valley where…"}
{"v":1,"start":1190.6,"end":2381.3,"title":"Parliament Dissolved Itself Again","excerpt":"parliament dissolved itself again amid furious objections…"}
```
@@ -49,91 +67,45 @@ garbled, needs OCR
$ for f in *.pdf; do pdftriage --quiet "$f" && ocrmypdf "$f" "ocr/$f"; done
```
## The tools
| | |
|---|---|
| `bibclean` | Catalogue records audited and repaired, with a reason for every change |
| `booksniff` | What a file actually is, whatever its name claims |
| `mdscrub` | Converted markdown in, clean markdown out: residual HTML, control characters, headings, page markers |
| `chunkmd` | Markdown into passages that carry their page, chapter and role |
| `pdftriage` | Whether a PDF's text layer is healthy, garbled, sparse, permuted, or absent — and so whether OCR is worth its cost |
| `cuenorm` | Any subtitle format in, one normalised cue stream out |
| `cueseg` | A timed transcript into topic chapters |
| `cuepack` | Cues into passages worth embedding, keeping the times they were said at |
| `emb` | Lines into vectors, against an embedding server |
| `rrf` | Several ranked lists into one, without the database |
## The contract
Every binary in this workspace obeys all of this, or it does not ship.
Every binary in this workspace obeys all of this, or it does not ship. The
whole of it is in [docs/contract.md](docs/contract.md).
1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
tools.
2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
diagnostics on stderr only — never content and chatter on the same stream.
diagnostics on stderr only.
3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
line, flushed as it is produced, so a slow stage never blocks the pipe.
4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell.
Failures use the `sysexits` range — `64` bad usage, `65` bad input, `69` a
missing external program, `70` the tool's own fault, `74` I/O — so "no" and
"broke" are never the same number. A consumer closing the pipe is not a
failure: `… | head -1` exits `0`, like every other filter.
5. **No network, no database, no config file, no state.** Two exceptions, both
declared in `--help`: `emb`, whose whole job is an HTTP call, and a tool
that shells out to a named external program and fails clearly when it is
absent — `pdftriage` needs `pdftotext`.
line, flushed as it is produced.
4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell;
failures use the `sysexits` range, so "no" and "broke" are never the same
number.
5. **No network, no database, no config file, no state**, with two declared
exceptions.
6. **No ambient knowledge.** No hostnames, no local paths, no private
vocabularies compiled in. General scheme data — Bible book names, ISBN check
digits — may be; anything local is a `--rules file.json`.
vocabularies compiled in.
7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
8. **`--json` everywhere.** The human-readable default is for a person at a
terminal; `--json` is the contract, and it does not change shape without a
major version.
terminal; `--json` is the contract.
## The records
Three shapes travel between the tools, defined in `corpus-core` and versioned
by a `v` field.
Three shapes travel between the tools, versioned, defined in `corpus-core` and
described in [docs/records.md](docs/records.md).
**`doc`** — what a file is.
```json
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
"pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
```
**`chunk`** — a retrievable passage with enough provenance to cite it.
```json
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
"role":"body","start_time":null,"end_time":null,"flags":[]}
```
`page` and `chapter` for documents, `start_time` and `end_time` for
recordings. One shape either way, so a single index serves a book and a
lecture and returns both with a location you can cite.
**`hit`** — one scored candidate, on its way into a fusion.
```json
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
```
The JSON Schema for all three:
```sh
cargo run -p corpus-core -- schema all
$ cargo run -p corpus-core -- schema all
```
## Build
```sh
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check
./check.sh
```
Formatting, lints, tests and documentation, which is the same gate CI runs.
## Licence
MPL-2.0, library crates and binaries alike. The notice lives in `LICENSE` and