mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 03:07:13 +00:00
94 lines
3.2 KiB
Markdown
94 lines
3.2 KiB
Markdown
# catena
|
|
|
|
Small programs that each do one part of turning a pile of files into a
|
|
searchable corpus, and hand the next one text on stdout.
|
|
|
|
A catena is a commentary assembled out of excerpts drawn from many sources and
|
|
chained in order, each one carrying the name of who said it and where. That is
|
|
both what these programs produce and how they run.
|
|
|
|
Nothing here needs a database, a GPU, a queue or a config file. Each tool is a
|
|
filter: bytes in, records out, diagnostics on stderr, and an exit code that
|
|
means something.
|
|
|
|
## Install
|
|
|
|
```sh
|
|
cargo install --git https://git.pumpin.online/funkpower/catena corpus-core
|
|
```
|
|
|
|
Each tool installs by name, so you can take one without taking the rest.
|
|
|
|
## The contract
|
|
|
|
Every binary in this workspace obeys all of this, or it does not ship.
|
|
|
|
1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
|
|
tools.
|
|
2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
|
|
diagnostics on stderr only — never content and chatter on the same stream.
|
|
3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
|
|
line, flushed as it is produced, so a slow stage never blocks the pipe.
|
|
4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell.
|
|
Failures use the `sysexits` range — `64` bad usage, `65` bad input, `69` a
|
|
missing external program, `74` I/O — so "no" and "broke" are never the same
|
|
number.
|
|
5. **No network, no database, no config file, no state.** Two exceptions, both
|
|
declared in `--help`: a tool whose whole job is an HTTP call, and a tool
|
|
that shells out to a named external program and fails clearly when it is
|
|
absent.
|
|
6. **No ambient knowledge.** No hostnames, no local paths, no private
|
|
vocabularies compiled in. General scheme data — Bible book names, ISBN check
|
|
digits — may be; anything local is a `--rules file.json`.
|
|
7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
|
|
8. **`--json` everywhere.** The human-readable default is for a person at a
|
|
terminal; `--json` is the contract, and it does not change shape without a
|
|
major version.
|
|
|
|
## The records
|
|
|
|
Three shapes travel between the tools, defined in `corpus-core` and versioned
|
|
by a `v` field.
|
|
|
|
**`doc`** — what a file is.
|
|
|
|
```json
|
|
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
|
|
"pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
|
|
```
|
|
|
|
**`chunk`** — a retrievable passage with enough provenance to cite it.
|
|
|
|
```json
|
|
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
|
|
"role":"body","start_time":null,"end_time":null,"flags":[]}
|
|
```
|
|
|
|
`page` and `chapter` for documents, `start_time` and `end_time` for
|
|
recordings. One shape either way, so a single index serves a book and a
|
|
lecture and returns both with a location you can cite.
|
|
|
|
**`hit`** — one scored candidate, on its way into a fusion.
|
|
|
|
```json
|
|
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
|
|
```
|
|
|
|
The JSON Schema for all three:
|
|
|
|
```sh
|
|
cargo run -p corpus-core -- schema all
|
|
```
|
|
|
|
## Build
|
|
|
|
```sh
|
|
cargo test
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo fmt --check
|
|
```
|
|
|
|
## Licence
|
|
|
|
MPL-2.0. See `LICENSE`.
|