# catena

Small programs that each do one part of turning a pile of files into a
searchable corpus, and hand the next one text on stdout.

A catena is a commentary assembled out of excerpts drawn from many sources and
chained in order, each one carrying the name of who said it and where. That is
both what these programs produce and how they run.

Nothing here needs a database, a GPU, a queue or a config file. Each tool is a
filter: bytes in, records out, diagnostics on stderr, and an exit code that
means something.

## Install

```sh
cargo install --git https://git.pumpin.online/funkpower/catena corpus-core
```

Each tool installs by name, so you can take one without taking the rest.

## The contract

Every binary in this workspace obeys all of this, or it does not ship.

1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
   tools.
2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
   diagnostics on stderr only — never content and chatter on the same stream.
3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
   line, flushed as it is produced, so a slow stage never blocks the pipe.
4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell.
   Failures use the `sysexits` range — `64` bad usage, `65` bad input, `69` a
   missing external program, `74` I/O — so "no" and "broke" are never the same
   number.
5. **No network, no database, no config file, no state.** Two exceptions, both
   declared in `--help`: a tool whose whole job is an HTTP call, and a tool
   that shells out to a named external program and fails clearly when it is
   absent.
6. **No ambient knowledge.** No hostnames, no local paths, no private
   vocabularies compiled in. General scheme data — Bible book names, ISBN check
   digits — may be; anything local is a `--rules file.json`.
7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
8. **`--json` everywhere.** The human-readable default is for a person at a
   terminal; `--json` is the contract, and it does not change shape without a
   major version.

## The records

Three shapes travel between the tools, defined in `corpus-core` and versioned
by a `v` field.

**`doc`** — what a file is.

```json
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
 "pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
```

**`chunk`** — a retrievable passage with enough provenance to cite it.

```json
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
 "role":"body","start_time":null,"end_time":null,"flags":[]}
```

`page` and `chapter` for documents, `start_time` and `end_time` for
recordings. One shape either way, so a single index serves a book and a
lecture and returns both with a location you can cite.

**`hit`** — one scored candidate, on its way into a fusion.

```json
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
```

The JSON Schema for all three:

```sh
cargo run -p corpus-core -- schema all
```

## Build

```sh
cargo test
cargo clippy --all-targets -- -D warnings
cargo fmt --check
```

## Licence

MPL-2.0. See `LICENSE`.