workspace, licence, and the three record types

README.md

@@ -0,0 +1,93 @@
+# catena
+
+Small programs that each do one part of turning a pile of files into a
+searchable corpus, and hand the next one text on stdout.
+
+A catena is a commentary assembled out of excerpts drawn from many sources and
+chained in order, each one carrying the name of who said it and where. That is
+both what these programs produce and how they run.
+
+Nothing here needs a database, a GPU, a queue or a config file. Each tool is a
+filter: bytes in, records out, diagnostics on stderr, and an exit code that
+means something.
+
+## Install
+
+```sh
+cargo install --git https://git.pumpin.online/funkpower/catena corpus-core
+```
+
+Each tool installs by name, so you can take one without taking the rest.
+
+## The contract
+
+Every binary in this workspace obeys all of this, or it does not ship.
+
+1. **One job.** A tool that needs a `--mode` flag to change what it *is* is two
+   tools.
+2. **stdin to stdout.** Input on stdin or a path argument, result on stdout,
+   diagnostics on stderr only — never content and chatter on the same stream.
+3. **Streams, not documents.** Anything list-shaped is NDJSON: one record per
+   line, flushed as it is produced, so a slow stage never blocks the pipe.
+4. **Predicates answer with exit codes.** `0` yes, `1` no, `2` could not tell.
+   Failures use the `sysexits` range — `64` bad usage, `65` bad input, `69` a
+   missing external program, `74` I/O — so "no" and "broke" are never the same
+   number.
+5. **No network, no database, no config file, no state.** Two exceptions, both
+   declared in `--help`: a tool whose whole job is an HTTP call, and a tool
+   that shells out to a named external program and fails clearly when it is
+   absent.
+6. **No ambient knowledge.** No hostnames, no local paths, no private
+   vocabularies compiled in. General scheme data — Bible book names, ISBN check
+   digits — may be; anything local is a `--rules file.json`.
+7. **Deterministic.** Same bytes in, same bytes out, across runs and machines.
+8. **`--json` everywhere.** The human-readable default is for a person at a
+   terminal; `--json` is the contract, and it does not change shape without a
+   major version.
+
+## The records
+
+Three shapes travel between the tools, defined in `corpus-core` and versioned
+by a `v` field.
+
+**`doc`** — what a file is.
+
+```json
+{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
+ "pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
+```
+
+**`chunk`** — a retrievable passage with enough provenance to cite it.
+
+```json
+{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
+ "role":"body","start_time":null,"end_time":null,"flags":[]}
+```
+
+`page` and `chapter` for documents, `start_time` and `end_time` for
+recordings. One shape either way, so a single index serves a book and a
+lecture and returns both with a location you can cite.
+
+**`hit`** — one scored candidate, on its way into a fusion.
+
+```json
+{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
+```
+
+The JSON Schema for all three:
+
+```sh
+cargo run -p corpus-core -- schema all
+```
+
+## Build
+
+```sh
+cargo test
+cargo clippy --all-targets -- -D warnings
+cargo fmt --check
+```
+
+## Licence
+
+MPL-2.0. See `LICENSE`.