catena (bindings)
The catena libraries behind one surface, for calling in process rather than as a subprocess.
Part of catena.
What it does
Every tool in this workspace is a library with a thin command on top. A pipeline written in another language can have either.
A subprocess is the right answer most of the time: it is a real process boundary, it composes with a shell, and it costs a few milliseconds. In process is the right answer when the caller is already holding the text — a stage that would otherwise write a file so that it can read it straight back.
What comes back is the same record shapes the commands write to stdout, as dictionaries rather than lines of JSON. Those shapes are documented and versioned; the structs behind them are free to move.
Python
pip install catena
import catena
chunks = catena.chunk_markdown(markdown, target=512, overlap=64)
chunks[0]["page"] # 41
chunks[0]["chapter"] # 'III. The Second Estate'
clean = catena.scrub(raw_markdown, pages=True)
catena.count_tokens(clean)
cues = catena.parse_subtitles(open("episode.srt").read())
passages = catena.pack_cues(cues, seconds=75)
catena.identify("atlas.pdf", open("atlas.pdf", "rb").read(1024))["kind"]
split_collection raises ValueError when the document is one work rather
than a collection — the usual answer, with the reason each candidate was
rejected:
try:
works = catena.split_collection(markdown)
except ValueError as why:
works = [{"idx": 0, "title": title, "text": markdown}]
What is bound
| Function | The tool it is |
|---|---|
chunk_markdown |
chunkmd |
scrub |
mdscrub |
count_tokens |
textkit |
parse_subtitles |
cuenorm |
pack_cues |
cuepack |
split_collection |
mdsplit |
fuse_hits |
rrf |
identify |
booksniff |
The tools that are not here are the ones a binding would gain nothing over a subprocess for: they read files, shell out to another program, or talk to a server, and none of that is cheaper in process.
Building the wheel
cd crates/bindings && maturin build --release
cargo build --features python does not work and is not supposed to: pyo3's
extension-module leaves the Python symbols to be resolved by the interpreter
at load time, so there is nothing to link against until an interpreter loads
it. cargo clippy --features python type-checks it without linking, which is
what the gate runs.
As a Rust library
The same surface, without a binding in the way:
let chunks = catena::chunk_markdown(&markdown, catena::ChunkBudgets::default());
Nothing here grows logic of its own. If a binding needs something the libraries cannot do, that is a missing library function rather than a binding feature.
See also
- The records — the shapes these hand back.
- Recipes — the same pipelines from a shell.