catena/crates/chunkmd/README.md

chunkmd

Markdown into passages that carry their page, chapter and role.

Part of catena.

What it does

Cuts a document into pieces of about the size an embedding model wants, and puts on each one the page it came from, the chapter it is in, and what part of the work it belongs to.

Writes chunk records, one per line.

When you would use it

• Before embedding anything. A whole chapter is too big to embed usefully and a sentence is too small to mean anything; this is the step in between.
• When a result has to be citable. A passage a reader cannot locate in the book is a passage they cannot check, so the page and chapter are carried from the first stage rather than reconstructed later.
• To keep the apparatus out of search results. An index, a table of contents and a bibliography all match a query beautifully and answer nothing — they are lists of the words somebody is searching for. Marking them lets a search exclude them while keeping them addressable for the consumers that want them.

Install

cargo install --git https://git.pumpin.online/funkpower/catena chunkmd

Example

$ mdscrub chapter.md | chunkmd --min-tokens 20
{"v":1,"idx":0,"text":"The obligation ran both ways, and contemporaries said so plainly\nwhenever they were asked.4 A lord who could not\nprotect was a lord who could not command, and the men who owed him service\nsaid as much in language that has survived in the court rolls.","tokens":56,"page":41,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"What the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested…","tokens":143,"page":42,"chapter":"III. The Second Estate","role":"body","start_time":null,"end_time":null,"flags":[]}

Just the text, for an embedder:

mdscrub chapter.md | chunkmd | jq -r .text | emb --model bge-m3 --dim 1024

Only the body, leaving the bibliography and the index behind:

chunkmd book.md | jq -c 'select(.role == "body")'

Options

Option	Default	Effect
--target <TOKENS>	512	Flush a chunk at this many tokens
--overlap <TOKENS>	64	Carry this much of each chunk into the next
--min-tokens <TOKENS>	50	Drop chunks smaller than this
--hard-max <TOKENS>	768	Never emit a chunk larger than this
--man	Write this page as a manual page, in roff

The defaults suit a model with a 512-token comfortable window. A larger model wants roughly double everything:

chunkmd --target 1024 --overlap 128 --min-tokens 80 --hard-max 1280

Budgets that cannot produce anything sensible — an overlap at or above the target, a hard maximum below the target — are refused before any input is read.

Exit codes

Code	Meaning
0	Done
64	The command line was wrong
65	The input was not text
74	The file could not be read

How it cuts

Page first, paragraph second, sentence only when a paragraph is itself too big.

1. Split on page markers, so a chunk never spans a page and its page number is exact rather than inferred. Text before the first marker is kept, with no page number — it is still the book.
2. Pack paragraphs toward the token target within each page, carrying an overlap forward so a sentence that answers a question is not orphaned from the sentence that asked it.
3. Enforce the hard maximum on token boundaries, whatever the structure said.
4. Drop what is too small to be worth an embedding.
5. Classify roles across the whole document at once, because deciding a stray chunk needs to see its neighbours: a short run of body chunks sandwiched between two bibliography chunks is part of the bibliography.

A chunk whose text is a page of numbered notes is flagged footnote_citations, because footnotes are citations in disguise — they read as body text, rank as body text, and answer a query with a publisher and a page number.

A run of entries with nothing numbering them counts too, because a reference list often arrives with its markup gone: three or more lines that open on an inverted name and each carry a year, a page range or a publisher. The opening may be a bullet — a converted bibliography is frequently a bulleted list, and the bullet sits exactly where the surname would.

The roles

Role	What it is
body	Running text
bibliography	A works-cited or references list
index	A back-of-book index
toc	A table of contents
frontmatter	Title page, copyright page, dedication

As a library

use chunkmd::Budgets;

for chunk in chunkmd::chunk(&markdown, &Budgets::default()) {
    println!("{} tokens on page {:?}", chunk.tokens, chunk.page);
}

See also

• mdscrub — what to run before this.
• mdsplit — when the file holds more than one work.
• cuepack — the same record, from a recording.
• emb — turning the text into vectors.
• The records — the shape this writes.