funkpower
99cc059310
An arrow pointing back to where the note was called, inside escaped brackets, sometimes as a link and sometimes not. Neither pattern knew it, so a page of notes set this way read as body prose: the chunk was never flagged, the citation stage never selected it, and the book came out with nothing. Ten books of the corpus use it and eight of them had zero citations — including both volumes of *The Decline of the West*, whose 830 endnotes are exactly what this is for. 142 of the 199 affected chunks are those two books. Recorded in parity/KNOWN.md: the Python this was ported from did not know the marker either, and no fixture carries one, so the suite agrees either way.
parity
Differential tests against the Python implementations these tools were ported from.
A reimplementation that is 98% right is worse than no reimplementation, because the missing 2% is silent: a chunk boundary moves, a verdict flips on one class of file, and nothing fails — the index is just quietly worse. These tests exist to make that loud.
How it works
Each ported function has a golden file under golden/, holding what the
Python produced for a fixture. The default test run compares Rust against
those files and needs nothing but Rust:
cargo test -p parity
With an interpreter that has the Python dependencies installed, the oracles can be re-run live, which catches a golden file drifting away from a Python implementation that has moved on:
CATENA_PARITY_PYTHON=/path/to/venv/bin/python \
CATENA_PARITY_PYTHONPATH=/path/to/book-pipeline/src:/path/to/podcast-pipeline/src \
cargo test -p parity --features python
SOURCE.md records which commit of that source the golden files came from.
Pointing the oracles at an older checkout will fail, and should.
Regenerating
Golden files are never regenerated by a test — a golden file that rewrites itself when it fails is a test that cannot fail. Regenerate deliberately, read the diff, and commit it with the reason:
CATENA_PARITY_PYTHON=/path/to/venv/bin/python \
CATENA_PARITY_PYTHONPATH=/path/to/book-pipeline/src:/path/to/podcast-pipeline/src \
cargo run -p parity --bin regen
Fixtures
Generated, synthetic, or public domain — nothing else. src/sample.rs
generates its fixture deterministically from a fixed seed, so both sides
demonstrably read the same bytes and the clone stays small.
Where a heuristic was tuned against copyrighted text, commit the derived signal and not the text: the page character counts, the ASCII scores, the word-shape tuples, the cue timings. A parity test over a text-layer classifier needs the score inputs, not the book.
Divergences
Where Rust and Python legitimately differ, record it in KNOWN.md with the
reason. Do not loosen a test to accommodate one.