parity harness, pinned to the python tokeniser over 10k lines

parity/README.md

@@ -0,0 +1,53 @@
+# parity
+
+Differential tests against the Python implementations these tools were ported
+from.
+
+A reimplementation that is 98% right is worse than no reimplementation,
+because the missing 2% is silent: a chunk boundary moves, a verdict flips on
+one class of file, and nothing fails — the index is just quietly worse. These
+tests exist to make that loud.
+
+## How it works
+
+Each ported function has a **golden file** under `golden/`, holding what the
+Python produced for a fixture. The default test run compares Rust against
+those files and needs nothing but Rust:
+
+```sh
+cargo test -p parity
+```
+
+With an interpreter that has the Python dependencies installed, the oracles
+can be re-run live, which catches a golden file drifting away from a Python
+implementation that has moved on:
+
+```sh
+CATENA_PARITY_PYTHON=/path/to/venv/bin/python cargo test -p parity --features python
+```
+
+## Regenerating
+
+Golden files are never regenerated by a test — a golden file that rewrites
+itself when it fails is a test that cannot fail. Regenerate deliberately,
+read the diff, and commit it with the reason:
+
+```sh
+CATENA_PARITY_PYTHON=/path/to/venv/bin/python cargo run -p parity --bin regen
+```
+
+## Fixtures
+
+Generated, synthetic, or public domain — nothing else. `src/sample.rs`
+generates its fixture deterministically from a fixed seed, so both sides
+demonstrably read the same bytes and the clone stays small.
+
+Where a heuristic was tuned against copyrighted text, commit the **derived
+signal** and not the text: the page character counts, the ASCII scores, the
+word-shape tuples, the cue timings. A parity test over a text-layer classifier
+needs the score inputs, not the book.
+
+## Divergences
+
+Where Rust and Python legitimately differ, record it in `KNOWN.md` with the
+reason. Do not loosen a test to accommodate one.