catena/parity/KNOWN.md

# Known divergences

Places where the Rust and the Python deliberately do not agree, each with the
reason it is acceptable. A divergence that is not written down here is a bug,
and loosening a test instead of adding an entry here is how a port rots.

Format:

```
## <function> — <one-line summary>

**Python:** what it does.
**Rust:** what it does instead.
**Why this is acceptable:** the argument, including what input class is
affected and how often it occurs in practice.
**Measured:** how many records out of how many differ.
```

---

## mdscrub — un-paginated text is not given a page 1 marker

**Python:** `_normalize_marker_pages` ends by wrapping any text it found no
pagination in as `<!-- Page 1 -->\n\n…`, so that downstream can assume a
marker exists.

**Rust:** text with no page separators and no form feeds is returned
unchanged.

**Why this is acceptable:** a cleanup filter has to be safe to run twice, and
the Python's version is not — a second pass wraps the first pass's marker in
another one. The assumption it exists to serve belongs to the converter that
knows it just read a PDF, not to a filter that is handed markdown of unknown
provenance. The chunker already handles a document with no page markers by
emitting chunks with no page number, which is the honest answer.

**Measured:** not exercised by any parity case, since the Python function is
only reachable from the PDF conversion path. Covered by
`mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was`.

## chunkmd, role and footnote patterns — `\s` excludes four control characters

**Python:** `\s` in a `str` pattern matches the Unicode `White_Space`
property *and* `U+001C`–`U+001F`, the four ASCII information separators.

**Rust:** `regex` and `fancy-regex` match the `White_Space` property only, so
those four characters are not whitespace to a ported pattern.

**Why this is acceptable:** every one of those codepoints is deleted by
`mdscrub` before a chunker sees it — they fall inside the control-character
class that a text column rejects. For the pattern to behave differently, a
document would have to reach `chunkmd` without having been cleaned, and carry
an information separator in exactly the position where a pattern expects
whitespace.

Note that this does **not** apply to the sentence and paragraph splitters in
`textkit`, which match Python's definition exactly — see
`textkit::segment::is_space`. Those decide chunk boundaries, so the four
characters are handled there rather than argued about.

**Measured:** zero differences over the parity corpus; the sample in
`parity/src/sample.rs` includes `U+001C` deliberately and the token counts
agree over all ten thousand lines.