2.7 KiB
Known divergences
Places where the Rust and the Python deliberately do not agree, each with the reason it is acceptable. A divergence that is not written down here is a bug, and loosening a test instead of adding an entry here is how a port rots.
Format:
## <function> — <one-line summary>
**Python:** what it does.
**Rust:** what it does instead.
**Why this is acceptable:** the argument, including what input class is
affected and how often it occurs in practice.
**Measured:** how many records out of how many differ.
mdscrub — un-paginated text is not given a page 1 marker
Python: _normalize_marker_pages ends by wrapping any text it found no
pagination in as <!-- Page 1 -->\n\n…, so that downstream can assume a
marker exists.
Rust: text with no page separators and no form feeds is returned unchanged.
Why this is acceptable: a cleanup filter has to be safe to run twice, and the Python's version is not — a second pass wraps the first pass's marker in another one. The assumption it exists to serve belongs to the converter that knows it just read a PDF, not to a filter that is handed markdown of unknown provenance. The chunker already handles a document with no page markers by emitting chunks with no page number, which is the honest answer.
Measured: not exercised by any parity case, since the Python function is
only reachable from the PDF conversion path. Covered by
mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was.
chunkmd, role and footnote patterns — \s excludes four control characters
Python: \s in a str pattern matches the Unicode White_Space
property and U+001C–U+001F, the four ASCII information separators.
Rust: regex and fancy-regex match the White_Space property only, so
those four characters are not whitespace to a ported pattern.
Why this is acceptable: every one of those codepoints is deleted by
mdscrub before a chunker sees it — they fall inside the control-character
class that a text column rejects. For the pattern to behave differently, a
document would have to reach chunkmd without having been cleaned, and carry
an information separator in exactly the position where a pattern expects
whitespace.
Note that this does not apply to the sentence and paragraph splitters in
textkit, which match Python's definition exactly — see
textkit::segment::is_space. Those decide chunk boundaries, so the four
characters are handled there rather than argued about.
Measured: zero differences over the parity corpus; the sample in
parity/src/sample.rs includes U+001C deliberately and the token counts
agree over all ten thousand lines.