mdscrub: residual html, control characters, headings, page markers

parity/KNOWN.md

@@ -18,4 +18,65 @@ affected and how often it occurs in practice.
 
 ---
 
-None recorded yet.
+## mdscrub — a line-break tag becomes a line break, not nothing
+
+**Python:** `strip_residual_html` removes every inline tag, `<br>` included,
+so `the<br>northern end` becomes `thenorthern end`.
+
+**Rust:** `<br>`, `<br/>` and `<br />` become a newline; every other tag is
+still removed.
+
+**Why this is acceptable:** the welded word is not a formatting difference,
+it is a word that does not exist. It cannot be searched for, it embeds as a
+token sequence no query will produce, and it is invisible in a result because
+the surrounding sentence reads normally. The divergence only affects
+documents that use `<br>` at all — mostly verse, addresses and tables of
+contents converted out of HTML — and within those, only the joins.
+
+**Measured:** `parity/cases/mdscrub/breaks.md`; both answers are committed as
+`golden/mdscrub/breaks.md` and `golden/mdscrub/breaks.rust.md`. Three welds
+in eight lines.
+
+## mdscrub — un-paginated text is not given a page 1 marker
+
+**Python:** `_normalize_marker_pages` ends by wrapping any text it found no
+pagination in as `<!-- Page 1 -->\n\n…`, so that downstream can assume a
+marker exists.
+
+**Rust:** text with no page separators and no form feeds is returned
+unchanged.
+
+**Why this is acceptable:** a cleanup filter has to be safe to run twice, and
+the Python's version is not — a second pass wraps the first pass's marker in
+another one. The assumption it exists to serve belongs to the converter that
+knows it just read a PDF, not to a filter that is handed markdown of unknown
+provenance. The chunker already handles a document with no page markers by
+emitting chunks with no page number, which is the honest answer.
+
+**Measured:** not exercised by any parity case, since the Python function is
+only reachable from the PDF conversion path. Covered by
+`mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was`.
+
+## chunkmd, role and footnote patterns — `\s` excludes four control characters
+
+**Python:** `\s` in a `str` pattern matches the Unicode `White_Space`
+property *and* `U+001C`–`U+001F`, the four ASCII information separators.
+
+**Rust:** `regex` and `fancy-regex` match the `White_Space` property only, so
+those four characters are not whitespace to a ported pattern.
+
+**Why this is acceptable:** every one of those codepoints is deleted by
+`mdscrub` before a chunker sees it — they fall inside the control-character
+class that a text column rejects. For the pattern to behave differently, a
+document would have to reach `chunkmd` without having been cleaned, and carry
+an information separator in exactly the position where a pattern expects
+whitespace.
+
+Note that this does **not** apply to the sentence and paragraph splitters in
+`textkit`, which match Python's definition exactly — see
+`textkit::segment::is_space`. Those decide chunk boundaries, so the four
+characters are handled there rather than argued about.
+
+**Measured:** zero differences over the parity corpus; the sample in
+`parity/src/sample.rs` includes `U+001C` deliberately and the token counts
+agree over all ten thousand lines.