catena/parity/KNOWN.md

Known divergences

Places where the Rust and the Python deliberately do not agree, each with the reason it is acceptable. A divergence that is not written down here is a bug, and loosening a test instead of adding an entry here is how a port rots.

Format:

## <function> — <one-line summary>

**Python:** what it does.
**Rust:** what it does instead.
**Why this is acceptable:** the argument, including what input class is
affected and how often it occurs in practice.
**Measured:** how many records out of how many differ.

---

cuenorm — a cue line with trailing settings still reads

Original: the timestamp line is split on the arrow and each half is read whole, so 00:00:01,000 --> 00:00:03,000 X1:040 X2:560 gives an end time that is not a number and the cue is dropped entirely.

Rust: each half is reduced to the one token that could be a timestamp, so the trailing text is ignored and the cue reads.

Why this is acceptable: it is what makes the same reader take WebVTT, where positioning settings after the end time are not a malformation but the format — 00:00:01.000 --> 00:00:03.000 line:0 position:20% is an ordinary cue. Without it the tool would need a second parser that differed in one line. The divergence can only ever recover a cue that the original discarded; there is no input where it changes a cue the original read.

Measured: parity/cases/cuenorm/coordinates.srt — one cue of two recovered. Across the other eight cases, including deliberately malformed timestamps, the two agree cue for cue.

pdftriage — a newer Unicode table knows letters an older one does not

Python: str.isalpha is true for the letter categories of the Unicode version the interpreter was built against.

Rust: the same test against a table built from a later Unicode version, which knows 4,924 code points the older one does not.

Why this is acceptable: the newer answer is the correct one, and the difference is entirely one-directional — every code point the two disagree about is a letter that Rust knows and Python does not, never the reverse. It matters because the letter share is what calls a text layer dead: a book typeset in Cyrillic Extended-C is judged garbled by the older table, which sends a perfectly readable text layer for an hour of OCR that can only make it worse. Erring toward not re-OCRing a book whose text is fine is the right direction for this gate to err in.

Both tables will keep moving, which is the other reason this entry exists: the golden files pin one pair of Unicode versions, and a change to either shows up as a parity failure rather than as a silently different verdict.

Measured: parity/cases/pdftriage/samples.ndjson, case cyrillic-extended-c: letter share 1.00 here against 0.00 there, which is the difference between healthy and garbled. Brute-forced over the whole code space: 4,924 disagreements, none of them in the other direction.

mdscrub — un-paginated text is not given a page 1 marker

Python: _normalize_marker_pages ends by wrapping any text it found no pagination in as <!-- Page 1 -->\n\n…, so that downstream can assume a marker exists.

Rust: page boundaries are not touched at all unless --pages asks for them, and even then text with no separators and no form feeds is returned unchanged.

Why this is acceptable: a cleanup filter has to be safe to run twice, and the Python's version is not — a second pass wraps the first pass's marker in another one. The assumption it exists to serve belongs to the converter that knows it just read a PDF, not to a filter that is handed markdown of unknown provenance: a form feed in a code listing is a control character, and a dash rule in a quoted document is not a page break. The chunker already handles a document with no page markers by emitting chunks with no page number, which is the honest answer.

Separating the two passes is also what makes the cleanup idempotent. Removing a tag can leave a dash rule flush against a newline, and a pagination pass running afterwards would find a page boundary on the second run that was not there on the first.

Measured: not exercised by any parity case, since the Python function is only reachable from the PDF conversion path. Covered by mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was and mdscrub::tests::nothing_manufactures_a_page_boundary_on_a_second_run.

Every ported pattern — the two engines disagree about \d and (?i)

Python: re compiles against the Unicode version of the interpreter; (?i) folds ı and İ onto i.

Rust: regex compiles against the Unicode version it was built with, and (?i) uses simple case folding, which leaves the dotless and dotted i alone.

Why this is acceptable: \s, \w and \b were the differences that mattered and they are closed — every ported pattern now spells Python's classes explicitly, from textkit::pattern. What is left is a \d that disagrees only on decimal digits added between the two Unicode versions (Garay, added in Unicode 16, is the current example), and a case fold that disagrees only for Turkish dotless i. A chapter heading reading Bıblıography is classified as body text here and as a bibliography there; a page marker written in Garay digits reads here and does not there.

Both will drift again at every dependency bump, which is the real reason this entry exists: the golden files pin one pair of Unicode versions, and a change to either shows up as a parity failure rather than as a silent difference.

Measured: zero differences over the parity corpus. Found by brute-forcing every code point below U+11000 through both engines: \d differs on ten, (?i) on three.

chunkmd — a page number too large for the field saturates

Python: integers are unbounded, so <!-- Page 99999999999 --> records page 99,999,999,999.

Rust: the page field is a 32-bit number and saturates at 4,294,967,295.

Why this is acceptable: the number is a page number. A document with four billion pages does not exist, and the alternative — a 64-bit field — would make every chunk record wider to carry a value nothing can produce. The saturation is deliberate rather than a wrap, so the wrong answer is at least obviously wrong.

Note that this applies only where a page number becomes a number. Where one is copied into text — a page anchor in the body of a document — it is carried through exactly as it was written, padding and all.

Measured: no document in any corpus this was run against has a page marker above five digits.

chunkmd — a hard maximum of zero yields nothing rather than raising

Python: _enforce_hard_max with hard_max=0 raises ValueError: range() arg 3 must not be zero.

Rust: chunkmd::chunk returns no chunks, since no piece could satisfy the limit.

Why this is acceptable: the command-line tool rejects it before reading any input, so this is reachable only by a library caller, and a filter that returns nothing for a budget that permits nothing is the more useful answer of the two. Every other degenerate budget — overlap at or above target, a hard maximum below the target, a minimum of zero — agrees with the Python exactly.

Measured: checked against the oracle for all of those combinations; only the zero hard maximum differs.

avplan — the encoder is found rather than configured

Original: reads COMPATIBLE_VIDEO_ENCODER and trusts it, defaulting to h264_nvenc. A machine with no NVIDIA card and an unset variable builds an NVENC command line that fails the moment it runs.

Rust: asks ffmpeg -encoders what is installed and takes the best of them, hardware first.

Why this is acceptable: an environment variable naming the encoder is per-host configuration, and a filter that reads configuration is a filter whose answer depends on where it ran. Detection reaches the same answer on every machine the variable was set correctly on, and a better one on the machines it was not. The encoder table itself — which arguments each encoder gets — is identical, and --encoder names one explicitly for anyone who needs to.

Measured: all four branches of the encoder table are compared against the original in cases/avplan, each with the encoder named, so the argument lists agree exactly.

avplan — the output path is the destination, not a temporary

Original: writes to <output>.tmp.mp4 and renames on success, so a transcode killed halfway leaves no file that looks finished.

Rust: the command line names the destination.

Why this is acceptable: avplan does not run the command, so it cannot do the rename half of the pattern, and a command line that writes to a temporary nobody moves is worse than one that writes where it says. Anything running these in a loop can write to a temporary of its own choosing with --output.

avplan — a failed probe is an answer, not a fallback

Original: when ffprobe cannot read a file it falls back to the codec recorded in the database, which came from the downloader's pre-download metadata and is wrong often enough that the comment above it says so.

Rust: exits 2, could not tell.

Why this is acceptable: there is no database, and inventing a codec is how a compatible file gets re-encoded for forty minutes. Saying so is the honest answer and the cheap one.

pdfgutter — a different rasteriser, so the profiles are compared rather than the pixels

Python: renders each sampled page with pdfium at a scale that makes it a thousand pixels wide, and counts dark pixels per column.

Rust: asks pdftoppm for the same thousand-pixel-wide greymap and counts the same way.

Why this is acceptable: two rasterisers antialias differently, so a pixel on the edge of a letterform can land either side of the ink threshold in one and not the other. No amount of care makes two renderers agree pixel for pixel, and a parity test that demanded it would be a test of pdfium rather than of this.

What is compared instead is everything that decides: the aspect ratio, which pages get sampled, the reduction from pixels to a column profile, whether a given profile has a gutter, and the verdict a set of ratios and gutters adds up to. Both sides are handed the same pixels and the same profiles, and have to agree exactly.

The two renderers do have to be pointed at the same rectangle, which is what -cropbox is for — see the next note.

Measured: 94 cases, including profiles either side of the ink threshold by one ten-thousandth, profiles too narrow to hold the three bands the test looks at, and greymaps whose heights do not divide evenly so that both sides have to round a fraction the same way.

pdfgutter — the page is the crop box, consistently

Python: takes the aspect ratio from the media box and the ink profile from page.render(), which draws the crop box. On a page that carries both — most scanned books do — those are two different rectangles, so the shape is measured on one and the ink on the other. The bands the gutter is looked for in then fall somewhere other than the middle of what was rendered, and a real gutter is missed.

Rust: the crop box is what a reader shows, so it is what everything measures: the ratio, the render (pdftoppm -cropbox), and where the cut goes. A page with no crop box is its media box, which is what the spec says and what most files are.

Why this is acceptable: the original is not internally consistent, and on a wide sheet cropped to show only one of its two pages it gets the answer wrong in the destructive direction — the sheet still measures wide, so it is split again, down the middle of a page rather than between two. Measuring one rectangle throughout is the only version of this that is right.

Measured: a wide sheet cropped to one page is no longer a split candidate, and a cropped spread's gutter lands where it is looked for. The arithmetic that both sides share is compared exactly, as above.

pdfgutter — a page tree that does not add up is refused

Python: pdfium enumerates the pages and the question never comes up.

Rust: the page tree is walked directly, and a walk is stricter than a renderer: a node missing its /Type, a /Type written as a reference, a kid that is a dictionary rather than a reference to one — each of those is a page a strict walk does not see and a reader does. So the number of pages found is checked against the number the file says it has, and a document where they disagree is refused with exit 2.

Why this is acceptable: splitting replaces the page tree, and a page left out of the new tree is a page deleted. A file this does not fully understand is one it must not rewrite. Refusing costs a document that could perhaps have been handled; not refusing costs pages, silently, in the copy you keep.

Measured: five shapes that a strict walk under-counts, each refused rather than written short. A document with nothing wide enough to cut is not rewritten at all, so every page comes through however the tree is built.

pdfgutter — the thresholds are arguments, not environment variables

Python: reads BOOK_PIPELINE_2UP_MIN_RATIO and BOOK_PIPELINE_2UP_GUTTER_FRAC at import.

Rust: --min-ratio and --gutter-fraction, with the same defaults.

Why this is acceptable: an environment variable read at import makes the answer depend on the shell that happened to launch the process, which is the one thing a filter must not do. A flag says the same thing where a reader can see it, and shows up in the record --json writes.

pdfgutter — a sheet on its side is cut the way it is read

Python: takes the media box and ignores /Rotate entirely. A sheet stored tall and displayed wide is never a split candidate; one stored wide and displayed tall is a candidate, and would be cut down the middle of an axis that has nothing down the middle of it.

Rust: turns the media box by whatever /Rotate asks for before measuring it, and cuts across rather than down for a quarter turn, in the order a reader would read the halves.

Why this is acceptable: the wrong half of a rotated sheet is a page with the top of one page and the bottom of another on it, and no pipeline downstream can recover from that. The rotation is in the file precisely so that readers honour it. Documents with no rotation — nearly all of them — behave identically.

pdfgutter — an unreadable file is an answer, not a false

Python: is_double_page catches everything and returns False, so a corrupt file, an encrypted one and an ordinary single-page book are the same answer.

Rust: exit 2 for a file it cannot judge, exit 74 for one it cannot read, and exit 1 only for a document it looked at and found to be one page per sheet.

Why this is acceptable: "no" and "could not tell" are different answers and the contract gives them different numbers. A caller that wants the Python's behaviour treats both as no; a caller that wants to know which files it failed on can now find out.

avplan — separators in a path, and a destination that starts with a dash

Original: builds the destination by byte arithmetic — the path, minus as many bytes as the extension is long, plus .h264.mp4. Its extension function ignores trailing separators while the subtraction does not, so x.mp4// becomes x.m.h264.mp4. Duplicate separators are carried through, and a destination beginning with - is emitted bare.

Rust: takes the path apart properly. film.mp4/. is a directory and has no extension rather than an mp4 one; /talk.webm becomes /talk.h264.mp4 rather than //talk.h264.mp4; x.mp4// becomes x.h264.mp4. A destination beginning with - is written ./-name, which is the same file and cannot be read as an option.

Why this is acceptable: every one of these is a path the original gets wrong. A double leading separator is left to the system to interpret and some systems interpret it as something else; x.m.h264.mp4 names a file nobody asked for; a directory reported as a playable MP4 is a wrong answer outright; and a command line whose destination is read as an option fails on a file that was otherwise fine. The original never met any of them because its paths were always joined onto an absolute root.

Measured: 32 cases, including a file at the root, a name beginning with a dash, a name that is only a dot, a non-ASCII name and an extension that is only a dot. Ordinary paths — the overwhelming majority — are byte-identical.

---

chunkmd — the back-reference marker a whole class of EPUBs leaves

looks_like_citations recognises one note marker the Python it was ported from did not:

\[←788 \]
\[ [←193] (index_split_011.xhtml#citation-193) \]

(the second written with spaces here so it is not read as a link)

An arrow pointing back to where the note was called, sometimes as a link and sometimes not, wrapped in escaped brackets. It is what a common EPUB conversion leaves behind, and ten books of the corpus use it — including both volumes of The Decline of the West, whose endnotes run to 830 entries of Krumbacher, op. cit., pp. 969-90 and were being read as body prose. Every one of those books had zero citations extracted.

The parity suite does not cover the footnote signal and no fixture carries the marker, so the tests agree either way. This is recorded because it is a deliberate difference from the Python, not because anything caught it.

citefind strips the same marker when it cuts a page of notes into notes.

---

chunkmd — a reading list with nothing marking it is citation material

looks_like_citations flags a chunk carrying a run of reference entries that nothing announces — no heading over them, no numbers beside them:

    Block, N. 1981. Psychologism and behaviorism. Philosophical Review 90:5-43.
    Dennett, D. C. 1987. The Intentional Stance. MIT Press.
    Geach, P. 1957. Mental Acts. Routledge and Kegan Paul.

That is how a reader or a companion sets the further-reading list at the end of each chapter, indented under the last paragraph of the essay. The Python this was ported from looked only for note markers, so those chunks stayed role='body' with no flag, the citation stage never selected them, and books made entirely of such chapters came out with one or two citations.

The shape is also an index line — Newton, Isaac, 5, 93, 141 opens identically. What tells them apart is that a reference names a work: five words of three letters or more on the entry's own line, and not two page numbers written as links.

Over 15,925 unflagged body chunks sampled from the corpus this fires on 2.6%, and seven of eight sampled were real reference lists.

The parity harness allows this one difference by name: a chunk the Rust flags and the Python does not, identical in every other field. A flag the Python sets and the Rust does not is still a failure.