a missing word, a hidden dependency, and a second pass that changed things

parity/KNOWN.md

@@ -38,25 +38,60 @@ there is no input where it changes a cue the original read.
 recovered. Across the other eight cases, including deliberately malformed
 timestamps, the two agree cue for cue.
 
+## pdftriage — a newer Unicode table knows letters an older one does not
+
+**Python:** `str.isalpha` is true for the letter categories of the Unicode
+version the interpreter was built against.
+
+**Rust:** the same test against a table built from a later Unicode version,
+which knows 4,924 code points the older one does not.
+
+**Why this is acceptable:** the newer answer is the correct one, and the
+difference is entirely one-directional — every code point the two disagree
+about is a letter that Rust knows and Python does not, never the reverse. It
+matters because the letter share is what calls a text layer dead: a book
+typeset in Cyrillic Extended-C is judged garbled by the older table, which
+sends a perfectly readable text layer for an hour of OCR that can only make it
+worse. Erring toward not re-OCRing a book whose text is fine is the right
+direction for this gate to err in.
+
+Both tables will keep moving, which is the other reason this entry exists: the
+golden files pin one pair of Unicode versions, and a change to either shows up
+as a parity failure rather than as a silently different verdict.
+
+**Measured:** `parity/cases/pdftriage/samples.ndjson`, case
+`cyrillic-extended-c`: letter share 1.00 here against 0.00 there, which is the
+difference between healthy and garbled. Brute-forced over the whole code
+space: 4,924 disagreements, none of them in the other direction.
+
 ## mdscrub — un-paginated text is not given a page 1 marker
 
 **Python:** `_normalize_marker_pages` ends by wrapping any text it found no
 pagination in as `<!-- Page 1 -->\n\n…`, so that downstream can assume a
 marker exists.
 
-**Rust:** text with no page separators and no form feeds is returned
+**Rust:** page boundaries are not touched at all unless `--pages` asks for
+them, and even then text with no separators and no form feeds is returned
 unchanged.
 
 **Why this is acceptable:** a cleanup filter has to be safe to run twice, and
 the Python's version is not — a second pass wraps the first pass's marker in
 another one. The assumption it exists to serve belongs to the converter that
 knows it just read a PDF, not to a filter that is handed markdown of unknown
-provenance. The chunker already handles a document with no page markers by
-emitting chunks with no page number, which is the honest answer.
+provenance: a form feed in a code listing is a control character, and a dash
+rule in a quoted document is not a page break. The chunker already handles a
+document with no page markers by emitting chunks with no page number, which is
+the honest answer.
+
+Separating the two passes is also what makes the cleanup idempotent. Removing
+a tag can leave a dash rule flush against a newline, and a pagination pass
+running afterwards would find a page boundary on the second run that was not
+there on the first.
 
 **Measured:** not exercised by any parity case, since the Python function is
 only reachable from the PDF conversion path. Covered by
-`mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was`.
+`mdscrub::pages::tests::text_with_no_page_boundaries_is_left_exactly_as_it_was`
+and `mdscrub::tests::nothing_manufactures_a_page_boundary_on_a_second_run`.
 
 ## Every ported pattern — the two engines disagree about `\d` and `(?i)`
 
@@ -97,6 +132,10 @@ make every chunk record wider to carry a value nothing can produce. The
 saturation is deliberate rather than a wrap, so the wrong answer is at least
 obviously wrong.
 
+Note that this applies only where a page number becomes a *number*. Where one
+is copied into text — a page anchor in the body of a document — it is carried
+through exactly as it was written, padding and all.
+
 **Measured:** no document in any corpus this was run against has a page
 marker above five digits.
 

parity/cases/pdftriage/samples.ndjson

@@ -22,3 +22,5 @@
 {"name": "nbsp-prose", "text": "The second estate held its land in exchange for service, and the service was military. That much is agreed; what follows is not. The court rolls do not say how often the obligation was tested, because the clerk who recorded it had no vocabulary for the difference between a rebellion and a bad harvest. This is the central difficulty of the sources, and everything that follows depends on reading around that limitation rather than through it. The second estate held its land in exchange for service, and the service was military. That much is agreed; what follows is not. The court rolls do not say how often the obligation was tested, because the clerk who recorded it had no vocabulary for the difference between a rebellion and a bad harvest. This is the central difficulty of the sources, and everything that follows depends on reading around that limitation rather than through it. "}
 {"name": "control-chars", "text": "text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c text\u0001with\u000bcontrol\u001fchars\u001c "}
 {"name": "ligatures", "text": "The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. The first flight of the figure was confirmed. "}
+{"name": "function-words", "text": "a all an and are as at be been but by for from had have his in is it more not of on one or that the there they this to was we were which with you a all an and are as at be been but by for from had have his in is it more not of on one or that the there they this to was we were which with you a all an and are as at be been but by for from had have his in is it more not of on one or that the there they this to was we were which with you a all an and are as at be been but by for from had have his in is it more not of on one or that the there they this to was we were which with you "}
+{"name": "cyrillic-extended-c", "text": "ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ ᲉᲉᲉᲉᲉ "}

parity/golden/pdftriage/samples.ndjson

@@ -22,3 +22,5 @@
 {"ascii": 1.0, "body": 736, "decoded": null, "english": 0.42105263157894735, "latin1": 0.0, "letters": 0.9782608695652174, "name": "nbsp-prose", "tidy": 1.0, "vowelled": 1.0}
 {"ascii": 0.9523809523809523, "body": 630, "decoded": null, "english": 0.0, "latin1": 0.0, "letters": 0.9523809523809523, "name": "control-chars", "tidy": 1.0, "vowelled": 1.0}
 {"ascii": 0.8823529411764706, "body": 680, "decoded": null, "english": 0.5, "latin1": 0.0, "letters": 0.9705882352941176, "name": "ligatures", "tidy": 1.0, "vowelled": 0.875}
+{"ascii": 1.0, "body": 436, "decoded": null, "english": 1.0, "latin1": 0.0, "letters": 1.0, "name": "function-words", "tidy": 1.0, "vowelled": 1.0}
+{"ascii": 0.0, "body": 500, "decoded": null, "english": 0.0, "latin1": 0.0, "letters": 0.0, "name": "cyrillic-extended-c", "tidy": null, "vowelled": null}

parity/tests/pdftriage.rs

@@ -10,6 +10,12 @@
 
 use serde_json::Value;
 
+/// Measurements that are meant to differ, each argued for in `KNOWN.md`.
+///
+/// The two implementations compile against different Unicode versions, and
+/// the newer one knows letterforms the older one does not.
+const DIVERGENT: &[(&str, &str)] = &[("cyrillic-extended-c", "letters")];
+
 /// Both sides compute a count divided by a count, so agreement is exact or
 /// the counts differ.
 fn same(ours: f64, theirs: &Value) -> bool {
@@ -46,6 +52,9 @@ fn every_measurement_matches_python() {
         let body = pdftriage::score::body(text);
 
         let mut note = |field: &str, ours: String, theirs: &Value| {
+            if DIVERGENT.contains(&(name, field)) {
+                return;
+            }
             wrong.push(format!("{name}.{field}: python {theirs}, rust {ours}"));
         };
 
@@ -100,6 +109,29 @@ fn every_measurement_matches_python() {
         }
     }
 
+    // A divergence that has quietly gone away is a note in KNOWN.md that is
+    // no longer true, which is worth knowing about.
+    for (name, field) in DIVERGENT {
+        let case = cases
+            .iter()
+            .find(|case| case["name"].as_str() == Some(name))
+            .unwrap_or_else(|| panic!("{name} is recorded as divergent but is not a case"));
+        let theirs = golden
+            .iter()
+            .find(|record| record["name"].as_str() == Some(name))
+            .expect("a golden answer");
+        let body = pdftriage::score::body(case["text"].as_str().expect("text"));
+        let ours = match *field {
+            "letters" => pdftriage::score::letter_share(&body),
+            other => panic!("nothing measures {other}"),
+        };
+        assert_ne!(
+            Some(ours),
+            theirs[field].as_f64(),
+            "{name}.{field} no longer diverges — update KNOWN.md"
+        );
+    }
+
     assert!(
         wrong.is_empty(),
         "{} measurements differ:\n{}",