mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
pdfgutter: two facing pages photographed as one, found and cut
This commit is contained in:
funkpower
@@ -23,6 +23,7 @@ cueseg.workspace = true
|
||||
mdscrub.workspace = true
|
||||
mdsplit.workspace = true
|
||||
avplan.workspace = true
|
||||
pdfgutter.workspace = true
|
||||
pdftriage.workspace = true
|
||||
rrf.workspace = true
|
||||
corpus-core.workspace = true
|
||||
|
||||
@@ -201,3 +201,70 @@ metadata and is wrong often enough that the comment above it says so.
|
||||
**Why this is acceptable:** there is no database, and inventing a codec is
|
||||
how a compatible file gets re-encoded for forty minutes. Saying so is the
|
||||
honest answer and the cheap one.
|
||||
|
||||
## pdfgutter — a different rasteriser, so the profiles are compared rather than the pixels
|
||||
|
||||
**Python:** renders each sampled page with pdfium at a scale that makes it a
|
||||
thousand pixels wide, and counts dark pixels per column.
|
||||
|
||||
**Rust:** asks `pdftoppm` for the same thousand-pixel-wide greymap and counts
|
||||
the same way.
|
||||
|
||||
**Why this is acceptable:** two rasterisers antialias differently, so a pixel
|
||||
on the edge of a letterform can land either side of the ink threshold in one
|
||||
and not the other. No amount of care makes two renderers agree pixel for
|
||||
pixel, and a parity test that demanded it would be a test of pdfium rather
|
||||
than of this.
|
||||
|
||||
What is compared instead is everything that decides: the aspect ratio, which
|
||||
pages get sampled, whether a given ink profile has a gutter, and the verdict
|
||||
a set of ratios and gutters adds up to. Both sides are handed the same
|
||||
profiles and have to agree exactly.
|
||||
|
||||
**Measured:** 72 cases, including profiles either side of the ink threshold by
|
||||
one ten-thousandth and profiles too narrow to hold the three bands the test
|
||||
looks at.
|
||||
|
||||
## pdfgutter — the thresholds are arguments, not environment variables
|
||||
|
||||
**Python:** reads `BOOK_PIPELINE_2UP_MIN_RATIO` and
|
||||
`BOOK_PIPELINE_2UP_GUTTER_FRAC` at import.
|
||||
|
||||
**Rust:** `--min-ratio` and `--gutter-fraction`, with the same defaults.
|
||||
|
||||
**Why this is acceptable:** an environment variable read at import makes the
|
||||
answer depend on the shell that happened to launch the process, which is the
|
||||
one thing a filter must not do. A flag says the same thing where a reader can
|
||||
see it, and shows up in the record `--json` writes.
|
||||
|
||||
## pdfgutter — a sheet on its side is cut the way it is read
|
||||
|
||||
**Python:** takes the media box and ignores `/Rotate` entirely. A sheet stored
|
||||
tall and displayed wide is never a split candidate; one stored wide and
|
||||
displayed tall is a candidate, and would be cut down the middle of an axis
|
||||
that has nothing down the middle of it.
|
||||
|
||||
**Rust:** turns the media box by whatever `/Rotate` asks for before measuring
|
||||
it, and cuts across rather than down for a quarter turn, in the order a
|
||||
reader would read the halves.
|
||||
|
||||
**Why this is acceptable:** the wrong half of a rotated sheet is a page with
|
||||
the top of one page and the bottom of another on it, and no pipeline
|
||||
downstream can recover from that. The rotation is in the file precisely so
|
||||
that readers honour it. Documents with no rotation — nearly all of them —
|
||||
behave identically.
|
||||
|
||||
## pdfgutter — an unreadable file is an answer, not a false
|
||||
|
||||
**Python:** `is_double_page` catches everything and returns `False`, so a
|
||||
corrupt file, an encrypted one and an ordinary single-page book are the same
|
||||
answer.
|
||||
|
||||
**Rust:** exit 2 for a file it cannot judge, exit 74 for one it cannot read,
|
||||
and exit 1 only for a document it looked at and found to be one page per
|
||||
sheet.
|
||||
|
||||
**Why this is acceptable:** "no" and "could not tell" are different answers
|
||||
and the contract gives them different numbers. A caller that wants the
|
||||
Python's behaviour treats both as no; a caller that wants to know which files
|
||||
it failed on can now find out.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,72 @@
|
||||
{"kind": "ratio", "name": "two-portraits-side-by-side", "ratio": 1.4}
|
||||
{"kind": "ratio", "name": "a4-landscape", "ratio": 1.4142756349952963}
|
||||
{"kind": "ratio", "name": "a4-portrait", "ratio": 0.7070757462376319}
|
||||
{"kind": "ratio", "name": "us-letter", "ratio": 0.7727272727272727}
|
||||
{"kind": "ratio", "name": "square", "ratio": 1.0}
|
||||
{"kind": "ratio", "name": "exactly-at-the-threshold", "ratio": 1.15}
|
||||
{"kind": "ratio", "name": "a-hair-over", "ratio": 1.1501388888888888}
|
||||
{"kind": "ratio", "name": "no-height", "ratio": 0.0}
|
||||
{"kind": "ratio", "name": "no-width", "ratio": 0.0}
|
||||
{"kind": "ratio", "name": "a-very-long-foldout", "ratio": 5.714285714285714}
|
||||
{"gutter": true, "kind": "gutter", "name": "two-columns"}
|
||||
{"gutter": false, "kind": "gutter", "name": "one-wide-block"}
|
||||
{"gutter": false, "kind": "gutter", "name": "a-narrow-centred-column"}
|
||||
{"gutter": false, "kind": "gutter", "name": "blank"}
|
||||
{"gutter": false, "kind": "gutter", "name": "text-on-the-left-only"}
|
||||
{"gutter": false, "kind": "gutter", "name": "text-on-the-right-only"}
|
||||
{"gutter": true, "kind": "gutter", "name": "a-single-blank-column"}
|
||||
{"gutter": true, "kind": "gutter", "name": "a-gutter-off-centre"}
|
||||
{"gutter": false, "kind": "gutter", "name": "text-just-at-the-ink-threshold"}
|
||||
{"gutter": true, "kind": "gutter", "name": "text-just-over-it"}
|
||||
{"gutter": false, "kind": "gutter", "name": "a-gutter-that-is-not-quite-blank"}
|
||||
{"gutter": false, "kind": "gutter", "name": "empty"}
|
||||
{"gutter": false, "kind": "gutter", "name": "two-columns-too-few"}
|
||||
{"gutter": true, "kind": "gutter", "name": "odd-width"}
|
||||
{"gutter": true, "kind": "gutter", "name": "a-shadow-at-the-edges"}
|
||||
{"gutter": false, "kind": "gutter", "name": "tiny-7"}
|
||||
{"gutter": true, "kind": "gutter", "name": "tiny-13"}
|
||||
{"gutter": true, "kind": "gutter", "name": "tiny-25"}
|
||||
{"gutter": true, "kind": "gutter", "name": "tiny-101"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-0"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-1"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-2"}
|
||||
{"gutter": false, "kind": "gutter", "name": "random-3"}
|
||||
{"gutter": false, "kind": "gutter", "name": "random-4"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-5"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-6"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-7"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-8"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-9"}
|
||||
{"gutter": false, "kind": "gutter", "name": "random-10"}
|
||||
{"gutter": true, "kind": "gutter", "name": "random-11"}
|
||||
{"indices": [], "kind": "sample", "name": "pages-0"}
|
||||
{"indices": [0], "kind": "sample", "name": "pages-1"}
|
||||
{"indices": [0, 1], "kind": "sample", "name": "pages-2"}
|
||||
{"indices": [0, 1, 2, 3, 4], "kind": "sample", "name": "pages-5"}
|
||||
{"indices": [0, 1, 2, 3, 4, 5], "kind": "sample", "name": "pages-6"}
|
||||
{"indices": [2, 3, 4], "kind": "sample", "name": "pages-7"}
|
||||
{"indices": [2, 3, 4, 5], "kind": "sample", "name": "pages-8"}
|
||||
{"indices": [2, 3, 4, 5, 6], "kind": "sample", "name": "pages-9"}
|
||||
{"indices": [2, 3, 4, 5, 6, 7, 8], "kind": "sample", "name": "pages-11"}
|
||||
{"indices": [2, 3, 4, 5, 6, 7, 8, 9], "kind": "sample", "name": "pages-12"}
|
||||
{"indices": [2, 3, 4, 5, 6, 7, 8, 9], "kind": "sample", "name": "pages-16"}
|
||||
{"indices": [2, 3, 4, 5, 6, 7, 8, 9], "kind": "sample", "name": "pages-17"}
|
||||
{"indices": [2, 3, 4, 5, 6, 7, 8, 9], "kind": "sample", "name": "pages-20"}
|
||||
{"indices": [2, 7, 12, 17, 22, 27, 32, 37], "kind": "sample", "name": "pages-50"}
|
||||
{"indices": [2, 13, 24, 35, 46, 57, 68, 79], "kind": "sample", "name": "pages-99"}
|
||||
{"indices": [2, 13, 24, 35, 46, 57, 68, 79], "kind": "sample", "name": "pages-100"}
|
||||
{"indices": [2, 32, 62, 92, 122, 152, 182, 212], "kind": "sample", "name": "pages-250"}
|
||||
{"indices": [2, 64, 126, 188, 250, 312, 374, 436], "kind": "sample", "name": "pages-501"}
|
||||
{"indices": [2, 126, 250, 374, 498, 622, 746, 870], "kind": "sample", "name": "pages-1000"}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.4, "name": "a-clear-two-up", "two_up": true}
|
||||
{"gutter_fraction": 0.0, "kind": "verdict", "median_ratio": 0.68, "name": "a-clear-single", "two_up": false}
|
||||
{"gutter_fraction": 0.0, "kind": "verdict", "median_ratio": 1.4, "name": "wide-but-no-gutters", "two_up": false}
|
||||
{"gutter_fraction": 0.6, "kind": "verdict", "median_ratio": 1.4, "name": "exactly-six-in-ten", "two_up": true}
|
||||
{"gutter_fraction": 0.5, "kind": "verdict", "median_ratio": 1.4, "name": "five-in-ten", "two_up": false}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.4, "name": "mixed-shapes", "two_up": true}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.04, "name": "an-even-split-of-shapes", "two_up": false}
|
||||
{"gutter_fraction": 0.0, "kind": "verdict", "median_ratio": 0.0, "name": "nothing-sampled", "two_up": false}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.4, "name": "one-page", "two_up": true}
|
||||
{"gutter_fraction": 0.6666666666666666, "kind": "verdict", "median_ratio": 1.4, "name": "two-of-three", "two_up": true}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.1501, "name": "just-over-the-ratio", "two_up": true}
|
||||
{"gutter_fraction": 1.0, "kind": "verdict", "median_ratio": 1.15, "name": "exactly-the-ratio", "two_up": false}
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Decide 2-up scans the way the book pipeline did, for comparison.
|
||||
|
||||
The four functions below are the originals, copied out of the preprocessing
|
||||
module verbatim — importing it would pull in a PDF renderer and a page tree,
|
||||
and none of that decides anything. What decides is the arithmetic, which is
|
||||
what this compares.
|
||||
|
||||
python3 oracle/pdfgutter.py < cases.ndjson
|
||||
|
||||
One case per line, tagged with what it is asking about. Writes one JSON
|
||||
answer per line.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
_MIN_RATIO = 1.15
|
||||
_MIN_GUTTER_FRAC = 0.6
|
||||
_TEXT_COL_INK = 0.02
|
||||
|
||||
|
||||
def _ratio(width: float, height: float) -> float:
|
||||
return width / height if height else 0.0
|
||||
|
||||
|
||||
def _has_gutter(ink: np.ndarray) -> bool:
|
||||
w = len(ink)
|
||||
center = ink[int(0.46 * w):int(0.54 * w)]
|
||||
left = ink[int(0.10 * w):int(0.44 * w)]
|
||||
right = ink[int(0.56 * w):int(0.90 * w)]
|
||||
if center.size == 0 or left.size == 0 or right.size == 0:
|
||||
return False
|
||||
return bool(
|
||||
center.min() < _TEXT_COL_INK
|
||||
and np.median(left) > _TEXT_COL_INK
|
||||
and np.median(right) > _TEXT_COL_INK
|
||||
)
|
||||
|
||||
|
||||
def _sample_indices(n: int, k: int = 8) -> list[int]:
|
||||
if n <= 6:
|
||||
return list(range(n))
|
||||
lo, hi = 2, n - 3
|
||||
step = max(1, (hi - lo) // k)
|
||||
return list(range(lo, hi + 1, step))[:k]
|
||||
|
||||
|
||||
def _verdict(ratios: list[float], gutters: list[bool]) -> dict:
|
||||
median_ratio = float(np.median(ratios)) if ratios else 0.0
|
||||
gutter_frac = (sum(gutters) / len(gutters)) if gutters else 0.0
|
||||
return {
|
||||
"median_ratio": median_ratio,
|
||||
"gutter_fraction": gutter_frac,
|
||||
"two_up": bool(median_ratio > _MIN_RATIO and gutter_frac >= _MIN_GUTTER_FRAC),
|
||||
}
|
||||
|
||||
|
||||
def answer(case: dict) -> dict:
|
||||
kind = case["kind"]
|
||||
if kind == "ratio":
|
||||
return {"ratio": _ratio(case["width"], case["height"])}
|
||||
if kind == "gutter":
|
||||
return {"gutter": _has_gutter(np.asarray(case["ink"], dtype=float))}
|
||||
if kind == "sample":
|
||||
return {"indices": _sample_indices(case["pages"])}
|
||||
if kind == "verdict":
|
||||
return _verdict(case["ratios"], case["gutters"])
|
||||
raise SystemExit(f"no such case: {kind}")
|
||||
|
||||
|
||||
for line in sys.stdin:
|
||||
if not line.strip():
|
||||
continue
|
||||
case = json.loads(line)
|
||||
out = {"name": case["name"], "kind": case["kind"]}
|
||||
out.update(answer(case))
|
||||
sys.stdout.write(json.dumps(out, sort_keys=True) + "\n")
|
||||
+14
-1
@@ -19,7 +19,7 @@ type Written = Result<Vec<String>, String>;
|
||||
type Job = (&'static str, fn() -> Written);
|
||||
|
||||
fn main() -> ExitCode {
|
||||
let jobs: [Job; 16] = [
|
||||
let jobs: [Job; 17] = [
|
||||
("tokens", tokens),
|
||||
("unescape", unescape),
|
||||
("chunkmd", chunkmd),
|
||||
@@ -36,6 +36,7 @@ fn main() -> ExitCode {
|
||||
("vocab", vocab),
|
||||
("mdsplit", mdsplit),
|
||||
("avplan", avplan),
|
||||
("pdfgutter", pdfgutter),
|
||||
];
|
||||
|
||||
for (name, job) in jobs {
|
||||
@@ -218,6 +219,18 @@ fn cueseg() -> Written {
|
||||
Ok(written)
|
||||
}
|
||||
|
||||
fn pdfgutter() -> Written {
|
||||
let dir = parity::golden("pdfgutter");
|
||||
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
|
||||
|
||||
let cases = parity::dir().join("cases/pdfgutter/cases.ndjson");
|
||||
let text = std::fs::read_to_string(&cases).map_err(|e| e.to_string())?;
|
||||
let answers = parity::run_oracle("pdfgutter.py", &text)?;
|
||||
let path = dir.join("cases.ndjson");
|
||||
std::fs::write(&path, answers).map_err(|e| e.to_string())?;
|
||||
Ok(vec![path.display().to_string()])
|
||||
}
|
||||
|
||||
fn avplan() -> Written {
|
||||
let dir = parity::golden("avplan");
|
||||
std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
//! The arithmetic that decides a two-up scan must match the Python exactly.
|
||||
//!
|
||||
//! The consequence of getting it wrong is not a wrong page count — it is a
|
||||
//! book whose text comes out interleaved line by line between two pages,
|
||||
//! which reads as prose and is not. Nobody notices that in a corpus until
|
||||
//! they read a retrieved passage and find it is nonsense.
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
fn records(text: &str) -> Vec<Value> {
|
||||
text.lines()
|
||||
.filter(|line| !line.trim().is_empty())
|
||||
.map(|line| serde_json::from_str(line).expect("valid JSON"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn cases() -> Vec<Value> {
|
||||
let text = std::fs::read_to_string(parity::dir().join("cases/pdfgutter/cases.ndjson"))
|
||||
.expect("a readable case file");
|
||||
records(&text)
|
||||
}
|
||||
|
||||
fn floats(value: &Value) -> Vec<f64> {
|
||||
value
|
||||
.as_array()
|
||||
.expect("an array")
|
||||
.iter()
|
||||
.map(|number| number.as_f64().expect("a number"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_answer_matches_python() {
|
||||
let golden = records(&parity::read_golden_text("pdfgutter/cases.ndjson"));
|
||||
let cases = cases();
|
||||
assert_eq!(cases.len(), golden.len(), "case count moved — regenerate");
|
||||
|
||||
let mut wrong: Vec<String> = Vec::new();
|
||||
for (case, theirs) in cases.iter().zip(&golden) {
|
||||
let name = case["name"].as_str().expect("a name");
|
||||
let kind = case["kind"].as_str().expect("a kind");
|
||||
assert_eq!(theirs["name"].as_str(), Some(name), "cases out of order");
|
||||
assert_eq!(theirs["kind"].as_str(), Some(kind), "cases out of order");
|
||||
|
||||
let mut note = |field: &str, ours: String, theirs: &Value| {
|
||||
wrong.push(format!("{name}.{field}: python {theirs}, rust {ours}"));
|
||||
};
|
||||
|
||||
match kind {
|
||||
"ratio" => {
|
||||
let ours = pdfgutter::aspect_ratio(
|
||||
case["width"].as_f64().expect("a width"),
|
||||
case["height"].as_f64().expect("a height"),
|
||||
);
|
||||
if Some(ours) != theirs["ratio"].as_f64() {
|
||||
note("ratio", ours.to_string(), &theirs["ratio"]);
|
||||
}
|
||||
}
|
||||
"gutter" => {
|
||||
let ours = pdfgutter::has_gutter(&floats(&case["ink"]));
|
||||
if Some(ours) != theirs["gutter"].as_bool() {
|
||||
note("gutter", ours.to_string(), &theirs["gutter"]);
|
||||
}
|
||||
}
|
||||
"sample" => {
|
||||
let ours = pdfgutter::sample_indices(
|
||||
case["pages"].as_u64().expect("a page count") as usize
|
||||
);
|
||||
let theirs_indices: Vec<usize> = theirs["indices"]
|
||||
.as_array()
|
||||
.expect("an array")
|
||||
.iter()
|
||||
.map(|index| index.as_u64().expect("a number") as usize)
|
||||
.collect();
|
||||
if ours != theirs_indices {
|
||||
note("indices", format!("{ours:?}"), &theirs["indices"]);
|
||||
}
|
||||
}
|
||||
"verdict" => {
|
||||
let ratios = floats(&case["ratios"]);
|
||||
let gutters: Vec<bool> = case["gutters"]
|
||||
.as_array()
|
||||
.expect("an array")
|
||||
.iter()
|
||||
.map(|found| found.as_bool().expect("a boolean"))
|
||||
.collect();
|
||||
|
||||
let median = pdfgutter::median(&ratios);
|
||||
if Some(median) != theirs["median_ratio"].as_f64() {
|
||||
note("median_ratio", median.to_string(), &theirs["median_ratio"]);
|
||||
}
|
||||
let fraction = pdfgutter::gutter_fraction(&gutters);
|
||||
if Some(fraction) != theirs["gutter_fraction"].as_f64() {
|
||||
note(
|
||||
"gutter_fraction",
|
||||
fraction.to_string(),
|
||||
&theirs["gutter_fraction"],
|
||||
);
|
||||
}
|
||||
let two_up = pdfgutter::is_two_up(&ratios, &gutters);
|
||||
if Some(two_up) != theirs["two_up"].as_bool() {
|
||||
note("two_up", two_up.to_string(), &theirs["two_up"]);
|
||||
}
|
||||
}
|
||||
other => panic!("nothing answers a {other} case"),
|
||||
}
|
||||
}
|
||||
|
||||
assert!(
|
||||
wrong.is_empty(),
|
||||
"{} answers differ:\n{}",
|
||||
wrong.len(),
|
||||
wrong
|
||||
.iter()
|
||||
.take(20)
|
||||
.map(|line| line.chars().take(200).collect::<String>())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
/// All four questions have to be asked, or the file proves less than it looks
|
||||
/// like it does.
|
||||
#[test]
|
||||
fn every_kind_of_case_is_present() {
|
||||
let cases = cases();
|
||||
for kind in ["ratio", "gutter", "sample", "verdict"] {
|
||||
assert!(
|
||||
cases.iter().any(|case| case["kind"] == kind),
|
||||
"no {kind} case"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Catches a golden file drifting away from a Python that has moved on.
|
||||
#[test]
|
||||
#[cfg(feature = "python")]
|
||||
fn the_golden_file_still_matches_live_python() {
|
||||
let cases = std::fs::read_to_string(parity::dir().join("cases/pdfgutter/cases.ndjson"))
|
||||
.expect("a readable case file");
|
||||
let live = parity::run_oracle("pdfgutter.py", &cases).expect("the oracle runs");
|
||||
assert_eq!(live, parity::read_golden_text("pdfgutter/cases.ndjson"));
|
||||
}
|
||||
Reference in New Issue
Block a user