langid and tocfind: what language a book is in, and what it says its chapters are

Cargo.lock

@@ -907,6 +907,18 @@ dependencies = [
  "wasm-bindgen",
 ]
 
+[[package]]
+name = "langid"
+version = "0.1.0"
+dependencies = [
+ "catena-cli",
+ "clap",
+ "corpus-core",
+ "schemars",
+ "serde",
+ "serde_json",
+]
+
 [[package]]
 name = "lazy_static"
 version = "1.5.0"
@@ -1699,6 +1711,20 @@ version = "1.13.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "fd3ca314f692efd6c868f8408f53fe444634a845f96c028b97d35f6a1f79f0ee"
 
+[[package]]
+name = "tocfind"
+version = "0.1.0"
+dependencies = [
+ "catena-cli",
+ "clap",
+ "corpus-core",
+ "lopdf",
+ "schemars",
+ "serde",
+ "serde_json",
+ "zip",
+]
+
 [[package]]
 name = "ttf-parser"
 version = "0.25.1"

Cargo.toml

@@ -30,6 +30,8 @@ seriesof = { path = "crates/seriesof", version = "0.1.0" }
 idnorm = { path = "crates/idnorm", version = "0.1.0" }
 bibfetch = { path = "crates/bibfetch", version = "0.1.0" }
 bookmeta = { path = "crates/bookmeta", version = "0.1.0" }
+langid = { path = "crates/langid", version = "0.1.0" }
+tocfind = { path = "crates/tocfind", version = "0.1.0" }
 booksniff = { path = "crates/booksniff", version = "0.1.0" }
 emb = { path = "crates/emb", version = "0.1.0" }
 

README.md

@@ -25,6 +25,8 @@ Each tool installs by name, so you can take one without taking the rest.
 |---|---|
 | [`booksniff`](crates/booksniff/README.md) | What a file actually is, whatever its name claims |
 | [`bookmeta`](crates/bookmeta/README.md) | The metadata a file already carries |
+| [`tocfind`](crates/tocfind/README.md) | A book's own table of contents |
+| [`langid`](crates/langid/README.md) | What language a document is in |
 | [`pdfgutter`](crates/pdfgutter/README.md) | Books scanned two pages to a sheet, split back into pages |
 | [`pdftriage`](crates/pdftriage/README.md) | Whether a PDF's text layer is worth reading, or whether it has to be OCRed |
 | [`bibclean`](crates/bibclean/README.md) | Catalogue records audited and repaired, with a reason for every change |

crates/langid/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+name = "langid"
+description = "What language a document is in, from its script and its commonest words."
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+repository.workspace = true
+
+[dependencies]
+catena-cli.workspace = true
+clap.workspace = true
+corpus-core.workspace = true
+schemars.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+
+[lints]
+workspace = true

crates/langid/README.md

@@ -0,0 +1,123 @@
+# langid
+
+What language a document is in.
+
+Part of [catena](../../README.md).
+
+## What it does
+
+Two passes, because one will not do. The writing system settles most of the
+world outright — a text in Hangul is Korean — and for the Latin and Cyrillic
+alphabets it only narrows the field, so the commonest words decide between
+what is left.
+
+No model, no download, no network. Function words make up a third of any
+running text, and counting them is decisive on a paragraph. On a single line
+it is not, which is why a close call is refused rather than guessed at.
+
+## When you would use it
+
+- **Before OCR**, to know which language to hand it.
+- **Before embedding**, where a multilingual shelf wants different models.
+- **Filling the `language` field** on a `doc` record, which nothing else sets.
+- **Finding the odd book out** in a library you assumed was all one language.
+
+## Install
+
+```sh
+cargo install --git https://git.pumpin.online/funkpower/catena langid
+```
+
+## Example
+
+```
+$ langid converted/*.md | jq -r '[.name, (.confidence*100|round|tostring + "%"), .path] | @tsv'
+English	77%	converted/enlightenment.md
+English	76%	converted/hagakure.md
+French	81%	converted/meditations.md
+Latin	68%	converted/summa.md
+```
+
+Just the code:
+
+```
+$ langid --quiet book.md
+en
+```
+
+The whole record, which carries the bibliographic code the `doc` record
+wants:
+
+```
+$ langid book.md
+{"v":1,"path":"book.md","language":"en","bibliographic":"eng","name":"English","script":"Latin","confidence":0.77}
+```
+
+The ones that are not what you assumed:
+
+```sh
+langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv'
+```
+
+Reading from a pipe:
+
+```sh
+pdftotext -l 5 scan.pdf - | langid --quiet
+```
+
+## Options
+
+| Option | Effect |
+|---|---|
+| `--quiet` | Print the code alone, one per line |
+| `--man` | Write this page as a manual page, in roff |
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Every file was placed |
+| `1` | At least one could not be |
+| `64` | The command line was wrong |
+| `74` | A file could not be read |
+
+## How it decides
+
+**The script first.** Kana, Hangul, Hebrew, Arabic and Devanagari each belong
+to one language as far as a library is concerned, and no amount of word
+counting improves on that. Kana is checked before Han, because Japanese is
+mostly Han characters with kana between them and it is the kana that tell it
+from Chinese.
+
+A quarter of a text's letters agreeing is enough to call it a script: a book
+carries page numbers, names and quotations in other alphabets, and an English
+word in a Russian paragraph does not make the paragraph English.
+
+**Then the words.** Thirteen languages across Latin, Cyrillic and Greek, each
+with its thirty commonest function words. The score is the share of a text's
+words found in each list, and the lists are the same length so the scores
+compare.
+
+**A close call is no answer.** The winner must be a quarter ahead of the
+runner-up. `de` is among the commonest words of four languages at once, and on
+its own settles none of them — so a text that short comes back with the script
+named, no language, and a note saying which two it could not choose between.
+
+Only the first twenty thousand characters are read. A book's language does not
+change after the first few pages.
+
+## As a library
+
+```rust
+let found = langid::detect("book.md", text);
+assert_eq!(found.language.as_deref(), Some("en"));
+```
+
+`langid::script::detect` answers the writing system on its own, and
+`langid::words::score` ranks every language of a given script.
+
+## See also
+
+- [`booksniff`](../booksniff/README.md) — what the file is.
+- [`bookmeta`](../bookmeta/README.md) — what the file says its language is,
+  which is worth more when it says anything.

crates/langid/src/lib.rs

@@ -0,0 +1,235 @@
+//! What language a document is in.
+//!
+//! Two passes, because one does not do. The writing system settles most of
+//! the world outright — a text in Hangul is Korean — and for the Latin and
+//! Cyrillic alphabets it only narrows the field, so the commonest words
+//! decide between what is left.
+//!
+//! No model, no download, no network. Function words make up a third of any
+//! running text, and on a paragraph or more counting them is decisive. On a
+//! single line it is not, which is why every answer carries a confidence and
+//! why "could not tell" is one of the answers.
+
+pub mod script;
+pub mod words;
+
+use schemars::JsonSchema;
+use serde::{Deserialize, Serialize};
+
+pub use script::Script;
+
+/// How much text is worth looking at.
+///
+/// A book's language does not change after the first few pages, and reading
+/// all of it to say so is waste.
+pub const SAMPLE_CHARS: usize = 20_000;
+
+/// What language a text turned out to be.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
+pub struct Verdict {
+    /// Record version.
+    #[serde(default = "version")]
+    pub v: u32,
+    /// What was read.
+    pub path: String,
+    /// ISO 639-1, the two-letter code. Absent when it could not be told.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub language: Option<String>,
+    /// ISO 639-2/B, which is what the `doc` record carries.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub bibliographic: Option<String>,
+    /// What the language is called in English.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+    /// The writing system, which is an answer even when the language is not.
+    pub script: String,
+    /// How far ahead the winner was, from zero to one. One means nothing else
+    /// scored at all.
+    #[serde(with = "corpus_core::finite")]
+    #[schemars(with = "f64")]
+    pub confidence: f64,
+    /// What was worth saying about the reading.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub notes: Vec<String>,
+}
+
+fn version() -> u32 {
+    corpus_core::V
+}
+
+impl corpus_core::Record for Verdict {
+    fn version(&self) -> u32 {
+        self.v
+    }
+}
+
+/// Read a text's language.
+pub fn detect(path: &str, text: &str) -> Verdict {
+    let sample: String = text.chars().take(SAMPLE_CHARS).collect();
+    let (script, share) = script::detect(&sample);
+
+    let mut verdict = Verdict {
+        v: corpus_core::V,
+        path: path.to_string(),
+        language: None,
+        bibliographic: None,
+        name: None,
+        script: name_of(script).to_string(),
+        confidence: 0.0,
+        notes: Vec::new(),
+    };
+
+    // A script only one language uses is the whole answer, and no amount of
+    // word counting would improve on it.
+    if let Some(sole) = sole_language(script) {
+        verdict.language = Some(sole.0.to_string());
+        verdict.bibliographic = Some(sole.1.to_string());
+        verdict.name = Some(sole.2.to_string());
+        verdict.confidence = share;
+        verdict
+            .notes
+            .push(format!("{} is written only in {}", sole.2, name_of(script)));
+        return verdict;
+    }
+
+    let scored = words::score(&sample, script);
+    let Some((best, best_score)) = scored.first() else {
+        verdict.notes.push(match script {
+            Script::Other => "no letters to go on".to_string(),
+            script => format!(
+                "nothing here knows the languages written in {}",
+                name_of(script)
+            ),
+        });
+        return verdict;
+    };
+
+    if *best_score == 0.0 {
+        verdict
+            .notes
+            .push("none of the commonest words of any language it knows".to_string());
+        return verdict;
+    }
+
+    // How far ahead of the runner-up, so a close call can be refused.
+    let runner_up = scored.get(1).map_or(0.0, |(_, score)| *score);
+    let ratio = match runner_up > 0.0 {
+        true => best_score / runner_up,
+        false => f64::INFINITY,
+    };
+    if ratio < words::MARGIN {
+        verdict.notes.push(format!(
+            "{} and {} score too alike to choose between",
+            best.name, scored[1].0.name
+        ));
+        return verdict;
+    }
+
+    verdict.language = Some(best.code.to_string());
+    verdict.bibliographic = Some(best.bibliographic.to_string());
+    verdict.name = Some(best.name.to_string());
+    verdict.confidence = match runner_up > 0.0 {
+        true => (1.0 - runner_up / best_score).clamp(0.0, 1.0),
+        false => 1.0,
+    };
+    verdict
+}
+
+/// The one language a script is used for, where there is only one.
+fn sole_language(script: Script) -> Option<(&'static str, &'static str, &'static str)> {
+    Some(match script {
+        Script::Kana => ("ja", "jpn", "Japanese"),
+        Script::Han => ("zh", "chi", "Chinese"),
+        Script::Hangul => ("ko", "kor", "Korean"),
+        Script::Hebrew => ("he", "heb", "Hebrew"),
+        Script::Arabic => ("ar", "ara", "Arabic"),
+        Script::Devanagari => ("hi", "hin", "Hindi"),
+        _ => return None,
+    })
+}
+
+/// What a script is called.
+pub fn name_of(script: Script) -> &'static str {
+    match script {
+        Script::Latin => "Latin",
+        Script::Cyrillic => "Cyrillic",
+        Script::Greek => "Greek",
+        Script::Arabic => "Arabic",
+        Script::Hebrew => "Hebrew",
+        Script::Han => "Han",
+        Script::Kana => "Kana",
+        Script::Hangul => "Hangul",
+        Script::Devanagari => "Devanagari",
+        Script::Other => "unknown",
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn a_paragraph_of_english_is_english() {
+        let found = detect(
+            "x.md",
+            "The obligation ran both ways, and contemporaries said so plainly whenever they were asked about it at all.",
+        );
+        assert_eq!(found.language.as_deref(), Some("en"));
+        assert_eq!(found.bibliographic.as_deref(), Some("eng"));
+        assert_eq!(found.script, "Latin");
+        assert!(found.confidence > 0.0);
+    }
+
+    #[test]
+    fn a_script_only_one_language_uses_needs_no_words() {
+        for (text, code) in [
+            ("안녕하세요 세계 반갑습니다 오늘은", "ko"),
+            ("これは日本語の文章です", "ja"),
+            ("מרחבי שלום עולם וברוך הבא", "he"),
+        ] {
+            assert_eq!(detect("x", text).language.as_deref(), Some(code), "{text}");
+        }
+    }
+
+    #[test]
+    fn nothing_to_read_is_no_answer_rather_than_a_wrong_one() {
+        let found = detect("x", "12345 -- ...");
+        assert_eq!(found.language, None);
+        assert_eq!(found.script, "unknown");
+        assert_eq!(found.notes, ["no letters to go on"]);
+    }
+
+    #[test]
+    fn latin_words_no_language_claims_are_refused() {
+        let found = detect("x", "zzzz qqqq xxxx vvvv wwww kkkk jjjj");
+        assert_eq!(found.language, None);
+        assert!(found.notes[0].starts_with("none of the commonest words"));
+    }
+
+    #[test]
+    fn a_close_call_is_refused_rather_than_guessed() {
+        // "de" is among the commonest words of four of these languages at
+        // once, and on its own settles none of them.
+        let found = detect("x", "de de de");
+        assert_eq!(found.language, None, "{:?}", found.notes);
+        assert!(found.notes[0].contains("too alike"), "{:?}", found.notes);
+    }
+
+    #[test]
+    fn only_the_opening_is_read() {
+        let long = format!(
+            "{}{}",
+            "The obligation ran both ways. ".repeat(2000),
+            "xyzzy"
+        );
+        assert!(long.len() > SAMPLE_CHARS * 2);
+        assert_eq!(detect("x", &long).language.as_deref(), Some("en"));
+    }
+
+    #[test]
+    fn the_script_is_reported_even_when_the_language_is_not() {
+        let found = detect("x", "zzzz qqqq xxxx vvvv");
+        assert_eq!(found.script, "Latin");
+        assert_eq!(found.language, None);
+    }
+}

crates/langid/src/main.rs

@@ -0,0 +1,93 @@
+//! What language a document is in.
+
+use std::path::PathBuf;
+use std::process::ExitCode;
+
+use clap::Parser;
+use corpus_core::exit;
+
+/// Say what language a text is in, one record per file.
+///
+/// The writing system settles most of the world outright; for the Latin and
+/// Cyrillic alphabets the commonest words decide between what is left.
+///
+/// No model, no download, no network. A paragraph is plenty; a single line
+/// often is not, and a close call is refused rather than guessed at.
+#[derive(Parser)]
+#[command(
+    version,
+    about,
+    after_help = "\
+Exit codes: 0 every file was placed, 1 at least one could not be,
+74 a file could not be read.
+
+Example:
+
+    langid book.md
+    langid converted/*.md | jq -r 'select(.language != \"en\") | [.path, .name] | @tsv'"
+)]
+struct Args {
+    /// Files to read. Reads stdin when absent.
+    paths: Vec<PathBuf>,
+
+    /// Print the code alone, for one file.
+    #[arg(long)]
+    quiet: bool,
+}
+
+fn main() -> ExitCode {
+    let args: Args = match catena_cli::parse() {
+        Ok(args) => args,
+        Err(code) => return code,
+    };
+
+    let mut verdicts = Vec::new();
+    let mut unplaced = 0usize;
+
+    if args.paths.is_empty() {
+        let text = match catena_cli::read_lossy(None) {
+            Ok(text) => text,
+            Err((code, why)) => {
+                eprintln!("langid: {why}");
+                return ExitCode::from(code);
+            }
+        };
+        verdicts.push(langid::detect("-", &text));
+    } else {
+        for path in &args.paths {
+            let text = match catena_cli::read_lossy(Some(path)) {
+                Ok(text) => text,
+                Err((code, why)) => {
+                    eprintln!("langid: {why}");
+                    return ExitCode::from(code);
+                }
+            };
+            verdicts.push(langid::detect(&path.display().to_string(), &text));
+        }
+    }
+    unplaced += verdicts.iter().filter(|v| v.language.is_none()).count();
+
+    if args.quiet {
+        let codes: String = verdicts
+            .iter()
+            .map(|v| format!("{}\n", v.language.as_deref().unwrap_or("")))
+            .collect();
+        let wrote = catena_cli::text("langid", &codes);
+        if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
+            return wrote;
+        }
+        return ExitCode::from(match unplaced {
+            0 => exit::OK,
+            _ => exit::NO,
+        });
+    }
+
+    let wrote = catena_cli::stream("langid", &verdicts);
+    if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
+        return wrote;
+    }
+    ExitCode::from(match unplaced {
+        0 => exit::OK,
+        _ => exit::NO,
+    })
+}

crates/langid/src/script.rs

@@ -0,0 +1,175 @@
+//! What writing system a text is in.
+//!
+//! For most of the world this is the whole answer: a text in Hangul is
+//! Korean, a text in Hiragana is Japanese. For the Latin and Cyrillic
+//! alphabets it narrows the field and no more, and the words have to decide.
+
+use serde::{Deserialize, Serialize};
+
+/// A writing system, as far as telling languages apart needs one.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub enum Script {
+    /// The alphabet most of Europe writes in.
+    Latin,
+    /// Russian, Ukrainian, Bulgarian, Serbian and their neighbours.
+    Cyrillic,
+    /// Greek.
+    Greek,
+    /// Arabic, Persian, Urdu.
+    Arabic,
+    /// Hebrew and Yiddish.
+    Hebrew,
+    /// Chinese characters, which Japanese also uses.
+    Han,
+    /// Japanese syllabaries, which only Japanese uses.
+    Kana,
+    /// Korean.
+    Hangul,
+    /// Devanagari: Hindi, Marathi, Sanskrit.
+    Devanagari,
+    /// Something else, or too little to tell.
+    Other,
+}
+
+/// How much of a text has to be in one script before it counts as that
+/// script.
+///
+/// Well under half, because a book carries page numbers, names and quotations
+/// in other alphabets, and an English word in a Russian paragraph does not
+/// make it English.
+const SHARE: f64 = 0.25;
+
+/// The script a text is written in, and what share of its letters agree.
+///
+/// Counts letters only: digits and punctuation belong to every language and
+/// would drown the signal in a text full of citations.
+pub fn detect(text: &str) -> (Script, f64) {
+    let mut counts = [0usize; 10];
+    let mut letters = 0usize;
+
+    for c in text.chars() {
+        let Some(script) = of(c) else { continue };
+        letters += 1;
+        counts[index(script)] += 1;
+    }
+    if letters == 0 {
+        return (Script::Other, 0.0);
+    }
+
+    // Kana before Han: a Japanese text is mostly Han characters with kana
+    // between them, and it is the kana that say which language it is.
+    if counts[index(Script::Kana)] as f64 / letters as f64 > 0.05 {
+        return (
+            Script::Kana,
+            counts[index(Script::Kana)] as f64 / letters as f64,
+        );
+    }
+
+    let (at, count) = counts
+        .iter()
+        .enumerate()
+        .max_by_key(|(_, count)| **count)
+        .expect("ten scripts");
+    let share = *count as f64 / letters as f64;
+    match share >= SHARE {
+        true => (from_index(at), share),
+        false => (Script::Other, share),
+    }
+}
+
+fn index(script: Script) -> usize {
+    match script {
+        Script::Latin => 0,
+        Script::Cyrillic => 1,
+        Script::Greek => 2,
+        Script::Arabic => 3,
+        Script::Hebrew => 4,
+        Script::Han => 5,
+        Script::Kana => 6,
+        Script::Hangul => 7,
+        Script::Devanagari => 8,
+        Script::Other => 9,
+    }
+}
+
+fn from_index(at: usize) -> Script {
+    [
+        Script::Latin,
+        Script::Cyrillic,
+        Script::Greek,
+        Script::Arabic,
+        Script::Hebrew,
+        Script::Han,
+        Script::Kana,
+        Script::Hangul,
+        Script::Devanagari,
+        Script::Other,
+    ][at]
+}
+
+/// The script one character belongs to, or `None` if it is not a letter.
+///
+/// By code point range rather than by a Unicode property table, because the
+/// ranges that matter here are few, stable, and cheaper than carrying the
+/// table.
+pub fn of(c: char) -> Option<Script> {
+    let code = c as u32;
+    Some(match code {
+        0x41..=0x5A | 0x61..=0x7A => Script::Latin,
+        0xC0..=0x24F => Script::Latin,    // Latin-1 through Extended-B
+        0x1E00..=0x1EFF => Script::Latin, // Latin Extended Additional
+        0x370..=0x3FF | 0x1F00..=0x1FFF => Script::Greek,
+        0x400..=0x52F => Script::Cyrillic,
+        0x590..=0x5FF => Script::Hebrew,
+        0x600..=0x6FF | 0x750..=0x77F => Script::Arabic,
+        0x900..=0x97F => Script::Devanagari,
+        0x3040..=0x30FF => Script::Kana, // hiragana and katakana, adjacent
+        0x4E00..=0x9FFF | 0x3400..=0x4DBF => Script::Han,
+        0xAC00..=0xD7AF | 0x1100..=0x11FF => Script::Hangul,
+        _ => return None,
+    })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn each_script_is_recognised_from_its_own_words() {
+        for (text, script) in [
+            ("The obligation ran both ways", Script::Latin),
+            ("Молчание было единственным ответом", Script::Cyrillic),
+            ("Καλημέρα κόσμε καὶ πάντες", Script::Greek),
+            ("مرحبا بالعالم وأهلا", Script::Arabic),
+            ("שלום עולם וברוך", Script::Hebrew),
+            ("안녕하세요 세계 반갑습니다", Script::Hangul),
+            ("नमस्ते दुनिया सब", Script::Devanagari),
+        ] {
+            assert_eq!(detect(text).0, script, "{text}");
+        }
+    }
+
+    #[test]
+    fn kana_settles_japanese_against_chinese() {
+        // Both are mostly Han characters. The kana between them are the tell.
+        assert_eq!(detect("これは日本語の文章です").0, Script::Kana);
+        assert_eq!(detect("這是中文的句子沒有假名").0, Script::Han);
+    }
+
+    #[test]
+    fn a_stray_foreign_word_does_not_change_the_script() {
+        let mostly = "Молчание было единственным ответом на его вопрос about that";
+        assert_eq!(detect(mostly).0, Script::Cyrillic);
+    }
+
+    #[test]
+    fn digits_and_punctuation_are_not_evidence() {
+        assert_eq!(detect("123 456 -- ... 789").0, Script::Other);
+        assert_eq!(detect("").0, Script::Other);
+    }
+
+    #[test]
+    fn accented_latin_is_still_latin() {
+        assert_eq!(detect("naïve café résumé Grüße piñata").0, Script::Latin);
+    }
+}

crates/langid/src/words.rs

@@ -0,0 +1,355 @@
+//! Telling apart the languages that share an alphabet, by their commonest
+//! words.
+//!
+//! Function words — articles, prepositions, conjunctions, pronouns — make up
+//! a third of any running text and differ sharply between languages even
+//! closely related ones. Counting them needs no model, no training and no
+//! network, and on a paragraph or more it is decisive.
+//!
+//! It is weak on a single line, which is why what comes out carries a margin
+//! and a caller can refuse a close call.
+
+use crate::script::Script;
+
+/// One language, and the words that give it away.
+pub struct Language {
+    /// ISO 639-1, the two-letter code everything else uses.
+    pub code: &'static str,
+    /// ISO 639-2/B, for the `doc` record's language field.
+    pub bibliographic: &'static str,
+    /// What it is called in English.
+    pub name: &'static str,
+    /// The script it is written in.
+    pub script: Script,
+    /// Its commonest function words, lowercased.
+    pub common: &'static [&'static str],
+}
+
+/// The languages this can tell apart.
+///
+/// General reference data, of the same kind as the HTML character references
+/// and the ISBN check digit: true everywhere, not a fact about one
+/// collection.
+pub const LANGUAGES: &[Language] = &[
+    Language {
+        code: "en",
+        bibliographic: "eng",
+        name: "English",
+        script: Script::Latin,
+        common: &[
+            "the", "of", "and", "to", "in", "that", "is", "was", "it", "for", "with", "as", "his",
+            "on", "be", "at", "by", "this", "had", "not", "are", "but", "from", "or", "have", "an",
+            "they", "which", "were", "their",
+        ],
+    },
+    Language {
+        code: "fr",
+        bibliographic: "fre",
+        name: "French",
+        script: Script::Latin,
+        common: &[
+            "le", "de", "un", "et", "être", "à", "il", "avoir", "ne", "je", "son", "que", "se",
+            "qui", "ce", "dans", "en", "du", "elle", "au", "pour", "pas", "que", "vous", "par",
+            "sur", "les", "des", "une", "plus",
+        ],
+    },
+    Language {
+        code: "de",
+        bibliographic: "ger",
+        name: "German",
+        script: Script::Latin,
+        common: &[
+            "der", "die", "und", "in", "den", "von", "zu", "das", "mit", "sich", "des", "auf",
+            "für", "ist", "im", "dem", "nicht", "ein", "eine", "als", "auch", "es", "an", "werden",
+            "aus", "er", "hat", "dass", "sie", "nach",
+        ],
+    },
+    Language {
+        code: "es",
+        bibliographic: "spa",
+        name: "Spanish",
+        script: Script::Latin,
+        common: &[
+            "de", "la", "que", "el", "en", "y", "a", "los", "del", "se", "las", "por", "un",
+            "para", "con", "no", "una", "su", "al", "es", "lo", "como", "más", "pero", "sus", "le",
+            "ya", "o", "este", "sí",
+        ],
+    },
+    Language {
+        code: "it",
+        bibliographic: "ita",
+        name: "Italian",
+        script: Script::Latin,
+        common: &[
+            "di", "che", "e", "il", "la", "in", "un", "a", "per", "non", "una", "è", "con", "si",
+            "da", "le", "del", "i", "sono", "come", "al", "della", "nel", "più", "lo", "ma", "se",
+            "gli", "alla", "anche",
+        ],
+    },
+    Language {
+        code: "pt",
+        bibliographic: "por",
+        name: "Portuguese",
+        script: Script::Latin,
+        common: &[
+            "de", "a", "o", "que", "e", "do", "da", "em", "um", "para", "com", "não", "uma", "os",
+            "no", "se", "na", "por", "mais", "as", "dos", "como", "mas", "ao", "ele", "das", "seu",
+            "sua", "ou", "quando",
+        ],
+    },
+    Language {
+        code: "nl",
+        bibliographic: "dut",
+        name: "Dutch",
+        script: Script::Latin,
+        common: &[
+            "de", "van", "het", "een", "en", "in", "is", "dat", "op", "te", "zijn", "met", "voor",
+            "niet", "aan", "er", "om", "maar", "dan", "of", "door", "over", "ze", "bij", "ook",
+            "tot", "uit", "naar", "heeft", "werd",
+        ],
+    },
+    Language {
+        code: "la",
+        bibliographic: "lat",
+        name: "Latin",
+        script: Script::Latin,
+        common: &[
+            "et", "in", "est", "non", "ad", "cum", "qui", "quod", "ut", "sed", "de", "si", "ex",
+            "quae", "per", "hoc", "esse", "enim", "autem", "nec", "atque", "sunt", "eius", "ab",
+            "quam", "tamen", "ita", "etiam", "ipse", "sive",
+        ],
+    },
+    Language {
+        code: "pl",
+        bibliographic: "pol",
+        name: "Polish",
+        script: Script::Latin,
+        common: &[
+            "w", "i", "z", "na", "się", "do", "nie", "to", "że", "jest", "o", "a", "od", "po",
+            "za", "jak", "ale", "tym", "tego", "przez", "przy", "dla", "co", "który", "była",
+            "były", "być", "ma", "już", "tylko",
+        ],
+    },
+    Language {
+        code: "sv",
+        bibliographic: "swe",
+        name: "Swedish",
+        script: Script::Latin,
+        common: &[
+            "och", "att", "det", "som", "en", "på", "är", "av", "för", "med", "till", "den", "har",
+            "de", "inte", "om", "ett", "han", "men", "var", "från", "vi", "så", "kan", "man",
+            "när", "hon", "sig", "eller", "efter",
+        ],
+    },
+    Language {
+        code: "ru",
+        bibliographic: "rus",
+        name: "Russian",
+        script: Script::Cyrillic,
+        common: &[
+            "и", "в", "не", "на", "что", "он", "с", "как", "это", "по", "но", "они", "к", "у",
+            "же", "за", "из", "от", "то", "все", "был", "была", "было", "так", "его", "её", "для",
+            "бы", "или", "мы",
+        ],
+    },
+    Language {
+        code: "uk",
+        bibliographic: "ukr",
+        name: "Ukrainian",
+        script: Script::Cyrillic,
+        common: &[
+            "і", "в", "не", "на", "що", "з", "як", "до", "це", "за", "від", "але", "той", "його",
+            "вона", "там", "бути", "має", "для", "та", "він", "ми", "ви", "чи", "або", "щоб",
+            "коли", "де", "так", "ще",
+        ],
+    },
+    Language {
+        code: "el",
+        bibliographic: "gre",
+        name: "Greek",
+        script: Script::Greek,
+        common: &[
+            "και",
+            "το",
+            "της",
+            "των",
+            "του",
+            "στο",
+            "με",
+            "για",
+            "που",
+            "στη",
+            "είναι",
+            "από",
+            "σε",
+            "τα",
+            "ο",
+            "η",
+            "δεν",
+            "να",
+            "ως",
+            "κατά",
+            "μια",
+            "ένα",
+            "αλλά",
+            "αυτό",
+            "όπως",
+            "θα",
+            "τον",
+            "την",
+            "οι",
+            "στον",
+        ],
+    },
+];
+
+/// How far ahead the best language must be before the answer is worth giving.
+///
+/// A ratio rather than a difference, so it means the same on a paragraph and
+/// on a book.
+pub const MARGIN: f64 = 1.25;
+
+/// Score every language whose script matches, best first.
+///
+/// The score is the share of a text's words that are among a language's
+/// commonest. It is comparable across languages because each list is the same
+/// length.
+pub fn score(text: &str, script: Script) -> Vec<(&'static Language, f64)> {
+    let words = tokens(text);
+    if words.is_empty() {
+        return Vec::new();
+    }
+
+    let mut scored: Vec<(&'static Language, f64)> = LANGUAGES
+        .iter()
+        .filter(|language| language.script == script)
+        .map(|language| {
+            let hits = words
+                .iter()
+                .filter(|word| language.common.contains(&word.as_str()))
+                .count();
+            (language, hits as f64 / words.len() as f64)
+        })
+        .collect();
+
+    scored.sort_by(|a, b| b.1.total_cmp(&a.1));
+    scored
+}
+
+/// A text's words, lowercased, with everything that is not a letter dropped.
+fn tokens(text: &str) -> Vec<String> {
+    text.split(|c: char| crate::script::of(c).is_none())
+        .filter(|word| !word.is_empty())
+        .map(|word| word.to_lowercase())
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn best(text: &str, script: Script) -> Option<&'static str> {
+        score(text, script)
+            .first()
+            .map(|(language, _)| language.code)
+    }
+
+    #[test]
+    fn each_language_is_recognised_from_a_sentence_of_it() {
+        for (text, code) in [
+            (
+                "The obligation ran both ways, and the lord who could not protect was not a lord at all in that time",
+                "en",
+            ),
+            (
+                "Le seigneur qui ne pouvait pas protéger n'était pas un seigneur, et c'est ce que les rôles disent de lui",
+                "fr",
+            ),
+            (
+                "Der Herr, der nicht schützen konnte, war kein Herr, und das ist es, was die Urkunden von ihm sagen",
+                "de",
+            ),
+            (
+                "El señor que no podía proteger no era un señor, y eso es lo que dicen los registros de él en la época",
+                "es",
+            ),
+            (
+                "Il signore che non poteva proteggere non era un signore, e questo è ciò che dicono i documenti di lui",
+                "it",
+            ),
+            (
+                "O senhor que não podia proteger não era um senhor, e é isso que os registos dizem dele naquele tempo",
+                "pt",
+            ),
+            (
+                "De heer die niet kon beschermen was geen heer, en dat is wat de bronnen over hem zeggen in die tijd",
+                "nl",
+            ),
+            (
+                "Dominus qui non poterat protegere non erat dominus, et hoc est quod de eo dicunt scripta illius aetatis",
+                "la",
+            ),
+        ] {
+            assert_eq!(best(text, Script::Latin), Some(code), "{text}");
+        }
+    }
+
+    #[test]
+    fn the_cyrillic_pair_are_told_apart() {
+        assert_eq!(
+            best(
+                "Молчание было единственным ответом на его вопрос, и это всё что он услышал",
+                Script::Cyrillic
+            ),
+            Some("ru")
+        );
+        assert_eq!(
+            best(
+                "Мовчання було єдиною відповіддю на його запитання, і це все що він почув від них",
+                Script::Cyrillic
+            ),
+            Some("uk")
+        );
+    }
+
+    #[test]
+    fn greek_is_read_as_greek() {
+        assert_eq!(
+            best(
+                "Και το φως εν τη σκοτία φαίνει και η σκοτία αυτό δεν κατέλαβεν από την αρχή",
+                Script::Greek
+            ),
+            Some("el")
+        );
+    }
+
+    #[test]
+    fn a_text_with_no_words_scores_nothing() {
+        assert!(score("", Script::Latin).is_empty());
+        assert!(score("123 456", Script::Latin).is_empty());
+    }
+
+    #[test]
+    fn only_languages_in_the_right_script_are_considered() {
+        let latin = score("the and of", Script::Latin);
+        assert!(
+            latin
+                .iter()
+                .all(|(language, _)| language.script == Script::Latin)
+        );
+        assert!(!latin.is_empty());
+    }
+
+    #[test]
+    fn every_language_has_the_same_number_of_words_so_scores_compare() {
+        let first = LANGUAGES[0].common.len();
+        for language in LANGUAGES {
+            assert_eq!(
+                language.common.len(),
+                first,
+                "{} has a different list",
+                language.code
+            );
+        }
+    }
+}

crates/tocfind/Cargo.toml

@@ -0,0 +1,21 @@
+[package]
+name = "tocfind"
+description = "A book's own table of contents, out of its navigation document or outline."
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+repository.workspace = true
+
+[dependencies]
+catena-cli.workspace = true
+clap.workspace = true
+corpus-core.workspace = true
+lopdf.workspace = true
+schemars.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+zip.workspace = true
+
+[lints]
+workspace = true

crates/tocfind/README.md

@@ -0,0 +1,137 @@
+# tocfind
+
+A book's own table of contents.
+
+Part of [catena](../../README.md).
+
+## What it does
+
+A book usually knows its own chapters. An EPUB keeps them in a navigation
+document, a PDF in an outline, and both are put there by the publisher — which
+makes them better than any heading guessed at from converted text.
+
+Entries come out in reading order, each carrying how deeply it is nested, so a
+flat stream still has the shape of the book.
+
+## When you would use it
+
+- **Real chapter titles** for chunks, instead of the extractive ones inferred
+  from converted markdown.
+- **Splitting a book** at the seams its publisher declared.
+- **Judging a conversion**: a book whose contents list forty chapters and
+  whose markdown has three was converted badly.
+
+## Install
+
+```sh
+cargo install --git https://git.pumpin.online/funkpower/catena tocfind
+```
+
+## Example
+
+```
+$ tocfind enlightenment.epub --outline
+Cover page
+Halftitle page
+Title page
+Copyright page
+Contents
+1. Introduction: Radical Enlightenment and “Modernity” (1650–1850)
+  1.1. Basic Argument and Scope
+  1.2. Definitions and Categories
+```
+
+As records:
+
+```
+$ tocfind enlightenment.epub | head -2
+{"v":1,"idx":0,"depth":0,"title":"Cover page","target":"000_ACover.xhtml"}
+{"v":1,"idx":1,"depth":0,"title":"Halftitle page","target":"002_FM_halftitle.xhtml"}
+```
+
+A PDF's outline carries page numbers instead of documents:
+
+```
+$ tocfind history.pdf --outline
+Part One  (1)
+  I. The Setting  (3)
+  II. The Second Estate  (41)
+Part Two  (118)
+```
+
+Only the real chapters, skipping the front matter every book opens with:
+
+```sh
+tocfind book.epub | jq -r 'select(.depth == 0) | .title' | tail -n +5
+```
+
+The books whose conversion lost the structure:
+
+```sh
+for f in *.epub; do
+  declared=$(tocfind "$f" | wc -l)
+  found=$(grep -c '^# ' "converted/${f%.epub}.md")
+  [ "$declared" -gt $(( found * 3 )) ] && echo "$f: $declared entries, $found headings"
+done
+```
+
+## Options
+
+| Option | Effect |
+|---|---|
+| `--outline` | Print an indented list rather than records |
+| `--quiet`, `-q` | Say nothing; the exit code is the answer |
+| `--man` | Write this page as a manual page, in roff |
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | It had a table of contents |
+| `1` | It had none |
+| `64` | The command line was wrong |
+| `65` | Not a format that keeps one |
+| `74` | The file could not be read |
+
+Having none is an answer; not being able to have one is not, which is why
+those are different codes.
+
+## Where it reads from
+
+| Format | Where |
+|---|---|
+| EPUB | The NCX, or an EPUB 3 `<nav epub:type="toc">` |
+| PDF | The catalogue's outline |
+
+**The NCX first**, even for books that declare themselves EPUB 3. It is what
+books actually carry — every one of thirteen real books sampled had one and
+none had a nav document — and its labels are plain text where a nav
+document's are markup.
+
+From a nav document only the contents are read, never the page list or the
+landmarks, and markup inside a title is not part of the title.
+
+A PDF outline entry resolves to a page where its destination names one
+directly. A named destination is not chased: that is a second index, and an
+entry without a page is still an entry.
+
+## As a library
+
+```rust
+match tocfind::read(std::path::Path::new("book.epub")) {
+    Ok(entries) => for entry in entries {
+        println!("{}{}", "  ".repeat(entry.depth as usize), entry.title);
+    },
+    Err(why) => eprintln!("no contents: {why}"),
+}
+```
+
+`tocfind::epub::from_ncx` and `from_nav` read a navigation document already in
+hand.
+
+## See also
+
+- [`mdsplit`](../mdsplit/README.md) — splitting a collection by its seams.
+- [`chunkmd`](../chunkmd/README.md) — which infers a chapter when the book
+  does not say.
+- [`bookmeta`](../bookmeta/README.md) — the rest of what a file carries.

crates/tocfind/src/epub.rs

@@ -0,0 +1,287 @@
+//! An EPUB's navigation document.
+//!
+//! Two shapes, because the format changed. EPUB 3 keeps a `<nav>` of nested
+//! `<ol>` lists; EPUB 2 keeps an NCX of nested `<navPoint>` elements. Every
+//! real book in a library has the second, including most that declare
+//! themselves EPUB 3, because converters keep writing it for readers that
+//! never caught up.
+
+use std::io::Read;
+
+use crate::{Entry, Missing, number, tidy};
+
+/// How large a navigation document is worth reading.
+const MAX_NAV: u64 = 4 * 1024 * 1024;
+
+/// Read an EPUB's table of contents.
+pub fn read(bytes: &[u8]) -> Result<Vec<Entry>, Missing> {
+    let cursor = std::io::Cursor::new(bytes);
+    let mut archive =
+        zip::ZipArchive::new(cursor).map_err(|e| Missing::Unreadable(e.to_string()))?;
+
+    let names: Vec<String> = (0..archive.len())
+        .filter_map(|at| {
+            archive
+                .by_index(at)
+                .ok()
+                .map(|file| file.name().to_string())
+        })
+        .collect();
+
+    // The NCX first: it is what books actually carry, and its labels are
+    // plain text where a nav document's are markup.
+    if let Some(name) = names
+        .iter()
+        .find(|name| name.to_lowercase().ends_with(".ncx"))
+    {
+        if let Some(text) = member(&mut archive, name) {
+            let mut entries = from_ncx(&text);
+            if !entries.is_empty() {
+                number(&mut entries);
+                return Ok(entries);
+            }
+        }
+    }
+    for name in names.iter().filter(|name| {
+        let lower = name.to_lowercase();
+        lower.ends_with("nav.xhtml") || lower.ends_with("nav.html") || lower.ends_with("toc.xhtml")
+    }) {
+        if let Some(text) = member(&mut archive, name) {
+            let mut entries = from_nav(&text);
+            if !entries.is_empty() {
+                number(&mut entries);
+                return Ok(entries);
+            }
+        }
+    }
+    Err(Missing::None("navigation document"))
+}
+
+fn member<R: Read + std::io::Seek>(archive: &mut zip::ZipArchive<R>, name: &str) -> Option<String> {
+    let file = archive.by_name(name).ok()?;
+    if file.size() > MAX_NAV {
+        return None;
+    }
+    let mut text = String::new();
+    file.take(MAX_NAV).read_to_string(&mut text).ok()?;
+    Some(text)
+}
+
+/// Walk an NCX, where depth is how deeply `<navPoint>` elements nest.
+pub fn from_ncx(ncx: &str) -> Vec<Entry> {
+    let mut entries = Vec::new();
+    let mut depth: u32 = 0;
+    let mut rest = ncx;
+
+    // Element by element rather than with a parser: the shape is three tags
+    // deep and a parser would be the largest dependency in the crate.
+    while let Some(at) = rest.find('<') {
+        let after = &rest[at..];
+        let Some(end) = after.find('>') else { break };
+        let tag = &after[1..end];
+
+        if tag.starts_with("navPoint") && !tag.starts_with("navPoint/") {
+            let point = &after[end + 1..];
+            if let Some(title) = between(point, "<text>", "</text>").map(tidy) {
+                if !title.is_empty() {
+                    entries.push(Entry {
+                        v: corpus_core::V,
+                        idx: 0,
+                        depth,
+                        title,
+                        target: between(point, "src=\"", "\"")
+                            .or_else(|| between(point, "src='", "'"))
+                            .map(|src| src.to_string()),
+                        page: None,
+                    });
+                }
+            }
+            // A self-closing navPoint has no children and no depth to add.
+            if !tag.ends_with('/') {
+                depth += 1;
+            }
+        } else if tag.starts_with("/navPoint") {
+            depth = depth.saturating_sub(1);
+        }
+        rest = &after[end + 1..];
+    }
+    entries
+}
+
+/// Walk an EPUB 3 navigation document, where depth is `<ol>` nesting.
+pub fn from_nav(nav: &str) -> Vec<Entry> {
+    // Only the table of contents, not the page list or the landmarks.
+    let scope = match nav
+        .find("epub:type=\"toc\"")
+        .or_else(|| nav.find("epub:type='toc'"))
+    {
+        Some(at) => {
+            let from = &nav[at..];
+            let end = from.find("</nav>").map_or(from.len(), |end| end + 6);
+            &from[..end]
+        }
+        None => nav,
+    };
+
+    let mut entries = Vec::new();
+    let mut depth: i64 = -1;
+    let mut rest = scope;
+
+    while let Some(at) = rest.find('<') {
+        let after = &rest[at..];
+        let Some(end) = after.find('>') else { break };
+        let tag = &after[1..end];
+        let lower = tag.to_lowercase();
+
+        if lower.starts_with("ol") && !lower.starts_with("ol/") {
+            depth += 1;
+        } else if lower.starts_with("/ol") {
+            depth -= 1;
+        } else if lower.starts_with("a ") || lower.starts_with("a>") {
+            let body = &after[end + 1..];
+            if let Some(close) = body.find("</a>") {
+                let title = tidy(&strip_tags(&body[..close]));
+                if !title.is_empty() {
+                    entries.push(Entry {
+                        v: corpus_core::V,
+                        idx: 0,
+                        depth: depth.max(0) as u32,
+                        title,
+                        target: attribute(tag, "href"),
+                        page: None,
+                    });
+                }
+            }
+        }
+        rest = &after[end + 1..];
+    }
+    entries
+}
+
+/// What sits between two markers.
+fn between<'a>(text: &'a str, open: &str, close: &str) -> Option<&'a str> {
+    let start = text.find(open)? + open.len();
+    let end = text[start..].find(close)? + start;
+    Some(&text[start..end])
+}
+
+/// An attribute out of one tag's text.
+fn attribute(tag: &str, name: &str) -> Option<String> {
+    for quote in ['"', '\''] {
+        let needle = format!("{name}={quote}");
+        if let Some(at) = tag.find(&needle) {
+            let rest = &tag[at + needle.len()..];
+            if let Some(end) = rest.find(quote) {
+                return Some(rest[..end].to_string());
+            }
+        }
+    }
+    None
+}
+
+/// Inner text, with the markup a title is wrapped in removed.
+fn strip_tags(text: &str) -> String {
+    let mut out = String::with_capacity(text.len());
+    let mut inside = false;
+    for c in text.chars() {
+        match c {
+            '<' => inside = true,
+            '>' => inside = false,
+            c if !inside => out.push(c),
+            _ => {}
+        }
+    }
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const NCX: &str = r#"<ncx><navMap>
+  <navPoint id="a" playOrder="1">
+    <navLabel><text>Part One</text></navLabel>
+    <content src="part1.xhtml"/>
+    <navPoint id="b" playOrder="2">
+      <navLabel><text>I. The Setting</text></navLabel>
+      <content src="ch1.xhtml"/>
+    </navPoint>
+    <navPoint id="c" playOrder="3">
+      <navLabel><text>II. The Second Estate</text></navLabel>
+      <content src="ch2.xhtml"/>
+    </navPoint>
+  </navPoint>
+  <navPoint id="d" playOrder="4">
+    <navLabel><text>Part Two</text></navLabel>
+    <content src="part2.xhtml"/>
+  </navPoint>
+</navMap></ncx>"#;
+
+    #[test]
+    fn nesting_becomes_depth() {
+        let entries = from_ncx(NCX);
+        assert_eq!(
+            entries
+                .iter()
+                .map(|e| (e.depth, e.title.as_str()))
+                .collect::<Vec<_>>(),
+            [
+                (0, "Part One"),
+                (1, "I. The Setting"),
+                (1, "II. The Second Estate"),
+                (0, "Part Two"),
+            ]
+        );
+    }
+
+    #[test]
+    fn each_entry_keeps_where_it_points() {
+        assert_eq!(from_ncx(NCX)[1].target.as_deref(), Some("ch1.xhtml"));
+    }
+
+    #[test]
+    fn a_nav_document_nests_by_its_lists() {
+        let nav = r#"<nav epub:type="toc"><ol>
+          <li><a href="p1.xhtml">Part One</a>
+            <ol><li><a href="c1.xhtml">I. The Setting</a></li></ol>
+          </li>
+          <li><a href="p2.xhtml">Part Two</a></li>
+        </ol></nav>"#;
+        let entries = from_nav(nav);
+        assert_eq!(
+            entries
+                .iter()
+                .map(|e| (e.depth, e.title.as_str()))
+                .collect::<Vec<_>>(),
+            [(0, "Part One"), (1, "I. The Setting"), (0, "Part Two")]
+        );
+    }
+
+    #[test]
+    fn only_the_contents_are_read_and_not_the_page_list() {
+        let nav = r#"
+          <nav epub:type="toc"><ol><li><a href="c1.xhtml">Chapter One</a></li></ol></nav>
+          <nav epub:type="page-list"><ol><li><a href="c1.xhtml#p1">1</a></li></ol></nav>"#;
+        let entries = from_nav(nav);
+        assert_eq!(entries.len(), 1);
+        assert_eq!(entries[0].title, "Chapter One");
+    }
+
+    #[test]
+    fn markup_inside_a_title_is_not_part_of_the_title() {
+        let nav = r#"<nav epub:type="toc"><ol><li><a href="c.xhtml">
+          <span class="num">1</span> <em>The Setting</em></a></li></ol></nav>"#;
+        assert_eq!(from_nav(nav)[0].title, "1 The Setting");
+    }
+
+    #[test]
+    fn an_entry_with_no_label_is_not_an_entry() {
+        let ncx = "<navMap><navPoint><navLabel><text>  </text></navLabel></navPoint></navMap>";
+        assert!(from_ncx(ncx).is_empty());
+    }
+
+    #[test]
+    fn what_is_not_a_zip_says_so() {
+        assert!(matches!(read(b"not a zip"), Err(Missing::Unreadable(_))));
+    }
+}

crates/tocfind/src/lib.rs

@@ -0,0 +1,132 @@
+//! A book's own table of contents.
+//!
+//! A book usually knows its own chapters. An EPUB carries them in a
+//! navigation document, a PDF in an outline, and both are put there by the
+//! publisher — which makes them better than any heading a converter guesses
+//! at afterwards, because they are what the book itself says its parts are.
+//!
+//! What comes back is the entries in reading order, each with how deeply it
+//! is nested, so a flat stream still carries the shape of the book.
+
+pub mod epub;
+pub mod pdf;
+
+use schemars::JsonSchema;
+use serde::{Deserialize, Serialize};
+
+/// One entry of a table of contents.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
+pub struct Entry {
+    /// Record version.
+    #[serde(default = "version")]
+    pub v: u32,
+    /// Position in reading order, from zero.
+    pub idx: u32,
+    /// How deeply nested: nought for a part, one for a chapter under it.
+    pub depth: u32,
+    /// What the book calls it.
+    pub title: String,
+    /// Where it points: a document within the archive, or a page number.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub target: Option<String>,
+    /// The page it begins on, where the source counts in pages.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub page: Option<u32>,
+}
+
+fn version() -> u32 {
+    corpus_core::V
+}
+
+impl corpus_core::Record for Entry {
+    fn version(&self) -> u32 {
+        self.v
+    }
+}
+
+/// Why a file has no table of contents to read.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Missing {
+    /// The format keeps one, and this file has none.
+    None(&'static str),
+    /// Not a format that keeps one.
+    Unsupported,
+    /// The file could not be opened or parsed.
+    Unreadable(String),
+}
+
+impl std::fmt::Display for Missing {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Missing::None(where_) => write!(f, "no {where_}"),
+            Missing::Unsupported => write!(f, "not a format that carries a table of contents"),
+            Missing::Unreadable(why) => write!(f, "{why}"),
+        }
+    }
+}
+
+impl std::error::Error for Missing {}
+
+/// Read a file's table of contents.
+pub fn read(path: &std::path::Path) -> Result<Vec<Entry>, Missing> {
+    let bytes = std::fs::read(path).map_err(|e| Missing::Unreadable(e.to_string()))?;
+    from_bytes(&bytes)
+}
+
+/// The same, for bytes already in hand.
+pub fn from_bytes(bytes: &[u8]) -> Result<Vec<Entry>, Missing> {
+    if bytes.starts_with(b"PK\x03\x04") {
+        return epub::read(bytes);
+    }
+    if bytes.starts_with(b"%PDF") {
+        return pdf::read(bytes);
+    }
+    Err(Missing::Unsupported)
+}
+
+/// Number the entries in reading order.
+///
+/// Done once at the end rather than while walking, so that a reader of a
+/// nested structure does not have to thread a counter through it.
+pub fn number(entries: &mut [Entry]) {
+    for (at, entry) in entries.iter_mut().enumerate() {
+        entry.idx = at as u32;
+    }
+}
+
+/// Collapse runs of whitespace, which navigation documents are full of.
+pub fn tidy(text: &str) -> String {
+    text.split_whitespace().collect::<Vec<_>>().join(" ")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn a_format_with_nowhere_to_keep_one_says_so() {
+        assert_eq!(from_bytes(b"just some text"), Err(Missing::Unsupported));
+    }
+
+    #[test]
+    fn entries_are_numbered_in_reading_order() {
+        let mut entries: Vec<Entry> = (0..3)
+            .map(|_| Entry {
+                v: corpus_core::V,
+                idx: 99,
+                depth: 0,
+                title: "x".into(),
+                target: None,
+                page: None,
+            })
+            .collect();
+        number(&mut entries);
+        assert_eq!(entries.iter().map(|e| e.idx).collect::<Vec<_>>(), [0, 1, 2]);
+    }
+
+    #[test]
+    fn whitespace_in_a_title_is_collapsed() {
+        assert_eq!(tidy("  Chapter\n  One  "), "Chapter One");
+        assert_eq!(tidy(""), "");
+    }
+}

crates/tocfind/src/main.rs

@@ -0,0 +1,81 @@
+//! A book's own table of contents.
+
+use std::path::PathBuf;
+use std::process::ExitCode;
+
+use clap::Parser;
+use corpus_core::exit;
+
+/// Read a book's table of contents, one record per entry.
+///
+/// A book usually knows its own chapters: an EPUB keeps them in a navigation
+/// document, a PDF in an outline. Both are put there by the publisher, which
+/// makes them better than any heading guessed at from converted text.
+///
+/// Entries come out in reading order, each carrying how deeply it is nested,
+/// so a flat stream still has the shape of the book.
+#[derive(Parser)]
+#[command(
+    version,
+    about,
+    after_help = "\
+Exit codes: 0 it had one, 1 it had none, 65 it is not a format that keeps one,
+74 the file could not be read.
+
+Example:
+
+    tocfind book.epub | jq -r '(\"  \" * .depth) + .title'
+    for f in *.epub; do tocfind --quiet \"$f\" || echo \"no contents: $f\"; done"
+)]
+struct Args {
+    /// The book to read.
+    path: PathBuf,
+
+    /// Say nothing; the exit code is the answer.
+    #[arg(long, short)]
+    quiet: bool,
+
+    /// Print the contents as an indented list rather than as records.
+    #[arg(long)]
+    outline: bool,
+}
+
+fn main() -> ExitCode {
+    let args: Args = match catena_cli::parse() {
+        Ok(args) => args,
+        Err(code) => return code,
+    };
+
+    let entries = match tocfind::read(&args.path) {
+        Ok(entries) => entries,
+        Err(why) => {
+            if !args.quiet {
+                eprintln!("tocfind: {}: {why}", args.path.display());
+            }
+            return ExitCode::from(match why {
+                // Having none is an answer; not being able to have one is not.
+                tocfind::Missing::None(_) => exit::NO,
+                tocfind::Missing::Unsupported => exit::DATA,
+                tocfind::Missing::Unreadable(_) => exit::IO,
+            });
+        }
+    };
+
+    if args.quiet {
+        return ExitCode::from(exit::OK);
+    }
+    if args.outline {
+        let text: String = entries
+            .iter()
+            .map(|entry| {
+                let indent = "  ".repeat(entry.depth as usize);
+                match entry.page {
+                    Some(page) => format!("{indent}{}  ({page})\n", entry.title),
+                    None => format!("{indent}{}\n", entry.title),
+                }
+            })
+            .collect();
+        return catena_cli::text("tocfind", &text);
+    }
+    catena_cli::stream("tocfind", &entries)
+}

crates/tocfind/src/pdf.rs

@@ -0,0 +1,240 @@
+//! A PDF's outline, which is what a reader shows in its sidebar.
+//!
+//! The catalogue names an outline whose items nest, each with a title and a
+//! destination. Resolving a destination to a page number means finding which
+//! page object it names, which is worth doing: a chapter that knows its page
+//! can be cited, and one that knows only an object number cannot.
+
+use lopdf::{Document, Object, ObjectId};
+
+use crate::{Entry, Missing, number, tidy};
+
+/// How deep an outline is followed before something is assumed to be wrong.
+const MAX_DEPTH: u32 = 16;
+
+/// How many items are read before stopping.
+///
+/// A real outline is hundreds; a malformed one that loops is unbounded, and
+/// the depth cap alone does not catch a cycle among siblings.
+const MAX_ITEMS: usize = 10_000;
+
+/// Read a PDF's outline.
+pub fn read(bytes: &[u8]) -> Result<Vec<Entry>, Missing> {
+    let document = Document::load_mem(bytes).map_err(|e| Missing::Unreadable(e.to_string()))?;
+    if document.is_encrypted() {
+        return Err(Missing::Unreadable("it is encrypted".into()));
+    }
+
+    let pages: Vec<ObjectId> = document.get_pages().into_values().collect();
+    let Some(root) = outline_root(&document) else {
+        return Err(Missing::None("outline"));
+    };
+
+    let mut entries = Vec::new();
+    let mut seen: Vec<ObjectId> = Vec::new();
+    walk(&document, &pages, root, 0, &mut entries, &mut seen);
+
+    match entries.is_empty() {
+        true => Err(Missing::None("outline entries")),
+        false => {
+            number(&mut entries);
+            Ok(entries)
+        }
+    }
+}
+
+/// The first item of the catalogue's outline.
+fn outline_root(document: &Document) -> Option<ObjectId> {
+    let outlines = document.catalog().ok()?.get(b"Outlines").ok()?;
+    let (_, object) = document.dereference(outlines).ok()?;
+    object
+        .as_dict()
+        .ok()?
+        .get(b"First")
+        .ok()?
+        .as_reference()
+        .ok()
+}
+
+/// Follow a chain of siblings, descending into each one's children.
+fn walk(
+    document: &Document,
+    pages: &[ObjectId],
+    first: ObjectId,
+    depth: u32,
+    entries: &mut Vec<Entry>,
+    seen: &mut Vec<ObjectId>,
+) {
+    if depth > MAX_DEPTH {
+        return;
+    }
+    let mut at = Some(first);
+    while let Some(id) = at {
+        if entries.len() >= MAX_ITEMS || seen.contains(&id) {
+            return;
+        }
+        seen.push(id);
+
+        let Ok(item) = document.get_dictionary(id) else {
+            return;
+        };
+        if let Some(title) = item.get(b"Title").ok().and_then(|t| text(document, t)) {
+            let title = tidy(&title);
+            if !title.is_empty() {
+                entries.push(Entry {
+                    v: corpus_core::V,
+                    idx: 0,
+                    depth,
+                    title,
+                    target: None,
+                    page: page_of(document, pages, item),
+                });
+            }
+        }
+
+        if let Some(child) = item.get(b"First").ok().and_then(|f| f.as_reference().ok()) {
+            walk(document, pages, child, depth + 1, entries, seen);
+        }
+        at = item
+            .get(b"Next")
+            .ok()
+            .and_then(|next| next.as_reference().ok());
+    }
+}
+
+/// Which page an item points at, counting from one.
+///
+/// A destination is either an array beginning with a page reference or a
+/// named one this does not chase, because a name table is a second index and
+/// an entry without a page is still an entry.
+fn page_of(document: &Document, pages: &[ObjectId], item: &lopdf::Dictionary) -> Option<u32> {
+    let destination = match item.get(b"Dest") {
+        Ok(dest) => dest.clone(),
+        Err(_) => {
+            // An action rather than a destination: `/A << /D [...] >>`.
+            let action = item.get(b"A").ok()?;
+            let (_, action) = document.dereference(action).ok()?;
+            action.as_dict().ok()?.get(b"D").ok()?.clone()
+        }
+    };
+    let (_, destination) = document.dereference(&destination).ok()?;
+    let target = match destination.as_array().ok()?.first()? {
+        Object::Reference(id) => *id,
+        _ => return None,
+    };
+    pages
+        .iter()
+        .position(|page| *page == target)
+        .map(|at| at as u32 + 1)
+}
+
+/// A PDF text string, in either of the two encodings.
+fn text(document: &Document, object: &Object) -> Option<String> {
+    let (_, object) = document.dereference(object).ok()?;
+    let bytes = match object {
+        Object::String(bytes, _) => bytes,
+        _ => return None,
+    };
+    if bytes.starts_with(&[0xFE, 0xFF]) {
+        let units: Vec<u16> = bytes[2..]
+            .chunks_exact(2)
+            .map(|pair| u16::from_be_bytes([pair[0], pair[1]]))
+            .collect();
+        return Some(String::from_utf16_lossy(&units));
+    }
+    Some(bytes.iter().map(|byte| *byte as char).collect())
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// A PDF whose objects are the ones given, with a cross-reference table
+    /// that points at them. lopdf needs a real one, as any reader would.
+    fn pdf(objects: &[(usize, &str)]) -> Vec<u8> {
+        let mut out = b"%PDF-1.4\n".to_vec();
+        let top = objects.iter().map(|(n, _)| *n).max().unwrap_or(0) + 1;
+        let mut offsets = vec![0usize; top];
+        for (number, body) in objects {
+            offsets[*number] = out.len();
+            out.extend_from_slice(format!("{number} 0 obj\n{body}\nendobj\n").as_bytes());
+        }
+        let start = out.len();
+        out.extend_from_slice(format!("xref\n0 {top}\n0000000000 65535 f \n").as_bytes());
+        for offset in offsets.iter().take(top).skip(1) {
+            out.extend_from_slice(format!("{offset:010} 00000 n \n").as_bytes());
+        }
+        out.extend_from_slice(
+            format!("trailer\n<< /Size {top} /Root 1 0 R >>\nstartxref\n{start}\n%%EOF\n")
+                .as_bytes(),
+        );
+        out
+    }
+
+    #[test]
+    fn a_pdf_with_no_outline_says_so_rather_than_failing() {
+        let minimal = pdf(&[
+            (1, "<< /Type /Catalog /Pages 2 0 R >>"),
+            (2, "<< /Type /Pages /Count 1 /Kids [3 0 R] >>"),
+            (3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>"),
+        ]);
+        assert_eq!(read(&minimal), Err(Missing::None("outline")));
+    }
+
+    #[test]
+    fn an_outline_comes_back_nested_and_with_its_pages() {
+        let doc = pdf(&[
+            (1, "<< /Type /Catalog /Pages 2 0 R /Outlines 10 0 R >>"),
+            (2, "<< /Type /Pages /Count 2 /Kids [3 0 R 4 0 R] >>"),
+            (3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>"),
+            (4, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>"),
+            (10, "<< /Type /Outlines /First 11 0 R /Count 2 >>"),
+            (
+                11,
+                "<< /Title (Part One) /Dest [3 0 R /Fit] /First 12 0 R /Next 13 0 R >>",
+            ),
+            (
+                12,
+                "<< /Title (I. The Setting) /Dest [3 0 R /Fit] /Parent 11 0 R >>",
+            ),
+            (
+                13,
+                "<< /Title (Part Two) /Dest [4 0 R /Fit] /Parent 10 0 R >>",
+            ),
+        ]);
+        let entries = read(&doc).expect("an outline");
+        assert_eq!(
+            entries
+                .iter()
+                .map(|e| (e.depth, e.title.as_str(), e.page))
+                .collect::<Vec<_>>(),
+            [
+                (0, "Part One", Some(1)),
+                (1, "I. The Setting", Some(1)),
+                (0, "Part Two", Some(2)),
+            ]
+        );
+    }
+
+    #[test]
+    fn an_outline_that_points_at_itself_does_not_loop_forever() {
+        let doc = pdf(&[
+            (1, "<< /Type /Catalog /Pages 2 0 R /Outlines 10 0 R >>"),
+            (2, "<< /Type /Pages /Count 1 /Kids [3 0 R] >>"),
+            (3, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>"),
+            (10, "<< /Type /Outlines /First 11 0 R >>"),
+            (11, "<< /Title (Round) /Next 12 0 R >>"),
+            (12, "<< /Title (And round) /Next 11 0 R >>"),
+        ]);
+        let entries = read(&doc).expect("an outline");
+        assert_eq!(entries.len(), 2, "a cycle was followed more than once");
+    }
+
+    #[test]
+    fn what_is_not_a_pdf_says_why() {
+        assert!(matches!(
+            read(b"%PDF-1.4 truncated"),
+            Err(Missing::Unreadable(_))
+        ));
+    }
+}