langid and tocfind: what language a book is in, and what it says its chapters are

crates/langid/Cargo.toml

@@ -0,0 +1,19 @@
+[package]
+name = "langid"
+description = "What language a document is in, from its script and its commonest words."
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+repository.workspace = true
+
+[dependencies]
+catena-cli.workspace = true
+clap.workspace = true
+corpus-core.workspace = true
+schemars.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+
+[lints]
+workspace = true

crates/langid/README.md

@@ -0,0 +1,123 @@
+# langid
+
+What language a document is in.
+
+Part of [catena](../../README.md).
+
+## What it does
+
+Two passes, because one will not do. The writing system settles most of the
+world outright — a text in Hangul is Korean — and for the Latin and Cyrillic
+alphabets it only narrows the field, so the commonest words decide between
+what is left.
+
+No model, no download, no network. Function words make up a third of any
+running text, and counting them is decisive on a paragraph. On a single line
+it is not, which is why a close call is refused rather than guessed at.
+
+## When you would use it
+
+- **Before OCR**, to know which language to hand it.
+- **Before embedding**, where a multilingual shelf wants different models.
+- **Filling the `language` field** on a `doc` record, which nothing else sets.
+- **Finding the odd book out** in a library you assumed was all one language.
+
+## Install
+
+```sh
+cargo install --git https://git.pumpin.online/funkpower/catena langid
+```
+
+## Example
+
+```
+$ langid converted/*.md | jq -r '[.name, (.confidence*100|round|tostring + "%"), .path] | @tsv'
+English	77%	converted/enlightenment.md
+English	76%	converted/hagakure.md
+French	81%	converted/meditations.md
+Latin	68%	converted/summa.md
+```
+
+Just the code:
+
+```
+$ langid --quiet book.md
+en
+```
+
+The whole record, which carries the bibliographic code the `doc` record
+wants:
+
+```
+$ langid book.md
+{"v":1,"path":"book.md","language":"en","bibliographic":"eng","name":"English","script":"Latin","confidence":0.77}
+```
+
+The ones that are not what you assumed:
+
+```sh
+langid converted/*.md | jq -r 'select(.language != "en") | [.path, .name] | @tsv'
+```
+
+Reading from a pipe:
+
+```sh
+pdftotext -l 5 scan.pdf - | langid --quiet
+```
+
+## Options
+
+| Option | Effect |
+|---|---|
+| `--quiet` | Print the code alone, one per line |
+| `--man` | Write this page as a manual page, in roff |
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | Every file was placed |
+| `1` | At least one could not be |
+| `64` | The command line was wrong |
+| `74` | A file could not be read |
+
+## How it decides
+
+**The script first.** Kana, Hangul, Hebrew, Arabic and Devanagari each belong
+to one language as far as a library is concerned, and no amount of word
+counting improves on that. Kana is checked before Han, because Japanese is
+mostly Han characters with kana between them and it is the kana that tell it
+from Chinese.
+
+A quarter of a text's letters agreeing is enough to call it a script: a book
+carries page numbers, names and quotations in other alphabets, and an English
+word in a Russian paragraph does not make the paragraph English.
+
+**Then the words.** Thirteen languages across Latin, Cyrillic and Greek, each
+with its thirty commonest function words. The score is the share of a text's
+words found in each list, and the lists are the same length so the scores
+compare.
+
+**A close call is no answer.** The winner must be a quarter ahead of the
+runner-up. `de` is among the commonest words of four languages at once, and on
+its own settles none of them — so a text that short comes back with the script
+named, no language, and a note saying which two it could not choose between.
+
+Only the first twenty thousand characters are read. A book's language does not
+change after the first few pages.
+
+## As a library
+
+```rust
+let found = langid::detect("book.md", text);
+assert_eq!(found.language.as_deref(), Some("en"));
+```
+
+`langid::script::detect` answers the writing system on its own, and
+`langid::words::score` ranks every language of a given script.
+
+## See also
+
+- [`booksniff`](../booksniff/README.md) — what the file is.
+- [`bookmeta`](../bookmeta/README.md) — what the file says its language is,
+  which is worth more when it says anything.

crates/langid/src/lib.rs

@@ -0,0 +1,235 @@
+//! What language a document is in.
+//!
+//! Two passes, because one does not do. The writing system settles most of
+//! the world outright — a text in Hangul is Korean — and for the Latin and
+//! Cyrillic alphabets it only narrows the field, so the commonest words
+//! decide between what is left.
+//!
+//! No model, no download, no network. Function words make up a third of any
+//! running text, and on a paragraph or more counting them is decisive. On a
+//! single line it is not, which is why every answer carries a confidence and
+//! why "could not tell" is one of the answers.
+
+pub mod script;
+pub mod words;
+
+use schemars::JsonSchema;
+use serde::{Deserialize, Serialize};
+
+pub use script::Script;
+
+/// How much text is worth looking at.
+///
+/// A book's language does not change after the first few pages, and reading
+/// all of it to say so is waste.
+pub const SAMPLE_CHARS: usize = 20_000;
+
+/// What language a text turned out to be.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
+pub struct Verdict {
+    /// Record version.
+    #[serde(default = "version")]
+    pub v: u32,
+    /// What was read.
+    pub path: String,
+    /// ISO 639-1, the two-letter code. Absent when it could not be told.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub language: Option<String>,
+    /// ISO 639-2/B, which is what the `doc` record carries.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub bibliographic: Option<String>,
+    /// What the language is called in English.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub name: Option<String>,
+    /// The writing system, which is an answer even when the language is not.
+    pub script: String,
+    /// How far ahead the winner was, from zero to one. One means nothing else
+    /// scored at all.
+    #[serde(with = "corpus_core::finite")]
+    #[schemars(with = "f64")]
+    pub confidence: f64,
+    /// What was worth saying about the reading.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub notes: Vec<String>,
+}
+
+fn version() -> u32 {
+    corpus_core::V
+}
+
+impl corpus_core::Record for Verdict {
+    fn version(&self) -> u32 {
+        self.v
+    }
+}
+
+/// Read a text's language.
+pub fn detect(path: &str, text: &str) -> Verdict {
+    let sample: String = text.chars().take(SAMPLE_CHARS).collect();
+    let (script, share) = script::detect(&sample);
+
+    let mut verdict = Verdict {
+        v: corpus_core::V,
+        path: path.to_string(),
+        language: None,
+        bibliographic: None,
+        name: None,
+        script: name_of(script).to_string(),
+        confidence: 0.0,
+        notes: Vec::new(),
+    };
+
+    // A script only one language uses is the whole answer, and no amount of
+    // word counting would improve on it.
+    if let Some(sole) = sole_language(script) {
+        verdict.language = Some(sole.0.to_string());
+        verdict.bibliographic = Some(sole.1.to_string());
+        verdict.name = Some(sole.2.to_string());
+        verdict.confidence = share;
+        verdict
+            .notes
+            .push(format!("{} is written only in {}", sole.2, name_of(script)));
+        return verdict;
+    }
+
+    let scored = words::score(&sample, script);
+    let Some((best, best_score)) = scored.first() else {
+        verdict.notes.push(match script {
+            Script::Other => "no letters to go on".to_string(),
+            script => format!(
+                "nothing here knows the languages written in {}",
+                name_of(script)
+            ),
+        });
+        return verdict;
+    };
+
+    if *best_score == 0.0 {
+        verdict
+            .notes
+            .push("none of the commonest words of any language it knows".to_string());
+        return verdict;
+    }
+
+    // How far ahead of the runner-up, so a close call can be refused.
+    let runner_up = scored.get(1).map_or(0.0, |(_, score)| *score);
+    let ratio = match runner_up > 0.0 {
+        true => best_score / runner_up,
+        false => f64::INFINITY,
+    };
+    if ratio < words::MARGIN {
+        verdict.notes.push(format!(
+            "{} and {} score too alike to choose between",
+            best.name, scored[1].0.name
+        ));
+        return verdict;
+    }
+
+    verdict.language = Some(best.code.to_string());
+    verdict.bibliographic = Some(best.bibliographic.to_string());
+    verdict.name = Some(best.name.to_string());
+    verdict.confidence = match runner_up > 0.0 {
+        true => (1.0 - runner_up / best_score).clamp(0.0, 1.0),
+        false => 1.0,
+    };
+    verdict
+}
+
+/// The one language a script is used for, where there is only one.
+fn sole_language(script: Script) -> Option<(&'static str, &'static str, &'static str)> {
+    Some(match script {
+        Script::Kana => ("ja", "jpn", "Japanese"),
+        Script::Han => ("zh", "chi", "Chinese"),
+        Script::Hangul => ("ko", "kor", "Korean"),
+        Script::Hebrew => ("he", "heb", "Hebrew"),
+        Script::Arabic => ("ar", "ara", "Arabic"),
+        Script::Devanagari => ("hi", "hin", "Hindi"),
+        _ => return None,
+    })
+}
+
+/// What a script is called.
+pub fn name_of(script: Script) -> &'static str {
+    match script {
+        Script::Latin => "Latin",
+        Script::Cyrillic => "Cyrillic",
+        Script::Greek => "Greek",
+        Script::Arabic => "Arabic",
+        Script::Hebrew => "Hebrew",
+        Script::Han => "Han",
+        Script::Kana => "Kana",
+        Script::Hangul => "Hangul",
+        Script::Devanagari => "Devanagari",
+        Script::Other => "unknown",
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn a_paragraph_of_english_is_english() {
+        let found = detect(
+            "x.md",
+            "The obligation ran both ways, and contemporaries said so plainly whenever they were asked about it at all.",
+        );
+        assert_eq!(found.language.as_deref(), Some("en"));
+        assert_eq!(found.bibliographic.as_deref(), Some("eng"));
+        assert_eq!(found.script, "Latin");
+        assert!(found.confidence > 0.0);
+    }
+
+    #[test]
+    fn a_script_only_one_language_uses_needs_no_words() {
+        for (text, code) in [
+            ("안녕하세요 세계 반갑습니다 오늘은", "ko"),
+            ("これは日本語の文章です", "ja"),
+            ("מרחבי שלום עולם וברוך הבא", "he"),
+        ] {
+            assert_eq!(detect("x", text).language.as_deref(), Some(code), "{text}");
+        }
+    }
+
+    #[test]
+    fn nothing_to_read_is_no_answer_rather_than_a_wrong_one() {
+        let found = detect("x", "12345 -- ...");
+        assert_eq!(found.language, None);
+        assert_eq!(found.script, "unknown");
+        assert_eq!(found.notes, ["no letters to go on"]);
+    }
+
+    #[test]
+    fn latin_words_no_language_claims_are_refused() {
+        let found = detect("x", "zzzz qqqq xxxx vvvv wwww kkkk jjjj");
+        assert_eq!(found.language, None);
+        assert!(found.notes[0].starts_with("none of the commonest words"));
+    }
+
+    #[test]
+    fn a_close_call_is_refused_rather_than_guessed() {
+        // "de" is among the commonest words of four of these languages at
+        // once, and on its own settles none of them.
+        let found = detect("x", "de de de");
+        assert_eq!(found.language, None, "{:?}", found.notes);
+        assert!(found.notes[0].contains("too alike"), "{:?}", found.notes);
+    }
+
+    #[test]
+    fn only_the_opening_is_read() {
+        let long = format!(
+            "{}{}",
+            "The obligation ran both ways. ".repeat(2000),
+            "xyzzy"
+        );
+        assert!(long.len() > SAMPLE_CHARS * 2);
+        assert_eq!(detect("x", &long).language.as_deref(), Some("en"));
+    }
+
+    #[test]
+    fn the_script_is_reported_even_when_the_language_is_not() {
+        let found = detect("x", "zzzz qqqq xxxx vvvv");
+        assert_eq!(found.script, "Latin");
+        assert_eq!(found.language, None);
+    }
+}

crates/langid/src/main.rs

@@ -0,0 +1,93 @@
+//! What language a document is in.
+
+use std::path::PathBuf;
+use std::process::ExitCode;
+
+use clap::Parser;
+use corpus_core::exit;
+
+/// Say what language a text is in, one record per file.
+///
+/// The writing system settles most of the world outright; for the Latin and
+/// Cyrillic alphabets the commonest words decide between what is left.
+///
+/// No model, no download, no network. A paragraph is plenty; a single line
+/// often is not, and a close call is refused rather than guessed at.
+#[derive(Parser)]
+#[command(
+    version,
+    about,
+    after_help = "\
+Exit codes: 0 every file was placed, 1 at least one could not be,
+74 a file could not be read.
+
+Example:
+
+    langid book.md
+    langid converted/*.md | jq -r 'select(.language != \"en\") | [.path, .name] | @tsv'"
+)]
+struct Args {
+    /// Files to read. Reads stdin when absent.
+    paths: Vec<PathBuf>,
+
+    /// Print the code alone, for one file.
+    #[arg(long)]
+    quiet: bool,
+}
+
+fn main() -> ExitCode {
+    let args: Args = match catena_cli::parse() {
+        Ok(args) => args,
+        Err(code) => return code,
+    };
+
+    let mut verdicts = Vec::new();
+    let mut unplaced = 0usize;
+
+    if args.paths.is_empty() {
+        let text = match catena_cli::read_lossy(None) {
+            Ok(text) => text,
+            Err((code, why)) => {
+                eprintln!("langid: {why}");
+                return ExitCode::from(code);
+            }
+        };
+        verdicts.push(langid::detect("-", &text));
+    } else {
+        for path in &args.paths {
+            let text = match catena_cli::read_lossy(Some(path)) {
+                Ok(text) => text,
+                Err((code, why)) => {
+                    eprintln!("langid: {why}");
+                    return ExitCode::from(code);
+                }
+            };
+            verdicts.push(langid::detect(&path.display().to_string(), &text));
+        }
+    }
+    unplaced += verdicts.iter().filter(|v| v.language.is_none()).count();
+
+    if args.quiet {
+        let codes: String = verdicts
+            .iter()
+            .map(|v| format!("{}\n", v.language.as_deref().unwrap_or("")))
+            .collect();
+        let wrote = catena_cli::text("langid", &codes);
+        if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
+            return wrote;
+        }
+        return ExitCode::from(match unplaced {
+            0 => exit::OK,
+            _ => exit::NO,
+        });
+    }
+
+    let wrote = catena_cli::stream("langid", &verdicts);
+    if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
+        return wrote;
+    }
+    ExitCode::from(match unplaced {
+        0 => exit::OK,
+        _ => exit::NO,
+    })
+}

crates/langid/src/script.rs

@@ -0,0 +1,175 @@
+//! What writing system a text is in.
+//!
+//! For most of the world this is the whole answer: a text in Hangul is
+//! Korean, a text in Hiragana is Japanese. For the Latin and Cyrillic
+//! alphabets it narrows the field and no more, and the words have to decide.
+
+use serde::{Deserialize, Serialize};
+
+/// A writing system, as far as telling languages apart needs one.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub enum Script {
+    /// The alphabet most of Europe writes in.
+    Latin,
+    /// Russian, Ukrainian, Bulgarian, Serbian and their neighbours.
+    Cyrillic,
+    /// Greek.
+    Greek,
+    /// Arabic, Persian, Urdu.
+    Arabic,
+    /// Hebrew and Yiddish.
+    Hebrew,
+    /// Chinese characters, which Japanese also uses.
+    Han,
+    /// Japanese syllabaries, which only Japanese uses.
+    Kana,
+    /// Korean.
+    Hangul,
+    /// Devanagari: Hindi, Marathi, Sanskrit.
+    Devanagari,
+    /// Something else, or too little to tell.
+    Other,
+}
+
+/// How much of a text has to be in one script before it counts as that
+/// script.
+///
+/// Well under half, because a book carries page numbers, names and quotations
+/// in other alphabets, and an English word in a Russian paragraph does not
+/// make it English.
+const SHARE: f64 = 0.25;
+
+/// The script a text is written in, and what share of its letters agree.
+///
+/// Counts letters only: digits and punctuation belong to every language and
+/// would drown the signal in a text full of citations.
+pub fn detect(text: &str) -> (Script, f64) {
+    let mut counts = [0usize; 10];
+    let mut letters = 0usize;
+
+    for c in text.chars() {
+        let Some(script) = of(c) else { continue };
+        letters += 1;
+        counts[index(script)] += 1;
+    }
+    if letters == 0 {
+        return (Script::Other, 0.0);
+    }
+
+    // Kana before Han: a Japanese text is mostly Han characters with kana
+    // between them, and it is the kana that say which language it is.
+    if counts[index(Script::Kana)] as f64 / letters as f64 > 0.05 {
+        return (
+            Script::Kana,
+            counts[index(Script::Kana)] as f64 / letters as f64,
+        );
+    }
+
+    let (at, count) = counts
+        .iter()
+        .enumerate()
+        .max_by_key(|(_, count)| **count)
+        .expect("ten scripts");
+    let share = *count as f64 / letters as f64;
+    match share >= SHARE {
+        true => (from_index(at), share),
+        false => (Script::Other, share),
+    }
+}
+
+fn index(script: Script) -> usize {
+    match script {
+        Script::Latin => 0,
+        Script::Cyrillic => 1,
+        Script::Greek => 2,
+        Script::Arabic => 3,
+        Script::Hebrew => 4,
+        Script::Han => 5,
+        Script::Kana => 6,
+        Script::Hangul => 7,
+        Script::Devanagari => 8,
+        Script::Other => 9,
+    }
+}
+
+fn from_index(at: usize) -> Script {
+    [
+        Script::Latin,
+        Script::Cyrillic,
+        Script::Greek,
+        Script::Arabic,
+        Script::Hebrew,
+        Script::Han,
+        Script::Kana,
+        Script::Hangul,
+        Script::Devanagari,
+        Script::Other,
+    ][at]
+}
+
+/// The script one character belongs to, or `None` if it is not a letter.
+///
+/// By code point range rather than by a Unicode property table, because the
+/// ranges that matter here are few, stable, and cheaper than carrying the
+/// table.
+pub fn of(c: char) -> Option<Script> {
+    let code = c as u32;
+    Some(match code {
+        0x41..=0x5A | 0x61..=0x7A => Script::Latin,
+        0xC0..=0x24F => Script::Latin,    // Latin-1 through Extended-B
+        0x1E00..=0x1EFF => Script::Latin, // Latin Extended Additional
+        0x370..=0x3FF | 0x1F00..=0x1FFF => Script::Greek,
+        0x400..=0x52F => Script::Cyrillic,
+        0x590..=0x5FF => Script::Hebrew,
+        0x600..=0x6FF | 0x750..=0x77F => Script::Arabic,
+        0x900..=0x97F => Script::Devanagari,
+        0x3040..=0x30FF => Script::Kana, // hiragana and katakana, adjacent
+        0x4E00..=0x9FFF | 0x3400..=0x4DBF => Script::Han,
+        0xAC00..=0xD7AF | 0x1100..=0x11FF => Script::Hangul,
+        _ => return None,
+    })
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn each_script_is_recognised_from_its_own_words() {
+        for (text, script) in [
+            ("The obligation ran both ways", Script::Latin),
+            ("Молчание было единственным ответом", Script::Cyrillic),
+            ("Καλημέρα κόσμε καὶ πάντες", Script::Greek),
+            ("مرحبا بالعالم وأهلا", Script::Arabic),
+            ("שלום עולם וברוך", Script::Hebrew),
+            ("안녕하세요 세계 반갑습니다", Script::Hangul),
+            ("नमस्ते दुनिया सब", Script::Devanagari),
+        ] {
+            assert_eq!(detect(text).0, script, "{text}");
+        }
+    }
+
+    #[test]
+    fn kana_settles_japanese_against_chinese() {
+        // Both are mostly Han characters. The kana between them are the tell.
+        assert_eq!(detect("これは日本語の文章です").0, Script::Kana);
+        assert_eq!(detect("這是中文的句子沒有假名").0, Script::Han);
+    }
+
+    #[test]
+    fn a_stray_foreign_word_does_not_change_the_script() {
+        let mostly = "Молчание было единственным ответом на его вопрос about that";
+        assert_eq!(detect(mostly).0, Script::Cyrillic);
+    }
+
+    #[test]
+    fn digits_and_punctuation_are_not_evidence() {
+        assert_eq!(detect("123 456 -- ... 789").0, Script::Other);
+        assert_eq!(detect("").0, Script::Other);
+    }
+
+    #[test]
+    fn accented_latin_is_still_latin() {
+        assert_eq!(detect("naïve café résumé Grüße piñata").0, Script::Latin);
+    }
+}

crates/langid/src/words.rs

@@ -0,0 +1,355 @@
+//! Telling apart the languages that share an alphabet, by their commonest
+//! words.
+//!
+//! Function words — articles, prepositions, conjunctions, pronouns — make up
+//! a third of any running text and differ sharply between languages even
+//! closely related ones. Counting them needs no model, no training and no
+//! network, and on a paragraph or more it is decisive.
+//!
+//! It is weak on a single line, which is why what comes out carries a margin
+//! and a caller can refuse a close call.
+
+use crate::script::Script;
+
+/// One language, and the words that give it away.
+pub struct Language {
+    /// ISO 639-1, the two-letter code everything else uses.
+    pub code: &'static str,
+    /// ISO 639-2/B, for the `doc` record's language field.
+    pub bibliographic: &'static str,
+    /// What it is called in English.
+    pub name: &'static str,
+    /// The script it is written in.
+    pub script: Script,
+    /// Its commonest function words, lowercased.
+    pub common: &'static [&'static str],
+}
+
+/// The languages this can tell apart.
+///
+/// General reference data, of the same kind as the HTML character references
+/// and the ISBN check digit: true everywhere, not a fact about one
+/// collection.
+pub const LANGUAGES: &[Language] = &[
+    Language {
+        code: "en",
+        bibliographic: "eng",
+        name: "English",
+        script: Script::Latin,
+        common: &[
+            "the", "of", "and", "to", "in", "that", "is", "was", "it", "for", "with", "as", "his",
+            "on", "be", "at", "by", "this", "had", "not", "are", "but", "from", "or", "have", "an",
+            "they", "which", "were", "their",
+        ],
+    },
+    Language {
+        code: "fr",
+        bibliographic: "fre",
+        name: "French",
+        script: Script::Latin,
+        common: &[
+            "le", "de", "un", "et", "être", "à", "il", "avoir", "ne", "je", "son", "que", "se",
+            "qui", "ce", "dans", "en", "du", "elle", "au", "pour", "pas", "que", "vous", "par",
+            "sur", "les", "des", "une", "plus",
+        ],
+    },
+    Language {
+        code: "de",
+        bibliographic: "ger",
+        name: "German",
+        script: Script::Latin,
+        common: &[
+            "der", "die", "und", "in", "den", "von", "zu", "das", "mit", "sich", "des", "auf",
+            "für", "ist", "im", "dem", "nicht", "ein", "eine", "als", "auch", "es", "an", "werden",
+            "aus", "er", "hat", "dass", "sie", "nach",
+        ],
+    },
+    Language {
+        code: "es",
+        bibliographic: "spa",
+        name: "Spanish",
+        script: Script::Latin,
+        common: &[
+            "de", "la", "que", "el", "en", "y", "a", "los", "del", "se", "las", "por", "un",
+            "para", "con", "no", "una", "su", "al", "es", "lo", "como", "más", "pero", "sus", "le",
+            "ya", "o", "este", "sí",
+        ],
+    },
+    Language {
+        code: "it",
+        bibliographic: "ita",
+        name: "Italian",
+        script: Script::Latin,
+        common: &[
+            "di", "che", "e", "il", "la", "in", "un", "a", "per", "non", "una", "è", "con", "si",
+            "da", "le", "del", "i", "sono", "come", "al", "della", "nel", "più", "lo", "ma", "se",
+            "gli", "alla", "anche",
+        ],
+    },
+    Language {
+        code: "pt",
+        bibliographic: "por",
+        name: "Portuguese",
+        script: Script::Latin,
+        common: &[
+            "de", "a", "o", "que", "e", "do", "da", "em", "um", "para", "com", "não", "uma", "os",
+            "no", "se", "na", "por", "mais", "as", "dos", "como", "mas", "ao", "ele", "das", "seu",
+            "sua", "ou", "quando",
+        ],
+    },
+    Language {
+        code: "nl",
+        bibliographic: "dut",
+        name: "Dutch",
+        script: Script::Latin,
+        common: &[
+            "de", "van", "het", "een", "en", "in", "is", "dat", "op", "te", "zijn", "met", "voor",
+            "niet", "aan", "er", "om", "maar", "dan", "of", "door", "over", "ze", "bij", "ook",
+            "tot", "uit", "naar", "heeft", "werd",
+        ],
+    },
+    Language {
+        code: "la",
+        bibliographic: "lat",
+        name: "Latin",
+        script: Script::Latin,
+        common: &[
+            "et", "in", "est", "non", "ad", "cum", "qui", "quod", "ut", "sed", "de", "si", "ex",
+            "quae", "per", "hoc", "esse", "enim", "autem", "nec", "atque", "sunt", "eius", "ab",
+            "quam", "tamen", "ita", "etiam", "ipse", "sive",
+        ],
+    },
+    Language {
+        code: "pl",
+        bibliographic: "pol",
+        name: "Polish",
+        script: Script::Latin,
+        common: &[
+            "w", "i", "z", "na", "się", "do", "nie", "to", "że", "jest", "o", "a", "od", "po",
+            "za", "jak", "ale", "tym", "tego", "przez", "przy", "dla", "co", "który", "była",
+            "były", "być", "ma", "już", "tylko",
+        ],
+    },
+    Language {
+        code: "sv",
+        bibliographic: "swe",
+        name: "Swedish",
+        script: Script::Latin,
+        common: &[
+            "och", "att", "det", "som", "en", "på", "är", "av", "för", "med", "till", "den", "har",
+            "de", "inte", "om", "ett", "han", "men", "var", "från", "vi", "så", "kan", "man",
+            "när", "hon", "sig", "eller", "efter",
+        ],
+    },
+    Language {
+        code: "ru",
+        bibliographic: "rus",
+        name: "Russian",
+        script: Script::Cyrillic,
+        common: &[
+            "и", "в", "не", "на", "что", "он", "с", "как", "это", "по", "но", "они", "к", "у",
+            "же", "за", "из", "от", "то", "все", "был", "была", "было", "так", "его", "её", "для",
+            "бы", "или", "мы",
+        ],
+    },
+    Language {
+        code: "uk",
+        bibliographic: "ukr",
+        name: "Ukrainian",
+        script: Script::Cyrillic,
+        common: &[
+            "і", "в", "не", "на", "що", "з", "як", "до", "це", "за", "від", "але", "той", "його",
+            "вона", "там", "бути", "має", "для", "та", "він", "ми", "ви", "чи", "або", "щоб",
+            "коли", "де", "так", "ще",
+        ],
+    },
+    Language {
+        code: "el",
+        bibliographic: "gre",
+        name: "Greek",
+        script: Script::Greek,
+        common: &[
+            "και",
+            "το",
+            "της",
+            "των",
+            "του",
+            "στο",
+            "με",
+            "για",
+            "που",
+            "στη",
+            "είναι",
+            "από",
+            "σε",
+            "τα",
+            "ο",
+            "η",
+            "δεν",
+            "να",
+            "ως",
+            "κατά",
+            "μια",
+            "ένα",
+            "αλλά",
+            "αυτό",
+            "όπως",
+            "θα",
+            "τον",
+            "την",
+            "οι",
+            "στον",
+        ],
+    },
+];
+
+/// How far ahead the best language must be before the answer is worth giving.
+///
+/// A ratio rather than a difference, so it means the same on a paragraph and
+/// on a book.
+pub const MARGIN: f64 = 1.25;
+
+/// Score every language whose script matches, best first.
+///
+/// The score is the share of a text's words that are among a language's
+/// commonest. It is comparable across languages because each list is the same
+/// length.
+pub fn score(text: &str, script: Script) -> Vec<(&'static Language, f64)> {
+    let words = tokens(text);
+    if words.is_empty() {
+        return Vec::new();
+    }
+
+    let mut scored: Vec<(&'static Language, f64)> = LANGUAGES
+        .iter()
+        .filter(|language| language.script == script)
+        .map(|language| {
+            let hits = words
+                .iter()
+                .filter(|word| language.common.contains(&word.as_str()))
+                .count();
+            (language, hits as f64 / words.len() as f64)
+        })
+        .collect();
+
+    scored.sort_by(|a, b| b.1.total_cmp(&a.1));
+    scored
+}
+
+/// A text's words, lowercased, with everything that is not a letter dropped.
+fn tokens(text: &str) -> Vec<String> {
+    text.split(|c: char| crate::script::of(c).is_none())
+        .filter(|word| !word.is_empty())
+        .map(|word| word.to_lowercase())
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn best(text: &str, script: Script) -> Option<&'static str> {
+        score(text, script)
+            .first()
+            .map(|(language, _)| language.code)
+    }
+
+    #[test]
+    fn each_language_is_recognised_from_a_sentence_of_it() {
+        for (text, code) in [
+            (
+                "The obligation ran both ways, and the lord who could not protect was not a lord at all in that time",
+                "en",
+            ),
+            (
+                "Le seigneur qui ne pouvait pas protéger n'était pas un seigneur, et c'est ce que les rôles disent de lui",
+                "fr",
+            ),
+            (
+                "Der Herr, der nicht schützen konnte, war kein Herr, und das ist es, was die Urkunden von ihm sagen",
+                "de",
+            ),
+            (
+                "El señor que no podía proteger no era un señor, y eso es lo que dicen los registros de él en la época",
+                "es",
+            ),
+            (
+                "Il signore che non poteva proteggere non era un signore, e questo è ciò che dicono i documenti di lui",
+                "it",
+            ),
+            (
+                "O senhor que não podia proteger não era um senhor, e é isso que os registos dizem dele naquele tempo",
+                "pt",
+            ),
+            (
+                "De heer die niet kon beschermen was geen heer, en dat is wat de bronnen over hem zeggen in die tijd",
+                "nl",
+            ),
+            (
+                "Dominus qui non poterat protegere non erat dominus, et hoc est quod de eo dicunt scripta illius aetatis",
+                "la",
+            ),
+        ] {
+            assert_eq!(best(text, Script::Latin), Some(code), "{text}");
+        }
+    }
+
+    #[test]
+    fn the_cyrillic_pair_are_told_apart() {
+        assert_eq!(
+            best(
+                "Молчание было единственным ответом на его вопрос, и это всё что он услышал",
+                Script::Cyrillic
+            ),
+            Some("ru")
+        );
+        assert_eq!(
+            best(
+                "Мовчання було єдиною відповіддю на його запитання, і це все що він почув від них",
+                Script::Cyrillic
+            ),
+            Some("uk")
+        );
+    }
+
+    #[test]
+    fn greek_is_read_as_greek() {
+        assert_eq!(
+            best(
+                "Και το φως εν τη σκοτία φαίνει και η σκοτία αυτό δεν κατέλαβεν από την αρχή",
+                Script::Greek
+            ),
+            Some("el")
+        );
+    }
+
+    #[test]
+    fn a_text_with_no_words_scores_nothing() {
+        assert!(score("", Script::Latin).is_empty());
+        assert!(score("123 456", Script::Latin).is_empty());
+    }
+
+    #[test]
+    fn only_languages_in_the_right_script_are_considered() {
+        let latin = score("the and of", Script::Latin);
+        assert!(
+            latin
+                .iter()
+                .all(|(language, _)| language.script == Script::Latin)
+        );
+        assert!(!latin.is_empty());
+    }
+
+    #[test]
+    fn every_language_has_the_same_number_of_words_so_scores_compare() {
+        let first = LANGUAGES[0].common.len();
+        for language in LANGUAGES {
+            assert_eq!(
+                language.common.len(),
+                first,
+                "{} has a different list",
+                language.code
+            );
+        }
+    }
+}