corpus-core

The record types and the stream plumbing everything shares.

Part of catena.

What it is

A library, not a command — though it carries a small one for printing schemas. It holds the three record types, the NDJSON reader and writer, and the exit-code conventions.

Everything else in the workspace depends on it, which is what makes the tools composable: they agree about the records before they agree about anything else.

When you would use it

Reading or writing these records in your own program, without shelling out to any of the tools.
Generating bindings in another language, from the JSON Schema.
Writing a new filter that joins the pipeline.

The records

Doc, Chunk and Hit. Each derives Serialize, Deserialize and JsonSchema, and implements Record, which is how a reader knows what version it is looking at.

use corpus_core::{Chunk, Kind, Doc};

let mut chunk = Chunk::new(0, "the mill stood at the northern end", 8);
chunk.page = Some(41);
chunk.chapter = Some("III. The Second Estate".into());
chunk.flag(corpus_core::chunk::flags::FOOTNOTE_CITATIONS);

let doc = Doc::new("scan.pdf", Kind::Pdf, 9_261_184);

Every field is emitted, including the empty ones, so a consumer can rely on the shape rather than on what a particular producer happened to fill in.

Reading and writing a stream

use corpus_core::ndjson;

for record in ndjson::read::<Chunk, _>(std::io::stdin().lock()) {
    let chunk = record?;
    println!("{} tokens", chunk.tokens);
}

ndjson::write_line(&mut std::io::stdout(), &chunk)?;

The reader is strict, and each strictness is there because the alternative is a silent failure:

A record from a version it does not know is refused. A reader that parses a newer record into whatever lines up will process truncated records rather than failing.
It stops at the first bad line. A bad line means the producer and the consumer disagree about the format, and reading on buries that.
Lines are bounded. A filter pointed at a binary file should fail, not allocate the file. The limit is sixteen megabytes and can be changed with read_with_limit.
Blank lines are skipped and a byte-order mark is ignored, so a stream that was concatenated or written on Windows still reads.
Every record is flushed as it is written, so a slow stage downstream sees a record as soon as it exists.

An error knows whether it was a closed pipe:

match ndjson::write_line(&mut out, &chunk) {
    Err(e) if e.is_broken_pipe() => return Ok(()),  // `| head` is normal
    other => other?,
}

Floats JSON cannot carry

JSON has no spelling for NaN or infinity, and the usual behaviour is to write them as null — which is a silent corruption in both directions: a score written as null fails to read back and kills the stream at the consumer, and a timestamp written as null reads back as "no timestamp" with nothing to say it was ever anything else.

So a non-finite number is refused where it is produced, which is the only place anyone can still fix it. Use the same guard on your own records:

#[derive(Serialize, Deserialize)]
struct Mine {
    #[serde(with = "corpus_core::finite")]
    score: f64,
    #[serde(default, with = "corpus_core::finite::option")]
    when: Option<f64>,
}

Exit codes

use corpus_core::exit;

std::process::ExitCode::from(exit::OK)             // 0
std::process::ExitCode::from(exit::NO)             // 1
std::process::ExitCode::from(exit::INDETERMINATE)  // 2
std::process::ExitCode::from(exit::USAGE)          // 64

Verdict is the three-answer form for a predicate:

use corpus_core::exit::Verdict;

let answer = Verdict::from(needs_ocr);
std::process::ExitCode::from(answer)

See the contract for the whole table.

The schemas

$ cargo run -p corpus-core -- schema chunk > chunk.schema.json
$ cargo run -p corpus-core -- schema all

doc, chunk, hit or all. The output is byte-identical across runs, so it can be committed and diffed.

4.5 KiB Raw Blame History