mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
143 lines
4.5 KiB
Markdown
143 lines
4.5 KiB
Markdown
# corpus-core
|
|
|
|
The record types and the stream plumbing everything shares.
|
|
|
|
Part of [catena](../../README.md).
|
|
|
|
## What it is
|
|
|
|
A library, not a command — though it carries a small one for printing
|
|
schemas. It holds the three [record types](../../docs/records.md), the NDJSON reader and
|
|
writer, and the exit-code conventions.
|
|
|
|
Everything else in the workspace depends on it, which is what makes the tools
|
|
composable: they agree about the records before they agree about anything
|
|
else.
|
|
|
|
## When you would use it
|
|
|
|
- **Reading or writing these records in your own program**, without shelling
|
|
out to any of the tools.
|
|
- **Generating bindings in another language**, from the JSON Schema.
|
|
- **Writing a new filter** that joins the pipeline.
|
|
|
|
## The records
|
|
|
|
[`Doc`](../../docs/records.md#doc--what-a-file-is),
|
|
[`Chunk`](../../docs/records.md#chunk--a-retrievable-passage) and
|
|
[`Hit`](../../docs/records.md#hit--one-scored-candidate). Each derives `Serialize`,
|
|
`Deserialize` and `JsonSchema`, and implements `Record`, which is how a reader
|
|
knows what version it is looking at.
|
|
|
|
```rust
|
|
use corpus_core::{Chunk, Kind, Doc};
|
|
|
|
let mut chunk = Chunk::new(0, "the mill stood at the northern end", 8);
|
|
chunk.page = Some(41);
|
|
chunk.chapter = Some("III. The Second Estate".into());
|
|
chunk.flag(corpus_core::chunk::flags::FOOTNOTE_CITATIONS);
|
|
|
|
let doc = Doc::new("scan.pdf", Kind::Pdf, 9_261_184);
|
|
```
|
|
|
|
Every field is emitted, including the empty ones, so a consumer can rely on
|
|
the shape rather than on what a particular producer happened to fill in.
|
|
|
|
## Reading and writing a stream
|
|
|
|
```rust
|
|
use corpus_core::ndjson;
|
|
|
|
for record in ndjson::read::<Chunk, _>(std::io::stdin().lock()) {
|
|
let chunk = record?;
|
|
println!("{} tokens", chunk.tokens);
|
|
}
|
|
|
|
ndjson::write_line(&mut std::io::stdout(), &chunk)?;
|
|
```
|
|
|
|
The reader is strict, and each strictness is there because the alternative is
|
|
a silent failure:
|
|
|
|
- **A record from a version it does not know is refused.** A reader that
|
|
parses a newer record into whatever lines up will process truncated records
|
|
rather than failing.
|
|
- **It stops at the first bad line.** A bad line means the producer and the
|
|
consumer disagree about the format, and reading on buries that.
|
|
- **Lines are bounded.** A filter pointed at a binary file should fail, not
|
|
allocate the file. The limit is sixteen megabytes and can be changed with
|
|
`read_with_limit`.
|
|
- **Blank lines are skipped and a byte-order mark is ignored**, so a stream
|
|
that was concatenated or written on Windows still reads.
|
|
- **Every record is flushed as it is written**, so a slow stage downstream
|
|
sees a record as soon as it exists.
|
|
|
|
An error knows whether it was a closed pipe:
|
|
|
|
```rust
|
|
match ndjson::write_line(&mut out, &chunk) {
|
|
Err(e) if e.is_broken_pipe() => return Ok(()), // `| head` is normal
|
|
other => other?,
|
|
}
|
|
```
|
|
|
|
## Floats JSON cannot carry
|
|
|
|
JSON has no spelling for NaN or infinity, and the usual behaviour is to write
|
|
them as `null` — which is a silent corruption in both directions: a score
|
|
written as `null` fails to read back and kills the stream at the consumer, and
|
|
a timestamp written as `null` reads back as "no timestamp" with nothing to say
|
|
it was ever anything else.
|
|
|
|
So a non-finite number is refused where it is produced, which is the only
|
|
place anyone can still fix it. Use the same guard on your own records:
|
|
|
|
```rust
|
|
#[derive(Serialize, Deserialize)]
|
|
struct Mine {
|
|
#[serde(with = "corpus_core::finite")]
|
|
score: f64,
|
|
#[serde(default, with = "corpus_core::finite::option")]
|
|
when: Option<f64>,
|
|
}
|
|
```
|
|
|
|
## Exit codes
|
|
|
|
```rust
|
|
use corpus_core::exit;
|
|
|
|
std::process::ExitCode::from(exit::OK) // 0
|
|
std::process::ExitCode::from(exit::NO) // 1
|
|
std::process::ExitCode::from(exit::INDETERMINATE) // 2
|
|
std::process::ExitCode::from(exit::USAGE) // 64
|
|
```
|
|
|
|
`Verdict` is the three-answer form for a predicate:
|
|
|
|
```rust
|
|
use corpus_core::exit::Verdict;
|
|
|
|
let answer = Verdict::from(needs_ocr);
|
|
std::process::ExitCode::from(answer)
|
|
```
|
|
|
|
See [the contract](../../docs/contract.md#exit-codes-carry-the-answer) for the
|
|
whole table.
|
|
|
|
## The schemas
|
|
|
|
```
|
|
$ cargo run -p corpus-core -- schema chunk > chunk.schema.json
|
|
$ cargo run -p corpus-core -- schema all
|
|
```
|
|
|
|
`doc`, `chunk`, `hit` or `all`. The output is byte-identical across runs, so
|
|
it can be committed and diffed.
|
|
|
|
## See also
|
|
|
|
- [The records](../../docs/records.md) — the shapes, described.
|
|
- [`catena-cli`](../catena-cli/README.md) — the rest of what a filter needs.
|
|
- [`textkit`](../textkit/README.md) — counting and splitting text.
|