4.6 KiB
The records
Five shapes travel between these tools. Each carries a version field v, so a
reader can refuse a stream it does not understand rather than parsing it into
whatever happens to line up.
Every list-shaped stream is NDJSON: one record per line, flushed as it is produced.
Part of catena.
doc — what a file is
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
"pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
| Field | Meaning |
|---|---|
path |
Where the file was read from, as given |
kind |
pdf, epub, djvu, azw3, mobi, txt, html, ole2, zip-wrapped-pdf, unknown |
declared_ext |
The extension the name claimed, lowercased, without the dot |
bytes |
Size on disk |
pages |
Page count, for paginated formats |
language |
Language code, when one was detected |
text_layer |
healthy, garbled, sparse, reflected-font, absent |
needs_ocr |
Whether the file has to be OCRed to be searchable |
notes |
Observations, in the order they were made |
A tool fills what it can and leaves the rest null: a format sniffer fills
kind and leaves text_layer empty, a triage tool does the reverse.
Written by booksniff and pdftriage.
chunk — a retrievable passage
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
"role":"body","start_time":null,"end_time":null,"flags":[]}
| Field | Meaning |
|---|---|
idx |
Position in the stream this chunk was cut from, from zero |
text |
The passage |
tokens |
Token count of text |
page |
Page the passage starts on, for paginated sources |
chapter |
Nearest enclosing heading, as written |
role |
body, bibliography, index, toc, frontmatter |
start_time, end_time |
Seconds from the start, for recordings |
flags |
Set tags, sorted and deduplicated |
page and chapter for documents, start_time and end_time for
recordings. One shape either way, which is what lets a single index serve a
book and a lecture and cite both.
Written by chunkmd and cuepack.
hit — one scored candidate
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
| Field | Meaning |
|---|---|
id |
Identifier of the thing retrieved, opaque to fusion |
channel |
vec, fts or meta |
rank |
Position within that channel's own ranking, from one |
score |
The channel's raw score, on whatever scale it uses |
payload |
Anything the producer wants carried through untouched |
rank is what fusion consumes. score is carried along because it is useful
to a person reading a trace and useless for comparing across channels.
Read and written by rrf.
cue — one line of a transcript
{"v":1,"start":12.5,"end":15.0,"text":"the mill stood at the northern end"}
Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few seconds and often half a sentence.
Written by cuenorm, read by cuepack and
cueseg.
segment — one topic chapter
{"v":1,"start":0.0,"end":1190.7,"title":"The Mill Stood At The Northern End",
"excerpt":"the mill stood at the northern end of the valley where…"}
The marks a scrubber needs, and the spans a summariser should be given one at a time.
Written by cueseg.
Records that belong to one tool
Not everything is shared. A tool whose answer nothing downstream consumes has a shape of its own, described where that tool is:
bibclean— catalogue records, author suggestions, slug proposals and vocabulary merge proposals.avplan— what has to happen to a recording, and the call that would do it.pdfgutter— whether a scan is two pages to a sheet, and what each sampled page measured.
Getting the schemas
The JSON Schema for the first three is printed by the record library itself:
$ cargo run -p corpus-core -- schema chunk
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Chunk",
…
}
See corpus-core.
See also the recipes, for what these rules buy when the tools are put in a row.