funkpower
2dfa370cd7
The README now lists the forms it handles with an example of each, says that where the italics sit is what decides whether they are the title or the journal, and sets out what it refuses and why — scripture, Migne, a contributor's biography, a note that only thanks people, a page footer. Recipes gains "What a book leans on": one bibliography read, then the same over a shelf, counted per citing book so a work cited forty times in one bibliography still counts once. Every command was run — thirty converted books gave 14,481 references, 10,536 naming a work, in about a second. `records.md` now points at the five tools whose record shapes live in their own pages rather than in `corpus-core`. And pandoc escapes an underscore, so the rule of them that stands in for a repeated author arrives as `\_\_\_\_\_` and was not being recognised.
5.0 KiB
The records
Five shapes travel between these tools. Each carries a version field v, so a
reader can refuse a stream it does not understand rather than parsing it into
whatever happens to line up.
Every list-shaped stream is NDJSON: one record per line, flushed as it is produced.
Part of catena.
doc — what a file is
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
"pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
| Field | Meaning |
|---|---|
path |
Where the file was read from, as given |
kind |
pdf, epub, djvu, azw3, mobi, txt, html, ole2, zip-wrapped-pdf, unknown |
declared_ext |
The extension the name claimed, lowercased, without the dot |
bytes |
Size on disk |
pages |
Page count, for paginated formats |
language |
Language code, when one was detected |
text_layer |
healthy, garbled, sparse, reflected-font, absent |
needs_ocr |
Whether the file has to be OCRed to be searchable |
notes |
Observations, in the order they were made |
A tool fills what it can and leaves the rest null: a format sniffer fills
kind and leaves text_layer empty, a triage tool does the reverse.
Written by booksniff and pdftriage.
chunk — a retrievable passage
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
"role":"body","start_time":null,"end_time":null,"flags":[]}
| Field | Meaning |
|---|---|
idx |
Position in the stream this chunk was cut from, from zero |
text |
The passage |
tokens |
Token count of text |
page |
Page the passage starts on, for paginated sources |
chapter |
Nearest enclosing heading, as written |
role |
body, bibliography, index, toc, frontmatter |
start_time, end_time |
Seconds from the start, for recordings |
flags |
Set tags, sorted and deduplicated |
page and chapter for documents, start_time and end_time for
recordings. One shape either way, which is what lets a single index serve a
book and a lecture and cite both.
Written by chunkmd and cuepack.
hit — one scored candidate
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
| Field | Meaning |
|---|---|
id |
Identifier of the thing retrieved, opaque to fusion |
channel |
vec, fts or meta |
rank |
Position within that channel's own ranking, from one |
score |
The channel's raw score, on whatever scale it uses |
payload |
Anything the producer wants carried through untouched |
rank is what fusion consumes. score is carried along because it is useful
to a person reading a trace and useless for comparing across channels.
Read and written by rrf.
cue — one line of a transcript
{"v":1,"start":12.5,"end":15.0,"text":"the mill stood at the northern end"}
Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few seconds and often half a sentence.
Written by cuenorm, read by cuepack and
cueseg.
segment — one topic chapter
{"v":1,"start":0.0,"end":1190.7,"title":"The Mill Stood At The Northern End",
"excerpt":"the mill stood at the northern end of the valley where…"}
The marks a scrubber needs, and the spans a summariser should be given one at a time.
Written by cueseg.
Records that belong to one tool
Not everything is shared. A tool whose answer nothing downstream consumes has a shape of its own, described where that tool is:
bibclean— catalogue records, author suggestions, slug proposals and vocabulary merge proposals.avplan— what has to happen to a recording, and the call that would do it.pdfgutter— whether a scan is two pages to a sheet, and what each sampled page measured.citefind— one reference out of a bibliography, and the fields it could not read off the entry.bookmeta— what a file says about itself.tocfind— one entry of a table of contents.seriesof— what a title says about its series.idnorm— an identifier as it arrived and as it should be stored.
Getting the schemas
The JSON Schema for the first three is printed by the record library itself:
$ cargo run -p corpus-core -- schema chunk
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Chunk",
…
}
See corpus-core.
See also the recipes, for what these rules buy when the tools are put in a row.