catena/docs/records.md

# The records

Five shapes travel between these tools. Each carries a version field `v`, so a
reader can refuse a stream it does not understand rather than parsing it into
whatever happens to line up.

Every list-shaped stream is [NDJSON](contract.md#streams-not-documents): one
record per line, flushed as it is produced.

Back to [the index](README.md).

## `doc` — what a file is

```json
{"v":1,"path":"scan.pdf","kind":"pdf","declared_ext":"pdf","bytes":9261184,
 "pages":412,"language":"eng","text_layer":"garbled","needs_ocr":true,"notes":[]}
```

| Field | Meaning |
|---|---|
| `path` | Where the file was read from, as given |
| `kind` | `pdf`, `epub`, `djvu`, `azw3`, `mobi`, `txt`, `html`, `ole2`, `zip-wrapped-pdf`, `unknown` |
| `declared_ext` | The extension the name claimed, lowercased, without the dot |
| `bytes` | Size on disk |
| `pages` | Page count, for paginated formats |
| `language` | Language code, when one was detected |
| `text_layer` | `healthy`, `garbled`, `sparse`, `reflected-font`, `absent` |
| `needs_ocr` | Whether the file has to be OCRed to be searchable |
| `notes` | Observations, in the order they were made |

A tool fills what it can and leaves the rest null: a format sniffer fills
`kind` and leaves `text_layer` empty, a triage tool does the reverse.

Written by [`booksniff`](booksniff.md) and [`pdftriage`](pdftriage.md).

## `chunk` — a retrievable passage

```json
{"v":1,"idx":7,"text":"…","tokens":512,"page":37,"chapter":"III. The Second Estate",
 "role":"body","start_time":null,"end_time":null,"flags":[]}
```

| Field | Meaning |
|---|---|
| `idx` | Position in the stream this chunk was cut from, from zero |
| `text` | The passage |
| `tokens` | Token count of `text` |
| `page` | Page the passage starts on, for paginated sources |
| `chapter` | Nearest enclosing heading, as written |
| `role` | `body`, `bibliography`, `index`, `toc`, `frontmatter` |
| `start_time`, `end_time` | Seconds from the start, for recordings |
| `flags` | Set tags, sorted and deduplicated |

`page` and `chapter` for documents, `start_time` and `end_time` for
recordings. One shape either way, which is what lets a single index serve a
book and a lecture and cite both.

Written by [`chunkmd`](chunkmd.md) and [`cuepack`](cuepack.md).

## `hit` — one scored candidate

```json
{"v":1,"id":"c81a","channel":"vec","rank":1,"score":0.83,"payload":{}}
```

| Field | Meaning |
|---|---|
| `id` | Identifier of the thing retrieved, opaque to fusion |
| `channel` | `vec`, `fts` or `meta` |
| `rank` | Position within that channel's own ranking, from one |
| `score` | The channel's raw score, on whatever scale it uses |
| `payload` | Anything the producer wants carried through untouched |

`rank` is what fusion consumes. `score` is carried along because it is useful
to a person reading a trace and useless for comparing across channels.

Read and written by [`rrf`](rrf.md).

## `cue` — one line of a transcript

```json
{"v":1,"start":12.5,"end":15.0,"text":"the mill stood at the northern end"}
```

Smaller than a chunk on purpose: a cue is what a subtitle file holds, a few
seconds and often half a sentence.

Written by [`cuenorm`](cuenorm.md), read by [`cuepack`](cuepack.md) and
[`cueseg`](cueseg.md).

## `segment` — one topic chapter

```json
{"v":1,"start":0.0,"end":1190.7,"title":"The Mill Stood At The Northern End",
 "excerpt":"the mill stood at the northern end of the valley where…"}
```

The marks a scrubber needs, and the spans a summariser should be given one at
a time.

Written by [`cueseg`](cueseg.md).

## Getting the schemas

The JSON Schema for the first three is printed by the record library itself:

```
$ cargo run -p corpus-core -- schema chunk
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Chunk",
  …
}
```

See [`corpus-core`](corpus-core.md).