mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 11:17:13 +00:00
funkpower
0b849c39c4
The hashed bag of words asks only whether two stretches repeat the same words, and cannot tell that ships and vessels are one subject. An embedding can. segment_with takes one vector per window and falls back to the words when it is given none, or the wrong number of them — a caller that embedded some other recording's windows is asking about that recording. windows_of gives the text to embed, in the order the vectors come back in, so nobody has to reimplement the windowing pass to use this. Bound in all three languages, and on the command line as --windows and --embeddings, which pipe through emb. The C ABI version goes to 2: catena_segment_cues gained an argument.
74 lines
2.3 KiB
Markdown
74 lines
2.3 KiB
Markdown
# catena
|
|
|
|
Turn a pile of files into a searchable corpus: chunking, scrubbing, subtitles
|
|
and rank fusion, as WebAssembly.
|
|
|
|
```sh
|
|
npm install catena
|
|
```
|
|
|
|
```js
|
|
import { chunkMarkdown, scrub, countTokens, fuseHits } from "catena";
|
|
|
|
const chunks = chunkMarkdown(markdown, 512, 64);
|
|
chunks[0].page; // 41
|
|
chunks[0].chapter; // 'III. The Second Estate'
|
|
|
|
const clean = scrub(rawMarkdown, true);
|
|
countTokens(clean);
|
|
|
|
fuseHits([...vectorHits, ...textHits], undefined, { vec: 2 }, 20);
|
|
```
|
|
|
|
Records come back as plain objects — the same documents the command-line
|
|
tools write, with the same field names and the same stability promise.
|
|
Nothing has to be freed.
|
|
|
|
`splitCollection` throws when the document is one work rather than a
|
|
collection, which is the usual answer:
|
|
|
|
```js
|
|
let works;
|
|
try {
|
|
works = splitCollection(markdown);
|
|
} catch (why) {
|
|
works = [{ idx: 0, title, text: markdown }];
|
|
}
|
|
```
|
|
|
|
## What is here
|
|
|
|
| | |
|
|
|---|---|
|
|
| `chunkMarkdown` | Markdown into passages that carry their page, chapter and role |
|
|
| `scrub` | Converted markdown in, clean markdown out |
|
|
| `countTokens` | Tokens, by the encoding the budgets count in |
|
|
| `parseSubtitles` | SubRip, WebVTT or ASS into cues |
|
|
| `packCues` | Cues into passages, keeping the times they were said at |
|
|
| `segmentCues` | A timed transcript into topic chapters |
|
|
| `cueWindows` | The windows `segmentCues` measures, to embed |
|
|
| `splitCollection` | One markdown file into the works inside it |
|
|
| `fuseHits` | Several ranked lists into one |
|
|
| `identify` | What a file is, from its first bytes |
|
|
| `seriesOf` | The series a book belongs to, and which volume this is |
|
|
| `parseIdentifier` | A DOI, ISSN, ISBN or arXiv id, read and checked |
|
|
|
|
The tools that read files or call another program are not here: a browser has
|
|
neither, and a binding that pretends otherwise fails at run time rather than
|
|
at compile time. Those stay command-line tools.
|
|
|
|
## Node and the web
|
|
|
|
```js
|
|
import { chunkMarkdown } from "catena"; // resolves per environment
|
|
import { chunkMarkdown } from "catena/node"; // or pick one
|
|
import { chunkMarkdown } from "catena/web";
|
|
```
|
|
|
|
Two builds because the loaders differ: node reads the `.wasm` off disk at
|
|
import, the web build takes a URL a bundler rewrites.
|
|
|
|
## Licence
|
|
|
|
MPL-2.0. Source: <https://git.pumpin.online/funkpower/catena>
|