funkpower
0b849c39c4
The hashed bag of words asks only whether two stretches repeat the same words, and cannot tell that ships and vessels are one subject. An embedding can. segment_with takes one vector per window and falls back to the words when it is given none, or the wrong number of them — a caller that embedded some other recording's windows is asking about that recording. windows_of gives the text to embed, in the order the vectors come back in, so nobody has to reimplement the windowing pass to use this. Bound in all three languages, and on the command line as --windows and --embeddings, which pipe through emb. The C ABI version goes to 2: catena_segment_cues gained an argument.
2.3 KiB
catena
Turn a pile of files into a searchable corpus: chunking, scrubbing, subtitles and rank fusion, as WebAssembly.
npm install catena
import { chunkMarkdown, scrub, countTokens, fuseHits } from "catena";
const chunks = chunkMarkdown(markdown, 512, 64);
chunks[0].page; // 41
chunks[0].chapter; // 'III. The Second Estate'
const clean = scrub(rawMarkdown, true);
countTokens(clean);
fuseHits([...vectorHits, ...textHits], undefined, { vec: 2 }, 20);
Records come back as plain objects — the same documents the command-line tools write, with the same field names and the same stability promise. Nothing has to be freed.
splitCollection throws when the document is one work rather than a
collection, which is the usual answer:
let works;
try {
works = splitCollection(markdown);
} catch (why) {
works = [{ idx: 0, title, text: markdown }];
}
What is here
chunkMarkdown |
Markdown into passages that carry their page, chapter and role |
scrub |
Converted markdown in, clean markdown out |
countTokens |
Tokens, by the encoding the budgets count in |
parseSubtitles |
SubRip, WebVTT or ASS into cues |
packCues |
Cues into passages, keeping the times they were said at |
segmentCues |
A timed transcript into topic chapters |
cueWindows |
The windows segmentCues measures, to embed |
splitCollection |
One markdown file into the works inside it |
fuseHits |
Several ranked lists into one |
identify |
What a file is, from its first bytes |
seriesOf |
The series a book belongs to, and which volume this is |
parseIdentifier |
A DOI, ISSN, ISBN or arXiv id, read and checked |
The tools that read files or call another program are not here: a browser has neither, and a binding that pretends otherwise fails at run time rather than at compile time. Those stay command-line tools.
Node and the web
import { chunkMarkdown } from "catena"; // resolves per environment
import { chunkMarkdown } from "catena/node"; // or pick one
import { chunkMarkdown } from "catena/web";
Two builds because the loaders differ: node reads the .wasm off disk at
import, the web build takes a URL a bundler rewrites.
Licence
MPL-2.0. Source: https://git.pumpin.online/funkpower/catena