funkpower
c071f84829
Cuts a reference list into entries — at the note numbers where there are any, at the layout where there are not — and reads each as far as its punctuation allows. Fields that would not parse are named in `unsettled` rather than guessed at, and `--prompt` asks a model about exactly those and no others. Over 202 real reference sections from the corpus: 8,566 references, 95% with a title, 82% with a year, 46% settling every field with no model involved. A title is refused where the entry was never divided, so no record reports its own text back as the work it cites. `ibid.` is flagged and left alone — it points at a work, and the entry does not say which. A rule of dashes is filled in, because the style defines it as the author above. chunkmd gains `note_starts` and `cites_something`: recognising a page of notes and splitting one are the same knowledge, and two copies of that pattern would drift.
catena
Turn a pile of files into a searchable corpus: chunking, scrubbing, subtitles and rank fusion, as WebAssembly.
npm install catena
import { chunkMarkdown, scrub, countTokens, fuseHits } from "catena";
const chunks = chunkMarkdown(markdown, 512, 64);
chunks[0].page; // 41
chunks[0].chapter; // 'III. The Second Estate'
const clean = scrub(rawMarkdown, true);
countTokens(clean);
fuseHits([...vectorHits, ...textHits], undefined, { vec: 2 }, 20);
Records come back as plain objects — the same documents the command-line tools write, with the same field names and the same stability promise. Nothing has to be freed.
splitCollection throws when the document is one work rather than a
collection, which is the usual answer:
let works;
try {
works = splitCollection(markdown);
} catch (why) {
works = [{ idx: 0, title, text: markdown }];
}
What is here
chunkMarkdown |
Markdown into passages that carry their page, chapter and role |
scrub |
Converted markdown in, clean markdown out |
countTokens |
Tokens, by the encoding the budgets count in |
parseSubtitles |
SubRip, WebVTT or ASS into cues |
packCues |
Cues into passages, keeping the times they were said at |
segmentCues |
A timed transcript into topic chapters |
cueWindows |
The windows segmentCues measures, to embed |
splitCollection |
One markdown file into the works inside it |
fuseHits |
Several ranked lists into one |
identify |
What a file is, from its first bytes |
seriesOf |
The series a book belongs to, and which volume this is |
parseIdentifier |
A DOI, ISSN, ISBN or arXiv id, read and checked |
The tools that read files or call another program are not here: a browser has neither, and a binding that pretends otherwise fails at run time rather than at compile time. Those stay command-line tools.
Node and the web
import { chunkMarkdown } from "catena"; // resolves per environment
import { chunkMarkdown } from "catena/node"; // or pick one
import { chunkMarkdown } from "catena/web";
Two builds because the loaders differ: node reads the .wasm off disk at
import, the web build takes a URL a bundler rewrites.
Licence
MPL-2.0. Source: https://git.pumpin.online/funkpower/catena