mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-23 03:07:13 +00:00
funkpower
57bc4356ab
Both registries already have a `catena`: an HPE blockchain client on PyPI,
and a JavaScript OOP wrapper at 2.2.0 on npm. Neither is going anywhere, so
the distributions take a name that is free on both.
The Python import is unaffected — a distribution name and a module name are
independent there, so `pip install catena-corpus` still gives `import
catena`, and nothing in the examples or the bindings changes. On npm the
package name is the import specifier, so that one does change: `import
{ chunkMarkdown } from "catena-corpus"`.
The crates keep their own names. `catena` is free on crates.io.
74 lines
2.3 KiB
Markdown
74 lines
2.3 KiB
Markdown
# catena
|
|
|
|
Turn a pile of files into a searchable corpus: chunking, scrubbing, subtitles
|
|
and rank fusion, as WebAssembly.
|
|
|
|
```sh
|
|
npm install catena-corpus
|
|
```
|
|
|
|
```js
|
|
import { chunkMarkdown, scrub, countTokens, fuseHits } from "catena-corpus";
|
|
|
|
const chunks = chunkMarkdown(markdown, 512, 64);
|
|
chunks[0].page; // 41
|
|
chunks[0].chapter; // 'III. The Second Estate'
|
|
|
|
const clean = scrub(rawMarkdown, true);
|
|
countTokens(clean);
|
|
|
|
fuseHits([...vectorHits, ...textHits], undefined, { vec: 2 }, 20);
|
|
```
|
|
|
|
Records come back as plain objects — the same documents the command-line
|
|
tools write, with the same field names and the same stability promise.
|
|
Nothing has to be freed.
|
|
|
|
`splitCollection` throws when the document is one work rather than a
|
|
collection, which is the usual answer:
|
|
|
|
```js
|
|
let works;
|
|
try {
|
|
works = splitCollection(markdown);
|
|
} catch (why) {
|
|
works = [{ idx: 0, title, text: markdown }];
|
|
}
|
|
```
|
|
|
|
## What is here
|
|
|
|
| | |
|
|
|---|---|
|
|
| `chunkMarkdown` | Markdown into passages that carry their page, chapter and role |
|
|
| `scrub` | Converted markdown in, clean markdown out |
|
|
| `countTokens` | Tokens, by the encoding the budgets count in |
|
|
| `parseSubtitles` | SubRip, WebVTT or ASS into cues |
|
|
| `packCues` | Cues into passages, keeping the times they were said at |
|
|
| `segmentCues` | A timed transcript into topic chapters |
|
|
| `cueWindows` | The windows `segmentCues` measures, to embed |
|
|
| `splitCollection` | One markdown file into the works inside it |
|
|
| `fuseHits` | Several ranked lists into one |
|
|
| `identify` | What a file is, from its first bytes |
|
|
| `seriesOf` | The series a book belongs to, and which volume this is |
|
|
| `parseIdentifier` | A DOI, ISSN, ISBN or arXiv id, read and checked |
|
|
|
|
The tools that read files or call another program are not here: a browser has
|
|
neither, and a binding that pretends otherwise fails at run time rather than
|
|
at compile time. Those stay command-line tools.
|
|
|
|
## Node and the web
|
|
|
|
```js
|
|
import { chunkMarkdown } from "catena-corpus"; // resolves per environment
|
|
import { chunkMarkdown } from "catena/node"; // or pick one
|
|
import { chunkMarkdown } from "catena/web";
|
|
```
|
|
|
|
Two builds because the loaders differ: node reads the `.wasm` off disk at
|
|
import, the web build takes a URL a bundler rewrites.
|
|
|
|
## Licence
|
|
|
|
MPL-2.0. Source: <https://git.pumpin.online/funkpower/catena>
|