mirror of
https://git.pumpin.online/funkpower/catena.git
synced 2026-10-22 18:57:14 +00:00
122 lines
4.2 KiB
Markdown
122 lines
4.2 KiB
Markdown
# textkit
|
||
|
||
Token counting, sentence and paragraph splitting.
|
||
|
||
Part of [catena](../../README.md).
|
||
|
||
## What it is
|
||
|
||
A library. Two jobs, both of which have to give the same answer everywhere or
|
||
chunk boundaries move: counting tokens the way an embedding stack counts them,
|
||
and cutting text into sentences and paragraphs.
|
||
|
||
## When you would use it
|
||
|
||
- **Deciding how much text fits** in a model's context, before sending it.
|
||
- **Writing your own chunker** that has to agree with
|
||
[`chunkmd`](../chunkmd/README.md) about where a boundary is.
|
||
- **Porting anything from Python** where the exact behaviour of `\s`, `\w` or
|
||
`\b` matters more than it looks like it should.
|
||
|
||
## Counting tokens
|
||
|
||
```rust
|
||
let tokens = textkit::count("the mill stood at the northern end");
|
||
let ids = textkit::encode("…");
|
||
let text = textkit::decode_lossy(&ids[..100]);
|
||
```
|
||
|
||
`cl100k_base`, loaded from data compiled into the binary — no download, no
|
||
cache directory, nothing to go stale. Counting against a different tokeniser
|
||
than the one an index was built with silently moves every chunk boundary,
|
||
which is why this is one function and not a parameter.
|
||
|
||
Special-token spellings are treated as ordinary text. Text out of a PDF can
|
||
contain anything, including the literal string `<|endoftext|>`, and encoding
|
||
it as a control token would be both wrong and a way for a document to reach
|
||
into a model's input.
|
||
|
||
Cutting a list of tokens at an arbitrary index can split a character in half,
|
||
so decoding a slice is lossy rather than fallible:
|
||
|
||
```rust
|
||
// The last resort of a chunker: guarantees a maximum, respects nothing else.
|
||
for piece in textkit::split_at_token_limit(&text, 512) {
|
||
assert!(textkit::count(&piece) <= 512);
|
||
}
|
||
```
|
||
|
||
## Splitting text
|
||
|
||
```rust
|
||
for paragraph in textkit::paragraphs(&markdown) {
|
||
for sentence in textkit::sentences(paragraph) {
|
||
// …
|
||
}
|
||
}
|
||
```
|
||
|
||
Both return borrowed slices, trimmed, with the empty pieces dropped.
|
||
|
||
A paragraph break is a newline, any run of whitespace, and a newline — so one
|
||
blank line and four of them separate equally, and trailing spaces on an
|
||
otherwise blank line do not stop it being blank.
|
||
|
||
A sentence break is whitespace following `.`, `!` or `?`. Crude on purpose: it
|
||
does not know that `Mr.` or `p. 41` are not sentence ends. It is the fallback
|
||
path for a paragraph too large to chunk whole, where a wrong cut costs a
|
||
little context and a clever cut that disagrees with an existing index costs a
|
||
re-embedding.
|
||
|
||
## Whitespace
|
||
|
||
```rust
|
||
let trimmed = text.trim_matches(textkit::is_space);
|
||
```
|
||
|
||
`is_space` is whitespace by Python's definition: the Unicode `White_Space`
|
||
property *and* the four ASCII information separators `U+001C`–`U+001F`.
|
||
Trimming with the narrower definition leaves a separator on the front of a
|
||
piece, which changes its token count, which moves a chunk boundary.
|
||
|
||
## Numbers
|
||
|
||
```rust
|
||
assert_eq!(textkit::decimal("<!-- Page 412 -->"), 412);
|
||
assert_eq!(textkit::decimal("٤٢"), 42);
|
||
```
|
||
|
||
Every decimal digit, not only the ASCII ones. Every Unicode decimal block runs
|
||
zero to nine in consecutive code points, which is what makes this possible
|
||
without a lookup table.
|
||
|
||
## Patterns
|
||
|
||
Character classes that mean in Rust what they mean in Python:
|
||
|
||
```rust
|
||
use textkit::pattern::{SPACE, WORD, WORD_START, WORD_END};
|
||
|
||
let year = fancy_regex::Regex::new(&format!(r"{WORD_START}(19|20)\d{{2}}{WORD_END}"))?;
|
||
let heading = regex::Regex::new(&format!(r"(?m)^(#{{1,3}}){SPACE}+(.+)$"))?;
|
||
```
|
||
|
||
The two regex engines disagree, and every disagreement is a heuristic that
|
||
fires on one side and not the other:
|
||
|
||
- Python's `\s` also matches `U+001C`–`U+001F`.
|
||
- Python's `\w` is the letters, the numbers and the underscore. Rust's adds
|
||
every combining mark and every connector punctuation mark, and leaves out
|
||
the letter-like and other numbers — so `Gödel` written in decomposed form is
|
||
one word to Rust and two to Python, and `²` is a word character to Python
|
||
and not to Rust.
|
||
- `\b` is defined in terms of `\w`, so it inherits all of that.
|
||
|
||
The boundary forms are lookarounds, so a pattern using them needs an engine
|
||
that has them.
|
||
|
||
## See also
|
||
|
||
- [`corpus-core`](../corpus-core/README.md) — the records these help produce.
|
||
- [`chunkmd`](../chunkmd/README.md) — the chunker built on this.
|