catena/crates/mdscrub/README.md

mdscrub

Converted markdown in, clean markdown out.

Part of catena.

What it does

Takes the markdown that falls out of an EPUB or a PDF converter and removes everything that means nothing: styling tags that survived, control characters a text column will reject, headings wrapped in links to their own anchors, page anchors that should be page markers.

Safe to run twice — the second run changes nothing.

When you would use it

• Between a converter and a chunker. Converted markdown is readable but not clean, and every piece of leftover markup ends up in an embedding, in a full-text index, and in every result a reader sees.
• Backfilling text you already stored. Running it over text that was ingested before a cleaning step existed is safe, because it is idempotent.
• Getting rid of the things that break a database write. A null byte in extracted text is rejected outright by most text columns; the rest of the C0 controls are storable and are extraction garbage.

Install

cargo install --git https://git.pumpin.online/funkpower/catena mdscrub

Example

A chapter as a converter leaves it:

$ cat chapter.md
# [<span id="page41"></span>III. The Second Estate](#ch3) {.unnumbered}

The obligation ran <em>both ways</em>, and contemporaries said so plainly
whenever they were asked.<a href="#n4"><sup>4</sup></a> A lord who could not
protect was a lord who could not command…

<span id="page42"></span>
…
## [Notes]{.section-title}

5. Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220&ndash;24.

and after:

$ mdscrub chapter.md
# III. The Second Estate

<!-- Page 41 -->

The obligation ran both ways, and contemporaries said so plainly
whenever they were asked.4 A lord who could not
protect was a lord who could not command…

<!-- Page 42 -->
…
## Notes

5. Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220–24.

The heading is readable, its page anchor became a marker on its own line, the superscript note reference became the digit it was, and – became the dash it names.

In a pipeline:

pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd > chunks.ndjson

Options

Option	Effect
--pages	First turn page separators and form feeds into page markers
--man	Write this page as a manual page, in roff

--pages is for text straight out of a PDF or DjVu extractor, which marks its page boundaries with a form feed or a rule of dashes. It is off by default because only the program that just read a PDF knows that a form feed in the text is a page break rather than a stray control character — in ordinary markdown it is a control character, and in a code listing it is part of the listing.

Exit codes

Code	Meaning
0	Done
64	The command line was wrong
65	The input was not text
74	The file could not be read

What it does, pass by pass

1. Headings. A heading wrapped in a link to its own anchor, an attribute block, a bracketed span, bold markers around a title set in capitals — all removed, leaving the title. A page anchor inside a heading is moved to a marker on the line after it, so the heading keeps its position relative to the body text.
2. Page anchors. <span id="page42"> in the body becomes <!-- Page 42 -->, with the number exactly as it was written.
3. Character references. <sup>107 is decoded first, because a tag escaped twice is invisible to any pattern looking for a tag — which is how markup survives into text.
4. Tags. Styling tags go and their text stays. A tag that stands for a line or block boundary — <br>, <p>, <li>, a table cell — becomes a space, because deleting it welds the words either side together: APPENDICES<br>OF<br>LIST would otherwise come out as APPENDICESOFLIST, which is a word no reader can search for. Genuinely inline tags go to nothing, so claim<sup>16</sup> stays claim16.
5. Fenced code. Left alone, so a book that quotes markup keeps its examples. A decorative rule of tildes is not a fence.
6. Control characters. Removed, tab and newline excepted.
7. Blank lines. Runs collapse to one.

HTML comments survive every pass, which is what lets page markers through.

As a library

let cleaned = mdscrub::scrub(&markdown);

// And, for text straight out of an extractor:
let cleaned = mdscrub::scrub(&mdscrub::pages::normalize(&raw));

The passes are also available one at a time — mdscrub::html::strip_tags, mdscrub::heading::clean, mdscrub::unescape::unescape — for a caller that wants only one of them.

See also

• mdsplit — when the file holds more than one work.
• chunkmd — what to do with the markdown once it is clean.
• pdftriage — whether the text was worth extracting at all.