funkpower
cfa86f778f
Two things between a converted EPUB and its chapter titles.
The heading level taken as the chapter map was the shallowest present. Pandoc
writes the book's title as the only `#` and every real chapter as a `##`, so
the shallowest level had one heading in it and the whole book came back as a
single chapter. The dividing level is now the shallowest one used more than
once — a level used once is a title.
And an empty bracketed span survived the heading cleaner, because the pattern
wanted at least one character between the brackets. Pandoc writes an anchor
as exactly that: `## []{#link2H_INTR} Introduction`.
Gibbon's first volume, converted from the Project Gutenberg EPUB: 2 distinct
chapter titles before, 61 after, and none of them carrying an element id.
mdscrub
Converted markdown in, clean markdown out.
Part of catena.
What it does
Takes the markdown that falls out of an EPUB or a PDF converter and removes everything that means nothing: styling tags that survived, control characters a text column will reject, headings wrapped in links to their own anchors, page anchors that should be page markers.
Safe to run twice — the second run changes nothing.
When you would use it
- Between a converter and a chunker. Converted markdown is readable but not clean, and every piece of leftover markup ends up in an embedding, in a full-text index, and in every result a reader sees.
- Backfilling text you already stored. Running it over text that was ingested before a cleaning step existed is safe, because it is idempotent.
- Getting rid of the things that break a database write. A null byte in extracted text is rejected outright by most text columns; the rest of the C0 controls are storable and are extraction garbage.
Install
cargo install --git https://git.pumpin.online/funkpower/catena mdscrub
Example
A chapter as a converter leaves it:
$ cat chapter.md
# [<span id="page41"></span>III. The Second Estate](#ch3) {.unnumbered}
The obligation ran <em>both ways</em>, and contemporaries said so plainly
whenever they were asked.<a href="#n4"><sup>4</sup></a> A lord who could not
protect was a lord who could not command…
<span id="page42"></span>
…
## [Notes]{.section-title}
5. Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220–24.
and after:
$ mdscrub chapter.md
# III. The Second Estate
<!-- Page 41 -->
The obligation ran both ways, and contemporaries said so plainly
whenever they were asked.4 A lord who could not
protect was a lord who could not command…
<!-- Page 42 -->
…
## Notes
5. Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220–24.
The heading is readable, its page anchor became a marker on its own line, the
superscript note reference became the digit it was, and – became the
dash it names.
In a pipeline:
pandoc -t commonmark chapter.xhtml | mdscrub | chunkmd > chunks.ndjson
Options
| Option | Effect |
|---|---|
--pages |
First turn page separators and form feeds into page markers |
--man |
Write this page as a manual page, in roff |
--pages is for text straight out of a PDF or DjVu extractor, which marks its
page boundaries with a form feed or a rule of dashes. It is off by default
because only the program that just read a PDF knows that a form feed in the
text is a page break rather than a stray control character — in ordinary
markdown it is a control character, and in a code listing it is part of the
listing.
Exit codes
| Code | Meaning |
|---|---|
0 |
Done |
64 |
The command line was wrong |
65 |
The input was not text |
74 |
The file could not be read |
What it does, pass by pass
- Headings. A heading wrapped in a link to its own anchor, an attribute block, a bracketed span, bold markers around a title set in capitals — all removed, leaving the title. A page anchor inside a heading is moved to a marker on the line after it, so the heading keeps its position relative to the body text.
- Page anchors.
<span id="page42">in the body becomes<!-- Page 42 -->, with the number exactly as it was written. - Character references.
<sup>107is decoded first, because a tag escaped twice is invisible to any pattern looking for a tag — which is how markup survives into text. - Tags. Styling tags go and their text stays. A tag that stands for a
line or block boundary —
<br>,<p>,<li>, a table cell — becomes a space, because deleting it welds the words either side together:APPENDICES<br>OF<br>LISTwould otherwise come out asAPPENDICESOFLIST, which is a word no reader can search for. Genuinely inline tags go to nothing, soclaim<sup>16</sup>staysclaim16. - Fenced code. Left alone, so a book that quotes markup keeps its examples. A decorative rule of tildes is not a fence.
- Control characters. Removed, tab and newline excepted.
- Blank lines. Runs collapse to one.
HTML comments survive every pass, which is what lets page markers through.
As a library
let cleaned = mdscrub::scrub(&markdown);
// And, for text straight out of an extractor:
let cleaned = mdscrub::scrub(&mdscrub::pages::normalize(&raw));
The passes are also available one at a time — mdscrub::html::strip_tags,
mdscrub::heading::clean, mdscrub::unescape::unescape — for a caller that
wants only one of them.