chunkmd: a passage is placed where it begins

The chapter a passage carries was resolved at the offset of the page it came
from, not the offset of the passage. A document with no page markers is a
single page beginning at zero, so every chunk in such a book answered with
whichever heading came first in the file — and took the role classification
with it, since that reads the chapter title.

Measured over a corpus of 5,386 books: all 1,232 unpaginated books with
twenty or more chunks carried exactly one chapter title from cover to cover,
as did 73% of the paginated ones. A bibliography chapter was therefore
invisible to anything selecting on the chapter title, which is the first of
the three ways the citation stage finds one.

The packer now hands back each passage with the offset it began at, and the
page's own offset is the true start of its text rather than the position of
the marker before it.

crates/chunkmd/src/lib.rs

@@ -33,7 +33,7 @@ pub fn chunk(markdown: &str, budgets: &Budgets) -> Vec<Chunk> {
     let mut chunks: Vec<Chunk> = Vec::new();
     for page in pages {
         for packed in pack::page(page.text, budgets) {
-            for (piece, tokens) in pack::enforce_hard_max(&packed, budgets.hard_max) {
+            for (piece, tokens) in pack::enforce_hard_max(&packed.text, budgets.hard_max) {
                 if tokens < budgets.min_tokens {
                     continue;
                 }
@@ -42,11 +42,15 @@ pub fn chunk(markdown: &str, budgets: &Budgets) -> Vec<Chunk> {
                 // that have headings but no page anchors. Page zero counts as
                 // no page at all, which is what the Python's truth test makes
                 // of it.
+                // The passage's own position, not the page's: an
+                // unpaginated document is a single page beginning at zero,
+                // so asking at the page put every chunk in the book under
+                // whichever heading came first in the file.
                 let chapter = page
                     .number
                     .filter(|number| *number != 0)
                     .and_then(|number| pages::chapter_of_page(&chapters, number))
-                    .or_else(|| by_offset.at(page.offset));
+                    .or_else(|| by_offset.at(page.offset + packed.offset));
 
                 let idx = chunks.len() as u32;
                 let notes = footnote::looks_like_citations(&piece);
@@ -110,6 +114,30 @@ mod tests {
         assert_eq!(chunks[0].chapter.as_deref(), Some("The Second Estate"));
     }
 
+    #[test]
+    fn every_chunk_answers_with_the_heading_it_sits_under() {
+        // An unpaginated document is one page beginning at zero, so asking
+        // the page where a chunk was rather than asking the chunk put the
+        // whole book under whichever heading came first in the file.
+        let markdown = "# Chapter One\n\nBody text long enough to be worth \
+                        keeping as a passage of its own.\n\n# Bibliography\n\n\
+                        Bloch, Marc. *Feudal Society*. London: Routledge, 1961.\n\n\
+                        Duby, Georges. *The Three Orders*. Chicago: Chicago, 1980.\n";
+        let budgets = Budgets {
+            target: 40,
+            overlap: 0,
+            min_tokens: 5,
+            ..Budgets::default()
+        };
+        let chunks = chunk(markdown, &budgets);
+        let under: Vec<&str> = chunks
+            .iter()
+            .map(|chunk| chunk.chapter.as_deref().unwrap_or("—"))
+            .collect();
+        assert!(under.contains(&"Chapter One"), "{under:?}");
+        assert!(under.contains(&"Bibliography"), "{under:?}");
+    }
+
     #[test]
     fn a_chapter_is_found_by_offset_when_there_are_no_page_markers() {
         let markdown = "# The Second Estate\n\nalpha beta gamma";