chunkmd: page and chapter aware chunking, roles, footnote flag

parity/Cargo.toml

@@ -14,6 +14,9 @@ publish = false
 python = []
 
 [dependencies]
+chunkmd.workspace = true
+corpus-core.workspace = true
+serde_json.workspace = true
 textkit.workspace = true
 
 [lints]

parity/cases/chunkmd/bibliography.md

@@ -0,0 +1,24 @@
+## Select Bibliography
+
+Arendt, Hannah. The Human Condition. Chicago: University of Chicago Press, 1958.
+
+Bloch, Marc. Feudal Society. Translated by L. A. Manyon. London: Routledge, 1961.
+
+Cassirer, Ernst. The Myth of the State. New Haven: Yale University Press, 1946.
+
+———. An Essay on Man. New Haven: Yale University Press, 1944.
+
+Duby, Georges. The Three Orders: Feudal Society Imagined. Chicago: University
+of Chicago Press, 1980.
+
+Ganshof, F. L. Feudalism. Translated by Philip Grierson. New York: Harper,
+1961.
+
+Hilton, R. H. Bond Men Made Free. London: Temple Smith, 1973.
+
+Maitland, F. W. Domesday Book and Beyond. Cambridge: Cambridge University
+Press, 1897.
+
+Postan, M. M. The Medieval Economy and Society. London: Weidenfeld, 1972.
+
+Vinogradoff, Paul. Villainage in England. Oxford: Clarendon Press, 1892.

parity/cases/chunkmd/contents.md

@@ -0,0 +1,12 @@
+# Contents
+
+Preface......vii
+One. The Setting......1
+Two. The Quarrel......22
+Three. The Break......45
+Four. The Settlement......70
+Five. After the Settlement......99
+Appendix A. The Renders......121
+Appendix B. The Mills......134
+Bibliography......150
+Index......161

parity/cases/chunkmd/epub-ish.md

@@ -0,0 +1,29 @@
+## Chapter One
+
+There are no page markers in this text, because the format it was converted
+from had none. A chunker that needs pages will produce nothing useful here,
+and one that falls back to character position will find the chapter correctly.
+
+The chapter itself concerns the difference between a right and a habit. A
+right can be pleaded; a habit can only be described. The men in these records
+pleaded habits as though they were rights, and the courts, with a patience
+that is easy to underestimate, let them.
+
+## Chapter Two
+
+The second chapter takes the same question into a period where the records are
+better. Better records make the problem worse rather than easier, because they
+show how much of what looked like custom was in fact a settlement of a dispute
+that had gone the other way a generation earlier.
+
+Custom, in this reading, is the name given to the outcome of a quarrel nobody
+alive remembers having. It is stable exactly as long as the memory of the
+quarrel is dead and the conditions that produced the settlement persist. When
+either changes, custom turns out to have been a treaty all along.
+
+## Chapter Three
+
+What remains is to say what this means for the wider argument, which is less
+than it might appear. The valley was not typical. It was chosen because its
+documents survive, and documents survive where somebody had a reason to keep
+them, which is itself a fact about the place.

parity/cases/chunkmd/index.md

@@ -0,0 +1,22 @@
+# Index
+
+abbey, 44, 47, 91
+advowson, 12
+agriculture, 12, 34, 102
+assize, 7, 19-22
+bishops, 7
+boon-work, 88
+charters, 88, 91
+commons, 14, 15, 16
+demesne, 14-16
+escheat, 120
+estates, 3, 44
+fealty, 201
+frankpledge, 55, 56
+heriot, 61
+homage, 3, 4, 201
+manor, 1-8, 14, 33
+merchet, 62
+mill, 71-79, 84
+villein, 90, 91, 104
+wardship, 118

parity/cases/chunkmd/notes-sentinel.md

@@ -0,0 +1,11 @@
+## Notes
+
+12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.
+
+13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.
+
+14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.
+
+15. Ibid., 91.
+
+16. https://example.org/records/valley, consulted 12 March.

parity/cases/chunkmd/notes.md

@@ -0,0 +1,11 @@
+## Notes
+
+12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.
+
+13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.
+
+14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.
+
+15. Ibid., 91.
+
+16. https://example.org/records/valley, consulted 12 March.

parity/cases/chunkmd/oversized.md

@@ -0,0 +1,3 @@
+<!-- Page 40 -->
+
+One sentence follows another without a paragraph break for the whole of this page, which is the case the sentence splitter exists for. The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest. Its wheel was undershot, which was wasteful, and nobody replaced it for two hundred years because the cost of replacing it fell on one household and the benefit was spread across four hundred. The abbey's mill, two miles downstream, was overshot and newer and better, and the abbey charged more to use it, which the valley resented and paid. When the lord's wheel finally failed, in a winter the chronicle calls hard without saying how hard, the valley ground at the abbey for eleven months and discovered that it preferred the abbey's flour. The lord rebuilt anyway. He had to: the right to compel grinding was worth more than the flour, because it was the visible sign that he could compel anything at all. A right that is never exercised is a right that is being tested, and he knew it, and so did they. The rebuilt mill ran for another century and was still running, badly, when the survey of 1439 recorded it as ruinous and in use. That phrase, ruinous and in use, is as good a description of the whole arrangement as anything a historian has managed since.

parity/cases/chunkmd/paged.md

@@ -0,0 +1,43 @@
+This preface was set before the pagination began, and any tool that starts
+reading at the first page marker will lose it. That has happened, and it cost
+a book all but two thousand of its characters.
+
+<!-- Page 1 -->
+
+# I. The Setting
+
+The valley is twelve miles long and never more than two wide. At its northern
+end the river enters through a gorge that no road followed until the last
+century; at its southern end it leaves through a marsh that no road follows
+yet. Everything that happened here happened to people who could not easily
+leave, and who knew it.
+
+The population in the first surviving survey is given as four hundred and six
+households, which at the usual multiplier means somewhere near two thousand
+people. That figure held, within a tenth either way, for three hundred years.
+
+<!-- Page 2 -->
+
+What changed was not how many people lived in the valley but what they owed
+for the privilege. The earliest schedule lists eleven separate renders, seven
+of them in kind. The latest lists two, both in coin. Between those two
+documents lies the whole of the argument.
+
+It would be convenient to say that the change was gradual. It was not. Two
+thirds of it happens in a single generation, in the middle of the fourteenth
+century, for reasons the documents do not give and which have to be supplied
+from outside.
+
+<!-- Page 3 -->
+
+# II. The Quarrel
+
+The quarrel began over a mill. It nearly always does. A mill is the one piece
+of capital in a village that everybody must use and only one person can own,
+and the law of the mill is therefore the law of the place in miniature.
+
+The lord's mill stood at the northern end, where the fall was steepest. The
+abbey's mill stood two miles downstream, on land the abbey had been given by a
+man whose grandson disputed the gift. Both houses claimed the right to compel
+the valley's tenants to grind at their own stones, and both were prepared to
+break the other's.

parity/cases/chunkmd/prose.md

@@ -0,0 +1,32 @@
+The argument of this chapter is that the obligation ran both ways, and that
+contemporaries said so plainly whenever they were asked. A lord who could not
+protect was a lord who could not command, and the men who owed him service
+said as much in language that has survived in the court rolls.
+
+What the rolls do not say, and what has to be inferred, is how often the
+obligation was actually tested. A tenant who withheld service in a year of
+famine was not making a constitutional argument; he was hungry. The clerk who
+recorded the withholding had no vocabulary for the difference, and so the
+record flattens the two into one offence with one penalty.
+
+This flattening is the central difficulty of the sources. It is not that they
+lie. It is that the categories they were written in were designed to produce a
+number at the bottom of a column, and a number at the bottom of a column
+cannot distinguish a rebellion from a bad harvest. Everything that follows
+depends on reading around that limitation rather than through it.
+
+Three approaches have been tried. The first counts the offences and treats the
+count as a proxy for discontent, which assumes the clerk's attention was
+constant across years when we know it was not. The second reads the language
+of the entries for signs of unusual trouble, which works where a clerk was
+talkative and fails entirely where he was not. The third, adopted here,
+compares the same manor across decades and treats changes in what was recorded
+as evidence about the recorder as much as about the recorded.
+
+None of the three is satisfactory, and the third has the particular weakness
+that it needs a long run of documents from a single place. There are not many
+such runs. The ones that exist are concentrated in the south and east, which
+is to say in exactly the regions where the argument is least likely to be
+surprising. A reader who wants to know what happened in the north will find
+this chapter unsatisfying, and the reason is that the documents do not exist,
+not that the question is uninteresting.

parity/cases/chunkmd/unicode.md

@@ -0,0 +1,17 @@
+<!-- Page 5 -->
+
+# Ἀθῆναι and Μόσχα
+
+Le café était naïve, and the first ligature in that sentence is a single
+codepoint rather than two letters, which is the sort of thing that moves a
+token boundary without moving a character boundary.
+
+東京から来た手紙には、日付が二つ書いてあった。One was the date it was written
+and the other was the date it was meant to be read, and the difference between
+them is the whole subject of this section.
+
+مرحبا — the paragraph above is followed by a right-to-left greeting, then by a
+combining acute on a bare e: é. None of this is unusual in a converted
+book, and all of it has broken a chunker at some point.
+
+Straße, Ægypt, first, ॐ. The list is arbitrary. Its purpose is to be awkward.

parity/golden/chunkmd/bibliography.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"## Select Bibliography\n\nArendt, Hannah. The Human Condition. Chicago: University of Chicago Press, 1958.\n\nBloch, Marc. Feudal Society. Translated by L. A. Manyon. London: Routledge, 1961.\n\nCassirer, Ernst. The Myth of the State. New Haven: Yale University Press, 1946.\n\n———. An Essay on Man. New Haven: Yale University Press, 1944.\n\nDuby, Georges. The Three Orders: Feudal Society Imagined. Chicago: University\nof Chicago Press, 1980.\n\nGanshof, F. L. Feudalism. Translated by Philip Grierson. New York: Harper,\n1961.\n\nHilton, R. H. Bond Men Made Free. London: Temple Smith, 1973.\n\nMaitland, F. W. Domesday Book and Beyond. Cambridge: Cambridge University\nPress, 1897.\n\nPostan, M. M. The Medieval Economy and Society. London: Weidenfeld, 1972.\n\nVinogradoff, Paul. Villainage in England. Oxford: Clarendon Press, 1892.","tokens":247,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/bibliography.small.ndjson

@@ -0,0 +1,9 @@
+{"v":1,"idx":0,"text":"## Select Bibliography\n\nArendt, Hannah. The Human Condition. Chicago: University of Chicago Press, 1958.\n\nBloch, Marc. Feudal Society. Translated by L. A. Manyon. London: Routledge, 1961.","tokens":55,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"Bloch, Marc. Feudal Society. Translated by L. A. Manyon. London: Routledge, 1961.\n\nCassirer, Ernst. The Myth of the State. New Haven: Yale University Press, 1946.","tokens":53,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"Cassirer, Ernst. The Myth of the State. New Haven: Yale University Press, 1946.\n\n———. An Essay on Man. New Haven: Yale University Press, 1944.","tokens":43,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":"———. An Essay on Man. New Haven: Yale University Press, 1944.\n\nDuby, Georges. The Three Orders: Feudal Society Imagined. Chicago: University\nof Chicago Press, 1980.","tokens":47,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"Duby, Georges. The Three Orders: Feudal Society Imagined. Chicago: University\nof Chicago Press, 1980.\n\nGanshof, F. L. Feudalism. Translated by Philip Grierson. New York: Harper,\n1961.","tokens":56,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":"Ganshof, F. L. Feudalism. Translated by Philip Grierson. New York: Harper,\n1961.\n\nHilton, R. H. Bond Men Made Free. London: Temple Smith, 1973.","tokens":49,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":"Hilton, R. H. Bond Men Made Free. London: Temple Smith, 1973.\n\nMaitland, F. W. Domesday Book and Beyond. Cambridge: Cambridge University\nPress, 1897.","tokens":47,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":7,"text":"Maitland, F. W. Domesday Book and Beyond. Cambridge: Cambridge University\nPress, 1897.\n\nPostan, M. M. The Medieval Economy and Society. London: Weidenfeld, 1972.","tokens":49,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":8,"text":"Postan, M. M. The Medieval Economy and Society. London: Weidenfeld, 1972.\n\nVinogradoff, Paul. Villainage in England. Oxford: Clarendon Press, 1892.","tokens":46,"page":null,"chapter":"Select Bibliography","role":"bibliography","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/contents.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"# Contents\n\nPreface......vii\nOne. The Setting......1\nTwo. The Quarrel......22\nThree. The Break......45\nFour. The Settlement......70\nFive. After the Settlement......99\nAppendix A. The Renders......121\nAppendix B. The Mills......134\nBibliography......150\nIndex......161","tokens":73,"page":null,"chapter":"Contents","role":"toc","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/contents.small.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"# Contents\n\nPreface......vii\nOne. The Setting......1\nTwo. The Quarrel......22\nThree. The Break......45\nFour. The Settlement......70\nFive. After the Settlement......99\nAppendix A. The Renders......121\nAppendix B. The Mills......134\nBibliography......150\nIndex......161","tokens":73,"page":null,"chapter":"Contents","role":"toc","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/epub-ish.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"## Chapter One\n\nThere are no page markers in this text, because the format it was converted\nfrom had none. A chunker that needs pages will produce nothing useful here,\nand one that falls back to character position will find the chapter correctly.\n\nThe chapter itself concerns the difference between a right and a habit. A\nright can be pleaded; a habit can only be described. The men in these records\npleaded habits as though they were rights, and the courts, with a patience\nthat is easy to underestimate, let them.\n\n## Chapter Two\n\nThe second chapter takes the same question into a period where the records are\nbetter. Better records make the problem worse rather than easier, because they\nshow how much of what looked like custom was in fact a settlement of a dispute\nthat had gone the other way a generation earlier.\n\nCustom, in this reading, is the name given to the outcome of a quarrel nobody\nalive remembers having. It is stable exactly as long as the memory of the\nquarrel is dead and the conditions that produced the settlement persist. When\neither changes, custom turns out to have been a treaty all along.\n\n## Chapter Three\n\nWhat remains is to say what this means for the wider argument, which is less\nthan it might appear. The valley was not typical. It was chosen because its\ndocuments survive, and documents survive where somebody had a reason to keep\nthem, which is itself a fact about the place.","tokens":298,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/epub-ish.small.ndjson

@@ -0,0 +1,9 @@
+{"v":1,"idx":0,"text":"## Chapter One\n\nThere are no page markers in this text, because the format it was converted\nfrom had none. A chunker that needs pages will produce nothing useful here,\nand one that falls back to character position will find the chapter correctly.","tokens":50,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"There are no page markers in this text, because the format it was converted\nfrom had none. A chunker that needs pages will produce nothing useful here,\nand one that falls back to character position will find the chapter correctly.\n\nThe chapter itself concerns the difference between a right and a habit. A\nright can be pleaded; a habit can only be described. The men in these records\npleaded habits as though they were rights, and the","tokens":90,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":" courts, with a patience\nthat is easy to underestimate, let them.","tokens":15,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":"The chapter itself concerns the difference between a right and a habit. A\nright can be pleaded; a habit can only be described. The men in these records\npleaded habits as though they were rights, and the courts, with a patience\nthat is easy to underestimate, let them.\n\n## Chapter Two","tokens":62,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"## Chapter Two\n\nThe second chapter takes the same question into a period where the records are\nbetter. Better records make the problem worse rather than easier, because they\nshow how much of what looked like custom was in fact a settlement of a dispute\nthat had gone the other way a generation earlier.","tokens":61,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":"The second chapter takes the same question into a period where the records are\nbetter. Better records make the problem worse rather than easier, because they\nshow how much of what looked like custom was in fact a settlement of a dispute\nthat had gone the other way a generation earlier.\n\nCustom, in this reading, is the name given to the outcome of a quarrel nobody\nalive remembers having. It is stable exactly as long as the memory of","tokens":90,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":" the\nquarrel is dead and the conditions that produced the settlement persist. When\neither changes, custom turns out to have been a treaty all along.","tokens":32,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":7,"text":"Custom, in this reading, is the name given to the outcome of a quarrel nobody\nalive remembers having. It is stable exactly as long as the memory of the\nquarrel is dead and the conditions that produced the settlement persist. When\neither changes, custom turns out to have been a treaty all along.\n\n## Chapter Three","tokens":68,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":8,"text":"## Chapter Three\n\nWhat remains is to say what this means for the wider argument, which is less\nthan it might appear. The valley was not typical. It was chosen because its\ndocuments survive, and documents survive where somebody had a reason to keep\nthem, which is itself a fact about the place.","tokens":63,"page":null,"chapter":"Chapter One","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/index.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"# Index\n\nabbey, 44, 47, 91\nadvowson, 12\nagriculture, 12, 34, 102\nassize, 7, 19-22\nbishops, 7\nboon-work, 88\ncharters, 88, 91\ncommons, 14, 15, 16\ndemesne, 14-16\nescheat, 120\nestates, 3, 44\nfealty, 201\nfrankpledge, 55, 56\nheriot, 61\nhomage, 3, 4, 201\nmanor, 1-8, 14, 33\nmerchet, 62\nmill, 71-79, 84\nvillein, 90, 91, 104\nwardship, 118","tokens":186,"page":null,"chapter":"Index","role":"index","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/index.small.ndjson

@@ -0,0 +1,3 @@
+{"v":1,"idx":0,"text":"# Index\n\nabbey, 44, 47, 91\nadvowson, 12\nagriculture, 12, 34, 102\nassize, 7, 19-22\nbishops, 7\nboon-work, 88\ncharters, 88, 91\ncommons, 14, 15, 16\ndemesne, 14-16\nesche","tokens":90,"page":null,"chapter":"Index","role":"index","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"at, 120\nestates, 3, 44\nfealty, 201\nfrankpledge, 55, 56\nheriot, 61\nhomage, 3, 4, 201\nmanor, 1-8, 14, 33\nmerchet, 62\nmill, 71-79, 84\nvillein, 90, 91, 104","tokens":90,"page":null,"chapter":"Index","role":"index","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"\nwardship, 118","tokens":6,"page":null,"chapter":"Index","role":"index","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/notes-sentinel.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"## Notes\n\n12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.\n\n13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.\n\n14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.\n\n15. Ibid., 91.\n\n16. https://example.org/records/valley, consulted 12 March.","tokens":170,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}

parity/golden/chunkmd/notes-sentinel.small.ndjson

@@ -0,0 +1,6 @@
+{"v":1,"idx":0,"text":"## Notes\n\n12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.","tokens":32,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":1,"text":"12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.\n\n13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.","tokens":65,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":2,"text":"13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.\n\n14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in","tokens":90,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":3,"text":" England* (Oxford: Clarendon Press, 1892), 88.","tokens":18,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.\n\n15. Ibid., 91.","tokens":82,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":5,"text":"15. Ibid., 91.\n\n16. https://example.org/records/valley, consulted 12 March.","tokens":30,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}

parity/golden/chunkmd/notes.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"## Notes\n\n12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.\n\n13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.\n\n14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.\n\n15. Ibid., 91.\n\n16. https://example.org/records/valley, consulted 12 March.","tokens":164,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}

parity/golden/chunkmd/notes.small.ndjson

@@ -0,0 +1,6 @@
+{"v":1,"idx":0,"text":"## Notes\n\n12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.","tokens":32,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":1,"text":"12. Arendt, *The Human Condition* (Chicago: University of Chicago Press, 1958), 44.\n\n13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.","tokens":65,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":2,"text":"13. Bloch, *Feudal Society*, vol. 2, p. 17. See also Duby, op. cit., 203.\n\n14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in","tokens":90,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":3,"text":" England* (Oxford: Clarendon Press, 1892), 88.","tokens":18,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"14. The point is made at length in Maitland, *Domesday Book and Beyond* (Cambridge, 1897), 220-24, and again, with qualifications, in Vinogradoff, *Villainage in England* (Oxford: Clarendon Press, 1892), 88.\n\n15. Ibid., 91.","tokens":79,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}
+{"v":1,"idx":5,"text":"15. Ibid., 91.\n\n16. https://example.org/records/valley, consulted 12 March.","tokens":24,"page":null,"chapter":"Notes","role":"bibliography","start_time":null,"end_time":null,"flags":["footnote_citations"]}

parity/golden/chunkmd/oversized.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"One sentence follows another without a paragraph break for the whole of this page, which is the case the sentence splitter exists for. The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest. Its wheel was undershot, which was wasteful, and nobody replaced it for two hundred years because the cost of replacing it fell on one household and the benefit was spread across four hundred. The abbey's mill, two miles downstream, was overshot and newer and better, and the abbey charged more to use it, which the valley resented and paid. When the lord's wheel finally failed, in a winter the chronicle calls hard without saying how hard, the valley ground at the abbey for eleven months and discovered that it preferred the abbey's flour. The lord rebuilt anyway. He had to: the right to compel grinding was worth more than the flour, because it was the visible sign that he could compel anything at all. A right that is never exercised is a right that is being tested, and he knew it, and so did they. The rebuilt mill ran for another century and was still running, badly, when the survey of 1439 recorded it as ruinous and in use. That phrase, ruinous and in use, is as good a description of the whole arrangement as anything a historian has managed since.","tokens":283,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/oversized.small.ndjson

@@ -0,0 +1,8 @@
+{"v":1,"idx":0,"text":"One sentence follows another without a paragraph break for the whole of this page, which is the case the sentence splitter exists for. The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest.","tokens":52,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"The mill stood at the northern end of the valley, where the fall of the river was steepest and the ice came latest. Its wheel was undershot, which was wasteful, and nobody replaced it for two hundred years because the cost of replacing it fell on one household and the benefit was spread across four hundred.","tokens":65,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"Its wheel was undershot, which was wasteful, and nobody replaced it for two hundred years because the cost of replacing it fell on one household and the benefit was spread across four hundred. The abbey's mill, two miles downstream, was overshot and newer and better, and the abbey charged more to use it, which the valley resented and paid.","tokens":74,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":"The abbey's mill, two miles downstream, was overshot and newer and better, and the abbey charged more to use it, which the valley resented and paid. When the lord's wheel finally failed, in a winter the chronicle calls hard without saying how hard, the valley ground at the abbey for eleven months and discovered that it preferred the abbey's flour.","tokens":78,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"When the lord's wheel finally failed, in a winter the chronicle calls hard without saying how hard, the valley ground at the abbey for eleven months and discovered that it preferred the abbey's flour. The lord rebuilt anyway.","tokens":47,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":"The lord rebuilt anyway. He had to: the right to compel grinding was worth more than the flour, because it was the visible sign that he could compel anything at all. A right that is never exercised is a right that is being tested, and he knew it, and so did they.","tokens":59,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":"A right that is never exercised is a right that is being tested, and he knew it, and so did they. The rebuilt mill ran for another century and was still running, badly, when the survey of 1439 recorded it as ruinous and in use.","tokens":54,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":7,"text":"The rebuilt mill ran for another century and was still running, badly, when the survey of 1439 recorded it as ruinous and in use.\n\nThat phrase, ruinous and in use, is as good a description of the whole arrangement as anything a historian has managed since.","tokens":56,"page":40,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/paged.default.ndjson

@@ -0,0 +1,3 @@
+{"v":1,"idx":0,"text":"# I. The Setting\n\nThe valley is twelve miles long and never more than two wide. At its northern\nend the river enters through a gorge that no road followed until the last\ncentury; at its southern end it leaves through a marsh that no road follows\nyet. Everything that happened here happened to people who could not easily\nleave, and who knew it.\n\nThe population in the first surviving survey is given as four hundred and six\nhouseholds, which at the usual multiplier means somewhere near two thousand\npeople. That figure held, within a tenth either way, for three hundred years.","tokens":122,"page":1,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"What changed was not how many people lived in the valley but what they owed\nfor the privilege. The earliest schedule lists eleven separate renders, seven\nof them in kind. The latest lists two, both in coin. Between those two\ndocuments lies the whole of the argument.\n\nIt would be convenient to say that the change was gradual. It was not. Two\nthirds of it happens in a single generation, in the middle of the fourteenth\ncentury, for reasons the documents do not give and which have to be supplied\nfrom outside.","tokens":112,"page":2,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"# II. The Quarrel\n\nThe quarrel began over a mill. It nearly always does. A mill is the one piece\nof capital in a village that everybody must use and only one person can own,\nand the law of the mill is therefore the law of the place in miniature.\n\nThe lord's mill stood at the northern end, where the fall was steepest. The\nabbey's mill stood two miles downstream, on land the abbey had been given by a\nman whose grandson disputed the gift. Both houses claimed the right to compel\nthe valley's tenants to grind at their own stones, and both were prepared to\nbreak the other's.","tokens":136,"page":3,"chapter":"II. The Quarrel","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/paged.small.ndjson

@@ -0,0 +1,11 @@
+{"v":1,"idx":0,"text":"This preface was set before the pagination began, and any tool that starts\nreading at the first page marker will lose it. That has happened, and it cost\na book all but two thousand of its characters.","tokens":44,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"# I. The Setting","tokens":5,"page":1,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"# I. The Setting\n\nThe valley is twelve miles long and never more than two wide. At its northern\nend the river enters through a gorge that no road followed until the last\ncentury; at its southern end it leaves through a marsh that no road follows\nyet. Everything that happened here happened to people who could not easily\nleave, and who knew it.","tokens":76,"page":1,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":"The valley is twelve miles long and never more than two wide. At its northern\nend the river enters through a gorge that no road followed until the last\ncentury; at its southern end it leaves through a marsh that no road follows\nyet. Everything that happened here happened to people who could not easily\nleave, and who knew it.\n\nThe population in the first surviving survey is given as four hundred and six\nhouseholds, which at","tokens":90,"page":1,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":" the usual multiplier means somewhere near two thousand\npeople. That figure held, within a tenth either way, for three hundred years.","tokens":26,"page":1,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":"What changed was not how many people lived in the valley but what they owed\nfor the privilege. The earliest schedule lists eleven separate renders, seven\nof them in kind. The latest lists two, both in coin. Between those two\ndocuments lies the whole of the argument.","tokens":56,"page":2,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":"What changed was not how many people lived in the valley but what they owed\nfor the privilege. The earliest schedule lists eleven separate renders, seven\nof them in kind. The latest lists two, both in coin. Between those two\ndocuments lies the whole of the argument.\n\nIt would be convenient to say that the change was gradual. It was not. Two\nthirds of it happens in a single generation, in the middle of the four","tokens":90,"page":2,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":7,"text":"teenth\ncentury, for reasons the documents do not give and which have to be supplied\nfrom outside.","tokens":22,"page":2,"chapter":"I. The Setting","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":8,"text":"# II. The Quarrel\n\nThe quarrel began over a mill. It nearly always does. A mill is the one piece\nof capital in a village that everybody must use and only one person can own,\nand the law of the mill is therefore the law of the place in miniature.","tokens":59,"page":3,"chapter":"II. The Quarrel","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":9,"text":"The quarrel began over a mill. It nearly always does. A mill is the one piece\nof capital in a village that everybody must use and only one person can own,\nand the law of the mill is therefore the law of the place in miniature.\n\nThe lord's mill stood at the northern end, where the fall was steepest. The\nabbey's mill stood two miles downstream, on land the abbey had been given by","tokens":90,"page":3,"chapter":"II. The Quarrel","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":10,"text":" a\nman whose grandson disputed the gift. Both houses claimed the right to compel\nthe valley's tenants to grind at their own stones, and both were prepared to\nbreak the other's.","tokens":39,"page":3,"chapter":"II. The Quarrel","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/prose.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"The argument of this chapter is that the obligation ran both ways, and that\ncontemporaries said so plainly whenever they were asked. A lord who could not\nprotect was a lord who could not command, and the men who owed him service\nsaid as much in language that has survived in the court rolls.\n\nWhat the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested. A tenant who withheld service in a year of\nfamine was not making a constitutional argument; he was hungry. The clerk who\nrecorded the withholding had no vocabulary for the difference, and so the\nrecord flattens the two into one offence with one penalty.\n\nThis flattening is the central difficulty of the sources. It is not that they\nlie. It is that the categories they were written in were designed to produce a\nnumber at the bottom of a column, and a number at the bottom of a column\ncannot distinguish a rebellion from a bad harvest. Everything that follows\ndepends on reading around that limitation rather than through it.\n\nThree approaches have been tried. The first counts the offences and treats the\ncount as a proxy for discontent, which assumes the clerk's attention was\nconstant across years when we know it was not. The second reads the language\nof the entries for signs of unusual trouble, which works where a clerk was\ntalkative and fails entirely where he was not. The third, adopted here,\ncompares the same manor across decades and treats changes in what was recorded\nas evidence about the recorder as much as about the recorded.\n\nNone of the three is satisfactory, and the third has the particular weakness\nthat it needs a long run of documents from a single place. There are not many\nsuch runs. The ones that exist are concentrated in the south and east, which\nis to say in exactly the regions where the argument is least likely to be\nsurprising. A reader who wants to know what happened in the north will find\nthis chapter unsatisfying, and the reason is that the documents do not exist,\nnot that the question is uninteresting.","tokens":433,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/prose.small.ndjson

@@ -0,0 +1,13 @@
+{"v":1,"idx":0,"text":"The argument of this chapter is that the obligation ran both ways, and that\ncontemporaries said so plainly whenever they were asked. A lord who could not\nprotect was a lord who could not command, and the men who owed him service\nsaid as much in language that has survived in the court rolls.","tokens":63,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"The argument of this chapter is that the obligation ran both ways, and that\ncontemporaries said so plainly whenever they were asked. A lord who could not\nprotect was a lord who could not command, and the men who owed him service\nsaid as much in language that has survived in the court rolls.\n\nWhat the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested. A","tokens":90,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":" tenant who withheld service in a year of\nfamine was not making a constitutional argument; he was hungry. The clerk who\nrecorded the withholding had no vocabulary for the difference, and so the\nrecord flattens the two into one offence with one penalty.","tokens":53,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":"What the rolls do not say, and what has to be inferred, is how often the\nobligation was actually tested. A tenant who withheld service in a year of\nfamine was not making a constitutional argument; he was hungry. The clerk who\nrecorded the withholding had no vocabulary for the difference, and so the\nrecord flattens the two into one offence with one penalty. This flattening is the central difficulty of the sources","tokens":90,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":". It is not that they\nlie. It is that the categories they were written in were designed to produce a\nnumber at the bottom of a column, and a number at the bottom of a column\ncannot distinguish a rebellion from a bad harvest. Everything that follows\ndepends on reading around that limitation rather than through it.","tokens":66,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":"This flattening is the central difficulty of the sources. It is not that they\nlie. It is that the categories they were written in were designed to produce a\nnumber at the bottom of a column, and a number at the bottom of a column\ncannot distinguish a rebellion from a bad harvest. Everything that follows\ndepends on reading around that limitation rather than through it. Three approaches have been tried.","tokens":82,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":"Three approaches have been tried. The first counts the offences and treats the\ncount as a proxy for discontent, which assumes the clerk's attention was\nconstant across years when we know it was not.","tokens":40,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":7,"text":"The first counts the offences and treats the\ncount as a proxy for discontent, which assumes the clerk's attention was\nconstant across years when we know it was not. The second reads the language\nof the entries for signs of unusual trouble, which works where a clerk was\ntalkative and fails entirely where he was not.","tokens":66,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":8,"text":"The second reads the language\nof the entries for signs of unusual trouble, which works where a clerk was\ntalkative and fails entirely where he was not. The third, adopted here,\ncompares the same manor across decades and treats changes in what was recorded\nas evidence about the recorder as much as about the recorded.","tokens":66,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":9,"text":"The third, adopted here,\ncompares the same manor across decades and treats changes in what was recorded\nas evidence about the recorder as much as about the recorded. None of the three is satisfactory, and the third has the particular weakness\nthat it needs a long run of documents from a single place.","tokens":62,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":10,"text":"None of the three is satisfactory, and the third has the particular weakness\nthat it needs a long run of documents from a single place. There are not many\nsuch runs.","tokens":36,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":11,"text":"There are not many\nsuch runs. The ones that exist are concentrated in the south and east, which\nis to say in exactly the regions where the argument is least likely to be\nsurprising.","tokens":41,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":12,"text":"The ones that exist are concentrated in the south and east, which\nis to say in exactly the regions where the argument is least likely to be\nsurprising.\n\nA reader who wants to know what happened in the north will find\nthis chapter unsatisfying, and the reason is that the documents do not exist,\nnot that the question is uninteresting.","tokens":72,"page":null,"chapter":null,"role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/unicode.default.ndjson

@@ -0,0 +1 @@
+{"v":1,"idx":0,"text":"# Ἀθῆναι and Μόσχα\n\nLe café était naïve, and the first ligature in that sentence is a single\ncodepoint rather than two letters, which is the sort of thing that moves a\ntoken boundary without moving a character boundary.\n\n東京から来た手紙には、日付が二つ書いてあった。One was the date it was written\nand the other was the date it was meant to be read, and the difference between\nthem is the whole subject of this section.\n\nمرحبا — the paragraph above is followed by a right-to-left greeting, then by a\ncombining acute on a bare e: é. None of this is unusual in a converted\nbook, and all of it has broken a chunker at some point.\n\nStraße, Ægypt, first, ॐ. The list is arbitrary. Its purpose is to be awkward.","tokens":207,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/golden/chunkmd/unicode.small.ndjson

@@ -0,0 +1,7 @@
+{"v":1,"idx":0,"text":"# Ἀθῆναι and Μόσχα","tokens":17,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":1,"text":"# Ἀθῆναι and Μόσχα\n\nLe café était naïve, and the first ligature in that sentence is a single\ncodepoint rather than two letters, which is the sort of thing that moves a\ntoken boundary without moving a character boundary.","tokens":63,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":2,"text":"Le café était naïve, and the first ligature in that sentence is a single\ncodepoint rather than two letters, which is the sort of thing that moves a\ntoken boundary without moving a character boundary.\n\n東京から来た手紙には、日付が二つ書いてあった。One was the date it was written\nand the other was the date it was meant to be read","tokens":90,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":3,"text":", and the difference between\nthem is the whole subject of this section.","tokens":15,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":4,"text":"東京から来た手紙には、日付が二つ書いてあった。One was the date it was written\nand the other was the date it was meant to be read, and the difference between\nthem is the whole subject of this section.\n\nمرحبا — the paragraph above is followed by a right-to-left greeting, then by a\ncombining acute on a bare e:","tokens":90,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":5,"text":" é. None of this is unusual in a converted\nbook, and all of it has broken a chunker at some point.","tokens":27,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
+{"v":1,"idx":6,"text":"مرحبا — the paragraph above is followed by a right-to-left greeting, then by a\ncombining acute on a bare e: é. None of this is unusual in a converted\nbook, and all of it has broken a chunker at some point.\n\nStraße, Ægypt, first, ॐ. The list is arbitrary. Its purpose is to be awkward.","tokens":84,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}

parity/oracle/chunkmd.py

@@ -0,0 +1,65 @@
+"""Chunk markdown the way the Python pipeline does, for comparison.
+
+Reproduces the production order exactly: chunk, then the footnote signal on
+the text *before* the structural marker is stripped, then role classification
+and smoothing across the whole document.
+
+    CATENA_PARITY_PYTHONPATH=/path/to/book-pipeline/src \
+        python3 oracle/chunkmd.py 512 64 50 768 < case.md
+
+Emits one JSON record per line, in the field order the Rust records use.
+"""
+
+import json
+import os
+import sys
+
+for path in os.environ.get("CATENA_PARITY_PYTHONPATH", "").split(os.pathsep):
+    if path:
+        sys.path.insert(0, path)
+
+from book_pipeline import chunk as chunker  # noqa: E402
+from book_pipeline.citations import looks_like_footnote_citations  # noqa: E402
+from book_pipeline.convert import FOOTNOTE_SENTINEL  # noqa: E402
+
+
+def main() -> int:
+    target, overlap, min_tokens, hard_max = (int(a) for a in sys.argv[1:5])
+    markdown = sys.stdin.buffer.read().decode("utf-8")
+
+    chunks = chunker.chunk_markdown(
+        markdown,
+        target=target,
+        overlap=overlap,
+        min_tokens=min_tokens,
+        hard_max=hard_max,
+    )
+
+    footnotes = {c.chunk_idx: looks_like_footnote_citations(c.text) for c in chunks}
+    for c in chunks:
+        c.text = c.text.replace(FOOTNOTE_SENTINEL, "")
+
+    roles = chunker.smooth_roles(
+        [chunker.classify_chunk_role(c.text, c.chapter_title) for c in chunks]
+    )
+
+    out = sys.stdout
+    for c, role in zip(chunks, roles, strict=True):
+        record = {
+            "v": 1,
+            "idx": c.chunk_idx,
+            "text": c.text,
+            "tokens": c.token_count,
+            "page": c.page,
+            "chapter": c.chapter_title,
+            "role": role,
+            "start_time": None,
+            "end_time": None,
+            "flags": ["footnote_citations"] if footnotes[c.chunk_idx] else [],
+        }
+        out.write(json.dumps(record, ensure_ascii=False, separators=(",", ":")) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

parity/src/bin/regen.rs

@@ -4,24 +4,40 @@
 //! it fails is a test that cannot fail. Run it deliberately, read the diff,
 //! and commit the change with the reason.
 //!
+//! ```text
+//! CATENA_PARITY_PYTHON=... CATENA_PARITY_PYTHONPATH=... \
 //!     cargo run -p parity --bin regen
+//! ```
 
 use std::process::ExitCode;
 
+/// What one oracle run produces: the golden files it wrote, or why it could
+/// not.
+type Written = Result<Vec<String>, String>;
+
+/// One oracle and the name it is reported under.
+type Job = (&'static str, fn() -> Written);
+
 fn main() -> ExitCode {
-    match regen_tokens() {
-        Ok(path) => {
-            eprintln!("wrote {path}");
-            ExitCode::SUCCESS
-        }
-        Err(e) => {
-            eprintln!("regen: {e}");
-            ExitCode::FAILURE
+    let jobs: [Job; 2] = [("tokens", tokens), ("chunkmd", chunkmd)];
+
+    for (name, job) in jobs {
+        match job() {
+            Ok(written) => {
+                for path in written {
+                    eprintln!("wrote {path}");
+                }
+            }
+            Err(e) => {
+                eprintln!("regen {name}: {e}");
+                return ExitCode::FAILURE;
+            }
         }
     }
+    ExitCode::SUCCESS
 }
 
-fn regen_tokens() -> Result<String, String> {
+fn tokens() -> Written {
     let header = format!(
         "# cl100k_base token counts, one per line of parity::sample::standard()\n\
          # seed={:#018x} lines={}\n",
@@ -31,5 +47,26 @@ fn regen_tokens() -> Result<String, String> {
     let counts = parity::run_oracle("tokens.py", &parity::sample::standard())?;
     let path = parity::golden("tokens.counts");
     std::fs::write(&path, format!("{header}{counts}")).map_err(|e| e.to_string())?;
-    Ok(path.display().to_string())
+    Ok(vec![path.display().to_string()])
+}
+
+fn chunkmd() -> Written {
+    let dir = parity::golden("chunkmd");
+    std::fs::create_dir_all(&dir).map_err(|e| e.to_string())?;
+
+    let mut written = Vec::new();
+    for case in parity::cases("chunkmd") {
+        let markdown = std::fs::read_to_string(&case).map_err(|e| e.to_string())?;
+        for (set, budgets) in parity::budgets::SETS {
+            let records = parity::run_oracle_with(
+                "chunkmd.py",
+                &parity::budgets::as_args(budgets),
+                &markdown,
+            )?;
+            let path = dir.join(format!("{}.{set}.ndjson", parity::case_name(&case)));
+            std::fs::write(&path, records).map_err(|e| e.to_string())?;
+            written.push(path.display().to_string());
+        }
+    }
+    Ok(written)
 }

parity/src/budgets.rs

@@ -0,0 +1,43 @@
+//! The budget sets the chunker is pinned at.
+//!
+//! Two of them: the defaults, and a deliberately small set that forces the
+//! flush, overlap and hard-maximum paths to run on a fixture small enough to
+//! read. Most of the interesting behaviour only appears at a boundary, and
+//! at the default budgets a fixture would have to be pages long to have one.
+
+use chunkmd::Budgets;
+
+/// Named budget sets, in the order goldens are generated.
+pub const SETS: &[(&str, Budgets)] = &[
+    (
+        "default",
+        Budgets {
+            target: 512,
+            overlap: 64,
+            min_tokens: 50,
+            hard_max: 768,
+        },
+    ),
+    (
+        "small",
+        Budgets {
+            target: 60,
+            overlap: 12,
+            min_tokens: 5,
+            hard_max: 90,
+        },
+    ),
+];
+
+/// The arguments the oracle takes for a budget set.
+pub fn as_args(budgets: &Budgets) -> Vec<String> {
+    [
+        budgets.target,
+        budgets.overlap,
+        budgets.min_tokens,
+        budgets.hard_max,
+    ]
+    .iter()
+    .map(usize::to_string)
+    .collect()
+}

parity/src/lib.rs

@@ -15,6 +15,7 @@
 //! committed here, and for copyrighted sources only the derived signal — the
 //! scores, the counts, the timings — is ever checked in.
 
+pub mod budgets;
 pub mod sample;
 
 use std::path::{Path, PathBuf};
@@ -49,12 +50,18 @@ pub fn python() -> String {
 /// directory, so nothing about where a Python pipeline lives on a particular
 /// machine is compiled in.
 pub fn run_oracle(script: &str, input: &str) -> Result<String, String> {
+    run_oracle_with(script, &[], input)
+}
+
+/// Run an oracle script with arguments.
+pub fn run_oracle_with(script: &str, args: &[String], input: &str) -> Result<String, String> {
     use std::io::Write as _;
     use std::process::{Command, Stdio};
 
     let script = oracle(script);
     let mut child = Command::new(python())
         .arg(&script)
+        .args(args)
         .stdin(Stdio::piped())
         .stdout(Stdio::piped())
         .stderr(Stdio::inherit())
@@ -76,6 +83,33 @@ pub fn run_oracle(script: &str, input: &str) -> Result<String, String> {
         .map_err(|e| format!("{} wrote invalid UTF-8: {e}", script.display()))
 }
 
+/// Every case file for a tool, in a fixed order.
+pub fn cases(tool: &str) -> Vec<PathBuf> {
+    let dir = dir().join("cases").join(tool);
+    let mut found: Vec<PathBuf> = std::fs::read_dir(&dir)
+        .unwrap_or_else(|e| panic!("reading {}: {e}", dir.display()))
+        .filter_map(Result::ok)
+        .map(|entry| entry.path())
+        .filter(|path| path.extension().is_some_and(|ext| ext == "md"))
+        .collect();
+    found.sort();
+    found
+}
+
+/// The stem of a case path, which names its golden file.
+pub fn case_name(path: &Path) -> String {
+    path.file_stem()
+        .expect("a case file has a name")
+        .to_string_lossy()
+        .into_owned()
+}
+
+/// Read a golden file whole.
+pub fn read_golden_text(name: &str) -> String {
+    let path = golden(name);
+    std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("reading {}: {e}", path.display()))
+}
+
 /// Read a golden file, dropping its `#` header lines.
 pub fn read_golden(name: &str) -> Vec<String> {
     let path = golden(name);

parity/tests/chunkmd.rs

@@ -0,0 +1,92 @@
+//! Chunk boundaries, token counts, provenance, roles and flags must match the
+//! Python exactly.
+//!
+//! A boundary that moves is not a cosmetic difference: every passage after it
+//! is a different passage than the one already embedded, so a re-ingest would
+//! silently replace an index rather than extend it.
+
+use serde_json::Value;
+
+fn records(text: &str) -> Vec<Value> {
+    text.lines()
+        .filter(|line| !line.trim().is_empty())
+        .map(|line| serde_json::from_str(line).expect("the oracle writes valid JSON"))
+        .collect()
+}
+
+fn rust_records(markdown: &str, budgets: &chunkmd::Budgets) -> Vec<Value> {
+    chunkmd::chunk(markdown, budgets)
+        .iter()
+        .map(|chunk| serde_json::to_value(chunk).expect("a chunk serialises"))
+        .collect()
+}
+
+fn compare(case: &str, set: &str, ours: &[Value], theirs: &[Value]) {
+    for (at, (ours, theirs)) in ours.iter().zip(theirs).enumerate() {
+        if ours != theirs {
+            let fields = ["idx", "page", "chapter", "role", "tokens", "flags", "text"];
+            let differing: Vec<String> = fields
+                .iter()
+                .filter(|field| ours.get(*field) != theirs.get(*field))
+                .map(|field| {
+                    format!(
+                        "  {field}: python {}, rust {}",
+                        theirs.get(field).unwrap_or(&Value::Null),
+                        ours.get(field).unwrap_or(&Value::Null)
+                    )
+                })
+                .collect();
+            panic!(
+                "{case} [{set}] chunk {at} differs:\n{}",
+                differing.join("\n")
+            );
+        }
+    }
+    assert_eq!(
+        ours.len(),
+        theirs.len(),
+        "{case} [{set}]: python produced {} chunks, rust {}",
+        theirs.len(),
+        ours.len()
+    );
+}
+
+#[test]
+fn every_case_chunks_the_same_as_python() {
+    let cases = parity::cases("chunkmd");
+    assert!(!cases.is_empty(), "no cases found");
+
+    for case in cases {
+        let name = parity::case_name(&case);
+        let markdown = std::fs::read_to_string(&case).expect("a readable case");
+        for (set, budgets) in parity::budgets::SETS {
+            let golden = parity::read_golden_text(&format!("chunkmd/{name}.{set}.ndjson"));
+            compare(
+                &name,
+                set,
+                &rust_records(&markdown, budgets),
+                &records(&golden),
+            );
+        }
+    }
+}
+
+/// Catches a golden file drifting away from a Python that has moved on.
+#[test]
+#[cfg(feature = "python")]
+fn the_golden_files_still_match_live_python() {
+    for case in parity::cases("chunkmd") {
+        let name = parity::case_name(&case);
+        let markdown = std::fs::read_to_string(&case).expect("a readable case");
+        for (set, budgets) in parity::budgets::SETS {
+            let live = parity::run_oracle_with(
+                "chunkmd.py",
+                &parity::budgets::as_args(budgets),
+                &markdown,
+            )
+            .expect("the oracle should run");
+            let golden = parity::read_golden_text(&format!("chunkmd/{name}.{set}.ndjson"));
+            assert_eq!(live, golden, "golden file for {name} [{set}] is stale");
+        }
+    }
+}