Files
catena/parity/golden/chunkmd/unicode.small.ndjson
T

8 lines
2.4 KiB
Plaintext

{"v":1,"idx":0,"text":"# Ἀθῆναι and Μόσχα","tokens":17,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":1,"text":"# Ἀθῆναι and Μόσχα\n\nLe café était naïve, and the first ligature in that sentence is a single\ncodepoint rather than two letters, which is the sort of thing that moves a\ntoken boundary without moving a character boundary.","tokens":63,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":2,"text":"Le café était naïve, and the first ligature in that sentence is a single\ncodepoint rather than two letters, which is the sort of thing that moves a\ntoken boundary without moving a character boundary.\n\n東京から来た手紙には、日付が二つ書いてあった。One was the date it was written\nand the other was the date it was meant to be read","tokens":90,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":3,"text":", and the difference between\nthem is the whole subject of this section.","tokens":15,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":4,"text":"東京から来た手紙には、日付が二つ書いてあった。One was the date it was written\nand the other was the date it was meant to be read, and the difference between\nthem is the whole subject of this section.\n\nمرحبا — the paragraph above is followed by a right-to-left greeting, then by a\ncombining acute on a bare e:","tokens":90,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":5,"text":" é. None of this is unusual in a converted\nbook, and all of it has broken a chunker at some point.","tokens":27,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}
{"v":1,"idx":6,"text":"مرحبا — the paragraph above is followed by a right-to-left greeting, then by a\ncombining acute on a bare e: é. None of this is unusual in a converted\nbook, and all of it has broken a chunker at some point.\n\nStraße, Ægypt, first, ॐ. The list is arbitrary. Its purpose is to be awkward.","tokens":84,"page":5,"chapter":"Ἀθῆναι and Μόσχα","role":"body","start_time":null,"end_time":null,"flags":[]}