{"v":1,"title":"The Human Condition","author":"Arendt, Hannah","isbn":"9780226586601","path":"/library/9780226586601.pdf","reasons":["allcaps_title","missing_isbn"],"severity":2,"changes":[{"field":"title","from":"THE HUMAN CONDITION","to":"The Human Condition","why":"allcaps_title"},{"field":"isbn","from":"","to":"9780226586601","why":"missing_isbn"}]}
{"v":1,"title":"Untitled","author":"Arguing About War","isbn":null,"path":null,"reasons":["placeholder_title","title_in_author","missing_isbn"],"severity":7,"changes":[]}
{"v":1,"title":"Microsoft Word - ch3.doc","author":"Oxford University Press","isbn":null,"path":null,"reasons":["filename_title","junk_title","publisher_as_author","missing_isbn"],"severity":9,"changes":[]}
{"v":1,"title":"The Fall: And the Rise","author":"Camus, Albert","isbn":null,"path":null,"shelf":"upstairs","reasons":["underscore_title","missing_isbn"],"severity":1,"changes":[{"field":"title","from":"The_Fall_ And the Rise_txt","to":"The Fall: And the Rise","why":"underscore_title"}]}
```
Three things to notice: the ISBN was recovered from the filename and the
check digit confirmed it; the second record was not touched, because what it
needs is a lookup and guessing would be worse; and `"shelf": "upstairs"` came
through untouched, because fields this does not know about are carried.
A book says who wrote it on its own copyright page. Hand in the opening pages
as `front_matter` and a record whose author is missing gets a name proposed,
with the evidence and how far it is worth trusting:
```
$ bibclean authors.ndjson
{"v":1,"title":"The Icon and the Axe","author":"Unknown",…,"reasons":["unknown_author","missing_isbn"],"severity":3,"changes":[],"author_proposal":{"author":"James H. Billington","method":"front_matter_copyright","confidence":0.9,"evidence":"…by James H. Billington","isbn":"0394708461"}}
{"v":1,"title":"Tragedy and Hope by Carroll Quigley","author":"",…,"reasons":["unknown_author"],"severity":3,"changes":[],"author_proposal":{"author":"Carroll Quigley","method":"title_by","confidence":0.7,"evidence":"by Carroll Quigley","isbn":null}}
```
Four signals, cheapest and most trustworthy first:
A slug is a public identifier — it reaches citations and notes, and outlives
whatever produced it. When a record carries one, along with the `id` that
makes it unique, its slug is judged too:
```
$ bibclean slugs.ndjson
{"v":1,"title":"Weaponizing Anthropology","author":"Laura Nader",…,"slug":"william-zinsser-on-writing-well-3451","id":3451,"reasons":[],"severity":0,"changes":[],"slug_proposal":{"current":"william-zinsser-on-writing-well-3451","proposed":"laura-nader-weaponizing-anthropology-3451","tier":"wrong-book","reasons":["only 0% of slug tokens appear in the current title/author — slug and metadata disagree"]}}
{"v":1,"title":"Mind and Brain","author":"Zinsser, William",…,"slug":"unknown-mind-and-brain-1234","id":1234,"reasons":[],"severity":0,"changes":[],"slug_proposal":{"current":"unknown-mind-and-brain-1234","proposed":"zinsser-william-mind-and-brain-1234","tier":"placeholder-author","reasons":["slug opens with placeholder author 'unknown'"]}}
```
Nothing is renamed. A proposal comes back beside the slug the record has, and
the **tier** says how much the evidence is worth:
| Tier | Evidence | Safe to apply in bulk? |
|---|---|---|
| `placeholder-author` | The slug opens with a placeholder, so it was minted without an author | yes |
| `filename-artefact` | The slug carries a mirror suffix, a wrapper format or a path | yes |
| `wrong-book` | Too few of the slug's words appear in the record at all | no |
| `noop-metadata-junk` | Regenerating gives the same slug: the record is the problem | no |
| `defer-new-slug-junk` | The new slug would be no better than the old one | no |
| `review-loses-volume` | The current slug names a volume the new one would lose | no |
| `review-low-recall` | The new slug recovers too little of the title | no |
The first two rest on evidence inside the slug itself and cannot be wrong
about it. The rest are *disagreements* between the slug and the record, and a
disagreement does not say which side is at fault — which is exactly the case
| `missing_apostrophe` | 2 | "Lossky s Argument" |
| `possible_swap` | 2 | Title and author look to be the wrong way round |
| `underscore_title` | 1 | Underscores standing in for spaces and colons |
| `missing_isbn` | 0 | Reported, but not a reason to do anything |
Each test is deliberately conservative. A real book called "The Document" or
"A Test of Faith" must not be swept up, so the placeholder tests only fire
when the *entire* field is the placeholder; and "Oxford University Press" in
an author field is wrong while "Oxford, Michael" is a person.
## The repairs it makes
| Fault | Repair |
|---|---|
| `underscore_title` | A trailing format name comes off, an underscore before a subtitle becomes a colon, the rest become spaces |
| `allcaps_title` | Title-cased, keeping small words small, hyphenated parts cased on both sides, and accented first letters capitalised rather than skipped |
| `missing_isbn` | Filled from the filename, if a checksum-valid ISBN is in it |
Everything else is named and left alone, because the fix needs a lookup and a
wrong guess in a catalogue is harder to find than a missing value.
{"kind":"tag","action":"delete","keep":null,"drop":"z-lib.org",…,"method":"junk_tag:format-or-watermark","confidence":0.92,"evidence":"format-or-watermark: 'z-lib.org' on 300 books"}
{"kind":"tag","action":"delete","keep":null,"drop":"9780198237907",…,"method":"junk_tag:isbn","confidence":0.92,"evidence":"isbn: '9780198237907' on 1 books"}
```
`18th Century` and `Philosophy` are left alone: the deletion rules are
high-precision on purpose, because a real subject swept up by one is a shelf
nobody can find again.
### Input
One heading per line — an `id`, a `name`, and the `count` of works under it:
```json
{"id":1,"name":"Dostoevsky, Fyodor","count":3}
```
### Options
| Option | Effect |
|---|---|
| `--kind <author\|publisher\|tag>` | What kind of heading the list holds |
| `--rules <FILE>` | Vocabulary specific to this collection |
Which subjects a collection uses is the most collection-specific thing there
is — a theology library and a science library agree on none of them — so it
arrives in the file rather than compiled in.
With one, every heading nothing else has already claimed gets an answer: a
spelling the scheme knows is a merge onto the spelling it uses, and a heading
the scheme does not have at all is a delete.
```
$ vocabmerge --kind tag --rules scheme.json tags.ndjson
{"kind":"tag","action":"merge","keep":"Philosophy","drop":"Philosophie",…,"method":"scheme_alias","confidence":0.95,"evidence":"the scheme files \"Philosophie\" under \"Philosophy\""}
{"kind":"tag","action":"delete","keep":null,"drop":"General",…,"method":"scheme_absent","confidence":0.8,"evidence":"\"General\" is not in the scheme, on 88 books"}
{"kind":"tag","action":"delete","keep":null,"drop":"Underwater Basketry",…,"method":"scheme_absent","confidence":0.8,"evidence":"\"Underwater Basketry\" is not in the scheme, on 1 books"}
{"kind":"tag","action":"merge","keep":"History","drop":"Church History",…,"method":"scheme_alias","confidence":0.95,"evidence":"the scheme files \"Church History\" under \"History\""}
```
The deletes are the lower confidence of the two on purpose: an alias is a
statement the scheme makes, while an absence only says the scheme does not
list it — which may equally mean the scheme is short a heading.
Four rules the scheme applies, for anything using it as a library:
- **Parents come along.** Phenomenology files under Philosophy and
Phenomenology, so a shelf browsed by the parent still finds the child.
- **Order is scheme order**, not the order the headings arrived in.
- **A study is not a novel.** A work already under a heading in
`blocks_fiction` — or whose title says "Companion to", "A Life of", "Diary",
"Very Short Introduction" — loses the `fiction_genres`. A historical novel
may still be both.
- **A book tagged only with headings the scheme rejects has no subject yet**,
which is what a catalogue built from filenames looks like.
