funkpower/catena
1
0
Fork 0
mirror of https://git.pumpin.online/funkpower/catena.git synced 2026-10-22 18:57:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

idnorm and bibfetch: an identifier read, and a register asked

Browse Source
This commit is contained in:
funkpower
2026-09-19 19:19:54 -04:00
parent 4d14544409
commit d47881ef3b
29 changed files with 1920 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/idnorm/Cargo.toml
+21
View File
@@ -0,0 +1,21 @@
[package]
name = "idnorm"
description = "The identifiers a bibliography is held together by, read and checked."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
bibclean.workspace = true
catena-cli.workspace = true
clap.workspace = true
corpus-core.workspace = true
fancy-regex.workspace = true
schemars.workspace = true
serde.workspace = true
serde_json.workspace = true
[lints]
workspace = true
crates/idnorm/README.md
+126
View File
@@ -0,0 +1,126 @@
# idnorm
The identifiers a bibliography is held together by, read and checked.
Part of [catena](../../README.md).
## What it does
A DOI arrives bare, as a URL, with a `doi:` prefix, with the full stop the
sentence around it left behind, in whatever case the publisher felt like. All
of those are one identifier. A catalogue that stores them as they arrived
cannot tell, so the same paper is three records and none of them join up.
`idnorm` reads one out of whatever it is wrapped in and writes it in the form
it should be stored and compared in.
## When you would use it
- **Before storing a reference**, so that two spellings of one DOI are one row.
- **Auditing a bibliography** for identifiers that cannot be right — an ISBN
and an ISSN carry a check digit, and a mistyped one fails it.
- **Pulling identifiers out of running text**, where the surrounding sentence
is attached to them.
## Install
```sh
cargo install --git https://git.pumpin.online/funkpower/catena idnorm
```
## Example
```
$ printf '10.1000/182\ndoi:10.1093/mind/LIX.236.433\n0028-0836\n978-0-306-40615-7\narXiv:2401.01234\n' | idnorm
{"v":1,"input":"10.1000/182","kind":"doi","id":"10.1000/182","checksum":null,"url":"https://doi.org/10.1000/182"}
{"v":1,"input":"doi:10.1093/mind/LIX.236.433","kind":"doi","id":"10.1093/mind/lix.236.433","checksum":null,"url":"https://doi.org/10.1093/mind/lix.236.433"}
{"v":1,"input":"0028-0836","kind":"issn","id":"0028-0836","checksum":true,"url":"https://portal.issn.org/resource/ISSN/0028-0836"}
{"v":1,"input":"978-0-306-40615-7","kind":"isbn","id":"9780306406157","checksum":true,"url":"https://openlibrary.org/isbn/9780306406157"}
{"v":1,"input":"arXiv:2401.01234","kind":"arxiv","id":"2401.01234","checksum":null,"url":"https://arxiv.org/abs/2401.01234"}
```
The ones that cannot be right:
```sh
idnorm refs.txt | jq -r 'select(.checksum == false) | .input'
```
What a line was not:
```
$ printf 'hello\n' | idnorm
idnorm: not an identifier: hello
$ echo $?
1
```
## Options
| Option | Effect |
|---|---|
| `--kind <doi\|issn\|isbn\|arxiv>` | Read every line as this kind rather than whatever it looks like |
| `--quiet` | Say nothing about the lines that are not identifiers |
| `--man` | Write this page as a manual page, in roff |
## Exit codes
| Code | Meaning |
|---|---|
| `0` | Every line was an identifier |
| `1` | At least one was not |
| `64` | The command line was wrong |
| `74` | The file could not be read |
## What it can and cannot tell you
**It can say a string is well formed.** An ISBN and an ISSN carry a check
digit, weighted by position, which catches one mistyped figure and two
transposed ones. A DOI and an arXiv identifier carry no checksum at all, so
for those `checksum` is `null` rather than `true` — there is nothing to check
against, and saying `true` would claim more than is known.
**It cannot say the work exists.** That is a fact about the world rather than
about the string, and the only way to learn it is to ask a register.
[`bibfetch`](../bibfetch/README.md) does that:
```sh
idnorm refs.txt | bibfetch | jq -r 'select(.found | not) | .id'
```
Those two together are what "validate against a catalogue" means, and they are
two tools because one of them touches the network and the other must not.
## The four kinds
**DOI.** A registrant prefix beginning `10.` and a suffix the registrant
chooses. Stored lowercased, because a DOI is case-insensitive and two
catalogues will disagree about the case of one. Read out of running text, with
the trailing full stop, comma or bracket the sentence left on it removed.
**ISSN.** Eight digits in two fours, the last a check digit that may be `X`
for ten. Stored hyphenated.
**ISBN.** Ten or thirteen digits. Stored as digits, without the hyphens, since
where a publisher puts them varies. Only where the line is the identifier and
nothing else — an ISBN has no prefix announcing it, so a sentence with a
thirteen-digit number in it would otherwise read as one.
**arXiv.** Both forms: `2401.01234` since 2007, and `math.GT/0309136` before
it. A version suffix is kept, because `v2` names a particular revision and
citing without one means the latest.
## As a library
```rust
let found = idnorm::parse("https://doi.org/10.1000/182").expect("an identifier");
assert_eq!(found.kind, idnorm::Kind::Doi);
assert_eq!(found.id, "10.1000/182");
```
`idnorm::doi`, `idnorm::issn` and `idnorm::arxiv` read one kind each, and
`idnorm::issn::valid` checks a serial number's check digit.
## See also
- [`bibfetch`](../bibfetch/README.md) — what a register says about one.
- [`bibclean`](../bibclean/README.md) — the rest of a catalogue record.
crates/idnorm/src/arxiv.rs
+122
View File
@@ -0,0 +1,122 @@
//! arXiv preprint identifiers.
//!
//! Two forms, because the scheme changed in 2007. The new one is four digits
//! of year and month, a dot, and a number: `2401.01234`. The old one is an
//! archive name, optionally a subject class, a slash, and seven digits:
//! `math.GT/0309136`.
//!
//! Either may carry a version — `v2` — which is part of the identifier: it
//! names a particular revision, and citing one without it means the latest.
use std::sync::LazyLock;
use fancy_regex::Regex;
/// The post-2007 form: `YYMM.NNNNN`, four or five digits after the dot.
static MODERN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?i)\b(\d{4}\.\d{4,5}(?:v\d+)?)\b").expect("a valid pattern"));
/// The pre-2007 form: an archive, an optional subject class, then seven
/// digits.
static LEGACY: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r"(?i)\b([a-z][a-z-]{1,20}(?:\.[A-Z]{2})?/\d{7}(?:v\d+)?)\b")
.expect("a valid pattern")
});
/// Read an arXiv identifier out of a string.
///
/// The old form keeps the case of its subject class, which is capitalised by
/// convention and is how the archives are written; the rest is lowercased.
pub fn parse(text: &str) -> Option<String> {
let text = text.trim();
let without = strip_prefix(text);
if let Some(found) = LEGACY.captures(without).ok().flatten() {
let id = found.get(1)?.as_str();
let (archive, number) = id.split_once('/')?;
return Some(format!(
"{}/{}",
lowercase_archive(archive),
number.to_lowercase()
));
}
let found = MODERN.captures(without).ok().flatten()?;
Some(found.get(1)?.as_str().to_lowercase())
}
/// The archive lowercased, but its subject class left as it was written.
fn lowercase_archive(archive: &str) -> String {
match archive.split_once('.') {
Some((name, class)) => format!("{}.{}", name.to_lowercase(), class.to_uppercase()),
None => archive.to_lowercase(),
}
}
/// Whatever introduced it.
fn strip_prefix(text: &str) -> &str {
for prefix in [
"https://arxiv.org/abs/",
"http://arxiv.org/abs/",
"arxiv:",
"arxiv ",
] {
if text.len() >= prefix.len() && text[..prefix.len()].eq_ignore_ascii_case(prefix) {
return text[prefix.len()..].trim();
}
}
text
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_modern_form_reads_as_itself() {
assert_eq!(parse("2401.01234").as_deref(), Some("2401.01234"));
// Four digits after the dot before 2015, five after it.
assert_eq!(parse("0704.0001").as_deref(), Some("0704.0001"));
}
#[test]
fn a_version_is_part_of_the_identifier() {
// It names a revision; without it a citation means the latest.
assert_eq!(parse("2401.01234v2").as_deref(), Some("2401.01234v2"));
assert_eq!(
parse("math.GT/0309136v1").as_deref(),
Some("math.GT/0309136v1")
);
}
#[test]
fn the_old_form_keeps_its_subject_class_capitalised() {
assert_eq!(parse("math.GT/0309136").as_deref(), Some("math.GT/0309136"));
assert_eq!(parse("MATH.gt/0309136").as_deref(), Some("math.GT/0309136"));
assert_eq!(parse("hep-th/9901001").as_deref(), Some("hep-th/9901001"));
}
#[test]
fn every_way_of_introducing_one_gives_the_same_identifier() {
for text in [
"arXiv:2401.01234",
"arxiv:2401.01234",
"arXiv 2401.01234",
"https://arxiv.org/abs/2401.01234",
] {
assert_eq!(parse(text).as_deref(), Some("2401.01234"), "{text}");
}
}
#[test]
fn what_is_not_an_arxiv_identifier_is_not_one() {
for text in ["", "hello", "24.01234", "2401.012", "2401.012345"] {
assert_eq!(parse(text), None, "{text:?}");
}
}
#[test]
fn a_doi_is_not_mistaken_for_one() {
// `10.1000/182` has a slash and digits, but not seven of them.
assert_eq!(parse("10.1000/182"), None);
}
}
crates/idnorm/src/doi.rs
+129
View File
@@ -0,0 +1,129 @@
//! Digital object identifiers.
//!
//! A DOI is a registrant prefix beginning `10.` and a suffix the registrant
//! chooses, separated by a slash. The suffix may be almost anything, which is
//! what makes reading one out of running text hard: there is no character
//! that reliably ends it.
use std::sync::LazyLock;
use fancy_regex::Regex;
/// The shape, wherever it appears and however it was introduced.
///
/// The suffix stops at whitespace or at one of the brackets and quotes that
/// wrap a citation, because those are never part of one in practice and
/// leaving them on makes two spellings of the same identifier.
static DOI: LazyLock<Regex> = LazyLock::new(|| {
Regex::new(r#"(?i)\b(10\.\d{4,9}/[^\s"'<>,;()\[\]{}]+)"#).expect("a valid pattern")
});
/// The prefixes a DOI arrives wearing.
const PREFIXES: &[&str] = &[
"https://doi.org/",
"http://doi.org/",
"https://dx.doi.org/",
"http://dx.doi.org/",
"doi:",
"doi ",
];
/// Read a DOI out of a string, in the form it should be stored in.
///
/// Lowercased, because a DOI is case-insensitive and two catalogues will
/// disagree about the case of the same one. The trailing punctuation a
/// sentence leaves behind is removed.
pub fn parse(text: &str) -> Option<String> {
let text = text.trim();
let mut without = text;
for prefix in PREFIXES {
if without.len() >= prefix.len() && without[..prefix.len()].eq_ignore_ascii_case(prefix) {
without = &without[prefix.len()..];
break;
}
}
let found = DOI.captures(without).ok().flatten()?;
let id = found.get(1)?.as_str();
// A full stop, comma or semicolon at the end belongs to the sentence
// rather than to the identifier. A closing bracket does too, unless the
// suffix opened one.
let id = id.trim_end_matches(['.', ',', ';', ':']);
if id.len() < 6 || !id.contains('/') {
return None;
}
// A suffix of nothing is not a suffix.
if id.split_once('/')?.1.is_empty() {
return None;
}
Some(id.to_lowercase())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_plain_form_reads_as_itself() {
assert_eq!(parse("10.1000/182").as_deref(), Some("10.1000/182"));
}
#[test]
fn every_way_of_introducing_one_gives_the_same_identifier() {
for text in [
"doi:10.1000/182",
"DOI: 10.1000/182",
"https://doi.org/10.1000/182",
"http://dx.doi.org/10.1000/182",
"HTTPS://DOI.ORG/10.1000/182",
] {
assert_eq!(parse(text).as_deref(), Some("10.1000/182"), "{text}");
}
}
#[test]
fn the_sentence_around_it_is_not_part_of_it() {
assert_eq!(
parse("See 10.1093/mind/LIX.236.433, which argues otherwise.").as_deref(),
Some("10.1093/mind/lix.236.433")
);
assert_eq!(parse("(10.1000/182)").as_deref(), Some("10.1000/182"));
assert_eq!(parse("\"10.1000/182\"").as_deref(), Some("10.1000/182"));
}
#[test]
fn case_is_not_part_of_it_either() {
// A DOI is case-insensitive, and two catalogues will disagree.
assert_eq!(
parse("10.1093/MIND/LIX.236.433").as_deref(),
parse("10.1093/mind/lix.236.433").as_deref()
);
}
#[test]
fn a_suffix_may_hold_almost_anything() {
assert_eq!(
parse("10.1371/journal.pone.0012345").as_deref(),
Some("10.1371/journal.pone.0012345")
);
assert_eq!(
parse("10.1016/j.cell.2020.01.001").as_deref(),
Some("10.1016/j.cell.2020.01.001")
);
}
#[test]
fn what_is_not_a_doi_is_not_one() {
for text in ["", "10.", "10.1000", "11.1000/182", "10.100/182", "hello"] {
assert_eq!(parse(text), None, "{text:?}");
}
}
#[test]
fn a_prefix_with_no_identifier_after_it_is_nothing() {
assert_eq!(parse("doi:"), None);
assert_eq!(parse("https://doi.org/"), None);
assert_eq!(parse("10.1000/"), None);
}
}
crates/idnorm/src/issn.rs
+129
View File
@@ -0,0 +1,129 @@
//! International standard serial numbers, which identify a journal.
//!
//! Eight digits in two groups of four, the last of which is a check digit
//! that may be `X` for ten.
use std::sync::LazyLock;
use fancy_regex::Regex;
/// Eight characters in two fours, with or without the hyphen, and with
/// whatever introduced it.
static ISSN: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?i)\b(\d{4})-?(\d{3}[\dX])\b").expect("a valid pattern"));
/// Read an ISSN out of a string, hyphenated as it should be stored.
pub fn parse(text: &str) -> Option<String> {
let text = text.trim();
let without = text
.strip_prefix("ISSN ")
.or_else(|| text.strip_prefix("issn "))
.or_else(|| text.strip_prefix("ISSN:"))
.or_else(|| text.strip_prefix("issn:"))
.unwrap_or(text)
.trim();
let found = ISSN.captures(without).ok().flatten()?;
Some(format!(
"{}-{}",
found.get(1)?.as_str(),
found.get(2)?.as_str().to_uppercase()
))
}
/// Whether the check digit agrees with the other seven.
///
/// Each of the first seven digits is weighted by its position counting down
/// from eight, and the total plus the check digit must be a multiple of
/// eleven. One mistyped digit, or two transposed, fails.
pub fn valid(issn: &str) -> bool {
let digits: Vec<char> = issn
.chars()
.filter(|c| !c.is_whitespace() && *c != '-')
.collect();
if digits.len() != 8 {
return false;
}
let mut total = 0u32;
for (index, digit) in digits[..7].iter().enumerate() {
let Some(value) = digit.to_digit(10) else {
return false;
};
total += value * (8 - index as u32);
}
let check = match digits[7] {
'X' | 'x' => 10,
digit => match digit.to_digit(10) {
Some(value) => value,
None => return false,
},
};
(total + check) % 11 == 0
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_real_issn_checks_out() {
// Nature.
assert!(valid("0028-0836"));
assert!(valid("00280836"));
// Science.
assert!(valid("0036-8075"));
}
#[test]
fn one_digit_out_does_not() {
assert!(!valid("0028-0837"));
assert!(!valid("0028-0835"));
}
#[test]
fn two_digits_swapped_does_not_either() {
// Which is the whole point of a weighted checksum.
assert!(!valid("0082-0836"));
}
#[test]
fn a_check_digit_of_ten_is_written_x() {
assert!(valid("1050-124X"));
assert!(valid("1050-124x"), "however it is cased");
}
#[test]
fn what_is_the_wrong_length_or_shape_is_not_valid() {
assert!(!valid(""));
assert!(!valid("0028-083"));
assert!(!valid("0028-08366"));
assert!(!valid("0028-083Y"));
assert!(!valid("ABCD-EFGH"));
}
#[test]
fn it_is_read_however_it_was_written() {
for text in [
"0028-0836",
"00280836",
"ISSN 0028-0836",
"ISSN: 0028-0836",
" 0028-0836 ",
] {
assert_eq!(parse(text).as_deref(), Some("0028-0836"), "{text}");
}
}
#[test]
fn a_lower_case_check_digit_is_stored_upper() {
assert_eq!(parse("1050-124x").as_deref(), Some("1050-124X"));
}
#[test]
fn what_is_not_an_issn_is_not_one() {
for text in ["", "hello", "123", "0028-08AB"] {
assert_eq!(parse(text), None, "{text:?}");
}
}
}
crates/idnorm/src/lib.rs
+242
View File
@@ -0,0 +1,242 @@
//! The identifiers a bibliography is held together by, read and checked.
//!
//! A DOI arrives as a bare string, a URL, a `doi:` prefix, with a trailing
//! full stop the sentence around it left behind, in whatever case the
//! publisher felt like. All of those are the same identifier, and a catalogue
//! that stores them as they arrived cannot tell that.
//!
//! Nothing here asks anybody whether an identifier exists. What it can say is
//! whether a string is *well formed*: an ISBN and an ISSN carry a check digit
//! that catches a mistyped one, and a DOI or an arXiv id has a shape. That is
//! a different question from whether the work is real, and the honest answer
//! to the second needs a catalogue.
pub mod arxiv;
pub mod doi;
pub mod issn;
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
/// What kind of identifier something is.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum Kind {
/// A digital object identifier: `10.1000/182`.
Doi,
/// An international standard serial number, for a journal: `0028-0836`.
Issn,
/// An international standard book number, for a book.
Isbn,
/// An arXiv preprint identifier: `2401.01234` or `math.GT/0309136`.
Arxiv,
}
impl Kind {
/// What it is called.
pub fn name(self) -> &'static str {
match self {
Kind::Doi => "doi",
Kind::Issn => "issn",
Kind::Isbn => "isbn",
Kind::Arxiv => "arxiv",
}
}
}
/// One identifier, as it arrived and as it should be stored.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize, JsonSchema)]
pub struct Identifier {
/// Record version.
#[serde(default = "version")]
pub v: u32,
/// The text it was read from.
pub input: String,
/// Which kind it turned out to be.
pub kind: Kind,
/// The identifier, in the form it should be stored and compared in.
pub id: String,
/// Whether the check digit agrees, where the kind carries one. `None`
/// where the kind has nothing to check against.
pub checksum: Option<bool>,
/// A URL that resolves it, where the kind has a resolver.
#[serde(skip_serializing_if = "Option::is_none")]
pub url: Option<String>,
}
fn version() -> u32 {
corpus_core::V
}
impl corpus_core::Record for Identifier {
fn version(&self) -> u32 {
self.v
}
}
/// Read whatever kind of identifier a string holds.
///
/// Tried in order of how distinctive each shape is: a DOI announces itself
/// with `10.`, an arXiv id with its own prefix or a dotted archive name, and
/// what is left is a run of digits that is either thirteen or ten long (an
/// ISBN) or eight (an ISSN).
pub fn parse(text: &str) -> Option<Identifier> {
parse_as(text, None)
}
/// Read a string as one particular kind, or as whatever it looks like.
pub fn parse_as(text: &str, kind: Option<Kind>) -> Option<Identifier> {
let input = text.trim();
if input.is_empty() {
return None;
}
// Order matters: a DOI announces itself with `10.`, an arXiv id with its
// own prefix, and what is left is a run of digits whose length decides
// between a book and a journal.
const EVERY: [Kind; 4] = [Kind::Doi, Kind::Arxiv, Kind::Isbn, Kind::Issn];
let one = [kind.unwrap_or(Kind::Doi)];
let kinds: &[Kind] = match kind {
Some(_) => &one,
None => &EVERY,
};
for kind in kinds {
let found = match kind {
Kind::Doi => doi::parse(input).map(|id| Identifier {
v: corpus_core::V,
url: Some(format!("https://doi.org/{id}")),
input: input.to_string(),
kind: Kind::Doi,
id,
checksum: None,
}),
Kind::Arxiv => arxiv::parse(input).map(|id| Identifier {
v: corpus_core::V,
url: Some(format!("https://arxiv.org/abs/{id}")),
input: input.to_string(),
kind: Kind::Arxiv,
id,
checksum: None,
}),
Kind::Issn => issn::parse(input).map(|id| Identifier {
v: corpus_core::V,
checksum: Some(issn::valid(&id)),
url: Some(format!("https://portal.issn.org/resource/ISSN/{id}")),
input: input.to_string(),
kind: Kind::Issn,
id,
}),
Kind::Isbn => isbn(input),
};
if let Some(found) = found {
return Some(found);
}
}
None
}
/// An ISBN, using the reader the catalogue tools already share.
fn isbn(input: &str) -> Option<Identifier> {
let digits = bibclean::isbn::digits(input);
if !matches!(digits.len(), 10 | 13) {
return None;
}
// Only where the input is the identifier and nothing else. An ISBN has no
// prefix that announces it, so a sentence holding a thirteen-digit number
// would otherwise read as one.
if !input
.chars()
.all(|c| c.is_ascii_digit() || "- xX".contains(c))
{
return None;
}
Some(Identifier {
v: corpus_core::V,
checksum: Some(bibclean::isbn::valid(&digits)),
url: Some(format!("https://openlibrary.org/isbn/{digits}")),
input: input.to_string(),
kind: Kind::Isbn,
id: digits,
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_doi_is_read_however_it_was_written() {
for text in [
"10.1000/182",
"doi:10.1000/182",
"https://doi.org/10.1000/182",
"http://dx.doi.org/10.1000/182",
" 10.1000/182. ",
] {
let found = parse(text).unwrap_or_else(|| panic!("{text}"));
assert_eq!(found.kind, Kind::Doi, "{text}");
assert_eq!(found.id, "10.1000/182", "{text}");
}
}
#[test]
fn an_issn_carries_a_check_digit() {
let found = parse("0028-0836").expect("an issn");
assert_eq!(found.kind, Kind::Issn);
assert_eq!(found.id, "0028-0836");
assert_eq!(found.checksum, Some(true));
let wrong = parse("0028-0837").expect("an issn");
assert_eq!(wrong.checksum, Some(false), "one digit out is caught");
}
#[test]
fn an_isbn_carries_one_too() {
let found = parse("978-0-306-40615-7").expect("an isbn");
assert_eq!(found.kind, Kind::Isbn);
assert_eq!(found.id, "9780306406157");
assert_eq!(found.checksum, Some(true));
}
#[test]
fn an_arxiv_identifier_is_read_in_both_its_forms() {
assert_eq!(parse("arXiv:2401.01234").expect("an id").id, "2401.01234");
assert_eq!(parse("2401.01234v2").expect("an id").id, "2401.01234v2");
assert_eq!(
parse("math.GT/0309136").expect("an id").id,
"math.GT/0309136"
);
}
#[test]
fn every_kind_resolves_somewhere() {
for text in [
"10.1000/182",
"0028-0836",
"978-0-306-40615-7",
"arXiv:2401.01234",
] {
assert!(parse(text).expect("an id").url.is_some(), "{text}");
}
}
#[test]
fn what_is_not_an_identifier_is_not_one() {
for text in ["", " ", "hello", "10.", "1234", "not/an/id"] {
assert_eq!(parse(text), None, "{text:?}");
}
}
#[test]
fn a_kind_can_be_insisted_on() {
// Eight digits are an ISSN; asked for a DOI, they are nothing.
assert!(parse_as("00280836", Some(Kind::Issn)).is_some());
assert_eq!(parse_as("00280836", Some(Kind::Doi)), None);
}
#[test]
fn a_sentence_that_merely_contains_digits_is_not_an_isbn() {
assert_eq!(parse("published in 9780306406157 by somebody"), None);
}
}
crates/idnorm/src/main.rs
+112
View File
@@ -0,0 +1,112 @@
//! The identifiers a bibliography is held together by, read and checked.
use std::path::PathBuf;
use std::process::ExitCode;
use clap::Parser;
use corpus_core::exit;
use idnorm::Kind;
/// Read identifiers and say what they are, one record per line.
///
/// A DOI arrives bare, as a URL, with a `doi:` prefix, with the full stop the
/// sentence around it left behind, in whatever case the publisher felt like.
/// All of those are one identifier, and a catalogue storing them as they
/// arrived cannot tell.
///
/// Asks nobody whether the thing exists. What it can say is whether the
/// string is well formed — an ISBN and an ISSN carry a check digit — which is
/// a different question, and the honest answer to the other one needs a
/// catalogue.
#[derive(Parser)]
#[command(
version,
about,
after_help = "\
Exit codes: 0 every line was an identifier, 1 at least one was not,
74 the file could not be read.
Example:
idnorm refs.txt | jq -r 'select(.checksum == false) | .input'
idnorm --kind doi refs.txt | jq -r .url"
)]
struct Args {
/// File of identifiers, one per line. Reads stdin when absent.
path: Option<PathBuf>,
/// Read every line as this kind, rather than whatever it looks like.
#[arg(long, value_enum)]
kind: Option<Sort>,
/// Say nothing about the lines that are not identifiers.
#[arg(long)]
quiet: bool,
}
/// The kinds a line can be insisted on being.
#[derive(Clone, Copy, clap::ValueEnum)]
enum Sort {
/// A digital object identifier.
Doi,
/// A journal's serial number.
Issn,
/// A book's number.
Isbn,
/// An arXiv preprint.
Arxiv,
}
impl From<Sort> for Kind {
fn from(sort: Sort) -> Self {
match sort {
Sort::Doi => Kind::Doi,
Sort::Issn => Kind::Issn,
Sort::Isbn => Kind::Isbn,
Sort::Arxiv => Kind::Arxiv,
}
}
}
fn main() -> ExitCode {
let args: Args = match catena_cli::parse() {
Ok(args) => args,
Err(code) => return code,
};
let text = match catena_cli::read_text(args.path.as_deref()) {
Ok(text) => text,
Err((code, why)) => {
eprintln!("idnorm: {why}");
return ExitCode::from(code);
}
};
let kind = args.kind.map(Kind::from);
let mut found = Vec::new();
let mut unread = 0usize;
for line in text.lines() {
if line.trim().is_empty() {
continue;
}
match idnorm::parse_as(line, kind) {
Some(identifier) => found.push(identifier),
None => {
unread += 1;
if !args.quiet {
eprintln!("idnorm: not an identifier: {line}");
}
}
}
}
let wrote = catena_cli::stream("idnorm", &found);
if format!("{wrote:?}") != format!("{:?}", ExitCode::from(exit::OK)) {
return wrote;
}
ExitCode::from(match unread {
0 => exit::OK,
_ => exit::NO,
})
}