"""tpc_biblio 0.1.0 (D170-D173 at Option 1, taken by delegation 2026-09-26;
readings RD-1 to RD-20 of tpm_biblio_resolution 0.1.0): the UWC
bibliography's identifier checks, key form, duplicate key, write check,
canonical serialization and digest, registry URL, registry comparison
with the CHECK and NOTICES lines, the LITERATURE VIEW's selection and the
HOLD record's bibliography list. 0.2.0 (increment 12; tpm_record_functions
0.1.0 section 1, tpm_record_schema 0.3.0 6.9 and 10.4): literature_view's
status is the record's status word W, one of the six (PROVISIONAL,
ESTABLISHED, REFUTED, WITHDRAWN, HELD, NOT GRADED); PROVISIONAL and
ESTABLISHED stand; any other word, STRUCK and VOID among them, raises
"records".
Reference implementation, Python standard library only, on tpc_digest
0.1.0 (D97.1) and tpc_zone 0.4.0 (the text rules of D99.1: LF line
endings, NFC, text safety). DATA for landing round 8; not a declared
TPC-PY version and not TPX code. Every function is a pure function of its
arguments. Every refused argument raises ValueError with a short fixed
message. render_*() give the canonical text that the PHP residence
(tpc_biblio.php) must reproduce byte for byte. Objects are JSON objects
(dict), lists are JSON arrays (list); ints are never bools. Every design
choice the sources leave open is a CONSTANT below and a reading in the
spec note (tpm_biblio_code 0.1.0 section 5).
doi_norm(text) -> dict (RD-5, BR F11, F19)
Strips one wrapper: "doi:" in any ASCII case, or an http or https URL
of doi.org or dx.doi.org (host in any ASCII case), the URL form
percent-decoded once (a "%" not followed by two hex digits, or bytes
that are not UTF-8, refuse). Then the zone text rules (NFC). Refuses
("doi") a non-string, an empty text, white space (class Z, TAB, LF,
CR), a control or format character (class C, F) or a disallowed code
point anywhere; then the shape: "10." + registrant code (digits, dots
between digit groups) + "/" + suffix (one or more graphic code points).
Other directory indicators are refused (a named limit; DOI Handbook
2025 says "usually 10"). A prefix 10.48550 is the arXiv DOI (DataCite's,
BR F19): its suffix must be "arXiv." in any ASCII case followed by an
arXiv id that arxiv_parse accepts; the result is kind ARXIV with that
id. Result {"doi": the DOI as given after the wrapper, "kind": DOI or
ARXIV, "id": the folded DOI (A-Z to a-z, the duplicate key) for DOI,
the arXiv id for ARXIV}. render "doi|<kind>|<doi>|<id>".
doi_shape(text) -> bool (RD-5)
The shape alone, no wrapper: what doi_norm's "doi" passes. A non-string
is false. render "shape|true" or "shape|false".
arxiv_parse(text) -> dict (RD-5, V-8)
Strips "arXiv:" in any ASCII case or an http or https URL prefix
"arxiv.org/abs/" (host in any ASCII case). NEW scheme YYMM.NNNN with
YYMM in 0704..1412 or YYMM.NNNNN with YYMM >= 1501, MM 01-12; OLD
scheme archive[.SC]/YYMMNNN, archive [a-z][a-z-]{1,15}, SC two capital
letters (a named limit: a lower-case subject class is refused), YY 91-99
or 00-07 and MM 01-12 with YYMM at most 0703; an optional vN, N a
positive integer without a leading zero. Refusal "arxiv". Result {"id":
the id with its version as given, "version": N or 0, "key": the id
without version, "scheme": NEW or OLD}. render "arxiv|<id>|<version>|
<key>|<scheme>".
isbn_check(text) -> dict (RD-5, V-13)
ASCII hyphens and spaces removed; ten characters (nine digits and a
digit or X in either case; weights 10..1, the sum a multiple of 11) or
thirteen digits (prefix 978 or 979; weights 1 and 3 alternating, the
sum a multiple of 10). An ISBN-10 is converted: 978, the nine digits,
the check recomputed. Refusal "isbn". Result {"isbn13", "given": "10"
or "13"}. render "isbn|<isbn13>|<given>".
issn_check(text) -> str (RD-5, V-14)
NNNN-NNNC, the hyphen required, C a digit or X in either case; weights
8..2 over the seven digits, the sum mod 11, no remainder 0 else 11 minus
the remainder, 10 as X. Refusal "issn". Returns the ISSN with X in upper
case. render "issn|<issn>".
pmid_check(text) -> str; pmcid_check(text) -> str (RD-5)
PMID: decimal digits, no leading zero. PMCID: "PMC" in any ASCII case
and decimal digits, returned with "PMC" upper. Refusals "pmid", "pmcid".
render "pmid|<v>", "pmcid|<v>".
biblio_key_form(text, kind) -> str (RD-9, V-2)
kind "title" or "name"; refusal "key" for another kind or a non-string.
KF0 title only: after leading white space, one marker of the closed set
RETRACTED, RETRACTED ARTICLE, WITHDRAWN, WITHDRAWN ARTICLE (any ASCII
case) followed by ":" is removed once. Other markers: a named limit.
KF1 every markup run "<" + letter or "/" ... ">" is removed.
KF2 & < > " ' and numeric &#N; / &#xH; (a scalar
value, not a surrogate, at most 10FFFF) are decoded; other named
entities stay (a named limit).
KF3 NFD; then every code point of class M (Mn), P (P*), S (S*), F (Cf)
and C (Cc, TAB LF CR excepted) is removed; TAB, LF, CR and every
class Z (Zs Zl Zp) code point become a space.
KF4 A-Z fold to a-z; a letter that does not decompose to ASCII keeps its
case (a named limit; U+00D8 stays).
KF5 NFC; runs of spaces collapse to one; leading and trailing removed.
Classes come from the table embedded in both residences, generated by
gen_biblio_unicode_table.py from Python's unicodedata at Unicode
15.1.0 and checked against its SHA-256 on first use: M Mn, P Pc Pd Ps
Pe Pi Pf Po, S Sm Sc Sk So, F Cf, C Cc, Z Zs Zl Zp; any other code
point has no class. render "key|<text>".
biblio_dupe_key(identifiers) -> dict (RD-20)
identifiers: an object with keys among DOI, ARXIV, ISBN, ISSN, PMID,
PMCID, each a string its check accepts (DOI must be kind DOI; refusal
"identifier:<KIND>"; another key or a non-object: "identifiers"). The
folded DOI; else the arXiv key (no version); else the ISBN-13; else
kind NONE with key "". render "dupe|<kind>|<key>".
biblio_check(entry, existing_keys, caps) -> dict (D171 (d); RD-3, RD-4, RD-5, RD-20)
entry: {"item": the CSL-JSON item, "identifiers": as above,
"supersedes": a BIB ID or null}; existing_keys: a list of [kind, key,
bib_id] over the UWC's landed entries; caps: {"entry_max" bytes,
"title", "text", "name", "url" code points, "names" per role}, positive
integers (the tuning option BIBLIO ENTRY MAX and RD-4's caps, supplied
by TPX, never fixed here). Checks in order; the first failure raises:
BC1 "subset:<field>": a key outside the 40-property subset CSL_KEYS; a
name with a key outside NAME_KEYS; a date with a key outside
DATE_KEYS.
BC2 "type:<field>": type not one of the 45 CSL types; a string field
that is not a non-empty string; a name role that is not a list of
objects each with non-empty string parts and a family or a literal;
a date that is not {date-parts: one or two lists of one to three
integers (year -9999..9999, month 1..12, day 1..31), circa?: bool}.
BC3 "text:<field>": a string with a disallowed code point (tpc_zone's
text safety, D99.1: category C except TAB and LF, U+2028, U+2029).
BC4 "required:<field>": type, title, issued (with a year), and author
or editor with at least one name ("required:author").
BC5 "cap:<field>": title over caps.title; URL over caps.url; any other
string field over caps.text; a name part over caps.name or a role
with more than caps.names names; lengths in code points after NFC.
BC6 "identifiers", "identifier:<KIND>": as biblio_dupe_key.
BC7 "mismatch:<KIND>": the item's DOI, ISBN, ISSN, PMID or PMCID, when
present, equals its IDENTIFIERS entry after normalization (DOI
folded; ISBN-10 converted); an item DOI of prefix 10.48550 requires
IDENTIFIERS.ARXIV with the same key and no IDENTIFIERS.DOI.
BC8 "duplicate:<bib_id>": the duplicate key is in existing_keys and
supersedes does not name that bib_id.
BC9 "size": the item's canonical JSON text (biblio_canon's) exceeds
caps.entry_max bytes of UTF-8.
"entry", "existing", "caps": a malformed argument. Result {"status":
"OK", "identifiers": the normalized identifiers (DOI as given, NFC;
ARXIV with its version; ISBN as 13; ISSN with X upper; PMID; PMCID
upper), "dupe": biblio_dupe_key's result}. render "check|OK|<dupe
kind>|<dupe key>" then one "id|<KIND>|<value>" per identifier in the
fixed order DOI, ARXIV, ISBN, ISSN, PMID, PMCID. A preprint and its
published version carry different keys and are two entries.
biblio_canon(bib_id, proposed_by, landed_by, landed, supersedes,
identifiers, item) -> dict (RD-11; D97.1, D99.1)
bib_id, proposed_by, landed_by, landed: non-empty texts passing text
safety ("bib_id", "proposed_by", "landed_by", "landed"); supersedes:
null or such a text ("supersedes"); identifiers as biblio_dupe_key;
item: BC1-BC3 (an item "id", when present, must equal bib_id: "id").
text = the item as canonical JSON: keys in CSL_KEYS order, "id" set to
bib_id, names with NAME_KEYS order, dates with date-parts then circa;
strings NFC with LF line endings (tpc_zone _text), escaped only for
'"' and '\\' and for a control character as \\u00XX lower-case hex; no
white space outside strings; integers in decimal; booleans true and
false. digest = digest_fields([TPC-BIB, 1, bib_id, proposed_by,
landed_by, landed, supersedes or "", DOI, doi or "", ARXIV, arxiv or "",
ISBN, ..., ISSN, ..., PMID, ..., PMCID, ..., text]) with the stored
identifier forms. The status lines CHECK, NOTICES and SUPERSEDED BY are
outside the digest. render "canon|<digest>" then the text.
biblio_preimage(...) -> str: the netstring the digest covers, as text.
render "preimage|<text>".
registry_url(declaration, identifier) -> str (RD-14, RD-15; D172 (a))
declaration: {"kind": DOI-RA, DOI-CROSSREF, DOI-DATACITE, ARXIV, PMID or
PMCID, "template": a URL, "name"?: text} ("declaration" otherwise).
The template is https with a host name and exactly one slot "{ID}" in
the path or the query, nothing else in braces, only RFC 3986
characters ("template"). The identifier is checked by its kind: a DOI
kind takes doi_norm of kind DOI (an arXiv DOI is never looked up as a
DOI), ARXIV takes arxiv_parse (the id with its version, RD-7), PMID and
PMCID their checks ("identifier"). The slot is filled with the checked
identifier percent-encoded: every UTF-8 byte as %XX upper-case except
A-Z a-z 0-9 "-" "." "_" "~" "/" ":" "(" ")" (V-1, V-2 observed; RFC 3986
per V-15), one rule for path and query slots. Because "/" and "." stay
raw, an identifier whose encoding leaves a "." or ".." segment in the
URL's path is refused ("identifier"): an HTTP client would remove the
segment (RFC 3986 5.2.4) and fetch another record; TPX lands it LOOKUP
FAILED and the falcon settles it (a named limit, taken by delegation).
render "url|<url>".
biblio_compare(entry, records) -> dict (D172 (b); RD-6, RD-7, RD-8, RD-10)
entry: {"item", "identifiers"} passing BC1-BC4, BC6 (the write check's
caps, duplicate and size are not re-run). records: one REGISTRY RECORD
per identifier TPX looked up, the parse of a registry payload being
TPX's (RD-8). Shapes:
DOI {"kind": "DOI", "identifier", "ra": {"status": FOUND | NOT
FOUND | FAILED, "ra": the agency's name or null}, "registry":
crossref | datacite | generic | null, "status": FOUND | NOT
FOUND | FAILED | null, "fields", "notices"}: ra Crossref takes
registry crossref, ra DataCite datacite, any other agency null
(FOUND UNCOMPARED) or generic when a REGISTRY DECLARATION names
its endpoint (RD-6).
ARXIV {"kind", "identifier", "registry": "arxiv", "status", "fields",
"notices"}; PMID and PMCID likewise with registry "pubmed" (a
PMCID through the ID converter, RD-6).
fields, when status is FOUND, else null: {"year": int | null,
"title": str | null, "subtitle": str | null, "family": str | null,
"name_form": FAMILY | GIVEN FAMILY | FAMILY INITIALS | null, "doi":
str | null, "issn": [str], "isbn": [str], "arxiv_doi": str | null,
"arxiv_id": str | null}; notices: a list, empty unless FOUND, of
{"type": text, "identifier": a token or null, "date": YYYY-MM-DD or
null, "source": text}. A record's identifier must equal the entry's
of that kind after normalization; one record per kind; a kind the
entry lacks is refused ("records").
CP1 PRIMARY: the first present of DOI, ARXIV, PMID, PMCID.
CP2 Per identifier: DOI: ra NOT FOUND -> NOT FOUND; ra FAILED -> LOOKUP
FAILED; ra FOUND with registry null -> FOUND UNCOMPARED; else the
registry's status: FOUND -> compared; NOT FOUND or FAILED -> LOOKUP
FAILED, never NOT FOUND (RD-6). ARXIV, PMID, PMCID: NOT FOUND,
FAILED -> LOOKUP FAILED, FOUND -> compared. An identifier a declared
registry takes with no record -> LOOKUP FAILED (the cap or the time
bound landed it so, D172 (g), RD-13). ISBN, ISSN: never looked up;
CONFIRMED when the primary's FOUND record lists it, FIELDS DIFFER
when that record lists others and not it, NO LOOKUP when no list
(WATCH: one word makes the second FOUND UNCOMPARED). No primary:
every identifier NO LOOKUP.
CP3 The comparison, every compared field equal for CONFIRMED, else
FIELDS DIFFER naming the fields (RD-8). Under the key form: title =
the record's title, or title + " " + subtitle where a subtitle
exists, against the entry's title, both as kind "title"; family =
the entry's first author's family (a literal name serving), or the
first editor's when there is no author, against the record's under
its name_form: FAMILY equal; GIVEN FAMILY the keyed record name ends
with the keyed family at a token boundary; FAMILY INITIALS begins
with it at a token boundary. year = the entry's issued year. doi:
folded, where both carry one. A DataCite record's arxiv_id against
the entry's ARXIV key where both carry one. A record whose year,
title or family is null differs on that field (fail closed).
An ARXIV record beside a primary DOI: arxiv_doi (folded) and family;
without arxiv_doi, CONFIRMED when title and family both equal, else
FOUND UNCOMPARED. A PubMed record beside a primary: year, title,
family (FAMILY INITIALS), doi where both carry one.
CP4 CHECK: the first present of NOT FOUND, FIELDS DIFFER, LOOKUP FAILED,
FOUND UNCOMPARED, NO LOOKUP, CONFIRMED.
CP5 NOTICES: every FOUND record's notices in results order, the type
under the key form (kind "name"), the identifier, date and source as
given (RD-10). A notice moves no result.
Result {"check", "results": [[kind, identifier, result, [fields]], ...]
in the order DOI, ARXIV, PMID, PMCID, ISBN, ISSN over the entry's
identifiers, "notices": [{type, identifier, date, source}, ...]}.
render "compare|<check>", one "<kind>|<identifier>|<result>|<fields
comma-joined>" per result, one "notice|<source>|<identifier or ->|<date
or ->|<type>" per notice, then "line|" + render_check_line and "line|"
+ render_notices_line ("line|-" when there is no notice).
render_check_line(d, date) -> str; render_notices_line(d, date) -> str
The status lines TPX writes into the entry's text (D172 (e)), date a
calendar date YYYY-MM-DD ("date"). CHECK: "CHECK <date> <value>" then
" ; <KIND> <identifier> <result>" per result. NOTICES: "NOTICES <date>"
then " ; <source> <identifier or -> <date or -> <type>" per notice; ""
when there is none (no line is written). Identifiers hold no white
space, so " ; " separates clauses unambiguously.
arxiv_withdrawn(comment) -> bool (RD-10, V-9)
The whole token "withdrawn" in any ASCII case in an arXiv comment (a
token: not adjoined by an ASCII letter or digit); null is false; another
non-string raises "comment". render "withdrawn|true" or "|false".
literature_view(records, entries, findings) -> dict (D172 (f); RD-18)
records: [{"id", "status", "literature": [bib ids]}]; entries:
[{"bib_id", "check": a CHECK value, "notices": count, "primary": bool}];
findings: [{"bib_id", "result": SOURCE AGREES | SOURCE DIFFERS | NOT
FOUND, "agreed": bool}] (FK-20's). status: one of STATUS_WORDS (else
"records"); standing: PROVISIONAL or ESTABLISHED. Each standing record whose LITERATURE names an entry with a
NOTICES line (count > 0), CHECK NOT FOUND or an agreed FK-20 NOT FOUND,
listed by id in code point order with its flagged entries by bib_id and
reason (NOTICES, NOT FOUND, FK-20 NOT FOUND, in that order). for_hdecl:
entries with a primary identifier, CHECK NOT FOUND and an agreed FK-20
NOT FOUND. A cited bib_id with no entry raises "literature" (TPX refuses
it at landing); malformed lists raise "records", "entries", "findings".
render "view|<n>", "record|<id>|<bib>:<reason>|..." per record,
"hdecl|<bib ids comma-joined>".
biblio_hold_list(entries, account, blocks, checks) -> list (D171 (j))
entries: [{"bib_id", "proposed_by", "landed_by", "block_id"}]; checks:
{bib_id: the CHECK value after the re-run lookup}. The entries the
account proposed or landed in a held block whose re-run CHECK is not
CONFIRMED, as [[bib_id, check], ...] sorted by bib_id; a selected entry
without a check raises "checks". render "holdlist|<n>" then
"<bib_id>|<check>" per entry.
"""
import bisect
import hashlib
import re
import unicodedata
import tpc_digest as G
import tpc_zone as Z
VERSION = "0.2.0" # increment 12 (S95: one MINOR bump over the declared 0.1.0): literature_view's closed status set
# ----------------------------------------------------------------- CONSTANTS
# Every design choice the sources leave open, in one place (spec note 5).
BIB_TAG = "TPC-BIB"
BIB_FORM = "1"
# RD-3: the CSL data schema 1.0 subset, in serialization order (40 keys).
CSL_KEYS = ("type", "id", "title", "title-short", "author", "editor",
"translator", "container-author", "collection-editor", "issued",
"accessed", "original-date", "container-title",
"container-title-short", "collection-title", "collection-number",
"publisher", "publisher-place", "volume", "issue", "page",
"number", "number-of-pages", "edition", "genre", "medium",
"event-title", "event-place", "chapter-number", "version",
"language", "DOI", "ISBN", "ISSN", "PMID", "PMCID", "URL",
"original-title", "archive", "archive_location")
NAME_ROLES = ("author", "editor", "translator", "container-author",
"collection-editor")
DATE_FIELDS = ("issued", "accessed", "original-date")
NAME_KEYS = ("family", "given", "dropping-particle", "non-dropping-particle",
"suffix", "literal")
DATE_KEYS = ("date-parts", "circa")
CSL_TYPES = ("article", "article-journal", "article-magazine",
"article-newspaper", "bill", "book", "broadcast", "chapter",
"classic", "collection", "dataset", "document", "entry",
"entry-dictionary", "entry-encyclopedia", "event", "figure",
"graphic", "hearing", "interview", "legal_case", "legislation",
"manuscript", "map", "motion_picture", "musical_score",
"pamphlet", "paper-conference", "patent", "performance",
"periodical", "personal_communication", "post", "post-weblog",
"regulation", "report", "review", "review-book", "software",
"song", "speech", "standard", "thesis", "treaty", "webpage")
# RD-4: the cap each string field falls under (caps are arguments).
CAP_TITLE = ("title",)
CAP_URL = ("URL",)
YEAR_MIN, YEAR_MAX = -9999, 9999
# RD-5: the identifier kinds in the digest's and the render's fixed order.
IDENT_KINDS = ("DOI", "ARXIV", "ISBN", "ISSN", "PMID", "PMCID")
ARXIV_DOI_PREFIX = "10.48550"
ARXIV_OLD_YY = ("91", "92", "93", "94", "95", "96", "97", "98", "99", "00",
"01", "02", "03", "04", "05", "06", "07")
# RD-9: the title markers, the decoded entities.
MARKERS = ("RETRACTED ARTICLE", "RETRACTED", "WITHDRAWN ARTICLE", "WITHDRAWN")
ENTITIES = {"amp": "&", "lt": "<", "gt": ">", "quot": '"', "apos": "'"}
KEY_REMOVE = "MPSFC" # table classes removed by the key form
KEY_SPACE = "Z" # table class read as a space
# RD-14, RD-15: the declaration's identifier kinds; the characters kept raw.
DECL_KINDS = ("DOI-RA", "DOI-CROSSREF", "DOI-DATACITE", "ARXIV", "PMID",
"PMCID")
URL_SLOT = "{ID}"
URL_SAFE = ("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
"-._~/:()")
# D172 (b): the results and the CHECK value's order; the registries.
RESULTS = ("NOT FOUND", "FIELDS DIFFER", "LOOKUP FAILED", "FOUND UNCOMPARED",
"NO LOOKUP", "CONFIRMED")
PRIMARY_ORDER = ("DOI", "ARXIV", "PMID", "PMCID")
RESULT_ORDER = ("DOI", "ARXIV", "PMID", "PMCID", "ISBN", "ISSN")
LOOKUP_STATUS = ("FOUND", "NOT FOUND", "FAILED")
REGISTRIES = ("crossref", "datacite", "generic", "arxiv", "pubmed")
RA_REGISTRY = {"Crossref": "crossref", "DataCite": "datacite"}
NAME_FORMS = ("FAMILY", "GIVEN FAMILY", "FAMILY INITIALS")
FIELD_KEYS = ("year", "title", "subtitle", "family", "name_form", "doi",
"issn", "isbn", "arxiv_doi", "arxiv_id")
NOTICE_KEYS = ("type", "identifier", "date", "source")
# RD-18: the LITERATURE VIEW's reasons and the standing test.
REASONS = ("NOTICES", "NOT FOUND", "FK-20 NOT FOUND")
STATUS_WORDS = ("PROVISIONAL", "ESTABLISHED", "REFUTED", "WITHDRAWN", "HELD", "NOT GRADED")
NOT_STANDING = ("REFUTED", "WITHDRAWN", "HELD", "NOT GRADED")
FK20_RESULTS = ("SOURCE AGREES", "SOURCE DIFFERS", "NOT FOUND")
WITHDRAWN_TOKEN = "withdrawn"
# ------------------------------------------------------------------ common
def _esc(s):
"""Render escape: "|" and "\\" backslashed; C0 controls and DEL as
\\xHH; every other character as it is."""
out = []
for ch in s:
o = ord(ch)
if ch == "|" or ch == "\\":
out.append("\\" + ch)
elif o < 0x20 or o == 0x7F:
out.append("\\x%02x" % o)
else:
out.append(ch)
return "".join(out)
def _is_int(x, lo, hi):
return type(x) is int and lo <= x <= hi
def _is_text(x):
"""A non-empty string with no disallowed code point (tpc_zone)."""
return type(x) is str and x != "" and Z._first_bad(x) is None
def _keys_within(o, allowed):
if type(o) is not dict:
return False
for k in o:
if k not in allowed:
return False
return True
def _fold(s):
"""A-Z to a-z; every other code point as it is (DOI equivalence and
the key form's fold, RD-5, RD-9)."""
return "".join(chr(ord(c) + 32) if "A" <= c <= "Z" else c for c in s)
def _lower_ascii(s):
return _fold(s)
# ----------------------------------------------------------- Unicode table
# The key form's character classes, generated at the pinned Unicode
# version by gen_biblio_unicode_table.py and embedded in both residences:
# M Mn, P P*, S S*, F Cf, C Cc, Z Zs Zl Zp; any other code point has none.
# BEGIN UNICODE TABLE (gen_biblio_unicode_table.py; do not edit)
UNICODE_VERSION = "15.1.0"
UNICODE_TABLE_SHA256 = (
"fda8fec9546959021219a4e401ca399ddd026bc1f1b66cbdf1fcb8e901e7f378")
_UTABLE = (
"0-1FC;20Z;21-23P;24S;25-2AP;2BS;2C-2FP;3A-3BP;3C-3ES;3F-40P;5B-5DP;5"
"ES;5FP;60S;7BP;7CS;7DP;7ES;7F-9FC;A0Z;A1P;A2-A6S;A7P;A8-A9S;ABP;ACS;"
"ADF;AE-B1S;B4S;B6-B7P;B8S;BBP;BFP;D7S;F7S;2C2-2C5S;2D2-2DFS;2E5-2EBS"
";2EDS;2EF-2FFS;300-36FM;375S;37EP;384-385S;387P;3F6S;482S;483-487M;5"
"5A-55FP;589-58AP;58D-58FS;591-5BDM;5BEP;5BFM;5C0P;5C1-5C2M;5C3P;5C4-"
"5C5M;5C6P;5C7M;5F3-5F4P;600-605F;606-608S;609-60AP;60BS;60C-60DP;60E"
"-60FS;610-61AM;61BP;61CF;61D-61FP;64B-65FM;66A-66DP;670M;6D4P;6D6-6D"
"CM;6DDF;6DES;6DF-6E4M;6E7-6E8M;6E9S;6EA-6EDM;6FD-6FES;700-70DP;70FF;"
"711M;730-74AM;7A6-7B0M;7EB-7F3M;7F6S;7F7-7F9P;7FDM;7FE-7FFS;816-819M"
";81B-823M;825-827M;829-82DM;830-83EP;859-85BM;85EP;888S;890-891F;898"
"-89FM;8CA-8E1M;8E2F;8E3-902M;93AM;93CM;941-948M;94DM;951-957M;962-96"
"3M;964-965P;970P;981M;9BCM;9C1-9C4M;9CDM;9E2-9E3M;9F2-9F3S;9FA-9FBS;"
"9FDP;9FEM;A01-A02M;A3CM;A41-A42M;A47-A48M;A4B-A4DM;A51M;A70-A71M;A75"
"M;A76P;A81-A82M;ABCM;AC1-AC5M;AC7-AC8M;ACDM;AE2-AE3M;AF0P;AF1S;AFA-A"
"FFM;B01M;B3CM;B3FM;B41-B44M;B4DM;B55-B56M;B62-B63M;B70S;B82M;BC0M;BC"
"DM;BF3-BFAS;C00M;C04M;C3CM;C3E-C40M;C46-C48M;C4A-C4DM;C55-C56M;C62-C"
"63M;C77P;C7FS;C81M;C84P;CBCM;CBFM;CC6M;CCC-CCDM;CE2-CE3M;D00-D01M;D3"
"B-D3CM;D41-D44M;D4DM;D4FS;D62-D63M;D79S;D81M;DCAM;DD2-DD4M;DD6M;DF4P"
";E31M;E34-E3AM;E3FS;E47-E4EM;E4FP;E5A-E5BP;EB1M;EB4-EBCM;EC8-ECEM;F0"
"1-F03S;F04-F12P;F13S;F14P;F15-F17S;F18-F19M;F1A-F1FS;F34S;F35M;F36S;"
"F37M;F38S;F39M;F3A-F3DP;F71-F7EM;F80-F84M;F85P;F86-F87M;F8D-F97M;F99"
"-FBCM;FBE-FC5S;FC6M;FC7-FCCS;FCE-FCFS;FD0-FD4P;FD5-FD8S;FD9-FDAP;102"
"D-1030M;1032-1037M;1039-103AM;103D-103EM;104A-104FP;1058-1059M;105E-"
"1060M;1071-1074M;1082M;1085-1086M;108DM;109DM;109E-109FS;10FBP;135D-"
"135FM;1360-1368P;1390-1399S;1400P;166DS;166EP;1680Z;169B-169CP;16EB-"
"16EDP;1712-1714M;1732-1733M;1735-1736P;1752-1753M;1772-1773M;17B4-17"
"B5M;17B7-17BDM;17C6M;17C9-17D3M;17D4-17D6P;17D8-17DAP;17DBS;17DDM;18"
"00-180AP;180B-180DM;180EF;180FM;1885-1886M;18A9M;1920-1922M;1927-192"
"8M;1932M;1939-193BM;1940S;1944-1945P;19DE-19FFS;1A17-1A18M;1A1BM;1A1"
"E-1A1FP;1A56M;1A58-1A5EM;1A60M;1A62M;1A65-1A6CM;1A73-1A7CM;1A7FM;1AA"
"0-1AA6P;1AA8-1AADP;1AB0-1ABDM;1ABF-1ACEM;1B00-1B03M;1B34M;1B36-1B3AM"
";1B3CM;1B42M;1B5A-1B60P;1B61-1B6AS;1B6B-1B73M;1B74-1B7CS;1B7D-1B7EP;"
"1B80-1B81M;1BA2-1BA5M;1BA8-1BA9M;1BAB-1BADM;1BE6M;1BE8-1BE9M;1BEDM;1"
"BEF-1BF1M;1BFC-1BFFP;1C2C-1C33M;1C36-1C37M;1C3B-1C3FP;1C7E-1C7FP;1CC"
"0-1CC7P;1CD0-1CD2M;1CD3P;1CD4-1CE0M;1CE2-1CE8M;1CEDM;1CF4M;1CF8-1CF9"
"M;1DC0-1DFFM;1FBDS;1FBF-1FC1S;1FCD-1FCFS;1FDD-1FDFS;1FED-1FEFS;1FFD-"
"1FFES;2000-200AZ;200B-200FF;2010-2027P;2028-2029Z;202A-202EF;202FZ;2"
"030-2043P;2044S;2045-2051P;2052S;2053-205EP;205FZ;2060-2064F;2066-20"
"6FF;207A-207CS;207D-207EP;208A-208CS;208D-208EP;20A0-20C0S;20D0-20DC"
"M;20E1M;20E5-20F0M;2100-2101S;2103-2106S;2108-2109S;2114S;2116-2118S"
";211E-2123S;2125S;2127S;2129S;212ES;213A-213BS;2140-2144S;214A-214DS"
";214FS;218A-218BS;2190-2307S;2308-230BP;230C-2328S;2329-232AP;232B-2"
"426S;2440-244AS;249C-24E9S;2500-2767S;2768-2775P;2794-27C4S;27C5-27C"
"6P;27C7-27E5S;27E6-27EFP;27F0-2982S;2983-2998P;2999-29D7S;29D8-29DBP"
";29DC-29FBS;29FC-29FDP;29FE-2B73S;2B76-2B95S;2B97-2BFFS;2CE5-2CEAS;2"
"CEF-2CF1M;2CF9-2CFCP;2CFE-2CFFP;2D70P;2D7FM;2DE0-2DFFM;2E00-2E2EP;2E"
"30-2E4FP;2E50-2E51S;2E52-2E5DP;2E80-2E99S;2E9B-2EF3S;2F00-2FD5S;2FF0"
"-2FFFS;3000Z;3001-3003P;3004S;3008-3011P;3012-3013S;3014-301FP;3020S"
";302A-302DM;3030P;3036-3037S;303DP;303E-303FS;3099-309AM;309B-309CS;"
"30A0P;30FBP;3190-3191S;3196-319FS;31C0-31E3S;31EFS;3200-321ES;322A-3"
"247S;3250S;3260-327FS;328A-32B0S;32C0-33FFS;4DC0-4DFFS;A490-A4C6S;A4"
"FE-A4FFP;A60D-A60FP;A66FM;A673P;A674-A67DM;A67EP;A69E-A69FM;A6F0-A6F"
"1M;A6F2-A6F7P;A700-A716S;A720-A721S;A789-A78AS;A802M;A806M;A80BM;A82"
"5-A826M;A828-A82BS;A82CM;A836-A839S;A874-A877P;A8C4-A8C5M;A8CE-A8CFP"
";A8E0-A8F1M;A8F8-A8FAP;A8FCP;A8FFM;A926-A92DM;A92E-A92FP;A947-A951M;"
"A95FP;A980-A982M;A9B3M;A9B6-A9B9M;A9BC-A9BDM;A9C1-A9CDP;A9DE-A9DFP;A"
"9E5M;AA29-AA2EM;AA31-AA32M;AA35-AA36M;AA43M;AA4CM;AA5C-AA5FP;AA77-AA"
"79S;AA7CM;AAB0M;AAB2-AAB4M;AAB7-AAB8M;AABE-AABFM;AAC1M;AADE-AADFP;AA"
"EC-AAEDM;AAF0-AAF1P;AAF6M;AB5BS;AB6A-AB6BS;ABE5M;ABE8M;ABEBP;ABEDM;F"
"B1EM;FB29S;FBB2-FBC2S;FD3E-FD3FP;FD40-FD4FS;FDCFS;FDFC-FDFFS;FE00-FE"
"0FM;FE10-FE19P;FE20-FE2FM;FE30-FE52P;FE54-FE61P;FE62S;FE63P;FE64-FE6"
"6S;FE68P;FE69S;FE6A-FE6BP;FEFFF;FF01-FF03P;FF04S;FF05-FF0AP;FF0BS;FF"
"0C-FF0FP;FF1A-FF1BP;FF1C-FF1ES;FF1F-FF20P;FF3B-FF3DP;FF3ES;FF3FP;FF4"
"0S;FF5BP;FF5CS;FF5DP;FF5ES;FF5F-FF65P;FFE0-FFE6S;FFE8-FFEES;FFF9-FFF"
"BF;FFFC-FFFDS;10100-10102P;10137-1013FS;10179-10189S;1018C-1018ES;10"
"190-1019CS;101A0S;101D0-101FCS;101FDM;102E0M;10376-1037AM;1039FP;103"
"D0P;1056FP;10857P;10877-10878S;1091FP;1093FP;10A01-10A03M;10A05-10A0"
"6M;10A0C-10A0FM;10A38-10A3AM;10A3FM;10A50-10A58P;10A7FP;10AC8S;10AE5"
"-10AE6M;10AF0-10AF6P;10B39-10B3FP;10B99-10B9CP;10D24-10D27M;10EAB-10"
"EACM;10EADP;10EFD-10EFFM;10F46-10F50M;10F55-10F59P;10F82-10F85M;10F8"
"6-10F89P;11001M;11038-11046M;11047-1104DP;11070M;11073-11074M;1107F-"
"11081M;110B3-110B6M;110B9-110BAM;110BB-110BCP;110BDF;110BE-110C1P;11"
"0C2M;110CDF;11100-11102M;11127-1112BM;1112D-11134M;11140-11143P;1117"
"3M;11174-11175P;11180-11181M;111B6-111BEM;111C5-111C8P;111C9-111CCM;"
"111CDP;111CFM;111DBP;111DD-111DFP;1122F-11231M;11234M;11236-11237M;1"
"1238-1123DP;1123EM;11241M;112A9P;112DFM;112E3-112EAM;11300-11301M;11"
"33B-1133CM;11340M;11366-1136CM;11370-11374M;11438-1143FM;11442-11444"
"M;11446M;1144B-1144FP;1145A-1145BP;1145DP;1145EM;114B3-114B8M;114BAM"
";114BF-114C0M;114C2-114C3M;114C6P;115B2-115B5M;115BC-115BDM;115BF-11"
"5C0M;115C1-115D7P;115DC-115DDM;11633-1163AM;1163DM;1163F-11640M;1164"
"1-11643P;11660-1166CP;116ABM;116ADM;116B0-116B5M;116B7M;116B9P;1171D"
"-1171FM;11722-11725M;11727-1172BM;1173C-1173EP;1173FS;1182F-11837M;1"
"1839-1183AM;1183BP;1193B-1193CM;1193EM;11943M;11944-11946P;119D4-119"
"D7M;119DA-119DBM;119E0M;119E2P;11A01-11A0AM;11A33-11A38M;11A3B-11A3E"
"M;11A3F-11A46P;11A47M;11A51-11A56M;11A59-11A5BM;11A8A-11A96M;11A98-1"
"1A99M;11A9A-11A9CP;11A9E-11AA2P;11B00-11B09P;11C30-11C36M;11C38-11C3"
"DM;11C3FM;11C41-11C45P;11C70-11C71P;11C92-11CA7M;11CAA-11CB0M;11CB2-"
"11CB3M;11CB5-11CB6M;11D31-11D36M;11D3AM;11D3C-11D3DM;11D3F-11D45M;11"
"D47M;11D90-11D91M;11D95M;11D97M;11EF3-11EF4M;11EF7-11EF8P;11F00-11F0"
"1M;11F36-11F3AM;11F40M;11F42M;11F43-11F4FP;11FD5-11FF1S;11FFFP;12470"
"-12474P;12FF1-12FF2P;13430-1343FF;13440M;13447-13455M;16A6E-16A6FP;1"
"6AF0-16AF4M;16AF5P;16B30-16B36M;16B37-16B3BP;16B3C-16B3FS;16B44P;16B"
"45S;16E97-16E9AP;16F4FM;16F8F-16F92M;16FE2P;16FE4M;1BC9CS;1BC9D-1BC9"
"EM;1BC9FP;1BCA0-1BCA3F;1CF00-1CF2DM;1CF30-1CF46M;1CF50-1CFC3S;1D000-"
"1D0F5S;1D100-1D126S;1D129-1D164S;1D167-1D169M;1D16A-1D16CS;1D173-1D1"
"7AF;1D17B-1D182M;1D183-1D184S;1D185-1D18BM;1D18C-1D1A9S;1D1AA-1D1ADM"
";1D1AE-1D1EAS;1D200-1D241S;1D242-1D244M;1D245S;1D300-1D356S;1D6C1S;1"
"D6DBS;1D6FBS;1D715S;1D735S;1D74FS;1D76FS;1D789S;1D7A9S;1D7C3S;1D800-"
"1D9FFS;1DA00-1DA36M;1DA37-1DA3AS;1DA3B-1DA6CM;1DA6D-1DA74S;1DA75M;1D"
"A76-1DA83S;1DA84M;1DA85-1DA86S;1DA87-1DA8BP;1DA9B-1DA9FM;1DAA1-1DAAF"
"M;1E000-1E006M;1E008-1E018M;1E01B-1E021M;1E023-1E024M;1E026-1E02AM;1"
"E08FM;1E130-1E136M;1E14FS;1E2AEM;1E2EC-1E2EFM;1E2FFS;1E4EC-1E4EFM;1E"
"8D0-1E8D6M;1E944-1E94AM;1E95E-1E95FP;1ECACS;1ECB0S;1ED2ES;1EEF0-1EEF"
"1S;1F000-1F02BS;1F030-1F093S;1F0A0-1F0AES;1F0B1-1F0BFS;1F0C1-1F0CFS;"
"1F0D1-1F0F5S;1F10D-1F1ADS;1F1E6-1F202S;1F210-1F23BS;1F240-1F248S;1F2"
"50-1F251S;1F260-1F265S;1F300-1F6D7S;1F6DC-1F6ECS;1F6F0-1F6FCS;1F700-"
"1F776S;1F77B-1F7D9S;1F7E0-1F7EBS;1F7F0S;1F800-1F80BS;1F810-1F847S;1F"
"850-1F859S;1F860-1F887S;1F890-1F8ADS;1F8B0-1F8B1S;1F900-1FA53S;1FA60"
"-1FA6DS;1FA70-1FA7CS;1FA80-1FA88S;1FA90-1FABDS;1FABF-1FAC5S;1FACE-1F"
"ADBS;1FAE0-1FAE8S;1FAF0-1FAF8S;1FB00-1FB92S;1FB94-1FBCAS;E0001F;E002"
"0-E007FF;E0100-E01EFM"
)
# END UNICODE TABLE
_UT = []
def _utable():
if not _UT:
if hashlib.sha256(_UTABLE.encode("ascii")).hexdigest() != \
UNICODE_TABLE_SHA256:
raise RuntimeError("unicode table")
starts, ends, cls = [], [], []
for item in _UTABLE.split(";"):
rng, c = item[:-1], item[-1]
a, _, b = rng.partition("-")
starts.append(int(a, 16))
ends.append(int(b or a, 16))
cls.append(c)
_UT.extend([starts, ends, cls])
return _UT
def ucls(ch):
"""The table class of one character, or "" for none."""
starts, ends, cls = _utable()
cp = ord(ch)
k = bisect.bisect_right(starts, cp) - 1
return cls[k] if k >= 0 and cp <= ends[k] else ""
def _is_space(ch):
return ch in "\t\n\r" or ucls(ch) == KEY_SPACE
# ------------------------------------------------------------ identifiers
_DOI_WRAP = re.compile(r"[dD][oO][iI]:")
_DOI_URL = re.compile(r"[hH][tT][tT][pP][sS]?://(?:[dD][xX]\.)?[dD][oO][iI]\.[oO][rR][gG]/")
_BADPCT = re.compile(r"%(?![0-9A-Fa-f]{2})")
_PCT = re.compile(r"%([0-9A-Fa-f]{2})")
_DOI_PREFIX = re.compile(r"10\.[0-9]+(?:\.[0-9]+)*/")
_ARXIV_WRAP = re.compile(r"[aA][rR][xX][iI][vV]:")
_ARXIV_URL = re.compile(r"[hH][tT][tT][pP][sS]?://[aA][rR][xX][iI][vV]\.[oO][rR][gG]/abs/")
_ARXIV_NEW = re.compile(r"([0-9]{2})([0-9]{2})\.([0-9]{4,5})(?:v([1-9][0-9]*))?")
_ARXIV_OLD = re.compile(r"([a-z][a-z-]{1,15})(\.[A-Z]{2})?/([0-9]{2})([0-9]{2})([0-9]{3})(?:v([1-9][0-9]*))?")
_ARXIV_DOI_SUFFIX = re.compile(r"[aA][rR][xX][iI][vV]\.")
_ISBN10 = re.compile(r"[0-9]{9}[0-9Xx]")
_ISBN13 = re.compile(r"97[89][0-9]{10}")
_ISSN = re.compile(r"([0-9]{4})-([0-9]{3})([0-9Xx])")
_PMID = re.compile(r"[1-9][0-9]*")
_PMCID = re.compile(r"[pP][mM][cC]([0-9]+)")
_DATE = re.compile(r"([0-9]{4})-([0-9]{2})-([0-9]{2})")
def _pct_decode_once(s):
"""One percent-decoding of a URL form; None when a "%" is not followed
by two hex digits or the bytes are not UTF-8."""
if _BADPCT.search(s) is not None:
return None
raw = bytearray()
i = 0
b = s.encode("utf-8")
while i < len(b):
if b[i:i + 1] == b"%":
raw += bytes([int(b[i + 1:i + 3].decode("ascii"), 16)])
i += 3
else:
raw.append(b[i])
i += 1
try:
return bytes(raw).decode("utf-8")
except UnicodeError:
return None
def _graphic(s):
"""Every code point graphic: no white space, control or format (RD-5,
DOI Handbook 2025 2.3), none disallowed by the zone's text safety."""
if Z._first_bad(s) is not None:
return False
for ch in s:
c = ucls(ch)
if ch in "\t\n\r" or (c != "" and c in "ZCF"):
return False
return True
def _doi_split(s):
"""(prefix, suffix) of a DOI-shaped string, or None."""
if type(s) is not str or s == "" or not _graphic(s):
return None
m = _DOI_PREFIX.match(s)
if m is None or m.end() == len(s):
return None
return s[:m.end() - 1], s[m.end():]
def doi_shape(text):
return _doi_split(text) is not None
def doi_norm(text):
if type(text) is not str:
raise ValueError("doi")
s = text
if _DOI_WRAP.match(s) is not None:
s = s[4:]
else:
m = _DOI_URL.match(s)
if m is not None:
s = _pct_decode_once(s[m.end():])
if s is None:
raise ValueError("doi")
s = Z._text(s)
parts = _doi_split(s)
if parts is None:
raise ValueError("doi")
prefix, suffix = parts
if prefix == ARXIV_DOI_PREFIX:
m = _ARXIV_DOI_SUFFIX.match(suffix)
if m is None:
raise ValueError("doi")
try:
a = arxiv_parse(suffix[m.end():])
except ValueError:
raise ValueError("doi")
return {"doi": s, "kind": "ARXIV", "id": a["id"]}
return {"doi": s, "kind": "DOI", "id": _fold(s)}
def render_doi(d):
return "doi|%s|%s|%s" % (d["kind"], _esc(d["doi"]), _esc(d["id"]))
def render_shape(b):
return "shape|" + ("true" if b else "false")
def arxiv_parse(text):
if type(text) is not str:
raise ValueError("arxiv")
s = text
if _ARXIV_WRAP.match(s) is not None:
s = s[6:]
else:
m = _ARXIV_URL.match(s)
if m is not None:
s = s[m.end():]
m = _ARXIV_NEW.fullmatch(s)
if m is not None:
yy, mm, nn, v = m.group(1), m.group(2), m.group(3), m.group(4)
yymm = int(yy + mm)
if not ("01" <= mm <= "12"):
raise ValueError("arxiv")
if len(nn) == 4 and not (704 <= yymm <= 1412):
raise ValueError("arxiv")
if len(nn) == 5 and yymm < 1501:
raise ValueError("arxiv")
key = yy + mm + "." + nn
return {"id": s, "version": int(v) if v is not None else 0,
"key": key, "scheme": "NEW"}
m = _ARXIV_OLD.fullmatch(s)
if m is not None:
arch, sc, yy, mm, nn, v = m.groups()
if yy not in ARXIV_OLD_YY or not ("01" <= mm <= "12"):
raise ValueError("arxiv")
if yy == "07" and mm > "03":
raise ValueError("arxiv")
key = arch + (sc or "") + "/" + yy + mm + nn
return {"id": s, "version": int(v) if v is not None else 0,
"key": key, "scheme": "OLD"}
raise ValueError("arxiv")
def render_arxiv(d):
return "arxiv|%s|%d|%s|%s" % (d["id"], d["version"], d["key"],
d["scheme"])
def isbn_check(text):
if type(text) is not str:
raise ValueError("isbn")
s = text.replace("-", "").replace(" ", "")
if _ISBN10.fullmatch(s) is not None:
total = 0
for i in range(9):
total += (10 - i) * int(s[i])
total += 10 if s[9] in "Xx" else int(s[9])
if total % 11 != 0:
raise ValueError("isbn")
nine = "978" + s[:9]
return {"isbn13": nine + _isbn13_check(nine), "given": "10"}
if _ISBN13.fullmatch(s) is not None:
if _isbn13_check(s[:12]) != s[12]:
raise ValueError("isbn")
return {"isbn13": s, "given": "13"}
raise ValueError("isbn")
def _isbn13_check(twelve):
total = 0
for i in range(12):
total += (1 if i % 2 == 0 else 3) * int(twelve[i])
return str((10 - total % 10) % 10)
def render_isbn(d):
return "isbn|%s|%s" % (d["isbn13"], d["given"])
def issn_check(text):
if type(text) is not str:
raise ValueError("issn")
m = _ISSN.fullmatch(text)
if m is None:
raise ValueError("issn")
seven = m.group(1) + m.group(2)
total = 0
for i in range(7):
total += (8 - i) * int(seven[i])
r = total % 11
c = 0 if r == 0 else 11 - r
want = "X" if c == 10 else str(c)
if m.group(3).upper() != want:
raise ValueError("issn")
return m.group(1) + "-" + m.group(2) + want
def render_issn(s):
return "issn|" + s
def pmid_check(text):
if type(text) is not str or _PMID.fullmatch(text) is None:
raise ValueError("pmid")
return text
def render_pmid(s):
return "pmid|" + s
def pmcid_check(text):
if type(text) is not str:
raise ValueError("pmcid")
m = _PMCID.fullmatch(text)
if m is None:
raise ValueError("pmcid")
return "PMC" + m.group(1)
def render_pmcid(s):
return "pmcid|" + s
# ---------------------------------------------------------------- key form
_MARKUP = re.compile(r"<[A-Za-z/][^>]*>")
_ENTITY = re.compile(r"&(#[0-9]+|#[xX][0-9A-Fa-f]+|[A-Za-z]+);")
def _entity(m):
e = m.group(1)
if e[:1] != "#":
return ENTITIES.get(e, m.group(0))
if e[1:2] in ("x", "X"):
digits = e[2:]
if len(digits) > 6:
return m.group(0)
cp = int(digits, 16)
else:
digits = e[1:]
if len(digits) > 7:
return m.group(0)
cp = int(digits, 10)
if cp > 0x10FFFF or 0xD800 <= cp <= 0xDFFF:
return m.group(0)
return chr(cp)
def _strip_marker(s):
i = 0
while i < len(s) and _is_space(s[i]):
i += 1
rest = s[i:]
up = _lower_ascii(rest)
for mk in MARKERS:
t = _lower_ascii(mk) + ":"
if up[:len(t)] == t:
return rest[len(t):]
return s
def biblio_key_form(text, kind):
if kind not in ("title", "name") or type(text) is not str:
raise ValueError("key")
s = text
for ch in s:
if 0xD800 <= ord(ch) <= 0xDFFF:
raise ValueError("key")
if kind == "title":
s = _strip_marker(s)
s = _MARKUP.sub("", s)
s = _ENTITY.sub(_entity, s)
s = unicodedata.normalize("NFD", s)
out = []
for ch in s:
if _is_space(ch):
out.append(" ")
elif ucls(ch) != "" and ucls(ch) in KEY_REMOVE:
continue
else:
out.append(ch)
s = _fold("".join(out))
s = unicodedata.normalize("NFC", s)
return " ".join(w for w in s.split(" ") if w != "")
def render_key(s):
return "key|" + _esc(s)
# ------------------------------------------------------------ CSL item
def _read_names(role, v):
"""The names of one role, NFC, each an object in NAME_KEYS order."""
if type(v) is not list:
raise ValueError("type:" + role)
out = []
for nm in v:
if not _keys_within(nm, NAME_KEYS):
if type(nm) is not dict:
raise ValueError("type:" + role)
raise ValueError("subset:" + role)
o = {}
for k in NAME_KEYS:
if k in nm:
if type(nm[k]) is not str or nm[k] == "":
raise ValueError("type:" + role)
if Z._first_bad(nm[k]) is not None:
raise ValueError("text:" + role)
o[k] = Z._text(nm[k])
if "family" not in o and "literal" not in o:
raise ValueError("type:" + role)
out.append(o)
return out
def _read_date(field, v):
if not _keys_within(v, DATE_KEYS):
if type(v) is not dict:
raise ValueError("type:" + field)
raise ValueError("subset:" + field)
dp = v.get("date-parts")
if type(dp) is not list or not 1 <= len(dp) <= 2:
raise ValueError("type:" + field)
parts = []
for p in dp:
if type(p) is not list or not 1 <= len(p) <= 3:
raise ValueError("type:" + field)
if not _is_int(p[0], YEAR_MIN, YEAR_MAX):
raise ValueError("type:" + field)
if len(p) > 1 and not _is_int(p[1], 1, 12):
raise ValueError("type:" + field)
if len(p) > 2 and not _is_int(p[2], 1, 31):
raise ValueError("type:" + field)
parts.append(list(p))
o = {"date-parts": parts}
if "circa" in v:
if type(v["circa"]) is not bool:
raise ValueError("type:" + field)
o["circa"] = v["circa"]
return o
def _read_item(item):
"""BC1-BC3: the item in CSL_KEYS order, strings NFC; raises."""
if type(item) is not dict:
raise ValueError("type:item")
for k in item:
if k not in CSL_KEYS:
raise ValueError("subset:" + (k if type(k) is str else "?"))
o = {}
for k in CSL_KEYS:
if k not in item:
continue
v = item[k]
if k in NAME_ROLES:
o[k] = _read_names(k, v)
elif k in DATE_FIELDS:
o[k] = _read_date(k, v)
else:
if type(v) is not str or v == "":
raise ValueError("type:" + k)
if Z._first_bad(v) is not None:
raise ValueError("text:" + k)
o[k] = Z._text(v)
if k == "type" and o[k] not in CSL_TYPES:
raise ValueError("type:type")
return o
def _require(o):
"""BC4."""
if "type" not in o:
raise ValueError("required:type")
if "title" not in o:
raise ValueError("required:title")
if "issued" not in o:
raise ValueError("required:issued")
if not o.get("author") and not o.get("editor"):
raise ValueError("required:author")
def _caps_ok(o, caps):
"""BC5."""
for k in o:
v = o[k]
if k in NAME_ROLES:
if len(v) > caps["names"]:
raise ValueError("cap:" + k)
for nm in v:
for p in nm:
if len(nm[p]) > caps["name"]:
raise ValueError("cap:" + k)
elif k in DATE_FIELDS:
continue
else:
lim = (caps["title"] if k in CAP_TITLE else
caps["url"] if k in CAP_URL else caps["text"])
if len(v) > lim:
raise ValueError("cap:" + k)
def _read_identifiers(ids):
"""BC6: the normalized identifiers, keys in IDENT_KINDS order."""
if not _keys_within(ids, IDENT_KINDS):
raise ValueError("identifiers")
o = {}
for k in IDENT_KINDS:
if k not in ids:
continue
v = ids[k]
try:
if k == "DOI":
d = doi_norm(v)
if d["kind"] != "DOI":
raise ValueError("doi")
o[k] = d["doi"]
elif k == "ARXIV":
o[k] = arxiv_parse(v)["id"]
elif k == "ISBN":
o[k] = isbn_check(v)["isbn13"]
elif k == "ISSN":
o[k] = issn_check(v)
elif k == "PMID":
o[k] = pmid_check(v)
else:
o[k] = pmcid_check(v)
except ValueError:
raise ValueError("identifier:" + k)
return o
def biblio_dupe_key(identifiers):
ids = _read_identifiers(identifiers)
return _dupe_of(ids)
def _dupe_of(ids):
if "DOI" in ids:
return {"kind": "DOI", "key": _fold(ids["DOI"])}
if "ARXIV" in ids:
return {"kind": "ARXIV", "key": arxiv_parse(ids["ARXIV"])["key"]}
if "ISBN" in ids:
return {"kind": "ISBN", "key": ids["ISBN"]}
return {"kind": "NONE", "key": ""}
def render_dupe(d):
return "dupe|%s|%s" % (d["kind"], _esc(d["key"]))
def _mismatch(o, ids):
"""BC7."""
if "DOI" in o:
try:
d = doi_norm(o["DOI"])
except ValueError:
raise ValueError("mismatch:DOI")
if d["kind"] == "ARXIV":
if "DOI" in ids or "ARXIV" not in ids or \
arxiv_parse(ids["ARXIV"])["key"] != arxiv_parse(d["id"])["key"]:
raise ValueError("mismatch:DOI")
elif "DOI" not in ids or _fold(ids["DOI"]) != d["id"]:
raise ValueError("mismatch:DOI")
if "ISBN" in o:
try:
v = isbn_check(o["ISBN"])["isbn13"]
except ValueError:
raise ValueError("mismatch:ISBN")
if ids.get("ISBN") != v:
raise ValueError("mismatch:ISBN")
for k, fn in (("ISSN", issn_check), ("PMID", pmid_check),
("PMCID", pmcid_check)):
if k in o:
try:
v = fn(o[k])
except ValueError:
raise ValueError("mismatch:" + k)
if ids.get(k) != v:
raise ValueError("mismatch:" + k)
def _read_caps(caps):
if not _keys_within(caps, ("entry_max", "title", "text", "name", "url",
"names")) or len(caps) != 6:
raise ValueError("caps")
for k in caps:
if not _is_int(caps[k], 1, 1000000000):
raise ValueError("caps")
return caps
def biblio_check(entry, existing_keys, caps):
if not _keys_within(entry, ("item", "identifiers", "supersedes")) or \
len(entry) != 3:
raise ValueError("entry")
sup = entry["supersedes"]
if sup is not None and not _is_text(sup):
raise ValueError("entry")
if type(existing_keys) is not list:
raise ValueError("existing")
for e in existing_keys:
if (type(e) is not list or len(e) != 3 or e[0] not in ("DOI", "ARXIV", "ISBN")
or type(e[1]) is not str or e[1] == "" or not _is_text(e[2])):
raise ValueError("existing")
caps = _read_caps(caps)
o = _read_item(entry["item"])
_require(o)
_caps_ok(o, caps)
ids = _read_identifiers(entry["identifiers"])
_mismatch(o, ids)
dupe = _dupe_of(ids)
if dupe["kind"] != "NONE":
for e in existing_keys:
if e[0] == dupe["kind"] and e[1] == dupe["key"] and sup != e[2]:
raise ValueError("duplicate:" + e[2])
if len(_canon_json(o).encode("utf-8")) > caps["entry_max"]:
raise ValueError("size")
return {"status": "OK", "identifiers": ids, "dupe": dupe}
def render_check(d):
out = ["check|OK|%s|%s" % (d["dupe"]["kind"], _esc(d["dupe"]["key"]))]
for k in IDENT_KINDS:
if k in d["identifiers"]:
out.append("id|%s|%s" % (k, _esc(d["identifiers"][k])))
return "\n".join(out)
# --------------------------------------------------------- canonical form
def _json_str(s):
out = ['"']
for ch in s:
o = ord(ch)
if ch == '"' or ch == "\\":
out.append("\\" + ch)
elif o < 0x20:
out.append("\\u%04x" % o)
else:
out.append(ch)
out.append('"')
return "".join(out)
def _json_value(v):
if type(v) is bool:
return "true" if v else "false"
if type(v) is int:
return str(v)
if type(v) is str:
return _json_str(v)
if type(v) is list:
return "[" + ",".join(_json_value(x) for x in v) + "]"
return "{" + ",".join(_json_str(k) + ":" + _json_value(v[k]) for k in v) + "}"
def _canon_json(o):
"""The read item (CSL_KEYS order, names and dates already ordered)."""
return _json_value(o)
def _canon_parts(bib_id, proposed_by, landed_by, landed, supersedes,
identifiers, item):
for nm, v in (("bib_id", bib_id), ("proposed_by", proposed_by),
("landed_by", landed_by), ("landed", landed)):
if not _is_text(v):
raise ValueError(nm)
if supersedes is not None and not _is_text(supersedes):
raise ValueError("supersedes")
ids = _read_identifiers(identifiers)
o = _read_item(item)
if "id" in o and o["id"] != bib_id:
raise ValueError("id")
o2 = {}
for k in CSL_KEYS:
if k == "id":
o2["id"] = bib_id
elif k in o:
o2[k] = o[k]
text = _canon_json(o2)
fields = [BIB_TAG, BIB_FORM, bib_id, proposed_by, landed_by, landed,
supersedes if supersedes is not None else ""]
for k in IDENT_KINDS:
fields += [k, ids.get(k, "")]
fields.append(text)
return text, fields
def biblio_preimage(bib_id, proposed_by, landed_by, landed, supersedes,
identifiers, item):
text, fields = _canon_parts(bib_id, proposed_by, landed_by, landed,
supersedes, identifiers, item)
return G.preimage_fields(fields).decode("utf-8")
def biblio_canon(bib_id, proposed_by, landed_by, landed, supersedes,
identifiers, item):
text, fields = _canon_parts(bib_id, proposed_by, landed_by, landed,
supersedes, identifiers, item)
return {"text": text, "digest": G.digest_fields(fields)}
def render_canon(d):
return "canon|" + d["digest"] + "\n" + _esc(d["text"])
def render_preimage(s):
return "preimage|" + _esc(s)
# ------------------------------------------------------------ registry URL
_AUTHORITY = re.compile(r"https://[A-Za-z0-9](?:[A-Za-z0-9.-]*[A-Za-z0-9])?(?::[0-9]+)?/")
_TEMPLATE = re.compile(r"https://[A-Za-z0-9](?:[A-Za-z0-9.-]*[A-Za-z0-9])?(?::[0-9]+)?/[A-Za-z0-9\-._~:/?#\[\]@!$&'()*+,;=%]*")
def _pct_encode(s):
out = []
for b in s.encode("utf-8"):
c = chr(b)
if b < 0x80 and c in URL_SAFE:
out.append(c)
else:
out.append("%%%02X" % b)
return "".join(out)
def registry_url(declaration, identifier):
if not _keys_within(declaration, ("kind", "template", "name")) or \
"kind" not in declaration or "template" not in declaration:
raise ValueError("declaration")
kind, tmpl = declaration["kind"], declaration["template"]
if kind not in DECL_KINDS or ("name" in declaration
and not _is_text(declaration["name"])):
raise ValueError("declaration")
if type(tmpl) is not str or tmpl.count(URL_SLOT) != 1:
raise ValueError("template")
plain = tmpl.replace(URL_SLOT, "x")
if "{" in plain or "}" in plain or _TEMPLATE.fullmatch(plain) is None \
or "#" in plain:
raise ValueError("template")
m = _AUTHORITY.match(tmpl)
if m is None or tmpl.find(URL_SLOT) < m.end():
raise ValueError("template") # the slot is in the path or the query, never the host
try:
if kind[:4] == "DOI-":
d = doi_norm(identifier)
if d["kind"] != "DOI":
raise ValueError("identifier")
v = d["doi"]
elif kind == "ARXIV":
v = arxiv_parse(identifier)["id"]
elif kind == "PMID":
v = pmid_check(identifier)
else:
v = pmcid_check(identifier)
except ValueError:
raise ValueError("identifier")
url = tmpl.replace(URL_SLOT, _pct_encode(v))
path = url[8:].split("?", 1)[0]
for seg in path.split("/"):
if seg == "." or seg == "..":
raise ValueError("identifier")
return url
def render_url(s):
return "url|" + _esc(s)
# ---------------------------------------------------------------- compare
def _read_notice(n):
if not _keys_within(n, NOTICE_KEYS) or len(n) != 4:
raise ValueError("records")
if not _is_text(n["type"]) or not _is_text(n["source"]):
raise ValueError("records")
ident = n["identifier"]
if ident is not None:
if not _is_text(ident) or ident == "-":
raise ValueError("records")
for ch in ident:
if _is_space(ch):
raise ValueError("records")
if n["date"] is not None and _calendar(n["date"]) is None:
raise ValueError("records")
return {"type": biblio_key_form(n["type"], "name"),
"identifier": ident, "date": n["date"], "source": n["source"]}
def _opt_text(v):
return v is None or _is_text(v)
def _read_fields(f):
if not _keys_within(f, FIELD_KEYS) or len(f) != 10:
raise ValueError("records")
if f["year"] is not None and not _is_int(f["year"], YEAR_MIN, YEAR_MAX):
raise ValueError("records")
for k in ("title", "subtitle", "family", "doi", "arxiv_doi", "arxiv_id"):
if not _opt_text(f[k]):
raise ValueError("records")
nf = f["name_form"]
if (nf is None) != (f["family"] is None) or (nf is not None and nf not in NAME_FORMS):
raise ValueError("records")
for k in ("issn", "isbn"):
if type(f[k]) is not list:
raise ValueError("records")
for x in f[k]:
if not _is_text(x):
raise ValueError("records")
return f
def _read_record(r, ids):
"""One REGISTRY RECORD, validated against the entry's identifiers."""
if type(r) is not dict or r.get("kind") not in PRIMARY_ORDER:
raise ValueError("records")
kind = r["kind"]
keys = ("kind", "identifier", "registry", "status", "fields", "notices")
if kind == "DOI":
keys = keys + ("ra",)
if not _keys_within(r, keys) or len(r) != len(keys):
raise ValueError("records")
if kind not in ids:
raise ValueError("records")
try:
if kind == "DOI":
d = doi_norm(r["identifier"])
same = d["kind"] == "DOI" and d["id"] == _fold(ids["DOI"])
elif kind == "ARXIV":
same = arxiv_parse(r["identifier"])["id"] == ids["ARXIV"]
elif kind == "PMID":
same = pmid_check(r["identifier"]) == ids["PMID"]
else:
same = pmcid_check(r["identifier"]) == ids["PMCID"]
except ValueError:
same = False
if not same:
raise ValueError("records")
reg, st = r["registry"], r["status"]
if kind == "DOI":
ra = r["ra"]
if not _keys_within(ra, ("status", "ra")) or len(ra) != 2 or \
ra["status"] not in LOOKUP_STATUS:
raise ValueError("records")
if ra["status"] == "FOUND":
if not _is_text(ra["ra"]):
raise ValueError("records")
want = RA_REGISTRY.get(ra["ra"])
if want is not None:
if reg != want:
raise ValueError("records")
elif reg not in (None, "generic"):
raise ValueError("records")
else:
if ra["ra"] is not None or reg is not None:
raise ValueError("records")
else:
if reg != ("arxiv" if kind == "ARXIV" else "pubmed"):
raise ValueError("records")
if reg is None:
if st is not None:
raise ValueError("records")
elif st not in LOOKUP_STATUS:
raise ValueError("records")
if type(r["notices"]) is not list:
raise ValueError("records")
if st == "FOUND":
fields = _read_fields(r["fields"])
notices = [_read_notice(n) for n in r["notices"]]
else:
if r["fields"] is not None or r["notices"] != []:
raise ValueError("records")
fields, notices = None, []
return {"kind": kind, "ra": r.get("ra"), "registry": reg, "status": st,
"fields": fields, "notices": notices}
def _calendar(s):
if type(s) is not str:
return None
m = _DATE.fullmatch(s)
if m is None:
return None
y, mo, d = int(m.group(1)), int(m.group(2)), int(m.group(3))
leap = y % 4 == 0 and (y % 100 != 0 or y % 400 == 0)
if mo < 1 or mo > 12:
return None
last = [31, 29 if leap else 28, 31, 30, 31, 30, 31, 31, 30, 31, 30,
31][mo - 1]
if d < 1 or d > last:
return None
return s
def _entry_family(o):
names = o.get("author") or o.get("editor")
nm = names[0]
return biblio_key_form(nm["family"] if "family" in nm else nm["literal"],
"name")
def _family_equal(entry_key, rec_family, form):
rk = biblio_key_form(rec_family, "name")
if entry_key == "" or rk == "":
return entry_key == rk
if form == "FAMILY":
return rk == entry_key
if form == "GIVEN FAMILY":
return rk == entry_key or rk.endswith(" " + entry_key)
return rk == entry_key or rk.startswith(entry_key + " ")
def _compare_fields(o, ids, rec, primary_kind):
"""The differing field names of a FOUND record (RD-8)."""
f = rec["fields"]
kind, reg = rec["kind"], rec["registry"]
diff = []
e_title = biblio_key_form(o["title"], "title")
e_family = _entry_family(o)
e_year = o["issued"]["date-parts"][0][0]
def cmp_title():
if f["title"] is None:
return False
t = f["title"] if f["subtitle"] is None else f["title"] + " " + f["subtitle"]
return biblio_key_form(t, "title") == e_title
def cmp_family():
if f["family"] is None:
return False
return _family_equal(e_family, f["family"], f["name_form"])
if kind == "ARXIV" and primary_kind == "DOI":
if f["arxiv_doi"] is None:
if cmp_title() and cmp_family():
return []
return None # FOUND UNCOMPARED
try:
d = doi_norm(f["arxiv_doi"])
ok = d["kind"] == "DOI" and d["id"] == _fold(ids["DOI"])
except ValueError:
ok = False
if not ok:
diff.append("arxiv_doi")
if not cmp_family():
diff.append("family")
return diff
if f["year"] != e_year:
diff.append("year")
if not cmp_title():
diff.append("title")
if not cmp_family():
diff.append("family")
if "DOI" in ids and f["doi"] is not None:
try:
d = doi_norm(f["doi"])
ok = d["kind"] == "DOI" and d["id"] == _fold(ids["DOI"])
except ValueError:
ok = False
if not ok:
diff.append("doi")
if reg == "datacite" and "ARXIV" in ids and f["arxiv_id"] is not None:
try:
ok = arxiv_parse(f["arxiv_id"])["key"] == arxiv_parse(ids["ARXIV"])["key"]
except ValueError:
ok = False
if not ok:
diff.append("arxiv")
return diff
def _in_list(kind, value, lst):
"""CONFIRMED, FIELDS DIFFER or NO LOOKUP for an ISSN or ISBN against
the primary record's list."""
if not lst:
return "NO LOOKUP"
for x in lst:
try:
v = issn_check(x) if kind == "ISSN" else isbn_check(x)["isbn13"]
except ValueError:
continue
if v == value:
return "CONFIRMED"
return "FIELDS DIFFER"
def biblio_compare(entry, records):
if not _keys_within(entry, ("item", "identifiers")) or len(entry) != 2:
raise ValueError("entry")
o = _read_item(entry["item"])
_require(o)
ids = _read_identifiers(entry["identifiers"])
if type(records) is not list:
raise ValueError("records")
recs = {}
for r in records:
rr = _read_record(r, ids)
if rr["kind"] in recs:
raise ValueError("records")
recs[rr["kind"]] = rr
primary = None
for k in PRIMARY_ORDER:
if k in ids:
primary = k
break
results = {}
for k in PRIMARY_ORDER:
if k not in ids:
continue
r = recs.get(k)
if r is None:
results[k] = ("LOOKUP FAILED", [])
continue
if k == "DOI":
if r["ra"]["status"] == "NOT FOUND":
results[k] = ("NOT FOUND", [])
continue
if r["ra"]["status"] == "FAILED":
results[k] = ("LOOKUP FAILED", [])
continue
if r["registry"] is None:
results[k] = ("FOUND UNCOMPARED", [])
continue
if r["status"] != "FOUND":
results[k] = ("LOOKUP FAILED", [])
continue
else:
if r["status"] == "NOT FOUND":
results[k] = ("NOT FOUND", [])
continue
if r["status"] == "FAILED":
results[k] = ("LOOKUP FAILED", [])
continue
diff = _compare_fields(o, ids, r, primary)
if diff is None:
results[k] = ("FOUND UNCOMPARED", [])
elif diff:
results[k] = ("FIELDS DIFFER", diff)
else:
results[k] = ("CONFIRMED", [])
prim = recs.get(primary) if primary is not None else None
pf = prim["fields"] if prim is not None and prim["status"] == "FOUND" else None
for k in ("ISBN", "ISSN"):
if k not in ids:
continue
if primary is None or pf is None:
results[k] = ("NO LOOKUP", [])
else:
results[k] = (_in_list(k, ids[k], pf["issn" if k == "ISSN" else "isbn"]), [])
out = []
for k in RESULT_ORDER:
if k in results:
out.append([k, ids[k], results[k][0], results[k][1]])
check = "NO LOOKUP"
present = set(x[2] for x in out)
for v in RESULTS:
if v in present:
check = v
break
notices = []
for k in RESULT_ORDER:
r = recs.get(k)
if r is not None and r["status"] == "FOUND":
notices.extend(r["notices"])
return {"check": check, "results": out, "notices": notices}
def render_check_line(d, date):
if _calendar(date) is None:
raise ValueError("date")
s = "CHECK " + date + " " + d["check"]
for kind, ident, res, _ in d["results"]:
s += " ; " + kind + " " + ident + " " + res
return s
def render_notices_line(d, date):
if _calendar(date) is None:
raise ValueError("date")
if not d["notices"]:
return ""
s = "NOTICES " + date
for n in d["notices"]:
s += " ; " + n["source"] + " " + (n["identifier"] or "-") + " " + \
(n["date"] or "-") + " " + n["type"]
return s
def render_compare(d, date):
out = ["compare|" + d["check"]]
for kind, ident, res, fields in d["results"]:
out.append("%s|%s|%s|%s" % (kind, _esc(ident), res, ",".join(fields)))
for n in d["notices"]:
out.append("notice|%s|%s|%s|%s" % (_esc(n["source"]),
_esc(n["identifier"] or "-"),
n["date"] or "-", _esc(n["type"])))
out.append("line|" + _esc(render_check_line(d, date)))
nl = render_notices_line(d, date)
out.append("line|" + (_esc(nl) if nl != "" else "-"))
return "\n".join(out)
def arxiv_withdrawn(comment):
if comment is None:
return False
if type(comment) is not str:
raise ValueError("comment")
low = _lower_ascii(comment)
i = low.find(WITHDRAWN_TOKEN)
while i >= 0:
j = i + len(WITHDRAWN_TOKEN)
before = low[i - 1] if i > 0 else ""
after = low[j] if j < len(low) else ""
if not _alnum(before) and not _alnum(after):
return True
i = low.find(WITHDRAWN_TOKEN, i + 1)
return False
def _alnum(c):
return c != "" and ("a" <= c <= "z" or "0" <= c <= "9")
def render_withdrawn(b):
return "withdrawn|" + ("true" if b else "false")
# --------------------------------------------------------- literature view
def _text_list(v):
if type(v) is not list:
return None
for x in v:
if not _is_text(x):
return None
return v
def literature_view(records, entries, findings):
if type(entries) is not list:
raise ValueError("entries")
ent = {}
for e in entries:
if not _keys_within(e, ("bib_id", "check", "notices", "primary")) or len(e) != 4:
raise ValueError("entries")
if (not _is_text(e["bib_id"]) or e["check"] not in RESULTS
or not _is_int(e["notices"], 0, 1000000000)
or type(e["primary"]) is not bool or e["bib_id"] in ent):
raise ValueError("entries")
ent[e["bib_id"]] = e
if type(findings) is not list:
raise ValueError("findings")
fk = set()
for f in findings:
if not _keys_within(f, ("bib_id", "result", "agreed")) or len(f) != 3:
raise ValueError("findings")
if (not _is_text(f["bib_id"]) or f["result"] not in FK20_RESULTS
or type(f["agreed"]) is not bool):
raise ValueError("findings")
if f["bib_id"] not in ent:
raise ValueError("literature")
if f["result"] == "NOT FOUND" and f["agreed"]:
fk.add(f["bib_id"])
if type(records) is not list:
raise ValueError("records")
seen = set()
rows = []
for r in records:
if not _keys_within(r, ("id", "status", "literature")) or len(r) != 3:
raise ValueError("records")
if not _is_text(r["id"]) or r["status"] not in STATUS_WORDS or r["id"] in seen:
raise ValueError("records")
lit = _text_list(r["literature"])
if lit is None or len(set(lit)) != len(lit):
raise ValueError("records")
seen.add(r["id"])
for b in lit:
if b not in ent:
raise ValueError("literature")
if r["status"] in NOT_STANDING:
continue
flagged = []
for b in sorted(lit):
e = ent[b]
if e["notices"] > 0:
flagged.append([b, "NOTICES"])
if e["check"] == "NOT FOUND":
flagged.append([b, "NOT FOUND"])
if b in fk:
flagged.append([b, "FK-20 NOT FOUND"])
if flagged:
rows.append([r["id"], flagged])
rows.sort(key=lambda x: x[0])
hd = sorted(b for b in ent if ent[b]["primary"] and ent[b]["check"] == "NOT FOUND" and b in fk)
return {"records": rows, "for_hdecl": hd}
def render_view(d):
out = ["view|%d" % len(d["records"])]
for rid, flagged in d["records"]:
out.append("record|" + _esc(rid) + "|"
+ "|".join(_esc(b) + ":" + reason for b, reason in flagged))
out.append("hdecl|" + ",".join(_esc(b) for b in d["for_hdecl"]))
return "\n".join(out)
# --------------------------------------------------------------- hold list
def biblio_hold_list(entries, account, blocks, checks):
if not _is_text(account):
raise ValueError("account")
bl = _text_list(blocks)
if bl is None:
raise ValueError("blocks")
if type(entries) is not list:
raise ValueError("entries")
if type(checks) is not dict:
raise ValueError("checks")
for k in checks:
if not _is_text(k) or checks[k] not in RESULTS:
raise ValueError("checks")
seen = set()
out = []
bset = set(bl)
for e in entries:
if not _keys_within(e, ("bib_id", "proposed_by", "landed_by", "block_id")) or len(e) != 4:
raise ValueError("entries")
for k in ("bib_id", "proposed_by", "landed_by", "block_id"):
if not _is_text(e[k]):
raise ValueError("entries")
if e["bib_id"] in seen:
raise ValueError("entries")
seen.add(e["bib_id"])
if (e["proposed_by"] == account or e["landed_by"] == account) and e["block_id"] in bset:
if e["bib_id"] not in checks:
raise ValueError("checks")
if checks[e["bib_id"]] != "CONFIRMED":
out.append([e["bib_id"], checks[e["bib_id"]]])
out.sort(key=lambda p: p[0])
return out
def render_holdlist(lst):
return "\n".join(["holdlist|%d" % len(lst)]
+ ["%s|%s" % (_esc(b), c) for b, c in lst])
# ------------------------------------------------------------ corpus case
FNS = ("doi", "shape", "arxiv", "isbn", "issn", "pmid", "pmcid", "key",
"dupe", "check", "canon", "preimage", "url", "compare", "withdrawn",
"view", "holdlist")
def run_case(fn, args):
"""One corpus case: the render, or "ERROR:" and the message."""
if fn not in FNS:
raise ValueError("fn")
try:
if fn == "doi":
return render_doi(doi_norm(args["text"]))
if fn == "shape":
return render_shape(doi_shape(args["text"]))
if fn == "arxiv":
return render_arxiv(arxiv_parse(args["text"]))
if fn == "isbn":
return render_isbn(isbn_check(args["text"]))
if fn == "issn":
return render_issn(issn_check(args["text"]))
if fn == "pmid":
return render_pmid(pmid_check(args["text"]))
if fn == "pmcid":
return render_pmcid(pmcid_check(args["text"]))
if fn == "key":
return render_key(biblio_key_form(args["text"], args["kind"]))
if fn == "dupe":
return render_dupe(biblio_dupe_key(args["identifiers"]))
if fn == "check":
return render_check(biblio_check(args["entry"], args["existing"],
args["caps"]))
if fn == "canon":
return render_canon(biblio_canon(
args["bib_id"], args["proposed_by"], args["landed_by"],
args["landed"], args["supersedes"], args["identifiers"],
args["item"]))
if fn == "preimage":
return render_preimage(biblio_preimage(
args["bib_id"], args["proposed_by"], args["landed_by"],
args["landed"], args["supersedes"], args["identifiers"],
args["item"]))
if fn == "url":
return render_url(registry_url(args["declaration"],
args["identifier"]))
if fn == "compare":
return render_compare(biblio_compare(args["entry"],
args["records"]),
args["date"])
if fn == "withdrawn":
return render_withdrawn(arxiv_withdrawn(args["comment"]))
if fn == "view":
return render_view(literature_view(args["records"],
args["entries"],
args["findings"]))
return render_holdlist(biblio_hold_list(args["entries"],
args["account"],
args["blocks"],
args["checks"]))
except ValueError as ex:
return "ERROR:" + str(ex)
